Repository: dask/dask

Issue:
get_dummies not executing
Dask `get_dummies` "runs" but it doesn't actually execute the get_dummies task. 

```
import pandas as pd
import dask.dataframe as dd

pandasData = pd.DataFrame({'var1': ['a', 'b', 'a'], 'var2': ['b', 'a', 'c'], 'var3': ['c', 'a', 'b']})
pd.get_dummies(pandasData)

daskData = dd.from_pandas(pandasData, npartitions=1)
daskData.head()

daskDataDummies = dd.get_dummies(daskData).compute()
daskDataDummies.head()

daskDataDummies.to_csv('daskDataDummies_out.csv', header=True, index=False)
```

There's no error message it simply doesn't transform the dataframe. 


Discussion:
Looking at the docstring it looks like these need to be categorical:

```
Convert categorical variable into dummy/indicator variables. Data must
have category dtype to infer result's ``columns``
```

Arguably we should err here, but I suspect that this is your problem.
Understood, but if this is trying to be 100% analogous to Pandas' get_dummies shouldn't this be baked into the functionality? 
I'm not sure that what you're proposing is possible without a full scan.
If you'd like to do some experimentation and submit a PR though that would
be great.

On Wed, Nov 23, 2016 at 7:43 PM, Nick Sharkey <notifications@github.com>
wrote:

> Understood, but if this is trying to be 100% analogous to Pandas'
> get_dummies shouldn't this be baked into the functionality?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/1809#issuecomment-262664124>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AASszJSryWCzAOcMvkLKNuQWaS-HDQpuks5rBN3OgaJpZM4K7CyU>
> .
>

I don't have the technical capabilities to do that. If it isn't something you or the community can do then close out the request. 
This can linger until someone with the time comes along.  At the least we should provide a more informative error message.
Adding to that parametrized test has been helpful so far.  Next time I take a look at this I'll probably dump in a bunch more dtypes and such.
Perhaps someone smarter than I can figure out a sampling technique to approximate the full file's dtypes. 
Sampling probably isn't sufficient.  We need to know *all* of the values throughout the file to determine the columns.  

Resolving patch:
diff --git a/dask/dataframe/reshape.py b/dask/dataframe/reshape.py
--- a/dask/dataframe/reshape.py
+++ b/dask/dataframe/reshape.py
@@ -50,14 +50,20 @@ def get_dummies(data, prefix=None, prefix_sep='_', dummy_na=False,
                               columns=columns, sparse=sparse,
                               drop_first=drop_first)
 
+    not_cat_msg = ("`get_dummies` with non-categorical dtypes is not "
+                   "supported. Please use `df.categorize()` beforehand to "
+                   "convert to categorical dtype.")
+
     if isinstance(data, Series) and not is_categorical_dtype(data):
-        raise ValueError('data must have category dtype')
+        raise NotImplementedError(not_cat_msg)
     elif isinstance(data, DataFrame):
         if columns is None:
+            if (data.dtypes == 'object').any():
+                raise NotImplementedError(not_cat_msg)
             columns = data._meta.select_dtypes(include=['category']).columns
         else:
             if not all(is_categorical_dtype(data[c]) for c in columns):
-                raise ValueError('target columns must have category dtype')
+                raise NotImplementedError(not_cat_msg)
 
     if sparse:
         raise NotImplementedError('sparse=True is not supported')


Test patch:
diff --git a/dask/dataframe/tests/test_reshape.py b/dask/dataframe/tests/test_reshape.py
--- a/dask/dataframe/tests/test_reshape.py
+++ b/dask/dataframe/tests/test_reshape.py
@@ -28,24 +28,21 @@ def test_get_dummies_object():
     df = pd.DataFrame({'a': pd.Categorical([1, 2, 3, 4, 4, 3, 2, 1]),
                        'b': list('abcdabcd'),
                        'c': pd.Categorical(list('abcdabcd'))})
-    # exclude object columns
-    exp = pd.get_dummies(df, columns=['a', 'c'])
-
     ddf = dd.from_pandas(df, 2)
-    res = dd.get_dummies(ddf)
+
+    # Explicitly exclude object columns
+    exp = pd.get_dummies(df, columns=['a', 'c'])
+    res = dd.get_dummies(ddf, columns=['a', 'c'])
     assert_eq(res, exp)
     tm.assert_index_equal(res.columns, exp.columns)
 
-    exp = pd.get_dummies(df, columns=['a'])
+    with pytest.raises(NotImplementedError):
+        dd.get_dummies(ddf)
 
-    ddf = dd.from_pandas(df, 2)
-    res = dd.get_dummies(ddf, columns=['a'])
-    assert_eq(res, exp)
-    tm.assert_index_equal(res.columns, exp.columns)
+    with pytest.raises(NotImplementedError):
+        dd.get_dummies(ddf.b)
 
-    # cannot target object columns
-    msg = 'target columns must have category dtype'
-    with tm.assertRaisesRegexp(ValueError, msg):
+    with pytest.raises(NotImplementedError):
         dd.get_dummies(ddf, columns=['b'])
 
 
@@ -88,8 +85,7 @@ def test_get_dummies_kwargs():
 
 
 def test_get_dummies_errors():
-    msg = 'data must have category dtype'
-    with tm.assertRaisesRegexp(ValueError, msg):
+    with pytest.raises(NotImplementedError):
         # not Categorical
         s = pd.Series([1, 1, 1, 2, 2, 1, 3, 4])
         ds = dd.from_pandas(s, 2)