Repository: dask/dask

Issue:
Categorical types in (empty) meta df's cause empty categories, which cause confusing downstream errors
Constructing a ddf from a meta (i.e. empty df) with categorical types results in a ddf with a "known" categorical that has 0 values, which then causes weird behaviors when you try to use it. Example:

```py
# Schema + rows, like you might get from file storage
>>> schema = {
...     'i': 'int',
...     'cats': 'category',
...     'strs': 'str',
... }
... rows = [
...     {'i': 1, 'cats': 'foo', 'strs': 'x'},
...     {'i': 2, 'cats': 'bar', 'strs': 'y'},
...     {'i': 3, 'cats': 'foo', 'strs': 'z'},
... ]

# Parse the file lines using a bag so we can convert to a ddf
>>> bag = dask.bag.from_sequence(rows)

# Make the ddf meta from the stored schema
>>> meta = pd.DataFrame({name: pd.Series(data=[], dtype=dtype) for name, dtype in schema.items()})

# Convert bag to ddf and then try to use the ddf, e.g. write to parquet:

# .to_dataframe without meta: no categorical types, and to_parquet works fine
>>> ddf = bag.to_dataframe()
>>> ddf._meta.cats
Series([], Name: cats, dtype: object)
>>> ddf.to_parquet('/tmp/foo')

# .to_dataframe with meta: we have a categorical type with 0 categories (!), and to_parquet fails
>>> ddf = bag.to_dataframe(columns=meta)
>>> ddf._meta.cats
Series([], Name: cats, dtype: category
Categories (0, float64): [])
>>> ddf.to_parquet('/tmp/foo')
ValueError: could not convert string to float: 'foo'

# .to_dataframe without meta, and then make the categorical type "unknown": to_parquet works fine again
#   - Also works if we .to_dataframe(columns=meta)
>>> ddf = bag.to_dataframe().astype({'cats': 'category'})
>>> ddf._meta.cats
Series([], Name: cats, dtype: category
Categories (1, object): [__UNKNOWN_CATEGORIES__])
>>> ddf.to_parquet('/tmp/foo')

# .to_dataframe without meta, and then .categorize to make the categorical types "known": to_parquet works here too
#   - However, if we .to_dataframe(columns=meta) then .categorize doesn't suffice to get us out of 0-categories territory
>>> ddf = bag.to_dataframe().categorize(['cats'])
>>> ddf._meta.cats
Series([], Name: cats, dtype: category
Categories (2, object): [foo, bar])
>>> ddf.to_parquet('/tmp/foo')
```

A few options I can imagine here:
1. Lightweight: fail instead of constructing a "known" category with 0 values
2. Lightweight: coerce "known" categories with 0 values into "unknown" categories
3. Heavyweight: rethink something higher level

(1) would have sufficed to avoid my confusion, and I'm not sure (2) would be any more helpful than (1), and maybe just more surprising.

# Versions
```sh
$ python --version
Python 3.6.0

$ cat requirements.txt | egrep 'dask|distributed|fastparquet'
git+https://github.com/dask/dask.git@d0aa50a
git+https://github.com/dask/distributed.git@312a41a
git+https://github.com/dask/fastparquet.git@0.1.0
```


Discussion:
Thanks for the report. I think it would be OK for

```python
>>> ddf = bag.to_dataframe(columns=meta)
```

to treat zero categories as unknown categories. If you *do* require zero categories for whatever reason, you can do that with a `.cat.as_unknown`, but treating zero categories as unknown seems like the better default.
Fastparquet is not able to write a categorical column without any categories (even if it has no rows or only null values), and I don't know if the parquet spec in general allows this. I think the specific error, though, is because `pd.Series([], dtype='category').cat.categories.dtype -> float64` and does not match the data.
 
The problem should be fixed before it gets to the parquet side, so I would go with @TomAugspurger 's comment above.
> treating zero categories as unknown seems like the better default.

Yeah, on second thought I do agree that (2) is a better default behavior than (1).

> The problem should be fixed before it gets to the parquet side, so I would go with @TomAugspurger 's comment above.

Yep, agreed. Zero categories are confusing in dask alone, even before you hit the parquet write path. I was using parquet writes here just as an example of where zero categories cause trouble.

Resolving patch:
diff --git a/dask/bag/core.py b/dask/bag/core.py
--- a/dask/bag/core.py
+++ b/dask/bag/core.py
@@ -1119,7 +1119,7 @@ def groupby(self, grouper, method=None, npartitions=None, blocksize=2**20,
             msg = "Shuffle method must be 'disk' or 'tasks'"
             raise NotImplementedError(msg)
 
-    def to_dataframe(self, columns=None):
+    def to_dataframe(self, meta=None, columns=None):
         """ Create Dask Dataframe from a Dask Bag
 
         Bag should contain tuples, dict records, or scalars.
@@ -1129,15 +1129,24 @@ def to_dataframe(self, columns=None):
 
         Parameters
         ----------
-        columns : pandas.DataFrame or list, optional
-            If a ``pandas.DataFrame``, it should mirror the column names and
-            dtypes of the output dataframe. If a list, it provides the desired
-            column names. If not provided or a list, a single element from
-            the first partition will be computed, triggering a potentially
-            expensive call to ``compute``. Providing a list is only useful for
-            selecting subset of columns, to avoid an internal compute call you
-            must provide a ``pandas.DataFrame`` as dask requires dtype knowledge
-            ahead of time.
+        meta : pd.DataFrame, dict, iterable, optional
+            An empty ``pd.DataFrame`` that matches the dtypes and column names
+            of the output. This metadata is necessary for many algorithms in
+            dask dataframe to work.  For ease of use, some alternative inputs
+            are also available. Instead of a ``DataFrame``, a ``dict`` of
+            ``{name: dtype}`` or iterable of ``(name, dtype)`` can be provided.
+            If not provided or a list, a single element from the first
+            partition will be computed, triggering a potentially expensive call
+            to ``compute``. This may lead to unexpected results, so providing
+            ``meta`` is recommended. For more information, see
+            ``dask.dataframe.utils.make_meta``.
+        columns : sequence, optional
+            Column names to use. If the passed data do not have names
+            associated with them, this argument provides names for the columns.
+            Otherwise this argument indicates the order of the columns in the
+            result (any names not found in the data will become all-NA
+            columns).  Note that if ``meta`` is provided, column names will be
+            taken from there and this parameter is invalid.
 
         Examples
         --------
@@ -1156,11 +1165,18 @@ def to_dataframe(self, columns=None):
         """
         import pandas as pd
         import dask.dataframe as dd
-        if isinstance(columns, pd.DataFrame):
-            meta = columns
+        if meta is None:
+            if isinstance(columns, pd.DataFrame):
+                warn("Passing metadata to `columns` is deprecated. Please "
+                     "use the `meta` keyword instead.")
+                meta = columns
+            else:
+                head = self.take(1)[0]
+                meta = pd.DataFrame([head], columns=columns)
+        elif columns is not None:
+            raise ValueError("Can't specify both `meta` and `columns`")
         else:
-            head = self.take(1)[0]
-            meta = pd.DataFrame([head], columns=columns)
+            meta = dd.utils.make_meta(meta)
         # Serializing the columns and dtypes is much smaller than serializing
         # the empty frame
         cols = list(meta.columns)


Test patch:
diff --git a/dask/bag/tests/test_bag.py b/dask/bag/tests/test_bag.py
--- a/dask/bag/tests/test_bag.py
+++ b/dask/bag/tests/test_bag.py
@@ -803,6 +803,9 @@ def check_parts(df, sol):
     df = b.to_dataframe(columns=['a', 'b'])
     dd.utils.assert_eq(df, sol, check_index=False)
     check_parts(df, sol)
+    df = b.to_dataframe(meta=[('a', 'i8'), ('b', 'i8')])
+    dd.utils.assert_eq(df, sol, check_index=False)
+    check_parts(df, sol)
 
     # Elements are dictionaries
     b = b.map(lambda x: dict(zip(['a', 'b'], x)))
@@ -812,14 +815,19 @@ def check_parts(df, sol):
     assert df._name == b.to_dataframe()._name
 
     # With metadata specified
-    df = b.to_dataframe(columns=sol)
-    dd.utils.assert_eq(df, sol, check_index=False)
-    check_parts(df, sol)
+    for meta in [sol, [('a', 'i8'), ('b', 'i8')]]:
+        df = b.to_dataframe(meta=meta)
+        dd.utils.assert_eq(df, sol, check_index=False)
+        check_parts(df, sol)
+
+    # Error to specify both columns and meta
+    with pytest.raises(ValueError):
+        b.to_dataframe(columns=['a', 'b'], meta=sol)
 
     # Single column
     b = b.pluck('a')
     sol = sol[['a']]
-    df = b.to_dataframe(columns=sol)
+    df = b.to_dataframe(meta=sol)
     dd.utils.assert_eq(df, sol, check_index=False)
     check_parts(df, sol)
 
@@ -827,7 +835,7 @@ def check_parts(df, sol):
     sol = pd.DataFrame({'a': range(100)})
     b = db.from_sequence(range(100), npartitions=5)
     for f in [iter, tuple]:
-        df = b.map_partitions(f).to_dataframe(columns=sol)
+        df = b.map_partitions(f).to_dataframe(meta=sol)
         dd.utils.assert_eq(df, sol, check_index=False)
         check_parts(df, sol)