Repository: dask/dask

Issue:
Bag.to_dataframe fails if bag is empty and meta not specified.
This is an *bit* of an edge case and I'm not entirely *entirely* sure the current behavior is incorrect, but I found the result surprising enough that it may be a bug. I'd be happy to contribute a fix if you agree that the behavior should be changed.

This scenario may occur when the bag pipeline contains some configurable components that either (a) filter values from the bag or (b) add data to elements of the bag. In the event that the "filtering" reduces the bag size to zero the call to Bag.to_dataframe will fail when attempting to infer column names. This is different than the expected result, which would be an "empty" dask DataFrame.

Evaluating the bag with `.compute()` and passing the results to `pandas.DataFrame` produces and empty dataframe. Similarly, calling `to_dataframe(meta=pandas.DataFrame())` yields and empty frame.

Repro:

````
import dask
import dask.bag
import pandas

def add_val(e):
    e["b"] = e["a"] + 10
    return e
    
def filter_val(e):
    return e["b"] > 100

source_bag = dask.bag.from_sequence([{"a" : i} for i in range(2)])

print("SOURCE VIA COMPUTE: ",
      pandas.DataFrame(source_bag.compute()))
print("MAPPED VIA COMPUTE: ",
      pandas.DataFrame(source_bag.map(add_val).compute()))
print("FILTERED VIA COMPUTE:",
      pandas.DataFrame(source_bag.map(add_val).filter(filter_val).compute()))

print("SOURCE VIA TO_DATAFRAME: ",
      source_bag.to_dataframe())
print("MAPPED VIA TO_DATAFRAME: ",
      source_bag.map(add_val).to_dataframe())
print("FILTERED VIA TO_DATAFRAME WITH META: ",
      source_bag.map(add_val).filter(filter_val).to_dataframe(meta=pandas.DataFrame()))
print("FILTERED VIA TO_DATAFRAME NO META: ",
      source_bag.map(add_val).filter(filter_val).to_dataframe())
````

Yields:
````
SOURCE VIA COMPUTE:     a
0  0
1  1
MAPPED VIA COMPUTE:     a   b
0  0  10
1  1  11
FILTERED VIA COMPUTE: Empty DataFrame
Columns: []
Index: []
SOURCE VIA TO_DATAFRAME:  Dask DataFrame Structure:
                   a
npartitions=2       
               int64
                 ...
                 ...
Dask Name: to_dataframe, 4 tasks
MAPPED VIA TO_DATAFRAME:  Dask DataFrame Structure:
                   a      b
npartitions=2              
               int64  int64
                 ...    ...
                 ...    ...
Dask Name: to_dataframe, 6 tasks
FILTERED VIA TO_DATAFRAME WITH META:  Dask DataFrame Structure:
Empty DataFrame
Columns: []
Index: []
Dask Name: to_dataframe, 6 tasks
/usr/local/miniconda3/envs/dev_base/lib/python3.6/site-packages/dask/bag/core.py:1973: UserWarning: Insufficient elements for `take`. 1 elements requested, only 0 elements available. Try passing larger `npartitions` to `take`.
  "to `take`.".format(n, len(r)))
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-44-7f461515d6f8> in <module>()
     26       source_bag.map(add_val).filter(filter_val).to_dataframe(meta=pandas.DataFrame()))
     27 print("FILTERED VIA TO_DATAFRAME NO META: ",
---> 28       source_bag.map(add_val).filter(filter_val).to_dataframe())

/usr/local/miniconda3/envs/dev_base/lib/python3.6/site-packages/dask/bag/core.py in to_dataframe(self, meta, columns)
   1195                 meta = columns
   1196             else:
-> 1197                 head = self.take(1)[0]
   1198                 meta = pd.DataFrame([head], columns=columns)
   1199         elif columns is not None:

IndexError: tuple index out of range
````


Discussion:
What would be the dtypes and columns of such a dataframe?  Empty as well?  If so then a fix for this should be somewhat simple by setting `meta = pd.DataFrame()`.  Would you be interested in submiting a PR for this?
Yes, I believe the output of `pandas.DataFrame(my_bag.compute())` should be the same as `my_bag.to_dataframe().compute()` in this case and `dask.DataFrame` should have empty columns. I believe the fix will be in [`Bag.to_dataframe`](https://github.com/dask/dask/blob/0.15.4/dask/bag/core.py#L1196) but I've already noticed that the classes of exceptions raised vary depending on how the bag has been generated and I'll need to investigate this a bit more.

I'm happy to file a PR for this change and will start on this shortly.
As a related question, coming from someone who has used `dask` a small bit and has little familiarity with the internals, would it be coherent to describe `dask.DataFrame` metadata as a "minimum" set of required columns in the frame but have the underlying frames in the computation include *additional* columns that are (a) returned by `.compute` and (b) saved when the frame is output?

I'm envisioning a sort of "generic" template for dataframe-based dask computations in which bag or delayed based code produces a collection of records, these are mapped to frames for inline analysis, and then remapped to bags or output. In this model you can specify at graph-generation time a set of columns needed for analysis that *must* be present, but you would like to pass through any additional columns which are provided by the previous generation logic.

For example, the current behavior:

````
import dask
import dask.bag

in_records = [{"a" : i, "b" : i % 2} for i in range(10)]

inbag = dask.bag.from_sequence(in_records)
inframe = inbag.to_dataframe(columns=["a"])
print("QUERY RESULT:\n", inframe.query("a > 5").compute())
````
Yields:
````
QUERY RESULT:
    a
0  6
0  7
0  8
0  9
````

But it would be very interesting to allow:

````
import dask
import dask.bag

in_records = [{"a" : i, "b" : i % 2} for i in range(10)]

inbag = dask.bag.from_sequence(in_records)
inframe = inbag.to_dataframe(min_columns=["a"])
print("QUERY RESULT:\n", inframe.query("a > 5").compute())
````

Yielding:

````
QUERY RESULT:
    a  b
0  6  0
0  7  1
0  8  0
0  9  1
````
Currently the answer is "no, dask.dataframe does not guarantee correct operation in this case".  However I can appreciate that it would be useful in a variety of practical settings.
I completely understand that this isn't supported now, but what's your gut instinct on whether this could be added to the current implementation? Is a strict definition a `dask.DataFrame`'s column layout baked into the implementation? Would this be a feature someone without a deep understanding of `dask` internals could work on as a learning project?
If we were to accept this we would need to have confidence that most (all?) dask.dataframe algorithms worked well even with extra columns.  I would not be very surprised to learn that this was or was not the case.

This *could* be a learning project, with the caveat that, because of the possibility above, it is somewhat likely that it will not be merged regardless.  

If you were curious and wanted to pursue this I would first create a dask.dataframe that had incomplete metadata, and then run through a bunch of operations with it, seeing where it performed well and where it broke.  This sort of exploration would probably help us to make a more informed decision.

```python
df = pd.DataFrame({'x': ..., 'y': ...})
value = dask.delayed(df)
ddf = dd.from_pandas([value], meta=df.iloc[:0][['x']])
# do stuff with ddf, see what works and what doesn't
```
As someone who wrote the current metadata handling for `dask.dataframe`, I'm going to be a fairly hard -1 on anything that makes knowing the complete metadata for each partition in a dask dataframe impossible. Many operations rely on knowing what the columns, dtypes, and indices are in order to function properly. Being strict about the metadata helps ensure correctness and eases the already large maintenance burden of dask dataframe.

If your goal as a user is just to not have to specify the metadata beforehand, you can either let dask compute it for you in `to_dataframe`, or somehow infer it yourself based on your query.

Alternatively, if your only goal is to turn a `bag` into an in-memory pandas dataframe (never using `dask.dataframe` operations), then you can probably write your own conversion function using `bag.to_delayed` and `dask.delayed` (applying `pd.DataFrame`, and `pd.concat` appropriately).
@jcrist I'm assuming you mean "knowing the complete metadata for each partition in a dask dataframe ***impossible***?"

Completely understand if you're the maintainer for section of the code and *don't* want to see this change. My instinct is that having incomplete metadata could make a number of dependent components quite tricky. Thank you for the clear 👎.

Resolving patch:
diff --git a/dask/bag/core.py b/dask/bag/core.py
--- a/dask/bag/core.py
+++ b/dask/bag/core.py
@@ -1,16 +1,15 @@
 from __future__ import absolute_import, division, print_function
 
-from collections import Iterable, Iterator, defaultdict
-from functools import wraps, partial
 import itertools
 import math
-from operator import getitem
 import types
 import uuid
-from random import Random
-from warnings import warn
+import warnings
+from collections import Iterable, Iterator, defaultdict
 from distutils.version import LooseVersion
-
+from functools import wraps, partial
+from operator import getitem
+from random import Random
 
 from toolz import (merge, take, reduce, valmap, map, partition_all, filter,
                    remove, compose, curry, first, second, accumulate, peek)
@@ -1050,7 +1049,7 @@ def combine2(acc, x):
 
         return type(self)(merge(self.dask, dsk), e, 1)
 
-    def take(self, k, npartitions=1, compute=True):
+    def take(self, k, npartitions=1, compute=True, warn=True):
         """ Take the first k elements.
 
         Parameters
@@ -1064,6 +1063,9 @@ def take(self, k, npartitions=1, compute=True):
             returned. Pass -1 to use all partitions.
         compute : bool, optional
             Whether to compute the result, default is True.
+        warn : bool, optional
+            Whether to warn if the number of elements returned is less than
+            requested, default is True.
 
         >>> b = from_sequence(range(10))
         >>> b.take(3)  # doctest: +SKIP
@@ -1087,9 +1089,9 @@ def take(self, k, npartitions=1, compute=True):
                 dsk[(name_p, i)] = (list, (take, k, (self.name, i)))
 
             concat = (toolz.concat, ([(name_p, i) for i in range(npartitions)]))
-            dsk[(name, 0)] = (safe_take, k, concat)
+            dsk[(name, 0)] = (safe_take, k, concat, warn)
         else:
-            dsk = {(name, 0): (safe_take, k, (self.name, 0))}
+            dsk = {(name, 0): (safe_take, k, (self.name, 0), warn)}
 
         b = Bag(merge(self.dask, dsk), name, 1)
 
@@ -1215,12 +1217,16 @@ def to_dataframe(self, meta=None, columns=None):
         import dask.dataframe as dd
         if meta is None:
             if isinstance(columns, pd.DataFrame):
-                warn("Passing metadata to `columns` is deprecated. Please "
-                     "use the `meta` keyword instead.")
+                warnings.warn("Passing metadata to `columns` is deprecated. "
+                              "Please use the `meta` keyword instead.")
                 meta = columns
             else:
-                head = self.take(1)[0]
-                meta = pd.DataFrame([head], columns=columns)
+                head = self.take(1, warn=False)
+                if len(head) == 0:
+                    raise ValueError("`dask.bag.Bag.to_dataframe` failed to "
+                                     "properly infer metadata, please pass in "
+                                     "metadata via the `meta` keyword")
+                meta = pd.DataFrame(list(head), columns=columns)
         elif columns is not None:
             raise ValueError("Can't specify both `meta` and `columns`")
         else:
@@ -1986,12 +1992,12 @@ def empty_safe_aggregate(func, parts, is_last):
     return empty_safe_apply(func, parts2, is_last)
 
 
-def safe_take(n, b):
+def safe_take(n, b, warn=True):
     r = list(take(n, b))
-    if len(r) != n:
-        warn("Insufficient elements for `take`. {0} elements requested, "
-             "only {1} elements available. Try passing larger `npartitions` "
-             "to `take`.".format(n, len(r)))
+    if len(r) != n and warn:
+        warnings.warn("Insufficient elements for `take`. {0} elements "
+                      "requested, only {1} elements available. Try passing "
+                      "larger `npartitions` to `take`.".format(n, len(r)))
     return r
 
 


Test patch:
diff --git a/dask/bag/tests/test_bag.py b/dask/bag/tests/test_bag.py
--- a/dask/bag/tests/test_bag.py
+++ b/dask/bag/tests/test_bag.py
@@ -530,17 +530,23 @@ def test_take_npartitions():
         b.take(1, npartitions=5)
 
 
-@pytest.mark.skipif(sys.version_info[:2] == (3,3),
-                    reason="Python3.3 uses pytest2.7.2, w/o warns method")
 def test_take_npartitions_warn():
-    with pytest.warns(None):
-        b.take(100)
+    # Use single-threaded scheduler so warnings are properly captured in the
+    # same process
+    with dask.set_options(get=dask.get):
+        with pytest.warns(UserWarning):
+            b.take(100)
+
+        with pytest.warns(UserWarning):
+            b.take(7)
 
-    with pytest.warns(None):
-        b.take(7)
+        with pytest.warns(None) as rec:
+            b.take(7, npartitions=2)
+        assert len(rec) == 0
 
-    with pytest.warns(None):
-        b.take(7, npartitions=2)
+        with pytest.warns(None) as rec:
+            b.take(7, warn=False)
+        assert len(rec) == 0
 
 
 def test_map_is_lazy():
@@ -775,6 +781,11 @@ def check_parts(df, sol):
     with pytest.raises(ValueError):
         b.to_dataframe(columns=['a', 'b'], meta=sol)
 
+    # Inference fails if empty first partition
+    b2 = b.filter(lambda x: x['a'] > 200)
+    with pytest.raises(ValueError):
+        b2.to_dataframe()
+
     # Single column
     b = b.pluck('a')
     sol = sol[['a']]