Repository: dask/dask

Issue:
Types of a fake partition in map_partition inferring are not exactly correct
Consider the following code:

``` python
import dask.dataframe
import pandas

def qq(partition_df):
    print(partition_df)
    print(partition_df['X'].dtype)
    return partition_df

df = dask.dataframe.from_pandas(pandas.DataFrame({'X': range(100)}, dtype='int32'), npartitions=1)
print(df['X'].dtype)
df.map_partitions(qq).compute()
```

The output is:

```
float32
float64
float32
```

while I expect to see:

```
float32
float32
float32
```

Can the types be just copied from the original dataframe? How can I distinguish fake partitions from true ones? I have a heavy computation which I would rather avoid on fake partitions. In Dask 0.10 fake partition was empty, but Dask 0.11 passes a partition with two records in it.



Discussion:
You would provide a `meta=` keyword argument to the `map_partitions` call.   

@jcrist maybe we should have a docpage about Dask.dataframe metadata?  Questions like these have been coming up a bit recently.  It'd be nice to have something to point people towards.

@mrocklin I would like to have types and column names in one place, that is inside the map function... With Dask 0.10 I could do the following:

``` python
def map_func(df):
    my_columns = ['a', 'b']
    if df.empty:
        return pandas.DataFrame([], columns=my_columns, dtype='float32')
    ...
    new_df = pandas.DataFrame(data, columns=my_columns, dtype='float32')
    ...
    return new_df
```

> @jcrist maybe we should have a docpage about Dask.dataframe metadata? Questions like these have been coming up a bit recently. It'd be nice to have something to point people towards.

That's a good idea, I'll write that up.

> Consider the following code: ...

I'm seeing `int64` where it should be `int32` (fixing now), but I'm not seeing floats anywhere. My experience:

`````` python
```python
In [1]: %load temp.py

In [2]: # %load temp.py
import dask.dataframe
import pandas

def qq(partition_df):
    print(partition_df['X'].dtype)
    return partition_df

df = dask.dataframe.from_pandas(pandas.DataFrame({'X': range(100)}, dtype='int32'), npartitions=3)
res = df.map_partitions(qq)
   ...:
int64

In [3]: t = res.compute()
int32
int32
int32

In [4]: t.dtypes
Out[4]:
X    int32
dtype: object
``````

> @mrocklin I would like to have types and column names in one place, that is inside the map function... 

I share that desire.  It competes with other desires though.  If you want dask.dataframe to be able to reason about your dtypes then we have three options:
1.  You need to provide them to it with a `meta=` keyword argument
2.  We call your function on a bit of data to guess what the dtypes are
3.  We inspect your function to determine your intent

Obviously 3 is very hard.  Until we figure something else out (suggestions and pull requests very welcome here) we're stuck with 1 or 2

Previous versions of dask.dataframe allowed the creation of dask.dataframes without known dtypes.  This is no longer allowed as it created some difficult to maintain situations.


Resolving patch:
diff --git a/dask/dataframe/utils.py b/dask/dataframe/utils.py
--- a/dask/dataframe/utils.py
+++ b/dask/dataframe/utils.py
@@ -229,7 +229,7 @@ def _nonempty_index(idx):
     'b': np.bool_(True),
     'V': np.void(b' '),
     'M': np.datetime64('1970-01-01'),
-    'm': np.timedelta64(1, 'D'),
+    'm': np.timedelta64(1),
     'S': np.str_('foo'),
     'a': np.str_('foo'),
     'U': np.unicode_('foo'),
@@ -242,7 +242,8 @@ def _scalar_from_dtype(dtype):
     elif dtype.kind == 'c':
         return dtype.type(complex(1, 0))
     elif dtype.kind in _simple_fake_mapping:
-        return _simple_fake_mapping[dtype.kind]
+        o = _simple_fake_mapping[dtype.kind]
+        return o.astype(dtype) if dtype.kind in ('m', 'M') else o
     else:
         raise TypeError("Can't handle dtype: {0}".format(dtype))
 
@@ -276,7 +277,7 @@ def _nonempty_series(s, idx):
                                ordered=s.cat.ordered)
     else:
         entry = _scalar_from_dtype(dtype)
-        data = [entry, entry]
+        data = np.array([entry, entry], dtype=dtype)
     return pd.Series(data, name=s.name, index=idx)
 
 


Test patch:
diff --git a/dask/dataframe/tests/test_utils_dataframe.py b/dask/dataframe/tests/test_utils_dataframe.py
--- a/dask/dataframe/tests/test_utils_dataframe.py
+++ b/dask/dataframe/tests/test_utils_dataframe.py
@@ -97,25 +97,29 @@ def test_meta_nonempty():
     df1 = pd.DataFrame({'A': pd.Categorical(['Alice', 'Bob', 'Carol']),
                         'B': list('abc'),
                         'C': 'bar',
-                        'D': 3.0,
-                        'E': pd.Timestamp('2016-01-01'),
-                        'F': pd.date_range('2016-01-01', periods=3,
+                        'D': np.float32(1),
+                        'E': np.int32(1),
+                        'F': pd.Timestamp('2016-01-01'),
+                        'G': pd.date_range('2016-01-01', periods=3,
                                            tz='America/New_York'),
-                        'G': pd.Timedelta('1 hours'),
-                        'H': np.void(b' ')},
-                       columns=list('DCBAHGFE'))
+                        'H': pd.Timedelta('1 hours', 'ms'),
+                        'I': np.void(b' ')},
+                       columns=list('DCBAHGFEI'))
     df2 = df1.iloc[0:0]
     df3 = meta_nonempty(df2)
     assert (df3.dtypes == df2.dtypes).all()
     assert df3['A'][0] == 'Alice'
     assert df3['B'][0] == 'foo'
     assert df3['C'][0] == 'foo'
-    assert df3['D'][0] == 1.0
-    assert df3['E'][0] == pd.Timestamp('1970-01-01 00:00:00')
-    assert df3['F'][0] == pd.Timestamp('1970-01-01 00:00:00',
+    assert df3['D'][0] == np.float32(1)
+    assert df3['D'][0].dtype == 'f4'
+    assert df3['E'][0] == np.int32(1)
+    assert df3['E'][0].dtype == 'i4'
+    assert df3['F'][0] == pd.Timestamp('1970-01-01 00:00:00')
+    assert df3['G'][0] == pd.Timestamp('1970-01-01 00:00:00',
                                        tz='America/New_York')
-    assert df3['G'][0] == pd.Timedelta('1 days')
-    assert df3['H'][0] == 'foo'
+    assert df3['H'][0] == pd.Timedelta('1', 'ms')
+    assert df3['I'][0] == 'foo'
 
     s = meta_nonempty(df2['A'])
     assert s.dtype == df2['A'].dtype