Repository: dask/dask

Issue:
Intermittent failure in pivot_table test
As reported by @chmp in #1678 and captured in [this travis.ci run](https://travis-ci.org/dask/dask/jobs/169042996).  cc @sinhrks 

```
[gw2] FAILED dask/dataframe/tests/test_reshape.py::test_pivot_table[count] 
```

``` python
=================================== FAILURES ===================================
___________________________ test_pivot_table[count] ____________________________
[gw2] linux -- Python 3.3.5 /home/travis/miniconda/envs/test-environment/bin/python3
aggfunc = 'count'
    @pytest.mark.parametrize('aggfunc', ['mean', 'sum', 'count'])
    def test_pivot_table(aggfunc):
        df = pd.DataFrame({'A': np.random.choice(list('XYZ'), size=100),
                           'B': np.random.randn(100),
                           'C': pd.Categorical(np.random.choice(list('abc'), size=100))})
        ddf = dd.from_pandas(df, 5)

        res = dd.pivot_table(ddf, index='A', columns='C', values='B',
                             aggfunc=aggfunc)
        exp = pd.pivot_table(df, index='A', columns='C', values='B',
                             aggfunc=aggfunc)
        if aggfunc == 'count':
            # dask result cannot be int64 dtype depending on divisions because of NaN
            exp = exp.astype(np.float64)

        if PANDAS_ge_0190:
            assert_eq(res, exp)
        else:
            # because of a pandas 0.18.x bug, categorical dtype is not preserved
>           assert_eq(res, exp, check_names=False, check_column_type=False)
dask/dataframe/tests/test_reshape.py:118: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dask/dataframe/utils.py:427: in assert_eq
    tm.assert_frame_equal(a, b, **kwargs)
../../../miniconda/envs/test-environment/lib/python3.3/site-packages/pandas/util/testing.py:1166: in assert_frame_equal
    obj='DataFrame.iloc[:, {0}]'.format(i))
../../../miniconda/envs/test-environment/lib/python3.3/site-packages/pandas/util/testing.py:1026: in assert_series_equal
    assert_attr_equal('dtype', left, right)
../../../miniconda/envs/test-environment/lib/python3.3/site-packages/pandas/util/testing.py:804: in assert_attr_equal
    left_attr, right_attr)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
obj = 'Attributes', message = 'Attribute "dtype" are different'
left = dtype('int64'), right = dtype('float64')
    def raise_assert_detail(obj, message, left, right):
        if isinstance(left, np.ndarray):
            left = pprint_thing(left)
        if isinstance(right, np.ndarray):
            right = pprint_thing(right)

        msg = """{0} are different

    {1}
    [left]:  {2}
    [right]: {3}""".format(obj, message, left, right)
>       raise AssertionError(msg)
E       AssertionError: Attributes are different
E       
E       Attribute "dtype" are different
E       [left]:  int64
E       [right]: float64
```



Discussion:
maybe depends on nans included in intermedfiate results. will check this weekend.

After I added additional tests in #1678, I encounter similar errors myself. At least for my code, I'm pretty sure it is caused by the fact that pandas tries to downcast the result back to the original type in groupby aggregations. The relevant code can be found in [_GroupBy._try_cast](https://github.com/pandas-dev/pandas/blob/master/pandas/core/groupby.py#L760). Maybe a similar process is at play here.

@chmp do you happen to understand this problem well enough to create a smaller reproducible example than the test linked to above?

Actually, I misread the test, I assumed it was related to the mean. For count, the explanation of @sinhrks should be right. 


Resolving patch:
diff --git a/dask/dataframe/methods.py b/dask/dataframe/methods.py
--- a/dask/dataframe/methods.py
+++ b/dask/dataframe/methods.py
@@ -102,5 +102,7 @@ def pivot_sum(df, index, columns, values):
 
 
 def pivot_count(df, index, columns, values):
+    # we cannot determine dtype until concatenationg all partitions.
+    # make dtype deterministic, always coerce to np.float64
     return pd.pivot_table(df, index=index, columns=columns,
-                          values=values, aggfunc='count')
+                          values=values, aggfunc='count').astype(np.float64)


Test patch:
diff --git a/dask/dataframe/tests/test_reshape.py b/dask/dataframe/tests/test_reshape.py
--- a/dask/dataframe/tests/test_reshape.py
+++ b/dask/dataframe/tests/test_reshape.py
@@ -118,6 +118,29 @@ def test_pivot_table(aggfunc):
         assert_eq(res, exp, check_names=False, check_column_type=False)
 
 
+def test_pivot_table_dtype():
+
+    df = pd.DataFrame({'A': list('AABB'),
+                       'B': pd.Categorical(list('ABAB')),
+                       'C': [1, 2, 3, 4]})
+    ddf = dd.from_pandas(df, 2)
+    res = dd.pivot_table(ddf, index='A', columns='B',
+                         values='C', aggfunc='count')
+
+    exp_index = pd.CategoricalIndex(['A', 'B'], name='B')
+    exp = pd.Series([np.float64] * 2, index=exp_index)
+    tm.assert_series_equal(res.dtypes, exp)
+
+    exp = pd.pivot_table(df, index='A', columns='B',
+                         values='C', aggfunc='count').astype(np.float64)
+
+    if PANDAS_ge_0190:
+        assert_eq(res, exp)
+    else:
+        # because of a pandas 0.18.x bug, categorical dtype is not preserved
+        assert_eq(res, exp, check_names=False, check_column_type=False)
+
+
 def test_pivot_table_errors():
     df = pd.DataFrame({'A': np.random.choice(list('abc'), size=10),
                        'B': np.random.randn(10),