Repository: dask/dask

Issue:
DataFrame correlations are broken?
These seem to break with unknown divisions (although perhaps this is appropriate):

``` python
>>> df.tip_amount.corr(df.payment_type == 2).compute()
/opt/anaconda/lib/python2.7/site-packages/dask/dataframe/core.pyc in corr(self, other, method, min_periods)
   1235             raise NotImplementedError("Only Pearson correlation has been "
   1236                                       "implemented")
-> 1237         df = concat([self, other], axis=1)
   1238         return cov_corr(df, min_periods, corr=True, scalar=True)
   1239 

/opt/anaconda/lib/python2.7/site-packages/dask/dataframe/multi.pyc in concat(dfs, axis, join, interleave_partitions)
    552     else:
    553         if axis == 1:
--> 554              raise ValueError('Unable to concatenate DataFrame with unknown '
    555                               'division specifying axis=1')
    556         else:

ValueError: Unable to concatenate DataFrame with unknown division specifying axis=1
```

Also I seem to be getting wrong results:

``` python
>>> df[['tip_amount', 'payment_type']].corr().compute()
               tip_amount    payment_type
tip_amount       0.999169       -0.033852
payment_type    -0.033852        8.420302

>>> df.head(1000)[['tip_amount', 'payment_type']].corr()
               tip_amount    payment_type
tip_amount       1.000000       -0.559584
payment_type    -0.559584        1.000000
```



Discussion:
cc @jcrist 

Another issue:

``` python
In [1]: import dask.dataframe as dd

In [2]: df = dd.demo.make_timeseries('2000', '2001', {'A': float, 'B': float}, freq='2H', partition_freq='1D', seed=1)

In [3]: import dask

In [4]: pdb
Automatic pdb calling has been turned ON

In [5]: df.corr().compute(get=dask.get)
Out[5]: 
          A         B
A  1.000000  0.009652
B  0.009652  1.000000

In [6]: df.A.corr(df.B > 0.5).compute(get=dask.get)
TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
> /home/mrocklin/workspace/dask/dask/dataframe/core.py(2234)cov_corr_chunk()
   2233     mat = df.values
-> 2234     mask = np.isfinite(mat)
   2235     keep = np.bitwise_and(mask[:, None, :], mask[:, :, None])

ipdb> pp mat
array([[-0.26907655414367415, True],
       [-0.2763089407924768, False],
       [-0.37464477680903263, False],
       [-0.587595215826626, False],
       [-0.7336550349057425, False],
       [0.21745538968855382, False],
       [0.6926140682880764, True],
       [0.8094715239590082, False],
       [-0.5834730068004101, True],
       [0.6651208612375974, False],
       [-0.5659759577457062, False],
       [0.6617252305707162, False]], dtype=object)
```

Example above depends on this:

```
--- a/dask/dataframe/demo.py
+++ b/dask/dataframe/demo.py
@@ -86,4 +86,5 @@ def make_timeseries(start, end, dtypes, freq, partition_freq, seed=No
     dsk = dict(((name, i), (make_timeseries_part, divisions[i], divisions[i + 1],
                                                  dtypes, freq, seeds[i]))
                 for i in range(len(divisions) - 1))
-    return DataFrame(dsk, name, sorted(dtypes), divisions)
+    head = make_timeseries_part('2000','2000', dtypes, '1H', 1)
+    return DataFrame(dsk, name, head, divisions)
```

``` python
In [1]: import dask.dataframe as dd

In [2]: df = dd.demo.make_timeseries('2000', '2050', {'A': float, 'B': float}, freq='1s', partition_freq='1D', seed=1)

In [3]: df.dtypes
Out[3]: 
A    float64
B    float64
dtype: object

In [4]: df.corr().compute()
Out[4]: 
          A         B
A  1.383751 -0.309378
B -0.309378  1.504100
```

This appears to be an overflow issue

I can only reproduce it with large datasets

This takes 2-3 minutes to run on my machine.  Smaller datasets don't have this issue.

Ok, going through the numerous issues here:
1. They must have known divisions, because `corr`/`cov` are index sensitive. We could throw a more informative error of course.
2. `df.A.corr(df.B > 0.5).compute(get=dask.get)` wouldn't work, even in pandas. `df.B > 0.5` is a boolean series.
3. The other issues seem to indicate some degree of numeric instability. I find that I can generate incorrect results with other smaller examples, but the smaller the example the closer the response gets to being correct. Will look into it.

1.  There is an important special case where they have the _same_ unknown divisions, such as in `df.A.corr(df.B)`.  We're still assured that things will work well in this case.  I'm not sure how best to support this.  Some options include allowing it to go through and then checking at runtime to ensure that the indexes match up or having dataframes keep track of their index separately (probably not worth the complexity).
2.  OK, I'll raise an issue in Pandas if I want this in the future.  For now I'm using `.astype(int)`
3.  Great, this is the only thing I can't easily work around.


Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -1510,7 +1510,9 @@ def to_bag(self, index=False):
         return to_bag(self, index)
 
     def _get_numeric_data(self, how='any', subset=None):
-
+        # If unknown dtype, need to infer from head.
+        if not self._known_dtype:
+            self.dtypes
         # calculate columns to avoid unnecessary calculation
         numerics = self._pd._get_numeric_data()
 
@@ -2258,18 +2260,21 @@ def cov_corr_agg(data, meta, min_periods=2, corr=False, scalar=False):
     sums = np.nan_to_num(data['sum'])
     counts = data['count']
 
-    totals = np.cumsum(sums, 0)
-    ns = np.cumsum(counts, 0)
+    cum_sums = np.cumsum(sums, 0)
+    cum_counts = np.cumsum(counts, 0)
+
+    s1 = cum_sums[:-1]
+    s2 = sums[1:]
+    n1 = cum_counts[:-1]
+    n2 = counts[1:]
+    d = (s2/n2) - (s1/n1)
+    C = (np.nansum((n1 * n2)/(n1 + n2) * (d * d.transpose((0, 2, 1))), 0) +
+         np.nansum(data['cov'], 0))
 
-    ts = ns[:-1] * sums[1:] - counts[1:] * totals[:-1]
-    dens = ns[1:] * ns[:-1] * counts[1:]
-    dens = np.where(dens, dens, np.nan)
-    C = (np.nansum(data['cov'], 0) +
-         np.nansum(ts * ts.transpose((0, 2, 1)) / dens, 0))
-    C[ns[-1] < min_periods] = np.nan
-    nobs = np.where(ns[-1], ns[-1], np.nan)
+    C[cum_counts[-1] < min_periods] = np.nan
+    nobs = np.where(cum_counts[-1], cum_counts[-1], np.nan)
     if corr:
-        mu = totals[-1] / nobs
+        mu = cum_sums[-1] / nobs
         counts_na = np.where(counts, counts, np.nan)
         m2 = np.nansum(data['m'] + counts*(sums/counts_na - mu)**2, axis=0)
         den = np.sqrt(m2 * m2.T)


Test patch:
diff --git a/dask/dataframe/tests/test_arithmetics_reduction.py b/dask/dataframe/tests/test_arithmetics_reduction.py
--- a/dask/dataframe/tests/test_arithmetics_reduction.py
+++ b/dask/dataframe/tests/test_arithmetics_reduction.py
@@ -779,6 +779,14 @@ def test_reductions_frame_dtypes():
     assert numerics._get_numeric_data().dask == numerics.dask
 
 
+def test_get_numeric_data_unknown_part():
+    df = pd.DataFrame({'a': range(5), 'b': range(5), 'c': list('abcde')})
+    ddf = dd.from_pandas(df, 3)
+    # Drop dtype information
+    ddf = dd.DataFrame(ddf.dask, ddf._name, ['a', 'b', 'c'], ddf.divisions)
+    assert eq(ddf._get_numeric_data(), df._get_numeric_data())
+
+
 def test_reductions_frame_nan():
     df = pd.DataFrame({'a': [1, 2, np.nan, 4, 5, 6, 7, 8],
                        'b': [1, 2, np.nan, np.nan, np.nan, 5, np.nan, np.nan],
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -1454,6 +1454,13 @@ def test_corr():
     pytest.raises(TypeError, lambda: da.corr(ddf))
 
 
+def test_cov_corr_stable():
+    df = pd.DataFrame(np.random.random((20000000, 2)) * 2 - 1, columns=['a', 'b'])
+    ddf = dd.from_pandas(df, npartitions=50)
+    assert eq(ddf.cov(), df.cov())
+    assert eq(ddf.corr(), df.corr())
+
+
 def test_apply_infer_columns():
     df = pd.DataFrame({'x': [1, 2, 3, 4], 'y': [10, 20, 30, 40]})
     ddf = dd.from_pandas(df, npartitions=2)