Repository: dask/dask

Issue:
Is there a hidden random seed in dask?
Running this twice on a dask dataframe returns different results in 'outfile' each time:

```
alldata = dd.read_csv(infile)
alldata = alldata.set_index('Id')
estimates = alldata.groupby(alldata.index).apply(myfunc, columns='Expected')
estimates.to_csv(outfile, header=True)
```

myfunc is deterministic.  Yet, if I run the script twice, I get different results.  The format of the input file is as follows:

```
Id,minutes_past,radardist_km,Ref,Ref_5x5_10th,Ref_5x5_50th,Ref_5x5_90th,RefComposite,RefComposite_5x5_10th,RefComposite_5
x5_50th,RefComposite_5x5_90th,RhoHV,RhoHV_5x5_10th,RhoHV_5x5_50th,RhoHV_5x5_90th,Zdr,Zdr_5x5_10th,Zdr_5x5_50th,Zdr_5x5_90
th,Kdp,Kdp_5x5_10th,Kdp_5x5_50th,Kdp_5x5_90th
1,1,8.0,,,,14.0,,,,15.0,,,,,,,,,,,,
1,5,8.0,10.0,,10.0,18.0,11.5,,11.0,18.0,,,,,,,,,,,,
2,0,15.0,,,,,,,,,,,,,,,,,,,,
2,4,15.0,,,,,,,,,,,,,,,,,,,,
2,8,15.0,,,,,,,,,,,,,,,,,,,,
2,12,15.0,,,,,,,,,,,,,,,,,,,,
3,3,11.0,22.5,15.0,25.5,34.5,29.5,15.0,27.5,36.5,0.885,0.86833334,0.95166665,0.9716667,0.5,-0.25,0.5625,2.375,1.4099884,-
5.3200073,0.34999084,3.869995
3,8,11.0,,,,22.0,,,,28.5,0.92833334,0.9316667,0.945,0.97833335,1.5625,-0.1875,0.875,1.625,-0.69999695,-5.6200104,-0.69999
695,2.119995
```

The file itself is 8 million lines long.



Discussion:
There isn't a random seed in this case, it's just multiple threads acting non-deterministically in the groupby's shuffle.

I suspect that the issue here is that we're not guaranteeing a fixed order within each group.  Perhaps we should sort groups before sending them to myfunc?

I'm curious, what is pandas' policy during a `groupby(...).apply(myfunc)`?  Do the dataframes sent to myfunc obey some canonical ordering?

> I'm curious, what is pandas' policy during a groupby(...).apply(myfunc)? Do the dataframes sent to myfunc obey some canonical ordering?

Yes, I'm pretty sure pandas preserves the original order of elements in a groupby/apply. I'm not sure if that's actually guaranteed as part of the API though.

Preserving original order is going to be _hard_.  

The two options I see are sort or ignore.  Thoughts?

Here's a timely reference for the pandas behavior: https://github.com/pydata/pandas/pull/10931/

Thanks! I think not having the original order preserved is
counterintuitive. However, if the original dataframe has some natural
ordering, users can do a sort in myfunc, so it is not the end of the world.

Perhaps, just document this difference in the API?

Sent from my phone
On Sep 4, 2015 10:55 PM, "Stephan Hoyer" notifications@github.com wrote:

> Here's a timely reference for the pandas behavior: pydata/pandas#10931
> https://github.com/pydata/pandas/pull/10931
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/blaze/dask/issues/665#issuecomment-137912944.


Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -1725,6 +1725,11 @@ def column_info(self):
         return self.df.columns
 
     def apply(self, func, columns=None):
+        """ Apply function to each group.
+
+        If the grouper does not align with the index then this causes a full
+        shuffle.  The order of rows within each group may not be preserved.
+        """
         if (isinstance(self.index, Series) and
             self.index._name == self.df.index._name):
             df = self.df
@@ -1787,6 +1792,11 @@ def column_info(self):
         return self.key
 
     def apply(self, func, columns=None):
+        """ Apply function to each group.
+
+        If the grouper does not align with the index then this causes a full
+        shuffle.  The order of rows within each group may not be preserved.
+        """
         # df = set_index(self.df, self.index, **self.kwargs)
         if self.index._name == self.df.index._name:
             df = self.df
diff --git a/dask/dataframe/shuffle.py b/dask/dataframe/shuffle.py
--- a/dask/dataframe/shuffle.py
+++ b/dask/dataframe/shuffle.py
@@ -14,8 +14,10 @@
 def set_index(df, index, npartitions=None, compute=True, **kwargs):
     """ Set DataFrame index to new column
 
-    Sorts index and realigns Dataframe to new sorted order.  This shuffles and
-    repartitions your data.
+    Sorts index and realigns Dataframe to new sorted order.
+
+    This shuffles and repartitions your data.  If done in parallel the
+    resulting order is non-deterministic.
     """
     npartitions = npartitions or df.npartitions
     if not isinstance(index, Series):
@@ -160,7 +162,8 @@ def shuffle(df, index, npartitions=None):
     full dataset read, serialization and shuffle.  This is expensive.  If
     possible you should avoid shuffles.
 
-    This does not preserve a meaningful index/partitioning scheme.
+    This does not preserve a meaningful index/partitioning scheme.  This is not
+    deterministic if done in parallel.
 
     See Also
     --------


Test patch: