Repository: dask/dask

Issue:
Support for randomized partitions of datasets
I'd love to see future capability that allows one to perform a randomized partition of a dataset and follow-on computation on the partitions. This is critical for simple hold-out tests and cross-validation to assess the generalization performance of models resulting from machine learning. 



Discussion:
What are your thoughts on reading through the data twice?  How performance critical is this operation in your workflow?

Do you need exactly a certain split or are you ok with a slightly uneven split?  E.g. you ask for 80/20 but you get 81/79.

Is there ever a case where you care about partitioning into more than two sets?

It'd be nice to avoid that but I can imagine this could be problematic without explicitly writing the partitioned datasets to disk. If you generate the partition index prior to streaming through the data, is it complicated to have multiple computational pipelines for the different partitions?

Regarding performance, it really all depends on the application. Nothing I'm thinking about is actually real-time currently but I'd prefer as much performance as possible. Otherwise there's really no reason not to load the data on a huge EC2 instance and load it all into RAM. I'm specifically thinking about using Blaze in the context of medium data problems. 

Regarding the partition, 80/20 might not always be exactly possible so no worries there. I just want to know exactly how many examples are in each partition when I'm computing test statistics. 

If I'm operating with so much data that I can't load it all into memory, it's very unlikely I'd be thinking about doing a k-fold cross-validation. More likely is partitioning the data three ways actually into training data, validation data and test data. 

>  If you generate the partition index prior to streaming through the data,

What I have in mind is using a hashing function rather than an index.  

> is it complicated to have multiple computational pipelines for the different partitions?

It isn't complicated.  We do this thing all the time in dask.array for more complex operations.  It becomes a bit more complex in dask.dataframe due to GIL and serialization issues if you try to parallelize things with multiprocessing.  My question about how much you care was just a question, not a statement of "this is really hard."  It sounds like your answer is "I care a non-trivial amount."

Splitting at least three ways for training, validation and test for medium/big data is the minimum requirement. One brings k-fold cross-validation to bear when data is constrained. Hard to imagine that being necessary when one has more data than can fit in RAM. Hope that helps!

@diehl are you datasets tabular or multidimensional arrays? I'm guessing tabular, because for out of core arrays (with N > 1) doing the split is pretty straightforward using the normal numpy/scitkit-learn methods.

@shoyer how do you define the distinction between array and tabular? 

> On May 31, 2015, at 3:06 PM, Stephan Hoyer notifications@github.com wrote:
> 
> @diehl are you datasets tabular or array based? I'm guessing tabular, because for out of core arrays (with N > 1) doing the split is pretty straightforward using the normal numpy/scitkit-learn methods.
> 
> —
> Reply to this email directly or view it on GitHub.

@diehl If you stored your data in a matrix, what would be the shape of that matrix? If the matrix has shape `(M, N)` and `M >> N`, then I would call the data tabular. This is a much harder problem than if `M ~ N` because in the later case you can easily store vectors of length `M`.

@shoyer I would definitely be using tabular data with Blaze. Odds are if M ~ N, then the data will likely fit into RAM. 


Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -24,7 +24,7 @@
 from ..async import get_sync
 from ..threaded import get as get_threaded
 from ..compatibility import unicode, apply
-from ..utils import repr_long_list, IndexCallable
+from ..utils import repr_long_list, IndexCallable, pseudorandom
 
 
 def _concat(args):
@@ -154,6 +154,32 @@ def map_blocks(self, func, columns=None):
         return type(self)(merge(dsk, self.dask), name,
                           columns, self.divisions)
 
+    def random_split(self, p, seed=None):
+        """ Pseudorandomly split dataframe into different pieces row-wise
+
+        50/50 split
+        >>> a, b = df.random_split([0.5, 0.5])  # doctest: +SKIP
+
+        80/10/10 split, consistent seed
+        >>> a, b, c = df.random_split([0.8, 0.1, 0.1], seed=123)  # doctest: +SKIP
+        """
+        if seed is not None:
+            np.random.seed(seed)
+        seeds = np.random.randint(0, 2**31, self.npartitions)
+        dsk_full = dict(((self._name + '-split-full', i),
+                         (pd_split, (self._name, i), p, seed))
+                       for i, seed in enumerate(seeds))
+
+        dsks = [dict(((self._name + '-split-%d' % i, j),
+                      (getitem, (self._name + '-split-full', j), i))
+                      for j in range(self.npartitions))
+                      for i in range(len(p))]
+        return [type(self)(merge(self.dask, dsk_full, dsk),
+                           self._name + '-split-%d' % i,
+                           self.column_info,
+                           self.divisions)
+                for i, dsk in enumerate(dsks)]
+
     def head(self, n=10, compute=True):
         """ First n rows of the dataset
 
@@ -454,11 +480,11 @@ def __getitem__(self, key):
     def __getattr__(self, key):
         try:
             return object.__getattribute__(self, key)
-        except AttributeError:
+        except AttributeError as e:
             try:
                 return self[key]
             except NotImplementedError:
-                raise AttributeError()
+                raise e
 
     def __dir__(self):
         return sorted(set(list(dir(type(self))) + list(self.columns)))
@@ -805,4 +831,28 @@ def get(dsk, keys, get=get_sync, **kwargs):
     return get(dsk2, keys, **kwargs)  # use synchronous scheduler for now
 
 
+def pd_split(df, p, seed=0):
+    """ Split DataFrame into multiple pieces pseudorandomly
+
+    >>> df = pd.DataFrame({'a': [1, 2, 3, 4, 5, 6],
+    ...                    'b': [2, 3, 4, 5, 6, 7]})
+
+    >>> a, b = pd_split(df, [0.5, 0.5], seed=123)  # roughly 50/50 split
+    >>> a
+       a  b
+    1  2  3
+    2  3  4
+    5  6  7
+
+    >>> b
+       a  b
+    0  1  2
+    3  4  5
+    4  5  6
+    """
+    p = list(p)
+    index = pseudorandom(len(df), p, seed)
+    return [df.iloc[index == i] for i in range(len(p))]
+
+
 from .shuffle import set_index, set_partition
diff --git a/dask/utils.py b/dask/utils.py
--- a/dask/utils.py
+++ b/dask/utils.py
@@ -193,3 +193,27 @@ def concrete(seq):
 
 def skip(func):
     pass
+
+
+def pseudorandom(n, p, key):
+    """ Pseudorandom array of integer indexes
+
+    >>> pseudorandom(5, [0.5, 0.5], key=123)
+    array([1, 0, 0, 1, 1], dtype=int8)
+
+    >>> pseudorandom(10, [0.5, 0.2, 0.2, 0.1], key=5)
+    array([0, 2, 0, 3, 0, 1, 2, 1, 0, 0], dtype=int8)
+    """
+    import numpy as np
+    p = list(p)
+    cp = np.cumsum([0] + p)
+    assert np.allclose(1, cp[-1])
+    assert len(p) < 256
+
+    np.random.seed(key)
+    x = np.random.random(n)
+    out = np.empty(n, dtype='i1')
+
+    for i, (low, high) in enumerate(zip(cp[:-1], cp[1:])):
+        out[(x >= low) & (x < high)] = i
+    return out


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -7,7 +7,7 @@
 import dask
 from dask.utils import raises
 import dask.dataframe as dd
-from dask.dataframe.core import get, concat
+from dask.dataframe.core import get, concat, pd_split
 
 
 def eq(a, b):
@@ -372,3 +372,11 @@ def test_dataframe_series_are_dillable():
     e = d.groupby(d.a).b.sum()
     f = dill.loads(dill.dumps(e))
     assert eq(e, f)
+
+
+def test_random_partitions():
+    a, b = d.random_split([0.5, 0.5])
+    assert isinstance(a, dd.DataFrame)
+    assert isinstance(b, dd.DataFrame)
+
+    assert len(a.compute()) + len(b.compute()) == len(full)