Repository: dask/dask

Issue:
Add to_csv() method to dask/dataframe/io.py
Wondering if to_csv() pandas wrapper could be added, much like what was already done with read_csv().  We'd be happy to test it :)



Discussion:
Would the desired output be a single CSV file or a collection of CSV files?  If a collection then what is the best way to specify the naming scheme?

I think the default should be a single file, just to replicate pandas directly.  I hadn't contemplated the collection output, but could see its applicability.  I'm not sure I can recommend a preferred way to specify a naming scheme, other than to find some other similar naming convention from another package with significant adoption.

Actually,  I think maybe an optional chunksize argument could be specified, and if the length of the output exceeds the chunksize, just use an auto-increment.

e.g.  If no chunksize, then name is 'foo.txt'  If chunksize is specified, then ['foo.txt', 'foo1.txt', 'foo2.txt'....]

I'm not sure if the first 'foo' should be 'foo', 'foo0', or 'foo1'.


Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -653,6 +653,11 @@ def to_castra(self, fn=None, categories=None):
         from .io import to_castra
         return to_castra(self, fn, categories)
 
+    @wraps(pd.DataFrame.to_csv)
+    def to_csv(self, filename, **kwargs):
+        from .io import to_csv
+        return to_csv(self, filename, **kwargs)
+
 
 def _assign(df, *pairs):
     kwargs = dict(partition(2, pairs))
diff --git a/dask/dataframe/io.py b/dask/dataframe/io.py
--- a/dask/dataframe/io.py
+++ b/dask/dataframe/io.py
@@ -593,3 +593,25 @@ def to_castra(df, fn=None, categories=None):
 
     c, _ = get(merge(dsk, df.dask), [(name, -1), list(dsk.keys())])
     return c
+
+
+def to_csv(df, filename, **kwargs):
+    myget = kwargs.pop('get')
+    name = 'to-csv' + next(tokens)
+
+    dsk = dict()
+    dsk[(name, 0)] = (apply, pd.DataFrame.to_csv,
+                        (tuple, [(df._name, 0), filename]),
+                        kwargs)
+
+    kwargs2 = kwargs.copy()
+    kwargs2['mode'] = 'a'
+    kwargs2['header'] = False
+
+    for i in range(1, df.npartitions):
+        dsk[(name, i)] = (_link, (name, i - 1),
+                           (apply, pd.DataFrame.to_csv,
+                             (tuple, [(df._name, i), filename]),
+                             kwargs2))
+
+    get(merge(dsk, df.dask), (name, i), get=myget)


Test patch:
diff --git a/dask/dataframe/tests/test_io.py b/dask/dataframe/tests/test_io.py
--- a/dask/dataframe/tests/test_io.py
+++ b/dask/dataframe/tests/test_io.py
@@ -457,3 +457,16 @@ def test_read_hdf():
         tm.assert_frame_equal(
               dd.read_hdf(fn, '/data', chunksize=2, start=1, stop=3).compute(),
               pd.read_hdf(fn, '/data', start=1, stop=3))
+
+
+def test_to_csv():
+    df = pd.DataFrame({'x': ['a', 'b', 'c', 'd'],
+                       'y': [1, 2, 3, 4]}, index=[1., 2., 3., 4.])
+    a = dd.from_pandas(df, 2)
+
+    with tmpfile('csv') as fn:
+        a.to_csv(fn, get=get_sync)
+
+        result = pd.read_csv(fn, index_col=0)
+
+        tm.assert_frame_equal(result, df)