Repository: dask/dask

Issue:
Reading csv with separator in python dask gives error
I am trying to create a DataFrame by reading a csv file separated by '#####' 5 hashes

The code is:

import dask.dataframe as dd
df = dd.read_csv('D:\temp.csv',sep='#####',engine='python')
res = df.compute()
Error is:

dask.async.ValueError:
Dask dataframe inspected the first 1,000 rows of your csv file to guess the
data types of your columns.  These first 1,000 rows led us to an incorrect
guess.

For example a column may have had integers in the first 1000
rows followed by a float or missing value in the 1,001-st row.

You will need to specify some dtype information explicitly using the
`dtype=` keyword argument for the right column names and dtypes.

```
df = dd.read_csv(..., dtype={'my-column': float})
```

Pandas has given us the following error when trying to parse the file:

  "The 'dtype' option is not supported with the 'python' engine"

Traceback

---

File "/home/ec2-user/anaconda3/lib/python3.4/site-packages/dask/async.py", line 263, in execute_task
result = _execute_task(task, data)
File "/home/ec2-user/anaconda3/lib/python3.4/site-packages/dask/async.py", line 245, in _execute_task
return func(*args2)
File "/home/ec2-user/anaconda3/lib/python3.4/site-packages/dask/dataframe/io.py", line 69, in _read_csv
raise ValueError(msg)
So how to get rid of that.

If i follow the error then i would have to give dtype for every column, but if I have a 100+ columns then that is of no use.

And if i am reading without separator,then everything goes fine but there is ##### everywhere. So after computing it to pandas DataFrame ,is there a way to get rid of that?

So help me in this.



Discussion:
Why did you choose the `engine='python'` option?

if i am executing 
     df = dd.read_csv('D:\temp.csv',sep='#####')
Then getting warning likely to use a engine python.
So suggest if any better way/style to achieve that.

ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators; you can avoid this warning by specifying engine='python'

@mrocklin please suggest a way to read csv with separator in dask dataframe

That's a warning/error from pandas, not dask. The `dtype=` and `sep=` arguments cannot be used at the same time. Two or more character separators are not possible with the C engine, while `dtype` is not possible with the python engine.

Your best bet is either to
1. preprocess the file to replace all occurrences of `'#####'` with a single character like `'#'`
2. read it in with dask using `'#'` and the separator, strip all the trailing `'####'` and convert  the dtypes appropriately using the typical [pandas methods](http://pandas.pydata.org/pandas-docs/version/0.17.1/text.html#text-string-methods). I don't thing there's anything for dask to do here.

`dask.dataframe.read_csv` uses the `dtype=` kwarg to internal calls to `pd.read_csv`.  We do this to ensure consistency across the different partitions.  It sounds like we shouldn't do this when certain other keywords are active.


Resolving patch:
diff --git a/dask/dataframe/io.py b/dask/dataframe/io.py
--- a/dask/dataframe/io.py
+++ b/dask/dataframe/io.py
@@ -110,6 +110,7 @@ def clean_kwargs(kwargs):
             or isinstance(col, (tuple, list)) and all(c in kwargs['usecols']
                                                       for c in col)
             or col in kwargs['usecols']]
+
     return kwargs
 
 
@@ -179,20 +180,21 @@ def fill_kwargs(fn, **kwargs):
         kwargs['parse_dates'] = [col for col in head.dtypes.index
                            if np.issubdtype(head.dtypes[col], np.datetime64)]
 
-    new_dtype = dict(head.dtypes)
-    dtype = kwargs.get('dtype', dict())
-    for k, v in dict(head.dtypes).items():
-        if k not in dtype:
-            dtype[k] = v
-
-    if kwargs.get('parse_dates'):
-        for col in kwargs['parse_dates']:
-            if isinstance(col, (tuple, list)):
-                del dtype['_'.join(col)]
-            else:
-                del dtype[col]
+    if 'sep' not in kwargs:
+        new_dtype = dict(head.dtypes)
+        dtype = kwargs.get('dtype', dict())
+        for k, v in dict(head.dtypes).items():
+            if k not in dtype:
+                dtype[k] = v
+
+        if kwargs.get('parse_dates'):
+            for col in kwargs['parse_dates']:
+                if isinstance(col, (tuple, list)):
+                    del dtype['_'.join(col)]
+                else:
+                    del dtype[col]
 
-    kwargs['dtype'] = dtype
+        kwargs['dtype'] = dtype
 
     return (head.columns.map(lambda s: s.strip() if isinstance(s, str) else s),
             kwargs)


Test patch:
diff --git a/dask/dataframe/tests/test_io.py b/dask/dataframe/tests/test_io.py
--- a/dask/dataframe/tests/test_io.py
+++ b/dask/dataframe/tests/test_io.py
@@ -845,3 +845,18 @@ def test_parse_dates_multi_column():
 
         assert (df.columns == ddf.columns).all()
         assert len(df) == len(ddf)
+
+
+sep_text = """
+name###amount
+alice###100
+bob###200
+charlie###300"""
+
+def test_read_csv_sep():
+    with filetext(sep_text) as fn:
+        ddf = dd.read_csv(fn, sep="###")
+        df = pd.read_csv(fn, sep="###")
+
+        assert (df.columns == ddf.columns).all()
+        assert len(df) == len(ddf)