Repository: dask/dask

Issue:
CParserError in read_csv after #1116
With #1116  merged I get a `CParserError` for some csv file I was able to read before:

```
CParserError: Too many columns specified: expected 16 and found 1
```

I still need to find the offending lines that cause the error as I can't share the original files, but I already wanted to state the problem here.



Discussion:
Can you say a bit more about the keyword arguments used in the read_csv call?  Anything non-trivial?  Are your line-terminators standard?

I usually debug these things by using the synchronous scheduler

``` python
import dask
dask.set_options(get=dask.get)
```

... and then see what keyword arguments and what block of bytes was given to `pd.read_csv`

Still though, the fact that new errors are popping up means that we still have work to do

I have `lineterminator='\n'` and `header=None`, even did not specify names for testing. Perhaps it's an '\r' inside the data, when I'm thinking about this, I had a problem once with that already ... will check soon on this.

I actually just ran into and fixed a `header=None` issue at https://github.com/dask/dask/pull/1136.  Perhaps related?

@wabu did #1136 resolve this?

nope, I still get the error. It actually is the handling of `lineterminator` and \r's:

```
with filetext("""
0,my
1,data
2,foo\rbar
""".strip()) as fn:    
    pandy = pd.read_csv(fn, header=None, sep=',', lineterminator='\n', names=['a','b'])
    dasky = dd.read_csv(fn, header=None, sep=',', lineterminator='\n', names=['a','b']).compute()
print(dasky)
print('--')
print(pandy)
```

```
     a     b
0    0    my
1    1  data
0    2   foo
1  bar   NaN
--
   a         b
0  0        my
1  1      data
2  2  foo\rbar
```

It can also cause error when the block boundaries are different:

```
with filetext("""0,my
1,data
""" * 1000 +
"""2,foo\rbar
""".strip()) as fn:
    dasky = dd.read_csv(fn, header=None, sep=',', lineterminator='\n', names=['a','b'], blocksize=200).compute()
```

Hrm, the first example works ok on my machine.  Can you list your platform, pandas version and dask git commit?

``` python
In [1]: !cat foo.csv
0,my
1,data
2,foo\rbar

In [2]: import pandas as pd

In [3]: pd.read_csv('foo.csv', header=None, sep=',', lineterminator='\n', names=['a', 'b'])
Out[3]: 
   a         b
0  0        my
1  1      data
2  2  foo\rbar

In [4]: import dask.dataframe as dd

In [5]: dd.read_csv('foo.csv', header=None, sep=',', lineterminator='\n', names=['a', 'b']).compute()
Out[5]: 
   a         b
0  0        my
1  1      data
2  2  foo\rbar
```

The second example does fail for me because it oddly interprets `foo\rbar` as `baroo`.  Not sure what's going on there.

``` python
In [6]: !head bar.csv
0,my
1,data
0,my
1,data
0,my
1,data
0,my
1,data
0,my
1,data

In [7]: !tail bar.csv
1,data
0,my
1,data
0,my
1,data
0,my
1,data
0,my
1,data
baroo
In [8]: len(pd.read_csv('bar.csv', header=None, sep=',', lineterminator='\n', names=['a', 'b']))
Out[8]: 2001

In [9]: len(dd.read_csv('bar.csv', header=None, sep=',', lineterminator='\n', names=['a', 'b'], blocksize=200))
ValueError: invalid literal for int() with base 10: 'bar'
```

Did your really put a the '\r' (0x0D) into the foo.csv, or a '\\r' (0x1C 0x32)? When I use cat on my foo.csv, the terminal interprets the linefeed, overwriting `2,f` with `bar` resulting in `baroo` showing up for me there too.

## 

dask: aab772d...
python: 3.5.1.final.0
python-bits: 64
OS: Linux
OS-release: 3.2.0-99-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
pandas: 0.18.0+99.ga1f5ef3

Ah, I probably generated the file as text rather than as bytes.  One sec.


Resolving patch:
diff --git a/dask/dataframe/csv.py b/dask/dataframe/csv.py
--- a/dask/dataframe/csv.py
+++ b/dask/dataframe/csv.py
@@ -141,6 +141,8 @@ def read_csv(filename, blocksize=2**25, chunkbytes=None,
     **kwargs: dict
         Options to pass down to ``pandas.read_csv``
     """
+    if len(lineterminator) == 1:
+        kwargs['lineterminator'] = lineterminator
     if chunkbytes is not None:
         warn("Deprecation warning: chunksize csv keyword renamed to blocksize")
         blocksize=chunkbytes


Test patch:
diff --git a/dask/dataframe/tests/test_io.py b/dask/dataframe/tests/test_io.py
--- a/dask/dataframe/tests/test_io.py
+++ b/dask/dataframe/tests/test_io.py
@@ -1044,3 +1044,10 @@ def test_to_hdf_kwargs():
     df2 = pd.read_hdf('tst.h5', 'foo4')
 
     tm.assert_frame_equal(df, df2)
+
+
+def test_read_csv_slash_r():
+    data = b'0,my\n1,data\n' * 1000 + b'2,foo\rbar'
+    with filetext(data, mode='wb') as fn:
+        dd.read_csv(fn, header=None, sep=',', lineterminator='\n',
+                    names=['a','b'], blocksize=200).compute(get=dask.get)