Repository: dask/dask

Issue:
Error when reading a .bz2 file with dask 0.7.5
Read plain CSV works.
Read bz2 compressed version fails, assume the pandas read_csv capability is not fully supported.
Note I also tested with dtype definition, same error.

partial strack trace
        df = dd.read_csv(..., dtype={'my-column': float})

```
Pandas has given us the following error when trying to parse the file:

  "utf-8"
```
# 
## Traceback

  File "lib\dask\async.py", line 263, in execute_task
    result = _execute_task(task, data)
  File "lib\dask\async.py", line 245, in _execute_task
    return func(*args2)
  File "lib\dask\dataframe\io.py", line 69, in _read_csv
    raise ValueError(msg)



Discussion:
Looks like you have an encoding issue. A few things to try:
- We just released dask 0.7.6. Try upgrading, and see if that fixes your problem.
- Try passing in the encoding directly to the read_csv call (`encoding='utf-8'`). The default is your system encoding, which might not be "utf-8".

Will do, however I'm seeing other odd behavior after I 'load' the data.
df.head() runs 'for a very long time' never returns a value, 6 million rows
4 columns.

On Fri, Jan 8, 2016 at 10:05 AM, Jim Crist notifications@github.com wrote:

> Looks like you have an encoding issue. A few things to try:
> - We just released dask 0.7.6. Try upgrading, and see if that fixes
>   your problem.
> - Try passing in the encoding directly to the read_csv call (
>   encoding='utf-8'). The default is your system encoding, which might
>   not be "utf-8".
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/blaze/dask/issues/909#issuecomment-170026237.

Yeah, that's probably due to how we're parsing the files. I'm unable to reproduce using a `bz2`, non ascii encoded csv that I quickly made. If you can create a small, reproducible test case (if your data isn't sharable, a small bz2 file of fake data that exhibits the same problem would also work) for us that would greatly help in debugging this.

The file encoding might also not be `'utf-8'` - pandas is just throwing a `ValueError('utf-8')`, which is less than informative. If you're comfortable mucking in source code, we could modify the line where dask swallows the traceback, which would help in debugging.

I can share a sample if i can get your email address.​

On Fri, Jan 8, 2016 at 10:19 AM, Jim Crist notifications@github.com wrote:

> Yeah, that's probably due to how we're parsing the files. I'm unable to
> reproduce using a bz2, non ascii encoded csv that I quickly made. If you
> can create a small, reproducible test case (if your data isn't sharable, a
> small bz2 file of fake data that exhibits the same problem would also work)
> for us that would greatly help in debugging this.
> 
> The file encoding might also not be 'utf-8' - pandas is just throwing a
> ValueError('utf-8'), which is less than informative. If you're
> comfortable mucking in source code, we could modify the line where dask
> swallows the traceback, which would help in debugging.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/blaze/dask/issues/909#issuecomment-170029392.

Note: I'm using Python 3.5.1.

On Fri, Jan 8, 2016 at 10:19 AM, Jim Crist notifications@github.com wrote:

> Yeah, that's probably due to how we're parsing the files. I'm unable to
> reproduce using a bz2, non ascii encoded csv that I quickly made. If you
> can create a small, reproducible test case (if your data isn't sharable, a
> small bz2 file of fake data that exhibits the same problem would also work)
> for us that would greatly help in debugging this.
> 
> The file encoding might also not be 'utf-8' - pandas is just throwing a
> ValueError('utf-8'), which is less than informative. If you're
> comfortable mucking in source code, we could modify the line where dask
> swallows the traceback, which would help in debugging.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/blaze/dask/issues/909#issuecomment-170029392.

> I can share a sample if i can get your email address.​

jcrist at continuum dot io

Note that there is also a [pretty serious performance flaw](https://github.com/blaze/dask/issues/901) with `dask.dataframe.read_csv` and compressed files.  If you have the space it's worthwhile to unzip ahead of time (also because bz2 decompression is very very slow, and something you only want to do once.)

Understand, I am testing, bz2 is a hit, compressed file system files on
windows seem optimal.

On Fri, Jan 8, 2016 at 10:36 AM, Matthew Rocklin notifications@github.com
wrote:

> Note that there is also a pretty serious performance flaw
> https://github.com/blaze/dask/issues/901 with dask.dataframe.read_csv
> and compressed files. If you have the space it's worthwhile to unzip ahead
> of time (also because bz2 decompression is very very slow, and something
> you only want to do once.)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/blaze/dask/issues/909#issuecomment-170033450.


Resolving patch:
diff --git a/dask/dataframe/io.py b/dask/dataframe/io.py
--- a/dask/dataframe/io.py
+++ b/dask/dataframe/io.py
@@ -16,7 +16,8 @@
 from toolz import merge, assoc, dissoc
 
 from ..compatibility import StringIO, unicode, range, apply
-from ..utils import textblock, file_size, get_bom, system_encoding
+from ..utils import (textblock, file_size, get_bom, system_encoding,
+                     infer_compression)
 from ..base import tokenize
 from .. import array as da
 from ..async import get_sync
@@ -27,7 +28,7 @@
 
 
 lock = Lock()
-csv_defaults = {'compression': None}
+csv_defaults = {'compression': 'infer'}
 
 
 def _read_csv(fn, i, chunkbytes, compression, kwargs, bom):
@@ -157,6 +158,9 @@ def fill_kwargs(fn, **kwargs):
             raise ValueError("No files found matching name %s" % fn)
         fn = filenames[0]
 
+    if kwargs['compression'] == 'infer':
+        kwargs['compression'] = infer_compression(fn)
+
     if 'names' not in kwargs:
         kwargs['names'] = csv_names(fn, **kwargs)
         if 'header' not in kwargs:


Test patch:
diff --git a/dask/dataframe/tests/test_io.py b/dask/dataframe/tests/test_io.py
--- a/dask/dataframe/tests/test_io.py
+++ b/dask/dataframe/tests/test_io.py
@@ -1,4 +1,5 @@
 import gzip
+from itertools import product
 import pandas as pd
 import numpy as np
 import pandas.util.testing as tm
@@ -18,7 +19,7 @@
         from_array, from_bcolz, from_dask_array)
 from dask.compatibility import StringIO, BZ2File, GzipFile
 
-from dask.utils import filetext, tmpfile, ignoring
+from dask.utils import filetext, tmpfile, ignoring, compressions
 from dask.async import get_sync
 
 from dask.dataframe.utils import eq
@@ -39,11 +40,15 @@
 """.strip()
 
 
-@pytest.mark.parametrize('myopen,compression',
-                         [(open, None), (GzipFile, 'gzip'), (BZ2File, 'bz2')])
-def test_read_csv(myopen, compression):
+@pytest.mark.parametrize('open_comp_pair,infer',
+                         list(product([(open, None), (GzipFile, 'gzip'),
+                                       (BZ2File, 'bz2')], (True, False))))
+def test_read_csv(open_comp_pair, infer):
+    myopen, compression = open_comp_pair
     text_ = text if compression is None else text.encode()
-    with filetext(text_, open=myopen) as fn:
+    ext = dict((v, k) for (k, v) in compressions.items()).get(compression, '')
+    with filetext(text_, open=myopen, extension=ext) as fn:
+        compression = 'infer' if infer else compression
         f = dd.read_csv(fn, chunkbytes=30, compression=compression,
                 lineterminator='\n')
         assert list(f.columns) == ['name', 'amount']