Repository: dask/dask

Issue:
Issue when reading csv file
Hello,

I have put some issues with dask/blaze for csv file reading (file can be read in pandas)...
Re-posting here:

When running this file here: 
https://goo.gl/xVOK4s

I got this message (even if tried the un-compressed csv file)
(I un gzip the file manually with winrar, it was ok.)

```
Traceback (most recent call last):
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\IPython\core\formatters.py", line 222, in catch_format_error
    r = method(self, *args, **kwargs)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\IPython\core\formatters.py", line 699, in __call__
    printer.pretty(obj)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\IPython\lib\pretty.py", line 383, in pretty
    return _default_pprint(obj, self, cycle)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\IPython\lib\pretty.py", line 503, in _default_pprint
    _repr_pprint(obj, p, cycle)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\IPython\lib\pretty.py", line 694, in _repr_pprint
    output = repr(obj)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\blaze\interactive.py", line 444, in _choose_repr
    return expr_repr(self)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\blaze\interactive.py", line 354, in expr_repr
    return repr(coerce_scalar(compute(expr), str(expr.dshape)))
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\multipledispatch\dispatcher.py", line 164, in __call__
    return func(*args, **kwargs)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\blaze\interactive.py", line 205, in compute
Out[7]:     return compute(expr, resources, **kwargs)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\multipledispatch\dispatcher.py", line 164, in __call__
    return func(*args, **kwargs)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\blaze\compute\core.py", line 394, in compute
    for e, dat in d2.items()
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\blaze\compute\core.py", line 395, in <genexpr>
    if e in expr2
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\multipledispatch\dispatcher.py", line 164, in __call__
    return func(*args, **kwargs)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\blaze\compute\csv.py", line 61, in pre_compute
    return dd.read_csv(data.path, **kwargs)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\dataframe\csv.py", line 226, in read_csv
    **(storage_options or {}))
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\bytes\core.py", line 203, in read_bytes
    ensure_protocol(protocol)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\bytes\core.py", line 387, in ensure_protocol
    raise ValueError("Unknown protocol %s" % protocol)
ValueError: Unknown protocol e
```

1) Do you have an idea how to read the file in Blaze ?

2) Also do you have regression tests on a seies of various real-life data file (even small size) to ensure that reading/writing various datatypes /files are fine (because I got another issue with HDF5 file writing before...,  it sounds in_out have some format reading/writing issues in Blaze...) ?




https://github.com/blaze/blaze/issues/1608


Discussion:
When reading gzipped CSV files you'll have to add the `compression='gzip'` keyword argument when calling `dask.dataframe.read_csv`

See https://github.com/dask/dask/blob/master/dask/dataframe/io/tests/test_csv.py for CSV tests.  Tests for other types are in that same directory.
Hello,

1)
Thanks, but this is directly the unzipped file where dask/blaze cannot read
(error is on tjhe de-compressed csv file).
Pandas can read it.


2) One possible way to bullet proof test on real dataset is to let community
upload dataset (reduced size) + write test code (standardize) into a folder.
And when testing, run the various folders.
Better than hand made small dataset for testing....(it is complementary kind of tests).




Matthew Rocklin wrote:
>
> When reading gzipped CSV files you'll have to add the 
> |compression='gzip'| keyword argument when calling 
> |dask.dataframe.read_csv|
>
> See 
> https://github.com/dask/dask/blob/master/dask/dataframe/io/tests/test_csv.py 
> for CSV tests. Tests for other types are in that same directory.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub 
> <https://github.com/dask/dask/issues/1885#issuecomment-270423229>, or 
> mute the thread 
> <https://github.com/notifications/unsubscribe-auth/AR10p9feUD94VrH4E8drIqsCdEmKJIn-ks5rO89igaJpZM4Lam3c>.
>

There are several files there.  Can you provide a minimal file and a `dask.dataframe.read_csv` call that reproduces your error?
File is   transactions.csv   (2Go compressed, 20Go uncompressed).

Code is 
```
import blaze as bz
file2= 'transactions_red3.csv'
df_bz= bz.data( file2)
df_bz.dshape
bz.by( df_bz.chain ,  col1=  df_bz.purchaseamount.sum() )
```

I did several tests (all on the un-compressed csv file).
  1) Reduced file (from initial file) to 10Mo :  Ok, no pb.
  2) Reduced file of 200 Mo: Ok, no pb.
  3) Reduced file of 1.7 Go: error happens.

it seems related to the size of the csv file

Upload the 1.7Go (195mo compressed) faulty file here : 
https://goo.gl/cBw0cy




Yeah, ideally you would reproduce the error with a small kilobyte sized csv file and just dask.dataframe calls.  I don't maintain Blaze or track its development, so including a failing example that depends on it means that you're much less likely to get much time from me trying to track things down.
File is here.
https://drive.google.com/drive/folders/0B0YzHEQ2aKc0T1hKUnhKdGhMN3c

Code with dask:

```
import dask
file3= 'transactions_red2.csv'
df= dask.dataframe.csv.read_csv(file3)

Traceback (most recent call last):
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\IPython\core\interactiveshell.py", line 2885, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-65-c0127464fa7c>", line 2, in <module>
    df= dask.dataframe.csv.read_csv(file3)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\dataframe\csv.py", line 226, in read_csv
    **(storage_options or {}))
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\bytes\core.py", line 203, in read_bytes
    ensure_protocol(protocol)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\bytes\core.py", line 387, in ensure_protocol
    raise ValueError("Unknown protocol %s" % protocol)
ValueError: Unknown protocol e
```








what is file3?

Ideally you provide a small file and a copy-pasteable example that I can use to reproduce the error on my machine, without downloading a very large file or understanding more than 5-10 lines of code.

See http://stackoverflow.com/help/mcve
Another example with csv file of 0.6 kb
File is available here :
https://drive.google.com/drive/folders/0B0YzHEQ2aKc0T1hKUnhKdGhMN3c?usp=sharing
```
# dask==0.13.0
df = dd.read_csv('user_groups.csv')

Traceback (most recent call last):
  File "<ipython-input-10-5da5f1ff8007>", line 1, in <module>
    df = dd.read_csv(in2 + 'user_groups.csv')
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\dataframe\io\csv.py", line 271, in read
    **kwargs)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\dataframe\io\csv.py", line 196, in read_pandas
    **(storage_options or {}))
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\bytes\core.py", line 145, in read_bytes
    None, **kwargs)

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\bytes\core.py", line 389, in get_fs_paths_myopen
    encoding=encoding, **kwargs)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\bytes\core.py", line 258, in __init__
    ensure_protocol(self.protocol)
  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\bytes\core.py", line 483, in ensure_protocol
    raise ValueError("Unknown protocol %s" % protocol)
```

Am forced to load it via Pandas  and re-save it in HDF....




I can load your file without any problems:
```
dask.__version__
>> 0.13
```

```
dd.read_csv('/absolute/path/to/user_groups.csv').compute()

 	id 	name 	created_at 	updated_at
0 	2 	Coke Sampling Phase1 	2016-09-28 03:32:21.930045 	2016-09-28 03:32:21.930045
1 	3 	DL month 201604 	2016-11-29 04:37:01.074817 	2016-11-29 04:37:01.074817
2 	4 	DL month 201605 	2016-11-29 04:44:51.55732 	2016-11-29 04:44:51.55732
3 	5 	DL month 201606 	2016-11-29 04:50:55.742028 	2016-11-29 04:50:55.742028
4 	6 	DL month 201607 	2016-11-29 05:13:12.856052 	2016-11-29 05:13:12.856052
5 	9 	DL month201608 	2016-11-30 06:10:53.632778 	2016-11-30 06:10:53.632778
6 	10 	DL month 201608 	2016-11-30 06:20:09.237565 	2016-11-30 06:20:09.237565
7 	11 	DL month 201609 	2016-12-01 12:01:34.039066 	2016-12-01 12:01:34.039066
```

Did you try to provide an absolute path?
Thank you !
Found the issue: related to the parsing of the path in windows.
Windows 7.0 64bits, Python 2.7 (Anaconda)
DASK 0.13  Pandas 0.19.1

Below does work : 
```
import dask.dataframe as dd
df = dd.read_csv(urlpath= 'E:\_data\csv\user_groups.csv'')  # DASK

df= pd.read_csv('E:/_data/csv/user_groups.csv')          #Pandas   slash /
df = pd.read_csv( 'E:\_data\csv\user_groups.csv')     #  Pandas    anti slash \
```

Does not work :   
```
df= dd.read_csv('E:/_data/csv/user_groups.csv'')          # Dask

Traceback (most recent call last):

  File "<ipython-input-17-f89f00b9a956>", line 1, in <module>
    df = dd.read_csv(urlpath= 'E:/_data/csv/user_groups.csv', sep=',')

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\dataframe\io\csv.py", line 271, in read
    **kwargs)

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\dataframe\io\csv.py", line 196, in read_pandas
    **(storage_options or {}))

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\bytes\core.py", line 145, in read_bytes
    None, **kwargs)

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\bytes\core.py", line 389, in get_fs_paths_myopen
    encoding=encoding, **kwargs)

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\bytes\core.py", line 258, in __init__
    ensure_protocol(self.protocol)

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dask\bytes\core.py", line 483, in ensure_protocol
    raise ValueError("Unknown protocol %s" % protocol)

ValueError: Unknown protocol e
```










cc @frol 
I will make a special handling for this case. Meanwhile, either use Windows backslashes or prefix the path with explicit `file://` (e.g. `file://E:/file.csv`)
generally you use ``r'file:\\server\location\file'`` on windows. This *should* be just a documentation  note.
Not really.
We use /folder1/folder2/  in windows in order to have  
the same folder suffix  than Linux/Unix suffix (slash ).

Reason:

1) FULL_path = OSPrefix + LocalPathSuffix
    (code can be executed without changing local path and OSPath is determined at startup).

2) Pandas manages the slash and anti-slash well, so make it useful.

3) We need to add hack to change path from Pandas, numpy,   ... to Dask, like this:
   filepath_dask=  filepath_pandas.replace('/', '\\')







On 11 Jan 2017, at 00:56, Jeff Reback <notifications@github.com> wrote:

generally you use r'file:\\server\location\file' on windows. This should be just a documentation note.

―
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.



Resolving patch:
diff --git a/dask/utils.py b/dask/utils.py
--- a/dask/utils.py
+++ b/dask/utils.py
@@ -800,15 +800,23 @@ def infer_storage_options(urlpath, inherit_storage_options=None):
     "url_query": "q=1", "extra": "value"}
     """
     # Handle Windows paths including disk name in this special case
-    if re.match(r'^[a-zA-Z]:\\', urlpath):
+    if re.match(r'^[a-zA-Z]:[\\/]', urlpath):
         return {'protocol': 'file',
                 'path': urlpath}
 
     parsed_path = urlsplit(urlpath)
+    protocol = parsed_path.scheme or 'file'
+    path = parsed_path.path
+    if protocol == 'file':
+        # Special case parsing file protocol URL on Windows according to:
+        # https://msdn.microsoft.com/en-us/library/jj710207.aspx
+        windows_path = re.match(r'^/([a-zA-Z])[:|]([\\/].*)$', path)
+        if windows_path:
+            path = '%s:%s' % windows_path.groups()
 
     inferred_storage_options = {
-        'protocol': parsed_path.scheme or 'file',
-        'path': parsed_path.path,
+        'protocol': protocol,
+        'path': path,
     }
 
     if parsed_path.netloc:


Test patch:
diff --git a/dask/tests/test_utils.py b/dask/tests/test_utils.py
--- a/dask/tests/test_utils.py
+++ b/dask/tests/test_utils.py
@@ -178,9 +178,18 @@ def test_infer_storage_options():
         infer_storage_options('hdfs:///bucket/file.csv', {'protocol': 'collide'})
 
 
-def test_infer_storage_options_c():
-    so = infer_storage_options(r'c:\foo\bar')
+@pytest.mark.parametrize('urlpath, expected_path', (
+    (r'c:\foo\bar', r'c:\foo\bar'),
+    (r'C:\\foo\bar', r'C:\\foo\bar'),
+    (r'c:/foo/bar', r'c:/foo/bar'),
+    (r'file:///c|\foo\bar', r'c:\foo\bar'),
+    (r'file:///C|/foo/bar', r'C:/foo/bar'),
+    (r'file:///C:/foo/bar', r'C:/foo/bar'),
+))
+def test_infer_storage_options_c(urlpath, expected_path):
+    so = infer_storage_options(urlpath)
     assert so['protocol'] == 'file'
+    assert so['path'] == expected_path
 
 
 def test_eq_strict():