Repository: dask/dask

Issue:
read_csv sampling logic is not a generalized solution
I have hit an issue with a dataset which has over 10k columns, so I obviously hit a bug with a misleading error message:

```
/opt/conda/lib/python3.5/site-packages/dask/async.py in get_async(apply_async, num_workers, dsk, result, cache, queue, get_id, raise_on_exception, rerun_exceptions_locally, callbacks, **kwargs)
    486                 _execute_task(task, data)  # Re-execute locally
    487             else:
--> 488                 raise(remote_exception(res, tb))
    489         state['cache'][key] = res
    490         finish_task(dsk, key, state, results, keyorder.get)

ValueError: ('Columns do not match', Index(['Unnamed: 0', 'Var1', 'Var2', 'Var3', 'Var4', 'Var5', 'Var6', 'Var7',
       'Var8', 'Var9',...
```

Once I bumped the `sample` size to `1MB` I got through the above error but hit another one described in a related issue #1166, when sample size is not enough to guess dtypes (I have very sparse character columns with a number of missings (>95%), which is not a so rare use-case).

I do understand that from a performance point of view it is not a great idea to scan the whole dataset twice, but can be there implemented an option to force full scan (in fact, early stopping once `object` dtype is discovered can save some time and full scan won't take so long)?



Discussion:
Regarding the first error, that can definitely be fixed, your csv was wide enough that it wasn't grabbing all of the columns in the sample.

If the dtypes are known, you can pass in the dtypes to `read_csv`, and that will avoid your issue. In general I'd prefer to avoid recommending a full read to users, so making that cheap is a bit undesirable. Passing in a very large number to `sample` should also work, although it's a bit of a hack.

@jcrist Passing a larger `sample` makes it work for this specific case, just like it did for @koverholt in #1166. However, increasing the size of `sample` will only make it harder to catch the issue next time.

Perhaps this issue can be resolved with a separate function that does do a full pass and determines dtypes.  This would be separate from read_csv, isolating an uncommon case, but would be mentioned somewhere in a docstring or exception, removing the burden from the user to figure out how to perform the full scan.

Thoughts?

@mrocklin This will be fine for me. However, I was thinking that there must be an option in the reader itself to ensure that a dataset will be read without weird issues... I would even assume this to be a default behavior with an option to switch to "hope we guessed correctly" mode.

In common cases this degrades performance sufficiently that it feels wrong to make it the default.  In general I'm completely for opting into performance and out of correctness.  But in the common case of reading data from disk `dd.read_csv` would start taking tens of minutes, pretty much destroying any chance of interactivity.  Personally I find the optimistic approach better, especially if we couple it with informative error messages.


Resolving patch:
diff --git a/dask/dataframe/io/csv.py b/dask/dataframe/io/csv.py
--- a/dask/dataframe/io/csv.py
+++ b/dask/dataframe/io/csv.py
@@ -213,22 +213,34 @@ def read_pandas(reader, urlpath, blocksize=AUTO_BLOCKSIZE, collection=True,
                                   compression)
 
     b_lineterminator = lineterminator.encode()
-    sample, values = read_bytes(urlpath, delimiter=b_lineterminator,
-                                blocksize=blocksize,
-                                sample=sample,
-                                compression=compression,
-                                **(storage_options or {}))
+    b_sample, values = read_bytes(urlpath, delimiter=b_lineterminator,
+                                  blocksize=blocksize,
+                                  sample=sample,
+                                  compression=compression,
+                                  **(storage_options or {}))
 
     if not isinstance(values[0], (tuple, list)):
         values = [values]
 
-    if kwargs.get('header', 'infer') is None:
-        header = b''
-    else:
-        header_row = kwargs.get('skiprows', 0)
-        header = sample.split(b_lineterminator)[header_row] + b_lineterminator
-
-    head = reader(BytesIO(sample), **kwargs)
+    # Get header row, and check that sample is long enough. If the file
+    # contains a header row, we need at least 2 nonempty rows + the number of
+    # rows to skip.
+    skiprows = kwargs.get('skiprows', 0)
+    header = kwargs.get('header', 'infer')
+    need = 1 if header is None else 2
+    parts = b_sample.split(b_lineterminator, skiprows + need)
+    # If the last partition is empty, don't count it
+    nparts = 0 if not parts else len(parts) - int(not parts[-1])
+
+    if nparts < skiprows + need and len(b_sample) >= sample:
+        raise ValueError("Sample is not large enough to include at least one "
+                         "row of data. Please increase the number of bytes "
+                         "in `sample` in the call to `read_csv`/`read_table`")
+
+    header = b'' if header is None else parts[skiprows] + b_lineterminator
+
+    # Use sample to infer dtypes
+    head = reader(BytesIO(b_sample), **kwargs)
 
     specified_dtypes = kwargs.get('dtype', {})
     if specified_dtypes is None:


Test patch:
diff --git a/dask/dataframe/io/tests/test_csv.py b/dask/dataframe/io/tests/test_csv.py
--- a/dask/dataframe/io/tests/test_csv.py
+++ b/dask/dataframe/io/tests/test_csv.py
@@ -761,6 +761,37 @@ def test_robust_column_mismatch():
         assert_eq(ddf, ddf)
 
 
+def test_error_if_sample_is_too_small():
+    text = ('AAAAA,BBBBB,CCCCC,DDDDD,EEEEE\n'
+            '1,2,3,4,5\n'
+            '6,7,8,9,10\n'
+            '11,12,13,14,15')
+    with filetext(text) as fn:
+        # Sample size stops mid header row
+        sample = 20
+        with pytest.raises(ValueError):
+            dd.read_csv(fn, sample=sample)
+
+        # Saying no header means this is fine
+        assert_eq(dd.read_csv(fn, sample=sample, header=None),
+                  pd.read_csv(fn, header=None))
+
+    skiptext = ('# skip\n'
+                '# these\n'
+                '# lines\n')
+
+    text = skiptext + text
+    with filetext(text) as fn:
+        # Sample size stops mid header row
+        sample = 20 + len(skiptext)
+        with pytest.raises(ValueError):
+            dd.read_csv(fn, sample=sample, skiprows=3)
+
+        # Saying no header means this is fine
+        assert_eq(dd.read_csv(fn, sample=sample, header=None, skiprows=3),
+                  pd.read_csv(fn, header=None, skiprows=3))
+
+
 ############
 #  to_csv  #
 ############