Repository: dask/dask

Issue:
nrows= in dd.read_csv should fail loudly
We don't support this keyword argument.  We should fail loudly if someone tries to use it.



Discussion:
You could add the 'bad' keywords as 'namedarg=None' and catch any of them as non-none, but then you're implying to people trying to tab-complete that they are valid parameters.

An anti-**slots** variable might work. This allows you to keep the **kwargs while checking them against known bad values and throwing an error on them.

I imagine this could be handled with fairly mundane technologies as follows:

``` python
for name in ['nrows', ... ]:
    if name in kwargs:
        raise ValueError(...)
```

I'd swap the order since individual kwargs _may belong to_ the set of  bad arguments.

We _do_ support `nrows`, but not with glob:

``` python
In [1]: %load test.py

In [2]: # %load test.py
import dask.dataframe as dd
import pandas as pd


cols = ['year', 'month', 'day_of_month', 'day_of_week', 'deptime',
        'crs_deptime', 'arrtime', 'crs_arrtime', 'unique_carrier',
        'flight_num', 'tail_num', 'actual_elapsed_time', 'crs_elapsed_time',
        'air_time', 'arrdelay', 'depdelay', 'origin', 'dest', 'distance',
        'taxi_in', 'taxi_out', 'cancelled', 'cancellation_code', 'diverted',
        'carrier_delay', 'weather_delay', 'nas_delay', 'security_delay',
        'late_aircraft_delay']

dtypes = {'cancellation_code': object, 'taxi_in': float, 'taxi_out': float,
          'cancelled': bool, 'diverted': bool, 'carrier_delay': float,
          'weather_delay': float, 'nas_delay': float, 'security_delay': float,
          'late_aircraft_delay': float, 'tail_num': object, 'crs_deptime':
          float, 'crs_arrtime': float, 'flight_num': float, 'crs_elapsed_time':
          float, 'distance': float}

ddata = dd.read_csv('2000.csv', nrows=1234, header=0, names=cols,
                    dtype=dtypes)
   ...:

In [3]: len(ddata)
Out[3]: 1234

In [4]: ddata
Out[4]: dd.DataFrame<from-de..., npartitions=1>
```

All rows are read into a single partition. Support is here: https://github.com/dask/dask/blob/master/dask/dataframe/csv.py#L227.

Should we raise or change behavior if someone tries to use nrows with glob?

I suggest we either:
- Remove this support, as I don't see the need for it in dask (although @ijstokes was using it here: https://github.com/dask/dask/issues/1497)
- Error informatively when using `nrows` with glob, as that use is non-intuitive.

I'm leaning towards the second, but either sounds fine.

Isn't the current behaviour in https://github.com/dask/dask/blob/master/dask/dataframe/csv.py#L227 actually wrong? It takes just the first partition, dropping the rest of rows even if `nrows` was larger than the number of rows in the first partition.

Example:

```
In [25]: len(read_csv('train_5M.csv'))
Out[25]: 4999999
In [26]: len(read_csv('train_5M.csv', nrows=3000000))
Out[26]: 1495477
```

I'd expect the second count to be 3000000 rows.

Can this be done "right" at a reasonable cost? I mean, to be able to know how many partitions must be inspected to get the desired number of rows, one has to count the number of rows of all partitions.

Suggestions:
- Remove support entirely.
- Keep it but document that only the first partition will be used in the arguments.

Both suggestions sound good to me.  I'm a bit in favor of removing support and pointing users towards the `.head()` method.


Resolving patch:
diff --git a/dask/dataframe/csv.py b/dask/dataframe/csv.py
--- a/dask/dataframe/csv.py
+++ b/dask/dataframe/csv.py
@@ -199,6 +199,10 @@ def read_csv(urlpath, blocksize=AUTO_BLOCKSIZE, chunkbytes=None,
     for kw in ['iterator', 'chunksize']:
         if kw in kwargs:
             raise ValueError("%s not supported for dd.read_csv" % kw)
+    if kwargs.get('nrows', None):
+        raise ValueError("The 'nrows' keyword is not supported by "
+                         "`dd.read_csv`. To achieve the same behavior, it's "
+                         "recommended to use `dd.read_csv(...).head(n=nrows)`")
     if isinstance(kwargs.get('skiprows'), list):
         raise TypeError("List of skiprows not supported for dd.read_csv")
     if isinstance(kwargs.get('header'), list):
@@ -224,9 +228,6 @@ def read_csv(urlpath, blocksize=AUTO_BLOCKSIZE, chunkbytes=None,
     if not isinstance(values[0], (tuple, list)):
         values = [values]
 
-    if 'nrows' in kwargs:
-        values = [[values[0][0]]]
-
     if kwargs.get('header', 'infer') is None:
         header = b''
     else:


Test patch:
diff --git a/dask/dataframe/tests/test_io.py b/dask/dataframe/tests/test_io.py
--- a/dask/dataframe/tests/test_io.py
+++ b/dask/dataframe/tests/test_io.py
@@ -1261,14 +1261,6 @@ def test_to_csv_series():
     assert (result.x == df0).all()
 
 
-def test_read_csv_with_nrows():
-    with filetext(text) as fn:
-        f = dd.read_csv(fn, nrows=3)
-        assert list(f.columns) == ['name', 'amount']
-        assert f.npartitions == 1
-        assert eq(dd.read_csv(fn, nrows=3), pd.read_csv(fn, nrows=3))
-
-
 def test_read_csv_raises_on_no_files():
     fn = '.not.a.real.file.csv'
     try: