Repository: dask/dask

Issue:
skiprows makes Dask skip rows in each partition
```python
import dask.dataframe as dd
import dask

print(dask.__version__)
# 0.14.0

d = dd.read_csv('data.txt', skiprows=1000, dtype=str)
print(d.npartitions)
# 6
print(d.compute().shape[0])
# 2644307
```
```sh
$ wc -l data.txt
2650308 data.txt
```

Apparently, Dask passes `skiprows` to each call of `pandas.read_csv`, whereas when reading a single file it should pass it only to the call that reads the first partition.


Discussion:
Clearly an issue.  Do you think we should pass it once to each file though
or only the first?

On Sat, Mar 11, 2017 at 3:55 AM, Eugene Pakhomov <notifications@github.com>
wrote:

> import dask.dataframe as ddimport dask
> print(dask.__version__)# 0.14.0
>
> d = dd.read_csv('data.txt', skiprows=1000, dtype=str)print(d.npartitions)# 6print(d.compute().shape[0])# 2644307
>
> $ wc -l data.txt
> 2650308 data.txt
>
> Apparently, Dask passes skiprows to each call of pandas.read_csv, whereas
> when reading a single file it should pass it only to the call that reads
> the first partition.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/2065>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszCUVJ2qXtxepZYT3byLlQ3D80yMTks5rkmF8gaJpZM4MaJc8>
> .
>

I'd say to each file. Having only a single file with both data and some not column related header info doesn't sound right.
Of course, someone still may have such situation, but if that someone wants such behavior to be implemented in Dask, I would treat it as a feature.
Do you have any interest in submitting a PR with a proposed fix?  I'd love to get some more hands in this project.  

The relevant function is probably [text_blocks_to_pandas](https://github.com/dask/dask/blob/master/dask/dataframe/io/csv.py#L88-L136) .  That function receives a list of delayed values that point to blocks of bytes from a single file.  You can see how the first block is already singled out.  We would have to do some logic there to manipulate the keyword arguments passed down to different blocks.
Thanks for the starting point. I think I'll be able to spend some time on this during the next week.
Thank you for the effort.

On Sat, Mar 11, 2017 at 8:01 AM, Eugene Pakhomov <notifications@github.com>
wrote:

> Thanks for the starting point. I think I'll be able to spend some time on
> this during the next week.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/2065#issuecomment-285865274>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AASszKPTzsiwl_qticJn69eweQAMZpFGks5rkpsjgaJpZM4MaJc8>
> .
>

And thank you for the bug report regardless.

On Sat, Mar 11, 2017 at 8:03 AM, Matthew Rocklin <mrocklin@continuum.io>
wrote:

> Thank you for the effort.
>
> On Sat, Mar 11, 2017 at 8:01 AM, Eugene Pakhomov <notifications@github.com
> > wrote:
>
>> Thanks for the starting point. I think I'll be able to spend some time on
>> this during the next week.
>>
>> —
>> You are receiving this because you commented.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/dask/dask/issues/2065#issuecomment-285865274>, or mute
>> the thread
>> <https://github.com/notifications/unsubscribe-auth/AASszKPTzsiwl_qticJn69eweQAMZpFGks5rkpsjgaJpZM4MaJc8>
>> .
>>
>
>

Wow, time sure flies.

The fix that involves just manipulation of the kwargs is trivial, but is, strictly speaking, not correct.
What if the number of lines I want to skip is larger than the number of lines in the first block? I know that a block's size is supposed to be in the order of 10-100MB which makes such situation unlikely, but it's still a possibility.
It's good to be aware of such issues, but generally we ignore them.  If we wanted to be very careful we could put in a runtime check that verified the data as we actually read it.  In practice I suspect that adding checks for all corner cases like this would as likely introduce bugs and identify them.

Resolving patch:
diff --git a/dask/dataframe/io/csv.py b/dask/dataframe/io/csv.py
--- a/dask/dataframe/io/csv.py
+++ b/dask/dataframe/io/csv.py
@@ -138,8 +138,10 @@ def text_blocks_to_pandas(reader, block_lists, header, head, kwargs,
                                       dtypes, columns, write_header=False,
                                       enforce=enforce)
         dfs.append(df)
+        rest_kwargs = kwargs.copy()
+        rest_kwargs.pop('skiprows', None)
         for b in blocks[1:]:
-            dfs.append(delayed_pandas_read_text(reader, b, header, kwargs,
+            dfs.append(delayed_pandas_read_text(reader, b, header, rest_kwargs,
                                                 dtypes, columns,
                                                 enforce=enforce))
 
@@ -219,7 +221,8 @@ def read_pandas(reader, urlpath, blocksize=AUTO_BLOCKSIZE, collection=True,
     if kwargs.get('header', 'infer') is None:
         header = b''
     else:
-        header = sample.split(b_lineterminator)[0] + b_lineterminator
+        header_row = kwargs.get('skiprows', 0)
+        header = sample.split(b_lineterminator)[header_row] + b_lineterminator
 
     head = reader(BytesIO(sample), **kwargs)
 


Test patch:
diff --git a/dask/dataframe/io/tests/test_csv.py b/dask/dataframe/io/tests/test_csv.py
--- a/dask/dataframe/io/tests/test_csv.py
+++ b/dask/dataframe/io/tests/test_csv.py
@@ -94,6 +94,11 @@ def normalize_text(s):
 expected = pd.concat([pd.read_csv(BytesIO(csv_files[k]))
                       for k in sorted(csv_files)])
 
+comment_header = b"""# some header lines
+# that may be present
+# in a data file
+# before any data"""
+
 
 csv_and_table = pytest.mark.parametrize('reader,files',
                                         [(pd.read_csv, csv_files),
@@ -191,6 +196,18 @@ def test_text_blocks_to_pandas_blocked(reader, files):
               expected2.reset_index(drop=True), check_dtype=False)
 
 
+@pytest.mark.parametrize('dd_read,pd_read,files',
+                         [(dd.read_csv, pd.read_csv, csv_files),
+                          (dd.read_table, pd.read_table, tsv_files)])
+def test_skiprows(dd_read, pd_read, files):
+    files = {name: comment_header + b'\n' + content for name, content in files.items()}
+    skip = len(comment_header.splitlines())
+    with filetexts(files, mode='b'):
+        df = dd_read('2014-01-*.csv', skiprows=skip)
+        expected_df = pd.concat([pd_read(n, skiprows=skip) for n in sorted(files)])
+        assert_eq(df, expected_df, check_dtype=False)
+
+
 csv_blocks = [[b'aa,bb\n1,1.0\n2,2.0', b'10,20\n30,40'],
               [b'aa,bb\n1,1.0\n2,2.0', b'10,20\n30,40']]
 
diff --git a/dask/tests/test_threaded.py b/dask/tests/test_threaded.py
--- a/dask/tests/test_threaded.py
+++ b/dask/tests/test_threaded.py
@@ -147,7 +147,7 @@ def test_interrupt():
             proc.send_signal(sigint)
             # Wait a bit for it to die
             start = time()
-            while time() - start < 0.5:
+            while time() - start < 2.0:
                 if proc.poll() is not None:
                     break
                 sleep(0.05)