Repository: pandas-dev/pandas

Issue:
Bug with read_table, skiprows, and C engine
I'm reading the file available at ftp://aftp.cmdl.noaa.gov/products/trends/co2/co2_mm_mlo.txt. The data start on line 73.

If I use the default C engine with `read_table` I have to specify `skiprows=85` to properly load the table:

``` python
pd.read_table(
        'co2_mm_mlo.txt.', sep=r'\s+', header=None, skiprows=85, engine='c',
        names=['year', 'month', 'dec_year', 'average', 'interpolated', 'trend', 'days'])
```

But if I use the Python engine then the expected `skiprows=72` works:

``` python
pd.read_table(
        'co2_mm_mlo.txt.', sep=r'\s+', header=None, skiprows=72, engine='python',
        names=['year', 'month', 'dec_year', 'average', 'interpolated', 'trend', 'days'])
```

The resulting DataFrame is expected to have 679 rows, but has 691 rows and data from the header if I use `skiprows=72` with the C engine.

I've confirmed this behavior on Mac OS X Yosemite with Pandas 0.15.0 and a checkout of master@5cf3d85a7d4c448519fa08f918a114209cfbdf2b.

memory error when skipping rows
I have a file with over 100Million rows. When I do 
pd.read_csv(filename, skiprows=100000000, iterator=True)
python crashes with a memory error. I have 32 gigs of memory and python eats up all that memory!!



Discussion:
so if you specify `comment='#'` the c-parser gives the same result. Must be an interactions on the skiprows/skip_blank_lines (e.g. somehow things its a 'real' line).

prob related #8661 

cc @mdmueller

This seems to be related to the problem with skiprows, header=None and lines ending in a trailing space.  I downloaded the file, removed all trailing spaces and was able to read the data correctly with skiprows=72 and engine='c'.  Mac OS X Yosemite, python 3.4.2 from python.org and pandas version '0.15.0-85-gaf2bfb7'

if you specify `chunksize=1000000` (or something I think it might help).

That said, still could be a bug. Would appreciate you having a deeper look if you can.

prob related #8661, #8679

cc @mdmueller

Hi,
I tried your suggestion. Unfortunately it did not really work. I dont have a dev version of pandas installed. However I was reading through the code, and I suspect the issue could be the use of lrange(skiprows)
        if com.is_integer(skiprows):
            skiprows = lrange(skiprows)
if skiprows is 100M, then lrange will generate a very large array. 

I know that skiprows can be a array of ints or a int. I think that if its simply an int, then we should use a more efficient code path to skip the rows instead of generating an 100M size array.

yep that sounds right
but would need some modifications in the code to deal with that

have a go!

Do you have any suggestions on how best to tackle this issue

Sent from my iPhone

> On Oct 30, 2014, at 9:03 AM, jreback notifications@github.com wrote:
> 
> yep that sounds right
> but would need some modifications in the code to deal with that
> 
> have a go!
> 
> —
> Reply to this email directly or view it on GitHub.

so currently the skipped rows are transformed to a list of rows with range (this is why this blows memory up). You could instead change the impl a bit to pass a list of a list was original passed (e.g. `skiprows=[0,3,5]`, and then pass say `slice` to represent the interval. This object appears as `.skiprows` in `parser.pyx` and then is directly passed to the `tokenizer.c`

You prob need to change the c-impl as well (so maybe make the original `skiprows_list`, and the new one `skiprows_slice`). The original is handled just as it was, the new can do a range change (this is all done in c-land). to just skip if the `start >= slice.start and end < slice.end`.

So a bit involved, but would be a nice change.

@cpcloud 


Resolving patch:
diff --git a/vb_suite/io_bench.py b/vb_suite/io_bench.py
--- a/vb_suite/io_bench.py
+++ b/vb_suite/io_bench.py
@@ -21,6 +21,22 @@
 read_csv_standard = Benchmark("read_csv('__test__.csv')", setup1,
                               start_date=datetime(2011, 9, 15))
 
+#----------------------------------
+# skiprows
+
+setup1 = common_setup + """
+index = tm.makeStringIndex(20000)
+df = DataFrame({'float1' : randn(20000),
+                'float2' : randn(20000),
+                'string1' : ['foo'] * 20000,
+                'bool1' : [True] * 20000,
+                'int1' : np.random.randint(0, 200000, size=20000)},
+               index=index)
+df.to_csv('__test__.csv')
+"""
+
+read_csv_skiprows = Benchmark("read_csv('__test__.csv', skiprows=10000)", setup1,
+                              start_date=datetime(2011, 9, 15))
 
 #----------------------------------------------------------------------
 # write_csv


Test patch:
diff --git a/pandas/io/tests/test_parsers.py b/pandas/io/tests/test_parsers.py
--- a/pandas/io/tests/test_parsers.py
+++ b/pandas/io/tests/test_parsers.py
@@ -3048,6 +3048,29 @@ def test_comment_skiprows(self):
         df = self.read_csv(StringIO(data), comment='#', skiprows=4)
         tm.assert_almost_equal(df.values, expected)
 
+    def test_trailing_spaces(self):
+        data = """skip
+random line with trailing spaces    
+skip
+1,2,3
+1,2.,4.
+random line with trailing tabs\t\t\t
+     
+5.,NaN,10.0
+"""
+        expected = pd.DataFrame([[1., 2., 4.],
+                    [5., np.nan, 10.]])
+        # this should ignore six lines including lines with trailing 
+        # whitespace and blank lines.  issues 8661, 8679
+        df = self.read_csv(StringIO(data.replace(',', '  ')), 
+                           header=None, delim_whitespace=True,
+                           skiprows=[0,1,2,3,5,6], skip_blank_lines=True)
+        tm.assert_frame_equal(df, expected)
+        df = self.read_table(StringIO(data.replace(',', '  ')), 
+                             header=None, delim_whitespace=True,
+                             skiprows=[0,1,2,3,5,6], skip_blank_lines=True)
+        tm.assert_frame_equal(df, expected)
+
     def test_comment_header(self):
         data = """# empty
 # second empty line