Repository: pandas-dev/pandas

Issue:
pd.io.parsers.read_csv ignores skiprows when parse_dates is set to a dict
For example:

```
pd.io.parsers.read_csv("http://www.datazoa.com/publish/export.asp?hash=yjPceG6fHL&uid=dzadmin&a=exportcsv",skiprows=range(1,13+1),skipfooter=4,parse_dates={"date":[0]})
```

has 907 rows.

As does:

```
pd.io.parsers.read_csv("http://www.datazoa.com/publish/export.asp?hash=yjPceG6fHL&uid=dzadmin&a=exportcsv",skipfooter=4,parse_dates={"date":[0]})
```

whereas:

```
pd.io.parsers.read_csv("http://www.datazoa.com/publish/export.asp?hash=yjPceG6fHL&uid=dzadmin&a=exportcsv",skiprows=range(1,13+1),skipfooter=4,parse_dates=[0])
```

has 894 rows.

I am on Pandas v0.11.0



Discussion:
This is a symptom of a bigger problem:

```
s = "a,b,c\n" + "\n".join([",".join([str(i), str(i+1), str(i+2)]) for i in xrange(500)])
print pd.read_csv(StringIO(s),skiprows=[200, 202], engine='python')
  <class 'pandas.core.frame.DataFrame'>
  Int64Index: 500 entries, 0 to 499
  Data columns (total 3 columns):
  a    500  non-null values
  b    500  non-null values
  c    500  non-null values
  dtypes: int64(3)
```

Somehow, skiprows got removed in python engine _except_ in the code that sniffs for the header. That's why the header isn't getting properly removed. Fix coming.

I assume the issue exists in both the python and non-python engines? 

Just python. However, since skiprows is defined in the example, read_csv silently fails over to the python engine.

This is a prime example of why I don't like to have silent failover to an unanticipated code path.


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -1283,7 +1283,6 @@ def __init__(self, f, **kwds):
 
         # needs to be cleaned/refactored
         # multiple date column thing turning into a real spaghetti factory
-
         if not self._has_complex_date_col:
             (index_names,
              self.orig_names, _) = self._get_index_name(self.columns)
@@ -1561,8 +1560,6 @@ def _get_index_name(self, columns):
         except StopIteration:
             next_line = None
 
-        index_name = None
-
         # implicitly index_col=0 b/c 1 fewer column names
         implicit_first_cols = 0
         if line is not None:
@@ -1647,11 +1644,20 @@ def _get_lines(self, rows=None):
                 if self.pos > len(source):
                     raise StopIteration
                 if rows is None:
-                    lines.extend(source[self.pos:])
-                    self.pos = len(source)
+                    new_rows = source[self.pos:]
+                    new_pos = len(source)
                 else:
-                    lines.extend(source[self.pos:self.pos + rows])
-                    self.pos += rows
+                    new_rows = source[self.pos:self.pos + rows]
+                    new_pos = self.pos + rows
+
+                # Check for stop rows. n.b.: self.skiprows is a set.
+                if self.skiprows:
+                    new_rows = [row for i, row in enumerate(new_rows)
+                                if i + self.pos not in self.skiprows]
+
+                lines.extend(new_rows)
+                self.pos = new_pos
+
             else:
                 new_rows = []
                 try:
@@ -1673,6 +1679,9 @@ def _get_lines(self, rows=None):
                                     raise Exception(msg)
                                 raise
                 except StopIteration:
+                    if self.skiprows:
+                        new_rows = [row for i, row in enumerate(new_rows)
+                                    if self.pos + i not in self.skiprows]
                     lines.extend(new_rows)
                     if len(lines) == 0:
                         raise


Test patch:
diff --git a/pandas/io/tests/test_parsers.py b/pandas/io/tests/test_parsers.py
--- a/pandas/io/tests/test_parsers.py
+++ b/pandas/io/tests/test_parsers.py
@@ -735,6 +735,14 @@ def test_skiprows_bug(self):
         tm.assert_frame_equal(data, expected)
         tm.assert_frame_equal(data, data2)
 
+    def test_deep_skiprows(self):
+        # GH #4382
+        text = "a,b,c\n" + "\n".join([",".join([str(i), str(i+1), str(i+2)]) for i in range(10)])
+        condensed_text = "a,b,c\n" + "\n".join([",".join([str(i), str(i+1), str(i+2)]) for i in [0, 1, 2, 3, 4, 6, 8, 9]])
+        data = self.read_csv(StringIO(text), skiprows=[6, 8])
+        condensed_data = self.read_csv(StringIO(condensed_text))
+        tm.assert_frame_equal(data, condensed_data)
+
     def test_detect_string_na(self):
         data = """A,B
 foo,bar