Repository: pandas-dev/pandas

Issue:
skipfooter doesn't really "skip" in read_csv
On `master`:

``` python
from pandas import read_csv
from pandas.compat import StringIO
data = 'a,b,c\ncat,foo,bar\ndog,foo,"baz'  # Note the stray quotation mark
read_csv(StringIO(data), engine='python', skipfooter=1)
...
_csv.Error: unexpected end of data
```

If we were truly "skipping" the last row, no error should have been raised.  However, this occurs because the data is all parsed <em>in memory</em> first with Python's `csv` library.

Whether this is intended behaviour or not has implications for the C engine in terms of implementing analogous `skipfooter` behaviour.  Or perhaps it has something to do with the fact that `error_bad_lines` and `error_warn_lines` parameters not with the Python engine?

xref #5232



Discussion:

Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -2411,14 +2411,23 @@ def _next_line(self):
                 try:
                     orig_line = next(self.data)
                 except csv.Error as e:
+                    msg = str(e)
+
                     if 'NULL byte' in str(e):
-                        raise csv.Error(
-                            'NULL byte detected. This byte '
-                            'cannot be processed in Python\'s '
-                            'native csv library at the moment, '
-                            'so please pass in engine=\'c\' instead.')
-                    else:
-                        raise
+                        msg = ('NULL byte detected. This byte '
+                               'cannot be processed in Python\'s '
+                               'native csv library at the moment, '
+                               'so please pass in engine=\'c\' instead')
+
+                    if self.skipfooter > 0:
+                        reason = ('Error could possibly be due to '
+                                  'parsing errors in the skipped footer rows '
+                                  '(the skipfooter keyword is only applied '
+                                  'after Python\'s csv library has parsed '
+                                  'all rows).')
+                        msg += '. ' + reason
+
+                    raise csv.Error(msg)
                 line = self._check_comments([orig_line])[0]
                 self.pos += 1
                 if (not self.skip_blank_lines and


Test patch:
diff --git a/pandas/io/tests/parser/python_parser_only.py b/pandas/io/tests/parser/python_parser_only.py
--- a/pandas/io/tests/parser/python_parser_only.py
+++ b/pandas/io/tests/parser/python_parser_only.py
@@ -221,3 +221,18 @@ def test_multi_char_sep_quotes(self):
             with tm.assertRaisesRegexp(ValueError, msg):
                 self.read_csv(StringIO(data), sep=',,',
                               quoting=csv.QUOTE_NONE)
+
+    def test_skipfooter_bad_row(self):
+        # see gh-13879
+
+        data = 'a,b,c\ncat,foo,bar\ndog,foo,"baz'
+        msg = 'parsing errors in the skipped footer rows'
+
+        with tm.assertRaisesRegexp(csv.Error, msg):
+            self.read_csv(StringIO(data), skipfooter=1)
+
+        # We expect no match, so there should be an assertion
+        # error out of the inner context manager.
+        with tm.assertRaises(AssertionError):
+            with tm.assertRaisesRegexp(csv.Error, msg):
+                self.read_csv(StringIO(data))