Repository: pandas-dev/pandas
Pull request: Fix correct warning with c engine when skipping lines
Fixed bug where c engine would not print warnings for lines
it skipped in case the skipped line had an inline comment.
Also, its accounting of number of fields in such lines would
be off by one.

 - [x] closes #16472 
 - [x] tests added
 - [x] tests passed
 - [x] passes ``git diff upstream/master --name-only -- '*.py' | flake8 --diff``
 - [x] whatsnew entry


Problem statement:
Incorrect skipping of lines with inline comments and printing warnings
#### Code Sample, a copy-pastable example if possible

```python
# Your code here
from io import StringIO
import numpy as np
import pandas as pd

test_input = u"""\
1 2
2 2 3
3 2 3 # 3 fields
4 2 3# 3 fields
5 2 # 2 fields
6 2# 2 fields
7 # 1 field, NaN
8# 1 field, NaN
9 2 3 # skipped line
# comment"""

df = pd.read_table(StringIO(test_input), comment='#', header=None,
                   delimiter='\\s+', skiprows=0, error_bad_lines=False)

print df
# Expected: only lines with <= 2 fields should appear in the df, others should be warned as skipped
assert (df == pd.DataFrame([[1, 2], [5, 2], [6, 2], [7, np.nan], [8, np.nan]],
                          index=list(range(5)), columns=[0,1])).all().all()
```

#### Problem description

Only lines with <= 2 fields should appear in the df, others should be skipped and their warning should be printed on stderr.

Output
```
Skipping line 2: expected 2 fields, saw 3
Skipping line 4: expected 2 fields, saw 6
Skipping line 6: expected 2 fields, saw 4

   0  1
0  1  2
1  7  8
```


Problems:

 - Lines skipped due to more fields than expected and which end with inline comments are never printed as skipped on stderr (lines 3-9)
- Lines which end with inline comment after a space count one more field than present, so incorrectly skip or not skip the line (lines 3, 5, 7, 9)
- The incorrect accounting joined the lines 7 and 8 as well which was not expected.

#### Expected Output

```
Skipping line 2: expected 2 fields, saw 3
Skipping line 3: expected 2 fields, saw 3
Skipping line 4: expected 2 fields, saw 3
Skipping line 9: expected 2 fields, saw 3

   0    1
0  1  2.0
1  5  2.0
2  6  2.0
3  7  NaN
4  8  NaN

```

#### Output of ``pd.show_versions()``

<details>
# Paste the output here pd.show_versions() here

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.13.final.0
python-bits: 64
OS: Linux
OS-release: 4.10.14-200.fc25.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.19.2
nose: 1.3.7
pip: 9.0.1
setuptools: 34.3.3
Cython: 0.25.2
numpy: 1.11.3
scipy: 0.19.0
statsmodels: 0.8.0
xarray: None
IPython: 5.3.0
sphinx: None
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.3
blosc: None
bottleneck: None
tables: 3.3.0
numexpr: 2.6.2
matplotlib: 2.0.0
openpyxl: 2.4.1
xlrd: None
xlwt: None
xlsxwriter: None
lxml: 3.7.3
bs4: 4.4.1
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 1.1.6
pymysql: None
psycopg2: None
jinja2: 2.9.5
boto: None
pandas_datareader: 0.2.1
</details>

Hints from issue discussion:

Commit message: Move test to CParserTests

The behavior is only applicable on the `c` engine.

Patch to review:
diff --git a/doc/source/whatsnew/v0.20.2.txt b/doc/source/whatsnew/v0.20.2.txt
index 6d6a148ed025f..6149fb0c392c1 100644
--- a/doc/source/whatsnew/v0.20.2.txt
+++ b/doc/source/whatsnew/v0.20.2.txt
@@ -53,6 +53,7 @@ Indexing
 I/O
 ^^^
 
+- Bug in skipping error lines with inline comments in space delimited text files (:issue:`16472`)
 - Bug that would force importing of the clipboard routines unnecessarily, potentially causing an import error on startup (:issue:`16288`)
 - Bug that raised IndexError HTML-rendering an empty DataFrame (:issue:`15953`)
 
diff --git a/pandas/_libs/src/parser/tokenizer.c b/pandas/_libs/src/parser/tokenizer.c
index 6b0775e54da0c..be23ebb023383 100644
--- a/pandas/_libs/src/parser/tokenizer.c
+++ b/pandas/_libs/src/parser/tokenizer.c
@@ -832,6 +832,9 @@ int tokenize_bytes(parser_t *self, size_t line_limit, int start_lines) {
                 } else if (IS_CARRIAGE(c)) {
                     self->state = EAT_CRNL;
                     break;
+                } else if (IS_COMMENT_CHAR(c)) {
+                    self->state = EAT_COMMENT;
+                    break;
                 } else if (!IS_WHITESPACE(c)) {
                     self->state = START_FIELD;
                     // fall through to subsequent state
diff --git a/pandas/tests/io/parser/c_parser_only.py b/pandas/tests/io/parser/c_parser_only.py
index 3e7a648474bc3..56ac10404b7b2 100644
--- a/pandas/tests/io/parser/c_parser_only.py
+++ b/pandas/tests/io/parser/c_parser_only.py
@@ -7,6 +7,8 @@
 further arguments when parsing.
 """
 
+import sys
+
 import pytest
 import numpy as np
 
@@ -417,3 +419,30 @@ def test_data_after_quote(self):
         expected = DataFrame({'a': ['1', 'ba']})
 
         tm.assert_frame_equal(result, expected)
+
+    @tm.capture_stderr
+    def test_comment_whitespace_delimited(self):
+        test_input = """\
+1 2
+2 2 3
+3 2 3 # 3 fields
+4 2 3# 3 fields
+5 2 # 2 fields
+6 2# 2 fields
+7 # 1 field, NaN
+8# 1 field, NaN
+9 2 3 # skipped line
+# comment"""
+        df = self.read_csv(StringIO(test_input), comment='#', header=None,
+                           delimiter='\\s+', skiprows=0,
+                           error_bad_lines=False)
+        error = sys.stderr.getvalue()
+        # skipped lines 2, 3, 4, 9
+        for line_num in (2, 3, 4, 9):
+            assert 'Skipping line {}'.format(line_num) in error, error
+        expected = DataFrame([[1, 2],
+                              [5, 2],
+                              [6, 2],
+                              [7, np.nan],
+                              [8, np.nan]])
+        tm.assert_frame_equal(df, expected)


Code review comments:
diff_hunk: @@ -53,6 +53,7 @@ Indexing
 I/O
 ^^^
 
+- Bug in skipping error lines with inline comments in space delimited text files (:issue:`16472`)
path: doc/source/whatsnew/v0.20.2.txt
text: @user1:
something like this: Bug in ``pd.read_csv()`` when ``comment`` is passed in space deliminted text files.