Repository: pandas-dev/pandas

Issue:
ENH: Option for reading files with a variable number of comment lines at start
http://stackoverflow.com/questions/14276661/python-pandas-read-file-skipping-commented



Discussion:
@wesm: I'm the author of the stackoverflow question. 

> Implementing this well would require dipping into the C tokenizer code. It's not as bad as it might sound.

If you can point me to where this is implemented, so I can give a look, see what I understand and if I can contribute?

`pandas/src/parser/tokenizer.c`

I have spent a number of hours in the last days trying to figure out which function calls which(, and is not an easy task).
The error that I get when reading a file (in my case with read_table, but my understanding is that it would be the same with read_csv) with commented line I get an error: 

```
pandas/io/parsers.py", line 1475, in _rows_to_cols
   raise ValueError(msg) 
ValueError: Expected 0 fields in line 7, saw 7content
```

The error comes from within the `PythonParser` class. Before calling `_rows_to_cols` the input file is stored in a list of lists, `content` and fully commented lines appears as an empty list.
So a possibility would be to add an extra keyword `skipcommentedlines` (or similar) and if `True` count how many empty lines from the beginning of the lines there are and take them out from `content`. If `self.orif_names` is also empty, count the number of fields, and fill it with `range(n_fields)`.

If this makes sense, I'll implement it asap.

It looks to me that the functions in `pandas/src/parser/tokenizer.c` are used only in `class CParserWrapper` when the 'c' engine is required. From a comment I understand that the 'c' engine is not supported yet. For this I think that I have found a couple of places where the skipping of commented lines can be implemented, although I have to understand if it makes sense and how to implement it.

At last. When reading a file of the kind

```
 # line1 (space before the comment)
# line2
# [...]
# line 6 
1 2 3 4 5 6 7
[...]
```

with `pd.read_table(fname, header=None, comment='#', sep='\s')` I have noticed that if I have 2 commented lines, `content` has two empty lists in the first two positions (as I would expecting), for 1 or >=3 commented lines I get n-1 empty lists in `content`. It does look quite strange to me. Any idea of what's going on?

Thanks for the time reading (and hopefully replying) to this long post

I would very much like to see this feature implemented as I have to deal with workarounds for it often.

Is there a reason that the above implementation would not work?  It seems simple enough.  I have never contributed to pandas before and would like some feedback before proceeding to implement the feature.

the c-parser is the primary parser; this change would have to be made there (actually in src/parser.pyx). its not that hard, but a bit non-trivial

I'll give a look at `src/parser.pyx` asap. I hope to have more luck than when I looked into `pandas/src/parser/tokenizer.c`

this is an already existing feature to skip comments at the end of the line

http://pandas.pydata.org/pandas-docs/dev/io.html#comments

IIUC you want to skip a line if there is a comment at the beginning?

e.g.

```
# comment
data.....
# comment
data....
   # comment
```

if you specified `comment='\s*#'` (I am not sure it takes re ATM though)

the last one would be tough FYI, tokenzier only takes a single character ATM

@montefra Whichever of us gets to it first then.  The race is on. ;-)

@jreback That is correct, I want to skip lines with a comment character at the beginning.  I would imagine the last case could not be handled in the event the file is space delimited.  Basically, comments probably need to either be declared at the very beginning or end of the line.

I have implemented comment skipping and also fixed a related problem with CSV format sniffing.  I did so by modifying `io/parsers.py` and have not yet begun work on the c-parser which apparently my install of pandas is not using by default.  Once I have finished making changes in `parser.pyx` I will push my changes.

A similar problem is whether pandas should ignore empty lines by default.  It would be very easy to implement as a slight extension of ignoring comments.  I will open up an issue related on this to get some feedback.

gre8!

you can put your changes up as a PR, be sure to enable travis, lmk if you need help

Only [one build failed](https://travis-ci.org/holocronweaver/pandas/builds/9918797) in Travis CI, but it does not appear to have anything to do with my changes.  Is this expected, or did I break something related to pytables somehow?

@holocronweaver rebase on master again I just pushed some code to 'fix' that failure (although net net the changes didn't actually do antything)...but seemed to fix it

to force travis to rebuild

```
git commit -amend -C HEAD
git push origin yourbranch -f
```

this resets the last commit to a new hash forcing a rebuild

I did as you suggested and the latest build (3) is all good.  I will go ahead and submit a pull request.

@montefra doing a PR for this?

@jreback: ehm. 
I did look into it but I kind of forgot after @holocronweaver did is PR. 
So the answer is no. 
When I did spend some time (about 9 months ago) I couldn't figure out much `pandas/src/parser/tokenizer.c` and proposed a solution, which is likely part of what @holocronweaver has done.

ok...thanks 

@jreback @montefra Yep, I have taken care of this, just need to find time to get everything ready for PR.

@holocronweaver @montefra can either of you update the PR for this?

@jreback Will do so the moment I have a chance.  Long time coming, I know.  =) High on my priority list.


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -64,9 +64,11 @@ class ParserWarning(Warning):
     pass ``header=0`` to be able to replace existing names. The header can be
     a list of integers that specify row locations for a multi-index on the
     columns E.g. [0,1,3]. Intervening rows that are not specified will be
-    skipped. (E.g. 2 in this example are skipped)
+    skipped (e.g. 2 in this example are skipped). Note that this parameter
+    ignores commented lines, so header=0 denotes the first line of
+    data rather than the first line of the file.
 skiprows : list-like or integer
-    Row numbers to skip (0-indexed) or number of rows to skip (int)
+    Line numbers to skip (0-indexed) or number of lines to skip (int)
     at the start of the file
 index_col : int or sequence or False, default None
     Column to use as the row labels of the DataFrame. If a sequence is given, a
@@ -106,8 +108,12 @@ class ParserWarning(Warning):
 thousands : str, default None
     Thousands separator
 comment : str, default None
-    Indicates remainder of line should not be parsed
-    Does not support line commenting (will return empty line)
+    Indicates remainder of line should not be parsed. If found at the
+    beginning of a line, the line will be ignored altogether. This parameter
+    must be a single character. Also, fully commented lines
+    are ignored by the parameter `header` but not by `skiprows`. For example,
+    if comment='#', parsing '#empty\n1,2,3\na,b,c' with `header=0` will
+    result in '1,2,3' being treated as the header.
 decimal : str, default '.'
     Character to recognize as decimal point. E.g. use ',' for European data
 nrows : int, default None
@@ -1313,6 +1319,7 @@ def __init__(self, f, **kwds):
         self.data = None
         self.buf = []
         self.pos = 0
+        self.line_pos = 0
 
         self.encoding = kwds['encoding']
         self.compression = kwds['compression']
@@ -1459,6 +1466,7 @@ class MyDialect(csv.Dialect):
                 line = self._check_comments([line])[0]
 
                 self.pos += 1
+                self.line_pos += 1
                 sniffed = csv.Sniffer().sniff(line)
                 dia.delimiter = sniffed.delimiter
                 if self.encoding is not None:
@@ -1566,7 +1574,7 @@ def _infer_columns(self):
         if self.header is not None:
             header = self.header
 
-            # we have a mi columns, so read and extra line
+            # we have a mi columns, so read an extra line
             if isinstance(header, (list, tuple, np.ndarray)):
                 have_mi_columns = True
                 header = list(header) + [header[-1] + 1]
@@ -1578,9 +1586,8 @@ def _infer_columns(self):
             for level, hr in enumerate(header):
                 line = self._buffered_line()
 
-                while self.pos <= hr:
+                while self.line_pos <= hr:
                     line = self._next_line()
-
                 unnamed_count = 0
                 this_columns = []
                 for i, c in enumerate(line):
@@ -1705,25 +1712,36 @@ def _buffered_line(self):
         else:
             return self._next_line()
 
+    def _empty(self, line):
+        return not line or all(not x for x in line)
+
     def _next_line(self):
         if isinstance(self.data, list):
             while self.pos in self.skiprows:
                 self.pos += 1
 
-            try:
-                line = self.data[self.pos]
-            except IndexError:
-                raise StopIteration
+            while True:
+                try:
+                    line = self._check_comments([self.data[self.pos]])[0]
+                    self.pos += 1
+                    # either uncommented or blank to begin with
+                    if self._empty(self.data[self.pos - 1]) or line:
+                        break
+                except IndexError:
+                    raise StopIteration
         else:
             while self.pos in self.skiprows:
                 next(self.data)
                 self.pos += 1
 
-            line = next(self.data)
-
-        line = self._check_comments([line])[0]
+            while True:
+                orig_line = next(self.data)
+                line = self._check_comments([orig_line])[0]
+                self.pos += 1
+                if self._empty(orig_line) or line:
+                    break
 
-        self.pos += 1
+        self.line_pos += 1
         self.buf.append(line)
 
         return line


Test patch:
diff --git a/pandas/io/tests/test_parsers.py b/pandas/io/tests/test_parsers.py
--- a/pandas/io/tests/test_parsers.py
+++ b/pandas/io/tests/test_parsers.py
@@ -1584,6 +1584,65 @@ def test_read_table_buglet_4x_multiindex(self):
             df = self.read_table(StringIO(text), sep='\s+')
             self.assertEqual(df.index.names, ('one', 'two', 'three', 'four'))
 
+    def test_line_comment(self):
+        data = """# empty
+A,B,C
+1,2.,4.#hello world
+#ignore this line
+5.,NaN,10.0
+"""
+        expected = [[1., 2., 4.],
+                    [5., np.nan, 10.]]
+        df = self.read_csv(StringIO(data), comment='#')
+        tm.assert_almost_equal(df.values, expected)
+
+    def test_comment_skiprows(self):
+        data = """# empty
+random line
+# second empty line
+1,2,3
+A,B,C
+1,2.,4.
+5.,NaN,10.0
+"""
+        expected = [[1., 2., 4.],
+                    [5., np.nan, 10.]]
+        # this should ignore the first four lines (including comments)
+        df = self.read_csv(StringIO(data), comment='#', skiprows=4)
+        tm.assert_almost_equal(df.values, expected)
+
+    def test_comment_header(self):
+        data = """# empty
+# second empty line
+1,2,3
+A,B,C
+1,2.,4.
+5.,NaN,10.0
+"""
+        expected = [[1., 2., 4.],
+                    [5., np.nan, 10.]]
+        # header should begin at the second non-comment line
+        df = self.read_csv(StringIO(data), comment='#', header=1)
+        tm.assert_almost_equal(df.values, expected)
+
+    def test_comment_skiprows_header(self):
+        data = """# empty
+# second empty line
+# third empty line
+X,Y,Z
+1,2,3
+A,B,C
+1,2.,4.
+5.,NaN,10.0
+"""
+        expected = [[1., 2., 4.],
+                    [5., np.nan, 10.]]
+        # skiprows should skip the first 4 lines (including comments), while
+        # header should start from the second non-commented line starting
+        # with line 5
+        df = self.read_csv(StringIO(data), comment='#', skiprows=4, header=1)
+        tm.assert_almost_equal(df.values, expected)
+
     def test_read_csv_parse_simple_list(self):
         text = """foo
 bar baz
@@ -2874,6 +2933,65 @@ def test_parse_dates_empty_string(self):
     def test_usecols(self):
         raise nose.SkipTest("Usecols is not supported in C High Memory engine.")
 
+    def test_line_comment(self):
+        data = """# empty
+A,B,C
+1,2.,4.#hello world
+#ignore this line
+5.,NaN,10.0
+"""
+        expected = [[1., 2., 4.],
+                    [5., np.nan, 10.]]
+        df = self.read_csv(StringIO(data), comment='#')
+        tm.assert_almost_equal(df.values, expected)
+
+    def test_comment_skiprows(self):
+        data = """# empty
+random line
+# second empty line
+1,2,3
+A,B,C
+1,2.,4.
+5.,NaN,10.0
+"""
+        expected = [[1., 2., 4.],
+                    [5., np.nan, 10.]]
+        # this should ignore the first four lines (including comments)
+        df = self.read_csv(StringIO(data), comment='#', skiprows=4)
+        tm.assert_almost_equal(df.values, expected)
+
+    def test_comment_header(self):
+        data = """# empty
+# second empty line
+1,2,3
+A,B,C
+1,2.,4.
+5.,NaN,10.0
+"""
+        expected = [[1., 2., 4.],
+                    [5., np.nan, 10.]]
+        # header should begin at the second non-comment line
+        df = self.read_csv(StringIO(data), comment='#', header=1)
+        tm.assert_almost_equal(df.values, expected)
+
+    def test_comment_skiprows_header(self):
+        data = """# empty
+# second empty line
+# third empty line
+X,Y,Z
+1,2,3
+A,B,C
+1,2.,4.
+5.,NaN,10.0
+"""
+        expected = [[1., 2., 4.],
+                    [5., np.nan, 10.]]
+        # skiprows should skip the first 4 lines (including comments), while
+        # header should start from the second non-commented line starting
+        # with line 5
+        df = self.read_csv(StringIO(data), comment='#', skiprows=4, header=1)
+        tm.assert_almost_equal(df.values, expected)
+
     def test_passing_dtype(self):
         # GH 6607
         # This is a copy which should eventually be merged into ParserTests