Repository: pandas-dev/pandas

Issue:
read_table fails with MultiIndex input and delim_whitespace=True
Related #6889

Example:

``` python
In [4]: text = """                      A       B       C       D        E
one two three   four
a   b   10.0032 5    -0.5109 -2.3358 -0.4645  0.05076  0.3640
a   q   20      4     0.4473  1.4152  0.2834  1.00661  0.1744
x   q   30      3    -0.6662 -0.5243 -0.3580  0.89145  2.5838"""

In [5]: pd.read_table(StringIO(text), delim_whitespace=True)
---------------------------------------------------------------------------
CParserError                              Traceback (most recent call last)
. . .
CParserError: Error tokenizing data. C error: Expected 6 fields in line 3, saw 9
```

This (partially) works if `delim_whitespace=True` is replaced with `sep='\s+', engine='python'` (although columns A-D are lost):

``` python
In [6]: pd.read_table(StringIO(text), sep='\s+', engine='python')
Out[6]: 
                           E
one two three   four        
a   b   10.0032 5     0.3640
    q   20.0000 4     0.1744
x   q   30.0000 3     2.5838

[3 rows x 1 columns]
```



Discussion:
@mcwitt can you address for 0.14?

Yep, I've made some progress on this. Should have a PR soon...

OK, I've fixed the bug in PythonParser. The C parser seems to have its own code for this, and I'm not sure yet whether it had been designed to support specifying a MultiIndex in this way.

hmm. its only the header detection code which is pretty straightforward (e.g. you have to read the header line-by-line based on the spec `header=[0,1]` whatever, one odd thing is you may or may not have an extra row for the row names depending on how it was written), then turning that to a MI is all the same for both parsers (and is python code)

> hmm. its only the header detection code

I think this is a separate issue from header detection. The original title of this issue was wrong, this doesn't have to do with MI columns. This has to do with automatically setting the index columns to the 2nd row when the sum of the 1st and 2nd rows equals the 3rd, i.e.

``` python
In [8]: text = """                      A       B       C       D        E
one two three   four
a   b   10.0032 5    -0.5109 -2.3358 -0.4645  0.05076  0.3640
a   q   20      4     0.4473  1.4152  0.2834  1.00661  0.1744
x   q   30      3    -0.6662 -0.5243 -0.3580  0.89145  2.5838"""

In [9]: pd.read_table(StringIO(text), sep='\s+', engine='python')
Out[9]: 
                           A       B       C        D       E
one two three   four                                         
a   b   10.0032 5    -0.5109 -2.3358 -0.4645  0.05076  0.3640
    q   20.0000 4     0.4473  1.4152  0.2834  1.00661  0.1744
x   q   30.0000 3    -0.6662 -0.5243 -0.3580  0.89145  2.5838

[3 rows x 5 columns]
```

The code for dealing with this in the python parser is in `_get_index_name`, but the C parser doesn't seem to have anything similar yet.

Made a PR for the fix for the python parser. The fix for the C parser will be a bit more involved so I think it would be good to do this in a separate PR. Essentially I think we need to duplicate the functionality in `PythonParser._get_index_name` in `CParserWrapper` initially, and then merge as much of the code as possible into a common function for both parsers.

can you detect the condition in c-parser and raise?

moving this to 0.14.1 (though we'll merge your pythonparse fix after release notes).

and if you can detect the condition in the c-parser would be great (and just raise for now)

@mcwitt any thoughts on fixing the c-parser for this?

I'm still thinking about this. I think I will try to implement the fix described above, but I will be too busy to look at it for the next two weeks (traveling to a conference). I should be able to get it done for 0.14.1 though.

gr8!


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -1079,6 +1079,10 @@ def __init__(self, src, **kwds):
 
         self.orig_names = self.names
 
+        # index_col may be specified on line after the header
+        if self.index_col is None:
+            self.index_col = self._reader.index_col
+
         if not self._has_complex_date_col:
             if (self._reader.leading_cols == 0 and
                     _is_index_col(self.index_col)):


Test patch:
diff --git a/pandas/io/tests/test_parsers.py b/pandas/io/tests/test_parsers.py
--- a/pandas/io/tests/test_parsers.py
+++ b/pandas/io/tests/test_parsers.py
@@ -1568,21 +1568,22 @@ def test_converter_return_string_bug(self):
         self.assertEqual(df2['Number1'].dtype, float)
 
     def test_read_table_buglet_4x_multiindex(self):
-        # GH 6607
-        # Parsing multi-level index currently causes an error in the C parser.
-        # Temporarily copied to TestPythonParser.
-        # Here test that CParserError is raised:
-
-        with tm.assertRaises(CParserError):
-            text = """                      A       B       C       D        E
+        text = """                      A       B       C       D        E
 one two three   four
 a   b   10.0032 5    -0.5109 -2.3358 -0.4645  0.05076  0.3640
 a   q   20      4     0.4473  1.4152  0.2834  1.00661  0.1744
 x   q   30      3    -0.6662 -0.5243 -0.3580  0.89145  2.5838"""
 
-            # it works!
-            df = self.read_table(StringIO(text), sep='\s+')
-            self.assertEqual(df.index.names, ('one', 'two', 'three', 'four'))
+        # it works!
+        df = self.read_table(StringIO(text), sep='\s+')
+        self.assertEqual(df.index.names, ('one', 'two', 'three', 'four'))
+
+        # GH 6893
+        data = '      A B C\na b c\n1 3 7 0 3 6\n3 1 4 1 5 9'
+        expected = DataFrame.from_records([(1,3,7,0,3,6), (3,1,4,1,5,9)],
+                columns=list('abcABC'), index=list('abc'))
+        actual = self.read_table(StringIO(data), sep='\s+')
+        tm.assert_frame_equal(actual, expected)
 
     def test_read_csv_parse_simple_list(self):
         text = """foo
@@ -2713,28 +2714,6 @@ def test_decompression_regex_sep(self):
             self.assertRaises(ValueError, self.read_csv,
                               path, compression='bz3')
 
-    def test_read_table_buglet_4x_multiindex(self):
-        # GH 6607
-        # This is a copy which should eventually be merged into ParserTests
-        # when the issue with multi-level index is fixed in the C parser.
-
-        text = """                      A       B       C       D        E
-one two three   four
-a   b   10.0032 5    -0.5109 -2.3358 -0.4645  0.05076  0.3640
-a   q   20      4     0.4473  1.4152  0.2834  1.00661  0.1744
-x   q   30      3    -0.6662 -0.5243 -0.3580  0.89145  2.5838"""
-
-        # it works!
-        df = self.read_table(StringIO(text), sep='\s+')
-        self.assertEqual(df.index.names, ('one', 'two', 'three', 'four'))
-
-        # GH 6893
-        data = '      A B C\na b c\n1 3 7 0 3 6\n3 1 4 1 5 9'
-        expected = DataFrame.from_records([(1,3,7,0,3,6), (3,1,4,1,5,9)],
-                columns=list('abcABC'), index=list('abc'))
-        actual = self.read_table(StringIO(data), sep='\s+')
-        tm.assert_frame_equal(actual, expected)
-
 class TestFwfColspaceSniffing(tm.TestCase):
     def test_full_file(self):
         # File with all values