Repository: pandas-dev/pandas

Issue:
Bug in read_csv when passing header kwarg?
Seems to lose a row when passing a list as header, check it out:

```
>>> from StringIO import StringIO
>>> import pandas
>>> text = """
a  a  a  b  c  c
q r s t u v
1 2 3 4 5 6
1 2 3 4 5 6
"""
>>> # expected behavior without header
>>> pandas.read_csv(StringIO(text.strip()), sep='\s+')
   a a.1 a.2  b  c c.1
0  q   r   s  t  u   v
1  1   2   3  4  5   6
2  1   2   3  4  5   6
>>> #drops row  with specified header
>>> pandas.read_csv(StringIO(text.strip()), sep='\s+', header=[0, 1])
   (a, q)  (a, r)  (a, s)  (b, t)  (c, u)  (c, v)
0       1       2       3       4       5       6
>>> pandas.read_csv(StringIO(text.strip()), sep='\s+', header=[0, 1], tupleize_cols=False)
   a        b  c
   q  r  s  t  u  v
0  1  2  3  4  5  6
```



Discussion:
interesting...wonder if this is related to the interaction between `header` and `skiprows`...

Nevermind on the last one :P I added a blank line.

So if you add extra rows, it specifically skips what ought to be the third row:

``` python
In [27]: text += """6 5 4 3 2 1
   ....: 6 5 4 3 2 1"""

In [28]:

In [28]: pandas.read_csv(StringIO(text.strip()), sep='\s+')
Out[28]:
   a a.1 a.2  b  c c.1
0  q   r   s  t  u   v
1  1   2   3  4  5   6
2  1   2   3  4  5   6
3  6   5   4  3  2   1
4  6   5   4  3  2   1

In [29]: pandas.read_csv(StringIO(text.strip()), sep='\s+', header=[0, 1], tupleize_cols=False)
Out[29]:
   a        b  c
   q  r  s  t  u  v
0  1  2  3  4  5  6
1  6  5  4  3  2  1
2  6  5  4  3  2  1
```

It's stripping what should be the second row I guess::

``` python
In [33]: text = """
   ....: a  a  a  b  c  c
   ....: q r s t u v
   ....: 0 0 0 0 0 1
   ....: 1 2 3 4 5 6
   ....: 1 2 3 4 5 6
   ....: """

In [34]: pandas.read_csv(StringIO(text.strip()), sep='\s+')
Out[34]:
   a a.1 a.2  b  c c.1
0  q   r   s  t  u   v
1  0   0   0  0  0   1
2  1   2   3  4  5   6
3  1   2   3  4  5   6

In [35]: pandas.read_csv(StringIO(text.strip()), sep='\s+', header=[0, 1], tupleize_cols=False)
Out[35]:
   a        b  c
   q  r  s  t  u  v
0  1  2  3  4  5  6
1  1  2  3  4  5  6
```

@cpcloud maybe it's an off-by-one error? (e.g., it passes a skiprows because of the header and it's passing one higher than it should)

no, this is expected, as you MUST have a row of row names (which it skips); I couldn't find a reliable way to detect that

http://pandas.pydata.org/pandas-docs/dev/io.html#reading-columns-with-a-multiindex

see the last note; I suppose in theory you could look for row names that don't have the full columns, but at the time seemed complicated.

coould be a bug too (though your format is not conforming to what its supposed to read).

I just misread. I thought it needed repeats, but instead it's looking for whitespace

Is this related: #4382?

@jreback no, I'm definitely just wrong. I read the csv intro as having repeated column names (instead of the empty column names to the right)

@jtratner I feel like I am seeing the same thing here (i.e. there is in fact a bug):

CSV:

```
a,,b,,c,
a1,a2,b1,b2,c1,c2
1,1,1,1,1,1
2,2,2,2,2,2
```

and then:

```
In [66]: pd.read_csv("test2.csv",header=[0,1],tupleize_cols=False)
Out[66]: 
    a  Unnamed: 1_level_0   b  Unnamed: 3_level_0   c  Unnamed: 5_level_0
   a1                  a2  b1                  b2  c1                  c2
0   2                   2   2                   2   2                   2
```

even if I clean up the CSV:

```
a,a,b,b,c,c
a1,a2,b1,b2,c1,c2
1,1,1,1,1,1
2,2,2,2,2,2
```

I still get:

```
In [67]: pd.read_csv("test3.csv",header=[0,1],tupleize_cols=False)
Out[67]: 
    a       b       c    
   a1  a2  b1  b2  c1  c2
0   2   2   2   2   2   2
```

@cancan101 well, that's certainly not what I would expect. @jreback why does it need to have a row of row names?

you can try and fix if u want, but couldn't find a way to make a general parser that could parse to_csv format (that does have all row and col names) and a format that is what u have above, without the user specifying more information that the header. bottom line is that maybe the above format should raise as an invalid format (unless you can find a way to parse it) without sacrificing the rest of the cases

Why would the above format be invalid?

it's not invalid per se but how do you know that the 3rd row is supposed to be row names (and just over specified) or values?

csv format is just not descriptive enough to figure it out

and that is what to_csv outputs
it HAS to assume it is row names in order to be round trip able

I am confused here. The parser tries to read the index values (ie the row names) from the third row?

Here are the cases this was setup to read. It certainly can be changed to read a mi columns with out a row of row-names, but I think you would need to have a way to tell the reader to do this. IIRC there is no easy way to 'figure' it out. Any suggestions?

```
In [27]: df = DataFrame([[1,1,1,1,1,1],[2,2,2,2,2,2]],columns=pd.MultiIndex.from_arrays([['a','a','b','b','c','c'],['a1','a2','b1','b2','c1','c2']]))

In [28]: df
Out[28]: 
    a       b       c    
   a1  a2  b1  b2  c1  c2
0   1   1   1   1   1   1
1   2   2   2   2   2   2

In [34]: df.to_csv('test.csv',tupleize_cols=False,index=None)

In [35]: !cat test.csv
a,a,b,b,c,c
a1,a2,b1,b2,c1,c2
,,,,,
1,1,1,1,1,1
2,2,2,2,2,2

In [38]: pd.read_csv('test.csv',header=[0,1],index_col=None,tupleize_cols=False)
Out[38]: 
    a       b       c    
   a1  a2  b1  b2  c1  c2
0   1   1   1   1   1   1
1   2   2   2   2   2   2
```

```
In [43]: df.to_csv('test.csv',tupleize_cols=False)

In [44]: !cat test.csv
,a,a,b,b,c,c
,a1,a2,b1,b2,c1,c2
,,,,,,
0,1,1,1,1,1,1
1,2,2,2,2,2,2

In [45]: pd.read_csv('test.csv',header=[0,1],index_col=0,tupleize_cols=False)
Out[45]: 
    a       b       c    
   a1  a2  b1  b2  c1  c2
0   1   1   1   1   1   1
1   2   2   2   2   2   2

```

@jreback I don't understand what you mean by "row of row names" - can you show an example of what the use case is for this?  I would think that the far left column would contain the row labels (if any) and its headers would be the name for the index.

Meant row if index name (or names if the index is a multi-index too). 

multi-index on columns

```
In [50]: df = DataFrame([[1,1,1,1,1,1],[2,2,2,2,2,2]],columns=pd.MultiIndex.from_arrays([['a','a','b','b','c','c'],['a1','a2','b1','b2','c1','c2']],names=['first','second']),index=Index([0,1],name='foo'))

In [51]: df.to_csv('test.csv',tupleize_cols=False)

In [52]: !cat test.csv
first,a,a,b,b,c,c
second,a1,a2,b1,b2,c1,c2
foo,,,,,,
0,1,1,1,1,1,1
1,2,2,2,2,2,2

In [56]: pd.read_csv('test.csv',header=[0,1],tupleize_cols=False,index_col=0)
Out[56]: 
first    a       b       c    
second  a1  a2  b1  b2  c1  c2
foo                           
0        1   1   1   1   1   1
1        2   2   2   2   2   2

```

multi index on both axes

```
In [57]: df = DataFrame([[1,1,1,1,1,1],[2,2,2,2,2,2]],columns=pd.MultiIndex.from_arrays([['a','a','b','b','c','c'],['a1','a2','b1','b2','c1','c2']],names=['first','second']),index=pd.MultiIndex.from_arrays([['A',0],['A',1]],names=['l1','l2']))

In [58]: df
Out[58]: 
first    a       b       c    
second  a1  a2  b1  b2  c1  c2
l1 l2                         
A  A     1   1   1   1   1   1
0  1     2   2   2   2   2   2

In [59]: df.to_csv('test.csv',tupleize_cols=False)

In [60]: !cat test.csv
first,,a,a,b,b,c,c
second,,a1,a2,b1,b2,c1,c2
l1,l2,,,,,,
A,A,1,1,1,1,1,1
0,1,2,2,2,2,2,2

In [61]: pd.read_csv('test.csv',header=[0,1],tupleize_cols=False,index_col=[0,1])
Out[61]: 
first    a       b       c    
second  a1  a2  b1  b2  c1  c2
l1 l2                         
A  A     1   1   1   1   1   1
0  1     2   2   2   2   2   2
```

So need to add an additional parameter to indicate that this 'extra' row need not be generated (in `to_csv`)
or skipped on the read back (in `read_csv`).

@jreback wow, that's a crazy case. Maybe we should just add a keyword argument that lets you skip it in read_csv and leave it alone aside from that.

Hey, I didn't follow everything in detail here, but i experienced the smae behavior, so I looked into the source files and found something with which i was able to resolve the loosing of the line. For me it seems as a real bug.
so the source file i mean is io.parsers.py, the relevant code is:

```
 def _infer_columns(self):
        names = self.names

        if self.header is not None:
            header = self.header

            # we have a mi columns, so read and extra line
            if isinstance(header,(list,tuple,np.ndarray)):
                have_mi_columns = True
                header = list(header) + [header[-1]+1]
            else:
                have_mi_columns = False
                header = [ header ]
```

so it just increases the line number of the last entry if one passes a list to the header kwarg, with the initial example:

```
>>> from StringIO import StringIO
>>> import pandas
>>> text = """
a  a  a  b  c  c
q r s t u v
1 2 3 4 5 6
1 2 3 4 5 6
"""

```

one could do:

```
>>> pandas.read_csv(StringIO(text.strip()), sep='\s+', header=[1, 0])
   (q, a)  (r, a)  (s, a)  (t, b)  (u, c)  (v, c)
0       1       2       3       4       5       6
0       1       2       3       4       5       6
```

to get the almost desired output, see that the header is now upside down, but nothing is skipped.

@mortbauer thanks for the example. This is more complicated that it looks because `to_csv` needs to write in this format; `read_csv` needs a keyword to determine whether to skip an extra line (which it needs now) or not.

If I'm reading this correctly, this is the same issue regarding ambiguity
of naming column levels or index levels. While I get why pandas wants to
have that nearly blank row for column names, many actual files are not
created like that. Usually you have a header for every column (or
potentially multiple levels of header).

Could we check for a row that has only the first N labels filled (where N
is the number of Index cols) occurring right after the last header row with
everything  blank after that. If we find it, then assume those are the
index labels and the header row has labels for the column levels.

s/labels/names above


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -569,7 +569,6 @@ def _clean_options(self, options, engine):
         skiprows = set() if skiprows is None else set(skiprows)
 
         # put stuff back
-        result['index_col'] = index_col
         result['names'] = names
         result['converters'] = converters
         result['na_values'] = na_values
@@ -641,7 +640,7 @@ def __init__(self, kwds):
         self.orig_names = None
         self.prefix = kwds.pop('prefix', None)
 
-        self.index_col = kwds.pop('index_col', None)
+        self.index_col = kwds.get('index_col', None)
         self.index_names = None
         self.col_names = None
 
@@ -1455,6 +1454,7 @@ def _convert_data(self, data):
     def _infer_columns(self):
         names = self.names
         num_original_columns = 0
+        clear_buffer = True
         if self.header is not None:
             header = self.header
 
@@ -1473,6 +1473,7 @@ def _infer_columns(self):
                 while self.pos <= hr:
                     line = self._next_line()
 
+                unnamed_count = 0
                 this_columns = []
                 for i, c in enumerate(line):
                     if c == '':
@@ -1480,6 +1481,7 @@ def _infer_columns(self):
                             this_columns.append('Unnamed: %d_level_%d' % (i, level))
                         else:
                             this_columns.append('Unnamed: %d' % i)
+                        unnamed_count += 1
                     else:
                         this_columns.append(c)
 
@@ -1490,12 +1492,25 @@ def _infer_columns(self):
                         if cur_count > 0:
                             this_columns[i] = '%s.%d' % (col, cur_count)
                         counts[col] = cur_count + 1
+                elif have_mi_columns:
+
+                    # if we have grabbed an extra line, but its not in our format
+                    # so save in the buffer, and create an blank extra line for the rest of the
+                    # parsing code
+                    if hr == header[-1]:
+                        lc = len(this_columns)
+                        ic = len(self.index_col) if self.index_col is not None else 0
+                        if lc != unnamed_count and lc-ic > unnamed_count:
+                            clear_buffer = False
+                            this_columns = [ None ] * lc
+                            self.buf = [ self.buf[-1] ]
 
                 columns.append(this_columns)
                 if len(columns) == 1:
                     num_original_columns = len(this_columns)
 
-            self._clear_buffer()
+            if clear_buffer:
+                self._clear_buffer()
 
             if names is not None:
                 if (self.usecols is not None and len(names) != len(self.usecols)) \


Test patch:
diff --git a/pandas/io/tests/test_parsers.py b/pandas/io/tests/test_parsers.py
--- a/pandas/io/tests/test_parsers.py
+++ b/pandas/io/tests/test_parsers.py
@@ -1215,29 +1215,113 @@ def test_header_multi_index(self):
 R_l0_g4,R_l1_g4,R4C0,R4C1,R4C2
 """
 
-        df = read_csv(StringIO(data), header=[0, 2, 3, 4], index_col=[0, 1], tupleize_cols=False)
+        df = self.read_csv(StringIO(data), header=[0, 2, 3, 4], index_col=[0, 1], tupleize_cols=False)
         tm.assert_frame_equal(df, expected)
 
         # skipping lines in the header
-        df = read_csv(StringIO(data), header=[0, 2, 3, 4], index_col=[0, 1], tupleize_cols=False)
+        df = self.read_csv(StringIO(data), header=[0, 2, 3, 4], index_col=[0, 1], tupleize_cols=False)
         tm.assert_frame_equal(df, expected)
 
         #### invalid options ####
 
         # no as_recarray
-        self.assertRaises(ValueError, read_csv, StringIO(data), header=[0,1,2,3],
+        self.assertRaises(ValueError, self.read_csv, StringIO(data), header=[0,1,2,3],
                           index_col=[0,1], as_recarray=True, tupleize_cols=False)
 
         # names
-        self.assertRaises(ValueError, read_csv, StringIO(data), header=[0,1,2,3],
+        self.assertRaises(ValueError, self.read_csv, StringIO(data), header=[0,1,2,3],
                           index_col=[0,1], names=['foo','bar'], tupleize_cols=False)
         # usecols
-        self.assertRaises(ValueError, read_csv, StringIO(data), header=[0,1,2,3],
+        self.assertRaises(ValueError, self.read_csv, StringIO(data), header=[0,1,2,3],
                           index_col=[0,1], usecols=['foo','bar'], tupleize_cols=False)
         # non-numeric index_col
-        self.assertRaises(ValueError, read_csv, StringIO(data), header=[0,1,2,3],
+        self.assertRaises(ValueError, self.read_csv, StringIO(data), header=[0,1,2,3],
                           index_col=['foo','bar'], tupleize_cols=False)
 
+    def test_header_multiindex_common_format(self):
+
+        df = DataFrame([[1,2,3,4,5,6],[7,8,9,10,11,12]],
+                       index=['one','two'],
+                       columns=MultiIndex.from_tuples([('a','q'),('a','r'),('a','s'),
+                                                       ('b','t'),('c','u'),('c','v')]))
+
+        # to_csv
+        data = """,a,a,a,b,c,c
+,q,r,s,t,u,v
+,,,,,,
+one,1,2,3,4,5,6
+two,7,8,9,10,11,12"""
+
+        result = self.read_csv(StringIO(data),header=[0,1],index_col=0)
+        tm.assert_frame_equal(df,result)
+
+        # common
+        data = """,a,a,a,b,c,c
+,q,r,s,t,u,v
+one,1,2,3,4,5,6
+two,7,8,9,10,11,12"""
+
+        result = self.read_csv(StringIO(data),header=[0,1],index_col=0)
+        tm.assert_frame_equal(df,result)
+
+        # common, no index_col
+        data = """a,a,a,b,c,c
+q,r,s,t,u,v
+1,2,3,4,5,6
+7,8,9,10,11,12"""
+
+        result = self.read_csv(StringIO(data),header=[0,1],index_col=None)
+        tm.assert_frame_equal(df.reset_index(drop=True),result)
+
+        # malformed case 1
+        expected = DataFrame(np.array([[ 2,  3,  4,  5,  6],
+                                       [ 8,  9, 10, 11, 12]]),
+                             index=Index([1, 7]),
+                             columns=MultiIndex(levels=[[u('a'), u('b'), u('c')], [u('r'), u('s'), u('t'), u('u'), u('v')]],
+                                                labels=[[0, 0, 1, 2, 2], [0, 1, 2, 3, 4]],
+                                                names=[u('a'), u('q')]))
+
+        data = """a,a,a,b,c,c
+q,r,s,t,u,v
+1,2,3,4,5,6
+7,8,9,10,11,12"""
+
+        result = self.read_csv(StringIO(data),header=[0,1],index_col=0)
+        tm.assert_frame_equal(expected,result)
+
+        # malformed case 2
+        expected = DataFrame(np.array([[ 2,  3,  4,  5,  6],
+                                       [ 8,  9, 10, 11, 12]]),
+                             index=Index([1, 7]),
+                             columns=MultiIndex(levels=[[u('a'), u('b'), u('c')], [u('r'), u('s'), u('t'), u('u'), u('v')]],
+                                                labels=[[0, 0, 1, 2, 2], [0, 1, 2, 3, 4]],
+                                                names=[None, u('q')]))
+
+        data = """,a,a,b,c,c
+q,r,s,t,u,v
+1,2,3,4,5,6
+7,8,9,10,11,12"""
+
+        result = self.read_csv(StringIO(data),header=[0,1],index_col=0)
+        tm.assert_frame_equal(expected,result)
+
+        # mi on columns and index (malformed)
+        expected = DataFrame(np.array([[ 3,  4,  5,  6],
+                                       [ 9, 10, 11, 12]]),
+                             index=MultiIndex(levels=[[1, 7], [2, 8]],
+                                              labels=[[0, 1], [0, 1]]),
+                             columns=MultiIndex(levels=[[u('a'), u('b'), u('c')], [u('s'), u('t'), u('u'), u('v')]],
+                                                labels=[[0, 1, 2, 2], [0, 1, 2, 3]],
+                                                names=[None, u('q')]))
+
+        data = """,a,a,b,c,c
+q,r,s,t,u,v
+1,2,3,4,5,6
+7,8,9,10,11,12"""
+
+        result = self.read_csv(StringIO(data),header=[0,1],index_col=[0, 1])
+        tm.assert_frame_equal(expected,result)
+
     def test_pass_names_with_index(self):
         lines = self.data1.split('\n')
         no_header = '\n'.join(lines[1:])
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -5348,15 +5348,19 @@ def test_to_csv_moar(self):
         def _do_test(df,path,r_dtype=None,c_dtype=None,rnlvl=None,cnlvl=None,
                      dupe_col=False):
 
+               kwargs = dict(parse_dates=False)
                if cnlvl:
-                   header = lrange(cnlvl)
+                   if rnlvl is not None:
+                       kwargs['index_col'] = lrange(rnlvl)
+                   kwargs['header'] = lrange(cnlvl)
                    with ensure_clean(path) as path:
                         df.to_csv(path,encoding='utf8',chunksize=chunksize,tupleize_cols=False)
-                        recons = DataFrame.from_csv(path,header=lrange(cnlvl),tupleize_cols=False,parse_dates=False)
+                        recons = DataFrame.from_csv(path,tupleize_cols=False,**kwargs)
                else:
+                   kwargs['header'] = 0
                    with ensure_clean(path) as path:
                        df.to_csv(path,encoding='utf8',chunksize=chunksize)
-                       recons = DataFrame.from_csv(path,header=0,parse_dates=False)
+                       recons = DataFrame.from_csv(path,**kwargs)
 
                def _to_uni(x):
                    if not isinstance(x, compat.text_type):
@@ -5366,7 +5370,7 @@ def _to_uni(x):
                    # read_Csv disambiguates the columns by
                    # labeling them dupe.1,dupe.2, etc'. monkey patch columns
                    recons.columns = df.columns
-               if rnlvl:
+               if rnlvl and not cnlvl:
                    delta_lvl = [recons.icol(i).values for i in range(rnlvl-1)]
                    ix=MultiIndex.from_arrays([list(recons.index)]+delta_lvl)
                    recons.index = ix
@@ -5417,7 +5421,7 @@ def _to_uni(x):
                         recons.columns = np.array(recons.columns,dtype=c_dtype )
                         df.columns = np.array(df.columns,dtype=c_dtype )
 
-               assert_frame_equal(df, recons,check_names=False,check_less_precise=True)
+               assert_frame_equal(df,recons,check_names=False,check_less_precise=True)
 
         N = 100
         chunksize=1000
@@ -5476,7 +5480,7 @@ def make_dtnat_arr(n,nnat=None):
             base = int((chunksize// ncols or 1) or 1)
             for nrows in [10,N-2,N-1,N,N+1,N+2,2*N-2,2*N-1,2*N,2*N+1,2*N+2,
                       base-1,base,base+1]:
-                print( nrows,ncols)
+                #print( nrows,ncols)
                 _do_test(mkdf(nrows, ncols),path)
 
         for nrows in [10,N-2,N-1,N,N+1,N+2]:
@@ -5498,7 +5502,7 @@ def make_dtnat_arr(n,nnat=None):
             base = int(chunksize//ncols)
             for nrows in [10,N-2,N-1,N,N+1,N+2,2*N-2,2*N-1,2*N,2*N+1,2*N+2,
                       base-1,base,base+1]:
-                print(nrows, ncols)
+                #print(nrows, ncols)
                 _do_test(mkdf(nrows, ncols,r_idx_nlevels=2),path,rnlvl=2)
                 _do_test(mkdf(nrows, ncols,c_idx_nlevels=2),path,cnlvl=2)
                 _do_test(mkdf(nrows, ncols,r_idx_nlevels=2,c_idx_nlevels=2),
@@ -5615,11 +5619,8 @@ def _make_frame(names=None):
             # dup column names?
             df = mkdf(5,3,r_idx_nlevels=3,c_idx_nlevels=4)
             df.to_csv(path,tupleize_cols=False)
-            result = read_csv(path,header=[0,1,2,3],index_col=[0,1],tupleize_cols=False)
-            result.columns = ['R2','A','B','C']
-            new_result = result.reset_index().set_index(['R0','R1','R2'])
-            new_result.columns = df.columns
-            assert_frame_equal(df,new_result)
+            result = read_csv(path,header=[0,1,2,3],index_col=[0,1,2],tupleize_cols=False)
+            assert_frame_equal(df,result)
 
             # writing with no index
             df = _make_frame()
@@ -9881,7 +9882,7 @@ def _check_stat_op(self, name, alternative, frame=None, has_skipna=True,
         if not ('max' in name or 'min' in name or 'count' in name):
             df = DataFrame({'b': date_range('1/1/2001', periods=2)})
             _f = getattr(df, name)
-            print(df)
+            #print(df)
             self.assertFalse(len(_f()))
 
             df['a'] = lrange(len(df))
@@ -11786,7 +11787,7 @@ def to_series(mi, level):
             if isinstance(v, Index):
                 assert v.is_(expected[k])
             elif isinstance(v, Series):
-                print(k)
+                #print(k)
                 tm.assert_series_equal(v, expected[k])
             else:
                 raise AssertionError("object must be a Series or Index")