Repository: pandas-dev/pandas

Issue:
BUG: read_excel with multi-indexed column ignores index_col=None
From SO: http://stackoverflow.com/questions/34020061/excel-to-pandas-dataframe-using-first-column-as-index

@chris-b1 another one on the multi-index excel issues .. :-)

Small test case: content of excel file:

| A | A | B | B |
| --- | --- | --- | --- |
| key | val | key | val |
| 1 | 2 | 3 | 4 |
| 1 | 2 | 3 | 4 |

gives:

```
In [2]: pd.read_excel("test_excel_index_col.xlsx", header=[0,1], index_col=None)
Out[2]:
A     A    B
key val key  val
1     2    3   4
1     2    3   4
```

It's not super clear in the formatting of the dataframe, but the [1, 1] is the index and [A, key] are seen as the level names of the multi-indexed columns.



Discussion:
We don't support writing to this format (multi-index columns w/ no row index) because it's ambiguous on the way back in.   But it seems reasonable to support reading it, I'll take a look.

I got the same problem
From https://github.com/pandas-dev/pandas/issues/15180#issuecomment-274065068 - consider changing  default of `index_col` to some kind of sentinel (e.g. `'infer'`) so that passing `None` means something.
@chris-b1 @jreback I have a proposed fix here: https://github.com/stephenrauch/pandas/commit/1204b315beadd68904a15a875b2254eb0aa68bc9  Before I do all the docs and stuff I wanna make sure I am headed in the right direction.  Thanks.
Per review comment from @jreback here is proposed api for read_excel.  It is the same as read_csv.

> index_col : int or sequence or ``False``, default ``None``

>  Column (0-indexed) to use as the row labels of the DataFrame. If a
  sequence is given, those columns will be combined into a ``MultiIndex``.
  If ``None`` (default), pandas will use the first column as the
  index. If ``False``, force pandas to *not* use the first column as the index
  (row names). 

Updates are here:

https://github.com/stephenrauch/pandas/commit/9b37ff94643296d489498138c79bb0244aaa3f79

So if this proposed API looks ok, I will do the PR.

Thanks @stephenrauch, that api looks good to me, please open it as a PR.  One subtlety for the docs, if not using a MultiIndex header, `index_col=None` won't always use the first column as an index, it takes an inference path based on the shape of the data (same as for `read_csv` IIRC).  e.g.

```
In [31]: df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 5]})

In [32]: df.to_excel('temp.xlsx', index=False)

In [33]: pd.read_excel('temp.xlsx')

   a  b
0  1  4
1  2  5
2  3  5
```

>  it takes an inference path based on the shape of the data

@chris-b1 @jreback  Are there any test cases (or maybe some other description) that show how `index_col=None` works for something other than the index in the first column? Either for the csv or excel cases? I am having trouble grokking what is meant by the *shape of the data* in this case. 
Not sure csv actually works this way.  But for current Excel behavior:

Sheet 1
![image](https://cloud.githubusercontent.com/assets/1924092/23959720/e97ee1dc-0973-11e7-8952-ec92c26d6ce1.png)

parses a row index
```python
In [297]: pd.read_excel('temp.xlsx', sheetname='Sheet1')
Out[297]: 
      a  b  c
row1  1  2  4
row2  1  2  4
row3  1  2  4
```

Sheet2
![image](https://cloud.githubusercontent.com/assets/1924092/23959767/0b7792ac-0974-11e7-96b1-34081d5dbb51.png)

No row index
```python
In [298]: pd.read_excel('temp.xlsx', sheetname='Sheet2')
Out[298]: 
   a  b  c
0  1  2  4
1  1  2  4
2  1  2  4
```
Has this problem been solved? I met the same question.
As of now I still see the same issue. when using multi headers with read_excel, pandas always assigns the first column as index.
Another vote for a fix. I'm running 0.22 and it seems so hackish to have to write the code to save the 'index' to a named column and then reset the index.

Pandas is a 'great' module - thank you -
Vote for index_col=False to fix this
Just encountered this issue and I am looking for a fix. @stephenrauch has the PR been made for this?
@rileymcdowell  no PRs have been made but would welcome any if you are interested
I'll put together a PR from @stephenrauch's work in the next couple of days.
I've dug into this and ran into a decision point. Consider the following spreadsheet (Taken from the test suite).

![snip](https://user-images.githubusercontent.com/6935571/42730150-4b9c8e14-87b3-11e8-8b73-b3c5f1dffc7a.png)

Right now, this is interpreted by the `read_excel` function as a `MultiIndex` where sheet cells `A1` and `A2` (values `c1` and `c2`) represent the level `name`s of the `MultiIndex`. 

```
MultiIndex(levels=[['bar', 'foo'], ['a', 'b']],
           labels=[[1, 1, 0, 0], [0, 1, 0, 1]],
           names=['c1', 'c2'])
```

The situation that brought me to find this github issue is that in this example, I expect cells `A1` and `A2` (values `c1` and `c2`) to be interpreted as another set of levels in the `MultiIndex`. For example

```
MultiIndex(levels=[['bar', 'c1', 'foo'], ['a', 'b', 'c2']],
           labels=[[1, 2, 2, 0, 0], [2, 0, 1, 0, 1]],
           names=['', ''])
```

The test suite explicitly covers the existing functionality of the former. This behavior differs from that of the `read_csv` function, which behaves most similarly to the latter.

A workaround is to allow a sentinel value of `index_col=False` as suggested in this issue, but a change to support this would also touch some of the core logic in `pandas.io.parsers`. For example [this check which limits non-null `index_col` settings to have numeric values](https://github.com/pandas-dev/pandas/blob/63eff04af6f343557b30e656e94d76030e039029/pandas/io/parsers.py#L1284-L1288). That seems like it could have far-reaching consequences.

Any thoughts about how best to tackle this?
> The situation that brought me to find this github issue is that in this example, I expect cells A1 and A2 (values c1 and c2) to be interpreted as another set of levels in the MultiIndex. For example

Hmm well I disagree since this representation matches what you'd see with a normal data frame representation, but regardless of opinions I think it just speaks to what @chris-b1 mentioned earlier that this is really ambiguous so there's not necessarily a right answer



> This behavior differs from that of the read_csv function, which behaves most similarly to the latter.

Can you clarify this with an example? CSV doesn't have the concept of a merged cell like you have with the `foo` and `bar` cells so I don't think this is apples-to-apples



> A workaround is to allow a sentinel value of index_col=False as suggested in this issue, but a change to support this would also touch some of the core logic in pandas.io.parsers. For example this check which limits non-null index_col settings to have numeric values. That seems like it could have far-reaching consequences.

Isn't `False` a valid value for read_csv? If so then there's got to be something with that implementation that isn't limited by what you've found

Thanks for the investigation!
@rileymcdowell I would agree with @WillAyd that the former of the two behaviors you describe is the most intuitive way to interpret an Excel file. If you can I would encourage you to submit a PR with this behavior.

Resolving patch:
diff --git a/pandas/io/excel.py b/pandas/io/excel.py
--- a/pandas/io/excel.py
+++ b/pandas/io/excel.py
@@ -630,11 +630,12 @@ def _parse_cell(cell_contents, cell_typ):
                     if is_integer(skiprows):
                         row += skiprows
 
-                    data[row], control_row = _fill_mi_header(
-                        data[row], control_row)
-                    header_name, _ = _pop_header_name(
-                        data[row], index_col)
-                    header_names.append(header_name)
+                    data[row], control_row = _fill_mi_header(data[row],
+                                                             control_row)
+
+                    if index_col is not None:
+                        header_name, _ = _pop_header_name(data[row], index_col)
+                        header_names.append(header_name)
 
             if is_list_like(index_col):
                 # Forward fill values for MultiIndex index.
@@ -682,7 +683,8 @@ def _parse_cell(cell_contents, cell_typ):
 
                 output[asheetname] = parser.read(nrows=nrows)
 
-                if not squeeze or isinstance(output[asheetname], DataFrame):
+                if ((not squeeze or isinstance(output[asheetname], DataFrame))
+                        and header_names):
                     output[asheetname].columns = output[
                         asheetname].columns.set_names(header_names)
             except EmptyDataError:
@@ -863,16 +865,30 @@ def _fill_mi_header(row, control_row):
 
 
 def _pop_header_name(row, index_col):
-    """ (header, new_data) for header rows in MultiIndex parsing"""
-    none_fill = lambda x: None if x == '' else x
-
-    if index_col is None:
-        # no index col specified, trim data for inference path
-        return none_fill(row[0]), row[1:]
-    else:
-        # pop out header name and fill w/ blank
-        i = index_col if not is_list_like(index_col) else max(index_col)
-        return none_fill(row[i]), row[:i] + [''] + row[i + 1:]
+    """
+    Pop the header name for MultiIndex parsing.
+
+    Parameters
+    ----------
+    row : list
+        The data row to parse for the header name.
+    index_col : int, list
+        The index columns for our data. Assumed to be non-null.
+
+    Returns
+    -------
+    header_name : str
+        The extracted header name.
+    trimmed_row : list
+        The original data row with the header name removed.
+    """
+    # Pop out header name and fill w/blank.
+    i = index_col if not is_list_like(index_col) else max(index_col)
+
+    header_name = row[i]
+    header_name = None if header_name == "" else header_name
+
+    return header_name, row[:i] + [''] + row[i + 1:]
 
 
 @add_metaclass(abc.ABCMeta)


Test patch:
diff --git a/pandas/tests/io/data/testmultiindex.xls b/pandas/tests/io/data/testmultiindex.xls
Binary files a/pandas/tests/io/data/testmultiindex.xls and b/pandas/tests/io/data/testmultiindex.xls differ
diff --git a/pandas/tests/io/data/testmultiindex.xlsm b/pandas/tests/io/data/testmultiindex.xlsm
Binary files a/pandas/tests/io/data/testmultiindex.xlsm and b/pandas/tests/io/data/testmultiindex.xlsm differ
diff --git a/pandas/tests/io/data/testmultiindex.xlsx b/pandas/tests/io/data/testmultiindex.xlsx
Binary files a/pandas/tests/io/data/testmultiindex.xlsx and b/pandas/tests/io/data/testmultiindex.xlsx differ
diff --git a/pandas/tests/io/test_excel.py b/pandas/tests/io/test_excel.py
--- a/pandas/tests/io/test_excel.py
+++ b/pandas/tests/io/test_excel.py
@@ -896,6 +896,17 @@ def test_read_excel_multiindex(self, ext):
                             header=[0, 1], skiprows=2)
         tm.assert_frame_equal(actual, expected)
 
+    def test_read_excel_multiindex_header_only(self, ext):
+        # see gh-11733.
+        #
+        # Don't try to parse a header name if there isn't one.
+        mi_file = os.path.join(self.dirpath, "testmultiindex" + ext)
+        result = read_excel(mi_file, "index_col_none", header=[0, 1])
+
+        exp_columns = MultiIndex.from_product([("A", "B"), ("key", "val")])
+        expected = DataFrame([[1, 2, 3, 4]] * 2, columns=exp_columns)
+        tm.assert_frame_equal(result, expected)
+
     @td.skip_if_no("xlsxwriter")
     def test_read_excel_multiindex_empty_level(self, ext):
         # see gh-12453