Bugfix for multilevel columns with empty strings in Python 2
State: closed

 Allow unicode empty strings to be used as placeholders in multilevel column names in Python 2.

The code in `Frame._getitem_multilevel()` only treats empty strings as placeholders in multilevel columns if they are of type `str`. Since in Python 2 this means bytestrings, this means a unicode empty string is not treated as a placeholder. This is a problem for Python 2 code that is trying to be forward-compatible with Python 3 by importing `unicode_literals` from `__future__` and using empty strings in DataFrame construction - those empty strings will be unicode strings and not treated as placeholders by pandas.

This patch changes the logic to allow anything that passes an equality check with `''` to be used as a placeholder, rather than checking its type, which means that after this patch either bytestring or unicode empty strings can be used as placeholders in Python 2, and the situation is unchanged in Python 3 (unicode strings, but not bytestrings, will be treated as placeholders).

Without this patch, the following code:

```python
from __future__ import print_function
import pandas

# Unicode string placeholder in multilevel column:
df = pandas.DataFrame({(u'test', u''): [1.0, 2.0, 3.0]})
print(df['test'])
print(type(df['test']))
print()

# native string type placeholder in multilevel column:
df = pandas.DataFrame({(u'test', ''): [1.0, 2.0, 3.0]})
print(df['test'])
print(type(df['test']))

```
Results in the following in Python 2:

```

0  1.0
1  2.0
2  3.0
<class 'pandas.core.frame.DataFrame'>

0    1.0
1    2.0
2    3.0
Name: test, dtype: float64
<class 'pandas.core.series.Series'>
```

Note the first blank line in the output when a unicode empty string placeholder was used. It's an empty string column label for the dataframe.

Whereas in Python 3 it results in:
```
0    1.0
1    2.0
2    3.0
Name: test, dtype: float64
<class 'pandas.core.series.Series'>

0    1.0
1    2.0
2    3.0
Name: test, dtype: float64
<class 'pandas.core.series.Series'>
```

After the patch, the Python 2 result is the same as the Python 3 result.

Labels:
name: MultiIndex

Comments:
Full disclosure - the bug is my fault! I've only contributed to pandas once before and it was the [exact code](https://github.com/pandas-dev/pandas2/commit/428896d59a1748278efea0acb44fb40f9508170f) that has this problem. I wasn't thinking about forward compatibility with Python 3 at the time.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=h1) Report
> Merging [#17099](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/073c14544436d95969258928e2554cb2fc093c99?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17099/graphs/tree.svg?height=150&width=650&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17099      +/-   ##
==========================================
- Coverage   91.03%   91.01%   -0.02%     
==========================================
  Files         162      162              
  Lines       49511    49510       -1     
==========================================
- Hits        45070    45060      -10     
- Misses       4441     4450       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.79% <100%> (-0.01%)` | :arrow_down: |
| #single | `40.26% <0%> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.71% <100%> (-0.1%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/resample.py](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzYW1wbGUucHk=) | `96.13% <0%> (-0.03%)` | :arrow_down: |
| [pandas/tseries/offsets.py](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvb2Zmc2V0cy5weQ==) | `97.12% <0%> (-0.01%)` | :arrow_down: |
| [pandas/core/ops.py](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvb3BzLnB5) | `91.86% <0%> (ø)` | :arrow_up: |
| [pandas/core/dtypes/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2dlbmVyaWMucHk=) | `100% <0%> (ø)` | :arrow_up: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `92.03% <0%> (ø)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `95.93% <0%> (ø)` | :arrow_up: |
| [pandas/core/tools/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvZGF0ZXRpbWVzLnB5) | `66.94% <0%> (+0.09%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=footer). Last update [073c145...f1edf68](https://codecov.io/gh/pandas-dev/pandas/pull/17099?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Hello @chrisjbillington! Thanks for updating the PR.





Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on August 10, 2017 at 05:02 Hours UTC
> Note the first blank line in the output when a unicode empty string placeholder was used.

A little hard to see given that you don't have anything printed before that but understood. :smile: 
Thanks for the (tentative) review!

I've made the requested changes.
Thanks! I've made the further requested changes.
Bump! Any chance of getting this into the next release? I'm in the process of porting an application to that strange dialect that is Python2.7/3.4+ and it would be great to be able to deploy my updates once this fix is in the conda repositories.
@chrisjbillington : Absolutely.  This looks pretty good to merge, but I think @jreback might want to look it over just in case there is anything else to fix.
Great, thanks!
Thanks! I've simplified the whatsnew note and parametrized the test.
@jreback : Thoughts?
thanks!
No worries, thank you!