BUG: DataFrame.stack(..., dropna=False) with partial MultiIndex.
State: closed

Closes #8844

Fixes `DataFrame.stack(..., dropna=False)` when the columns consist of a "partial" `MultiIndex`, i.e. one in which the `labels` don't reference all the `levels`.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
this looks fine

cc @rockg

@seth-p want to do a quick speed test (maybe even add a vbench?)

Afraid I'm not set up to run vbench (on windows). Can do simple speed tests Monday or Tuesday.

Any comments on my question at https://github.com/pydata/pandas/issues/8844#issuecomment-63724427?

You are stacking in the SAME order as presented yes (so if it happens to be sorted as is most of the time), then the output columns will be in that order, otherwise they will be in the indicated order, yes?

I think that is correct and fine. Granted you almost always have a sorted MultiIndex, but it shouldn't prevent it from working. I would guarantee that input order -> output order, and not required sortedness.

Yes, I am keeping them in the same order they appear in the MultiIndex.levels -- which is not necessarily the order in which they appear in the MultiIndex.labels. This reproduces the existing behavior when all levels are used.

The proposed change has no effect for a super-long `DataFrame` (as suggested in https://github.com/pydata/pandas/issues/8844#issuecomment-63553382):
Existing code:

```
In [12]: df1 = pd.DataFrame(np.arange(500000*10).reshape(500000,10),
                            columns=pd.MultiIndex.from_tuples([(100*x, -y) for x in range(5)
                                                                           for y in range(2)],
                                                              names=['X', 'Y']))

In [13]: %timeit df1.stack()
10 loops, best of 3: 156 ms per loop

In [14]: %timeit df1.stack(level=0)
1 loops, best of 3: 254 ms per loop

In [15]: %timeit df1.stack(level=1)
10 loops, best of 3: 156 ms per loop

In [17]: %timeit df1.stack(level=[0,1])
1 loops, best of 3: 382 ms per loop
```

With this PR:

```
In [12]: %timeit df1.stack()
10 loops, best of 3: 156 ms per loop

In [13]: %timeit df1.stack(level=0)
1 loops, best of 3: 252 ms per loop

In [14]: %timeit df1.stack(level=1)
10 loops, best of 3: 156 ms per loop

In [15]: %timeit df1.stack(level=[0,1])
1 loops, best of 3: 381 ms per loop
```

For a super-wide `DataFrame`, the new code is a tad slower, but I think tolerably so:
Existing code:

```
In [3]: df = pd.DataFrame(np.arange(5*500000).reshape(5,500000), index=list('abcde'),
                          columns=pd.MultiIndex.from_tuples([(100*x, -y) for x in range(500)
                                                                         for y in range(1000)],
                                                            names=['X', 'Y']))

In [4]: %timeit df.stack()
1 loops, best of 3: 782 ms per loop

In [6]: %timeit df.stack(level=0)
1 loops, best of 3: 1.43 s per loop

In [8]: %timeit df.stack(level=1)
1 loops, best of 3: 758 ms per loop

In [18]: %timeit df.stack(level=[0,1])
1 loops, best of 3: 1.51 s per loop
```

With this PR:

```
In [4]: %timeit df.stack()
1 loops, best of 3: 862 ms per loop

In [6]: %timeit df.stack(level=0)
1 loops, best of 3: 1.53 s per loop

In [8]: %timeit df.stack(level=1)
1 loops, best of 3: 867 ms per loop

In [16]: %timeit df.stack(level=[0,1])
1 loops, best of 3: 1.62 s per loop
```

thanks!