Preserve Alignment Between Index and Values for Non-Monotonic Stack
State: closed

- [X] closes #20945
- [X] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Not overly familiar with this code so submitting for review as there's probably a better way of going about it. The root cause of the referenced issue IIUC is that the index labels of the caller are non-monotonic. `stack` essentially takes values and labels from the level that is getting pushed down into the rows with an implicit assumption that both are monotonic, hence the index/values get misaligned.

This breaks at least one other test so not ready to merge, but looking for feedback on:

 - If there's a better way to align the values with the labels in this function AND/OR
 - If we make any guarantees about the order of the labels for the level(s) being moved in this function

Labels:

Comments:
Hello @WillAyd! Thanks for updating the PR.



 - In the file [`pandas/tests/frame/test_reshape.py`](https://github.com/pandas-dev/pandas/blob/a79eaf714c0466315760ec21472f77311e12cc0d/pandas/tests/frame/test_reshape.py), following are the PEP8 issues :

> [Line 427:80](https://github.com/pandas-dev/pandas/blob/a79eaf714c0466315760ec21472f77311e12cc0d/pandas/tests/frame/test_reshape.py#L427): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (81 > 79 characters)



##### Comment last updated on May 14, 2018 at 21:28 Hours UTC
Good to know, though I must have explained the issue incorrectly. Within the call to `_stack_multi_columns` on master here's how the column index of the failing example looks:

```python
MultiIndex(levels=[['A', 'B'], ['a', 'b', 'c', 'd']],
           labels=[[0, 0, 0, 0, 1, 1, 1, 1], [2, 1, 0, 3, 2, 1, 0, 3]],
           names=['dim2', 'foo'])
```

I built another DataFrame manually which was equivalent (at least according to `tm.assert_frame_equal`) and had a column index that look as follows:

```python
MultiIndex(levels=[['A', 'B'], ['c', 'b', 'a', 'd']],
           labels=[[0, 0, 0, 0, 1, 1, 1, 1], [0, 1, 2, 3, 0, 1, 2, 3]],
           names=['dim2', 'foo'])
```

The former yielded incorrect results at the end of the stack operation but the latter was fine, even though they were coming from two frames that look exactly the same. 

I believe the problem is that when iterating over the groups, `_stack_multi_columns` gets a slice of the frames' values. With the latter, getting a slice of the 4 values at a time keeps it aligned with the second level labels since they are sequential. That does not apply to the former example hence the issue.

Is the fact that the values of the DataFrame do not align with the labels of the column index by design?
eg.
```
In [2]: mi = pd.MultiIndex(levels=[['A', 'B'], ['c', 'b', 'a', 'd']],
   ...:            labels=[[0, 0, 0, 0, 1, 1, 1, 1], [0, 1, 2, 3, 0, 1, 2, 3]],
   ...:            names=['dim2', 'foo'])
   ...:            

In [3]: mi._sort_levels_monotonic()
Out[3]: 
MultiIndex(levels=[['A', 'B'], ['a', 'b', 'c', 'd']],
           labels=[[0, 0, 0, 0, 1, 1, 1, 1], [2, 1, 0, 3, 2, 1, 0, 3]],
           names=['dim2', 'foo'])
```
solved via #21043