Repository: pandas-dev/pandas

Issue:
BUG: Passing multiple levels to stack when having mixed integer/string level names
Related  #7770

Using the example of the docs (http://pandas.pydata.org/pandas-docs/stable/reshaping.html#multiple-levels):

```
columns = MultiIndex.from_tuples([('A', 'cat', 'long'), ('B', 'cat', 'long'), ('A', 'dog', 'short'), ('B', 'dog', 'short')], 
                                 names=['exp', 'animal', 'hair_length'])
df = DataFrame(randn(4, 4), columns=columns)
```

CONTEXT: `df.stack(level=['animal', 'hair_length'])` and `df.stack(level=[1, 2])` are equivalent (feature introduced in #7770). Mixing integers location and string names (eg `df.stack(level=['animal', 2])`) gives a ValueError.

**But** if you have level names of mixed types, some different (and wrong things) happen:
- With a total different number, it still works as it should:
  
  ```
  df.columns.names = ['exp', 'animal', 10]
  df.stack(level=['animal', 10])
  ```
- With the number 1, it treats the 1 as a level number instead of the level name, leading to a wrong result (two times the same level unstacked):
  
  ```
  In [42]: df.columns.names = ['exp', 'animal', 1]
  
  In [43]: df.stack(level=['animal', 1])
  Out[43]: 
  exp                     A         B
    animal animal                    
  0 cat    cat    -1.006065  0.401136
    dog    dog     0.526734 -1.753478
  1 cat    cat    -0.718401 -0.400386
    dog    dog    -0.951336 -1.074323
  2 cat    cat     1.119843 -0.606982
    dog    dog     0.371467 -1.837341
  3 cat    cat    -1.467968  1.114524
    dog    dog    -0.040112  0.240026
  ```
- With the number 0, it gives a strange error:
  
  ```
  In [46]: df.columns.names = ['exp', 'animal', 0]
  
  In [47]: df.stack(level=['animal', 0])
  ---------------------------------------------------------------------------
  KeyError                                  Traceback (most recent call last)
  <ipython-input-47-4e9507e0708f> in <module>()
  ----> 1 df.stack(level=['animal', 0])
  
  /home/joris/scipy/pandas/pandas/core/frame.pyc in stack(self, level, dropna)
  3390 
  3391         if isinstance(level, (tuple, list)):
  -> 3392             return stack_multiple(self, level, dropna=dropna)
  3393         else:
  3394             return stack(self, level, dropna=dropna)
  
  ....
  
  /home/joris/scipy/pandas/pandas/core/index.pyc in _partial_tup_index(self, tup, side)
  3820             raise KeyError('Key length (%d) was greater than MultiIndex'
  3821                            ' lexsort depth (%d)' %
  -> 3822                            (len(tup), self.lexsort_depth))
  3823 
  3824         n = len(tup)
  
  KeyError: 'Key length (2) was greater than MultiIndex lexsort depth (0)'
  ```



Discussion:
hmm, so this is an api issue then? I think we should be very strict on this as we cannot disambiguate easy (e.g. `.ix/.loc` issues).
- Integers must be treated always as positional and don't allow mixed integers / names 
- if integer-like level names then can be passed as strings (and not actual integers; not sure if this will break anything)

I think this are examples that we _can_ disambiguate.

I understood from that PR that the new logic was:
- if all entries (strings or ints) are found in the level names -> use as level names
- if not all found:
  - if all integers -> use as level locations
  - if not all integers -> raise ValueError

So following that, these case should / can work I think (and it does work in some cases, so at least it is a bit inconstent).

And if that logic is correct (it is the logic we want to follow), that should maybe be also mentioned in the docstring.

I agree that these two should work, since all the levels are in the level names:

```
In [42]: df.columns.names = ['exp', 'animal', 1]
In [43]: df.stack(level=['animal', 1])
```

And

```
In [46]: df.columns.names = ['exp', 'animal', 0]
In [47]: df.stack(level=['animal', 0])
```

I had a look tonight and I think I have a fix for both cases, we just need to be a bit more careful about when we're dealing with level names and when we're dealing with level numbers. If I can get these cases working, then I think the logic you've outlined (which was the original intent of the PR) still holds. Probably a good idea to add it to the docstring though.

The simplest solution I came up with for this involved adding an `as_level_numbers=False` flag to `MultiIndex.swaplevel()`, so I could use `as_level_numbers=True` to signal that the levels being passed were already level numbers, skipping the `_get_level_number()` step. 

Would this be OK to add to the API, or should I add this behaviour in a new method like `MultiIndex._swaplevel_using_level_numbers()`? Seems like it could be somewhat useful if you ever need to force `swaplevel` to deal with the passed levels as numbers, but it might break consistency.

@onesandzeroes you can make an internal function (leading '_') if you need, but this shouldn't be exposed


Resolving patch:
diff --git a/pandas/core/reshape.py b/pandas/core/reshape.py
--- a/pandas/core/reshape.py
+++ b/pandas/core/reshape.py
@@ -525,10 +525,10 @@ def stack(frame, level=-1, dropna=True):
             raise ValueError(msg)
 
     # Will also convert negative level numbers and check if out of bounds.
-    level = frame.columns._get_level_number(level)
+    level_num = frame.columns._get_level_number(level)
 
     if isinstance(frame.columns, MultiIndex):
-        return _stack_multi_columns(frame, level=level, dropna=dropna)
+        return _stack_multi_columns(frame, level_num=level_num, dropna=dropna)
     elif isinstance(frame.index, MultiIndex):
         new_levels = list(frame.index.levels)
         new_levels.append(frame.columns)
@@ -595,19 +595,43 @@ def stack_multiple(frame, level, dropna=True):
     return result
 
 
-def _stack_multi_columns(frame, level=-1, dropna=True):
+def _stack_multi_columns(frame, level_num=-1, dropna=True):
+    def _convert_level_number(level_num, columns):
+        """
+        Logic for converting the level number to something
+        we can safely pass to swaplevel:
+
+        We generally want to convert the level number into
+        a level name, except when columns do not have names,
+        in which case we must leave as a level number
+        """
+        if level_num in columns.names:
+            return columns.names[level_num]
+        else:
+            if columns.names[level_num] is None:
+                return level_num
+            else:
+                return columns.names[level_num]
+
     this = frame.copy()
 
     # this makes life much simpler
-    if level != frame.columns.nlevels - 1:
+    if level_num != frame.columns.nlevels - 1:
         # roll levels to put selected level at end
         roll_columns = this.columns
-        for i in range(level, frame.columns.nlevels - 1):
-            roll_columns = roll_columns.swaplevel(i, i + 1)
+        for i in range(level_num, frame.columns.nlevels - 1):
+            # Need to check if the ints conflict with level names
+            lev1 = _convert_level_number(i, roll_columns)
+            lev2 = _convert_level_number(i + 1, roll_columns)
+            roll_columns = roll_columns.swaplevel(lev1, lev2)
         this.columns = roll_columns
 
     if not this.columns.is_lexsorted():
-        this = this.sortlevel(0, axis=1)
+        # Workaround the edge case where 0 is one of the column names,
+        # which interferes with trying to sort based on the first
+        # level
+        level_to_sort = _convert_level_number(0, this.columns)
+        this = this.sortlevel(level_to_sort, axis=1)
 
     # tuple list excluding level for grouping columns
     if len(frame.columns.levels) > 2:
@@ -660,9 +684,9 @@ def _stack_multi_columns(frame, level=-1, dropna=True):
         new_labels = [np.arange(N).repeat(levsize)]
         new_names = [this.index.name]  # something better?
 
-    new_levels.append(frame.columns.levels[level])
+    new_levels.append(frame.columns.levels[level_num])
     new_labels.append(np.tile(np.arange(levsize), N))
-    new_names.append(frame.columns.names[level])
+    new_names.append(frame.columns.names[level_num])
 
     new_index = MultiIndex(levels=new_levels, labels=new_labels,
                            names=new_names, verify_integrity=False)


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -12110,6 +12110,70 @@ def test_stack_ints(self):
             df_named.stack(level=1).stack(level=1)
         )
 
+    def test_stack_mixed_levels(self):
+        columns = MultiIndex.from_tuples(
+            [('A', 'cat', 'long'), ('B', 'cat', 'long'),
+             ('A', 'dog', 'short'), ('B', 'dog', 'short')],
+            names=['exp', 'animal', 'hair_length']
+        )
+        df = DataFrame(randn(4, 4), columns=columns)
+
+        animal_hair_stacked = df.stack(level=['animal', 'hair_length'])
+        exp_hair_stacked = df.stack(level=['exp', 'hair_length'])
+
+        # GH #8584: Need to check that stacking works when a number
+        # is passed that is both a level name and in the range of
+        # the level numbers
+        df2 = df.copy()
+        df2.columns.names = ['exp', 'animal', 1]
+        assert_frame_equal(df2.stack(level=['animal', 1]),
+                           animal_hair_stacked,  check_names=False)
+        assert_frame_equal(df2.stack(level=['exp', 1]),
+                           exp_hair_stacked,  check_names=False)
+
+        # When mixed types are passed and the ints are not level
+        # names, raise
+        self.assertRaises(ValueError, df2.stack, level=['animal', 0])
+
+        # GH #8584: Having 0 in the level names could raise a
+        # strange error about lexsort depth
+        df3 = df.copy()
+        df3.columns.names = ['exp', 'animal', 0]
+        assert_frame_equal(df3.stack(level=['animal', 0]),
+                           animal_hair_stacked, check_names=False)
+
+    def test_stack_int_level_names(self):
+        columns = MultiIndex.from_tuples(
+            [('A', 'cat', 'long'), ('B', 'cat', 'long'),
+             ('A', 'dog', 'short'), ('B', 'dog', 'short')],
+            names=['exp', 'animal', 'hair_length']
+        )
+        df = DataFrame(randn(4, 4), columns=columns)
+
+        exp_animal_stacked = df.stack(level=['exp', 'animal'])
+        animal_hair_stacked = df.stack(level=['animal', 'hair_length'])
+        exp_hair_stacked = df.stack(level=['exp', 'hair_length'])
+
+        df2 = df.copy()
+        df2.columns.names = [0, 1, 2]
+        assert_frame_equal(df2.stack(level=[1, 2]), animal_hair_stacked,
+                           check_names=False )
+        assert_frame_equal(df2.stack(level=[0, 1]), exp_animal_stacked,
+                           check_names=False)
+        assert_frame_equal(df2.stack(level=[0, 2]), exp_hair_stacked,
+                           check_names=False)
+
+        # Out-of-order int column names
+        df3 = df.copy()
+        df3.columns.names = [2, 0, 1]
+        assert_frame_equal(df3.stack(level=[0, 1]), animal_hair_stacked,
+                           check_names=False)
+        assert_frame_equal(df3.stack(level=[2, 0]), exp_animal_stacked,
+                           check_names=False)
+        assert_frame_equal(df3.stack(level=[2, 1]), exp_hair_stacked,
+                           check_names=False)
+
+
     def test_unstack_bool(self):
         df = DataFrame([False, False],
                        index=MultiIndex.from_arrays([['a', 'b'], ['c', 'l']]),