Repository: pandas-dev/pandas

Issue:
BUG: .stack(dropna=False) looks through views incorrectly for dataframe views with multi-index columns
xref #8850

In the example below, `[11]` is incorrectly reflecting columns in `dfa` that should not be visible to `dfa1`. Note that this is not a problem when the columns are not a multi-index (`[5]` and `[6]`), or when `dropna=True` (the default; `[10]` and `[12]`).

```
Python 3.4.2 (v3.4.2:ab2c023a9432, Oct  6 2014, 22:16:31) [MSC v.1600 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 2.3.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import pandas as pd

In [2]: import numpy as np

In [3]: df = pd.DataFrame(np.zeros((2,5)), columns=list('ABCDE'))

In [4]: df1 = df[df.columns[:2]]

In [5]: df1.stack()
Out[5]:
0  A    0
   B    0
1  A    0
   B    0
dtype: float64

In [6]: df1.stack(dropna=False)
Out[6]:
0  A    0
   B    0
1  A    0
   B    0
dtype: float64

In [7]: dfa = pd.DataFrame(np.zeros((2,5)),
                           columns=pd.MultiIndex.from_tuples([(1,'A'), (1,'B'), (1,'C'), (1,'D'), (1,'E')],
                                                             names=['num', 'let']))

In [8]: dfa1 = dfa[dfa.columns[:2]]

In [10]: dfa1.stack()
Out[10]:
num    1
  let
0 A    0
  B    0
1 A    0
  B    0

In [11]: dfa1.stack(dropna=False)
Out[11]:
num     1
  let
0 A     0
  B     0
  C   NaN
  D   NaN
  E   NaN
1 A     0
  B     0
  C   NaN
  D   NaN
  E   NaN

In [12]: dfa1.stack(dropna=True)
Out[12]:
num    1
  let
0 A    0
  B    0
1 A    0
  B    0

In [13]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.2.final.0
python-bits: 64
OS: Windows
OS-release: 8
machine: AMD64
processor: Intel64 Family 6 Model 62 Stepping 4, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.15.1
nose: 1.3.4
Cython: 0.21.1
numpy: 1.9.1
scipy: 0.14.0
statsmodels: 0.6.0
IPython: 2.3.1
sphinx: None
patsy: 0.3.0
dateutil: 2.2
pytz: 2014.9
bottleneck: 0.8.0
tables: 3.1.1
numexpr: 2.4
matplotlib: 1.4.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: 0.6.3
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.8
pymysql: None
psycopg2: None
```



Discussion:
Curiously doing a deep copy doesn't help:

```
In [14]: dfa2 = dfa[dfa.columns[:2]].copy()

In [15]: dfa2.stack(dropna=False)
Out[15]:
num     1
  let
0 A     0
  B     0
  C   NaN
  D   NaN
  E   NaN
1 A     0
  B     0
  C   NaN
  D   NaN
  E   NaN
```

Though specifying all levels does:

```
In [46]: dfa1.stack(level=[0,1], dropna=False)
Out[46]:
   num  let
0  1    A      0
        B      0
1  1    A      0
        B      0
dtype: float64
```

Other variants:

```
In [51]: dfa1.stack(level=0, dropna=False)
Out[51]:
let    A  B
  num
0 1    0  0
1 1    0  0

In [52]: dfa1.stack(level=1, dropna=False)
Out[52]:
num     1
  let
0 A     0
  B     0
  C   NaN
  D   NaN
  E   NaN
1 A     0
  B     0
  C   NaN
  D   NaN
  E   NaN
```

More oddities I don't understand:

```
In [90]: dfb = pd.DataFrame(np.zeros((2,5)),
   ....:                    columns=pd.MultiIndex.from_tuples([(1,'A'), (1,'B'), (2,'C'), (2,'D'), (3,'E')],
   ....:                                                      names=['num', 'let']))

In [91]: dfb
Out[91]:
num  1     2     3
let  A  B  C  D  E
0    0  0  0  0  0
1    0  0  0  0  0

In [92]: dfb1 = dfb[dfb.columns[:3]]

In [93]: dfb1
Out[93]:
num  1     2
let  A  B  C
0    0  0  0
1    0  0  0

In [94]: dfb1.stack()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-94-a43c6a602bfd> in <module>()
----> 1 dfb1.stack()

C:\Python34\lib\site-packages\pandas\core\frame.py in stack(self, level, dropna)
   3430             return stack_multiple(self, level, dropna=dropna)
   3431         else:
-> 3432             return stack(self, level, dropna=dropna)
   3433
   3434     def unstack(self, level=-1):

C:\Python34\lib\site-packages\pandas\core\reshape.py in stack(frame, level, dropna)
    529
    530     if isinstance(frame.columns, MultiIndex):
--> 531         return _stack_multi_columns(frame, level=level, dropna=dropna)
    532     elif isinstance(frame.index, MultiIndex):
    533         new_levels = list(frame.index.levels)

C:\Python34\lib\site-packages\pandas\core\reshape.py in _stack_multi_columns(frame, level, dropna)
    648
    649     if len(drop_cols) > 0:
--> 650         new_columns = new_columns - drop_cols
    651
    652     N = len(this)

C:\Python34\lib\site-packages\pandas\core\index.py in _evaluate_numeric_binop(self, other)
   2139                 else:
   2140                     if not (com.is_float(other) or com.is_integer(other)):
-> 2141                         raise TypeError("can only perform ops with scalar values")
   2142
   2143                 # if we are a reversed non-communative op

TypeError: can only perform ops with scalar values

In [95]: dfb1.stack(level=0)
Out[95]:
let     A   B   C
  num
0 1     0   0 NaN
  2   NaN NaN   0
1 1     0   0 NaN
  2   NaN NaN   0

In [96]: dfb1.stack(level=1)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-96-c2c854f1a484> in <module>()
----> 1 dfb1.stack(level=1)

C:\Python34\lib\site-packages\pandas\core\frame.py in stack(self, level, dropna)
   3430             return stack_multiple(self, level, dropna=dropna)
   3431         else:
-> 3432             return stack(self, level, dropna=dropna)
   3433
   3434     def unstack(self, level=-1):

C:\Python34\lib\site-packages\pandas\core\reshape.py in stack(frame, level, dropna)
    529
    530     if isinstance(frame.columns, MultiIndex):
--> 531         return _stack_multi_columns(frame, level=level, dropna=dropna)
    532     elif isinstance(frame.index, MultiIndex):
    533         new_levels = list(frame.index.levels)

C:\Python34\lib\site-packages\pandas\core\reshape.py in _stack_multi_columns(frame, level, dropna)
    648
    649     if len(drop_cols) > 0:
--> 650         new_columns = new_columns - drop_cols
    651
    652     N = len(this)

C:\Python34\lib\site-packages\pandas\core\index.py in _evaluate_numeric_binop(self, other)
   2139                 else:
   2140                     if not (com.is_float(other) or com.is_integer(other)):
-> 2141                         raise TypeError("can only perform ops with scalar values")
   2142
   2143                 # if we are a reversed non-communative op

TypeError: can only perform ops with scalar values

In [97]: dfb1.stack(level=[0,1])
Out[97]:
   num  let
0  1    A      0
        B      0
   2    C      0
1  1    A      0
        B      0
   2    C      0
dtype: float64

```

And yet more oddities (I'd expect `[102]` to produce the same output as `[101]`):

```
In [98]: dfb
Out[98]:
num  1     2     3
let  A  B  C  D  E
0    0  0  0  0  0
1    0  0  0  0  0

In [99]: dfb2 = dfb[dfb.columns[:2]]

In [100]: dfb2
Out[100]:
num  1
let  A  B
0    0  0
1    0  0

In [101]: dfb2.stack(level=[0,1])
Out[101]:
   num  let
0  1    A      0
        B      0
1  1    A      0
        B      0
dtype: float64

In [102]: dfb2.stack(level=[0,1], dropna=False)
Out[102]:
   num  let
0  1    A       0
        B       0
   2    A     NaN
        B     NaN
   3    A     NaN
        B     NaN
1  1    A       0
        B       0
   2    A     NaN
        B     NaN
   3    A     NaN
        B     NaN
dtype: float64
```

Regarding your last comment, `dropna=True` by default so I expect [101] and [102] to be different.

But there are no NaNs, so I'd expect them to be the same.

this is somewhat related to this: https://github.com/pydata/pandas/issues/2770

though its an easy fix, just recompute the levels (rather than just take them as is). Index objects are shared, so you need to create a new one here. (rather than using the original and just appending levels). That's why you get the 'hidden' values showing up.

So far as I can tell, all the problems listed above are solved by first calling the following function:

```
def fix_multiindex_axes(obj):
    for axis_number, axis_index in enumerate(obj.axes):
        if isinstance(axis_index, pd.MultiIndex):
            obj.set_axis(axis_number,
                         pd.MultiIndex.from_tuples(axis_index.get_values(),
                                                   names=axis_index.names))
    return obj
```

@seth-p yes, that essentially recreates the index from scratch thereby removing an aliasing issues.

This should prob be done in unstack though. (with a method called out to MultiIndex, something like _deep_copy / _recreate or something similar).

fyi, similar issue in #8850 

like to do a PR?

And I think we need to be cognizant of speed issues here as well if we are talking about recreating the entire index again, namely, how does it perform on large frames (500,000x5 DataFrame)? 

OK, I'll give it a shot, though if what I have in mind is right I don't think it will address #8850.


Resolving patch:
diff --git a/pandas/core/reshape.py b/pandas/core/reshape.py
--- a/pandas/core/reshape.py
+++ b/pandas/core/reshape.py
@@ -648,7 +648,9 @@ def _convert_level_number(level_num, columns):
     # time to ravel the values
     new_data = {}
     level_vals = this.columns.levels[-1]
-    levsize = len(level_vals)
+    level_labels = sorted(set(this.columns.labels[-1]))
+    level_vals_used = level_vals[level_labels]
+    levsize = len(level_labels)
     drop_cols = []
     for key in unique_groups:
         loc = this.columns.get_loc(key)
@@ -661,7 +663,7 @@ def _convert_level_number(level_num, columns):
         elif slice_len != levsize:
             chunk = this.ix[:, this.columns[loc]]
             chunk.columns = level_vals.take(chunk.columns.labels[-1])
-            value_slice = chunk.reindex(columns=level_vals).values
+            value_slice = chunk.reindex(columns=level_vals_used).values
         else:
             if frame._is_mixed_type:
                 value_slice = this.ix[:, this.columns[loc]].values
@@ -685,7 +687,7 @@ def _convert_level_number(level_num, columns):
         new_names = [this.index.name]  # something better?
 
     new_levels.append(frame.columns.levels[level_num])
-    new_labels.append(np.tile(np.arange(levsize), N))
+    new_labels.append(np.tile(level_labels, N))
     new_names.append(frame.columns.names[level_num])
 
     new_index = MultiIndex(levels=new_levels, labels=new_labels,


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -12266,6 +12266,56 @@ def test_stack_datetime_column_multiIndex(self):
         expected = DataFrame([1, 2, 3, 4], index=eidx, columns=ecols)
         assert_frame_equal(result, expected)
 
+    def test_stack_partial_multiIndex(self):
+        # GH 8844
+        def _test_stack_with_multiindex(multiindex):
+            df = DataFrame(np.arange(3 * len(multiindex)).reshape(3, len(multiindex)),
+                           columns=multiindex)
+            for level in (-1, 0, 1, [0, 1], [1, 0]):
+                result = df.stack(level=level, dropna=False)
+
+                if isinstance(level, int):
+                    # Stacking a single level should not make any all-NaN rows,
+                    # so df.stack(level=level, dropna=False) should be the same
+                    # as df.stack(level=level, dropna=True).
+                    expected = df.stack(level=level, dropna=True)
+                    if isinstance(expected, Series):
+                        assert_series_equal(result, expected)
+                    else:
+                        assert_frame_equal(result, expected)
+
+                df.columns = MultiIndex.from_tuples(df.columns.get_values(),
+                                                    names=df.columns.names)
+                expected = df.stack(level=level, dropna=False)
+                if isinstance(expected, Series):
+                    assert_series_equal(result, expected)
+                else:
+                    assert_frame_equal(result, expected)
+
+        full_multiindex = MultiIndex.from_tuples([('B', 'x'), ('B', 'z'),
+                                                  ('A', 'y'),
+                                                  ('C', 'x'), ('C', 'u')],
+                                                 names=['Upper', 'Lower'])
+        for multiindex_columns in ([0, 1, 2, 3, 4],
+                                   [0, 1, 2, 3], [0, 1, 2, 4],
+                                   [0, 1, 2], [1, 2, 3], [2, 3, 4],
+                                   [0, 1], [0, 2], [0, 3],
+                                   [0], [2], [4]):
+            _test_stack_with_multiindex(full_multiindex[multiindex_columns])
+            if len(multiindex_columns) > 1:
+                multiindex_columns.reverse()
+                _test_stack_with_multiindex(full_multiindex[multiindex_columns])
+
+        df = DataFrame(np.arange(6).reshape(2, 3), columns=full_multiindex[[0, 1, 3]])
+        result = df.stack(dropna=False)
+        expected = DataFrame([[0, 2], [1, nan], [3, 5], [4, nan]],
+                             index=MultiIndex(levels=[[0, 1], ['u', 'x', 'y', 'z']],
+                                              labels=[[0, 0, 1, 1], [1, 3, 1, 3]],
+                                              names=[None, 'Lower']),
+                             columns=Index(['B', 'C'], name='Upper'),
+                             dtype=df.dtypes[0])
+        assert_frame_equal(result, expected)
+
     def test_repr_with_mi_nat(self):
         df = DataFrame({'X': [1, 2]},
                        index=[[pd.NaT, pd.Timestamp('20130101')], ['a', 'b']])