Repository: pandas-dev/pandas

Issue:
DataFrame.ix[idx, :] = value sets wrong values when idx is a MultiIndex and DataFrame.columns is also a MultiIndex
This code is broken in `0.17.0` but not in `0.15.2`:

``` python
import pandas as pd
import numpy as np

np.random.seed(1)

from itertools import product

from pandas.util.testing import assert_frame_equal

pd.show_versions()

idx = pd.MultiIndex.from_tuples(
    list(
        product(['A', 'B', 'C'], 
                pd.date_range('2015-01-01', '2015-04-01', freq='MS'))
    )
)

sub = pd.MultiIndex.from_tuples(
    [('A', pd.Timestamp('2015-01-01')), ('A', pd.Timestamp('2015-02-01'))]
)
# if cols = ['foo', 'bar', 'baz', 'quux'], there is no error. 
cols = pd.MultiIndex.from_tuples(
    list(
        product(['foo', 'bar'], 
                pd.date_range('2015-01-01', '2015-02-01', freq='MS'))
    )
)

test = pd.DataFrame(np.random.random((12, 4)), index=idx, columns=cols)
vals = pd.DataFrame(np.random.random((2, 4)), index=sub, columns=cols)
test.ix[sub, :] = vals

print test.ix[sub, :]
print vals

assert_frame_equal(test.ix[sub, :], vals)
```
### 0.17.0

``` python
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.17.0
nose: 1.3.7
pip: 7.1.0
setuptools: 18.0.1
Cython: 0.22
numpy: 1.10.1
scipy: 0.16.0
statsmodels: 0.6.1
IPython: 3.2.1
sphinx: 1.3.1
patsy: 0.4.0
dateutil: 2.4.1
pytz: 2015.4
blosc: None
bottleneck: 1.0.0
tables: 3.2.2
numexpr: 2.4.4
matplotlib: 1.4.3
openpyxl: None
xlrd: 0.9.4
xlwt: None
xlsxwriter: 0.7.3
lxml: None
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 1.0.7
pymysql: None
psycopg2: None
                    foo                   bar
             2015-01-01 2015-02-01 2015-01-01 2015-02-01
A 2015-01-01   0.287775   0.130029   0.019367   0.678836
  2015-02-01   0.287775   0.130029   0.019367   0.678836
                    foo                   bar
             2015-01-01 2015-02-01 2015-01-01 2015-02-01
A 2015-01-01   0.287775   0.130029   0.019367   0.678836
  2015-02-01   0.211628   0.265547   0.491573   0.053363
Traceback (most recent call last):
  File "c:\dev\code\sandbox\multiindex.py", line 41, in <module>
    assert_frame_equal(test.ix[sub, :], vals)
  File "c:\python\envs\pd017\lib\site-packages\pandas\util\testing.py", line 1028, in assert_frame_equal
    obj='DataFrame.iloc[:, {0}]'.format(i))
  File "c:\python\envs\pd017\lib\site-packages\pandas\util\testing.py", line 925, in assert_series_equal
    check_less_precise, obj='{0}'.format(obj))
  File "pandas\src\testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas\src\testing.c:3809)
  File "pandas\src\testing.pyx", line 147, in pandas._testing.assert_almost_equal (pandas\src\testing.c:2685)
  File "c:\python\envs\pd017\lib\site-packages\pandas\util\testing.py", line 798, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: DataFrame.iloc[:, 0] are different

DataFrame.iloc[:, 0] values are different (50.0 %)
[left]:  [0.287775338586, 0.287775338586]
[right]: [0.287775338586, 0.211628116]
```
### 0.15.2

``` python

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en_GB

pandas: 0.15.2
nose: 1.3.7
Cython: 0.22
numpy: 1.9.2
scipy: 0.15.1
statsmodels: None
IPython: 3.2.1
sphinx: 1.3.1
patsy: 0.3.0
dateutil: 2.4.1
pytz: 2015.4
bottleneck: 1.0.0
tables: 3.2.0
numexpr: 2.4.3
matplotlib: 1.4.3
openpyxl: 1.8.5
xlrd: 0.9.4
xlwt: 0.7.5
xlsxwriter: 0.7.3
lxml: 3.4.4
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: 1.0.7
pymysql: None
psycopg2: None
                    foo                   bar           
             2015-01-01 2015-02-01 2015-01-01 2015-02-01
A 2015-01-01   0.287775   0.130029   0.019367   0.678836
  2015-02-01   0.211628   0.265547   0.491573   0.053363
                    foo                   bar           
             2015-01-01 2015-02-01 2015-01-01 2015-02-01
A 2015-01-01   0.287775   0.130029   0.019367   0.678836
  2015-02-01   0.211628   0.265547   0.491573   0.053363
```



Discussion:
Indexing with a specific set of columns also gives the error:

Code sample:

``` python
import pandas as pd
import numpy as np

np.random.seed(1)

from itertools import product

from pandas.util.testing import assert_frame_equal

pd.show_versions()

idx = pd.MultiIndex.from_tuples(
    list(
        product(['A', 'B', 'C'], 
                pd.date_range('2015-01-01', '2015-04-01', freq='MS'))
    )
)
cols = pd.MultiIndex.from_tuples(
    list(
        product(['foo', 'bar'], 
                pd.date_range('2016-01-01', '2016-02-01', freq='MS'))
    )
)

# if cols = ['foo', 'bar', 'baz', 'quux'], there is no error. 

test = pd.DataFrame(np.random.random((12, 4)), index=idx, columns=cols)


subidx = pd.MultiIndex.from_tuples(
    [('A', pd.Timestamp('2015-01-01')), ('A', pd.Timestamp('2015-02-01'))]
)

subcols = pd.MultiIndex.from_tuples(
    [('foo', pd.Timestamp('2016-01-01')), ('foo', pd.Timestamp('2016-02-01'))]
)


vals = pd.DataFrame(np.random.random((2, 2)), index=subidx, columns=subcols)


test.ix[subidx, subcols] = vals

print test.ix[subidx, subcols]

print vals

assert_frame_equal(test.ix[subidx, subcols], vals)
```

### 0.17.0

``` python
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.17.0
nose: 1.3.7
pip: 7.1.0
setuptools: 18.0.1
Cython: 0.22
numpy: 1.10.1
scipy: 0.16.0
statsmodels: 0.6.1
IPython: 3.2.1
sphinx: 1.3.1
patsy: 0.4.0
dateutil: 2.4.1
pytz: 2015.4
blosc: None
bottleneck: 1.0.0
tables: 3.2.2
numexpr: 2.4.4
matplotlib: 1.4.3
openpyxl: None
xlrd: 0.9.4
xlwt: None
xlsxwriter: 0.7.3
lxml: None
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 1.0.7
pymysql: None
psycopg2: None
                    foo
             2016-01-01 2016-02-01
A 2015-01-01   0.287775   0.130029
  2015-02-01   0.287775   0.130029
                    foo
             2016-01-01 2016-02-01
A 2015-01-01   0.287775   0.130029
  2015-02-01   0.019367   0.678836
Traceback (most recent call last):
  File "c:\dev\code\sandbox\multiindex.py", line 48, in <module>
    assert_frame_equal(test.ix[subidx, subcols], vals)
  File "c:\python\envs\pd017\lib\site-packages\pandas\util\testing.py", line 1028, in assert_frame_equal
    obj='DataFrame.iloc[:, {0}]'.format(i))
  File "c:\python\envs\pd017\lib\site-packages\pandas\util\testing.py", line 925, in assert_series_equal
    check_less_precise, obj='{0}'.format(obj))
  File "pandas\src\testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas\src\testing.c:3809)
  File "pandas\src\testing.pyx", line 147, in pandas._testing.assert_almost_equal (pandas\src\testing.c:2685)
  File "c:\python\envs\pd017\lib\site-packages\pandas\util\testing.py", line 798, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: DataFrame.iloc[:, 0] are different

DataFrame.iloc[:, 0] values are different (50.0 %)
[left]:  [0.287775338586, 0.287775338586]
[right]: [0.287775338586, 0.0193669578703]
```

(Deleted- misread something, my previous suggestion was not really a fix)

hmm, surprised that broke. there is not must testing on that sub-section actually

The issue is here: https://github.com/pydata/pandas/blob/master/pandas/core/indexing.py#L450

`self._align_series` is called on a sub-section of the frame, but in the aligner, it looks at it and says oh you are a frame so gives back the wrong result.

So could prob pass in an additional parameter which would determine this.

Since I've already got two test cases, I'd be happy to have a go if I can be pointed in the right direction. I'll start by looking at the history of `indexing.py` and following any referenced issues / PRs

the pointer above is to the relevant issues.

the way to do this is to setup the test cases and the expected results (in test_indexing); they should fail before a fix, then you can step thru to see where to put a fix and go from there


Resolving patch:
diff --git a/pandas/core/indexing.py b/pandas/core/indexing.py
--- a/pandas/core/indexing.py
+++ b/pandas/core/indexing.py
@@ -443,11 +443,14 @@ def can_do_equal_len():
                 # we have an equal len Frame
                 if isinstance(value, ABCDataFrame) and value.ndim > 1:
                     sub_indexer = list(indexer)
+                    multiindex_indexer = isinstance(labels, MultiIndex)
 
                     for item in labels:
                         if item in value:
                             sub_indexer[info_axis] = item
-                            v = self._align_series(tuple(sub_indexer), value[item])
+                            v = self._align_series(
+                                tuple(sub_indexer), value[item], multiindex_indexer
+                            )
                         else:
                             v = np.nan
 
@@ -516,8 +519,28 @@ def can_do_equal_len():
             self.obj._data = self.obj._data.setitem(indexer=indexer, value=value)
             self.obj._maybe_update_cacher(clear=True)
 
-    def _align_series(self, indexer, ser):
-        # indexer to assign Series can be tuple, slice, scalar
+    def _align_series(self, indexer, ser, multiindex_indexer=False):
+        """
+        Parameters
+        ----------
+        indexer : tuple, slice, scalar
+            The indexer used to get the locations that will be set to 
+            `ser`
+
+        ser : pd.Series
+            The values to assign to the locations specified by `indexer`
+
+        multiindex_indexer : boolean, optional
+            Defaults to False. Should be set to True if `indexer` was from
+            a `pd.MultiIndex`, to avoid unnecessary broadcasting. 
+
+
+        Returns:
+        --------
+        `np.array` of `ser` broadcast to the appropriate shape for assignment
+        to the locations selected by `indexer`
+
+        """
         if isinstance(indexer, (slice, np.ndarray, list, Index)):
             indexer = tuple([indexer])
 
@@ -555,7 +578,7 @@ def _align_series(self, indexer, ser):
                 ser = ser.reindex(obj.axes[0][indexer[0]], copy=True)._values
 
                 # single indexer
-                if len(indexer) > 1:
+                if len(indexer) > 1 and not multiindex_indexer:
                     l = len(indexer[1])
                     ser = np.tile(ser, l).reshape(l, -1).T
 


Test patch:
diff --git a/pandas/tests/test_indexing.py b/pandas/tests/test_indexing.py
--- a/pandas/tests/test_indexing.py
+++ b/pandas/tests/test_indexing.py
@@ -762,32 +762,95 @@ def compare(result, expected):
         result2 = s.loc[0:3]
         assert_series_equal(result1,result2)
 
-    def test_loc_setitem_multiindex(self):
+    def test_setitem_multiindex(self):
+        for index_fn in ('ix', 'loc'):
+            def check(target, indexers, value, compare_fn, expected=None):
+                fn = getattr(target, index_fn)
+                fn.__setitem__(indexers, value)
+                result = fn.__getitem__(indexers)
+                if expected is None:
+                    expected = value
+                compare_fn(result, expected)
+            # GH7190
+            index = pd.MultiIndex.from_product([np.arange(0,100), np.arange(0, 80)], names=['time', 'firm'])
+            t, n = 0, 2
+            df = DataFrame(np.nan,columns=['A', 'w', 'l', 'a', 'x', 'X', 'd', 'profit'], index=index)
+            check(
+                target=df, indexers=((t,n), 'X'),
+                value=0, compare_fn=self.assertEqual
+            )
 
-        # GH7190
-        index = pd.MultiIndex.from_product([np.arange(0,100), np.arange(0, 80)], names=['time', 'firm'])
-        t, n = 0, 2
+            df = DataFrame(-999,columns=['A', 'w', 'l', 'a', 'x', 'X', 'd', 'profit'], index=index)
+            check(
+                target=df, indexers=((t,n), 'X'),
+                value=1, compare_fn=self.assertEqual
+            )
 
-        df = DataFrame(np.nan,columns=['A', 'w', 'l', 'a', 'x', 'X', 'd', 'profit'], index=index)
-        df.loc[(t,n),'X'] = 0
-        result = df.loc[(t,n),'X']
-        self.assertEqual(result, 0)
+            df = DataFrame(columns=['A', 'w', 'l', 'a', 'x', 'X', 'd', 'profit'], index=index)
+            check(
+                target=df, indexers=((t,n), 'X'),
+                value=2, compare_fn=self.assertEqual
+            )
 
-        df = DataFrame(-999,columns=['A', 'w', 'l', 'a', 'x', 'X', 'd', 'profit'], index=index)
-        df.loc[(t,n),'X'] = 1
-        result = df.loc[(t,n),'X']
-        self.assertEqual(result, 1)
+            # GH 7218, assinging with 0-dim arrays
+            df = DataFrame(-999,columns=['A', 'w', 'l', 'a', 'x', 'X', 'd', 'profit'], index=index)
+            check(
+                target=df, indexers=((t,n), 'X'),
+                value=np.array(3), compare_fn=self.assertEqual,
+                expected=3,
+            )
 
-        df = DataFrame(columns=['A', 'w', 'l', 'a', 'x', 'X', 'd', 'profit'], index=index)
-        df.loc[(t,n),'X'] = 2
-        result = df.loc[(t,n),'X']
-        self.assertEqual(result, 2)
+            # GH5206
+            df = pd.DataFrame(
+                np.arange(25).reshape(5, 5), columns='A,B,C,D,E'.split(','),
+                dtype=float
+            )
+            df['F'] = 99
+            row_selection = df['A'] % 2 == 0
+            col_selection = ['B', 'C']
+            df.ix[row_selection, col_selection] = df['F']
+            output = pd.DataFrame(99., index=[0, 2, 4], columns=['B', 'C'])
+            assert_frame_equal(df.ix[row_selection, col_selection], output)
+            check(
+                target=df, indexers=(row_selection, col_selection),
+                value=df['F'], compare_fn=assert_frame_equal,
+                expected=output,
+            )
 
-        # GH 7218, assinging with 0-dim arrays
-        df = DataFrame(-999,columns=['A', 'w', 'l', 'a', 'x', 'X', 'd', 'profit'], index=index)
-        df.loc[(t,n), 'X'] = np.array(3)
-        result = df.loc[(t,n),'X']
-        self.assertEqual(result,3)
+            # GH11372
+            idx = pd.MultiIndex.from_product([
+                ['A', 'B', 'C'],
+                pd.date_range('2015-01-01', '2015-04-01', freq='MS')
+            ])
+            cols = pd.MultiIndex.from_product([
+                ['foo', 'bar'],
+                pd.date_range('2016-01-01', '2016-02-01', freq='MS')
+            ])
+            df = pd.DataFrame(np.random.random((12, 4)), index=idx, columns=cols)
+            subidx = pd.MultiIndex.from_tuples(
+                [('A', pd.Timestamp('2015-01-01')), ('A', pd.Timestamp('2015-02-01'))]
+            )
+            subcols = pd.MultiIndex.from_tuples(
+                [('foo', pd.Timestamp('2016-01-01')), ('foo', pd.Timestamp('2016-02-01'))]
+            )
+            vals = pd.DataFrame(np.random.random((2, 2)), index=subidx, columns=subcols)
+            check(
+                target=df, indexers=(subidx, subcols),
+                value=vals, compare_fn=assert_frame_equal,
+            )
+            # set all columns
+            vals = pd.DataFrame(np.random.random((2, 4)), index=subidx, columns=cols)
+            check(
+                target=df, indexers=(subidx, slice(None, None, None)),
+                value=vals, compare_fn=assert_frame_equal,
+            )
+            # identity
+            copy = df.copy()
+            check(
+                target=df, indexers=(df.index, df.columns),
+                value=df, compare_fn=assert_frame_equal,
+                expected=copy
+            )
 
     def test_indexing_with_datetime_tz(self):