BUG: Fixed handling of boolean indexing with 2-d ndarrays
State: closed

- [x] closes #18582
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Looks like just one error. The rest are the same as before the change:
```
 1 failed, 17689 passed, 454 skipped, 47 xfailed, 1 xpassed, 44 warnings in 1607.96 seconds
```
```
================================== FAILURES ===================================
_________________ TestDataFrameIndexing.test_setitem_boolean __________________

self = <pandas.tests.frame.test_indexing.TestDataFrameIndexing object at 0x000002C9E29D2A90>

    def test_setitem_boolean(self):
        df = self.frame.copy()
        values = self.frame.values

        df[df['A'] > 0] = 4
        values[values[:, 0] > 0] = 4
        assert_almost_equal(df.values, values)

        # test that column reindexing works
        series = df['A'] == 4
        series = series.reindex(df.index[::-1])
        df[series] = 1
        values[values[:, 0] == 4] = 1
        assert_almost_equal(df.values, values)

        df[df > 0] = 5
        values[values > 0] = 5
        assert_almost_equal(df.values, values)

        df[df == 5] = 0
        values[values == 5] = 0
        assert_almost_equal(df.values, values)

        # a df that needs alignment first
        df[df[:-1] < 0] = 2
        np.putmask(values[:-1], values[:-1] < 0, 2)
        assert_almost_equal(df.values, values)

        # indexed with same shape but rows-reversed df
        df[df[::-1] == 2] = 3
        values[values == 2] = 3
        assert_almost_equal(df.values, values)

        with tm.assert_raises_regex(TypeError, 'Must pass '
                                    'DataFrame with '
                                    'boolean values only'):
>           df[df * 0] = 2

pandas\tests\frame\test_indexing.py:528:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self =                    A         B    C         D
YZQTztQ8Ng  0.000000  0.000000  0.0  0.000000
lYi3UlJmyD  3.000000  3.00...00  0.000000  0.0  0.000000
8RLavvGvaC  3.000000  0.000000  0.0  3.000000
S4V8aOm8PB -2.325562 -0.382488  0.0 -0.010254
key =               A    B    C    D
YZQTztQ8Ng  0.0  0.0  0.0  0.0
lYi3UlJmyD  0.0  0.0  0.0  0.0
PAZ8DZYTxG  0.0  0.0  0.0...rCoQFA  0.0  0.0  0.0  0.0
O4jF5wCtd2  0.0  0.0  0.0  0.0
8RLavvGvaC  0.0  0.0  0.0  0.0
S4V8aOm8PB -0.0 -0.0  0.0 -0.0
value = 2

    def __setitem__(self, key, value):
        key = com._apply_if_callable(key, self)

        # see if we can slice the rows
        indexer = convert_to_index_sliceable(self, key)
        if indexer is not None:
            return self._setitem_slice(indexer, value)

        if isinstance(key, DataFrame) or getattr(key, 'ndim', None) == 2:
>           self._setitem_frame(key, value)

pandas\core\frame.py:2534:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self =                    A         B    C         D
YZQTztQ8Ng  0.000000  0.000000  0.0  0.000000
lYi3UlJmyD  3.000000  3.00...00  0.000000  0.0  0.000000
8RLavvGvaC  3.000000  0.000000  0.0  3.000000
S4V8aOm8PB -2.325562 -0.382488  0.0 -0.010254
key =               A    B    C    D
YZQTztQ8Ng  0.0  0.0  0.0  0.0
lYi3UlJmyD  0.0  0.0  0.0  0.0
PAZ8DZYTxG  0.0  0.0  0.0...rCoQFA  0.0  0.0  0.0  0.0
O4jF5wCtd2  0.0  0.0  0.0  0.0
8RLavvGvaC  0.0  0.0  0.0  0.0
S4V8aOm8PB -0.0 -0.0  0.0 -0.0
value = 2

    def _setitem_frame(self, key, value):
        # support boolean setting with DataFrame input, e.g.
        # df[df > df2] = 0
        if isinstance(key, np.ndarray):
            if key.shape != self.shape:
                raise ValueError(
                    'Array conditional must be same shape as self'
                )
            key = self._constructor(key, **self._construct_axes_dict())

        if key.values.size and not is_bool_dtype(key.values):
            raise TypeError(
>               'Must pass DataFrame or 2-d ndarray with boolean values only'
            )
E           TypeError: Must pass DataFrame or 2-d ndarray with boolean values only

pandas\core\frame.py:2578: TypeError

During handling of the above exception, another exception occurred:

self = <pandas.tests.frame.test_indexing.TestDataFrameIndexing object at 0x000002C9E29D2A90>

    def test_setitem_boolean(self):
        df = self.frame.copy()
        values = self.frame.values

        df[df['A'] > 0] = 4
        values[values[:, 0] > 0] = 4
        assert_almost_equal(df.values, values)

        # test that column reindexing works
        series = df['A'] == 4
        series = series.reindex(df.index[::-1])
        df[series] = 1
        values[values[:, 0] == 4] = 1
        assert_almost_equal(df.values, values)

        df[df > 0] = 5
        values[values > 0] = 5
        assert_almost_equal(df.values, values)

        df[df == 5] = 0
        values[values == 5] = 0
        assert_almost_equal(df.values, values)

        # a df that needs alignment first
        df[df[:-1] < 0] = 2
        np.putmask(values[:-1], values[:-1] < 0, 2)
        assert_almost_equal(df.values, values)

        # indexed with same shape but rows-reversed df
        df[df[::-1] == 2] = 3
        values[values == 2] = 3
        assert_almost_equal(df.values, values)

        with tm.assert_raises_regex(TypeError, 'Must pass '
                                    'DataFrame with '
                                    'boolean values only'):
>           df[df * 0] = 2

pandas\tests\frame\test_indexing.py:528:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pandas\util\testing.py:2489: in __exit__
    return self.exception_matches(exc_type, exc_value, trace_back)
pandas\util\testing.py:2525: in exception_matches
    raise_with_traceback(e, trace_back)
pandas\compat\__init__.py:385: in raise_with_traceback
    raise exc.with_traceback(traceback)
pandas\tests\frame\test_indexing.py:528: in test_setitem_boolean
    df[df * 0] = 2
pandas\core\frame.py:2534: in __setitem__
    self._setitem_frame(key, value)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self =                    A         B    C         D
YZQTztQ8Ng  0.000000  0.000000  0.0  0.000000
lYi3UlJmyD  3.000000  3.00...00  0.000000  0.0  0.000000
8RLavvGvaC  3.000000  0.000000  0.0  3.000000
S4V8aOm8PB -2.325562 -0.382488  0.0 -0.010254
key =               A    B    C    D
YZQTztQ8Ng  0.0  0.0  0.0  0.0
lYi3UlJmyD  0.0  0.0  0.0  0.0
PAZ8DZYTxG  0.0  0.0  0.0...rCoQFA  0.0  0.0  0.0  0.0
O4jF5wCtd2  0.0  0.0  0.0  0.0
8RLavvGvaC  0.0  0.0  0.0  0.0
S4V8aOm8PB -0.0 -0.0  0.0 -0.0
value = 2

    def _setitem_frame(self, key, value):
        # support boolean setting with DataFrame input, e.g.
        # df[df > df2] = 0
        if isinstance(key, np.ndarray):
            if key.shape != self.shape:
                raise ValueError(
                    'Array conditional must be same shape as self'
                )
            key = self._constructor(key, **self._construct_axes_dict())

        if key.values.size and not is_bool_dtype(key.values):
            raise TypeError(
>               'Must pass DataFrame or 2-d ndarray with boolean values only'
            )
E           AssertionError: "Must pass DataFrame with boolean values only" does not match "Must pass DataFrame or 2-d ndarray with boolean values only"

pandas\core\frame.py:2578: AssertionError
```
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18645?src=pr&el=h1) Report
> Merging [#18645](https://codecov.io/gh/pandas-dev/pandas/pull/18645?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c24a0d2815b717a33edc5de9d3f09eeecc810670?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `87.5%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18645/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/18645?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18645      +/-   ##
==========================================
- Coverage   91.57%   91.55%   -0.03%     
==========================================
  Files         150      150              
  Lines       48933    48937       +4     
==========================================
- Hits        44812    44803       -9     
- Misses       4121     4134      +13
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.91% <87.5%> (-0.03%)` | :arrow_down: |
| #single | `41.75% <25%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18645?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18645/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.63% <87.5%> (-0.05%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/18645/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.22% <0%> (-1.74%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18645?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18645?src=pr&el=footer). Last update [c24a0d2...c2da170](https://codecov.io/gh/pandas-dev/pandas/pull/18645?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Hello @dhirschfeld! Thanks for updating the PR.





Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on December 30, 2017 at 14:24 Hours UTC
@jreback - test moved to it's own function and whatsnew entry added
Just thinking about this - as a bug fix wouldn't it be more appropriate for 0.21.1? 
> Just thinking about this - as a bug fix wouldn't it be more appropriate for 0.21.1?

yes its a bug fix, but it is substantially different, so put in 0.22. folks may be relying on this behavior (event though its wrong).
Fair enough. I think it's good to go then if there are no more changes you'd like?
pls rebase & ping on green (the 3.6 build is failing unrelated so that may fail the build), but otherwise.
@jreback - should be good to go I think...
pushed a change to the more common form of parametrization. ping on green.
thanks @dhirschfeld !
Thanks for getting this over the line @jreback!