Repository: pandas-dev/pandas

Issue:
Boolean masks on dataframes with indexes with timezones introduces NaNs
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
import numpy as np
mask = np.array([False, True, True, False])

idx = pd.date_range('20010101', '20020101')[:4].tz_localize('UTC')
df = pd.DataFrame({'a' : np.arange(4)}, index=idx).astype('float64')
fill = 2 * df
df.loc[mask, :] = fill.loc[mask, :]
print df

idx = pd.date_range('20010101', '20020101')[:4]
df = pd.DataFrame({'a' : np.arange(4)}, index=idx).astype('float64')
fill = 2 * df
df.loc[mask, :] = fill.loc[mask, :]
print df
```
#### Problem description

This is the output that I get:

```
                            a
2001-01-01 00:00:00+00:00   0
2001-01-02 00:00:00+00:00 NaN
2001-01-03 00:00:00+00:00 NaN
2001-01-04 00:00:00+00:00   3
            a
2001-01-01  0
2001-01-02  2
2001-01-03  4
2001-01-04  3

```


In the first case, a timezone of UTC seems to cause the masking operation to introduce NaNs.  The second case is correct.  This problem seems to be introduced in pandas 0.17.1, whereas with 0.16 I get the following output below

#### Expected Output

```
                           a
2001-01-01 00:00:00+00:00  0
2001-01-02 00:00:00+00:00  2
2001-01-03 00:00:00+00:00  4
2001-01-04 00:00:00+00:00  3
            a
2001-01-01  0
2001-01-02  2
2001-01-03  4
2001-01-04  3
```

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.13.final.0
python-bits: 64
OS: Darwin
OS-release: 14.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.17.1
nose: None
pip: 9.0.1
setuptools: 27.2.0
Cython: None
numpy: 1.10.2
scipy: 0.16.0
statsmodels: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.4.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
Jinja2: None

</details>



Discussion:
The problem seems to be related to how `.loc` is being used.  The following code fixes the issue

```
df.loc[mask] = fill.loc[mask]
```
So i've dug a bit further.  This line, 

https://github.com/pandas-dev/pandas/blob/master/pandas/core/indexing.py#L764

converts the DatetimeIndex into a datetime64[ns] NumPy array.

reindexing a tz-aware DatetimeIndex  with a datetime64[ns] NumPy array does not work, whereas reindexing a non tz-aware DatetimeIndex with a datetime64[ns] NumPy array does work.  It's unclear to me whether the initial conversion to the NumPy array is the problem, or if it's the re-indexing operation which should work.

This little snippet illustrates the reindexing behavior:

```
idx = pd.date_range('20010101', '20020101')[:4].tz_localize('UTC')
df = pd.DataFrame({'a' : np.arange(4)}, index=idx).astype('float64')
print df.reindex(index=df.index[[0, 1]])
print df.reindex(index=df.index[[0, 1]].values)

idx = pd.date_range('20010101', '20020101')[:4]
df2 = pd.DataFrame({'a' : np.arange(4)}, index=idx).astype('float64')
print df2.reindex(index=df2.index[[0, 1]])
print df2.reindex(index=df2.index[[0, 1]].values)
```
The output I get is:

```
                             a
2001-01-01 00:00:00+00:00  0.0
2001-01-02 00:00:00+00:00  1.0
             a
2001-01-01 NaN
2001-01-02 NaN
              a
2001-01-01  0.0
2001-01-02  1.0
              a
2001-01-01  0.0
2001-01-02  1.0
```
I'm concerned by this - This is pretty core functionality, and it seems to have been broken from 0.17-0.20.  Should we stop using tz-aware DatetimeIndexes?

Thanks for any guidance


> I'm concerned by this - This is pretty core functionality, and it seems to have been broken from 0.17-0.20. Should we stop using tz-aware DatetimeIndexes?

well DTI tz-aware were refactored in 0.17 to make them way more performant. Sure there may be some bugs out there. PR's to fix are welcome.

```
(pandas) bash-3.2$ git diff
diff --git a/pandas/core/indexing.py b/pandas/core/indexing.py
index ae0aaf9..4fe8942 100755
--- a/pandas/core/indexing.py
+++ b/pandas/core/indexing.py
@@ -761,9 +761,9 @@ class _NDFrameIndexer(object):
                 ax = self.obj.axes[i]
                 if is_sequence(ix) or isinstance(ix, slice):
                     if idx is None:
-                        idx = ax[ix].ravel()
+                        idx = ax[ix]
                     elif cols is None:
-                        cols = ax[ix].ravel()
+                        cols = ax[ix]
                     else:
                         break
                 else:
```
fixes this case (though need to check whether other things are break).


Thanks, I'll give it a shot
taking out the ravel breaks some of the iloc test cases - it seems that data comes in sometimes looking like the output of np.ix_

```
(array([[0],
       [1]]), array([[1, 2]]))
```

However if I do this, all the indexing tests pass (as well as my broken example)

```
            for i, ix in enumerate(indexer):
                ax = self.obj.axes[i]
                if is_sequence(ix) or isinstance(ix, slice):
                    if isinstance(ix, np.ndarray):
                        ix = ix.ravel()
                    if idx is None:
                        idx = ax[ix]
                    elif cols is None:
                        cols = ax[ix]
                    else:
                        break
                else:
                    sindexers.append(i)
```

This seems more correct to me, but I'm not to familiar with how pandas indexing is implemented

Resolving patch:
diff --git a/pandas/core/indexing.py b/pandas/core/indexing.py
--- a/pandas/core/indexing.py
+++ b/pandas/core/indexing.py
@@ -760,10 +760,12 @@ def _align_frame(self, indexer, df):
             for i, ix in enumerate(indexer):
                 ax = self.obj.axes[i]
                 if is_sequence(ix) or isinstance(ix, slice):
+                    if isinstance(ix, np.ndarray):
+                        ix = ix.ravel()
                     if idx is None:
-                        idx = ax[ix].ravel()
+                        idx = ax[ix]
                     elif cols is None:
-                        cols = ax[ix].ravel()
+                        cols = ax[ix]
                     else:
                         break
                 else:


Test patch:
diff --git a/pandas/tests/indexing/test_datetime.py b/pandas/tests/indexing/test_datetime.py
--- a/pandas/tests/indexing/test_datetime.py
+++ b/pandas/tests/indexing/test_datetime.py
@@ -8,6 +8,33 @@
 
 class TestDatetimeIndex(object):
 
+    def test_setitem_with_datetime_tz(self):
+        # 16889
+        # support .loc with alignment and tz-aware DatetimeIndex
+        mask = np.array([True, False, True, False])
+
+        idx = pd.date_range('20010101', periods=4, tz='UTC')
+        df = pd.DataFrame({'a': np.arange(4)}, index=idx).astype('float64')
+
+        result = df.copy()
+        result.loc[mask, :] = df.loc[mask, :]
+        tm.assert_frame_equal(result, df)
+
+        result = df.copy()
+        result.loc[mask] = df.loc[mask]
+        tm.assert_frame_equal(result, df)
+
+        idx = pd.date_range('20010101', periods=4)
+        df = pd.DataFrame({'a': np.arange(4)}, index=idx).astype('float64')
+
+        result = df.copy()
+        result.loc[mask, :] = df.loc[mask, :]
+        tm.assert_frame_equal(result, df)
+
+        result = df.copy()
+        result.loc[mask] = df.loc[mask]
+        tm.assert_frame_equal(result, df)
+
     def test_indexing_with_datetime_tz(self):
 
         # 8260