Repository: pandas-dev/pandas

Issue:
Where method does not properly handle values with datetimes with TZ
#### Code Sample, a copy-pastable example if possible
Series input:
```python
    import pandas as pd
    dts1 = pd.date_range('20150101', '20150105', tz='America/New_York')
    df1 = pd.DataFrame({'date': dts1})
    dts2 = pd.date_range('20150103', '20150107', tz='America/New_York')
    df2 = pd.DataFrame({'date': dts2})
    ser_result = df1.date.where(df1.date < df1.date[3], df2.date)
    ser_result
```
Series output:
```python
0    2015-01-01 00:00:00-05:00
1    2015-01-02 00:00:00-05:00
2    2015-01-03 00:00:00-05:00
3          1420520400000000000
4          1420606800000000000
Name: date, dtype: object
```

DataFrame input:
```python
import pandas as pd                                                                  
dts1 = pd.date_range('20150101', '20150105', tz='America/New_York')                  
df1 = pd.DataFrame({'date': dts1, 'x':np.arange(5), 'y':dts1.tz_localize(None)})     
dts2 = pd.date_range('20150103', '20150107', tz='America/New_York')                  
df2 = pd.DataFrame({'date': dts2, 'x':np.arange(3, 8), 'y':dts2.tz_localize(None)})  
mask = pd.DataFrame(True, index=df1.index, columns=df2.columns)                      
mask.iloc[3:] = False                                                                
df_result = df1.where(mask, df2)             
df_result                                           
```

DataFrame output:
```python
                       date  x          y
0 2015-01-01 00:00:00-05:00  0 2015-01-01
1 2015-01-02 00:00:00-05:00  1 2015-01-02
2 2015-01-03 00:00:00-05:00  2 2015-01-03
3 2015-01-04 00:00:00-05:00  6 2015-01-06
4 2015-01-05 00:00:00-05:00  7 2015-01-07
  2015-01-01 00:00:00-05:00
  2015-01-02 00:00:00-05:00
  2015-01-03 00:00:00-05:00
  2015-01-04 00:00:00-05:00
  2015-01-05 00:00:00-05:00
  2015-01-01 00:00:00-05:00
  2015-01-02 00:00:00-05:00
  2015-01-03 00:00:00-05:00
  2015-01-04 00:00:00-05:00
  2015-01-05 00:00:00-05:00
  2015-01-03 00:00:00-05:00
  2015-01-04 00:00:00-05:00
  2015-01-05 00:00:00-05:00
  2015-01-06 00:00:00-05:00
  2015-01-07 00:00:00-05:00
  2015-01-03 00:00:00-05:00
  2015-01-04 00:00:00-05:00
  2015-01-05 00:00:00-05:00
  2015-01-06 00:00:00-05:00
  2015-01-07 00:00:00-05:00
```
#### Problem description

`where` fails on both Series and DataFrame when given values that have datetime-tz dtype. Both work fine with naive datetime values. Series ends up with a mix of datetime-tz and what appear to be i8 values; datetime-tz columns in DataFrames end up with the wrong shape (perhaps a concat along the wrong axis is occurring?).

#### Expected Output

Series:
```python
0    2015-01-01 00:00:00-05:00
1    2015-01-02 00:00:00-05:00
2    2015-01-03 00:00:00-05:00
3    2015-01-06 00:00:00-05:00
4    2015-01-07 00:00:00-05:00
Name: date, dtype: datetime64[ns, America/New_York]
```

DataFrame:
```python
                       date  x          y
0 2015-01-01 00:00:00-05:00  0 2015-01-01
1 2015-01-02 00:00:00-05:00  1 2015-01-02
2 2015-01-03 00:00:00-05:00  2 2015-01-03
3 2015-01-06 00:00:00-05:00  6 2015-01-06
4 2015-01-07 00:00:00-05:00  7 2015-01-07
```

#### Output of ``pd.show_versions()``
<details>

INSTALLED VERSIONS
------------------
commit: 2f4d393d6e0b7cfeed0691c3230ea6ece418bb20
python: 3.6.5.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 62 Stepping 4, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.24.0.dev0+113.g263386389
pytest: 3.6.0
pip: 10.0.1
setuptools: 39.2.0
Cython: 0.28.3
numpy: 1.14.2
scipy: 1.0.0
pyarrow: 0.8.0
xarray: 0.10.6
IPython: 6.4.0
sphinx: 1.7.5
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: 3.4.3
numexpr: 2.6.4
feather: 0.4.0
matplotlib: 2.2.2
openpyxl: 2.5.3
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.5
lxml: 4.1.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.8
pymysql: 0.8.1
psycopg2: None
jinja2: 2.10
s3fs: 0.1.5
fastparquet: 0.1.5
pandas_gbq: None
pandas_datareader: None
</details>

Series (but not DataFrame) combine_first() loses timezone information
#### Code Sample, a copy-pastable example if possible

```python
dts1 = pd.date_range('20150101','20150105',tz='UTC')  
df1 = pd.DataFrame({'DATE':dts1})                     
dts2 = pd.date_range('20150103','20150105',tz='UTC')  
df2 = pd.DataFrame({'DATE':dts2})                     
df = df1.combine_first(df2)                           
df.DATE[0].tz # returns<UTC>, 10567 fixed

ser = df1['DATE'].combine_first(df2['DATE'])
ser[0].tz  # returns None, should be <UTC> as above

```
#### Problem description

Calling `Series.combine_first` on two tz-localized datetime Series returns a non-localized Series.

#10567 handled the case when running `DataFrame.combine_first` on DataFrames with datetime tz columns. Oddly, this does not work for Series. This behavior is the same under at least both 0.19.2 and latest master so it appears it may never have been fixed with #10567.

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit: 576d5c6b76e039a411a7cc4c0de29813e2de0149
python: 3.6.5.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 62 Stepping 4, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.24.0.dev0+103.g576d5c6b7
pytest: 3.6.0
pip: 10.0.1
setuptools: 39.2.0
Cython: 0.28.3
numpy: 1.14.2
scipy: 1.0.0
pyarrow: 0.8.0
xarray: 0.10.6
IPython: 6.4.0
sphinx: 1.7.5
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: 3.4.3
numexpr: 2.6.4
feather: 0.4.0
matplotlib: 2.2.2
openpyxl: 2.5.3
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.5
lxml: 4.1.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.8
pymysql: 0.8.1
psycopg2: None
jinja2: 2.10
s3fs: 0.1.5
fastparquet: 0.1.5
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
xrefing my https://github.com/pandas-dev/pandas/issues/21469#issuecomment-398626548 and probably the root cause for #21469
@mroeschke Do you have an idea why there's the weirdly shaped result from DataFrame.where with a tz column? I think you're more familiar with the internals here than I am.
@mroeschke I think I figured out at least the weirdly-shaped result issue here (not sure about the 1-D case). As indicated at [your comment on #21469](https://github.com/pandas-dev/pandas/issues/21469#issuecomment-398626548) the issue is in `Block.where`. The current implementation implicitly assumes that `Block.values` for a DataFrame is a 2-D ndarray. However, if the `Block` is DatetimeTZ-type, `Block.values` gives back a (1-D) DatetimeIndex with the correct TZ. This in turn is apparently confusing the `where` function into thinking broadcasting is required.
Thanks for investigating @Liam3851. 

IIRC DatetimeTZBlocks only contain data (`.values`) in 1D because tz may not be conserved across both axis in a 2D sense e.g. one column has UTC data and other has US/Eastern data. This _may_ be remedied once `ExtentionArrays` for DatetimeTZ data are implemented, Otherwise, this may need to be patched by defining or patching a `where` method for `DatetimeTZBlock`s 
@mroeschke Been digging into this a bit further. Do you know, what is the contract regarding the dimension of `Block.values`, `Block.get_values`, and `Block.internal_values`? It's not entirely clear from the pandas.core.internals source.

In particular, `NonConsolidatableMixin.get_values` has the comment:

''' need to to_dense myself (**and always return a ndim sized object**) '''

DatetimeTZBlock overrides `NonConsolidatableMixin.get_values`, and has 1-dimensional `get_values`/`values` (a DatetimeIndex), but for blocks inside a DataFrame, `self.ndim==2`. That seems to violate the comment above, but maybe that's more a description than a required contract for subclasses?

Similarly in `DatetimeTZBlock._try_coerce_args`, right now we have a call:
```
values = _block_shape(values.asi8, ndim=self.ndim)
```
which ensures that values is self.ndim (2) dimensions. However, if `other` is also a DatetimeIndex we just get the 1-dimensional values of `other`. Seems clear these should be consistent, just not sure whether in general `_try_coerce_args` should always return something 

1. the same dimension/type as `self.values` (internal), or
1. the same dimension as `self.ndim`

Thoughts?

@jreback architected the `DatetimeTZBlock` (and feel free to correct me), but from what I understand, `DatetimeTZBlock` holds its `.values` as a 1D array (i.e. DatetimeIndex) while other block types can hold its values in 1 or 2D arrays (numpy arrays mostly). 

From what I understand at a high level for DataFrames, it's composed from a group of Blocks that groups the values of the DataFrame by dtype along the columns. In most cases these Blocks can store 2D values, so overall the collection (concatination) of 2D values from the Blocks comprise the overall 2D dimension of the DataFrame.

The reason why `DatetimeTZBlock` doesn't hold 2D arrays is that the timezone is assumed to be consistent along the columns and not necessarily the rows; therefore, you need 1 DatetimeTZBlock per DataFrame column with a Datetimetz dtype. Other block types AFAIK don't have similar limitations so 1 Block can hold values for multiple DataFrame columns.

Beyond that, I am not familiar with how the internals makes sure the dimensions line up or routines that would coerce `DatetimeTZBlock.values` from 1D to 2D (and if there were it would probably strip the tz information)
@mroeschke expl is correct. yeah its currently a bummer that we aren't super consistent internally how we handle naive datetimes (2D) vs tz-aware (1D), but that's the direction we have been moving. So various compat things exist to handle this. 

We basically have too many accessors that do slightly different things, but the cleanup is non-trivial / hard. For this issue, prob just need to see where its getting coerced to 2D (or vice-versa) and maybe override the method in ``DatetimeTZBlock`` or put comething that can generically handle.

here's a patch which works on the above example. doesn't seem to break any series tests, but have a go. (breaks something in dataframe tests :<)

```
(pandas) bash-3.2$ git diff
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
index fe508dc1b..f2b732b57 100644
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -1495,7 +1495,7 @@ class Block(PandasObject):
                 return values
 
             values, values_mask, other, other_mask = self._try_coerce_args(
-                values, other)
+                values, orig_other)
 
             try:
                 return self._try_coerce_result(expressions.where(
```
I'm +1 for consistency.  Investigation and PR are welcome!
Hmm.. looks like this might be the result of a more general issue with `where` when `other` is a Series-- we're losing type info. `combine_first` appears to be delegating its implementation to `where`'s internals.

```
dts1 = pd.date_range('20150101','20150105',tz='UTC')  
df1 = pd.DataFrame({'date':dts1})                     
dts2 = pd.date_range('20150103','20150107',tz='UTC')  
df2 = pd.DataFrame({'date':dts2})                     
df1.date.where(df1.date < df1.date[3], df2.date)

Out[42]:
0    2015-01-01 00:00:00+00:00
1    2015-01-02 00:00:00+00:00
2    2015-01-03 00:00:00+00:00
3          1420502400000000000
4          1420588800000000000
Name: date, dtype: object
```

Actually it looks like there where issue might only be tangentially related... `Series.combine_first` refers to `pd.core.common._where_compat`, but despite the name `_where_compat` is not referenced in `where`.
If I were to guess, this may be a problem in the `where` property defined in the `SingleBlockManager`

https://github.com/pandas-dev/pandas/blob/5fbb683712ce0312e35e06152cf8410c33cee330/pandas/core/internals.py#L1449

In general, data is operated as numpy arrays and therefore tz information will be discarded (and not appropriated considered when remerging data)

Resolving patch:
diff --git a/pandas/core/common.py b/pandas/core/common.py
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -14,7 +14,7 @@
 from pandas.compat import long, zip, iteritems, PY36, OrderedDict
 from pandas.core.config import get_option
 from pandas.core.dtypes.generic import ABCSeries, ABCIndex
-from pandas.core.dtypes.common import _NS_DTYPE, is_integer
+from pandas.core.dtypes.common import is_integer
 from pandas.core.dtypes.inference import _iterable_not_string
 from pandas.core.dtypes.missing import isna, isnull, notnull  # noqa
 from pandas.core.dtypes.cast import construct_1d_object_array_from_listlike
@@ -410,19 +410,6 @@ def _apply_if_callable(maybe_callable, obj, **kwargs):
     return maybe_callable
 
 
-def _where_compat(mask, arr1, arr2):
-    if arr1.dtype == _NS_DTYPE and arr2.dtype == _NS_DTYPE:
-        new_vals = np.where(mask, arr1.view('i8'), arr2.view('i8'))
-        return new_vals.view(_NS_DTYPE)
-
-    if arr1.dtype == _NS_DTYPE:
-        arr1 = tslib.ints_to_pydatetime(arr1.view('i8'))
-    if arr2.dtype == _NS_DTYPE:
-        arr2 = tslib.ints_to_pydatetime(arr2.view('i8'))
-
-    return np.where(mask, arr1, arr2)
-
-
 def _dict_compat(d):
     """
     Helper function to convert datetimelike-keyed dicts to Timestamp-keyed dict
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -1476,7 +1476,7 @@ def where(self, other, cond, align=True, errors='raise',
         if transpose:
             values = values.T
 
-        other = getattr(other, 'values', other)
+        other = getattr(other, '_values', getattr(other, 'values', other))
         cond = getattr(cond, 'values', cond)
 
         # If the default broadcasting would go in the wrong direction, then
@@ -1484,6 +1484,8 @@ def where(self, other, cond, align=True, errors='raise',
         if getattr(other, 'ndim', 0) >= 1:
             if values.ndim - 1 == other.ndim and axis == 1:
                 other = other.reshape(tuple(other.shape + (1, )))
+            elif transpose and values.ndim == self.ndim - 1:
+                cond = cond.T
 
         if not hasattr(cond, 'shape'):
             raise ValueError("where must have a condition that is ndarray "
@@ -2888,8 +2890,8 @@ def _try_coerce_args(self, values, other):
         elif isinstance(other, self._holder):
             if other.tz != self.values.tz:
                 raise ValueError("incompatible or non tz-aware value")
-            other = other.asi8
-            other_mask = isna(other)
+            other_mask = _block_shape(isna(other), ndim=self.ndim)
+            other = _block_shape(other.asi8, ndim=self.ndim)
         elif isinstance(other, (np.datetime64, datetime, date)):
             other = tslib.Timestamp(other)
             tz = getattr(other, 'tz', None)
diff --git a/pandas/core/series.py b/pandas/core/series.py
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -22,6 +22,7 @@
     is_float_dtype,
     is_extension_type,
     is_extension_array_dtype,
+    is_datetimelike,
     is_datetime64tz_dtype,
     is_timedelta64_dtype,
     is_object_dtype,
@@ -78,6 +79,7 @@
 from pandas._libs import index as libindex, tslib as libts, lib, iNaT
 from pandas.core.config import get_option
 from pandas.core.strings import StringMethods
+from pandas.core.tools.datetimes import to_datetime
 
 import pandas.plotting._core as gfx
 
@@ -2303,10 +2305,10 @@ def combine_first(self, other):
         new_index = self.index.union(other.index)
         this = self.reindex(new_index, copy=False)
         other = other.reindex(new_index, copy=False)
-        # TODO: do we need name?
-        name = ops.get_op_result_name(self, other)  # noqa
-        rs_vals = com._where_compat(isna(this), other._values, this._values)
-        return self._constructor(rs_vals, index=new_index).__finalize__(self)
+        if is_datetimelike(this) and not is_datetimelike(other):
+            other = to_datetime(other)
+
+        return this.where(notna(this), other)
 
     def update(self, other):
         """


Test patch:
diff --git a/pandas/tests/frame/test_indexing.py b/pandas/tests/frame/test_indexing.py
--- a/pandas/tests/frame/test_indexing.py
+++ b/pandas/tests/frame/test_indexing.py
@@ -2936,6 +2936,20 @@ def test_where_callable(self):
         tm.assert_frame_equal(result,
                               (df + 2).where((df + 2) > 8, (df + 2) + 10))
 
+    def test_where_tz_values(self, tz_naive_fixture):
+        df1 = DataFrame(DatetimeIndex(['20150101', '20150102', '20150103'],
+                                      tz=tz_naive_fixture),
+                        columns=['date'])
+        df2 = DataFrame(DatetimeIndex(['20150103', '20150104', '20150105'],
+                                      tz=tz_naive_fixture),
+                        columns=['date'])
+        mask = DataFrame([True, True, False], columns=['date'])
+        exp = DataFrame(DatetimeIndex(['20150101', '20150102', '20150105'],
+                                      tz=tz_naive_fixture),
+                        columns=['date'])
+        result = df1.where(mask, df2)
+        assert_frame_equal(exp, result)
+
     def test_mask(self):
         df = DataFrame(np.random.randn(5, 3))
         cond = df > 0
diff --git a/pandas/tests/indexing/test_coercion.py b/pandas/tests/indexing/test_coercion.py
--- a/pandas/tests/indexing/test_coercion.py
+++ b/pandas/tests/indexing/test_coercion.py
@@ -580,12 +580,11 @@ def test_where_series_datetime64(self, fill_val, exp_dtype):
         values = pd.Series(pd.date_range(fill_val, periods=4))
         if fill_val.tz:
             exp = pd.Series([pd.Timestamp('2011-01-01'),
-                             pd.Timestamp('2012-01-02 05:00'),
+                             pd.Timestamp('2012-01-02 00:00', tz='US/Eastern'),
                              pd.Timestamp('2011-01-03'),
-                             pd.Timestamp('2012-01-04 05:00')])
-            self._assert_where_conversion(obj, cond, values, exp,
-                                          'datetime64[ns]')
-            pytest.xfail("ToDo: do not coerce to UTC, must be object")
+                             pd.Timestamp('2012-01-04 00:00',
+                                          tz='US/Eastern')])
+            self._assert_where_conversion(obj, cond, values, exp, exp_dtype)
 
         exp = pd.Series([pd.Timestamp('2011-01-01'), values[1],
                          pd.Timestamp('2011-01-03'), values[3]])
diff --git a/pandas/tests/series/indexing/test_boolean.py b/pandas/tests/series/indexing/test_boolean.py
--- a/pandas/tests/series/indexing/test_boolean.py
+++ b/pandas/tests/series/indexing/test_boolean.py
@@ -551,6 +551,18 @@ def test_where_datetime_conversion():
     assert_series_equal(rs, expected)
 
 
+def test_where_dt_tz_values(tz_naive_fixture):
+    ser1 = pd.Series(pd.DatetimeIndex(['20150101', '20150102', '20150103'],
+                                      tz=tz_naive_fixture))
+    ser2 = pd.Series(pd.DatetimeIndex(['20160514', '20160515', '20160516'],
+                                      tz=tz_naive_fixture))
+    mask = pd.Series([True, True, False])
+    result = ser1.where(mask, ser2)
+    exp = pd.Series(pd.DatetimeIndex(['20150101', '20150102', '20160516'],
+                                     tz=tz_naive_fixture))
+    assert_series_equal(exp, result)
+
+
 def test_mask():
     # compare with tested results in test_where
     s = Series(np.random.randn(5))
diff --git a/pandas/tests/series/test_combine_concat.py b/pandas/tests/series/test_combine_concat.py
--- a/pandas/tests/series/test_combine_concat.py
+++ b/pandas/tests/series/test_combine_concat.py
@@ -170,6 +170,20 @@ def get_result_type(dtype, dtype2):
                                     ]).dtype
                 assert result.kind == expected
 
+    def test_combine_first_dt_tz_values(self, tz_naive_fixture):
+        ser1 = pd.Series(pd.DatetimeIndex(['20150101', '20150102', '20150103'],
+                                          tz=tz_naive_fixture),
+                         name='ser1')
+        ser2 = pd.Series(pd.DatetimeIndex(['20160514', '20160515', '20160516'],
+                                          tz=tz_naive_fixture),
+                         index=[2, 3, 4], name='ser2')
+        result = ser1.combine_first(ser2)
+        exp_vals = pd.DatetimeIndex(['20150101', '20150102', '20150103',
+                                     '20160515', '20160516'],
+                                    tz=tz_naive_fixture)
+        exp = pd.Series(exp_vals, name='ser1')
+        assert_series_equal(exp, result)
+
     def test_concat_empty_series_dtypes(self):
 
         # booleans