Repository: pandas-dev/pandas

Issue:
BUG: selecting from HDFStore with a tz-aware level of a multi-index
I'm encountering a bug when I query for a multiindex dataframe with a timezoned DatetimeIndex in one of the multiindex levels.
This only happens
1) for a multiindex with one of the levels as timestamps with timezones (As seen in [1]). If timestamps have no timezone set, there is no issue (As seen in [2])
2) if the query returns no rows
3) in pandas 0.17.\*  This was working fine in pandas 0.16.*

``` python

In [1]: periods = 10
   ...: dts = pd.date_range('20151201', periods=periods, freq='D', tz='UTC') #WITH TIMEZONE
   ...: mi = pd.MultiIndex.from_arrays([dts, range(periods)], names = ['DATE', 'NO'])
   ...: df = pd.DataFrame({'MYCOL':0}, index=mi)
   ...: file_path = 'table.h5'
   ...: key = 'mykey'
   ...: with pd.HDFStore(file_path, 'w') as store:
   ...:     store.append(key, df, format='table', append=True)
   ...:     dfres = store.select(key, where="""DATE > '20151220'""")
   ...:     print(dfres)
   ...: 
   ...: 
Traceback (most recent call last):

  File "<ipython-input-1-e0b7db50fd4d>", line 9, in <module>
    dfres = store.select(key, where="""DATE > '20151220'""")

  File "/export/data/anaconda/anaconda3.2.4/lib/python3.5/site-packages/pandas/io/pytables.py", line 669, in select
    return it.get_result()

  File "/export/data/anaconda/anaconda3.2.4/lib/python3.5/site-packages/pandas/io/pytables.py", line 1352, in get_result
    results = self.func(self.start, self.stop, where)

  File "/export/data/anaconda/anaconda3.2.4/lib/python3.5/site-packages/pandas/io/pytables.py", line 662, in func
    columns=columns, **kwargs)

  File "/export/data/anaconda/anaconda3.2.4/lib/python3.5/site-packages/pandas/io/pytables.py", line 4170, in read
    df = super(AppendableMultiFrameTable, self).read(**kwargs)

  File "/export/data/anaconda/anaconda3.2.4/lib/python3.5/site-packages/pandas/io/pytables.py", line 4029, in read
    df = concat(frames, axis=1, verify_integrity=False).consolidate()

  File "/export/data/anaconda/anaconda3.2.4/lib/python3.5/site-packages/pandas/tools/merge.py", line 813, in concat
    return op.get_result()

  File "/export/data/anaconda/anaconda3.2.4/lib/python3.5/site-packages/pandas/tools/merge.py", line 995, in get_result
    mgrs_indexers, self.new_axes, concat_axis=self.axis, copy=self.copy)

  File "/export/data/anaconda/anaconda3.2.4/lib/python3.5/site-packages/pandas/core/internals.py", line 4456, in concatenate_block_managers
    for placement, join_units in concat_plan]

  File "/export/data/anaconda/anaconda3.2.4/lib/python3.5/site-packages/pandas/core/internals.py", line 4456, in <listcomp>
    for placement, join_units in concat_plan]

  File "/export/data/anaconda/anaconda3.2.4/lib/python3.5/site-packages/pandas/core/internals.py", line 4553, in concatenate_join_units
    for ju in join_units]

  File "/export/data/anaconda/anaconda3.2.4/lib/python3.5/site-packages/pandas/core/internals.py", line 4553, in <listcomp>
    for ju in join_units]

  File "/export/data/anaconda/anaconda3.2.4/lib/python3.5/site-packages/pandas/core/internals.py", line 4801, in get_reindexed_values
    missing_arr = np.empty(self.shape, dtype=empty_dtype)

TypeError: data type not understood


In [2]: periods = 10
   ...: dts = pd.date_range('20151201', periods=periods, freq='D') #WITHOUT TIMEZONE
   ...: mi = pd.MultiIndex.from_arrays([dts, range(periods)], names = ['DATE', 'NO'])
   ...: df = pd.DataFrame({'MYCOL':0}, index=mi)
   ...: file_path = 'table.h5'
   ...: key = 'mykey'
   ...: with pd.HDFStore(file_path, 'w') as store:
   ...:     store.append(key, df, format='table', append=True)
   ...:     dfres = store.select(key, where="""DATE > '20151220'""")
   ...:     print(dfres)
   ...: 
   ...: 
Empty DataFrame
Columns: [MYCOL]
Index: []

In [3]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.1.final.0
python-bits: 64
OS: Linux
OS-release: 2.6.32-431.11.2.el6.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.17.1
nose: 1.3.7
pip: 7.1.2
setuptools: 19.1.1
Cython: 0.23.4
numpy: 1.10.2
scipy: 0.16.1
statsmodels: None
IPython: 4.0.1
sphinx: 1.3.1
patsy: 0.4.0
dateutil: 2.4.2
pytz: 2015.7
blosc: None
bottleneck: 1.0.0
tables: 3.2.2
numexpr: 2.4.4
matplotlib: 1.5.0
openpyxl: 2.2.6
xlrd: 0.9.4
xlwt: 1.0.0
xlsxwriter: 0.7.7
lxml: 3.5.0
bs4: 4.4.1
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.10
pymysql: None
psycopg2: None
Jinja2: None

```



Discussion:
So its the readback, not the writing. I _think_ that its taking the wrong path on the dtype conversion.

```
import numpy as np
import pandas as pd

periods = 10
dts = pd.date_range('20151201', periods=periods, freq='D', tz='UTC') #WITH TIMEZONE
mi = pd.MultiIndex.from_arrays([dts, range(periods)], names = ['DATE', 'NO'])
df = pd.DataFrame({'MYCOL':0}, index=mi)

file_path = 'table.h5'
key = 'mykey'

with pd.HDFStore(file_path, 'w') as store:
   store.append(key, df, format='table', append=True)

print(pd.read_hdf(file_path, key))


dfres = pd.read_hdf(file_path, key, where="DATE > 20151220")
print(dfres)
```

Has there been any update to patch this? Any ideas on which commit broke this since 0.16\* -> 0.17*?

I'm encountering the same issue when selecting datetime64[ns, tz] data using an iterator. 

there are vast changes to the way tz's work in 0.17 vs. 0.16. see the whatsnew [here](http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#datetime-with-tz).

This is a relatively simple fix however. pull-requests are welcome.


Resolving patch:
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -23,7 +23,8 @@
 
 from pandas.core.dtypes.common import (
     ensure_object, is_categorical_dtype, is_datetime64_dtype,
-    is_datetime64tz_dtype, is_list_like, is_timedelta64_dtype)
+    is_datetime64tz_dtype, is_extension_type, is_list_like,
+    is_timedelta64_dtype)
 from pandas.core.dtypes.missing import array_equivalent
 
 from pandas import (
@@ -2647,6 +2648,9 @@ def write_multi_index(self, key, index):
                                                          index.codes,
                                                          index.names)):
             # write the level
+            if is_extension_type(lev):
+                raise NotImplementedError("Saving a MultiIndex with an "
+                                          "extension dtype is not supported.")
             level_key = '{key}_level{idx}'.format(key=key, idx=i)
             conv_level = _convert_index(lev, self.encoding, self.errors,
                                         self.format_type).set_name(level_key)


Test patch:
diff --git a/pandas/tests/indexes/datetimes/test_timezones.py b/pandas/tests/indexes/datetimes/test_timezones.py
--- a/pandas/tests/indexes/datetimes/test_timezones.py
+++ b/pandas/tests/indexes/datetimes/test_timezones.py
@@ -1094,6 +1094,18 @@ def test_dti_union_aware(self):
         assert result[0].tz.zone == 'US/Central'
         assert result[-1].tz.zone == 'US/Eastern'
 
+    def test_dti_union_mixed(self):
+        # GH 21671
+        rng = DatetimeIndex([pd.Timestamp('2011-01-01'), pd.NaT])
+        rng2 = pd.DatetimeIndex(['2012-01-01', '2012-01-02'], tz='Asia/Tokyo')
+        result = rng.union(rng2)
+        expected = Index([pd.Timestamp('2011-01-01'),
+                          pd.NaT,
+                          pd.Timestamp('2012-01-01', tz='Asia/Tokyo'),
+                          pd.Timestamp('2012-01-02', tz='Asia/Tokyo')],
+                         dtype=object)
+        tm.assert_index_equal(result, expected)
+
     @pytest.mark.parametrize('tz', [None, 'UTC', "US/Central",
                                     dateutil.tz.tzoffset(None, -28800)])
     @pytest.mark.usefixtures("datetime_tz_utc")
diff --git a/pandas/tests/indexes/multi/test_format.py b/pandas/tests/indexes/multi/test_format.py
--- a/pandas/tests/indexes/multi/test_format.py
+++ b/pandas/tests/indexes/multi/test_format.py
@@ -10,8 +10,8 @@
 def test_dtype_str(indices):
     with tm.assert_produces_warning(FutureWarning):
         dtype = indices.dtype_str
-        assert isinstance(dtype, str)
-        assert dtype == str(indices.dtype)
+    assert isinstance(dtype, str)
+    assert dtype == str(indices.dtype)
 
 
 def test_format(idx):
diff --git a/pandas/tests/io/pytables/test_pytables.py b/pandas/tests/io/pytables/test_pytables.py
--- a/pandas/tests/io/pytables/test_pytables.py
+++ b/pandas/tests/io/pytables/test_pytables.py
@@ -17,9 +17,9 @@
 
 import pandas as pd
 from pandas import (
-    Categorical, DataFrame, DatetimeIndex, Index, Int64Index, MultiIndex,
-    RangeIndex, Series, Timestamp, bdate_range, concat, date_range, isna,
-    timedelta_range)
+    Categorical, CategoricalIndex, DataFrame, DatetimeIndex, Index, Int64Index,
+    MultiIndex, RangeIndex, Series, Timestamp, bdate_range, concat, date_range,
+    isna, timedelta_range)
 import pandas.util.testing as tm
 from pandas.util.testing import (
     assert_frame_equal, assert_series_equal, set_timezone)
@@ -4749,6 +4749,19 @@ def test_select_empty_where(self, where):
                 result = pd.read_hdf(store, "df", where=where)
                 assert_frame_equal(result, df)
 
+    @pytest.mark.parametrize('idx', [
+        date_range('2019', freq='D', periods=3, tz='UTC'),
+        CategoricalIndex(list('abc'))
+    ])
+    def test_to_hdf_multiindex_extension_dtype(self, idx):
+        # GH 7775
+        mi = MultiIndex.from_arrays([idx, idx])
+        df = pd.DataFrame(0, index=mi, columns=['a'])
+        with ensure_clean_path(self.path) as path:
+            with pytest.raises(NotImplementedError,
+                               match="Saving a MultiIndex"):
+                df.to_hdf(path, 'df')
+
 
 class TestHDFComplexValues(Base):
     # GH10447
@@ -5170,3 +5183,20 @@ def test_dst_transitions(self):
                 store.append('df', df)
                 result = store.select('df')
                 assert_frame_equal(result, df)
+
+    def test_read_with_where_tz_aware_index(self):
+        # GH 11926
+        periods = 10
+        dts = pd.date_range('20151201', periods=periods,
+                            freq='D', tz='UTC')
+        mi = pd.MultiIndex.from_arrays([dts, range(periods)],
+                                       names=['DATE', 'NO'])
+        expected = pd.DataFrame({'MYCOL': 0}, index=mi)
+
+        key = 'mykey'
+        with ensure_clean_path(self.path) as path:
+            with pd.HDFStore(path) as store:
+                store.append(key, expected, format='table', append=True)
+            result = pd.read_hdf(path, key,
+                                 where="DATE > 20151130")
+            assert_frame_equal(result, expected)
diff --git a/pandas/tests/reshape/test_pivot.py b/pandas/tests/reshape/test_pivot.py
--- a/pandas/tests/reshape/test_pivot.py
+++ b/pandas/tests/reshape/test_pivot.py
@@ -429,6 +429,46 @@ def test_pivot_with_tz(self, method):
             pv = pd.pivot(df, index='dt1', columns='dt2', values='data1')
         tm.assert_frame_equal(pv, expected)
 
+    def test_pivot_tz_in_values(self):
+        # GH 14948
+        df = pd.DataFrame([{'uid': u'aa',
+                            'ts': pd.Timestamp('2016-08-12 13:00:00-0700',
+                                               tz='US/Pacific')},
+                           {'uid': u'aa',
+                            'ts': pd.Timestamp('2016-08-12 08:00:00-0700',
+                                               tz='US/Pacific')},
+                           {'uid': u'aa',
+                            'ts': pd.Timestamp('2016-08-12 14:00:00-0700',
+                                               tz='US/Pacific')},
+                           {'uid': u'aa',
+                            'ts': pd.Timestamp('2016-08-25 11:00:00-0700',
+                                               tz='US/Pacific')},
+                           {'uid': u'aa',
+                            'ts': pd.Timestamp('2016-08-25 13:00:00-0700',
+                                               tz='US/Pacific')}])
+
+        df = df.set_index('ts').reset_index()
+        mins = df.ts.map(lambda x: x.replace(hour=0, minute=0,
+                                             second=0, microsecond=0))
+
+        result = pd.pivot_table(df.set_index('ts').reset_index(),
+                                values='ts', index=['uid'], columns=[mins],
+                                aggfunc=np.min)
+        expected = pd.DataFrame(
+            [
+                [pd.Timestamp('2016-08-12 08:00:00-0700', tz='US/Pacific'),
+                 pd.Timestamp('2016-08-25 11:00:00-0700', tz='US/Pacific')]
+            ],
+            index=pd.Index(['aa'], name='uid'),
+            columns=pd.DatetimeIndex(
+                [
+                    pd.Timestamp('2016-08-12 00:00:00', tz='US/Pacific'),
+                    pd.Timestamp('2016-08-25 00:00:00', tz='US/Pacific')
+                ],
+                name='ts')
+        )
+        tm.assert_frame_equal(result, expected)
+
     @pytest.mark.parametrize('method', [True, False])
     def test_pivot_periods(self, method):
         df = DataFrame({'p1': [pd.Period('2013-01-01', 'D'),