Repository: pandas-dev/pandas
Pull request: Fix read of py27 pytables tz attribute, gh#26443
When created by python 2.7, the "tz" attribute will be created with CSET
H5T_CSET_ASCII instead of H5T_CSET_UTF8, therefore it is read as bytes
when string is expected.

- [ X] closes #26443
- [ X] tests added / passed
- [ X] passes `black pandas`
- [ X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ X] whatsnew entry


Problem statement:
Py3 Failed to read HDF5 created in Py2 with datetime timezone
#### Code Sample, a copy-pastable example if possible

```python
d = {"data": 123}
df = pd.DataFrame(d, index=[pd.Timestamp("2019-01-01T18:00").tz_localize('America/New_York')])

df.to_hdf('/tmp/test.h5', key="key") # write in Python 2

df = pd.read_hdf('/tmp/test.h5', key="key") # failed in py3

print(df)
```
#### Problem description

When a dataframe contains a column with datetime with timezone, and this dataframe is written as hd5 in Python 2.  Python 3 is unable to read it.  The error is 
```
...

~/miniconda3/envs/SOMEENV/lib/python3.7/site-packages/pandas/core/indexes/datetimelike.py in _format_with_header(self, header, **kwargs)
    431 
    432     def _format_with_header(self, header, **kwargs):
--> 433         return header + list(self._format_native_types(**kwargs))
    434 
    435     @property

~/miniconda3/envs/SOMEENV/lib/python3.7/site-packages/pandas/core/indexes/datetimes.py in _format_native_types(self, na_rep, date_format, **kwargs)
    450                                                 tz=self.tz,
    451                                                 format=fmt,
--> 452                                                 na_rep=na_rep)
    453 
    454     @property

pandas/_libs/tslib.pyx in pandas._libs.tslib.format_array_from_datetime()

pandas/_libs/tslibs/timestamps.pyx in pandas._libs.tslibs.timestamps.Timestamp.__new__()

pandas/_libs/tslibs/conversion.pyx in pandas._libs.tslibs.conversion.convert_to_tsobject()

TypeError: Cannot convert numpy.bytes_ to datetime.tzinfo
```

#### Expected Output

Python 3 should be able to read the dataframe written by Python 2

#### Output of ``pd.show_versions()``

Python 3

```
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.3.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-957.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.2
pytest: 4.4.2
pip: 19.1
setuptools: 41.0.1
Cython: 0.29.7
numpy: 1.16.3
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.5.0
sphinx: None
patsy: 0.5.1
dateutil: 2.8.0
pytz: 2019.1
blosc: 1.8.1
bottleneck: None
tables: 3.5.1
numexpr: 2.6.9
feather: None
matplotlib: 3.0.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: 2.8.2 (dt dec pq3 ext lo64)
jinja2: 2.10.1
s3fs: 0.2.1
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None
```

Python 2
```
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-957.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.24.2
pytest: 4.0.2
pip: 18.1
setuptools: 40.6.3
Cython: 0.29.2
numpy: 1.15.0
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 5.8.0
sphinx: None
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: 1.5.1
bottleneck: None
tables: 3.4.4
numexpr: 2.6.8
feather: None
matplotlib: 1.5.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: 2.7.5 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: 0.2.0
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None
```

Hints from issue discussion:
+1
Seeing the same issue.
Anyone interested in debugging this?

Commit message: Fix read of py27 pytables tz attribute, gh#26443

When created by python 2.7, the "tz" attribute will be created with CSET
H5T_CSET_ASCII instead of H5T_CSET_UTF8, therefore it is read as bytes
when string is expected.

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.0.rst b/doc/source/whatsnew/v1.0.0.rst
index 050a26cc86d42..ff74965812c8c 100644
--- a/doc/source/whatsnew/v1.0.0.rst
+++ b/doc/source/whatsnew/v1.0.0.rst
@@ -97,6 +97,7 @@ Datetimelike
 ^^^^^^^^^^^^
 - Bug in :meth:`Series.__setitem__` incorrectly casting ``np.timedelta64("NaT")`` to ``np.datetime64("NaT")`` when inserting into a :class:`Series` with datetime64 dtype (:issue:`27311`)
 - Bug in :meth:`Series.dt` property lookups when the underlying data is read-only (:issue:`27529`)
+- Bug in :meth:`HDFStore.__getitem__` incorrectly reading tz attribute created in Py2 (:issue:`26443`)
 -
 
 
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
index fbe413f820c90..1ff3400323e54 100644
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -2902,7 +2902,12 @@ def read_index_node(self, node, start=None, stop=None):
             kwargs["freq"] = node._v_attrs["freq"]
 
         if "tz" in node._v_attrs:
-            kwargs["tz"] = node._v_attrs["tz"]
+            if isinstance(node._v_attrs["tz"], bytes):
+                # created by python2
+                kwargs["tz"] = node._v_attrs["tz"].decode("utf-8")
+            else:
+                # created by python3
+                kwargs["tz"] = node._v_attrs["tz"]
 
         if kind in ("date", "datetime"):
             index = factory(
diff --git a/pandas/tests/io/data/legacy_hdf/gh26443.h5 b/pandas/tests/io/data/legacy_hdf/gh26443.h5
new file mode 100644
index 0000000000000..45aa64324530f
Binary files /dev/null and b/pandas/tests/io/data/legacy_hdf/gh26443.h5 differ
diff --git a/pandas/tests/io/pytables/test_pytables.py b/pandas/tests/io/pytables/test_pytables.py
index d67f2c3b7bd66..8dca807afec64 100644
--- a/pandas/tests/io/pytables/test_pytables.py
+++ b/pandas/tests/io/pytables/test_pytables.py
@@ -5446,3 +5446,18 @@ def test_read_with_where_tz_aware_index(self):
                 store.append(key, expected, format="table", append=True)
             result = pd.read_hdf(path, key, where="DATE > 20151130")
             assert_frame_equal(result, expected)
+
+    def test_py2_created_with_datetimez(self, datapath):
+        """
+        The test HDF5 file was created in Python2, but could not be read in
+        Python3.
+
+        GH26443
+        """
+        index = [pd.Timestamp("2019-01-01T18:00").tz_localize("America/New_York")]
+        expected = DataFrame({"data": 123}, index=index)
+        with ensure_clean_store(
+            datapath("io", "data", "legacy_hdf", "gh26443.h5"), mode="r"
+        ) as store:
+            result = store["key"]
+            assert_frame_equal(result, expected)


Code review comments:
diff_hunk: @@ -97,6 +97,7 @@ Datetimelike
 ^^^^^^^^^^^^
 - Bug in :meth:`Series.__setitem__` incorrectly casting ``np.timedelta64("NaT")`` to ``np.datetime64("NaT")`` when inserting into a :class:`Series` with datetime64 dtype (:issue:`27311`)
 - Bug in :meth:`Series.dt` property lookups when the underlying data is read-only (:issue:`27529`)
+- Bug in :meth:`HDFStore.__getitem__` incorrectly reading tz attribute created in Py2 (:issue:`26443`)
path: doc/source/whatsnew/v1.0.0.rst
text: @user1:
Py2 -> Python 2.

diff_hunk: @@ -97,6 +97,7 @@ Datetimelike
 ^^^^^^^^^^^^
 - Bug in :meth:`Series.__setitem__` incorrectly casting ``np.timedelta64("NaT")`` to ``np.datetime64("NaT")`` when inserting into a :class:`Series` with datetime64 dtype (:issue:`27311`)
 - Bug in :meth:`Series.dt` property lookups when the underlying data is read-only (:issue:`27529`)
+- Bug in :meth:`HDFStore.__getitem__` incorrectly reading tz attribute created in Py2 (:issue:`26443`)
path: doc/source/whatsnew/v1.0.0.rst
text: @user1:
I don't think magic methods are in the docs.

It also seems like we don't have a generic `HDFStore` reference, so just write this as double backticks rather than :meth:.

@author:
Ok, will commit again.