Repository: pandas-dev/pandas

Issue:
Py3 Failed to read HDF5 created in Py2 with datetime timezone
#### Code Sample, a copy-pastable example if possible

```python
d = {"data": 123}
df = pd.DataFrame(d, index=[pd.Timestamp("2019-01-01T18:00").tz_localize('America/New_York')])

df.to_hdf('/tmp/test.h5', key="key") # write in Python 2

df = pd.read_hdf('/tmp/test.h5', key="key") # failed in py3

print(df)
```
#### Problem description

When a dataframe contains a column with datetime with timezone, and this dataframe is written as hd5 in Python 2.  Python 3 is unable to read it.  The error is 
```
...

~/miniconda3/envs/SOMEENV/lib/python3.7/site-packages/pandas/core/indexes/datetimelike.py in _format_with_header(self, header, **kwargs)
    431 
    432     def _format_with_header(self, header, **kwargs):
--> 433         return header + list(self._format_native_types(**kwargs))
    434 
    435     @property

~/miniconda3/envs/SOMEENV/lib/python3.7/site-packages/pandas/core/indexes/datetimes.py in _format_native_types(self, na_rep, date_format, **kwargs)
    450                                                 tz=self.tz,
    451                                                 format=fmt,
--> 452                                                 na_rep=na_rep)
    453 
    454     @property

pandas/_libs/tslib.pyx in pandas._libs.tslib.format_array_from_datetime()

pandas/_libs/tslibs/timestamps.pyx in pandas._libs.tslibs.timestamps.Timestamp.__new__()

pandas/_libs/tslibs/conversion.pyx in pandas._libs.tslibs.conversion.convert_to_tsobject()

TypeError: Cannot convert numpy.bytes_ to datetime.tzinfo
```

#### Expected Output

Python 3 should be able to read the dataframe written by Python 2

#### Output of ``pd.show_versions()``

Python 3

```
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.3.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-957.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.2
pytest: 4.4.2
pip: 19.1
setuptools: 41.0.1
Cython: 0.29.7
numpy: 1.16.3
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.5.0
sphinx: None
patsy: 0.5.1
dateutil: 2.8.0
pytz: 2019.1
blosc: 1.8.1
bottleneck: None
tables: 3.5.1
numexpr: 2.6.9
feather: None
matplotlib: 3.0.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: 2.8.2 (dt dec pq3 ext lo64)
jinja2: 2.10.1
s3fs: 0.2.1
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None
```

Python 2
```
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-957.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.24.2
pytest: 4.0.2
pip: 18.1
setuptools: 40.6.3
Cython: 0.29.2
numpy: 1.15.0
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 5.8.0
sphinx: None
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: 1.5.1
bottleneck: None
tables: 3.4.4
numexpr: 2.6.8
feather: None
matplotlib: 1.5.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: 2.7.5 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: 0.2.0
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None
```


Discussion:
+1
Seeing the same issue.
Anyone interested in debugging this?

Resolving patch:
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -2902,7 +2902,12 @@ def read_index_node(self, node, start=None, stop=None):
             kwargs["freq"] = node._v_attrs["freq"]
 
         if "tz" in node._v_attrs:
-            kwargs["tz"] = node._v_attrs["tz"]
+            if isinstance(node._v_attrs["tz"], bytes):
+                # created by python2
+                kwargs["tz"] = node._v_attrs["tz"].decode("utf-8")
+            else:
+                # created by python3
+                kwargs["tz"] = node._v_attrs["tz"]
 
         if kind in ("date", "datetime"):
             index = factory(


Test patch:
diff --git a/pandas/tests/io/data/legacy_hdf/gh26443.h5 b/pandas/tests/io/data/legacy_hdf/gh26443.h5
new file mode 100644
Binary files /dev/null and b/pandas/tests/io/data/legacy_hdf/gh26443.h5 differ
diff --git a/pandas/tests/io/pytables/test_pytables.py b/pandas/tests/io/pytables/test_pytables.py
--- a/pandas/tests/io/pytables/test_pytables.py
+++ b/pandas/tests/io/pytables/test_pytables.py
@@ -5446,3 +5446,16 @@ def test_read_with_where_tz_aware_index(self):
                 store.append(key, expected, format="table", append=True)
             result = pd.read_hdf(path, key, where="DATE > 20151130")
             assert_frame_equal(result, expected)
+
+    def test_py2_created_with_datetimez(self, datapath):
+        # The test HDF5 file was created in Python 2, but could not be read in
+        # Python 3.
+        #
+        # GH26443
+        index = [pd.Timestamp("2019-01-01T18:00").tz_localize("America/New_York")]
+        expected = DataFrame({"data": 123}, index=index)
+        with ensure_clean_store(
+            datapath("io", "data", "legacy_hdf", "gh26443.h5"), mode="r"
+        ) as store:
+            result = store["key"]
+            assert_frame_equal(result, expected)