Repository: pandas-dev/pandas

Issue:
DataFrame with PeriodIndex written in Python2 gets an Int64Index when read back in Python3
#### Code Sample, a copy-pastable example if possible

```
df = pd.DataFrame([1.,2,3], index=pd.PeriodIndex(['2015-01-01', '2015-01-02', '2015-01-05'], freq='B'))
df.to_hdf('pind2.7', 'p')
```
#### Problem description

Writing this DataFrame out via HDF5 in Python2.7, then reading back in Python3.6 causes the DataFrame to have an Int64Index rather than a PeriodIndex:

**in python3.6:**
```
In [6]: pd.read_hdf('pind2.7').index
Out[6]: Int64Index([11740, 11741, 11742], dtype='int64')
```

#### Expected Output

The deserialized index should look like this:

**in python2.7**:
```
In [5]: pd.read_hdf('pind2.7').index
Out[5]: PeriodIndex(['2015-01-01', '2015-01-02', '2015-01-05'], dtype='period[B]', freq='B')
```

The index is correctly deserialized when reading it back in from Python2.7.  The index is also correctly deserialized when reading a file that was written in Python3.6.  I don't know if we guarantee 2.7 -> 3.6 compatibility for an on-disk HDF file?

I've attached an HDF file written in Python2.7 and a similar HDF file written in Python3.6.

[pind.zip](https://github.com/pandas-dev/pandas/files/1106157/pind.zip)


#### Output of ``pd.show_versions()``

<details>
pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.1.final.0
python-bits: 64
OS: Darwin
OS-release: 16.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.20.1
pytest: 3.0.7
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
xarray: None
IPython: 5.3.0
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.3.0
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.7
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.3
bs4: 4.6.0
html5lib: 0.999
sqlalchemy: 1.1.9
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
It's the index_class coming out as the bytes b'period' rather than the string 
period' in Python3 that seems to be the issue... this hack seems to fix the problem but I don't know what the right way is:

```
--- pytables.py 2017-05-08 14:40:21.000000000 -0700
+++ pytables.py 2017-06-27 12:37:50.000000000 -0700
@@ -2563,8 +2563,12 @@
         if 'name' in node._v_attrs:
             name = node._v_attrs.name

-        index_class = self._alias_to_class(getattr(node._v_attrs,
-                                                   'index_class', ''))
+        index_class = getattr(node._v_attrs, 'index_class', '')
+
+        if index_class == b'period':
+            index_class = 'period'
+
+        index_class = self._alias_to_class(index_class)
         factory = self._get_index_factory(index_class)

         kwargs = {}
```
I can confirm this on master

>  I don't know if we guarantee 2.7 -> 3.6 compatibility for an on-disk HDF file?

That's indeed the good question. Our docs at least don't mention it is not supported. 
@jreback ?
this is for a ``fixed`` store
we have very little testing or support for ``PeriodIndex``. That said would accept a PR to fix this.

The way this is done in the code is probably not ideal. The hard part here is make this backward compat. 
I think you can use

``_ensure_decoded(index_class)`` and this will just work.
Confirmed.. this fixes the issue for me:

```
--- pytables.py 2017-05-08 14:40:21.000000000 -0700
+++ pytables.py 2017-06-28 06:15:57.000000000 -0700
@@ -2563,8 +2563,8 @@
         if 'name' in node._v_attrs:
             name = node._v_attrs.name

-        index_class = self._alias_to_class(getattr(node._v_attrs,
-                                                   'index_class', ''))
+        index_class = self._alias_to_class(_ensure_decoded(
+            getattr(node._v_attrs, 'index_class', '')))
         factory = self._get_index_factory(index_class)

         kwargs = {}
```
great if u can put a PR with the above test and fix would be great
ok i'll work on it today.

Resolving patch:
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -2591,8 +2591,8 @@ def read_index_node(self, node, start=None, stop=None):
         if 'name' in node._v_attrs:
             name = _ensure_str(node._v_attrs.name)
 
-        index_class = self._alias_to_class(getattr(node._v_attrs,
-                                                   'index_class', ''))
+        index_class = self._alias_to_class(_ensure_decoded(
+            getattr(node._v_attrs, 'index_class', '')))
         factory = self._get_index_factory(index_class)
 
         kwargs = {}


Test patch:
diff --git a/pandas/tests/io/data/periodindex_0.20.1_x86_64_darwin_2.7.13.h5 b/pandas/tests/io/data/periodindex_0.20.1_x86_64_darwin_2.7.13.h5
new file mode 100644
Binary files /dev/null and b/pandas/tests/io/data/periodindex_0.20.1_x86_64_darwin_2.7.13.h5 differ
diff --git a/pandas/tests/io/test_pytables.py b/pandas/tests/io/test_pytables.py
--- a/pandas/tests/io/test_pytables.py
+++ b/pandas/tests/io/test_pytables.py
@@ -5264,6 +5264,26 @@ def test_fspath(self):
             with pd.HDFStore(path) as store:
                 assert os.fspath(store) == str(path)
 
+    def test_read_py2_hdf_file_in_py3(self):
+        # GH 16781
+
+        # tests reading a PeriodIndex DataFrame written in Python2 in Python3
+
+        # the file was generated in Python 2.7 like so:
+        #
+        # df = pd.DataFrame([1.,2,3], index=pd.PeriodIndex(
+        #              ['2015-01-01', '2015-01-02', '2015-01-05'], freq='B'))
+        # df.to_hdf('periodindex_0.20.1_x86_64_darwin_2.7.13.h5', 'p')
+
+        expected = pd.DataFrame([1., 2, 3], index=pd.PeriodIndex(
+            ['2015-01-01', '2015-01-02', '2015-01-05'], freq='B'))
+
+        with ensure_clean_store(
+                tm.get_data_path('periodindex_0.20.1_x86_64_darwin_2.7.13.h5'),
+                mode='r') as store:
+            result = store['p']
+            assert_frame_equal(result, expected)
+
 
 class TestHDFComplexValues(Base):
     # GH10447