Repository: pandas-dev/pandas
Pull request: BUG: to_hdf and HDFStore for subclasses
- [ ] closes #33748
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Fix the bug that the subclasses of a dataframe and a series cannot be saved ``to_hdf``.
The tests were added.
When testing I found that ``assert_frame_equal`` does not work on subclasses as well - probably need to fix it in a separate PR.
For now in the tests I compare values only.

Problem statement:
BUG: to_hdf and HDFStore raise KeyError for DataFrame subclasses
- [x ] I have checked that this issue has not already been reported.

- [x ] I have confirmed this bug exists on the latest version of pandas.
---
#### Code Sample, a copy-pastable example

```python
import pandas as pd
class SubDataFrame(pd.DataFrame):
    @property
    def _constructor(self):
        return SubDataFrame

# fails with KeyError
sdf = SubDataFrame({'a':[1,2], 'b':[3,4]})
sdf.to_hdf('test.h5', 'test')

with pd.HDFStore('test.h5') as store:
    store.put('test', sdf)
```

#### Problem description

``to_hdf()`` and ``HDFStore.put()`` fail for DataFrame subclasses.

This happens because in ``pandas/io/pytables.py`` in ``_create_storer`` line 1578 (or thereabouts), the ``_TYPE_MAP`` is accessed by ``type()``, whereas upon ``_create_storer`` entry, the check is done using ``isinstance()``. 

#### Expected Output

The check upon entry of ``_create_storer`` should at least be consistent with the way ``_TYPE_MAP`` is accessed. So if the choice is not to support writing DataFrame subclasses to HDF, instead of a ``KeyError``, a ``TypeError("value must be None, Series, or DataFrame")`` should be raised.

But ideally, storing subclasses of DataFrame to HDF should be supported.

#### Output of ``pd.show_versions()``
<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.10.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.14.111-1.el7.centos.x86_64
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.0.3
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 46.1.3.post20200325
Cython           : None
pytest           : 5.4.1
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : 2.8.4 (dt dec pq3 ext lo64)
jinja2           : 2.10.3
IPython          : 7.10.2
pandas_datareader: None
bs4              : 4.8.2
bottleneck       : None
fastparquet      : 0.3.2
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.1
numexpr          : 2.7.1
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : 5.4.1
pyxlsb           : None
s3fs             : 0.4.0
scipy            : 1.2.1
sqlalchemy       : 1.3.11
tables           : 3.6.1
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : 0.46.0

</details>

Hints from issue discussion:
this is possible but would need a community PR
to support

Well, I'd be happy to change line 1528 of ``_create_storer`` from
``if value is not None and not isinstance(value, (Series, DataFrame)):`` 
to
``if value is not None and type(value) not in (Series, DataFrame):``
to correctly catch that subclasses of ``Series`` or ``DataFrame`` are currently not supported (this is why I initially labeled it a bug -- so maybe this ticket could be split in two then, one for this correction, one for implementing the actual support?).


I have run into a similar error by just appending pandas chunk dfs to an HDF5 file. 
I have confirmed that the first chunk dtypes and the following chunk dtypes are exactly the same for all the columns. BTW, I have 130 columns and all of them are converted to object dtype before writing to hdf5 file. 
Still, I got this error message:

> `Traceback (most recent call last):
>   File "/home/yangyxt/anaconda3/lib/python3.6/site-packages/pandas/io/pytables.py", line 3881, in create_axes
>     b, b_items = by_items.pop(items)
> KeyError: ('1000g2015aug_all', 'AAChange.refGene', 'Alt', 'Alt.1', 'CADD_phred', 'CADD_raw', 'CADD_raw_rankscore', 'CLNALLELEID', 'CLNDISDB', 'CLNDN', 'CLNREVSTAT', 'CLNSIG', 'Chr', 'DANN_rankscore', 'DANN_score', 'Eigen-PC-raw', 'Eigen-r
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "/paedwy/disk1/yangyxt/ngs_scripts/Reformatting_hg19_multianno_txt.py", line 243, in reformat
>     if len(new_chunk) > 0: new_chunk.to_hdf(tmp_hdf5_path, key='table', format='table', append=True)
>   File "/home/yangyxt/anaconda3/lib/python3.6/site-packages/pandas/core/generic.py", line 2530, in to_hdf
>     pytables.to_hdf(path_or_buf, key, self, **kwargs)
>   File "/home/yangyxt/anaconda3/lib/python3.6/site-packages/pandas/io/pytables.py", line 278, in to_hdf
>     f(store)
>   File "/home/yangyxt/anaconda3/lib/python3.6/site-packages/pandas/io/pytables.py", line 269, in <lambda>
>     f = lambda store: store.append(key, value, **kwargs)
>   File "/home/yangyxt/anaconda3/lib/python3.6/site-packages/pandas/io/pytables.py", line 1059, in append
>     self._write_to_group(key, value, append=append, dropna=dropna, **kwargs)
>   File "/home/yangyxt/anaconda3/lib/python3.6/site-packages/pandas/io/pytables.py", line 1525, in _write_to_group
>     s.write(obj=value, append=append, complib=complib, **kwargs)
>   File "/home/yangyxt/anaconda3/lib/python3.6/site-packages/pandas/io/pytables.py", line 4615, in write
>     return super().write(obj=obj, data_columns=data_columns, **kwargs)
>   File "/home/yangyxt/anaconda3/lib/python3.6/site-packages/pandas/io/pytables.py", line 4194, in write
>     axes=axes, obj=obj, validate=append, min_itemsize=min_itemsize, **kwargs
>   File "/home/yangyxt/anaconda3/lib/python3.6/site-packages/pandas/io/pytables.py", line 3888, in create_axes
>     items=(",".join(pprint_thing(item) for item in items))
> ValueError: cannot match existing table structure for [1000g2015aug_all,AAChange.refGene,Alt,Alt.1,CADD_phred,CADD_raw,CADD_raw_rankscore,CLNALLELEID,CLNDISDB,CLNDN,CLNREVSTAT,CLNSIG,Chr,DANN_rankscore,DANN_score,Eigen-PC-raw,Eigen-raw,`

I didn't find any solutions in the issues. This issue post is the most relevant one. Pls help debug it. Thanks!
That doesn't look like it's the same issue. This thread is specifically about storing subclasses of DataFrame to HDF.

Commit message: TST: use pandas testing tools

Patch to review:
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
index d7ee4acc2e670..8f023d9e9fe87 100644
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -1646,8 +1646,10 @@ def error(t):
                         "nor a value are passed"
                     )
             else:
-                _TYPE_MAP = {Series: "series", DataFrame: "frame"}
-                pt = _TYPE_MAP[type(value)]
+                if isinstance(value, Series):
+                    pt = "series"
+                else:
+                    pt = "frame"
 
                 # we are actually a table
                 if format == "table":
diff --git a/pandas/tests/io/pytables/test_store.py b/pandas/tests/io/pytables/test_store.py
index c20955227d05e..ef254657ff354 100644
--- a/pandas/tests/io/pytables/test_store.py
+++ b/pandas/tests/io/pytables/test_store.py
@@ -4888,6 +4888,49 @@ def test_unsuppored_hdf_file_error(self, datapath):
         with pytest.raises(ValueError, match=message):
             pd.read_hdf(data_path)
 
+    def test_supported_for_subclasses_dataframe(self):
+        class SubDataFrame(DataFrame):
+            @property
+            def _constructor(self):
+                return SubDataFrame
+
+        data = {"a": [1, 2], "b": [3, 4]}
+        sdf = SubDataFrame(data, dtype=np.intp)
+
+        expected = np.array([[1, 3], [2, 4]], dtype=np.intp)
+
+        with ensure_clean_path("temp.h5") as path:
+            sdf.to_hdf(path, "df")
+            result = read_hdf(path, "df").values
+            tm.assert_numpy_array_equal(result, expected)
+
+        with ensure_clean_path("temp.h5") as path:
+            with HDFStore(path) as store:
+                store.put("df", sdf)
+            result = read_hdf(path, "df").values
+            tm.assert_numpy_array_equal(result, expected)
+
+    def test_supported_for_subclasses_series(self):
+        class SubSeries(Series):
+            @property
+            def _constructor(self):
+                return SubSeries
+
+        sser = SubSeries([1, 2, 3], dtype=np.intp)
+
+        expected = np.array([1, 2, 3], dtype=np.intp)
+
+        with ensure_clean_path("temp.h5") as path:
+            sser.to_hdf(path, "ser")
+            result = read_hdf(path, "ser").values
+            tm.assert_numpy_array_equal(result, expected)
+
+        with ensure_clean_path("temp.h5") as path:
+            with HDFStore(path) as store:
+                store.put("ser", sser)
+            result = read_hdf(path, "ser").values
+            tm.assert_numpy_array_equal(result, expected)
+
 
 @pytest.mark.parametrize("bad_version", [(1, 2), (1,), [], "12", "123"])
 def test_maybe_adjust_name_bad_version_raises(bad_version):


Code review comments:
diff_hunk: @@ -4888,6 +4888,49 @@ def test_unsuppored_hdf_file_error(self, datapath):
         with pytest.raises(ValueError, match=message):
             pd.read_hdf(data_path)
 
+    def test_supported_for_subclasses_dataframe(self):
+        class SubDataFrame(DataFrame):
+            @property
+            def _constructor(self):
+                return SubDataFrame
+
+        data = {"a": [1, 2], "b": [3, 4]}
+        sdf = SubDataFrame(data, dtype=np.intp)
+
+        expected = np.array([[1, 3], [2, 4]], dtype=np.intp)
+
+        with ensure_clean_path("temp.h5") as path:
+            sdf.to_hdf(path, "df")
+            result = read_hdf(path, "df").values
+            tm.assert_numpy_array_equal(result, expected)
+
+        with ensure_clean_path("temp.h5") as path:
+            with HDFStore(path) as store:
+                store.put("df", sdf)
+            result = read_hdf(path, "df").values
+            tm.assert_numpy_array_equal(result, expected)
+
+    def test_supported_for_subclasses_series(self):
+        class SubSeries(Series):
+            @property
+            def _constructor(self):
+                return SubSeries
+
+        sser = SubSeries([1, 2, 3], dtype=np.intp)
+
+        expected = np.array([1, 2, 3], dtype=np.intp)
+
+        with ensure_clean_path("temp.h5") as path:
+            sser.to_hdf(path, "ser")
+            result = read_hdf(path, "ser").values
+            tm.assert_numpy_array_equal(result, expected)
+
+        with ensure_clean_path("temp.h5") as path:
+            with HDFStore(path) as store:
+                store.put("ser", sser)
+            result = read_hdf(path, "ser").values
path: pandas/tests/io/pytables/test_store.py
text: @user1:
i remember something about preferring `to_numpy` over `values` but obv both work