Repository: pandas-dev/pandas

Issue:
BUG: Categorical data fails to load from hdf when all columns are NaN
#### Code Sample, a copy-pastable example if possible

```python
import numpy as np
import pandas as pd

df = pd.DataFrame({'a': ['a', 'b', 'c', np.nan], 'b': [np.nan, np.nan, np.nan, np.nan]})
df['a'] = df.a.astype('category')
df['b'] = df.b.astype('category')

df.to_hdf('foo.h5', 'bar', format='table')
pd.read_hdf('foo.h5', 'bar')

```
#### Problem description

While storing an hdf file with categorical data containing `np.nan`s works fine, loading the file back in to a DataFrame raises an exception.

```
  File "~/env/lib/python2.7/site-packages/pandas/io/pytables.py", line 372, in read_hdf
    return store.select(key, auto_close=auto_close, **kwargs)
  File "~/env/lib/python2.7/site-packages/pandas/io/pytables.py", line 742, in select
    return it.get_result()
  File "~/env/lib/python2.7/site-packages/pandas/io/pytables.py", line 1449, in get_result
    results = self.func(self.start, self.stop, where)
  File "~/env/lib/python2.7/site-packages/pandas/io/pytables.py", line 735, in func
    columns=columns, **kwargs)
  File "~/env/lib/python2.7/site-packages/pandas/io/pytables.py", line 4124, in read
    if not self.read_axes(where=where, **kwargs):
  File "~/env/lib/python2.7/site-packages/pandas/io/pytables.py", line 3329, in read_axes
    a.convert(values, nan_rep=self.nan_rep, encoding=self.encoding)
  File "~/env/lib/python2.7/site-packages/pandas/io/pytables.py", line 2133, in convert
    if mask.any():
AttributeError: 'bool' object has no attribute 'any'
```

This exception is related to storing a column that contains `np.nan` values only (column `a` stores and loads fine on its own).

The problem could already be in the way the `metadata` for column `b` (the `np.nan`-only column) is stored when calling `df.to_hdf()` as the `metadata` is `None` when loading. The relevant code for the `pd.read_hdf()` in `DataCol.convert`:

```python
elif meta == u('category'):

                # we have a categorical
                categories = self.metadata
                codes = self.data.ravel()

                # if we have stored a NaN in the categories
                # then strip it; in theory we could have BOTH
                # -1s in the codes and nulls :<
                mask = isnull(categories)
                if mask.any():
                    categories = categories[~mask]
                    codes[codes != -1] -= mask.astype(int).cumsum().values
```
has metadata set to None (`self.metadata == categories == None`) which in turn makes mask (=`isnull(None)`) a scalar value (`False`) and thus `.any()` fails.

#### Expected Output

No exception; dataframe loads as it would without categorical data.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.13.final.0
python-bits: 64
OS: Darwin
OS-release: 16.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_AU.UTF-8
LOCALE: None.None

pandas: 0.20.3
pytest: 3.1.3
pip: 9.0.1
setuptools: 35.0.2
Cython: 0.25.2
numpy: 1.13.1
scipy: 0.16.0
xarray: None
IPython: 5.4.1
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.0
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 1.4.3
openpyxl: 1.8.6
xlrd: 0.9.3
xlwt: None
xlsxwriter: 0.9.6
lxml: 3.7.3
bs4: None
html5lib: 0.999999999
sqlalchemy: 1.1.9
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
jinja2: 2.9.5
s3fs: 0.1.2
pandas_gbq: None
pandas_datareader: None

</details>

BUG: Categorical data fails to load from hdf when all columns are NaN
#### Code Sample, a copy-pastable example if possible

```python
import numpy as np
import pandas as pd

df = pd.DataFrame({'a': ['a', 'b', 'c', np.nan], 'b': [np.nan, np.nan, np.nan, np.nan]})
df['a'] = df.a.astype('category')
df['b'] = df.b.astype('category')

df.to_hdf('foo.h5', 'bar', format='table')
pd.read_hdf('foo.h5', 'bar')

```
#### Problem description

While storing an hdf file with categorical data containing `np.nan`s works fine, loading the file back in to a DataFrame raises an exception.

```
  File "~/env/lib/python2.7/site-packages/pandas/io/pytables.py", line 372, in read_hdf
    return store.select(key, auto_close=auto_close, **kwargs)
  File "~/env/lib/python2.7/site-packages/pandas/io/pytables.py", line 742, in select
    return it.get_result()
  File "~/env/lib/python2.7/site-packages/pandas/io/pytables.py", line 1449, in get_result
    results = self.func(self.start, self.stop, where)
  File "~/env/lib/python2.7/site-packages/pandas/io/pytables.py", line 735, in func
    columns=columns, **kwargs)
  File "~/env/lib/python2.7/site-packages/pandas/io/pytables.py", line 4124, in read
    if not self.read_axes(where=where, **kwargs):
  File "~/env/lib/python2.7/site-packages/pandas/io/pytables.py", line 3329, in read_axes
    a.convert(values, nan_rep=self.nan_rep, encoding=self.encoding)
  File "~/env/lib/python2.7/site-packages/pandas/io/pytables.py", line 2133, in convert
    if mask.any():
AttributeError: 'bool' object has no attribute 'any'
```

This exception is related to storing a column that contains `np.nan` values only (column `a` stores and loads fine on its own).

The problem could already be in the way the `metadata` for column `b` (the `np.nan`-only column) is stored when calling `df.to_hdf()` as the `metadata` is `None` when loading. The relevant code for the `pd.read_hdf()` in `DataCol.convert`:

```python
elif meta == u('category'):

                # we have a categorical
                categories = self.metadata
                codes = self.data.ravel()

                # if we have stored a NaN in the categories
                # then strip it; in theory we could have BOTH
                # -1s in the codes and nulls :<
                mask = isnull(categories)
                if mask.any():
                    categories = categories[~mask]
                    codes[codes != -1] -= mask.astype(int).cumsum().values
```
has metadata set to None (`self.metadata == categories == None`) which in turn makes mask (=`isnull(None)`) a scalar value (`False`) and thus `.any()` fails.

#### Expected Output

No exception; dataframe loads as it would without categorical data.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.13.final.0
python-bits: 64
OS: Darwin
OS-release: 16.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_AU.UTF-8
LOCALE: None.None

pandas: 0.20.3
pytest: 3.1.3
pip: 9.0.1
setuptools: 35.0.2
Cython: 0.25.2
numpy: 1.13.1
scipy: 0.16.0
xarray: None
IPython: 5.4.1
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.0
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 1.4.3
openpyxl: 1.8.6
xlrd: 0.9.3
xlwt: None
xlsxwriter: 0.9.6
lxml: 3.7.3
bs4: None
html5lib: 0.999999999
sqlalchemy: 1.1.9
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
jinja2: 2.9.5
s3fs: 0.1.2
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
Has anyone had a chance to look at this? Can it be reproduced? If so, is there a workaround or some suggestions?
looks like a bug. The issue is the categories are an empty array; when this is stored, pytables cannot write a zero-len array, so on readback the categories are ``None``. 

```
In [2]: pd.read_hdf('foo.h5', 'bar')
Out[2]: 
     a    b
0    a  NaN
1    b  NaN
2    c  NaN
3  NaN  NaN

In [3]: pd.read_hdf('foo.h5', 'bar').dtypes
Out[3]: 
a    category
b    category
dtype: object
```

patch
```
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
index 2a66aea..2ae3765 100644
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -2137,10 +2137,13 @@ class DataCol(IndexCol):
                 # if we have stored a NaN in the categories
                 # then strip it; in theory we could have BOTH
                 # -1s in the codes and nulls :<
-                mask = isna(categories)
-                if mask.any():
-                    categories = categories[~mask]
-                    codes[codes != -1] -= mask.astype(int).cumsum().values
+                if categories is None:
+                    categories = []
+                else:
+                    mask = isna(categories)
+                    if mask.any():
+                        categories = categories[~mask]
+                        codes[codes != -1] -= mask.astype(int).cumsum().values
 
                 self.data = Categorical.from_codes(codes,
                                                    categories=categories,
```

if you can submit a PR would be great.
Has anyone had a chance to look at this? Can it be reproduced? If so, is there a workaround or some suggestions?
looks like a bug. The issue is the categories are an empty array; when this is stored, pytables cannot write a zero-len array, so on readback the categories are ``None``. 

```
In [2]: pd.read_hdf('foo.h5', 'bar')
Out[2]: 
     a    b
0    a  NaN
1    b  NaN
2    c  NaN
3  NaN  NaN

In [3]: pd.read_hdf('foo.h5', 'bar').dtypes
Out[3]: 
a    category
b    category
dtype: object
```

patch
```
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
index 2a66aea..2ae3765 100644
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -2137,10 +2137,13 @@ class DataCol(IndexCol):
                 # if we have stored a NaN in the categories
                 # then strip it; in theory we could have BOTH
                 # -1s in the codes and nulls :<
-                mask = isna(categories)
-                if mask.any():
-                    categories = categories[~mask]
-                    codes[codes != -1] -= mask.astype(int).cumsum().values
+                if categories is None:
+                    categories = []
+                else:
+                    mask = isna(categories)
+                    if mask.any():
+                        categories = categories[~mask]
+                        codes[codes != -1] -= mask.astype(int).cumsum().values
 
                 self.data = Categorical.from_codes(codes,
                                                    categories=categories,
```

if you can submit a PR would be great.

Resolving patch:
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -2137,10 +2137,17 @@ def convert(self, values, nan_rep, encoding):
                 # if we have stored a NaN in the categories
                 # then strip it; in theory we could have BOTH
                 # -1s in the codes and nulls :<
-                mask = isna(categories)
-                if mask.any():
-                    categories = categories[~mask]
-                    codes[codes != -1] -= mask.astype(int).cumsum().values
+                if categories is None:
+                    # Handle case of NaN-only categorical columns in which case
+                    # the categories are an empty array; when this is stored,
+                    # pytables cannot write a zero-len array, so on readback
+                    # the categories would be None and `read_hdf()` would fail.
+                    categories = Index([], dtype=np.float64)
+                else:
+                    mask = isna(categories)
+                    if mask.any():
+                        categories = categories[~mask]
+                        codes[codes != -1] -= mask.astype(int).cumsum().values
 
                 self.data = Categorical.from_codes(codes,
                                                    categories=categories,


Test patch:
diff --git a/pandas/tests/io/test_pytables.py b/pandas/tests/io/test_pytables.py
--- a/pandas/tests/io/test_pytables.py
+++ b/pandas/tests/io/test_pytables.py
@@ -4928,6 +4928,25 @@ def test_categorical_conversion(self):
             result = read_hdf(path, 'df', where='obsids=B')
             tm.assert_frame_equal(result, expected)
 
+    def test_categorical_nan_only_columns(self):
+        # GH18413
+        # Check that read_hdf with categorical columns with NaN-only values can
+        # be read back.
+        df = pd.DataFrame({
+            'a': ['a', 'b', 'c', np.nan],
+            'b': [np.nan, np.nan, np.nan, np.nan],
+            'c': [1, 2, 3, 4],
+            'd': pd.Series([None] * 4, dtype=object)
+        })
+        df['a'] = df.a.astype('category')
+        df['b'] = df.b.astype('category')
+        df['d'] = df.b.astype('category')
+        expected = df
+        with ensure_clean_path(self.path) as path:
+            df.to_hdf(path, 'df', format='table', data_columns=True)
+            result = read_hdf(path, 'df')
+            tm.assert_frame_equal(result, expected)
+
     def test_duplicate_column_name(self):
         df = DataFrame(columns=["a", "a"], data=[[0, 0]])