Repository: pandas-dev/pandas

Issue:
Irregular errors when reading certain categorical strings from hdf
It seems that there is something bad happening when we use certain strings with special characters AND the empty string with categoricals:

``` python
    # -*- coding: latin-1 -*-
    import pandas
    import os

    examples = [
            pandas.Series(['EÉ, 17', '', 'a', 'b', 'c'], dtype='category'),
            pandas.Series(['EÉ, 17', 'a', 'b', 'c'], dtype='category'),
            pandas.Series(['', 'a', 'b', 'c'], dtype='category'),
            pandas.Series(['EE, 17', '', 'a', 'b', 'c'], dtype='category'),
            pandas.Series(['øü', 'a', 'b', 'c'], dtype='category'),
            pandas.Series(['Aøü', '', 'a', 'b', 'c'], dtype='category'),
            pandas.Series(['EÉ, 17', 'øü', 'a', 'b', 'c'], dtype='category')
            ]

    def test_hdf(s):
        f = 'testhdf.h5'
        if os.path.exists(f):
            os.remove(f)
        s.to_hdf(f, 'data', format='table')
        return pandas.read_hdf(f, 'data')

    for i, s in enumerate(examples):
        flag = True
        e = ''
        try:
            test_hdf(s)
        except Exception as ex:
            e = ex
            flag = False
        print('%d: %s\t%s\t%s' % (i, 'pass' if flag else 'fail', s.tolist(), e))
```

Results in:

```
    0: fail ['EÉ, 17', '', 'a', 'b', 'c']   Categorical categories must be unique
    1: pass ['EÉ, 17', 'a', 'b', 'c']
    2: pass ['', 'a', 'b', 'c']
    3: pass ['EE, 17', '', 'a', 'b', 'c']
    4: pass ['øü', 'a', 'b', 'c']
    5: fail ['Aøü', '', 'a', 'b', 'c']  Categorical categories must be unique
    6: pass ['EÉ, 17', 'øü', 'a', 'b', 'c']
```

Not sure if I am using this incorrectly or if this is actually a corner case.



Discussion:
On Linux/Python 2.7/pandas: 0.16.2-9-g7636c2c (master) I get

```
0: pass ['E\xc3\x89, 17', '', 'a', 'b', 'c']    
1: pass ['E\xc3\x89, 17', 'a', 'b', 'c']    
2: pass ['', 'a', 'b', 'c'] 
3: pass ['EE, 17', '', 'a', 'b', 'c']   
4: pass ['\xc3\xb8\xc3\xbc', 'a', 'b', 'c'] 
5: pass ['A\xc3\xb8\xc3\xbc', '', 'a', 'b', 'c']    
6: pass ['E\xc3\x89, 17', '\xc3\xb8\xc3\xbc', 'a', 'b', 'c']    
```

What does `pd.show_versions()` output?

@bashtage Sorry I forgot to mention any version platform information. I am on OSX and am getting the errors only in python3 (python2 behaves differently). I was on master but turned back to the latest release.

Notice the "-_coding_-" EDIT in the example I gave above. It changes the look of the output slightly.

Here is what I just tested now:

```
    $ python --version
    Python 3.4.3 :: Anaconda 2.2.0 (x86_64)

    $ python -c 'import pandas; pandas.show_versions()'

    INSTALLED VERSIONS
    ------------------
    commit: None
    python: 3.4.3.final.0
    python-bits: 64
    OS: Darwin
    OS-release: 14.3.0
    machine: x86_64
    processor: i386
    byteorder: little
    LC_ALL: None
    LANG: en_CA.UTF-8

    pandas: 0.16.2
    nose: None
    Cython: 0.22
    numpy: 1.9.2
    scipy: 0.15.1
    statsmodels: None
    IPython: 3.1.0
    sphinx: 1.2.3
    patsy: 0.3.0
    dateutil: 2.4.2
    pytz: 2015.4
    bottleneck: None
    tables: 3.1.1
    numexpr: 2.3.1
    matplotlib: 1.4.3
    openpyxl: 1.8.5
    xlrd: 0.9.3
    xlwt: None
    xlsxwriter: 0.6.7
    lxml: 3.4.2
    bs4: 4.3.2
    html5lib: 0.999
    httplib2: None
    apiclient: None
    sqlalchemy: 0.9.9
    pymysql: None
    psycopg2: None

    $ python debug.py
    0: fail ['EÃ\x89, 17', '', 'a', 'b', 'c']  Categorical categories must be unique
    1: pass ['EÃ\x89, 17', 'a', 'b', 'c']
    2: pass ['', 'a', 'b', 'c']
    3: pass ['EE, 17', '', 'a', 'b', 'c']
    4: pass ['Ã¸Ã¼', 'a', 'b', 'c']
    5: fail ['AÃ¸Ã¼', '', 'a', 'b', 'c']    Categorical categories must be unique
    6: fail ['EÃ\x89, 17', 'Ã¸Ã¼', 'a', 'b', 'c']  'utf-8' codec can't decode byte 0xc2 in position 6: unexpected end of data

    $ python --version
    Python 2.7.10 :: Continuum Analytics, Inc.

    $ python -c 'import pandas; pandas.show_versions()'

    INSTALLED VERSIONS
    ------------------
    commit: None
    python: 2.7.10.final.0
    python-bits: 64
    OS: Darwin
    OS-release: 14.3.0
    machine: x86_64
    processor: i386
    byteorder: little
    LC_ALL: None
    LANG: en_CA.UTF-8

    pandas: 0.16.2
    nose: 1.3.4
    Cython: 0.21.1
    numpy: 1.9.2
    scipy: 0.15.1
    statsmodels: 0.6.1
    IPython: 2.3.1
    sphinx: 1.2.3
    patsy: 0.3.0
    dateutil: 2.4.2
    pytz: 2015.4
    bottleneck: None
    tables: 3.1.1
    numexpr: 2.3.1
    matplotlib: 1.4.3
    openpyxl: 2.0.2
    xlrd: 0.9.3
    xlwt: None
    xlsxwriter: 0.6.4
    lxml: 3.4.1
    bs4: 4.3.2
    html5lib: None
    httplib2: 0.8
    apiclient: None
    sqlalchemy: 0.9.8
    pymysql: None
    psycopg2: None

    $ python debug.py
    0: pass ['E\xc3\x89, 17', '', 'a', 'b', 'c']
    1: pass ['E\xc3\x89, 17', 'a', 'b', 'c']
    2: pass ['', 'a', 'b', 'c']
    3: pass ['EE, 17', '', 'a', 'b', 'c']
    4: pass ['\xc3\xb8\xc3\xbc', 'a', 'b', 'c']
    5: pass ['A\xc3\xb8\xc3\xbc', '', 'a', 'b', 'c']
    6: pass ['E\xc3\x89, 17', '\xc3\xb8\xc3\xbc', 'a', 'b', 'c']
```

So this is a bug [here](https://github.com/pydata/pandas/blob/master/pandas/io/pytables.py#L3024).
The `Categoricals` are written as a separate table in a sub-node of the main table. The `encoding` and `nan_rep` are known on the table at this point, and should be passed thru. Since you already have a test you have the hardest part already done. pls submit a pull-request to fix!

@jreback Thanks. I think I can get to this in a week or so.

@cottrell gr8!


Resolving patch:
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -3039,7 +3039,8 @@ def write_metadata(self, key, values):
 
         """
         values = Series(values)
-        self.parent.put(self._get_metadata_path(key), values, format='table')
+        self.parent.put(self._get_metadata_path(key), values, format='table',
+                encoding=self.encoding, nan_rep=self.nan_rep)
 
     def read_metadata(self, key):
         """ return the meta data array for this key """
@@ -4389,11 +4390,23 @@ def _unconvert_index_legacy(data, kind, legacy=False, encoding=None):
 
 
 def _convert_string_array(data, encoding, itemsize=None):
+    """
+    we take a string-like that is object dtype and coerce to a fixed size string type
+
+    Parameters
+    ----------
+    data : a numpy array of object dtype
+    encoding : None or string-encoding
+    itemsize : integer, optional, defaults to the max length of the strings
+
+    Returns
+    -------
+    data in a fixed-length string dtype, encoded to bytes if needed
+    """
 
     # encode if needed
     if encoding is not None and len(data):
-        f = np.vectorize(lambda x: x.encode(encoding), otypes=[np.object])
-        data = f(data)
+        data = Series(data.ravel()).str.encode(encoding).values.reshape(data.shape)
 
     # create the sized dtype
     if itemsize is None:
@@ -4403,7 +4416,20 @@ def _convert_string_array(data, encoding, itemsize=None):
     return data
 
 def _unconvert_string_array(data, nan_rep=None, encoding=None):
-    """ deserialize a string array, possibly decoding """
+    """
+    inverse of _convert_string_array
+
+    Parameters
+    ----------
+    data : fixed length string dtyped array
+    nan_rep : the storage repr of NaN, optional
+    encoding : the encoding of the data, optional
+
+    Returns
+    -------
+    an object array of the decoded data
+
+    """
     shape = data.shape
     data = np.asarray(data.ravel(), dtype=object)
 
@@ -4412,16 +4438,16 @@ def _unconvert_string_array(data, nan_rep=None, encoding=None):
     encoding = _ensure_encoding(encoding)
     if encoding is not None and len(data):
 
-        try:
-            itemsize = lib.max_len_string_array(com._ensure_object(data.ravel()))
-            if compat.PY3:
-                dtype = "U{0}".format(itemsize)
-            else:
-                dtype = "S{0}".format(itemsize)
+        itemsize = lib.max_len_string_array(com._ensure_object(data))
+        if compat.PY3:
+            dtype = "U{0}".format(itemsize)
+        else:
+            dtype = "S{0}".format(itemsize)
+
+        if isinstance(data[0], compat.binary_type):
+            data = Series(data).str.decode(encoding).values
+        else:
             data = data.astype(dtype, copy=False).astype(object, copy=False)
-        except (Exception) as e:
-            f = np.vectorize(lambda x: x.decode(encoding), otypes=[np.object])
-            data = f(data)
 
     if nan_rep is None:
         nan_rep = 'nan'


Test patch:
diff --git a/pandas/io/tests/test_pytables.py b/pandas/io/tests/test_pytables.py
--- a/pandas/io/tests/test_pytables.py
+++ b/pandas/io/tests/test_pytables.py
@@ -930,6 +930,51 @@ def test_encoding(self):
             result = store.select('df',Term('columns=A',encoding='ascii'))
             tm.assert_frame_equal(result,expected)
 
+    def test_latin_encoding(self):
+
+        if compat.PY2:
+            self.assertRaisesRegexp(TypeError, '\[unicode\] is not implemented as a table column')
+            return
+
+        values = [[b'E\xc9, 17', b'', b'a', b'b', b'c'],
+                  [b'E\xc9, 17', b'a', b'b', b'c'],
+                  [b'EE, 17', b'', b'a', b'b', b'c'],
+                  [b'E\xc9, 17', b'\xf8\xfc', b'a', b'b', b'c'],
+                  [b'', b'a', b'b', b'c'],
+                  [b'\xf8\xfc', b'a', b'b', b'c'],
+                  [b'A\xf8\xfc', b'', b'a', b'b', b'c'],
+                  [np.nan, b'', b'b', b'c'],
+                  [b'A\xf8\xfc', np.nan, b'', b'b', b'c']]
+
+        def _try_decode(x, encoding='latin-1'):
+            try:
+                return x.decode(encoding)
+            except AttributeError:
+                return x
+        # not sure how to remove latin-1 from code in python 2 and 3
+        values = [[_try_decode(x) for x in y] for y in values]
+
+        examples = []
+        for dtype in ['category', object]:
+            for val in values:
+                examples.append(pandas.Series(val, dtype=dtype))
+
+        def roundtrip(s, key='data', encoding='latin-1', nan_rep=''):
+            with ensure_clean_path(self.path) as store:
+                s.to_hdf(store, key, format='table', encoding=encoding,
+                        nan_rep=nan_rep)
+                retr = read_hdf(store, key)
+                s_nan = s.replace(nan_rep, np.nan)
+                assert_series_equal(s_nan, retr)
+
+        for s in examples:
+            roundtrip(s)
+
+        # fails:
+        # for x in examples:
+        #     roundtrip(s, nan_rep=b'\xf8\xfc')
+
+
     def test_append_some_nans(self):
 
         with ensure_clean_store(self.path) as store: