Repository: pandas-dev/pandas

Issue:
BUG: HDFStore TypeError: decoding Unicode is not supported
On latest master `0.12.0-315-ga99b3be`, when I try to select a table from an HDFStore, I get a Unicode Error

```
In [4]: df = store.select('monthly/data/m1998_01')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-67c3fabe2905> in <module>()
----> 1 df = store.select('monthly/data/m1998_01')

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/io/pytables.py in select(self, key, where, start, stop, columns, iterator, chunksize, auto_close, **kwargs)
    579         where = _ensure_term(where)
    580         s = self._create_storer(group)
--> 581         s.infer_axes()
    582 
    583         # what we are actually going to do for a chunk

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/io/pytables.py in infer_axes(self)
   2001         if s is None:
   2002             return False
-> 2003         self.get_attrs()
   2004         return True
   2005 

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/io/pytables.py in get_attrs(self)
   2740         t = self.table
   2741         self.index_axes = [a.infer(t)
-> 2742                                          for a in self.indexables if a.is_an_indexable]
   2743         self.values_axes = [a.infer(t)
   2744                                          for a in self.indexables if not a.is_an_indexable]

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/io/pytables.py in indexables(self)
   2793 
   2794             self._indexables.extend(
-> 2795                 [f(i, c) for i, c in enumerate(self.attrs.values_cols)])
   2796 
   2797         return self._indexables

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/io/pytables.py in f(i, c)
   2790                 if c in dc:
   2791                     klass = DataIndexableCol
-> 2792                 return klass.create_for_block(i=i, name=c, pos=base_pos + i, version=self.version)
   2793 
   2794             self._indexables.extend(

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/io/pytables.py in create_for_block(cls, i, name, cname, version, **kwargs)
   1537             pass
   1538 
-> 1539         return cls(name=name, cname=cname, **kwargs)
   1540 
   1541     def __init__(self, values=None, kind=None, typ=None,

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/io/pytables.py in __init__(self, values, kind, typ, cname, data, block, **kwargs)
   1544             values=values, kind=kind, typ=typ, cname=cname, **kwargs)
   1545         self.dtype = None
-> 1546         self.dtype_attr = u("%s_dtype" % self.name)
   1547         self.set_data(data)
   1548 

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/compat/__init__.py in u(s)
    189 
    190     def u(s):
--> 191         return unicode(s, "unicode_escape")
    192 
    193 

TypeError: decoding Unicode is not supported
```

I'm still trying to whittle down where this behavior changed, but at least as of `a99b3be`, it worked.  I'm going to go through commits changing HDFStore, but maybe someone knows exactly where this might be from?

I'll also try to come up with a minimal example that reproduces.  Simple ones aren't doing it which is a good thing I guess.



Discussion:
what is in this file? can you post it?

are you using pytables 3.0.0?

and did you create it with a different version?

I've selected out a column and uploaded it here: https://www.dropbox.com/s/mjaomv981ue4cu6/bug.h5 (98 MB)

I'm on tables 3.0

Works fine as of `a2d8b3d`, so its pretty recent.

well..just merged in eval, (which changed pytables parsing)...could be a bug....

Seems to be here: error as of `50502fb CLN/DOC: remove __getitem__ from docs`, good as of `ba9c143 Merge pull request #4852 from jreback/seq2`.

I'll see if I can trim down that example.  98 MB isn't exactly minimal :)

hmmm....its a trivial fix....hold a sec


Resolving patch:
diff --git a/pandas/compat/__init__.py b/pandas/compat/__init__.py
--- a/pandas/compat/__init__.py
+++ b/pandas/compat/__init__.py
@@ -180,6 +180,8 @@ class to receive bound method
 
     def u(s):
         return s
+    def u_safe(s):
+        return s
 else:
     string_types = basestring,
     integer_types = (int, long)
@@ -190,6 +192,12 @@ def u(s):
     def u(s):
         return unicode(s, "unicode_escape")
 
+    def u_safe(s):
+        try:
+            return unicode(s, "unicode_escape")
+        except:
+            return s
+
 
 string_and_binary_types = string_types + (binary_type,)
 
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -29,7 +29,7 @@
 import pandas.core.common as com
 from pandas.tools.merge import concat
 from pandas import compat
-from pandas.compat import u, PY3, range, lrange
+from pandas.compat import u_safe as u, PY3, range, lrange
 from pandas.io.common import PerformanceWarning
 from pandas.core.config import get_option
 from pandas.computation.pytables import Expr, maybe_expression


Test patch: