Repository: statsmodels/statsmodels

Issue:
Avoid unicode names when working with np.recarray in py2
Fixes all but one of the test errors discussed in #3658


Discussion:

[![Coverage Status](https://coveralls.io/builds/11813144/badge)](https://coveralls.io/builds/11813144)

Coverage decreased (-0.004%) to 90.431% when pulling **3d43ef14374a2181b5df6c22dec2d9200d58669a on jbrockmendel:fix_3658** into **ac254645819494c0a5d3d57785d97c3e0a9ebfa3 on statsmodels:master**.

coveralls is reporting coverage decrease for python 3.6.  This is to be expected, as the relevant code added is conditioned on `is_py2`
recarrays support should DIAF.
> recarrays support should DIAF.

That's why this is a "fix the tests and move on with our lives" PR instead of a "carefully address the underlying issue" PR.

[![Coverage Status](https://coveralls.io/builds/11816499/badge)](https://coveralls.io/builds/11816499)

Coverage decreased (-0.005%) to 90.431% when pulling **af5041ebde5a12d123dd2a9852a55a84388faa95 on jbrockmendel:fix_3658** into **ac254645819494c0a5d3d57785d97c3e0a9ebfa3 on statsmodels:master**.

I started to look a bit at the details of the related issues (including numpy and pandas)

I think we should try to fix recarrays one last time, and then deprecate the support of them to avoid "wasting" more time with numpy and pandas compat issues for structured dtypes and recarrays.
If possible we need to fix it for Debian #3841 and users who still are in python 2 land.

OK.  What do you have in mind for "try to fix"?
You mentioned in the description that there is still one error that isn't fixed. I was planning to try out this PR and understand a bit better what is going on, and see whether we want bytes or string or a compat function in python 2 in the way or a similar way as you have in this PR.

Also, this PR needs a rebase on master which has as side effect that the tests will be run by pytest instead of nosetests.

(Because I had forgotten that I avoided the failing tests, I didn't realize before that this is high priority for 0.9, especially if numpy or pandas don't fix it for us.)

(The cheapest solution would be to skip the failing unit tests on python 2, which would be the last resort if not everything is fixed.)
Also, to see the results of this PR, we need to revert the two PRs that pin the pandas version on python 2  on travis and appveyor to one that "works".
OK.  I've stretched myself a bit too thin so will not be pro-active for a bit.  Please let me know if there's anything specific I can do to help.
appveyor has upgraded to pandas 0.20.3 and the unicode errors are back
https://ci.appveyor.com/project/josef-pkt/statsmodels/build/1.0.2294/job/32pr6h5xh4jexanh


I've forgotten: is this one of the problems expected to resolve itself eventually?  Or one of the ones where we need to stop supporting X?
I was planning to look into it this week and merge (I didn't get much done today)

My strategy:
- update pandas on travis and appveyor to see all errors again
- rebase this
- whatever tests keep failing will get a skiptest on python 2.7 (better unicode handling in statsmodels requires python 3)

It didn't look like it will go away by changes in pandas, because the problematic changes there were introduced to solve other problems. 
The problems for statsmodels will go away at some point in the future when we drop recarray and/or python 2.7 support.


Resolving patch:
diff --git a/statsmodels/compat/numpy.py b/statsmodels/compat/numpy.py
--- a/statsmodels/compat/numpy.py
+++ b/statsmodels/compat/numpy.py
@@ -40,11 +40,15 @@
 """
 
 from __future__ import absolute_import
-from .scipy import NumpyVersion
 import numpy as np
 
+from .python import PY3
+from .scipy import NumpyVersion
+
+
 np_matrix_rank = np.linalg.matrix_rank
 
+
 if NumpyVersion(np.__version__) >= '1.9.0':
     np_new_unique = np.unique
 else:
@@ -179,6 +183,19 @@ def recarray_select(recarray, fields):
     from pandas import DataFrame
     fields = [fields] if not isinstance(fields, (tuple, list)) else fields
     if len(fields) == len(recarray.dtype):
-        return recarray
-    recarray = DataFrame.from_records(recarray)
-    return recarray[fields].to_records(index=False)
+        selection = recarray
+    else:
+        recarray = DataFrame.from_records(recarray)
+        selection = recarray[fields].to_records(index=False)
+
+    _bytelike_dtype_names(selection)
+    return selection
+
+
+def _bytelike_dtype_names(arr):
+    # See # 3658
+    if not PY3:
+        dtype = arr.dtype
+        names = dtype.names
+        names = [bytes(name) if isinstance(name, unicode) else name for name in names]
+        dtype.names = names
diff --git a/statsmodels/tsa/tsatools.py b/statsmodels/tsa/tsatools.py
--- a/statsmodels/tsa/tsatools.py
+++ b/statsmodels/tsa/tsatools.py
@@ -1,4 +1,5 @@
-from statsmodels.compat.python import range, lrange, lzip, long
+
+from statsmodels.compat.python import range, lrange, lzip, long, PY3
 from statsmodels.compat.numpy import recarray_select
 
 import numpy as np
@@ -117,7 +118,19 @@ def safe_is_const(s):
         x = x.to_records(index=False, convert_datetime64=False)
         new_descr = x.dtype.descr
         extra_col = len(new_descr) - len(descr)
-        descr = new_descr[:extra_col] + descr if prepend else descr + new_descr[-extra_col:]
+        if prepend:
+            descr = new_descr[:extra_col] + descr
+        else:
+            descr = descr + new_descr[-extra_col:]
+
+        if not PY3:
+            # See 3658
+            names = [entry[0] for entry in descr]
+            dtypes = [entry[1] for entry in descr]
+            names = [bytes(name) for name in names]
+            # Fail loudly if there is a non-ascii name
+            descr = list(zip(names, dtypes))
+
         x = x.astype(np.dtype(descr))
 
     return x
@@ -172,10 +185,18 @@ def add_lag(x, col=None, lags=1, drop=False, insert=True):
             col = names[0]
         if isinstance(col, (int, long)):
             col = x.dtype.names[col]
+        if not PY3:
+            # TODO: Get rid of this kludge.  See GH # 3658
+            names = [bytes(name) if isinstance(name, unicode) else name for name in names]
+            # Fail loudly if there is a non-ascii name.
+            x.dtype.names = names
+            if isinstance(col, unicode):
+                col = bytes(col)
+
         contemp = x[col]
 
         # make names for lags
-        tmp_names = [col + '_'+'L(%i)' % i for i in range(1,lags+1)]
+        tmp_names = [col + '_'+'L(%i)' % i for i in range(1, lags+1)]
         ndlags = lagmat(contemp, maxlag=lags, trim='Both')
 
         # get index for return
@@ -202,8 +223,9 @@ def add_lag(x, col=None, lags=1, drop=False, insert=True):
         if first_names: # only do this if x isn't "empty"
             # Workaround to avoid NumPy FutureWarning
             _x = recarray_select(x, first_names)
-            first_arr = nprf.append_fields(_x[lags:],tmp_names, ndlags.T,
+            first_arr = nprf.append_fields(_x[lags:], tmp_names, ndlags.T,
                                            usemask=False)
+
         else:
             first_arr = np.zeros(len(x)-lags, dtype=lzip(tmp_names,
                 (x[col].dtype,)*lags))


Test patch:
diff --git a/statsmodels/tsa/vector_ar/tests/test_var.py b/statsmodels/tsa/vector_ar/tests/test_var.py
--- a/statsmodels/tsa/vector_ar/tests/test_var.py
+++ b/statsmodels/tsa/vector_ar/tests/test_var.py
@@ -448,6 +448,7 @@ def test_reorder(self):
     def test_pickle(self):
         fh = BytesIO()
         #test wrapped results load save pickle
+        del self.res.model.data.orig_endog
         self.res.save(fh)
         fh.seek(0,0)
         res_unpickled = self.res.__class__.load(fh)