Repository: statsmodels/statsmodels

Issue:
datasets.get_rdataset   string decode error on python 3
trying to run Skipper's example from issue #805 on python 3.3
fails with a UnicodeDecodeError

I guess we don't have a test for this.
I ran several examples on python 3 before the release, but none that used get_rdataset

```
import statsmodels.api as sm
from statsmodels.formula.api import ols

dta = sm.datasets.get_rdataset("Guerry", "HistData", cache=True)
# .... snip rest

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-15-6ce721e61c62> in <module>()
      3 from statsmodels.formula.api import ols
      4 
----> 5 dta = sm.datasets.get_rdataset("Guerry", "HistData", cache=True)
      6 df = dta.data[['Lottery', 'Literacy', 'Wealth', 'Region']].dropna()
      7 

E:\Josef\programs\WinPython-64bit-3.3.2.2\python-3.3.2.amd64\lib\site-packages\statsmodels\datasets\utils.py in get_rdataset(dataname, package, cache)
    250 
    251     title = _get_dataset_meta(dataname, package, cache)
--> 252     doc, _ = _get_data(docs_base_url, dataname, cache, "rst")
    253 
    254     return Dataset(data=data, __doc__=doc.read(), package=package, title=title,

E:\Josef\programs\WinPython-64bit-3.3.2.2\python-3.3.2.amd64\lib\site-packages\statsmodels\datasets\utils.py in _get_data(base_url, dataname, cache, extension)
    185     #Python 3, don't think there will be any unicode in r datasets
    186     if sys.version[0] == '3':  # pragma: no cover
--> 187         data = data.decode('ascii', errors='strict')
    188     return StringIO(data), from_cache
    189 

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1288: ordinal not in range(128)

```



Discussion:
Yeah, the error is intended because I didn't know how to properly do the decoding on Python 3 and wasn't necessarily expecting unicode from an R dataset.

We might be able to postpone this until the py2 py3 common code base conversion, where we have to decide and figure out what to do with strings.
But we need examples in the documentation that don't fail on this.

What are the expectations for if something is cached using 2.x that we're able to read it from 3.x? That's the only hold up for me right now. I'd rather not support it, but it should be doable and we probably should support this.

I could also mangle the cached name to include the python version, so the Python 3 will never find a Python 2 cached item.

This is the implicit expectation now actually. It was just never actually checked for.

Nevermind, it's an easy fix.

Is this related to caching, I'm using a new python 3.3 version with installed statsmodels. Do we share cache directories across python version and different statsmodels folders?

But in either case, we are just caching the csv file. Isn't that the same across python versions?


Resolving patch:
diff --git a/statsmodels/datasets/utils.py b/statsmodels/datasets/utils.py
--- a/statsmodels/datasets/utils.py
+++ b/statsmodels/datasets/utils.py
@@ -131,7 +131,9 @@ def _cache_it(data, cache_path):
     if sys.version_info[0] >= 3:
         # for some reason encode("zip") won't work for me in Python 3?
         import zlib
-        open(cache_path, "wb").write(zlib.compress(pickle.dumps(data)))
+        # use protocol 2 so can open with python 2.x if cached in 3.x
+        open(cache_path, "wb").write(zlib.compress(pickle.dumps(data,
+                                                                protocol=2)))
     else:
         open(cache_path, "wb").write(pickle.dumps(data).encode("zip"))
 
@@ -141,7 +143,8 @@ def _open_cache(cache_path):
         # Python 3 build
         import zlib
         data = zlib.decompress(open(cache_path, 'rb').read())
-        data = pickle.loads(data)
+        # return as bytes object encoded in utf-8 for cross-compat of cached
+        data = pickle.loads(data).encode('utf-8')
     else:
         data = open(cache_path, 'rb').read().decode('zip')
         data = pickle.loads(data)
@@ -181,9 +184,9 @@ def _get_data(base_url, dataname, cache, extension="csv"):
         else:
             raise err
 
-    #Python 3, don't think there will be any unicode in r datasets
+    #Python 3, always decode as unicode
     if sys.version[0] == '3':  # pragma: no cover
-        data = data.decode('ascii', errors='strict')
+        data = data.decode('utf-8', errors='strict')
     return StringIO(data), from_cache
 
 


Test patch: