Repository: statsmodels/statsmodels

Issue:
get_rdataset with cache is broken (perhaps only py3)
Results in internet always being used (and failing if not present)

Issue is that the incorrect method is called on the bytes object: should be `decode` (like in the `else` block just below it)

Below I show the usage of the correct method within `ipdb`

```
In [78]: import os

In [79]: cache_path = '/home/dlovell/statsmodels_data/raw.github.com,vincentarelbundock,Rdatasets,master,csv,datasets,sunspots.csv.zip'

In [80]: os.path.exists(cache_path)
Out[80]: True

In [81]: data = sdu._open_cache(cache_path)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-81-fb74558ddc07> in <module>()
----> 1 data = sdu._open_cache(cache_path)

~/anaconda3/envs/pymc3/lib/python3.6/site-packages/statsmodels/datasets/utils.py in _open_cache(cache_path)
    185         data = zlib.decompress(open(cache_path, 'rb').read())
    186         # return as bytes object encoded in utf-8 for cross-compat of cached
--> 187         data = cPickle.loads(data).encode('utf-8')
    188     else:
    189         data = open(cache_path, 'rb').read().decode('zip')

AttributeError: 'bytes' object has no attribute 'encode'
> /home/dlovell/anaconda3/envs/pymc3/lib/python3.6/site-packages/statsmodels/datasets/utils.py(187)_open_cache()
    185         data = zlib.decompress(open(cache_path, 'rb').read())
    186         # return as bytes object encoded in utf-8 for cross-compat of cached
--> 187         data = cPickle.loads(data).encode('utf-8')
    188     else:
    189         data = open(cache_path, 'rb').read().decode('zip')

ipdb> len(cPickle.loads(data).decode('utf-8'))
79218
```


Discussion:
Actually, _get_data does the decode, the solution is just to remove the call to `encode`
temporarily prio-high as reminder to look at it
Something is broken, but I don't understand why the current unit test doesn't fail.
If I run the unit test, then it passes. Travis had and internet connection during testing, which I guess it still has, but the unit test doesn't fail. 
There is an `assert_(duncan.from_cache)` and from_cache is True in the unit tests.

When I run an example in the interpreter with a different cache directory, then I get the same error as in the description.

This seems to work with **old** cached files.
checking: we have the cache files in the repo, which I assume were created with python 2
https://github.com/statsmodels/statsmodels/tree/master/statsmodels/datasets/tests

So, maybe the underlying problem is that the cache writing with py3 uses a different format/encoding than with py 2.
If I didn't make a mistake in shuffling around cache files:
The proposed fix of removing encode causes an exception when reading old (py2) cache files when I run the unit test.
```
    def _get_data(base_url, dataname, cache, extension="csv"):
        url = base_url + (dataname + ".%s") % extension
        try:
            data, from_cache = _urlopen_cached(url, cache)
        except HTTPError as err:
            if '404' in str(err):
                raise ValueError("Dataset %s was not found." % dataname)
            else:
                raise err

>       data = data.decode('utf-8', 'strict')
E       AttributeError: 'str' object has no attribute 'decode'
```




Resolving patch:
diff --git a/statsmodels/datasets/utils.py b/statsmodels/datasets/utils.py
--- a/statsmodels/datasets/utils.py
+++ b/statsmodels/datasets/utils.py
@@ -171,6 +171,7 @@ def _cache_it(data, cache_path):
         # for some reason encode("zip") won't work for me in Python 3?
         import zlib
         # use protocol 2 so can open with python 2.x if cached in 3.x
+        data = data.decode('utf-8')
         open(cache_path, "wb").write(zlib.compress(cPickle.dumps(data,
                                                                  protocol=2)))
     else:


Test patch:
diff --git a/statsmodels/datasets/tests/test_utils.py b/statsmodels/datasets/tests/test_utils.py
--- a/statsmodels/datasets/tests/test_utils.py
+++ b/statsmodels/datasets/tests/test_utils.py
@@ -8,6 +8,7 @@
 
 cur_dir = os.path.dirname(os.path.abspath(__file__))
 
+
 def test_get_rdataset():
     # smoke test
     test_url = "https://raw.githubusercontent.com/vincentarelbundock/Rdatasets/master/csv/datasets/cars.csv"
@@ -18,6 +19,17 @@ def test_get_rdataset():
     assert_(isinstance(duncan, utils.Dataset))
     assert_(duncan.from_cache)
 
+    # test writing and reading cache
+    guerry = get_rdataset("Guerry", "HistData", cache=cur_dir)
+    assert_(guerry.from_cache is False)
+    guerry2 = get_rdataset("Guerry", "HistData", cache=cur_dir)
+    assert_(guerry2.from_cache is True)
+    fn = "raw.github.com,vincentarelbundock,Rdatasets,master,csv,HistData,Guerry.csv.zip"
+    os.remove(os.path.join(cur_dir, fn))
+    fn = "raw.github.com,vincentarelbundock,Rdatasets,master,doc,HistData,rst,Guerry.rst.zip"
+    os.remove(os.path.join(cur_dir, fn))
+
+
 def test_webuse():
     # test copied and adjusted from iolib/tests/test_foreign
     from statsmodels.iolib.tests.results.macrodata import macrodata_result as res2