Repository: statsmodels/statsmodels

Issue:
formula example notebook broken
When I try to follow the _example_formulas_ example notebook, line number 6

```
dta = sm.datasets.get_rdataset("Guerry", "HistData", cache=True)
```

triggers the following error:

```
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1288: ordinal not in range(128)
```

I'm using statsmodels 0.5.0 on python 3.3.2.



Discussion:
Thanks for the report and pointing out an example case, see #1324 

---

Bytes or Unicode - that's the question

I guess the meta csv is utf8 and should be decoded that way instead of ascii

Also, there are some encoding inconsistencies, opening the cache is a different decoding

running on python 3.3.0

with cache:

```
>>> import statsmodels.api as sm
>>> dta = sm.datasets.get_rdataset("Guerry", "HistData", cache=True)
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    dta = sm.datasets.get_rdataset("Guerry", "HistData", cache=True)
  File "C:\Programs\Python33\lib\site-packages\statsmodels-0.5.0-py3.3-win32.egg\statsmodels\datasets\utils.py", line 245, in get_rdataset
    title = _get_dataset_meta(dataname, package, cache)
  File "C:\Programs\Python33\lib\site-packages\statsmodels-0.5.0-py3.3-win32.egg\statsmodels\datasets\utils.py", line 193, in _get_dataset_meta
    data = data.decode('ascii', errors='strict')
AttributeError: 'str' object has no attribute 'decode'
```

without cache it's the same as Andreas's

```
>>> dta = sm.datasets.get_rdataset("Guerry", "HistData", cache=False)
Traceback (most recent call last):
  File "<pyshell#3>", line 1, in <module>
    dta = sm.datasets.get_rdataset("Guerry", "HistData", cache=False)
  File "C:\Programs\Python33\lib\site-packages\statsmodels-0.5.0-py3.3-win32.egg\statsmodels\datasets\utils.py", line 246, in get_rdataset
    doc, _ = _get_data(docs_base_url, dataname, cache, "rst")
  File "C:\Programs\Python33\lib\site-packages\statsmodels-0.5.0-py3.3-win32.egg\statsmodels\datasets\utils.py", line 181, in _get_data
    data = data.decode('ascii', errors='strict')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1288: ordinal not in range(128)
```

```
Python 3.3.2+ (default, Oct  9 2013, 14:50:09) 
[GCC 4.8.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import statsmodels.api as sm
>>> dta = sm.datasets.get_rdataset("Guerry", "HistData", cache=False)
>>> dta = sm.datasets.get_rdataset("Guerry", "HistData", cache=True)
```

What gives?

I only have statsmodels 0.5.0 in python 3.3.0 right now. (and no spyder for it, so I don't know how to use pdb easily)

Trying out, I get different exceptions, or some cases work, depending on which and how the url file is obtained.

one case, I tried to reconstruct from the backtrace

```
>>> url
'https://raw.github.com/vincentarelbundock/Rdatasets/master/doc/HistData/rst/Guerry.rst'
>>> cache
'C:\\Users\\josef\\statsmodels_data'
>>> import statsmodels.datasets.utils as du
>>> datam, tmp = du._urlopen_cached(url, cache)
>>> type(datam)
<class 'bytes'>
>>> ms = datam.decode('ascii')
Traceback (most recent call last):
  File "<pyshell#80>", line 1, in <module>
    ms = datam.decode('ascii')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1288: ordinal not in range(128)
>>> datam[1250:1300]
b' against persons. Source: A2 (Compte g\xc3\xa9n\xc3\xa9ral,\n  '
```

I also looked at the python 3 source of statsmodels, and I don't see anything that 2to3 might have changed.

I'm using an installed statsmodels 0.5 on python 3.3 (and I don't have spyder for this to conveniently use pdb)

trying different versions, I get different exceptions or some cases it works

following up on one exception, this should show one "broken" case with non-ascii data:

```
>>> url
'https://raw.github.com/vincentarelbundock/Rdatasets/master/doc/HistData/rst/Guerry.rst'
>>> cache
'C:\\Users\\josef\\statsmodels_data'
>>> import statsmodels.datasets.utils as du
>>> datam, tmp = du._urlopen_cached(url, cache)
>>> type(datam)
<class 'bytes'>
>>> ms = datam.decode('ascii')
Traceback (most recent call last):
  File "<pyshell#80>", line 1, in <module>
    ms = datam.decode('ascii')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1288: ordinal not in range(128)
>>> datam[1250:1300]
b' against persons. Source: A2 (Compte g\xc3\xa9n\xc3\xa9ral,\n  '
```

I was trying to get through the cache and download code, but there are too many possibilities for me to understand. for example whether things have been cached from py 2 or py 3.
I also don't see if the system encoding is used at some point to write to disk, which on Windows wouldn't be utf-8, at least not on python 2  I guess.

Now I wrote it twice, I thought I didn't save/Comment the first version of the comment I wrote. 
firefox browser cache problems and switching tabs ?

my "impression" is that there are two "bugs"    (just a rough guess)

we are decoding the metadata with ascii even if it contains unicode.
we are pickle dumping the decoded data which might have a system encoding on python 2

Python2:

```
$ python
Python 2.7.6 |Anaconda 1.8.0 (64-bit)| (default, Nov 11 2013, 10:47:18) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import statsmodels.api as sm
>>> dta = sm.datasets.get_rdataset("Guerry", "HistData", cache=False)
>>> dta = sm.datasets.get_rdataset("Guerry", "HistData", cache=True)
>>>
```

Python3:

```
$ python
Python 3.3.2 |Anaconda 1.8.0 (64-bit)| (default, Aug  5 2013, 15:04:35) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import statsmodels.api as sm
>>> dta = sm.datasets.get_rdataset("Guerry", "HistData", cache=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/raid/home2/hilboll/SCIATRAN/no2prof/pyenv/lib/python3.3/site-packages/statsmodels/datasets/utils.py", line 253, in get_rdataset
    doc, _ = _get_data(docs_base_url, dataname, cache, "rst")
  File "/raid/home2/hilboll/SCIATRAN/no2prof/pyenv/lib/python3.3/site-packages/statsmodels/datasets/utils.py", line 187, in _get_data
    data = data.decode('ascii', errors='strict')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1288: ordinal not in range(128)
>>> dta = sm.datasets.get_rdataset("Guerry", "HistData", cache=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/raid/home2/hilboll/SCIATRAN/no2prof/pyenv/lib/python3.3/site-packages/statsmodels/datasets/utils.py", line 248, in get_rdataset
    data, from_cache = _get_data(data_base_url, dataname, cache)
  File "/raid/home2/hilboll/SCIATRAN/no2prof/pyenv/lib/python3.3/site-packages/statsmodels/datasets/utils.py", line 187, in _get_data
    data = data.decode('ascii', errors='strict')
AttributeError: 'str' object has no attribute 'decode'
>>>
```

BTW: is there a reason why you didn't use `gzip` to write an archive file. 
It doesn't really matter for the internal use, but I tried to open one of the cache ".zip" files and it is an invalid format. I guess because of missing metadata for archive files.

and the 3rd bug candidate is that we try to decode cached data twice.

There's no reason I didn't use gzip. AFAICT, zlib uses gzip for the compression but gzip provides file-like handles, which I don't use.

I don't see why a system encoding should matter for pickle.

The decoding to ascii of the meta data is a bug. I don't know why it doesn't show up for me or why it doesn't always decode to utf-8 like the data.


Resolving patch:
diff --git a/statsmodels/datasets/utils.py b/statsmodels/datasets/utils.py
--- a/statsmodels/datasets/utils.py
+++ b/statsmodels/datasets/utils.py
@@ -198,7 +198,7 @@ def _get_dataset_meta(dataname, package, cache):
     data, _ = _urlopen_cached(index_url, cache)
     #Python 3
     if sys.version[0] == '3':  # pragma: no cover
-        data = data.decode('ascii', errors='strict')
+        data = data.decode('utf-8', errors='strict')
     index = read_csv(StringIO(data))
     idx = np.logical_and(index.Item == dataname, index.Package == package)
     dataset_meta = index.ix[idx]


Test patch: