Repository: pandas-dev/pandas

Issue:
Unable to import Stata 13 database files with read_stata()
pandas v0.14.0 (May 31 , 2014) seems uncapable of importing Stata 13 datasets although according to this http://pandas.pydata.org/pandas-docs/stable/whatsnew.html, it should. Stata 12 files can be imported without problems.

The output of running this

``` python
import pandas
pandas.show_versions()
dta = pandas.io.stata.read_stata('D:\\Datos\\rferrer\\Desktop\\myauto.dta')
```

follows:

``` python
%run D:/Datos/RFERRER/Desktop/import_stata13.py

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 45 Stepping 7, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.14.0
nose: 1.3.0
Cython: 0.19.2
numpy: 1.8.0
scipy: 0.14.0
statsmodels: 0.5.0
IPython: 1.2.1
sphinx: 1.2.2
patsy: 0.2.0
scikits.timeseries: 0.91.3
dateutil: 2.2
pytz: 2013.8
bottleneck: None
tables: 2.4.0
numexpr: 2.2.2
matplotlib: 1.3.1
openpyxl: 1.8.5
xlrd: 0.9.2
xlwt: 0.7.5
xlsxwriter: None
lxml: 3.2.3
bs4: None
html5lib: 0.95-dev
bq: None
apiclient: None
rpy2: None
sqlalchemy: 0.8.3
pymysql: None
psycopg2: None
C:\Users\rferrer\AppData\Local\Enthought\Canopy\User\lib\site-packages\openpyxl\__init__.py:31: UserWarning: The installed version of lxml is too old to be used with openpyxl
  warnings.warn("The installed version of lxml is too old to be used with openpyxl")
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
C:\Users\rferrer\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.4.0.1938.win-x86_64\lib\site-packages\IPython\utils\py3compat.pyc in execfile(fname, glob, loc)
    195             else:
    196                 filename = fname
--> 197             exec compile(scripttext, filename, 'exec') in glob, loc
    198     else:
    199         def execfile(fname, *where):

D:\Datos\RFERRER\Desktop\import_stata13.py in <module>()
      3 pandas.show_versions()
      4 
----> 5 dta = pandas.io.stata.read_stata('D:\\Datos\\rferrer\\Desktop\\myauto.dta')

C:\Users\rferrer\AppData\Local\Enthought\Canopy\User\lib\site-packages\pandas\io\stata.pyc in read_stata(filepath_or_buffer, convert_dates, convert_categoricals, encoding, index)
     45         identifier of column that should be used as index of the DataFrame
     46     """
---> 47     reader = StataReader(filepath_or_buffer, encoding)
     48 
     49     return reader.data(convert_dates, convert_categoricals, index)

C:\Users\rferrer\AppData\Local\Enthought\Canopy\User\lib\site-packages\pandas\io\stata.pyc in __init__(self, path_or_buf, encoding)
    455             self.path_or_buf = path_or_buf
    456 
--> 457         self._read_header()
    458 
    459     def _read_header(self):

C:\Users\rferrer\AppData\Local\Enthought\Canopy\User\lib\site-packages\pandas\io\stata.pyc in _read_header(self)
    657 
    658         """Calculate size of a data record."""
--> 659         self.col_sizes = lmap(lambda x: self._calcsize(x), self.typlist)
    660 
    661     def _calcsize(self, fmt):

C:\Users\rferrer\AppData\Local\Enthought\Canopy\User\lib\site-packages\pandas\io\stata.pyc in <lambda>(x)
    657 
    658         """Calculate size of a data record."""
--> 659         self.col_sizes = lmap(lambda x: self._calcsize(x), self.typlist)
    660 
    661     def _calcsize(self, fmt):

C:\Users\rferrer\AppData\Local\Enthought\Canopy\User\lib\site-packages\pandas\io\stata.pyc in _calcsize(self, fmt)
    661     def _calcsize(self, fmt):
    662         return (type(fmt) is int and fmt
--> 663                 or struct.calcsize(self.byteorder + fmt))
    664 
    665     def _col_size(self, k=None):

TypeError: cannot concatenate 'str' and 'NoneType' objects
```

The dataset `myauto.dta` is just the `auto` dataset made available running `sysuse auto` within Stata.

The problem is originally documented here: http://stackoverflow.com/questions/24053652/pandas-and-stata-13-files.

My Python is set up with [Enthough Canopy 1.4.0 (64 bit)](https://www.enthought.com/products/canopy/).

Error when writing non-ascii allowed characters to Stata dta
When trying to write a Stata dataset with strings containing upper latin-1 characters (which are allowed by the Stata format), I get an encoding error.

``` python
import pandas.io.stata as sta
sr = sta.StataReader('pandas/pandas/io/tests/data/stata1_encoding.dta')
df = sr.data()
sw = sta.StataWriter('stata1_encoding_dup.dta', df)
sw.write_file()
```

I get the following output:

``` python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\pandas\io\stata.py", line 1242, in write_file
    self._write_data_nodates()
  File "C:\Python27\lib\site-packages\pandas\io\stata.py", line 1326, in _write_data_nodates
    self._write(var)
  File "C:\Python27\lib\site-packages\pandas\io\stata.py", line 1104, in _write
    self._file.write(to_write)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 1:ordinal not in range(128)
```

Machine info:

```
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.6.final.0
python-bits: 32
OS: Windows
OS-release: 7
machine: AMD64
processor: AMD64 Family 16 Model 6 Stepping 3, AuthenticAMD
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.14.0
nose: 1.3.3
Cython: 0.20.1
numpy: 1.8.1
scipy: None
statsmodels: None
IPython: None
sphinx: None
patsy: None
scikits.timeseries: None
dateutil: 2.2
pytz: 2014.3
bottleneck: 0.8.0
tables: None
numexpr: None
matplotlib: None
openpyxl: 1.8.6
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
bq: None
apiclient: None
rpy2: None
sqlalchemy: None
pymysql: None
psycopg2: None
```



Discussion:
cc @bashtage

Will try and look into this weekend. I don't have Stata 13 so I can't easily generate any data files, which makes debugging hard.  Ideally there would be data files for the same dta files that are in the test data.

@refp16 where do you see this in the whatsnew?

i think he's referring to https://github.com/pydata/pandas/issues/4291 and #4662

@jreback Just search for "GH4291" in http://pandas.pydata.org/pandas-docs/stable/whatsnew.html.

This may be clear from error message but it looks like it happens with a `stata 13` data file when there is a string column:

```
  ___  ____  ____  ____  ____ (R)
 /__    /   ____/   /   ____/
___/   /   /___/   /   /___/   13.0   Copyright 1985-2013 StataCorp LP
  Statistics/Data Analysis            StataCorp
                                      4905 Lakeway Drive
     MP - Parallel Edition            College Station, Texas 77845 USA
                                      800-STATA-PC        http://www.stata.com
                                      979-696-4600        stata@stata.com
                                      979-696-4601 (fax)

5-user 6-core Stata network perpetual license:

. insheet using foo.csv
(3 vars, 5 obs)

. list

     +---------------+
     | x1   x2    x3 |
     |---------------|
  1. |  1    5   'a' |
  2. |  2    4   'b' |
  3. |  3    3   'c' |
  4. |  4    2   'd' |
  5. |  5    1   'e' |
     +---------------+

. save all.dta, replace
(note: file all.dta not found)
file all.dta saved

. drop x3

. save just_numeric.dta, replace
(note: file just_numeric.dta not found)
file just_numeric.dta saved
```

Now reading them into pandas 0.14

``` python


In [6]: df = pd.read_stata('just_numeric.dta')

In [7]: df
Out[7]: 
   x1  x2
0   1   5
1   2   4
2   3   3
3   4   2
4   5   1

In [8]: df = pd.read_stata('all.dta')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-8c472969f1f6> in <module>()
----> 1 df = pd.read_stata('all.dta')

/usr/lib64/python2.7/site-packages/pandas/io/stata.pyc in read_stata(filepath_or_buffer, convert_dates, convert_categoricals, encoding, index)
     45         identifier of column that should be used as index of the DataFrame
     46     """
---> 47     reader = StataReader(filepath_or_buffer, encoding)
     48 
     49     return reader.data(convert_dates, convert_categoricals, index)

/usr/lib64/python2.7/site-packages/pandas/io/stata.pyc in __init__(self, path_or_buf, encoding)
    455             self.path_or_buf = path_or_buf
    456 
--> 457         self._read_header()
    458 
    459     def _read_header(self):

/usr/lib64/python2.7/site-packages/pandas/io/stata.pyc in _read_header(self)
    657 
    658         """Calculate size of a data record."""
--> 659         self.col_sizes = lmap(lambda x: self._calcsize(x), self.typlist)
    660 
    661     def _calcsize(self, fmt):

/usr/lib64/python2.7/site-packages/pandas/io/stata.pyc in <lambda>(x)
    657 
    658         """Calculate size of a data record."""
--> 659         self.col_sizes = lmap(lambda x: self._calcsize(x), self.typlist)
    660 
    661     def _calcsize(self, fmt):

/usr/lib64/python2.7/site-packages/pandas/io/stata.pyc in _calcsize(self, fmt)
    661     def _calcsize(self, fmt):
    662         return (type(fmt) is int and fmt
--> 663                 or struct.calcsize(self.byteorder + fmt))
    664 
    665     def _col_size(self, k=None):

TypeError: cannot concatenate 'str' and 'NoneType' objects
```

Can you post the problematic dta somewhere?  Dropbox link, or email me?

@bashtage Try here:

http://www.evernote.com/shard/s11/sh/1a28ae6d-512c-4a21-8353-32bb7d5956d3/ab5b783d9711c5e3f800afe5e3ff3a5f

Those are two files. One problematic (with string variable), the other one fine.

Thanks, got it.

Just a note:

Long strings (or `strL`) were introduced in Stata 13. But notice that the string variable does not have to be an `strL` type (holds up to 2000000000 bytes) to generate the problem. In my previous comment the problematic file holds a simple `str1` variable (holds 1 byte).

cc @bashtage

hmm. it seems it not being encoded to the passed encoding, odd (and is not tested).

the reading however works

FYI, you can/should use:

`df = pd.read_stata(.......,encoding='latin-1')`
`df.to_stata(......,encoding='latin-1')`
(thought this last seems to be a bug)


Resolving patch:
diff --git a/pandas/io/stata.py b/pandas/io/stata.py
--- a/pandas/io/stata.py
+++ b/pandas/io/stata.py
@@ -512,8 +512,10 @@ def _read_header(self):
             try:
                 i = 0
                 for typ in typlist:
-                    if typ <= 2045 or typ == 32768:
-                        self.typlist[i] = None
+                    if typ <= 2045:
+                        self.typlist[i] = typ
+                    elif typ == 32768:
+                        raise ValueError("Long strings are not supported")
                     else:
                         self.typlist[i] = self.TYPE_MAP_XML[typ]
                     i += 1
@@ -1326,7 +1328,10 @@ def _write_data_nodates(self):
                         var = _pad_bytes('', typ)
                     if len(var) < typ:
                         var = _pad_bytes(var, typ)
-                    self._write(var)
+                    if compat.PY3:
+                        self._write(var)
+                    else:
+                        self._write(var.encode(self._encoding))
                 else:
                     try:
                         self._file.write(struct.pack(byteorder + TYPE_MAP[typ],
@@ -1356,7 +1361,10 @@ def _write_data_dates(self):
                 if typ <= 244:  # we've got a string
                     if len(var) < typ:
                         var = _pad_bytes(var, typ)
-                    self._write(var)
+                    if compat.PY3:
+                        self._write(var)
+                    else:
+                        self._write(var.encode(self._encoding))
                 else:
                     self._file.write(struct.pack(byteorder+TYPE_MAP[typ], var))
 


Test patch:
diff --git a/pandas/io/tests/data/stata5_117.dta b/pandas/io/tests/data/stata5_117.dta
new file mode 100644
Binary files /dev/null and b/pandas/io/tests/data/stata5_117.dta differ
diff --git a/pandas/io/tests/data/stata6_117.dta b/pandas/io/tests/data/stata6_117.dta
new file mode 100644
Binary files /dev/null and b/pandas/io/tests/data/stata6_117.dta differ
diff --git a/pandas/io/tests/test_stata.py b/pandas/io/tests/test_stata.py
--- a/pandas/io/tests/test_stata.py
+++ b/pandas/io/tests/test_stata.py
@@ -60,11 +60,13 @@ def setUp(self):
         self.dta14_113 = os.path.join(self.dirpath, 'stata5_113.dta')
         self.dta14_114 = os.path.join(self.dirpath, 'stata5_114.dta')
         self.dta14_115 = os.path.join(self.dirpath, 'stata5_115.dta')
+        self.dta14_117 = os.path.join(self.dirpath, 'stata5_117.dta')
 
         self.csv15 = os.path.join(self.dirpath, 'stata6.csv')
         self.dta15_113 = os.path.join(self.dirpath, 'stata6_113.dta')
         self.dta15_114 = os.path.join(self.dirpath, 'stata6_114.dta')
         self.dta15_115 = os.path.join(self.dirpath, 'stata6_115.dta')
+        self.dta15_117 = os.path.join(self.dirpath, 'stata6_117.dta')
 
     def read_dta(self, file):
         return read_stata(file, convert_dates=True)
@@ -281,6 +283,11 @@ def test_encoding(self):
             self.assertEqual(result, expected)
             self.assertIsInstance(result, unicode)
 
+        with tm.ensure_clean() as path:
+            encoded.to_stata(path,encoding='latin-1', write_index=False)
+            reread_encoded = read_stata(path, encoding='latin-1')
+            tm.assert_frame_equal(encoded, reread_encoded)
+
     def test_read_write_dta11(self):
         original = DataFrame([(1, 2, 3, 4)],
                              columns=['good', compat.u('b\u00E4d'), '8number', 'astringwithmorethan32characters______'])
@@ -354,9 +361,12 @@ def test_read_write_reread_dta14(self):
         parsed_114.index.name = 'index'
         parsed_115 = self.read_dta(self.dta14_115)
         parsed_115.index.name = 'index'
+        parsed_117 = self.read_dta(self.dta14_117)
+        parsed_117.index.name = 'index'
 
         tm.assert_frame_equal(parsed_114, parsed_113)
         tm.assert_frame_equal(parsed_114, parsed_115)
+        tm.assert_frame_equal(parsed_114, parsed_117)
 
         with tm.ensure_clean() as path:
             parsed_114.to_stata(path, {'date_td': 'td'})
@@ -375,10 +385,12 @@ def test_read_write_reread_dta15(self):
         parsed_113 = self.read_dta(self.dta15_113)
         parsed_114 = self.read_dta(self.dta15_114)
         parsed_115 = self.read_dta(self.dta15_115)
+        parsed_117 = self.read_dta(self.dta15_117)
 
         tm.assert_frame_equal(expected, parsed_114)
         tm.assert_frame_equal(parsed_113, parsed_114)
         tm.assert_frame_equal(parsed_114, parsed_115)
+        tm.assert_frame_equal(parsed_114, parsed_117)
 
     def test_timestamp_and_label(self):
         original = DataFrame([(1,)], columns=['var'])