Repository: pandas-dev/pandas

Issue:
Pandas 0.14.1 StataReader seems to read .dta files 10x slower than Pandas 0.13.1
Hello, I recently upgraded to Pandas 0.14.1 from Pandas 0.13.1, and am having trouble reading Stata .dta files using StataReader.  Files that used to take 20 seconds to read now take 300 seconds, and files that used to take 220 seconds are not read even after 20 minutes.

I would really like to use the newer version of Pandas for these large datasets, and import a .dta file rather than a .csv or other filetype in order to maintain my value labels from Stata.

Steps for reproduction:

First, create a large dataset in Stata 13:

```
clear
set obs 11500
forvalues i = 1/8000{
gen var`i' = 1
}

saveold bigdataset, replace
```

Second, try to read it into pandas using StataReader:

```
from pandas.io.stata import StataReader

reader = StataReader('bigdataset.dta')
data = reader.data()
```

Using pandas 0.13.1, this takes around 220 seconds, which is ok, but using pandas 0.14.1, nothing has happened even after waiting around 20 minutes.

When I test this using a smaller dataset:

```
clear
set obs 11500
forvalues i = 1/1000{
gen var`i' = 1
}

saveold smalldataset, replace
```

```
from pandas.io.stata import StataReader

reader = StataReader('smalldataset.dta')
data = reader.data()
```

Using pandas 0.13.1, this takes around 20 seconds, but using pandas 0.14.1, this takes around 300 seconds.

Thanks for reading!  Here is the output from show_versions(), if relevant:
## INSTALLED VERSIONS

commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-32-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.13.1
Cython: 0.20.2
numpy: 1.8.1
scipy: 0.13.3
statsmodels: 0.5.0
IPython: 1.2.1
sphinx: 1.2.2
patsy: 0.2.1
scikits.timeseries: None
dateutil: 1.5
pytz: 2012c
bottleneck: None
tables: 3.1.1
numexpr: 2.2.2
matplotlib: 1.3.1
openpyxl: 1.7.0
xlrd: 0.9.2
xlwt: 0.7.5
xlsxwriter: None
sqlalchemy: None
lxml: 3.3.3
bs4: 4.2.1
html5lib: 0.999
bq: None
apiclient: None



Discussion:
cc @bashtage want to have a look?

Also a vbench for this would be nice to have. I didn't see any in `io_bench` or `parser_bench`.

should go in `vb_suite/packers.py` but yes would be nice as well

hmm, seems ok to me

@Ffamran can you show `df.info()` for the frame?

```
In [1]: df = DataFrame(np.random.randn(1000,2))

In [3]: df['C'] = 'foo'

In [4]: df.to_stata('test.stata')
```

master/0.15.0

```
In [5]: %timeit pd.read_stata('test.stata')
100 loops, best of 3: 13.2 ms per loop
```

0.14.1

```
In [4]: %timeit pd.read_stata('test.stata')
100 loops, best of 3: 13 ms per loop
```

13.1

```
In [9]: %timeit pd.read_stata('test.stata')
100 loops, best of 3: 20.4 ms per loop
```

This is probably the problem

```
        self.VALID_RANGE = \
            {
                'b': (-127, 100),
                'h': (-32767, 32740),
                'l': (-2147483647, 2147483620),
                'f': (np.float32(struct.unpack('<f', float32_min)[0]),
                      np.float32(struct.unpack('<f', float32_max)[0])),
                'd': (np.float64(struct.unpack('<d', float64_min)[0]),
                      np.float64(struct.unpack('<d', float64_max)[0]))
            }
```

This is called for every value, which then calls `unpack` 4 times per value and float.  These need to be converted to some other form that is faster to parse.

Is an easy fix - this is static can can be moved outside of the class and called once.

could also easily vectorize this (the range checking)

e.g. separate by dtype and then do the call (you can do this in blocks)

e.g.

```
for b in df.blocks:
    values = b.values.ravel()

    if b.is_float:
       # float block
    elsif b.is_integer:
       # integer
       result = values([values>max_int) | (values<min_int)]
       if result.any():
           raise out-of-bounds

# etc
```

True, but I think restoring this to simply be static would return performance to 0.13 levels.  This is what a profile shows (4 slowest):

0.14

```
1005027 0.708   0   0.708   0   {method 'read' of 'file' objects}       0.708   0.708
1000000 1.247   0   13.438  0   stata.py:721(<lambda>)      13.438  1.247
1998    6.835   0.003   6.922   0.003   function_base.py:3112(delete)       6.922   6.835
1000000 11.014  0   11.292  0   stata.py:677(_unpack)       11.292  11.014
```

0.13

```
999002  0.484   0   1.215   0   internals.py:180(__contains__)  0.484
1007002 0.673   0   0.741   0   index.py:604(__contains__)  0.673
1000000 0.954   0   2.716   0   stata.py:601(<lambda>)  0.954
1000000 0.972   0   1.16    0   stata.py:557(_unpack)   0.972

```

So around 12x slower.  There is another line that is also slow that will need some checking - not sure what `function_base.py:3112` is or what is triggering it.

ok either way. pls add a vbench while you are add it (for both reading and writing), see `vb_suite/packers.py`


Resolving patch:
diff --git a/pandas/io/stata.py b/pandas/io/stata.py
--- a/pandas/io/stata.py
+++ b/pandas/io/stata.py
@@ -9,7 +9,6 @@
 You can find more information on http://presbrey.mit.edu/PyDTA and
 http://statsmodels.sourceforge.net/devel/
 """
-# TODO: Fix this module so it can use cross-compatible zip, map, and range
 import numpy as np
 
 import sys
@@ -20,14 +19,16 @@
 from pandas.core.categorical import Categorical
 import datetime
 from pandas import compat
-from pandas.compat import long, lrange, lmap, lzip, text_type, string_types
+from pandas.compat import lrange, lmap, lzip, text_type, string_types, range, \
+    zip
 from pandas import isnull
 from pandas.io.common import get_filepath_or_buffer
 from pandas.lib import max_len_string_array, is_string_array
 from pandas.tslib import NaT
 
 def read_stata(filepath_or_buffer, convert_dates=True,
-               convert_categoricals=True, encoding=None, index=None):
+               convert_categoricals=True, encoding=None, index=None,
+               convert_missing=False):
     """
     Read Stata file into DataFrame
 
@@ -44,10 +45,19 @@ def read_stata(filepath_or_buffer, convert_dates=True,
         support unicode. None defaults to cp1252.
     index : identifier of index column
         identifier of column that should be used as index of the DataFrame
+    convert_missing : boolean, defaults to False
+        Flag indicating whether to convert missing values to their Stata
+        representations.  If False, missing values are replaced with nans.
+        If True, columns containing missing values are returned with
+        object data types and missing values are represented by
+        StataMissingValue objects.
     """
     reader = StataReader(filepath_or_buffer, encoding)
 
-    return reader.data(convert_dates, convert_categoricals, index)
+    return reader.data(convert_dates,
+                       convert_categoricals,
+                       index,
+                       convert_missing)
 
 _date_formats = ["%tc", "%tC", "%td", "%d", "%tw", "%tm", "%tq", "%th", "%ty"]
 
@@ -291,35 +301,76 @@ class StataMissingValue(StringMixin):
 
     Parameters
     -----------
-    offset
-    value
+    value : int8, int16, int32, float32 or float64
+        The Stata missing value code
 
     Attributes
     ----------
-    string
-    value
+    string : string
+        String representation of the Stata missing value
+    value : int8, int16, int32, float32 or float64
+        The original encoded missing value
 
     Notes
     -----
     More information: <http://www.stata.com/help.cgi?missing>
+
+    Integer missing values make the code '.', '.a', ..., '.z' to the ranges
+    101 ... 127 (for int8), 32741 ... 32767  (for int16) and 2147483621 ...
+    2147483647 (for int32).  Missing values for floating point data types are
+    more complex but the pattern is simple to discern from the following table.
+
+    np.float32 missing values (float in Stata)
+    0000007f    .
+    0008007f    .a
+    0010007f    .b
+    ...
+    00c0007f    .x
+    00c8007f    .y
+    00d0007f    .z
+
+    np.float64 missing values (double in Stata)
+    000000000000e07f    .
+    000000000001e07f    .a
+    000000000002e07f    .b
+    ...
+    000000000018e07f    .x
+    000000000019e07f    .y
+    00000000001ae07f    .z
     """
-    # TODO: Needs test
-    def __init__(self, offset, value):
+
+    # Construct a dictionary of missing values
+    MISSING_VALUES = {}
+    bases = (101, 32741, 2147483621)
+    for b in bases:
+        MISSING_VALUES[b] = '.'
+        for i in range(1, 27):
+            MISSING_VALUES[i + b] = '.' + chr(96 + i)
+
+    base = b'\x00\x00\x00\x7f'
+    increment = struct.unpack('<i', b'\x00\x08\x00\x00')[0]
+    for i in range(27):
+        value = struct.unpack('<f', base)[0]
+        MISSING_VALUES[value] = '.'
+        if i > 0:
+            MISSING_VALUES[value] += chr(96 + i)
+        int_value = struct.unpack('<i', struct.pack('<f', value))[0] + increment
+        base = struct.pack('<i', int_value)
+
+    base = b'\x00\x00\x00\x00\x00\x00\xe0\x7f'
+    increment = struct.unpack('q', b'\x00\x00\x00\x00\x00\x01\x00\x00')[0]
+    for i in range(27):
+        value = struct.unpack('<d', base)[0]
+        MISSING_VALUES[value] = '.'
+        if i > 0:
+            MISSING_VALUES[value] += chr(96 + i)
+        int_value = struct.unpack('q', struct.pack('<d', value))[0] + increment
+        base = struct.pack('q', int_value)
+
+    def __init__(self, value):
         self._value = value
-        value_type = type(value)
-        if value_type in int:
-            loc = value - offset
-        elif value_type in (float, np.float32, np.float64):
-            if value <= np.finfo(np.float32).max:  # float32
-                conv_str, byte_loc, scale = '<f', 1, 8
-            else:
-                conv_str, byte_loc, scale = '<d', 5, 1
-            value_bytes = struct.pack(conv_str, value)
-            loc = (struct.unpack('<b', value_bytes[byte_loc])[0] / scale) + 0
-        else:
-            # Should never be hit
-            loc = 0
-        self._str = loc is 0 and '.' or ('.' + chr(loc + 96))
+        self._str = self.MISSING_VALUES[value]
+
     string = property(lambda self: self._str,
                       doc="The Stata representation of the missing value: "
                           "'.', '.a'..'.z'")
@@ -333,6 +384,10 @@ def __repr__(self):
         # not perfect :-/
         return "%s(%s)" % (self.__class__, self)
 
+    def __eq__(self, other):
+        return (isinstance(other, self.__class__)
+                and self.string == other.string and self.value == other.value)
+
 
 class StataParser(object):
     _default_encoding = 'cp1252'
@@ -711,15 +766,7 @@ def _col_size(self, k=None):
             return self.col_sizes[k]
 
     def _unpack(self, fmt, byt):
-        d = struct.unpack(self.byteorder + fmt, byt)[0]
-        if fmt[-1] in self.VALID_RANGE:
-            nmin, nmax = self.VALID_RANGE[fmt[-1]]
-            if d < nmin or d > nmax:
-                if self._missing_values:
-                    return StataMissingValue(nmax, d)
-                else:
-                    return None
-        return d
+        return struct.unpack(self.byteorder + fmt, byt)[0]
 
     def _null_terminate(self, s):
         if compat.PY3 or self._encoding is not None:  # have bytes not strings,
@@ -752,16 +799,15 @@ def _next(self):
                     )
             return data
         else:
-            return list(
-                map(
+            return lmap(
                     lambda i: self._unpack(typlist[i],
                                            self.path_or_buf.read(
                                                self._col_size(i)
                                            )),
                     range(self.nvar)
-                )
             )
 
+
     def _dataset(self):
         """
         Returns a Python generator object for iterating over the dataset.
@@ -853,7 +899,8 @@ def _read_strls(self):
             self.GSO[v_o] = self.path_or_buf.read(length-1)
             self.path_or_buf.read(1)  # zero-termination
 
-    def data(self, convert_dates=True, convert_categoricals=True, index=None):
+    def data(self, convert_dates=True, convert_categoricals=True, index=None,
+             convert_missing=False):
         """
         Reads observations from Stata file, converting them into a dataframe
 
@@ -866,11 +913,18 @@ def data(self, convert_dates=True, convert_categoricals=True, index=None):
             variables
         index : identifier of index column
             identifier of column that should be used as index of the DataFrame
+        convert_missing : boolean, defaults to False
+            Flag indicating whether to convert missing values to their Stata
+            representation.  If False, missing values are replaced with
+            nans.  If True, columns containing missing values are returned with
+            object data types and missing values are represented by
+            StataMissingValue objects.
 
         Returns
         -------
         y : DataFrame instance
         """
+        self._missing_values = convert_missing
         if self._data_read:
             raise Exception("Data has already been read.")
         self._data_read = True
@@ -894,18 +948,62 @@ def data(self, convert_dates=True, convert_categoricals=True, index=None):
         if convert_categoricals:
             self._read_value_labels()
 
+        # TODO: Refactor to use a dictionary constructor and the correct dtype from the start?
         if len(data)==0:
             data = DataFrame(columns=self.varlist, index=index)
         else:
             data = DataFrame(data, columns=self.varlist, index=index)
 
         cols_ = np.where(self.dtyplist)[0]
+
+        # Convert columns (if needed) to match input type
+        index = data.index
+        requires_type_conversion = False
+        data_formatted = []
         for i in cols_:
             if self.dtyplist[i] is not None:
                 col = data.columns[i]
-                if data[col].dtype is not np.dtype(object):
-                    data[col] = Series(data[col], data[col].index,
-                                       self.dtyplist[i])
+                dtype = data[col].dtype
+                if (dtype != np.dtype(object)) and (dtype != self.dtyplist[i]):
+                    requires_type_conversion = True
+                    data_formatted.append((col, Series(data[col], index, self.dtyplist[i])))
+                else:
+                    data_formatted.append((col, data[col]))
+        if requires_type_conversion:
+            data = DataFrame.from_items(data_formatted)
+        del data_formatted
+
+        # Check for missing values, and replace if found
+        for i, colname in enumerate(data):
+            fmt = self.typlist[i]
+            if fmt not in self.VALID_RANGE:
+                continue
+
+            nmin, nmax = self.VALID_RANGE[fmt]
+            series = data[colname]
+            missing = np.logical_or(series < nmin, series > nmax)
+
+            if not missing.any():
+                continue
+
+            if self._missing_values:  # Replacement follows Stata notation
+                missing_loc = np.argwhere(missing)
+                umissing, umissing_loc = np.unique(series[missing],
+                                                   return_inverse=True)
+                replacement = Series(series, dtype=np.object)
+                for i, um in enumerate(umissing):
+                    missing_value = StataMissingValue(um)
+
+                    loc = missing_loc[umissing_loc == i]
+                    replacement.iloc[loc] = missing_value
+            else:  # All replacements are identical
+                dtype = series.dtype
+                if dtype not in (np.float32, np.float64):
+                    dtype = np.float64
+                replacement = Series(series, dtype=dtype)
+                replacement[missing] = np.nan
+
+            data[colname] = replacement
 
         if convert_dates:
             cols = np.where(lmap(lambda x: x in _date_formats,
diff --git a/vb_suite/packers.py b/vb_suite/packers.py
--- a/vb_suite/packers.py
+++ b/vb_suite/packers.py
@@ -121,3 +121,25 @@ def remove(f):
 packers_write_json_date_index = Benchmark("df.to_json(f,orient='split')", setup, cleanup="remove(f)", start_date=start_date)
 setup = setup + setup_int_index
 packers_write_json = Benchmark("df.to_json(f,orient='split')", setup, cleanup="remove(f)", start_date=start_date)
+
+#----------------------------------------------------------------------
+# stata
+
+setup = common_setup + """
+df.to_stata(f, {'index': 'tc'})
+"""
+packers_read_stata = Benchmark("pd.read_stata(f)", setup, start_date=start_date)
+
+packers_write_stata = Benchmark("df.to_stata(f, {'index': 'tc'})", setup, cleanup="remove(f)", start_date=start_date)
+
+setup = common_setup + """
+df['int8_'] = [randint(-127,100) for _ in range(N)]
+df['int16_'] = [randint(-127,100) for _ in range(N)]
+df['int32_'] = [randint(-127,100) for _ in range(N)]
+df['float32_'] = np.array(randn(N), dtype=np.float32)
+df.to_stata(f, {'index': 'tc'})
+"""
+
+packers_read_stata_with_int = Benchmark("pd.read_stata(f)", setup, start_date=start_date)
+
+packers_write_stata_with_int = Benchmark("df.to_stata(f, {'index': 'tc'})", setup, cleanup="remove(f)", start_date=start_date)


Test patch:
diff --git a/pandas/io/tests/data/stata8_113.dta b/pandas/io/tests/data/stata8_113.dta
new file mode 100644
Binary files /dev/null and b/pandas/io/tests/data/stata8_113.dta differ
diff --git a/pandas/io/tests/data/stata8_115.dta b/pandas/io/tests/data/stata8_115.dta
new file mode 100644
Binary files /dev/null and b/pandas/io/tests/data/stata8_115.dta differ
diff --git a/pandas/io/tests/data/stata8_117.dta b/pandas/io/tests/data/stata8_117.dta
new file mode 100644
Binary files /dev/null and b/pandas/io/tests/data/stata8_117.dta differ
diff --git a/pandas/io/tests/test_stata.py b/pandas/io/tests/test_stata.py
--- a/pandas/io/tests/test_stata.py
+++ b/pandas/io/tests/test_stata.py
@@ -5,16 +5,18 @@
 import os
 import warnings
 import nose
+import struct
 import sys
 from distutils.version import LooseVersion
 
 import numpy as np
 
 import pandas as pd
+from pandas.compat import iterkeys
 from pandas.core.frame import DataFrame, Series
 from pandas.io.parsers import read_csv
 from pandas.io.stata import (read_stata, StataReader, InvalidColumnName,
-    PossiblePrecisionLoss)
+    PossiblePrecisionLoss, StataMissingValue)
 import pandas.util.testing as tm
 from pandas.util.misc import is_little_endian
 from pandas import compat
@@ -71,6 +73,10 @@ def setUp(self):
         self.dta16_115 = os.path.join(self.dirpath, 'stata7_115.dta')
         self.dta16_117 = os.path.join(self.dirpath, 'stata7_117.dta')
 
+        self.dta17_113 = os.path.join(self.dirpath, 'stata8_113.dta')
+        self.dta17_115 = os.path.join(self.dirpath, 'stata8_115.dta')
+        self.dta17_117 = os.path.join(self.dirpath, 'stata8_117.dta')
+
     def read_dta(self, file):
         return read_stata(file, convert_dates=True)
 
@@ -589,6 +595,50 @@ def test_excessively_long_string(self):
             with tm.ensure_clean() as path:
                 original.to_stata(path)
 
+    def test_missing_value_generator(self):
+        types = ('b','h','l')
+        df = DataFrame([[0.0]],columns=['float_'])
+        with tm.ensure_clean() as path:
+            df.to_stata(path)
+            valid_range = StataReader(path).VALID_RANGE
+        expected_values = ['.' + chr(97 + i) for i in range(26)]
+        expected_values.insert(0, '.')
+        for t in types:
+            offset = valid_range[t][1]
+            for i in range(0,27):
+                val = StataMissingValue(offset+1+i)
+                self.assertTrue(val.string == expected_values[i])
+
+        # Test extremes for floats
+        val = StataMissingValue(struct.unpack('<f',b'\x00\x00\x00\x7f')[0])
+        self.assertTrue(val.string == '.')
+        val = StataMissingValue(struct.unpack('<f',b'\x00\xd0\x00\x7f')[0])
+        self.assertTrue(val.string == '.z')
+
+        # Test extremes for floats
+        val = StataMissingValue(struct.unpack('<d',b'\x00\x00\x00\x00\x00\x00\xe0\x7f')[0])
+        self.assertTrue(val.string == '.')
+        val = StataMissingValue(struct.unpack('<d',b'\x00\x00\x00\x00\x00\x1a\xe0\x7f')[0])
+        self.assertTrue(val.string == '.z')
+
+    def test_missing_value_conversion(self):
+        columns = ['int8_', 'int16_', 'int32_', 'float32_', 'float64_']
+        smv = StataMissingValue(101)
+        keys = [key for key in iterkeys(smv.MISSING_VALUES)]
+        keys.sort()
+        data = []
+        for i in range(27):
+            row = [StataMissingValue(keys[i+(j*27)]) for j in range(5)]
+            data.append(row)
+        expected = DataFrame(data,columns=columns)
+
+        parsed_113 = read_stata(self.dta17_113, convert_missing=True)
+        parsed_115 = read_stata(self.dta17_115, convert_missing=True)
+        parsed_117 = read_stata(self.dta17_117, convert_missing=True)
+
+        tm.assert_frame_equal(expected, parsed_113)
+        tm.assert_frame_equal(expected, parsed_115)
+        tm.assert_frame_equal(expected, parsed_117)
 
 if __name__ == '__main__':
     nose.runmodule(argv=[__file__, '-vvs', '-x', '--pdb', '--pdb-failure'],