Repository: pandas-dev/pandas

Issue:
to_stata always stores strings as str244
Copied from #7858:

I am still getting this bug (pandas 0.15.1, numpy 1.9.1, Stata 13.1 on Windows 7). The written DTA file still stores the strings as str244 even though the strings themselves have length 1. This also means that when they're read back into pandas the DataFrame looks the same. I think the only way to detect it from within Pandas is to look at the size of the DTA file itself.

``` python
df = pd.DataFrame(['a', 'b', 'c'], columns=['alpha'])
df.to_stata('test.dta')
df2 = pd.read_stata('test.dta')
assert (df['alpha'] == df2['alpha']).min()
```

But when loading in Stata:

``` stata
. use test
. describe

Contains data from D:\data\Pollution\test.dta
  obs:             3                          
 vars:             2                          02 Dec 2014 10:30
 size:           744                          
--------------------------------------------------------------------------------------------
              storage   display    value
variable name   type    format     label      variable label
--------------------------------------------------------------------------------------------
index           long    %12.0g                
alpha           str244  %1s                   
--------------------------------------------------------------------------------------------
Sorted by:  

. compress
  index was long now byte
  alpha was str244 now str1
  (738 bytes saved)

. describe

Contains data from D:\data\Pollution\test.dta
  obs:             3                          
 vars:             2                          02 Dec 2014 10:30
 size:             6                          
--------------------------------------------------------------------------------------------
              storage   display    value
variable name   type    format     label      variable label
--------------------------------------------------------------------------------------------
index           byte    %12.0g                
alpha           str1    %9s                   
--------------------------------------------------------------------------------------------
Sorted by:  

```

I don't know how critical this is since there are workarounds. You can either use `compress; save, replace` in Stata after every use of pandas or, if the 244 problem makes the DTA exceed your memory limit (which causes quite the system error lightshow as I just experienced), you could pass it through a CSV first.

It's just a matter of convenience.



Discussion:
Solved in reader in #7858, but not in writer.


Resolving patch:
diff --git a/pandas/io/stata.py b/pandas/io/stata.py
--- a/pandas/io/stata.py
+++ b/pandas/io/stata.py
@@ -1409,7 +1409,7 @@ def _maybe_convert_to_int_keys(convert_dates, varlist):
     return new_dict
 
 
-def _dtype_to_stata_type(dtype):
+def _dtype_to_stata_type(dtype, column):
     """
     Converts dtype types to stata types. Returns the byte of the given ordinal.
     See TYPE_MAP and comments for an explanation. This is also explained in
@@ -1425,13 +1425,14 @@ def _dtype_to_stata_type(dtype):
     If there are dates to convert, then dtype will already have the correct
     type inserted.
     """
-    #TODO: expand to handle datetime to integer conversion
+    # TODO: expand to handle datetime to integer conversion
     if dtype.type == np.string_:
         return chr(dtype.itemsize)
     elif dtype.type == np.object_:  # try to coerce it to the biggest string
                                     # not memory efficient, what else could we
                                     # do?
-        return chr(244)
+        itemsize = max_len_string_array(column.values)
+        return chr(max(itemsize, 1))
     elif dtype == np.float64:
         return chr(255)
     elif dtype == np.float32:
@@ -1461,6 +1462,7 @@ def _dtype_to_default_stata_fmt(dtype, column):
     int16   -> "%8.0g"
     int8    -> "%8.0g"
     """
+    # TODO: Refactor to combine type with format
     # TODO: expand this to handle a default datetime format?
     if dtype.type == np.object_:
         inferred_dtype = infer_dtype(column.dropna())
@@ -1470,8 +1472,7 @@ def _dtype_to_default_stata_fmt(dtype, column):
         itemsize = max_len_string_array(column.values)
         if itemsize > 244:
             raise ValueError(excessive_string_length_error % column.name)
-
-        return "%" + str(itemsize) + "s"
+        return "%" + str(max(itemsize, 1)) + "s"
     elif dtype == np.float64:
         return "%10.0g"
     elif dtype == np.float32:
@@ -1718,10 +1719,11 @@ def _prepare_pandas(self, data):
                     self._convert_dates[key]
                 )
                 dtypes[key] = np.dtype(new_type)
-        self.typlist = [_dtype_to_stata_type(dt) for dt in dtypes]
+        self.typlist = []
         self.fmtlist = []
         for col, dtype in dtypes.iteritems():
             self.fmtlist.append(_dtype_to_default_stata_fmt(dtype, data[col]))
+            self.typlist.append(_dtype_to_stata_type(dtype, data[col]))
 
         # set the given format for the datetime cols
         if self._convert_dates is not None:


Test patch:
diff --git a/pandas/io/tests/test_stata.py b/pandas/io/tests/test_stata.py
--- a/pandas/io/tests/test_stata.py
+++ b/pandas/io/tests/test_stata.py
@@ -593,10 +593,12 @@ def test_minimal_size_col(self):
         with tm.ensure_clean() as path:
             original.to_stata(path, write_index=False)
             sr = StataReader(path)
+            typlist = sr.typlist
             variables = sr.varlist
             formats = sr.fmtlist
-            for variable, fmt in zip(variables, formats):
+            for variable, fmt, typ in zip(variables, formats, typlist):
                 self.assertTrue(int(variable[1:]) == int(fmt[1:-1]))
+                self.assertTrue(int(variable[1:]) == typ)
 
     def test_excessively_long_string(self):
         str_lens = (1, 244, 500)
@@ -850,7 +852,6 @@ def test_categorical_order(self):
         # Check identity of codes
         for col in expected:
             if is_categorical_dtype(expected[col]):
-                print(col)
                 tm.assert_series_equal(expected[col].cat.codes,
                                        parsed_115[col].cat.codes)
                 tm.assert_index_equal(expected[col].cat.categories,