Repository: pandas-dev/pandas

Issue:
to_stata results in unnecessarily long strings
I'm writing some simple data, including a string of len()=5 using to_stata.
In Stata, I then have trouble with it due to its length. codebook shows that my variable as been encoded as a string of length 244:
of type  "string (str244), but longest is str5"

If tthe longest string is length 5, so should be the output variable type in Stata.



Discussion:
cc @bashtage

@cpbl can you post a self-contained example

 Here's the code:

```
#!/usr/bin/python

fout=open('tmp.tsv','wt')
fout.write("""A B   C
dummy   32.470986000    -86.487033000
01001   32.470986000    -86.487033000
01001   32.466056000    -86.472934000
01001   32.474035000    -86.457764000
01001   32.466794000    -86.445569000
01001   32.454933000    -86.425025000
01001   32.439950000    -86.478442000
01001   32.438025000    -86.443068000
""")
fout.close()
import pandas as pd
df=pd.read_table('tmp.tsv')
df.to_stata('tmp.dta')
fout=open('tmp.do','wt')
fout.write("""
log using tmp.log, text
use tmp
codebook
log close
""")
```

And an excerpt from the output in stata ("tmp.log")

```
------------------------------------------------------------------------------------------
A                                                                              (unlabeled)
------------------------------------------------------------------------------------------

                  type:  string (str244), but longest is str5

         unique values:  2                        missing "":  0/8

            tabulation:  Freq.  Value
                             7  "01001"
                             1  "dummy"

```

can you put the stata version up as well

This seems to be something related to how pandas stores strings now.

Consider this simple example

``` python
>>> data = pd.DataFrame(np.array(['a','b','c']))
>>> data.dtypes[0].type
numpy.object_
```

`StataWriter` assumes all `numpy.object_` are 244 character strings since there isn't an easy way to tell its size.  Note that htis is different from how numpy stores the same array

``` python
>>>arr = np.array(['a','b','c'])
>>> arr.dtype
dtype('S1')
```

which is a string with a fixed size.

Maybe @jreback can comment on how to force pandas to store fixed width strings as `Sxx` where `xx` is the width?

@bashtage 

in pandas you _cannot_ store anything string-like that is not `object` dtype. iow all strings are variable-width.

However, you _could_ on storage, convert to a numpy str of fixed width. (This is done in pytables as the underlying storage mechanism ONLY stores fixed-width). 

If you are writing in one go (e.g. no chunking), then this is pretty easy to do (e.g. `lib.max_len_string_array(....)`). With chunking pytables has a `min_itemsize` to make sure that the table is a minimum width.

If stata supports a fixed-width character size you could store like that.

@jreback said "can you put the stata version up as well". Was that to me? 
stata version is 12.1 for Unix (Linux 64-bit x86-64
Revision 23 Jan 2014

If you mean to include the stata file... I don't see how to attach files.
And this discussion is already way over my head.
Thanks.
c

@cpbl yes that was to you as stata has many different formats for the files, some newer than others.

Dont' need a stata file - this issue is how the string dimension is determined in `StataWriter` which uses only the `DataFrams.dtypes` and not the actual data.  Can probably change, although this isn't so much a bug as an omitted optimization.

@jreback  So did you want a file or the version? Still unsure. File is here (today): http://barringtonleigh.net/tmp.dta

@bashtage Not a bug: Oh, maybe. I was interpreting my resultant inability to merge as a result of the strlen in the dta, but I guess I had a more fundamental error.

@bashtage yep. I'll mark as an enhancement. To add fixed-width strings in addition/in lieu of variable-wdith. You make the call if you should simply switch this totally / or make it optional. I don't now how this would affect the stata side (I don't think a round-trip would be affected on the pandas side at all).

Though dealing with unicode could be a bit of an issue (you need to astype to `U123` rather than `S123`)

I don't think Stata handles unicode anyway - just different encodings of 1 byte strings (e.g. Latin1)

@cpbl You probably need to tell stata to treat it is a len 5 string, and then you can do things in Stata - or get Stata 13 which supports variable length strings (although `StataWriter` does not)

@jreback It is probably useful, so will see what I can work in.

@bashtage gr8!

thinking further I think you might want to make an option to turn this on (could be default or not, that's a separate question). maybe `strings='fixed|variable'`

I wouldn't want to use the expression `variable` since this has a very different meaning in Stata 13 - it would be more correct to describe the difference as `optimal` (adaptive to string being written, writing the minimum).

hmm, so stata has notions of `fixed_width` and `variable_width` ? or more possibliites?

side issue. since the `Categorical` type has been in master a bit. I am not sure this is mapped correctly to stata (it is tested, but not sure).


Resolving patch:
diff --git a/pandas/io/stata.py b/pandas/io/stata.py
--- a/pandas/io/stata.py
+++ b/pandas/io/stata.py
@@ -23,6 +23,7 @@
 from pandas.compat import long, lrange, lmap, lzip, text_type, string_types
 from pandas import isnull
 from pandas.io.common import get_filepath_or_buffer
+from pandas.lib import max_len_string_array, is_string_array
 from pandas.tslib import NaT
 
 def read_stata(filepath_or_buffer, convert_dates=True,
@@ -181,6 +182,11 @@ def _datetime_to_stata_elapsed(date, fmt):
         raise ValueError("fmt %s not understood" % fmt)
 
 
+excessive_string_length_error = """
+Fixed width strings in Stata .dta files are limited to 244 (or fewer) characters.
+Column '%s' does not satisfy this restriction.
+"""
+
 class PossiblePrecisionLoss(Warning):
     pass
 
@@ -1040,12 +1046,14 @@ def _dtype_to_stata_type(dtype):
                          "Please report an error to the developers." % dtype)
 
 
-def _dtype_to_default_stata_fmt(dtype):
+def _dtype_to_default_stata_fmt(dtype, column):
     """
     Maps numpy dtype to stata's default format for this type. Not terribly
     important since users can change this in Stata. Semantics are
 
     string  -> "%DDs" where DD is the length of the string
+    object  -> "%DDs" where DD is the length of the string, if a string, or 244
+                for anything that cannot be converted to a string.
     float64 -> "%10.0g"
     float32 -> "%9.0g"
     int64   -> "%9.0g"
@@ -1055,9 +1063,21 @@ def _dtype_to_default_stata_fmt(dtype):
     """
     #TODO: expand this to handle a default datetime format?
     if dtype.type == np.string_:
+        if max_len_string_array(column.values) > 244:
+            raise ValueError(excessive_string_length_error % column.name)
+
         return "%" + str(dtype.itemsize) + "s"
     elif dtype.type == np.object_:
-        return "%244s"
+        try:
+            # Try to use optimal size if available
+            itemsize = max_len_string_array(column.values)
+        except:
+            # Default size
+            itemsize = 244
+        if itemsize > 244:
+            raise ValueError(excessive_string_length_error % column.name)
+
+        return "%" + str(itemsize) + "s"
     elif dtype == np.float64:
         return "%10.0g"
     elif dtype == np.float32:
@@ -1264,7 +1284,9 @@ def __iter__(self):
                 )
                 dtypes[key] = np.dtype(new_type)
         self.typlist = [_dtype_to_stata_type(dt) for dt in dtypes]
-        self.fmtlist = [_dtype_to_default_stata_fmt(dt) for dt in dtypes]
+        self.fmtlist = []
+        for col, dtype in dtypes.iteritems():
+            self.fmtlist.append(_dtype_to_default_stata_fmt(dtype, data[col]))
         # set the given format for the datetime cols
         if self._convert_dates is not None:
             for key in self._convert_dates:


Test patch:
diff --git a/pandas/io/tests/test_stata.py b/pandas/io/tests/test_stata.py
--- a/pandas/io/tests/test_stata.py
+++ b/pandas/io/tests/test_stata.py
@@ -565,6 +565,30 @@ def test_variable_labels(self):
             self.assertTrue(k in keys)
             self.assertTrue(v in labels)
 
+    def test_minimal_size_col(self):
+        str_lens = (1, 100, 244)
+        s = {}
+        for str_len in str_lens:
+            s['s' + str(str_len)] = Series(['a' * str_len, 'b' * str_len, 'c' * str_len])
+        original = DataFrame(s)
+        with tm.ensure_clean() as path:
+            original.to_stata(path, write_index=False)
+            sr = StataReader(path)
+            variables = sr.varlist
+            formats = sr.fmtlist
+            for variable, fmt in zip(variables, formats):
+                self.assertTrue(int(variable[1:]) == int(fmt[1:-1]))
+
+    def test_excessively_long_string(self):
+        str_lens = (1, 244, 500)
+        s = {}
+        for str_len in str_lens:
+            s['s' + str(str_len)] = Series(['a' * str_len, 'b' * str_len, 'c' * str_len])
+        original = DataFrame(s)
+        with tm.assertRaises(ValueError):
+            with tm.ensure_clean() as path:
+                original.to_stata(path)
+
 
 if __name__ == '__main__':
     nose.runmodule(argv=[__file__, '-vvs', '-x', '--pdb', '--pdb-failure'],