Repository: statsmodels/statsmodels

Issue:
tools.categorical truncates numerical categories
When statsmodels.tools.categorical creates the column names for the categories as follows:

(line 157)

```
if not tmp_arr.dtype.names:
        tmp_arr = np.squeeze(tmp_arr).astype('str').tolist()
elif tmp_arr.dtype.names:
        tmp_arr = np.squeeze(tmp_arr.tolist()).astype('str').tolist()
```

the astype('str')-call results in dtype "|S1" so it uses only the first character of the string. This causes later problems with duplicate names in situations, when multiple group names start with same characters.

A fix that seems to work is eg. tmp_arr = [str(item) for item in np.squeeze(tmp_arr)], however I'm hesitant in sending a patch as I'm not really familiar with the codebase.

An example that shows the failure:

```
from scikits.statsmodels.tools import categorical
import numpy as np

arr = np.rec.fromrecords([[10], [11]], names='group')
categorical(arr)
```



Discussion:
Hi, thanks for the report. What version of numpy? What version of statsmodels? I do not see this behavior. Can you also post the output you see?

```
np.version.full_version
# '1.7.0.dev-fd78546'
sm.version.full_version
# '0.5.0.dev-d61c8cb'
arr = np.rec.fromrecords([[10], [11]], names='group')
sm.tools.categorical(arr)
# rec.array([(10, 1.0, 0.0), (11, 0.0, 1.0)], 
#      dtype=[('group', '<i8'), ('group_10', '<f8'), ('group_11', '<f8')])
```

Ah, sorry, I thought this was universal.

I'm using statsmodels 0.3.1(-3ubuntu1) and numpy 1.6.1(-6ubuntu1), but I checked that the code hasn't changed in the git HEAD.

And the problem is in the fifth line

```
categorical(arr)
```

If I run it as a script, the output is:

```
Traceback (most recent call last):
  File "/tmp/breaksm.py", line 5, in <module>
    categorical(arr)
  File "/usr/lib/pymodules/python2.7/scikits/statsmodels/tools/tools.py", line 183, in categorical
    usemask=False, asrecarray=type(data) is np.recarray)
  File "/usr/lib/python2.7/dist-packages/numpy/lib/recfunctions.py", line 633, in append_fields
    fill_value=fill_value)
  File "/usr/lib/python2.7/dist-packages/numpy/lib/recfunctions.py", line 410, in merge_arrays
    newdtype = zip_descr(seqarrays, flatten=flatten)
  File "/usr/lib/python2.7/dist-packages/numpy/lib/recfunctions.py", line 185, in zip_descr
    return np.dtype(newdtype).descr
ValueError: two fields with the same name
```

Ah, ok. I see this in numpy 1.6.1. If you want to submit a pull request that would be very welcome.

I asked on the numpy list, currently numpy cannot detect the string length with astype('str').

So, the only alternative is list comprehension or using map, which might require list(map(str, ... )) to be python 3 compatible.

I have a test and patch on my laptop (not with me at the moment) using list comprehension, if you don't want to duplicate work.

Thanks, I can wait. Also I never looked at what the range of allowed structured arrays are.


Resolving patch:
diff --git a/statsmodels/tools/tools.py b/statsmodels/tools/tools.py
--- a/statsmodels/tools/tools.py
+++ b/statsmodels/tools/tools.py
@@ -132,8 +132,14 @@ def categorical(data, col=None, dictnames=False, drop=False, ):
 
     >>> design2 = sm.tools.categorical(struct_ar, col='str_instr', drop=True)
     '''
-
-#TODO: add a NameValidator function
+    if isinstance(col, (list, tuple)):
+        try:
+            assert len(col) == 1
+            col = col[0]
+        except:
+            raise ValueError("Can only convert one column at a time")
+
+    #TODO: add a NameValidator function
     # catch recarrays and structured arrays
     if data.dtype.names or data.__class__ is np.recarray:
         if not col and np.squeeze(data).ndim > 1:
@@ -154,10 +160,10 @@ def categorical(data, col=None, dictnames=False, drop=False, ):
         if _swap:
             tmp_dummy = np.squeeze(tmp_dummy).swapaxes(1,0)
 
-        if not tmp_arr.dtype.names:
-            tmp_arr = np.squeeze(tmp_arr).astype('str').tolist()
+        if not tmp_arr.dtype.names: # how do we get to this code path?
+            tmp_arr = [str(item) for item in np.squeeze(tmp_arr)]
         elif tmp_arr.dtype.names:
-            tmp_arr = np.squeeze(tmp_arr.tolist()).astype('str').tolist()
+            tmp_arr = [str(item) for item in np.squeeze(tmp_arr.tolist())]
 
 # prepend the varname and underscore, if col is numeric attribute lookup
 # is lost for recarrays...
@@ -172,8 +178,6 @@ def categorical(data, col=None, dictnames=False, drop=False, ):
 #TODO: test this for rec and structured arrays!!!
 
         if drop is True:
-            # if len(data.dtype) is 1 then we have a 1 column array
-#            if len(data.dtype) == 1:
             if len(data.dtype) <= 1:
                 if tmp_dummy.shape[0] < tmp_dummy.shape[1]:
                     tmp_dummy = np.squeeze(tmp_dummy).swapaxes(1,0)


Test patch:
diff --git a/statsmodels/tools/tests/test_tools.py b/statsmodels/tools/tests/test_tools.py
--- a/statsmodels/tools/tests/test_tools.py
+++ b/statsmodels/tools/tests/test_tools.py
@@ -306,6 +306,21 @@ def test_arraylike2d_drop(self):
     def test_arraylike1d_drop(self):
         pass
 
+def test_rec_issue302():
+    arr = np.rec.fromrecords([[10], [11]], names='group')
+    actual = tools.categorical(arr)
+    expected = np.rec.array([(10, 1.0, 0.0), (11, 0.0, 1.0)],
+        dtype=[('group', '<i8'), ('group_10', '<f8'), ('group_11', '<f8')])
+    assert_array_equal(actual, expected)
+
+def test_issue302():
+    arr = np.rec.fromrecords([[10, 12], [11, 13]], names=['group', 'whatever'])
+    actual = tools.categorical(arr, col=['group'])
+    expected = np.rec.array([(10, 12, 1.0, 0.0), (11, 13, 0.0, 1.0)],
+        dtype=[('group', '<i8'), ('whatever', '<i8'), ('group_10', '<f8'),
+               ('group_11', '<f8')])
+    assert_array_equal(actual, expected)
+
 def test_pandas_const_series():
     dta = longley.load_pandas()
     series = dta.exog['GNP']