Repository: statsmodels/statsmodels

Issue:
categorical does not work with Pandas DataFrames
```
In [9]: sm.categorical(pd.DataFrame({'a':[1,2,12], 'b':['a','b','a']}), col='a')
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-9-5966a3ee6951> in <module>()
----> 1 sm.categorical(pd.DataFrame({'a':[1,2,12], 'b':['a','b','a']}), col='a')

/usr/local/lib/python2.7/dist-packages/statsmodels-0.6.0-py2.7-linux-x86_64.egg/statsmodels/tools/tools.pyc in categorical(data, col, dictnames, drop)
    143     #TODO: add a NameValidator function
    144     # catch recarrays and structured arrays
--> 145     if data.dtype.names or data.__class__ is np.recarray:
    146         if not col and np.squeeze(data).ndim > 1:
    147             raise IndexError("col is None and the input array is not 1d")

/usr/local/lib/python2.7/dist-packages/pandas-0.13.0_285_gfcfaa7d-py2.7-linux-x86_64.egg/pandas/core/generic.pyc in __getattr__(self, name)
   1802                 return self[name]
   1803             raise AttributeError("'%s' object has no attribute '%s'" %
-> 1804                                  (type(self).__name__, name))
   1805 
   1806     def __setattr__(self, name, value):

AttributeError: 'DataFrame' object has no attribute 'dtype'
```

an alternative is to use `pd.get_dummies`:

```
pd.get_dummies(pd.DataFrame({'a':[1,2,12], 'b':['a','b','a']}).b )
```



Discussion:
This was written before pandas and patsy and was mainly to support numpy arrays and structured dtypes.

I wonder if this still has (m)any users left. It's still a nice function because it inserts the dummies back into the array.

I do like the fact that the variables are reinserted back into the array. In Pandas I have to do a concat with the results. Perhaps an explicit check for a `DataFrame` that recommends using the pandas `get_dummies`? 

I think, given that it still has a use case, we should extend it to handle DataFrames in the same way as recarrays.

Do you want to give it a try?

I probably can't get to this for a while.
On Jan 28, 2014 10:29 PM, "Josef Perktold" notifications@github.com wrote:

> I think, given that it still has a use case, we should extend it to handle
> DataFrames in the same way as recarrays.
> 
> Do you want to give it a try?
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/statsmodels/statsmodels/issues/1342#issuecomment-33553334
> .

I think any solution should use patsy and allow use of all the contrasts. If we even want to keep this (given that patsy handles this almost entirely), we basically need a wrapper around 1-2 lines of patsy calls to handle the merging to DataFrames or arrays. I'm not really sure I care about targeting recarrays anymore given that pandas exists now. 

Hi!
Any updates on fixing this or workarounds?
Thanks!

Edit: I guess it's related to https://github.com/statsmodels/statsmodels/issues/540

AFAIK, nobody worked on this.  And with the categorical support in pandas it might not have a large audience.

However, given that we still have the function, we should still add pandas support as in other similar functions. PR welcome.

> And with the categorical support in pandas it might not have a large audience.

Actually, my DataFrame here has 2 category columns. Given that, I guess something is wrong and that's why I'm reactivating this issue.

I have submitted a PR #3203 on this issue.  It should handle DataFrames as well as other data formats.

`categorical` still doesn't work with a DataFrame, which is a bit annoying for updating older code.
e.g. test_glm #4432
`categorical` is still nice when we quickly want to get an exog without going through the formulas or the full pandas categorical type.
In pandas.get_dummies, dtype attribute does not work. Please help!

pd.get_dummies(pd.Series(list('abc')), dtype=float)
TypeError: get_dummies() got an unexpected keyword argument 'dtype'

Resolving patch:
diff --git a/statsmodels/tools/tools.py b/statsmodels/tools/tools.py
--- a/statsmodels/tools/tools.py
+++ b/statsmodels/tools/tools.py
@@ -1,12 +1,13 @@
 '''
 Utility functions models code
 '''
-from statsmodels.compat.python import reduce, lzip, lmap, asstr2, range, long
 import numpy as np
 import numpy.lib.recfunctions as nprf
 import numpy.linalg as L
 import pandas as pd
 
+from statsmodels.compat.python import (reduce, lzip, lmap, asstr2, range,
+                                       long, string_types)
 from statsmodels.tools.data import _is_using_pandas, _is_recarray
 
 
@@ -17,7 +18,7 @@ def _make_dictnames(tmp_arr, offset=0):
     """
     col_map = {}
     for i, col_name in enumerate(tmp_arr):
-        col_map.update({i+offset : col_name})
+        col_map[i + offset] = col_name
     return col_map
 
 
@@ -60,20 +61,22 @@ def drop_missing(Y, X=None, axis=1):
 # ie., if you still have a string variable in your array you don't
 # want to cast it to float
 # TODO: add name validator (ie., bad names for datasets.grunfeld)
-def categorical(data, col=None, dictnames=False, drop=False, ):
+def categorical(data, col=None, dictnames=False, drop=False):
     '''
     Returns a dummy matrix given an array of categorical variables.
 
     Parameters
     ----------
     data : array
-        A structured array, recarray, or array.  This can be either
-        a 1d vector of the categorical variable or a 2d array with
+        A structured array, recarray, array, Series or DataFrame.  This can be
+        either a 1d vector of the categorical variable or a 2d array with
         the column specifying the categorical variable specified by the col
         argument.
-    col : 'string', int, or None
-        If data is a structured array or a recarray, `col` can be a string
-        that is the name of the column that contains the variable.  For all
+    col : {str, int, None}
+        If data is a DataFrame col must in a column of data. If data is a
+        Series, col must be either the name of the Series or None. If data is a
+        structured array or a recarray, `col` can be a string that is the name
+        of the column that contains the variable.  For all other
         arrays `col` can be an int that is the (zero-based) column index
         number.  `col` can only be None for a 1d array.  The default is None.
     dictnames : bool, optional
@@ -135,16 +138,43 @@ def categorical(data, col=None, dictnames=False, drop=False, ):
 
     >>> design2 = sm.tools.categorical(struct_ar, col='str_instr', drop=True)
     '''
+    # TODO: add a NameValidator function
     if isinstance(col, (list, tuple)):
-        try:
-            assert len(col) == 1
+        if len(col) == 1:
             col = col[0]
-        except:
+        else:
             raise ValueError("Can only convert one column at a time")
+    if (not isinstance(data, (pd.DataFrame, pd.Series)) and
+            not isinstance(col, (string_types, int)) and
+            col is not None):
+        raise TypeError('col must be a str, int or None')
 
-    # TODO: add a NameValidator function
+    # Pull out a Series from a DataFrame if provided
+    if isinstance(data, pd.DataFrame):
+        if col is None:
+            raise TypeError('col must be a str or int when using a DataFrame')
+        elif col not in data:
+            raise ValueError('Column \'{0}\' not found in data'.format(col))
+        data = data[col]
+        # Set col to None since we not have a Series
+        col = None
+
+    if isinstance(data, pd.Series):
+        if col is not None and data.name != col:
+            raise ValueError('data.name does not match col '
+                             '\'{0}\''.format(col))
+        data_cat = data.astype('category')
+        dummies = pd.get_dummies(data_cat)
+        col_map = {i: cat for i, cat in enumerate(data_cat.cat.categories) if
+                   cat in dummies}
+        if not drop:
+            dummies.columns = list(dummies.columns)
+            dummies = pd.concat([dummies, data], 1)
+        if dictnames:
+            return dummies, col_map
+        return dummies
     # catch recarrays and structured arrays
-    if data.dtype.names or data.__class__ is np.recarray:
+    elif data.dtype.names or data.__class__ is np.recarray:
         if not col and np.squeeze(data).ndim > 1:
             raise IndexError("col is None and the input array is not 1d")
         if isinstance(col, (int, long)):
@@ -196,11 +226,10 @@ def categorical(data, col=None, dictnames=False, drop=False, ):
                                   asrecarray=type(data) is np.recarray)
         return data
 
-    # handle ndarrays and catch array-like for an error
-    elif data.__class__ is np.ndarray or not isinstance(data, np.ndarray):
-        if not isinstance(data, np.ndarray):
-            raise NotImplementedError("Array-like objects are not supported")
-
+    # Catch array-like for an error
+    elif not isinstance(data, np.ndarray):
+        raise NotImplementedError("Array-like objects are not supported")
+    else:
         if isinstance(col, (int, long)):
             offset = data.shape[1]          # need error catching here?
             tmp_arr = np.unique(data[:, col])


Test patch:
diff --git a/statsmodels/tools/tests/test_tools.py b/statsmodels/tools/tests/test_tools.py
--- a/statsmodels/tools/tests/test_tools.py
+++ b/statsmodels/tools/tests/test_tools.py
@@ -2,6 +2,9 @@
 Test functions for models.tools
 """
 from statsmodels.compat.python import lrange, range
+
+import string
+
 import numpy as np
 from numpy.random import standard_normal
 from numpy.testing import (assert_equal, assert_array_equal,
@@ -15,6 +18,19 @@
 from statsmodels.tools.tools import pinv_extended
 
 
+@pytest.fixture('module')
+def string_var():
+    string_var = [string.ascii_lowercase[0:5],
+                  string.ascii_lowercase[5:10],
+                  string.ascii_lowercase[10:15],
+                  string.ascii_lowercase[15:20],
+                  string.ascii_lowercase[20:25]]
+    string_var *= 5
+    string_var = np.asarray(sorted(string_var))
+    series = pd.Series(string_var, name='string_var')
+    return series
+
+
 class TestTools(object):
 
     def test_add_constant_list(self):
@@ -211,8 +227,12 @@ def test_array2d(self):
 
     def test_array1d(self):
         des = tools.categorical(self.instr)
-        assert_array_equal(des[:,-5:], self.dummy)
-        assert_equal(des.shape[1],6)
+        assert_array_equal(des[:, -5:], self.dummy)
+        assert_equal(des.shape[1], 6)
+
+    def test_array1d_col_error(self):
+        with pytest.raises(TypeError, match='col must be a str, int or None'):
+            tools.categorical(self.instr, col={'a': 1})
 
     def test_array2d_drop(self):
         des = np.column_stack((self.des, self.instr, self.des))
@@ -232,6 +252,11 @@ def test_recarray2d(self):
         assert_array_equal(test_des, self.dummy)
         assert_equal(len(des.dtype.names), 9)
 
+    def test_recarray2d_error(self):
+        arr = np.c_[self.recdes, self.recdes]
+        with pytest.raises(IndexError, match='col is None and the input'):
+            tools.categorical(arr, col=None)
+
     def test_recarray2dint(self):
         des = tools.categorical(self.recdes, col=2)
         test_des = np.column_stack(([des[_] for _ in des.dtype.names[-5:]]))
@@ -606,6 +631,50 @@ def test_numpy(self):
         assert_array_equal(results[0], self.ndarray)
         assert_equal(results[1], None)
 
-        results = tools._ensure_2d(self.ndarray[:,0])
-        assert_array_equal(results[0], self.ndarray[:,[0]])
+        results = tools._ensure_2d(self.ndarray[:, 0])
+        assert_array_equal(results[0], self.ndarray[:, [0]])
         assert_equal(results[1], None)
+
+
+def test_categorical_pandas_errors(string_var):
+    with pytest.raises(ValueError, match='data.name does not match col'):
+        tools.categorical(string_var, 'unknown')
+
+    df = pd.DataFrame(string_var)
+    with pytest.raises(TypeError, match='col must be a str or int'):
+        tools.categorical(df, None)
+    with pytest.raises(ValueError, match='Column \'unknown\' not found in '
+                                         'data'):
+        tools.categorical(df, 'unknown')
+
+
+def test_categorical_series(string_var):
+    design = tools.categorical(string_var, drop=True)
+    dummies = pd.get_dummies(pd.Categorical(string_var))
+    assert_frame_equal(design, dummies)
+    design = tools.categorical(string_var, drop=False)
+    dummies.columns = list(dummies.columns)
+    assert_frame_equal(design.iloc[:, :5], dummies)
+    assert_series_equal(design.iloc[:, 5], string_var)
+    _, dictnames = tools.categorical(string_var, drop=False, dictnames=True)
+    for i, c in enumerate(pd.Categorical(string_var).categories):
+        assert i in dictnames
+        assert dictnames[i] == c
+
+
+def test_categorical_dataframe(string_var):
+    df = pd.DataFrame(string_var)
+    design = tools.categorical(df, 'string_var', drop=True)
+    dummies = pd.get_dummies(pd.Categorical(string_var))
+    assert_frame_equal(design, dummies)
+
+    df = pd.DataFrame({'apple': string_var, 'ban': string_var})
+    design = tools.categorical(df, 'apple', drop=True)
+    assert_frame_equal(design, dummies)
+
+
+def test_categorical_errors(string_var):
+    with pytest.raises(ValueError, match='Can only convert one column'):
+        tools.categorical(string_var, (0, 1))
+    with pytest.raises(ValueError, match='data.name does not match col'):
+        tools.categorical(string_var, {'a': 1})