Repository: pandas-dev/pandas

Issue:
ENH: support .astype('category') on DataFrame / aka co-factorization
xref to #10696, #8709

We don't allow an astype of a DataFrame to category directly

```
In [44]: df.astype('category')
NotImplementedError: > 1 ndim Categorical are not supported at this time
```

Instead you can apply the astype per-column.

```
In [35]: df = DataFrame({'A' : list('aabcda'), 'B' : list('bcdaae')})

In [36]: df
Out[36]: 
   A  B
0  a  b
1  a  c
2  b  d
3  c  a
4  d  a
5  a  e

In [37]: df.apply(lambda x: x.astype('category'))
Out[37]: 
   A  B
0  a  b
1  a  c
2  b  d
3  c  a
4  d  a
5  a  e

In [38]: df.apply(lambda x: x.astype('category')).B
Out[38]: 
0    b
1    c
2    d
3    a
4    a
5    e
Name: B, dtype: category
Categories (5, object): [a, b, c, d, e]

In [39]: df.apply(lambda x: x.astype('category')).A
Out[39]: 
0    a
1    a
2    b
3    c
4    d
5    a
Name: A, dtype: category
Categories (4, object): [a, b, c, d]
```

But if you have 'similar' cateogories then you would usually do this, automatically 
astyping with the same uniques.

```
In [41]: uniques = np.sort(pd.unique(df.values.ravel()))

In [42]: df.apply(lambda x: x.astype('category', categories=uniques)).A
Out[42]: 
0    a
1    a
2    b
3    c
4    d
5    a
Name: A, dtype: category
Categories (5, object): [a, b, c, d, e]

In [43]: df.apply(lambda x: x.astype('category', categories=uniques)).B
Out[43]: 
0    b
1    c
2    d
3    a
4    a
5    e
Name: B, dtype: category
Categories (5, object): [a, b, c, d, e]
```

This is failry straightforward to actually implement, and I think is a nice easy way of coding, w/o having to actually support 2D categoricals internally (and we are moving away from internal 2-d structures anyhow).



Discussion:
@TomAugspurger @sinhrks @shoyer @jorisvandenbossche 
cc @janschulz 

I'm not so sure what you are proposing here? That `df.astype('category' , ...)` would internally be mapped to `df.apply(lambda x: x.astype('category', ...))`?

For my usecase `df.astype(...)` is not necessary, I usually have different types of columns and doing a astype on the complete df would just destroy the df... But if others have the need...

My usecase is more:

```
lickert_columns= [...] # a few of the columns in my df
for col in lickert_columns:
    df[col] = df[col].astype("category", categories=lickert_scale, ordered=True)
```

well, you would oftentimes do this on a sub-set I think, e.g. `df[['A','B']].astype(...)`

the reason I bring this up is whether we should form the uniques FIRST before conversions, IOW

if `categories=None` is passed (which is the default), then we would create it explicity from ALL the passed values.

As opposed to individually create them per-column.

IMO constructing the categories from all uniques makes sense. 

[How would one merge these subset back into the original DF? dropping the old columns and merging the new ones back in? Sounds like a lot of work which ends up as long as the for loop?]

[example from SO](https://stackoverflow.com/questions/46761978/factorize-values-across-dataframe-columns-with-consistent-mappings)

Here is a complete example
```
In [9]: np.random.seed(1234)

In [10]: import string

In [11]: df = pd.DataFrame([np.random.choice(list(string.ascii_lowercase), 10) for i in range(5)])

In [12]: df
Out[12]: 
   0  1  2  3  4  5  6  7  8  9
0  p  t  g  v  m  u  y  z  p  r
1  x  j  l  m  w  y  q  f  q  j
2  w  p  s  q  m  f  c  g  d  h
3  l  a  j  l  q  d  c  t  m  b
4  l  t  l  r  o  t  h  k  l  o

In [14]: In [16]: b = pd.unique(df.values.T.reshape(-1, )) 
    ...: df.apply(lambda x: pd.Categorical(x, b).codes)
    ...: 
    ...: 
Out[14]: 
   0  1  2   3   4   5   6   7   8   9
0  0  4  7   9  10  14  15  20   0  12
1  1  5  3  10   2  15  11  16  11   5
2  2  0  8  11  10  16  18   7  17  19
3  3  6  5   3  11  17  18   4  10  22
4  3  4  3  12  13   4  19  21   3  13
```
Note this can actually be implemented in a more performant way via https://github.com/pandas-dev/pandas/blob/master/pandas/core/reshape/merge.py#L1453

Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -18,6 +18,7 @@
     is_number,
     is_integer, is_bool,
     is_bool_dtype,
+    is_categorical_dtype,
     is_numeric_dtype,
     is_datetime64_dtype,
     is_timedelta64_dtype,
@@ -4429,14 +4430,18 @@ def astype(self, dtype, copy=True, errors='raise', **kwargs):
                 if col_name not in self:
                     raise KeyError('Only a column name can be used for the '
                                    'key in a dtype mappings argument.')
-            from pandas import concat
             results = []
             for col_name, col in self.iteritems():
                 if col_name in dtype:
                     results.append(col.astype(dtype[col_name], copy=copy))
                 else:
                     results.append(results.append(col.copy() if copy else col))
-            return concat(results, axis=1, copy=False)
+            return pd.concat(results, axis=1, copy=False)
+
+        elif is_categorical_dtype(dtype) and self.ndim > 1:
+            # GH 18099: columnwise conversion to categorical
+            results = (self[col].astype(dtype, copy=copy) for col in self)
+            return pd.concat(results, axis=1, copy=False)
 
         # else, only a single dtype is given
         new_data = self._data.astype(dtype=dtype, copy=copy, errors=errors,


Test patch:
diff --git a/pandas/tests/frame/test_dtypes.py b/pandas/tests/frame/test_dtypes.py
--- a/pandas/tests/frame/test_dtypes.py
+++ b/pandas/tests/frame/test_dtypes.py
@@ -8,11 +8,11 @@
 
 import numpy as np
 from pandas import (DataFrame, Series, date_range, Timedelta, Timestamp,
-                    compat, concat, option_context)
+                    Categorical, compat, concat, option_context)
 from pandas.compat import u
 from pandas import _np_version_under1p14
 
-from pandas.core.dtypes.dtypes import DatetimeTZDtype
+from pandas.core.dtypes.dtypes import DatetimeTZDtype, CategoricalDtype
 from pandas.tests.frame.common import TestData
 from pandas.util.testing import (assert_series_equal,
                                  assert_frame_equal,
@@ -619,12 +619,21 @@ def test_astype_duplicate_col(self):
         expected = concat([a1_str, b, a2_str], axis=1)
         assert_frame_equal(result, expected)
 
-    @pytest.mark.parametrize('columns', [['x'], ['x', 'y'], ['x', 'y', 'z']])
-    def test_categorical_astype_ndim_raises(self, columns):
-        # GH 18004
-        msg = '> 1 ndim Categorical are not supported at this time'
-        with tm.assert_raises_regex(NotImplementedError, msg):
-            DataFrame(columns=columns).astype('category')
+    @pytest.mark.parametrize('dtype', [
+        'category',
+        CategoricalDtype(),
+        CategoricalDtype(ordered=True),
+        CategoricalDtype(ordered=False),
+        CategoricalDtype(categories=list('abcdef')),
+        CategoricalDtype(categories=list('edba'), ordered=False),
+        CategoricalDtype(categories=list('edcb'), ordered=True)], ids=repr)
+    def test_astype_categorical(self, dtype):
+        # GH 18099
+        d = {'A': list('abbc'), 'B': list('bccd'), 'C': list('cdde')}
+        df = DataFrame(d)
+        result = df.astype(dtype)
+        expected = DataFrame({k: Categorical(d[k], dtype=dtype) for k in d})
+        tm.assert_frame_equal(result, expected)
 
     @pytest.mark.parametrize("cls", [
         pd.api.types.CategoricalDtype,