Repository: pandas-dev/pandas

Issue:
API: preferred way to check if column/Series has Categorical dtype
From http://stackoverflow.com/questions/26924904/check-if-dataframe-column-is-categorical/26925340#26925340

What is the preferred way to check for categorical dtype?

I now answered:

```
In [42]: isinstance(df.cat_column.dtype, pd.core.common.CategoricalDtype)
Out[42]: True

In [43]: pd.core.common.is_categorical_dtype(df.cat_column)
Out[43]: True
```

But:
- this seems somewhat buried in pandas. Should there be a more top-level function to do this?
- we should add the preferred way to the categorical docs.



Discussion:
That was me asking the question. I originally started writing up the question because I was working on a PR for pandas, and while writing I discovered `is_categorical_dtype()`. Since I was working on internal pandas code that works for my current usage.

Having something that's not so deeply buried would be good though. I tried `df.col.dtype == 'category'` because I thought that was a pretty standard way of doing a quick type check. Unless there are strong reasons not to use this method, it should probably work the same for categoricals as it does for other types (e.g. `df.col.dtype == 'float64'`)

`df.col.dtype == 'category'` _does_ appear to work for me on pandas 0.15.1. As @onesandzeroes says, I think should be the preferred way to check for categorical types.

It does looks like `Categorical.dtype.__ne__` needs to be defined, though -- currently it's not, so Python defaults to something arbitrary.

@shoyer as you can see in my answer on SO, it does indeed work, but the problem is it raises for other dtypes instead of giving False, which is not very handy (and that is a numpy thing). 
With the example of SO:

```
In [86]: df
Out[86]:
  cat_column   x   y
0          c   0   0
1          d  10   4
2          f  20   8
3          a  30  12
4          b  40  16
5          e  50  20

In [87]: df.cat_column.dtype == 'category'
Out[87]: True

In [88]: df.x.dtype
Out[88]: dtype('float64')

In [89]: df.x.dtype == 'category'
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-89-aff611b16544> in <module>()
----> 1 df.x.dtype == 'category'

TypeError: data type "category" not understood
```

So exactly because that is not working (as you would expect: returning False), I think we should provide a common way to do this (or at least document this in the categorical docs what is the best way to do this)

Ah, I see. A reasonable solution might be to wrap the dtype in `str`, e.g., `str(df.x.dtype) == 'category'`.

cc @JanSchulz

```
In [10]: df = DataFrame({'A' : np.random.randn(5), 'B' : Series(list('aabbc')).astype('category')})

In [11]: df
Out[11]: 
          A  B
0 -0.064981  a
1  0.852717  a
2  0.693611  b
3  0.411486  b
4 -1.425537  c

In [12]: df.dtypes
Out[12]: 
A     float64
B    category
dtype: object

In [13]: df.dtypes == 'category'
Out[13]: 
A    False
B     True
dtype: bool

In [14]: df.select_dtypes(include=['category'])
Out[14]: 
   B
0  a
1  a
2  b
3  b
4  c

n [16]: pd.core.common.is_categorical_dtype(df.A.dtype)
Out[16]: False
```

So the preferred method of 'cheking' dtypes is simply to use `select_dtypes` or `[13]` works as well. Check a `np.dtype('float64')=='category'` blows up - I think maybe should create a bug report to have this fixed upstream. Not much we can do for this. Of course as @jtratner pointed out
``np.dtype('float64').dtype.name=='category' (will work correctly with numpy dtypes).

So I don't think it necessary to have the user actually use anything internal.

If pressed, `com.is_categorical_dtype(...)` would be ok

would not suggestion any mention/use of `com.CategoricalDType` (as an instance check) though - this is TOO internal.

To be honest this rarely should if ever come up. If the OP is trying to check individual dtypes for category then this is the wrong approach (and mostly certainly `.select_dtypes()` is the correct method.

So if someone wants to add a small doc section, ok.

Shouldn't this work?

```
if s.cat: 
   print("It's a categorical!")
```

@JanSchulz 

that will raise for non-cat types (as does `.dt`)

prevents user error. I think this is correct (these _should_ raise)

@JanSchulz no, because it gives a TypeError instead of False if it is not a categorical:

```
In [143]: df
Out[143]:
          A  B
0  0.299586  a
1  0.335853  a
2 -0.135405  b
3  1.247738  b
4 -0.232270  c

In [144]: bool(df.B.cat)
Out[144]: True

In [145]: bool(df.A.cat)
....
TypeError: Can only use .cat accessor with a 'category' dtype
```

I think is an issue that should be raised to numpy

```
In [9]: np.dtype('i8') == 'foo'
TypeError: data type "foo" not understood
```

closing on pandas side as this is sane on the pandas side

well, for all who care, I tried to push upstream. The user is now subject to random numpyisms that are really hard to fix downstream (impossible in this case).

The string coercion for dtype equality _is_ is an ugly API, and @njsmith is right that we are probably misusing the API here with `dtype == 'category'`. The category dtype should probably include all the metadata for the type of category (i.e., the categories and sorted-ness). This is how it currently works in dynd, for example.

So it think it would indeed be better to do this differently. Perhaps `dtype.kind == 'C'`? Or we could even make `pd.is_categorical` part of the API. Both options are compatible with numpy and not too terrible, IMO (the first has even fewer characters).

Either way, I think this should probably change evaluate to False (because the categories are different):

```
In [9]: pd.Categorical([1]).dtype == pd.Categorical([2]).dtype
Out[9]: True
```

If the the last example should work (I think that was discussed during the design of Categoricals), then we have to put the `categories` into the dtype.

@shoyer I disagree.

DyND does support categorical as a full-fledged datashape (see [here](http://nbviewer.ipython.org/github/ContinuumIO/dynd-python/blob/master/docs/notebooks/CategoricalType.ipynb). But using that impl is prob a ways away.

`CategoricalDtype` is basically a super-type for categories. You _could_ implement a concrete sub-class that allows an categories comparison, and maybe we should do that; it is nicer from a theoretical point of view.

But to be honest its a fair amount of complexity and not sure how much gain from that.

I am not sure anything is actually gained from explicty type checking with a `pd.is_categorical()`.
pandas is meant to be practical, and I think `s.dtype == 'category'` is useful and in the spirit of all other numpy dtype comparisons. 

@jreback  It's one thing for pandas to take a pragmatic approach instead of waiting for a full solution, but designing an API that is incompatible with that full solution seems like a bad idea. `s.dtype == 'category'` is quite practical but it probably will/should break when we switch to dynd. The dynd API is certainly more flexible, but IMO Nathaniel raised some good points that will likely apply there as well.

In any case, perhaps it was premature to close this issue? `s.dtype != 'category'` does _not_ currently work -- do we have a preferred alternative? I do understand that you are frustrated with the response from upstream, but even if numpy changed things tomorrow this would still be an issue.

(I do agree it's probably not worth refining `CategoricalDtype` given that it's pretty well hidden from the public API.)

@shoyer I'll buy that `s.dtype != 'category'` should work. Pls create a separate issue for that.

This is closed because pandas has does all it can to facilitate `s.dtype == 'category'` and provides many solns which don't need that.

Changing to use DyND type system will likely cause a bit of pain all around (good pain though). And will have to be revisited when DyND is more of a fixture.

If you have a better API idea which doesn't break anything, all ears.

> This is closed because pandas has does all it can to facilitate `s.dtype == 'category'` and provides many solns which don't need that.

The reason I think this should not be closed already is the reason I initially opened this issue: just to document this issue in the categorical.rst docs. 
And this isn't done yet, and we all know `s.dtype == 'category'` has it problems (and whether this are limitations on the numpy side or not is another dicussion but does not really matter for the _current situation and users who have this problem_). 

So I can do quick PR to include this in the docs, but therefore, just make a quick choice what I put in there:
- `pd.core.common.is_categorical_dtype(df['cat'])`
- `df['cat'].dtype.name == 'category'`
- ..

Or provide this `is_categorical_dtype` (or `is_categorical`) as a top-level function.

well, neither of those are preferred at all

```
df.dtypes == 'category'
df.select_dtypes(include=['category'])
```

are the most correct ways to do this
if you want to mention in a very small not that `df['cat'].dtype.name == 'category'` then ok with that
using `com.is_categorical_dtype(...)` is actually ok too, but that is just so completely different for the avg user then it should be advertised.

of course `s.dtype == 'category'` WILL work if its actually a categorical type.....

amzing that this works!

```
In [2]: Series([1,2,3],dtype='int32').dtype=='i123'
Out[2]: True
```

going to bump this


Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -78,10 +78,11 @@ class NDFrame(PandasObject):
     copy : boolean, default False
     """
     _internal_names = ['_data', '_cacher', '_item_cache', '_cache',
-                       'is_copy', 'dt', 'cat', 'str', '_subtyp', '_index',
+                       'is_copy', '_subtyp', '_index',
                        '_default_kind', '_default_fill_value',
                        '__array_struct__','__array_interface__']
     _internal_names_set = set(_internal_names)
+    _accessors = frozenset([])
     _metadata = []
     is_copy = None
 
@@ -1957,9 +1958,9 @@ def __getattr__(self, name):
         # Note: obj.x will always call obj.__getattribute__('x') prior to
         # calling obj.__getattr__('x').
 
-        if name in self._internal_names_set:
-            return object.__getattribute__(self, name)
-        elif name in self._metadata:
+        if (name in self._internal_names_set
+                or name in self._metadata
+                or name in self._accessors):
             return object.__getattribute__(self, name)
         else:
             if name in self._info_axis:
diff --git a/pandas/core/series.py b/pandas/core/series.py
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -113,6 +113,7 @@ class Series(base.IndexOpsMixin, generic.NDFrame):
         Copy input data
     """
     _metadata = ['name']
+    _accessors = frozenset(['dt', 'cat', 'str'])
     _allow_index_ops = True
 
     def __init__(self, data=None, index=None, dtype=None, name=None,


Test patch:
diff --git a/pandas/tests/test_categorical.py b/pandas/tests/test_categorical.py
--- a/pandas/tests/test_categorical.py
+++ b/pandas/tests/test_categorical.py
@@ -1354,13 +1354,13 @@ def test_sequence_like(self):
     def test_series_delegations(self):
 
         # invalid accessor
-        self.assertRaises(TypeError, lambda : Series([1,2,3]).cat)
-        tm.assertRaisesRegexp(TypeError,
+        self.assertRaises(AttributeError, lambda : Series([1,2,3]).cat)
+        tm.assertRaisesRegexp(AttributeError,
                               r"Can only use .cat accessor with a 'category' dtype",
                               lambda : Series([1,2,3]).cat)
-        self.assertRaises(TypeError, lambda : Series(['a','b','c']).cat)
-        self.assertRaises(TypeError, lambda : Series(np.arange(5.)).cat)
-        self.assertRaises(TypeError, lambda : Series([Timestamp('20130101')]).cat)
+        self.assertRaises(AttributeError, lambda : Series(['a','b','c']).cat)
+        self.assertRaises(AttributeError, lambda : Series(np.arange(5.)).cat)
+        self.assertRaises(AttributeError, lambda : Series([Timestamp('20130101')]).cat)
 
         # Series should delegate calls to '.categories', '.codes', '.ordered' and the
         # methods '.set_categories()' 'drop_unused_categories()' to the categorical