Categorical fixups
State: closed

Some fixups for Categoricals.

Fixes: #3678 
- [x] Maybe change Series.cat after the discussion in https://github.com/pydata/pandas/issues/7207 ?
- [ ] remove Series.cat from tab completition if Series is not of dtype category
- [x] fix for the "FIXME" in unittests
- [x] Look at problems in docs (-> hdf support)
- [x] Fixup Comparison thingies...


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Categorical
description: Categorical Data Type

Comments:
see here: https://github.com/jreback/pandas/commit/1fc80e6c2c16b992e9f04ff9d4c66462d773e4b9

I don't buy the 2 NaN output case. very odd thing to do. A nan is a nan

This currently does not work:

``` python
cat = pd.Categorical([1,2,3, np.nan], levels=[1,2,3])
cat.levels = [1,2,3, np.nan]
cat[1] = np.nan
exp = np.array([0,3,2,-1])
self.assert_numpy_array_equal(cat.codes, exp)
```

I'm not sure if this here is a bug or actually as intended:

```
In[18]: idx = _ensure_index([1,2,3,np.nan]) 
In[19]: idx
Out[19]: Float64Index([1.0, 2.0, 3.0, nan], dtype='float64')
In[20]: idx.get_indexer([np.nan])
Out[20]: array([-1])
```

with a Floatindex u can't do that
u have to use hasnans and a check indexer 

u really can only have 1 nan if you intend to do any kind of indexing operation rlse how would u know where it goes?

it not a value but an indicator of a missing value 

so u can have more than 1 nan but then indexing (eg setitem/getitem) is impossible by value (but u can take by position)

Re "two times nan in describe": R has the same problem with "NA as a level vs NA as missing value": http://stat.ethz.ch/R-manual/R-devel/library/base/html/factor.html

> If NA is a level, the way to set a code to be missing (as opposed to the code of the missing level) is to use is.na on the left-hand-side of an assignment (as in is.na(f)[i] <- TRUE; indexing inside is.na does not work). Under those circumstances missing values are currently printed as <NA>, i.e., identical to entries of level NA.

R does the same thing, printing NA twice, but once indicating that it is a level `<NA>` and once that it is missing `NA's`:

``` R
> f <- factor(c(1,2,3,NA), exclude=F)
> f
[1] 1    2    3    <NA>
Levels: 1 2 3 <NA>
> is.na(f)[1] <- TRUE
> f
[1] <NA> 2    3    <NA>
Levels: 1 2 3 <NA>
> table(f)
f
   1    2    3 <NA> 
   0    1    1    1 
> summary(f)
   1    2    3 <NA> NA's 
   0    1    1    1    1 
```

@jreback Please have a look ... I added a workaround for `NaN in index` in 39a7531and implemented all the stuff which came up in the end of #7217.

So, if Travis passes and you are satisfied, I think this is ready to merge.

I want to fix the issue with nan first

@JanSchulz relating to your comment from #7207 
What more do you want here? I think this groups very nicely, don't show excess stuff.

```
In [1]: s = Series(list('aabbccd')).astype('category')

In [2]: s
Out[2]: 
0    a
1    a
2    b
3    b
4    c
5    c
6    d
dtype: category
Levels (4, object): [a < b < c < d]

In [3]: s.values.
s.values.T                     s.values.describe              s.values.from_array            s.values.max                   s.values.order                 s.values.reorder_levels        s.values.take_nd               
s.values.argsort               s.values.dtype                 s.values.from_codes            s.values.min                   s.values.ordered               s.values.shape                 s.values.to_dense              
s.values.codes                 s.values.equals                s.values.get_values            s.values.mode                  s.values.ravel                 s.values.sort                  s.values.unique                
s.values.copy                  s.values.fillna                s.values.levels                s.values.ndim                  s.values.remove_unused_levels  s.values.take                  s.values.view                  

In [3]: s.values.
```

[The follwoing is just about `Series.cat`, not abaout `Series.values` which should be the underlying data structure (either `numpy.array` or `pd.Categorical`)]

from my perspective, the above has way to many methods visible (and some relevant are not even visible): when you have a Series of type `category`, only a few methods should be available under `Series.cat`: `.levels` `.reorder_levels()`, `.remove_unused_levels()` and `.codes`. Everything else is an implementation detail and the functionality is exposed via the normal `Series` methods (`unique`, `min`, `max`,...).

Only declaring the above 4 methods/properties as API (and only exposing them) will mean that we can switch to a numpy category implementation (which I expect to happen) much more easily without breaking user code and it makes for a nicer tab completion...

numpy cat implementation is pie-in-the-sky. if it happens great. but don't hold your breath. And if it actually does, and we actually switch to it (a big IF, why would it really be any different/better?), so we make a change, so what.

what methods are relevant but not visible?

what about `.codes,fillna,copy,astype` these seem relevant?

yes they are available (and delegate from series). but as a user wouldn't I expect to see what I can do with the object?

`.codes` is, but `.fillna` is done via `Series.fillna()` and not via `Series.cat.fillna()` (same for `copy` and `astype`). If you know what you are doing, fine, just go for `Series.values` and do whatever you want with it, but you also have to watch out for bugs then. 

From my perspective (using pandas for some not-so-advanced data munging and statistics) I never noticed how numpy arrays are wrapped until I worked on Categorical, so I also never tried to work with `Series.values` and I also think it would be wrong to use methods on the numpy array directly because that would (as far as my understanding goes) lead to strange results like in the `Series(numpyarray)`-case where you can manipulate the `numpyarray` and the Series changes in some cases (depending on view or not).

I'm also a fan of only promoting to API as few methods/properties as possible to have some room for future developments, but that's probably not as problematic in python as it is in java.

ok, I am on-board with this then. this is actually really easy, 

just provide a new `Categorical.__dir__()` with the methods/properties you want (just a list of strings)

It was a bit non-trivial with datetime, because ipython search all of the base classes (which complicates things). Here base class is pretty trivial so its not an issue (and they ignore anything that starts with '_')).

and need a test as well (see what I did for the .dt tests)

But that has a problem too: Looking at the `Categorical` object is like looking at the numpy array and that object should expose every method. So `S.cat.<tab>` should give you a different list of methods than `s.values.<tab>` or `Categorical(...).<tab>`. So I still think it makes sense to replace `s.cat` with a similar object like it is now done with `s.dt` (which has the same objective to hide the internals of the implementation from the user).

oh, I c what you want to do, like a `CategoricalDelegate` or something that is like `DatetimelikeProperties`, delgating as needed (but this is only for a limited purpose).

ok then go for it, should be straightforward.

I current try my idea with the "generate the dt accessors" and if that works I will base the cat access on that.

I already did it. https://github.com/pydata/pandas/pull/7953/files

Just saw it. Damn, you are too fast! :-)

If `_add_accessors(cls)` get's changed to `_add_accessors(cls, names)`, this will work for cat as well. Then it can also be put into a more generic place.

Will push in a minute...

https://github.com/jreback/pandas/commit/a6828b22160b52d859da8f419634b5fc9d35bc2e will form a nice base

See here: https://github.com/JanSchulz/pandas/commit/3ad8faad2e3859345a2c06a6102f47548783e1ee

I think this makes it more generic (and allow CategoricalDelgate to inherit now) : https://github.com/jreback/pandas/commit/e0d8e8b5af99634f428fa3ae62e4933db9d3b159

Damn, you are always a few minutes faster (and have the better result...): just for the reference, here is mine: https://github.com/JanSchulz/pandas/commit/886313005fcacea9d1d2e7185e890d18c0469d0e

It would be nice of "Properties" would move to a more generic place, IMO it's not appropriate to import that from `tseries` in `categorical.py`...

I will base the category_fixup branch on your dt branch

create a `CategoricalProperties` which subclasses `PandasDelegate`. You don't need to mess with the `Properties` I created in tseries.datetimelike. Its too specific for what Categorical need (and also not really general enough).

something like Properties

ok, will do, thanks!

Damn:

``` python
In[50]: cat = pd.Series(pd.Categorical(["a","b","c"], levels=["c","b","a"]))
In[51]: cat2 = pd.Series(pd.Categorical(["b","b","b"], levels=["c","b","a"]))
In[52]: cat3 = pd.Series(pd.Categorical(["a","b","c"]))
In[56]: cat4 = pd.Series(pd.Categorical(["b","b","b"]))
# this is correct
In[58]: cat3 > cat4
Out[57]: 
0    False
1    False
2     True
dtype: bool
# This is wrong, should be True, False, False because level order is reversed!
In[57]: cat > cat2
Out[56]: 
0    False
1    False
2     True
dtype: bool
# This should raise?
In[59]: cat > cat3
Out[58]: 
0    False
1    False
2    False
dtype: bool
```

BTW, why does this not raise:

``` python
In[61]: pd.Series(list("abc")) < pd.Series([1,2,3])
Out[60]: 
0    False
1    False
2    False
dtype: bool
In[63]: pd.Series(list("abc")) > pd.Series([1,2,3])
Out[62]: 
0    True
1    True
2    True
dtype: bool
```

the last is a pythonism, 'a' > 1 is True. I suppose its an error.

the comparisons need a delegation wrapper.....coming right up!

https://github.com/jreback/pandas/commit/35afdc925f2106f1beeb8b00404253362b974699

fixes the boolean comparison issue

need tests for comparisons with a scalar

e.g. 

`cat3 > 5`

in theory your last example works (`cat > cat2`), as ordering is always for the lhs

fyi, ordered vs unordered should prob raise.

I came up with some other implementation:

``` python
def _cat_compare_op(op):
    def f(self, other):
        if isinstance(other, Categorical):
            # Two categoricals can only be be compared if the levels are the same
            if not ((self.levels == other.levels).all()):
                raise TypeError("Categoricals can only be compared if 'levels' are the same")
            if not (self.ordered == other.ordered):
                raise TypeError("Categoricals can only be compared if 'ordered' is the same")
            na_mask = (self._codes == -1) | (other._codes == -1)
            f = getattr(self._codes, op)
            ret = f(other._codes)
            if na_mask.any():
                # In other series, the leads to False, so do that here too
                ret[na_mask] = False
            return ret
        elif isinstance(other, (np.ndarray, com.ABCSeries)):
            values = np.asarray(self)
            f = getattr(values, op)
            return f(np.asarray(other))
        else:
            if other in self.levels:
                i = self.levels.get_loc(other)
                return getattr(self._codes, op)(i)
            else:
                return np.repeat(False, len(self))

    f.__name__ = op

    return f
```

On the other hand: I haven't found any comparison on Series, which was throwing an error, so according to that, we shouldn't raise at all? Different levels are undefined as `1 > object()`, but even worse, as for all other types the comparison outcome is predictable, but not for Categorical.

Aren't such comparisons raising on Py3? 

[Edit: yep:]

``` python
[ggplot_py33] C:\Windows\System32>python
Python 3.3.3 |Continuum Analytics, Inc.| (default, Dec 13 2013, 10:31:18) [MSC v
.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> 1 > object()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: int() > object()
>>> import pandas as pd
>>> pd.Series(list("abc")) < pd.Series([1,2,3])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\portabel\miniconda\envs\ggplot_py33\lib\site-packages\pandas\core\ops
.py", line 543, in wrapper
    return self._constructor(na_op(self.values, other.values),
  File "C:\portabel\miniconda\envs\ggplot_py33\lib\site-packages\pandas\core\ops
.py", line 522, in na_op
    result = lib.vec_compare(x, y.astype(np.object_), op)
  File "lib.pyx", line 661, in pandas.lib.vec_compare (pandas\lib.c:11253)
TypeError: unorderable types: str() < int()
```

your impl looks done too

hmm not sure about the series comparisons
(in py3)

I don't think they are tested as they r not well defined 

though for categoricals u can be very strict 

So, "being strict" (=the Py3 behaviour) would mean that we only compare "cat vs cat", if levels and ordered are the same and additionally "cat vs scalar". Everything else raises.

The Py2 behaviour would be the above but instead of raising, fallback on every other case to comparing values (that's at least predictable...)

Ah, and for all comparisons apart from `!=` and `==` this must also raise/fallback if the Categorical is unordered. 

Any comments? should the "fallback" stay on py2 or should the implementation be strict under all python versions?