CLN/DOC/TST: Categorical fixups (GH7768)
State: closed

replaces #7768, #8006 
closes #3678 

Categorical: preserve ints when NaN are present

  `Categorical([1, np.nan])` would end up with a single `1.` float level.
  This commit ensures that if `values` is a list of ints and contains np.nan,
  the float conversation does not take place.

Categorical: fix describe with np.nan

Categorical: ensure that one can assign np.nan

Categorical: fix assigning NaN if NaN in levels

API: change default Categorical.from_codes() to ordered=False

  In the normal constructor `ordered=True` is only assumed if the levels
  are given or the values are sortable (which is most of the cases), but
  in `from_codes(...)` we can't asssume this so the default should be
  `False`.

Categorical: add some links to Categorical in the other docs

Categorical: use s.values when calling private methods

  s.values is the underlying Categorical object, s.cat will be changed
  to only expose the API methods/properties.

Categorical: Change series.cat to only expose the API

Categorical: Fix order and na_position

Categorical: Fix comparison of Categoricals and Series|Categorical|np.array

  Categorical can only be comapred to another Categorical with the same levels
  and the same ordering or to a scalar value.

  If the Categorical has no order defined (cat.ordered == False), only equal
  (and not equal) are defined.

Categorical: Tab completition tests

Categorical: Fix for NA handling/float converting in levels

Categorical: make sure fillna gets both missing values and NA-levels

Categorical: add back labels as deprecated property

Categorical: Fix assigment to a Series(Categorical)) and remove Series.cat.codes

Categorical: declare most methods in Categorical NON-API


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: API Design
name: Categorical
description: Categorical Data Type

Comments:
FYI, their is no way easy IMHO to NOT have `s.cat` show up when tab completing on `s.?` . I mean you can do it (but we don't for `.dt/.str`), when its a TypeError. Mainly because you have to test it (for categorical its trivial but for dt its not because you sometimes have to convert it which can take a small but non-zero amount of time). and I don't think it distracts that much anyhow (their are lots of methods).

ok, so leave that as is.

Let's see what travis has to say about this, seems that it quite overworked these days..

btw, #7820 is not longer required right? because you put a `nan` level in an object array?

No, the workaround for #7820 is still in there: https://github.com/pydata/pandas/pull/8007/files#diff-d295fd3c83b52dcaabdf13701ead1a41R893

Floats + nan still end up as nan in the index:

```
In[18]: cat = pd.Categorical([1., 2., np.nan], levels=[1., 2., np.nan])
In[19]: cat.levels
Out[19]: Float64Index([1.0, 2.0, nan], dtype='float64')
```

yeah i have to get back to that 

Damn, and ints end up as float levels too:

```
In[20]: cat = pd.Categorical([1, 2, np.nan], levels=[1, 2, np.nan])
In[21]: cat.levels
Out[21]: Float64Index([1.0, 2.0, nan], dtype='float64')
```

Should that end up as object levels? That would mean that ints are preserved when you get back to the values and have _no_ NaN in the values (probably quite rare case) 

> That would mean that ints are preserved when you get back to the values and have no NaN in the values (probably quite rare case)

what are values here? is that the actual entries in the `Categorical`? or is it values of `cat.levels`?

I think you might have to make that `object` dtype (to have the ints and the float nan)

I refer to values like `values = np.asarray(cat)`

That would be without nans for cases like `np.asarray(cat[cat.notnull()])`. On the other hand statsmodels/ggplot will use the levels as factors or in bar plots, so this might be a good idea to preserve ints...

Because this is not anymore visible above: should the "Non-API" methods be merged with the API methods on Categorical or be removed?

```
To create compatibility with `pandas.Series` and `numpy` arrays, the following (non-API) methods
are also introduced and available when `Categorical` is used directly.

.. autosummary::
   :toctree: generated/

   Categorical.from_array
   Categorical.get_values
   Categorical.copy
   Categorical.dtype
   Categorical.ndim
   Categorical.sort
   Categorical.equals
   Categorical.unique
   Categorical.order
   Categorical.argsort
   Categorical.fillna
   Categorical.notnull
   Categorical.isnull
```

Not sure what's interesting for an outsider...? 

And another question: what should `fillna` do if there is an `NaN` in the levels? `isnull`takes both into account, `fillna` currently only fills "missing values", not "NA in level".

On the non-api methods: what was the reason they were introduced? Only for internal use? Or also for users? In the first case, make them private (or make them not show up in tab completion)? In the second case, they should be included, but with the guarantee they stay?

These methods on `Categorical` all got added to make Categorical work as a block, but potentially some of them are useful if you work with Categoricals directly (Who would want that? I've not found a usecase which would need to touch categoricals instead of `Series(Categorical(...))`)

CC @jseabold @njsmith @shoyer Maybe you guys have one? 

I don't really care what the data type for holding categorical data is, but
I can certainly see the advantage of having just one data type. And if so
then Series seems like a good choice for that.
On 12 Aug 2014 23:16, "Jan Schulz" notifications@github.com wrote:

> These methods on Categorical all got added to make Categorical work as a
> block, but potentially some of them are useful if you work with
> Categoricals directly (Who would want that? I've not found a usecase which
> would need to touch categoricals instead of Series(Categorical(...)))
> 
> CC @jseabold https://github.com/jseabold @njsmith
> https://github.com/njsmith @shoyer https://github.com/shoyer Maybe
> you guys have one?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/8007#issuecomment-51985421.

```
In[6]: cat = pd.Categorical([1, 2, np.nan], levels=[1, 2, np.nan])
In[7]: cat.levels
Out[7]: Index([1, 2, nan], dtype='object')
```

Not sure if the hack in `_validate_levels(...)` is too hackish... It's only triggered if you specify a level with NAs explicitly (`Categorical(<vals>,levels=[...,np.nan])`).

I will do the `fillna` tomorrow: I think that `fillna` should fill NA no matter if they are missing or levels...

My use case for working with `Categorical` directly would be so I can put it inside my own N-dimensional `Series`-like container (e.g., in a `DataArray` for my project [xray](http://xray.readthedocs.org/en/latest/index.html), which is basically "N-dimensional pandas").

I actually haven't gotten around to wrapping Categorical yet, but it should be pretty straightforward as long as it has the main ndarray-like (`shape`, `ndim`, `copy`, `dtype`, etc) attributes. I am OK with these not being guaranteed by the API docs, but they do need to be public for ndarray compatibility.

I have also made a bit of progress on N-dimensional categorical (https://github.com/pydata/pandas/pull/8012) but don't let that stop you from merging this PR first.

Seems that the numpy skip is still needed:

```
======================================================================
FAIL: test_comparisons (pandas.tests.test_categorical.TestCategorical)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/pandas/tests/test_categorical.py", line 319, in test_comparisons
    self.assertRaises(TypeError, lambda: a < cat)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/pandas/util/testing.py", line 168, in assertRaises
    assertRaises(_exception, _callable, *args, **kwargs)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/pandas/util/testing.py", line 1476, in assertRaises
    _callable(*args, **kwargs)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/pandas/util/testing.py", line 1540, in __exit__
    raise AssertionError("{0} not raised.".format(name))
AssertionError: TypeError not raised.
```

-> put back...

@jreback: seems that a numpy check is still needed :-( Tried with '1.7', but only ' 1.7.1' seems to work :-(

You may need to change the test. 

@JanSchulz 

check using:

```
if LooseVersion(np.__version__) > 1.7.1:
  # do the test
```

updated, rebased and squashed. Lets see what travis says... 

The only thing open is the decision what to include in api.rst

@JanSchulz I think adding a labels property (to simply return codes)
with. DeprecationWarning is good

the statsmodels guys can then change tests to do

`labels = getattr(categorical,'codes' ,categorical.labels)`

to provide version compat

@jseabold 

cc @josef-pkt

done in pandas

`FutureWarning` or `DeprecationWarning`? Categorical currently uses FutureWarning, but pandas has both

I think `DeprecationWarning`

Don't we use `FutureWarning`s for deprecation? Sounds not logical, I know, but `DepreceationWarning` is suppresed by default, so no user will see it (https://docs.python.org/2/library/warnings.html#warning-categories).

The rule that python and numpy use is: ask, what happens if people ignore
the warning and fail to update their code before the change is made? if
their code will fail noisely (raise an exception, e.g. b/c it's calling a
function that no longer exists), then use DeprecationWarning. If their code
will silently do something bad (e.g. return the wrong results), then use
FutureWarning.

Of course pandas can use whatever rule it prefers, but that's one approach.

On Thu, Aug 14, 2014 at 2:16 PM, Joris Van den Bossche <
notifications@github.com> wrote:

> Don't we use FutureWarnings for deprecation? Sounds not logical, I know,
> but DepreceationWarning is suppresed by default, so no user will see it.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/8007#issuecomment-52181287.

## 

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org