API: deprecate setting of .ordered directly (GH9347, GH9190)
State: closed

closes #9347
closes #9190
closes #9148 

so this is option 4).

Default is now `ordered=False` (independently of whether you specify categories or not)
The ordering is still what factorize does (seen order, if no categories are specified).

Add `set_ordered` to change the `ordered` flag, which by default returns a new object.

```
In [1]: cat = pd.Categorical([0,1,2])
In [2]: cat
Out[2]: 
[0, 1, 2]
Categories (3, int64): [0, 1, 2]

In [3]: cat.ordered
Out[3]: False

In [4]: cat.ordered=True
pandas/core/categorical.py:452: FutureWarning: Setting 'ordered' directly is deprecated, use 'set_ordered'

In [5]: cat = cat.set_ordered(True)

In [6]: cat
Out[6]: 
[0, 1, 2]
Categories (3, int64): [0 < 1 < 2]

In [7]: cat.ordered
Out[7]: True

In [8]: cat = pd.Categorical([0,1,2],ordered=True)

In [9]: cat
Out[9]: 
[0, 1, 2]
Categories (3, int64): [0 < 1 < 2]

In [12]: cat.ordered
Out[12]: True
```

Further added the ability for astype to pass on keywords to the constructor

```
In [1]: Series(["a","b","c","a"]).astype('category',ordered=True)
Out[1]: 
0    a
1    b
2    c
3    a
dtype: category
Categories (3, object): [a < b < c]

In [2]: Series(["a","b","c","a"]).astype('category',categories=list('abcdef'),ordered=False)
Out[2]: 
0    a
1    b
2    c
3    a
dtype: category
Categories (6, object): [a, b, c, d, e, f]
```

Furthermore I put a warning as seemingly simple operations will fail because the user had an _implicity_ ordered Categorical.

```
In [1]: df = DataFrame({ 'A' : Series(list('aabc')).astype('category'), 'B' : np.arange(4) })

In [2]: df['A'].order()
TypeError: Categorical not ordered
you can use .set_ordered(True) to change the Categorical to an ordered one

In [3]: df.groupby('A').sum()
ValueError: cannot sort by an unordered Categorical in the grouper
you can set sort=False in the groupby expression or
make the categorical ordered by using .set_ordered(True)
```


Labels:
name: API Design
name: Categorical
description: Categorical Data Type

Comments:
One point to note.

supplying a `categories` DOES NOT make the categorical ordered. Ordering is a separate and distinct property.

```
In [3]: pd.Categorical(["a","a","b","b"], categories=["a","b","z"])
Out[3]: 
[a, a, b, b]
Categories (3, object): [a, b, z]

In [4]: pd.Categorical(["a","a","b","b"], categories=["a","b","z"], ordered=True)
Out[4]: 
[a, a, b, b]
Categories (3, object): [a < b < z]
```

cc @JanSchulz 
@jorisvandenbossche 
@shoyer 
cc @jseabold 

@JanSchulz  https://github.com/jreback/pandas/commit/0becba3086289fc452f304e7d628645f9f12a7c7

The original discussion in #9347, which the current impl.

```
In [14]: cat1 = Series(pd.Categorical(["a","c","b"], ordered=False))

In [16]: cat1
Out[16]: 
0    a
1    c
2    b
dtype: category
Categories (3, object): [a, c, b]

In [15]: cat2 = Series(pd.Categorical(["a","c","b"], ordered=True))

In [17]: cat2
Out[17]: 
0    a
1    c
2    b
dtype: category
Categories (3, object): [a < b < c]

In [19]: cat3 = Series(pd.Categorical(["a","c","b"], categories=['b','c','a'], ordered=True))

In [20]: cat3
Out[20]: 
0    a
1    c
2    b
dtype: category
Categories (3, object): [b < c < a]
```

@JanSchulz does this look right?

No, the first example is what #9347 is/was all about and which was kind of rejected by all "option 4" votes:

> 4) leave this alone, but make ordered=None -> ordered=False, e.g. if you don't specify an ordering then your ordering is lexographic but you are not considered ordered (this is the same as 1), but we don't default the order

ok, so I should leave the sorting alone, then just set the `ordered` flag. ok.

ok, if I change back to what was there before (e.g. this should be option 4)

```
In [4]: cat1
Out[4]: 
0    a
1    c
2    b
dtype: category
Categories (3, object): [a, b, c]

In [5]: cat2
Out[5]: 
0    a
1    c
2    b
dtype: category
Categories (3, object): [a < b < c]

In [6]: cat3
Out[6]: 
0    a
1    c
2    b
dtype: category
Categories (3, object): [b < c < a]
```

yep

for completeness

```
In [13]: cat1 = pd.Categorical(["a","c","b"], ordered=False)

In [14]: cat1
Out[14]: 
[a, c, b]
Categories (3, object): [a, b, c]

In [15]: cat2 = pd.Categorical(["a","c","b"], categories=['b','c','a'], ordered=False)

In [16]: cat2
Out[16]: 
[a, c, b]
Categories (3, object): [b, c, a]

In [17]: cat3 = pd.Categorical(["a","c","b"], ordered=True)

In [18]: cat3
Out[18]: 
[a, c, b]
Categories (3, object): [a < b < c]

In [19]: cat4 = pd.Categorical(["a","c","b"], categories=['b','c','a'], ordered=True)

In [20]: cat4
Out[20]: 
[a, c, b]
Categories (3, object): [b < c < a]
```

These change also makes this part of categorical.rst obsolete:

```
By passing a :class:`pandas.Categorical` object to a `Series` or assigning it to a `DataFrame`.
This is the only possibility to specify differently ordered categories (or no order at all) at
creation time and the only reason to use :class:`pandas.Categorical` directly:

.. ipython:: python

    raw_cat = Categorical(["a","b","c","a"], categories=["b","c","d"],
                             ordered=False)
    s = Series(raw_cat)
    s
    df = DataFrame({"A":["a","b","c","a"]})
    df["B"] = raw_cat
    df
```

```
You can also specify differently ordered categories or make the resulting data ordered, by 
passing these arguments to ``astype()``:

.. ipython:: python

    s = Series(["a","b","c","a"])
    s_cat = s.astype("category", categories=["b","c","d"], ordered=False)
    s_cat

.. note::
    It's also possible to pass a :class:`pandas.Categorical` object directly to a `Series` or assigning it to 
    a `DataFrame`. This is discouraged, as :class:`pandas.Categorical` is an implementation detail 
    which can change.
```

@JanSchulz ok, I think I addressed all of the points and fixed up the docs.

I didn't make `set_ordered` an API function (though you can certainly still call it). For the reason you mentioned, possibly user confusing, prob best left off of the public (and .cat API).

ok fixed up

I had some more comments, not sure if you saw them. Unfortunately I can't find them anymore due to the forced push :-/ 

[EDIT]found it: https://github.com/pydata/pandas/commit/42c9b81e1271711ec92fe8ce2d85e9d14190c276#commitcomment-10087039 

-> all problems fixed, only open are the minor suggestions s/Categorical/categorical data/ and 

```
The solution is to make 'A' orderable, e.g. by supplying `ordered=True` at creation time
(`...Series(list('aabc')).astype('category', ordered=True)...`) or by changing it afterwards 
via `df['A'] = df['A'].cat.as_ordered()`
```

both in whatsnew...

I changed what u mentioned 

prob need a once over after we merge this anyhow 

@jreback @JanSchulz I don't have time today to look at this in more detail, so if possible, can you keep it open a bit longer?

But already one thing: getting errors with a plain groupby does really not look nice in my opinion. Can't we just sort it in the 'order' of the categories? I know it is not 'ordered' in the sense that the first element is regarded as lower as the last etc, but still, you have the 'order' of the categories in the `categories` attribute.

So here is a possible compromise. automatically ordering basically violates the API guarantee that we shouldn't allow sorting (or sort-like operations) on unordered Categoricals.

But could simply do an implicit `.as_ordered()` inside the operation and warn (OrderingWarning) or something, that what they are doing is bad. I think we could do this for all current raises for non-ordering operations, e.g. in groupby, min, max, order, argosrt, searchsorted.

```
In [1]: df = DataFrame({ 'A' : Series(list('aabc')).astype('category'), 'B' : np.arange(4) })

In [2]: df['A'].order()
pandas/core/categorical.py:1012: OrderingWarning: Categorical is not ordered
sort will be in the order of the categories
you can use .as_ordered() to change the Categorical to an ordered one

  warn("Categorical is not ordered\n"
Out[2]: 
0    a
1    a
2    b
3    c
Name: A, dtype: category
Categories (3, object): [a, b, c]
```

latest commit changes all of the raises to `OrderingWarning`. So users can do what they were doing, now will get a warning.

```
In [1]: s = Series([1,2,3,1], dtype="category")

In [2]: s.min()
pandas/core/categorical.py:843: OrderingWarning: Categorical is not ordered
sort will be in the order of the categories
you can use .as_ordered() to change the Categorical to an ordered one

  OrderingWarning)
Out[2]: 1
```

I don't like the "warn + change instead of raise"-thing... The old thing (raise in groupby) is not so nice, but at least it's not changing a datatype. The way I think about categorical data is like an user defined integer: you have certain valid values (0...INTMAX), maybe an order (0<1, etc). If we had a "unorderable int", it would be bad if `min()` just issues a warning and than proceeds as if it is a normal int.

The only place I find that "not so nice" is the groupby and I think that is more of a problem of the default of the groupby. 

I see several ways to fix that, but all have drawbacks:
- Use this "warn + change" approach as in the before last commit (see above...).
- keep `ordered=True` as the implicit default -> makes the simple groupby case happy, but not so consistent with R and other expectations
- change to `sort=False` in groupby -> API change :-(
- change to `sort=Null` and only make that False (with a warning) if the input is a unordered Categorical
- simple raise an error for the "simple case", which lets the user know that he should always use explicit ordering information in the creation if a categorical: `astype("category", ordered=True)`

My preference would be `sort=Null` or `raise if unordered`...

I need to look into this more deeply, but my inclination would be to switch to `sort=None` in groupby, which could be interpreted as "sort if possible".  In practice, that would mean "sort everything except unordered categoricals." I would certainly _not_ change categorical dtypes in groupby operations if possible.... though in the current implementation (without `CategoricalIndex`), we basically do that anyways.

this commit [here](https://github.com/jreback/pandas/commit/3625dfe45115163b5433d556a9c456bae7f02110) implementes the `sort=None` option.

and then we have the groupby back to 'working' (its doing what it was doing before we changed the default), but you force `sort=True` then it WILL raise.

`.order()/min/max` will still raise

```
In [1]: df = DataFrame({ 'A' : Series(list('aabc')).astype('category'), 'B' : np.arange(4) })

In [2]: df.groupby('A').sum()
Out[2]: 
   B
A   
a  1
b  2
c  3

In [3]: df['A'].order()
TypeError: Categorical not ordered
you can use .as_ordered() to change the Categorical to an ordered one

```

@jorisvandenbossche @shoyer 

If you guys can have a look today would be gr8. Need this for the rc.

@JanSchulz made the list above.

I think `sort=None` is a reasonable approach to this.

I think there is another option that was not included in @JanSchulz his list for possible fixes:
- allow sorting of unordered categoricals. And then we don't have to change anything in the groupby. 
  This maybe sounds strange, but it is also what R does: it allows sorting, but not max/min eg:
  
  ```
  > cat <- as.factor(c('b', 'a', 'c', 'b'))
  > cat
  [1] b a c b
  Levels: a b c
  > sort(cat)
  [1] a b b c
  Levels: a b c
  > min(cat)
  Error in Summary.factor(c(2L, 1L, 3L, 2L), na.rm = FALSE) : 
  min not meaningful for factors
  > cat2 <- as.ordered(c('b', 'a', 'c', 'b'))
  > cat2
  [1] b a c b
  Levels: a < b < c
  > sort(cat2)
  [1] a b b c
  Levels: a < b < c
  > min(cat2)
  [1] a
  Levels: a < b < c
  ```
  
  If this is good or bad, I can't really say. But the question maybe is what then is the added value of using an ordered categorical above an unorderd.

But to put in the other way around: why not sorting an unordered categorical? It is a convenience to be able to do this (see eg the groupby case), and what harm does it do that this works without having to convert it to an ordered categorical? Just sorting the values in the order as it is in the `categories`.

@jreback In the case of using `sort=None` in groupby, what is the order of the result when grouping by an unordered categorical? The order of occurrence? 

@jorisvandenbossche `sort=None` -> `sort=True` for everything but unordered categoricals. 

I like your soln. It basically doesn't care on groupby's which is fine; they will be ordered as they are ordered now (and not custom ordered like ordered categoricals).

If you think about it we actually have 2 dtypes here (ordered & unorderded) categoricals (kind of like a sub-dtype).

closing in favor of #9622