Repository: pandas-dev/pandas

Issue:
BUG/API: Unordered Categorical should ignore order in comparisons?
I think that when comparing two unordered categorical-dtyped series with categories which differ only by ordered should compare equal

#### Code Sample, a copy-pastable example if possible

```python
In [1]: import pandas as pd
c
In [2]: c1 = pd.Series(pd.Categorical(['a', 'b']))

In [3]: c2 = pd.Series(pd.Categorical(['a', 'b'], categories=['b', 'a']))

In [4]: c1
Out[4]:
0    a
1    b
dtype: category
Categories (2, object): [a, b]

In [5]: c2
Out[5]:
0    a
1    b
dtype: category
Categories (2, object): [b, a]

In [6]: c1 == c2
```

```pytb
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-d8d43a43a02a> in <module>()
----> 1 c1 == c2

/Users/taugspurger/Envs/statsmodels-dev/lib/python3.6/site-packages/pandas/core/ops.py in wrapper(self, other, axis)
    811                 msg = 'Can only compare identically-labeled Series objects'
    812                 raise ValueError(msg)
--> 813             return self._constructor(na_op(self.values, other.values),
    814                                      index=self.index, name=name)
    815         elif isinstance(other, pd.DataFrame):  # pragma: no cover

/Users/taugspurger/Envs/statsmodels-dev/lib/python3.6/site-packages/pandas/core/ops.py in na_op(x, y)
    752         # in either operand
    753         if is_categorical_dtype(x):
--> 754             return op(x, y)
    755         elif is_categorical_dtype(y) and not isscalar(y):
    756             return op(y, x)

/Users/taugspurger/Envs/statsmodels-dev/lib/python3.6/site-packages/pandas/core/categorical.py in f(self, other)
     55             if ((len(self.categories) != len(other.categories)) or
     56                     not ((self.categories == other.categories).all())):
---> 57                 raise TypeError("Categoricals can only be compared if "
     58                                 "'categories' are the same")
     59             if not (self.ordered == other.ordered):

TypeError: Categoricals can only be compared if 'categories' are the same
```

#### Expected Output

I think this should return True. Unordered categories shouldn't care about the order :)

cc @janschulz thoughts?


Discussion:
yes this seems reasonable. creating a ``Categorical`` w/o cateogories creates categories in the order they are found, but that's an implementation detail.

if you change this how much breakage on tests? (IOW this is going to affect ``is_dtype_equal`` as well)
Once upon a time, categoricals were modeled after factor in R and there, levels (=categories) were sorted:

```
> levels(factor(c('b','a', 'c')) )
[1] "a" "b" "c"
```

In that case, both cases would result in the same categories.

I think the sorting was removed ("categories in order of apearance") and this was overlooked? [edit] None of the commits show this (as far as my github blame skill reach...), so it either happend  in one of the early discussions or in my imagination and this was simpy overlooked in the implementation.[/] 
>  creating a Categorical w/o cateogories creates categories in the order they are found, but that's an implementation detail.

@jreback no, when no categories are specified, the found values are sorted:

In [1]: pd.Categorical(['b', 'a'])
Out[1]: 
[b, a]
Categories (2, object): [a, b]

> None of the commits show this (as far as my github blame skill reach...), so it either happend in one of the early discussions or in my imagination and this was simpy overlooked in the implementation

It just didn't happen AFAIK :-)
right, forgot we *sort* :>

so the question is why are we differentiating between passed categories and categories created from direct factorizing in the ``ordered=False`` case?

For ``ordered=False``
 - I think leaving the passed categories as is is fine (e.g. no sorting)
- a bit indifferent to whether we *should* sort factorized categories (a) (I know R does), or show them in order of appearance (b)

But these *all* should be equivalent
i
I'll play around with it when I'm doing the CategoricalDtype stuff. For ease of implementation, we'll have to do something. Having an "ordering" function that deterministically orders a given set of categories is necessary so that the codes match up.

```
In [8]: pd.Categorical(['b', 'a', 1])
Out[8]:
[b, a, 1]
Categories (3, object): [1, a, b]

In [9]: pd.Categorical(['b', 'a', 1, pd.Timestamp('2017')])
Out[9]:
[b, a, 1, 2017-01-01 00:00:00]
Categories (4, object): [b, a, 1, 2017-01-01 00:00:00]
```
If we are comparing to R, they do not care about order of categories when they are not ordered:

```
> f1 <- factor(c('b','a', 'c'))
> f2 <- factor(c('b','a', 'c'), levels = c('b', 'a', 'c'))
> f1
[1] b a c
Levels: a b c
> f2
[1] b a c
Levels: b a c
> f1 == f2
[1] TRUE TRUE TRUE
```

but ... they also don't care when they *are* ordered:

```
> f1 <- factor(c('b','a', 'c'), ordered = T)
> f1
[1] b a c
Levels: a < b < c
> f2 <- factor(c('b','a', 'c'), levels = c('b', 'a', 'c'), ordered = T)
> f2
[1] b a c
Levels: b < a < c
> f1 == f2
[1] TRUE TRUE TRUE
```

If we change something, I would personally keep it so that this second case (ordered categories with different order) raises when trying to compare. 
For the unordered case, I am fine with changing this.

> a bit indifferent to whether we should sort factorized categories (a) (I know R does), or show them in order of appearance (b)

I certainly think we should *not* change the current behaviour, which is have sorted categories when you do not specify them manually.

> so the question is why are we differentiating between passed categories and categories created from direct factorizing in the ordered=False case?

Because it is a different case. In case of passed categories, you explicitly pass them, so I find it logical that pandas respects the passed values by the user.

BTW, there is no difference between the `ordered` is `False` vs `True` here. In both cases, if you do not specify categories yourself, they are derived from the unique values and sorted:

```
In [3]: pd.Categorical(['b', 'a'])
Out[3]: 
[b, a]
Categories (2, object): [a, b]

In [4]: pd.Categorical(['b', 'a'], ordered=True)
Out[4]: 
[b, a]
Categories (2, object): [a < b]
```
> Because it is a different case. In case of passed categories, you explicitly pass them, so I find it logical that pandas respects the passed values by the user.

this is all fine, but then they *should* compare equally. The point of ``ordered=False` is the don't *care* about the ordering (so its a set rather than a list comparison).

> If we change something, I would personally keep it so that this second case (ordered categories with different order) raises when trying to compare.
For the unordered case, I am fine with changing this.

Agreed

> they are derived from the unique values and sorted:

Well... usually :)

```python
In [1]: import pandas as pd

In [2]: pd.Categorical(['b', 'a', 1])
Out[2]:
[b, a, 1]
Categories (3, object): [1, a, b]

In [3]: pd.Categorical(['b', 'a', 1, pd.Timestamp('2017')])
Out[3]:
[b, a, 1, 2017-01-01 00:00:00]
Categories (4, object): [b, a, 1, 2017-01-01 00:00:00]
```

But I agree that the current behavior around *building* categorical shouldn't change, just the behavior when comparing them. I *think* we're all in agreement on that.
> But I agree that the current behavior around building categorical shouldn't change, just the behavior when comparing them. I think we're all in agreement on that.

Yep!

Your corner case is a bit strange in that in the first case it seems to be sorted and in the second not, while in *both* cases the values are not sortable (eg if you would put them in an index or series and sort, you get an error).  

Resolving patch:
diff --git a/pandas/core/categorical.py b/pandas/core/categorical.py
--- a/pandas/core/categorical.py
+++ b/pandas/core/categorical.py
@@ -55,17 +55,31 @@ def f(self, other):
                                 "equality or not")
         if isinstance(other, Categorical):
             # Two Categoricals can only be be compared if the categories are
-            # the same
-            if ((len(self.categories) != len(other.categories)) or
-                    not ((self.categories == other.categories).all())):
-                raise TypeError("Categoricals can only be compared if "
-                                "'categories' are the same")
+            # the same (maybe up to ordering, depending on ordered)
+
+            msg = ("Categoricals can only be compared if "
+                   "'categories' are the same.")
+            if len(self.categories) != len(other.categories):
+                raise TypeError(msg + " Categories are different lengths")
+            elif (self.ordered and not (self.categories ==
+                                        other.categories).all()):
+                raise TypeError(msg)
+            elif not set(self.categories) == set(other.categories):
+                raise TypeError(msg)
+
             if not (self.ordered == other.ordered):
                 raise TypeError("Categoricals can only be compared if "
                                 "'ordered' is the same")
-            na_mask = (self._codes == -1) | (other._codes == -1)
+            if not self.ordered and not self.categories.equals(
+                    other.categories):
+                # both unordered and different order
+                other_codes = _get_codes_for_values(other, self.categories)
+            else:
+                other_codes = other._codes
+
+            na_mask = (self._codes == -1) | (other_codes == -1)
             f = getattr(self._codes, op)
-            ret = f(other._codes)
+            ret = f(other_codes)
             if na_mask.any():
                 # In other series, the leads to False, so do that here too
                 ret[na_mask] = False


Test patch:
diff --git a/pandas/tests/test_categorical.py b/pandas/tests/test_categorical.py
--- a/pandas/tests/test_categorical.py
+++ b/pandas/tests/test_categorical.py
@@ -3821,6 +3821,43 @@ def test_cat_equality(self):
         pytest.raises(TypeError, lambda: a > b)
         pytest.raises(TypeError, lambda: b > a)
 
+    @pytest.mark.parametrize('ctor', [
+        lambda *args, **kwargs: Categorical(*args, **kwargs),
+        lambda *args, **kwargs: Series(Categorical(*args, **kwargs)),
+    ])
+    def test_unordered_different_order_equal(self, ctor):
+        # https://github.com/pandas-dev/pandas/issues/16014
+        c1 = ctor(['a', 'b'], categories=['a', 'b'], ordered=False)
+        c2 = ctor(['a', 'b'], categories=['b', 'a'], ordered=False)
+        assert (c1 == c2).all()
+
+        c1 = ctor(['a', 'b'], categories=['a', 'b'], ordered=False)
+        c2 = ctor(['b', 'a'], categories=['b', 'a'], ordered=False)
+        assert (c1 != c2).all()
+
+        c1 = ctor(['a', 'a'], categories=['a', 'b'], ordered=False)
+        c2 = ctor(['b', 'b'], categories=['b', 'a'], ordered=False)
+        assert (c1 != c2).all()
+
+        c1 = ctor(['a', 'a'], categories=['a', 'b'], ordered=False)
+        c2 = ctor(['a', 'b'], categories=['b', 'a'], ordered=False)
+        result = c1 == c2
+        tm.assert_numpy_array_equal(np.array(result), np.array([True, False]))
+
+    def test_unordered_different_categories_raises(self):
+        c1 = Categorical(['a', 'b'], categories=['a', 'b'], ordered=False)
+        c2 = Categorical(['a', 'c'], categories=['c', 'a'], ordered=False)
+        with tm.assert_raises_regex(TypeError,
+                                    "Categoricals can only be compared"):
+            c1 == c2
+
+    def test_compare_different_lengths(self):
+        c1 = Categorical([], categories=['a', 'b'])
+        c2 = Categorical([], categories=['a'])
+        msg = "Categories are different lengths"
+        with tm.assert_raises_regex(TypeError, msg):
+            c1 == c2
+
     def test_concat_append(self):
         cat = pd.Categorical(["a", "b"], categories=["a", "b"])
         vals = [1, 2]