Repository: pandas-dev/pandas

Issue:
API: relax categorical equality when comparing against object
from [SO](http://stackoverflow.com/questions/27164380/merging-pandas-dataframes-on-categorical-series)

```
In [1]: a = pd.Series(['a','b','c'],dtype="category")
In [2]: b = pd.Series(['a','b','c'],dtype="object")
In [3]: c = pd.Series(['a','b','cc'],dtype="object")
In [5]: a==b
TypeError: Cannot compare a Categorical for op <built-in function eq> with type <type 'numpy.ndarray'>. If you want to 
compare values, use 'series <op> np.asarray(cat)'.
```

```
In [6]: A = pd.DataFrame({'A':a,'B':[1,2,3]})
In [7]: B = pd.DataFrame({'A':b,'C':[4,5,6]})

In [9]: A.merge(B,on='A') 
Out[9]: 
   A  B  C
0  a  1  4
1  b  2  5
2  c  3  6

In [10]: A.merge(B,on='A').dtypes
Out[10]: 
A    object
B     int64
C     int64
dtype: object

In [11]: A.dtypes
Out[11]: 
A    category
B       int64
dtype: object

In [12]: B.dtypes
Out[12]: 
A    object
C     int64
dtype: object
```



Discussion:
cc @JanSchulz
@jorisvandenbossche 
cc @shoyer
cc @immerrr

I think the merging is the correct behavior (@immerrr and I worked on this for a while to make it correct).

as a single categorical merging with either an equal categorical (meaning the same categories) or an object array is OK, though you do lose the 'character', meaning the Categorical of the dtypes. But this is like downgrading a type, e.g. going from integer to float when you are merging a float. So I don't think this is that big of a deal.

Note that we _could_ try to infer the categorical if possible after merging (which is possible if the object happens to not contain new categories).

Agreed, the first example should work. 

Yup, it seems natural to enable that.

The problem is when there is a custom ordering, then a object array with `"a"<"b"<"c"` is not the same as an categorical with `"a">"b">"c"`, so what should `"a" < "b"` comparison do: use the object order or the categorical one?

In my head I think about categorical elements as a new type and under python 3, different types are not comparable per default and I went with that thought in the implementation.

@JanSchulz I agree with you on the ordering case and < or >. Clearly they are not comparable.

However, merging and equality are are about positional equality.

and we already support comparisons against a scalar

```
In [2]: s = Series(list('abc'),dtype='category')

In [3]: s
Out[3]: 
0    a
1    b
2    c
dtype: category
Categories (3, object): [a < b < c]

In [4]: s=='a'
Out[4]: 
0     True
1    False
2    False
dtype: bool
```

I'll put up a PR


Resolving patch:
diff --git a/pandas/core/categorical.py b/pandas/core/categorical.py
--- a/pandas/core/categorical.py
+++ b/pandas/core/categorical.py
@@ -64,6 +64,12 @@ def f(self, other):
             else:
                 return np.repeat(False, len(self))
         else:
+
+            # allow categorical vs object dtype array comparisons for equality
+            # these are only positional comparisons
+            if op in ['__eq__','__ne__']:
+                return getattr(np.array(self),op)(np.array(other))
+
             msg = "Cannot compare a Categorical for op {op} with type {typ}. If you want to \n" \
                   "compare values, use 'np.asarray(cat) <op> other'."
             raise TypeError(msg.format(op=op,typ=type(other)))
diff --git a/pandas/core/ops.py b/pandas/core/ops.py
--- a/pandas/core/ops.py
+++ b/pandas/core/ops.py
@@ -541,10 +541,13 @@ def _comp_method_SERIES(op, name, str_rep, masker=False):
     """
     def na_op(x, y):
 
-        if com.is_categorical_dtype(x) != (not np.isscalar(y) and com.is_categorical_dtype(y)):
-            msg = "Cannot compare a Categorical for op {op} with type {typ}. If you want to \n" \
-                  "compare values, use 'series <op> np.asarray(cat)'."
-            raise TypeError(msg.format(op=op,typ=type(y)))
+        # dispatch to the categorical if we have a categorical
+        # in either operand
+        if com.is_categorical_dtype(x):
+            return op(x,y)
+        elif com.is_categorical_dtype(y) and not lib.isscalar(y):
+            return op(y,x)
+
         if x.dtype == np.object_:
             if isinstance(y, list):
                 y = lib.list_to_object_array(y)
@@ -586,33 +589,33 @@ def wrapper(self, other):
                 msg = "Cannot compare a Categorical for op {op} with Series of dtype {typ}.\n"\
                       "If you want to compare values, use 'series <op> np.asarray(other)'."
                 raise TypeError(msg.format(op=op,typ=self.dtype))
-        else:
 
-            mask = isnull(self)
 
-            values = self.get_values()
-            other = _index.convert_scalar(values,_values_from_object(other))
+        mask = isnull(self)
 
-            if issubclass(values.dtype.type, (np.datetime64, np.timedelta64)):
-                values = values.view('i8')
+        values = self.get_values()
+        other = _index.convert_scalar(values,_values_from_object(other))
 
-            # scalars
-            res = na_op(values, other)
-            if np.isscalar(res):
-                raise TypeError('Could not compare %s type with Series'
-                                % type(other))
+        if issubclass(values.dtype.type, (np.datetime64, np.timedelta64)):
+            values = values.view('i8')
 
-            # always return a full value series here
-            res = _values_from_object(res)
+        # scalars
+        res = na_op(values, other)
+        if np.isscalar(res):
+            raise TypeError('Could not compare %s type with Series'
+                            % type(other))
 
-            res = pd.Series(res, index=self.index, name=self.name,
-                            dtype='bool')
+        # always return a full value series here
+        res = _values_from_object(res)
 
-            # mask out the invalids
-            if mask.any():
-                res[mask] = masker
+        res = pd.Series(res, index=self.index, name=self.name,
+                        dtype='bool')
+
+        # mask out the invalids
+        if mask.any():
+            res[mask] = masker
 
-            return res
+        return res
     return wrapper
 
 


Test patch:
diff --git a/pandas/tests/test_categorical.py b/pandas/tests/test_categorical.py
--- a/pandas/tests/test_categorical.py
+++ b/pandas/tests/test_categorical.py
@@ -2211,11 +2211,63 @@ def f():
             tm.assert_series_equal(res, exp)
 
         # And test NaN handling...
-        cat = pd.Series(pd.Categorical(["a","b","c", np.nan]))
+        cat = Series(Categorical(["a","b","c", np.nan]))
         exp = Series([True, True, True, False])
         res = (cat == cat)
         tm.assert_series_equal(res, exp)
 
+    def test_cat_equality(self):
+
+        # GH 8938
+        # allow equality comparisons
+        a = Series(list('abc'),dtype="category")
+        b = Series(list('abc'),dtype="object")
+        c = Series(['a','b','cc'],dtype="object")
+        d = Series(list('acb'),dtype="object")
+        e = Categorical(list('abc'))
+        f = Categorical(list('acb'))
+
+        # vs scalar
+        self.assertFalse((a=='a').all())
+        self.assertTrue(((a!='a') == ~(a=='a')).all())
+
+        self.assertFalse(('a'==a).all())
+        self.assertTrue((a=='a')[0])
+        self.assertTrue(('a'==a)[0])
+        self.assertFalse(('a'!=a)[0])
+
+        # vs list-like
+        self.assertTrue((a==a).all())
+        self.assertFalse((a!=a).all())
+
+        self.assertTrue((a==list(a)).all())
+        self.assertTrue((a==b).all())
+        self.assertTrue((b==a).all())
+        self.assertTrue(((~(a==b))==(a!=b)).all())
+        self.assertTrue(((~(b==a))==(b!=a)).all())
+
+        self.assertFalse((a==c).all())
+        self.assertFalse((c==a).all())
+        self.assertFalse((a==d).all())
+        self.assertFalse((d==a).all())
+
+        # vs a cat-like
+        self.assertTrue((a==e).all())
+        self.assertTrue((e==a).all())
+        self.assertFalse((a==f).all())
+        self.assertFalse((f==a).all())
+
+        self.assertTrue(((~(a==e)==(a!=e)).all()))
+        self.assertTrue(((~(e==a)==(e!=a)).all()))
+        self.assertTrue(((~(a==f)==(a!=f)).all()))
+        self.assertTrue(((~(f==a)==(f!=a)).all()))
+
+        # non-equality is not comparable
+        self.assertRaises(TypeError, lambda: a < b)
+        self.assertRaises(TypeError, lambda: b < a)
+        self.assertRaises(TypeError, lambda: a > b)
+        self.assertRaises(TypeError, lambda: b > a)
+
     def test_concat(self):
         cat = pd.Categorical(["a","b"], categories=["a","b"])
         vals = [1,2]