Repository: pandas-dev/pandas

Issue:
BUG: ordered categorical comparison with missing values evaluates to True
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
pd.Categorical(["1", "2", "3", None], categories=["1", "2", "3"], ordered=True) <= "2"
# => array([ True,  True, False,  True])
```

#### Problem description

Here a missing entry is being evaluated as `None <= "2" == True`. Shouldn't missing values always be evaluate to `False` in any comparison?
I think this is related to #4537

#### Expected Output

```python
import pandas as pd
pd.Categorical(["1", "2", "3", None], categories=["1", "2", "3"], ordered=True) <= "2"
# => array([ True,  True, False,  False])
```

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.8.final.0
python-bits: 64
OS: Darwin
OS-release: 17.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8
LOCALE: en_GB.UTF-8

pandas: 0.24.2
pytest: 3.6.2
pip: 10.0.1
setuptools: 39.2.0
Cython: 0.28.3
numpy: 1.16.3
scipy: 1.2.1
pyarrow: None
xarray: None
IPython: 7.1.1
sphinx: 1.7.5
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: 3.4.4
numexpr: 2.6.9
feather: None
matplotlib: 3.0.3
openpyxl: 2.5.4
xlrd: 1.1.0
xlwt: 1.2.0
xlsxwriter: 1.0.5
lxml.etree: 4.2.2
bs4: 4.6.0
html5lib: 0.9999999
sqlalchemy: 1.2.8
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None
</details>



Discussion:
Yep, should probably be False. It looks like we correctly mask for categorical : categorical comparisons, but not categorical : scalar. 


```python
In [16]: c
Out[16]:
[1, 2, 3, NaN]
Categories (3, object): [1 < 2 < 3]

In [17]: c <= c
Out[17]: array([ True,  True,  True, False])


```

This fix will be somewhere in https://github.com/pandas-dev/pandas/blob/65eac80dd4113ddbcee876029bfe03b4b3f11a3d/pandas/core/arrays/categorical.py#L100, similar to what we do above.
@TomAugspurger I am new here. I would like to contribute to this issue. Can you tell me what needs to be done?

The `Categorical` object has the `_codes` property, which is used for comparison.

```python
In [1]: import pandas as pd

In [2]: a = pd.Categorical([1, 2, 3, 4], categories=[1, 2, 3], ordered=Tr
   ...: ue)

In [3]: a._codes
Out[3]: array([ 0,  1,  2, -1], dtype=int8)

In [4]: a._codes < 2
Out[4]: array([ True,  True, False,  True])
```
The `_codes` takes `NaN` to be `-1` leading to incorrect comparison

-----

Perhaps the fix lies in correct construction of the `Categorical` object and not in the comparison itself.

Resolving patch:
diff --git a/pandas/core/arrays/categorical.py b/pandas/core/arrays/categorical.py
--- a/pandas/core/arrays/categorical.py
+++ b/pandas/core/arrays/categorical.py
@@ -89,18 +89,23 @@ def f(self, other):
             else:
                 other_codes = other._codes
 
-            na_mask = (self._codes == -1) | (other_codes == -1)
+            mask = (self._codes == -1) | (other_codes == -1)
             f = getattr(self._codes, op)
             ret = f(other_codes)
-            if na_mask.any():
+            if mask.any():
                 # In other series, the leads to False, so do that here too
-                ret[na_mask] = False
+                ret[mask] = False
             return ret
 
         if is_scalar(other):
             if other in self.categories:
                 i = self.categories.get_loc(other)
-                return getattr(self._codes, op)(i)
+                ret = getattr(self._codes, op)(i)
+
+                # check for NaN in self
+                mask = (self._codes == -1)
+                ret[mask] = False
+                return ret
             else:
                 if op == '__eq__':
                     return np.repeat(False, len(self))


Test patch:
diff --git a/pandas/tests/arrays/categorical/test_operators.py b/pandas/tests/arrays/categorical/test_operators.py
--- a/pandas/tests/arrays/categorical/test_operators.py
+++ b/pandas/tests/arrays/categorical/test_operators.py
@@ -1,4 +1,5 @@
 import operator
+import warnings
 
 import numpy as np
 import pytest
@@ -17,7 +18,6 @@ def test_categories_none_comparisons(self):
         tm.assert_categorical_equal(factor, self.factor)
 
     def test_comparisons(self):
-
         result = self.factor[self.factor == 'a']
         expected = self.factor[np.asarray(self.factor) == 'a']
         tm.assert_categorical_equal(result, expected)
@@ -186,6 +186,36 @@ def test_comparison_with_unknown_scalars(self):
         tm.assert_numpy_array_equal(cat != 4,
                                     np.array([True, True, True]))
 
+    def test_comparison_of_ordered_categorical_with_nan_to_scalar(
+            self, compare_operators_no_eq_ne):
+        # https://github.com/pandas-dev/pandas/issues/26504
+        # BUG: fix ordered categorical comparison with missing values (#26504 )
+        # and following comparisons with scalars in categories with missing
+        # values should be evaluated as False
+
+        cat = Categorical([1, 2, 3, None], categories=[1, 2, 3], ordered=True)
+        scalar = 2
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", RuntimeWarning)
+            expected = getattr(np.array(cat),
+                               compare_operators_no_eq_ne)(scalar)
+        actual = getattr(cat, compare_operators_no_eq_ne)(scalar)
+        tm.assert_numpy_array_equal(actual, expected)
+
+    def test_comparison_of_ordered_categorical_with_nan_to_listlike(
+            self, compare_operators_no_eq_ne):
+        # https://github.com/pandas-dev/pandas/issues/26504
+        # and following comparisons of missing values in ordered Categorical
+        # with listlike should be evaluated as False
+
+        cat = Categorical([1, 2, 3, None], categories=[1, 2, 3], ordered=True)
+        other = Categorical([2, 2, 2, 2], categories=[1, 2, 3], ordered=True)
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", RuntimeWarning)
+            expected = getattr(np.array(cat), compare_operators_no_eq_ne)(2)
+        actual = getattr(cat, compare_operators_no_eq_ne)(other)
+        tm.assert_numpy_array_equal(actual, expected)
+
     @pytest.mark.parametrize('data,reverse,base', [
         (list("abc"), list("cba"), list("bbb")),
         ([1, 2, 3], [3, 2, 1], [2, 2, 2])]