Repository: pandas-dev/pandas
Pull request: BUG: Fix construction of Categorical from pd.NA
- [x] closes #31927 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Problem statement:
Construction of Categorical from array with pd.NA failing
So creating a Categorical from an array with pd.NA fails:

```
In [10]: pd.Categorical(np.array(["a", "b", pd.NA], dtype=object))  
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/scipy/pandas/pandas/core/arrays/categorical.py in __init__(self, values, categories, ordered, dtype, fastpath)
    354             try:
--> 355                 codes, categories = factorize(values, sort=True)
    356             except TypeError:

~/scipy/pandas/pandas/core/algorithms.py in factorize(values, sort, na_sentinel, size_hint)
    635         codes, uniques = _factorize_array(
--> 636             values, na_sentinel=na_sentinel, size_hint=size_hint, na_value=na_value
    637         )

~/scipy/pandas/pandas/core/algorithms.py in _factorize_array(values, na_sentinel, size_hint, na_value)
    483     table = hash_klass(size_hint or len(values))
--> 484     uniques, codes = table.factorize(values, na_sentinel=na_sentinel, na_value=na_value)
    485 

~/scipy/pandas/pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.factorize()

~/scipy/pandas/pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable._unique()

~/scipy/pandas/pandas/_libs/missing.pyx in pandas._libs.missing.NAType.__bool__()

TypeError: boolean value of NA is ambiguous
```

while from a list actually "works":

```
In [11]: pd.Categorical(["a", "b", pd.NA]) 
Out[11]: 
[a, b, NaN]
Categories (2, object): [a, b]
```

This also means that creating a Categorical from a StringArray with missing values won't work (with the same error as above). 
I am only not sure if it should work (at least before we can have a "string" dtype index as the categories of the created Categorical).

Hints from issue discussion:
Not too familiar with this part of the code, but could the issue be here: https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/hashtable_class_helper.pxi.in#L433? Can that comparison be short-circuited somehow by checking if `value` is identically `na_value2` _or_ equal to `na_value2`?
Yes, that sounds correct (only that it's in the PyObect or String hashtable lower in the file, but with similar code)

Commit message: Add MultiIndex test

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.2.rst b/doc/source/whatsnew/v1.0.2.rst
index 0216007ea5ba8..860463a99fa2f 100644
--- a/doc/source/whatsnew/v1.0.2.rst
+++ b/doc/source/whatsnew/v1.0.2.rst
@@ -31,6 +31,7 @@ Bug fixes
 **Categorical**
 
 - Fixed bug where :meth:`Categorical.from_codes` improperly raised a ``ValueError`` when passed nullable integer codes. (:issue:`31779`)
+- Fixed bug where :meth:`Categorical` constructor would raise a ``TypeError`` when given a numpy array containing ``NA``. (:issue:`31927`)
 
 **I/O**
 
diff --git a/pandas/_libs/hashtable_class_helper.pxi.in b/pandas/_libs/hashtable_class_helper.pxi.in
index 6671375f628e7..01ce78199b60a 100644
--- a/pandas/_libs/hashtable_class_helper.pxi.in
+++ b/pandas/_libs/hashtable_class_helper.pxi.in
@@ -1032,7 +1032,8 @@ cdef class PyObjectHashTable(HashTable):
             val = values[i]
             hash(val)
 
-            if ignore_na and ((val != val or val is None)
+            if ignore_na and (((val == val) is (val != val))
+                              or (val != val or val is None)
                               or (use_na_value and val == na_value)):
                 # if missing values do not count as unique values (i.e. if
                 # ignore_na is True), skip the hashtable entry for them, and
diff --git a/pandas/tests/arrays/categorical/test_constructors.py b/pandas/tests/arrays/categorical/test_constructors.py
index dbd8fd8df67c1..3854bbb633c65 100644
--- a/pandas/tests/arrays/categorical/test_constructors.py
+++ b/pandas/tests/arrays/categorical/test_constructors.py
@@ -458,6 +458,14 @@ def test_constructor_with_categorical_categories(self):
         result = Categorical(["a", "b"], categories=CategoricalIndex(["a", "b", "c"]))
         tm.assert_categorical_equal(result, expected)
 
+    def test_construction_with_na(self):
+        # https://github.com/pandas-dev/pandas/issues/31927
+        values = ["a", pd.NA]
+        result = Categorical(np.array(values, dtype=object))
+        expected = Categorical(values)
+
+        tm.assert_categorical_equal(result, expected)
+
     def test_from_codes(self):
 
         # too few categories
diff --git a/pandas/tests/indexing/multiindex/test_multiindex.py b/pandas/tests/indexing/multiindex/test_multiindex.py
index 0064187a94265..da4307477fb8e 100644
--- a/pandas/tests/indexing/multiindex/test_multiindex.py
+++ b/pandas/tests/indexing/multiindex/test_multiindex.py
@@ -111,3 +111,13 @@ def test_nested_tuples_duplicates(self):
         df3 = df.copy(deep=True)
         df3.loc[[(dti[0], "a")], "c2"] = 1.0
         tm.assert_frame_equal(df3, expected)
+
+    def test_multiindex_from_product_contains_na(self):
+        # https://github.com/pandas-dev/pandas/issues/31883
+        values1 = [np.array([0.0, pd.NA], dtype="object"), ["a", "b"]]
+        values2 = [np.array([0.0, np.nan], dtype="object"), ["a", "b"]]
+
+        result = pd.MultiIndex.from_product(values1)
+        expected = pd.MultiIndex.from_product(values2)
+
+        tm.assert_index_equal(result, expected)


Code review comments:
diff_hunk: @@ -1032,7 +1032,8 @@ cdef class PyObjectHashTable(HashTable):
             val = values[i]
             hash(val)
 
-            if ignore_na and ((val != val or val is None)
+            if ignore_na and (((val == val) is (val != val))
path: pandas/_libs/hashtable_class_helper.pxi.in
text: @user1:
I think would be clearer if you just added `or val is pd.NA`

@author:
Are we able to do that here? I'm not too familiar with `pxi.in` files

@user1:
Yea import machinery should still be the same

@author:
Ok yeah that's way nicer

diff_hunk: @@ -458,6 +458,14 @@ def test_constructor_with_categorical_categories(self):
         result = Categorical(["a", "b"], categories=CategoricalIndex(["a", "b", "c"]))
         tm.assert_categorical_equal(result, expected)
 
+    def test_construction_with_na(self):
path: pandas/tests/arrays/categorical/test_constructors.py
text: @user1:
If you rename test to `test_construction_with_null` and use the `nulls_fixture` I think would be more comprehensive (will have to skip pd.NaT)