Repository: pandas-dev/pandas
Pull request: REGR: Categorical with np.str_ categories
- [x] closes #31499
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
Pandas 1.0 no longer handles `numpy.str_`s as catgories
#### Code Sample

```python
import pandas as pd
pd.Categorical(['1', '0', '1'], [np.str_('0'), np.str_('1')])
```

```pytb
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/angerer/Dev/Python/venvs/env-pandas-1/lib/python3.8/site-packages/pandas/core/arrays/categorical.py", line 385, in __init__
    codes = _get_codes_for_values(values, dtype.categories)
  File "/home/angerer/Dev/Python/venvs/env-pandas-1/lib/python3.8/site-packages/pandas/core/arrays/categorical.py", line 2576, in _get_codes_for_values
    t.map_locations(cats)
  File "pandas/_libs/hashtable_class_helper.pxi", line 1403, in pandas._libs.hashtable.StringHashTable.map_locations
TypeError: Expected unicode, got numpy.str_
```

#### Problem description
I know that having a list of `numpy.str_`s seems weird, but it easily happens when you use non-numpy algorithms on numpy arrays (e.g. `natsort.natsorted` in our case), or via comprehensions or so:

```py
>>> np.array(['1', '0'])[0].__class__
<class 'numpy.str_'>
>>> [type(s) for s in np.array(['1', '0'])]
[<class 'numpy.str_'>, <class 'numpy.str_'>]
```
#### Expected Output
A normal pd.Categorical

#### Pandas version

pandas 1.0

Hints from issue discussion:
This changed from 0.25.3?

Are you able to pin down what change caused it?
Yes, in 0.25.3 this worked. 

At least, it gives a categorical with object categories with those numpy strings (but with Series constructor, we also preserve the numpy strings, and don't convert to python strings, so that seems the "expected" behaviour).
My guess is that it's related to https://github.com/pandas-dev/pandas/pull/30419 which changed `get_c_string` implementation (which is used in StringHashTable to get the c string from the string object) cc @jbrockmendel 
maybe if we're lucky it will be good enough to change L706 in hashtable_class_helper.pxi.in from `v = get_c_string(val)` to `v = get_c_string(<str>val)`, but this is really a PITA because the previous line is precisely a check for `isintance(val, str)` which is True for np.str_ objects

Commit message: whatsnew

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.1.rst b/doc/source/whatsnew/v1.0.1.rst
index ff8433c7cafd9..745a0a66e4b65 100644
--- a/doc/source/whatsnew/v1.0.1.rst
+++ b/doc/source/whatsnew/v1.0.1.rst
@@ -19,7 +19,7 @@ Bug fixes
 
 Categorical
 ^^^^^^^^^^^
-
+- Bug in :class:`Categorical` construction with ``numpy.str_`` categories (:issue:`31499`)
 -
 -
 
diff --git a/pandas/_libs/hashtable_class_helper.pxi.in b/pandas/_libs/hashtable_class_helper.pxi.in
index 7d57c67e70b58..6671375f628e7 100644
--- a/pandas/_libs/hashtable_class_helper.pxi.in
+++ b/pandas/_libs/hashtable_class_helper.pxi.in
@@ -670,7 +670,9 @@ cdef class StringHashTable(HashTable):
             val = values[i]
 
             if isinstance(val, str):
-                v = get_c_string(val)
+                # GH#31499 if we have a np.str_ get_c_string wont recognize
+                #  it as a str, even though isinstance does.
+                v = get_c_string(<str>val)
             else:
                 v = get_c_string(self.na_string_sentinel)
             vecs[i] = v
@@ -703,7 +705,9 @@ cdef class StringHashTable(HashTable):
             val = values[i]
 
             if isinstance(val, str):
-                v = get_c_string(val)
+                # GH#31499 if we have a np.str_ get_c_string wont recognize
+                #  it as a str, even though isinstance does.
+                v = get_c_string(<str>val)
             else:
                 v = get_c_string(self.na_string_sentinel)
             vecs[i] = v
diff --git a/pandas/tests/arrays/categorical/test_constructors.py b/pandas/tests/arrays/categorical/test_constructors.py
index cfba3da354d44..70e1421c8dcf4 100644
--- a/pandas/tests/arrays/categorical/test_constructors.py
+++ b/pandas/tests/arrays/categorical/test_constructors.py
@@ -408,6 +408,11 @@ def test_constructor_str_unknown(self):
         with pytest.raises(ValueError, match="Unknown dtype"):
             Categorical([1, 2], dtype="foo")
 
+    def test_constructor_np_strs(self):
+        # GH#31499 Hastable.map_locations needs to work on np.str_ objects
+        cat = pd.Categorical(["1", "0", "1"], [np.str_("0"), np.str_("1")])
+        assert all(isinstance(x, np.str_) for x in cat.categories)
+
     def test_constructor_from_categorical_with_dtype(self):
         dtype = CategoricalDtype(["a", "b", "c"], ordered=True)
         values = Categorical(["a", "b", "d"])


Code review comments:
diff_hunk: @@ -19,7 +19,7 @@ Bug fixes
 
 Categorical
 ^^^^^^^^^^^
-
+- Bug in :class:`Categorical` construction with ``numpy.str_`` categories (:issue:`31499`)
path: doc/source/whatsnew/v1.0.1.rst
text: @user1:
```suggestion
- Fixed regression in :class:`Categorical` construction with ``numpy.str_`` categories (:issue:`31499`)
```