Repository: pandas-dev/pandas

Issue:
BUG: CategoricalIndex.searchsorted doesn't return a scalar if input was scalar
`CategoricalIndex.searchsorted` returns the wrong shape for scalar input. Numpy arrays and all other index types return a scalar if the input is a scalar, but the `CategoricalIndex` does not

For example
```
>>> import numpy as np
>>> np.array([1, 2, 3]).searchsorted(1)
0
>>> np.array([1, 2, 3]).searchsorted([1])
array([0])
>>> import pandas as pd
>>> pd.Index([1, 2, 3]).searchsorted(1)
0
>>> pd.Index([1, 2, 3]).searchsorted([1])
array([0])
```

This issue also affects slicing on sorted/ordered categoricals, which is why I've written another test for the slicing.


- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry
- [ ] example in categoricals.rst


Discussion:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21019?src=pr&el=h1) Report
> Merging [#21019](https://codecov.io/gh/pandas-dev/pandas/pull/21019?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/0c65c57a279e755ab7093db925d1e580f9878dae?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21019/graphs/tree.svg?height=150&width=650&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/21019?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #21019      +/-   ##
==========================================
+ Coverage   91.85%   91.85%   +<.01%     
==========================================
  Files         153      153              
  Lines       49549    49551       +2     
==========================================
+ Hits        45512    45514       +2     
  Misses       4037     4037
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.25% <100%> (ø)` | :arrow_up: |
| #single | `41.87% <0%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21019?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/category.py](https://codecov.io/gh/pandas-dev/pandas/pull/21019/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9jYXRlZ29yeS5weQ==) | `97.03% <ø> (ø)` | :arrow_up: |
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/21019/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.7% <100%> (+0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21019?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21019?src=pr&el=footer). Last update [0c65c57...04ca52f](https://codecov.io/gh/pandas-dev/pandas/pull/21019?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

does this have an associated issue? (pls do a search if not)
Could this be a reason the slowness seen in #20395? (i.e. searchsorted returning wrong type, and then pandas taking a different, slower, code path...)
@jreback The only issue I could find which seems to be related is https://github.com/pandas-dev/pandas/issues/9748 where an open TODO is the slicing on categoricals. From what I can see slicing is still not working for integers, though.

@topper-123 I don't think these are related. You should only hit the `searchsorted` code path during range slicing, e.g. `df[1:2]`
@fjetter, actually that operation is/should be slicing and use searchsorted, as the index is a monotonic index, and ``'b'`` is not unique.

I'll look into that again with this angle. I won't high-jack this thread anymore.
I refactored the tests and hope the intention is a bit clearer now. Slicing of the categorical should behave similar to a ordinary index (at least if it is ordered)
Tests fail because of an import error of `geopandas`. I have no idea what might cause this, though.
pls rebase on master (ci failures are fixed)
Hey @fjetter , want to follow this to the door?
can you merge master and can see where this is

Resolving patch:
diff --git a/pandas/core/arrays/categorical.py b/pandas/core/arrays/categorical.py
--- a/pandas/core/arrays/categorical.py
+++ b/pandas/core/arrays/categorical.py
@@ -1344,15 +1344,13 @@ def searchsorted(self, value, side='left', sorter=None):
                              "ordered one")
 
         from pandas.core.series import Series
+        codes = _get_codes_for_values(Series(value).values, self.categories)
+        if -1 in codes:
+            raise KeyError("Value(s) to be inserted must be in categories.")
 
-        values_as_codes = _get_codes_for_values(Series(value).values,
-                                                self.categories)
+        codes = codes[0] if is_scalar(value) else codes
 
-        if -1 in values_as_codes:
-            raise ValueError("Value(s) to be inserted must be in categories.")
-
-        return self.codes.searchsorted(values_as_codes, side=side,
-                                       sorter=sorter)
+        return self.codes.searchsorted(codes, side=side, sorter=sorter)
 
     def isna(self):
         """


Test patch:
diff --git a/pandas/tests/arrays/categorical/test_analytics.py b/pandas/tests/arrays/categorical/test_analytics.py
--- a/pandas/tests/arrays/categorical/test_analytics.py
+++ b/pandas/tests/arrays/categorical/test_analytics.py
@@ -85,10 +85,10 @@ def test_searchsorted(self):
 
         # Searching for single item argument, side='left' (default)
         res_cat = c1.searchsorted('apple')
+        assert res_cat == 2
+
         res_ser = s1.searchsorted('apple')
-        exp = np.array([2], dtype=np.intp)
-        tm.assert_numpy_array_equal(res_cat, exp)
-        tm.assert_numpy_array_equal(res_ser, exp)
+        assert res_ser == 2
 
         # Searching for single item array, side='left' (default)
         res_cat = c1.searchsorted(['bread'])
@@ -105,13 +105,13 @@ def test_searchsorted(self):
         tm.assert_numpy_array_equal(res_ser, exp)
 
         # Searching for a single value that is not from the Categorical
-        pytest.raises(ValueError, lambda: c1.searchsorted('cucumber'))
-        pytest.raises(ValueError, lambda: s1.searchsorted('cucumber'))
+        pytest.raises(KeyError, lambda: c1.searchsorted('cucumber'))
+        pytest.raises(KeyError, lambda: s1.searchsorted('cucumber'))
 
         # Searching for multiple values one of each is not from the Categorical
-        pytest.raises(ValueError,
+        pytest.raises(KeyError,
                       lambda: c1.searchsorted(['bread', 'cucumber']))
-        pytest.raises(ValueError,
+        pytest.raises(KeyError,
                       lambda: s1.searchsorted(['bread', 'cucumber']))
 
         # searchsorted call for unordered Categorical