BUG: CategoricalIndex.get_indexer issue with NaNs (#45361)
State: closed

Bug Example:
```python
ci = pd.CategoricalIndex([1, 2, np.nan, 3])
other = pd.Index([2, 3, 4])

res = ci.get_indexer(other)

>>> res
array([1, 3, 2])
```

`ci.get_indexer(other)` yields incorrect results when `ci` contains NaNs, and 
`other` does not. The reason is that, if `other` contains elements that do not 
match any category in `ci`, they are replaced by NaNs. In such a situation, 
if `ci` also has NaNs, then the corresponding elements in `ci` are mapped to 
an index that is not -1

eg:
```python
ci = pd.CategoricalIndex([1, 2, np.nan, 3])
other = pd.Index([2, 3, 4])
ci.get_indexer(other)
```
In the implementation of get_indexer, `other` becomes [2, 3, NaN]
which is mapped to index 2, in `ci`

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Categorical
description: Categorical Data Type

Comments:
Hey, thanks for the PR! 
Is this issue reported (if yes, could you please link to the issue), otherwise please consider creating an issue (for tracking, discussion, etc).
@debnathshoham this is actually my first PR. Could you tell me what needs to be done? I'll look into it right away.
So you can find a lot of open issues in the issues tab (around 3.3k). What it seems to me is you have identified an issue, and want to put a fix to that.
First step should be to check if there are any already open issue that someone else has opened, and point this PR to that particular issue (in closs #xxxx). If there is no such issue, please create a new issue with detailed description and reproducible examples.
@Shashank-Shet thanks for taking a look at this.

For any PR fixing a bug, there should always be a test for the fixed behavior.  In this case it would go somewhere like tests/indexes/categorical/tests_indexing.py

It looks like this broke some existing tests.  That suggests that either the existing tests have a problem, or this fix isn't quite right.

If this is your first PR, I suggest looking for issues with the "Good first issue" label.
@debnathshoham will do. I'll work on identifying whether this is a new issue and reporting if necessary.
@jbrockmendel  yes, the fix has some flaws. I'll look into them as well. Worst case, I can at least let another person know what I've dug up. 
@jbrockmendel I have made a small change to the commit. Now all but one check is passing. Unfortunately, I cannot get much data regarding the testcase itself. Apparently, it seems to be a build error (timeout).
> Unfortunately, I cannot get much data regarding the testcase itself. Apparently, it seems to be a build error (timeout).

ill take a look.  the timeout is unrelated, is affecting all PRs ATM.
Please add test(s) for the behavior you are trying to fix.

@jbrockmendel I have added a test case (let me know if I need to add more). Also, the checks failing on this build seem to be related to a file I haven't changed (parquet.py). Some test cases which are failing are apparently due to a new test case added called `test_unsupported_float16_cleanup`. While I am not sure whether these tests are failing because of my commit, is it possible to find out if this is plaguing other PRs as well?
> it possible to find out if this is plaguing other PRs as well?

it is.  You can ignore this failure for now.
Long term solution is #37930
Looks good.  Can you add a whatsnew note for 1.5.0
> Looks good. Can you add a whatsnew note for 1.5.0

I am not sure what this entails. May I know what I am required to add?
>> Looks good. Can you add a whatsnew note for 1.5.0

> I am not sure what this entails. May I know what I am required to add?

In doc/source/whatsnew/v1.5.0.rst there should be a section for Indexing bugs.  Add an entry describing the bug this fixes.  (use the existing entries to get an idea for how much to write)
> > > Looks good. Can you add a whatsnew note for 1.5.0
> 
> > I am not sure what this entails. May I know what I am required to add?
> 
> In doc/source/whatsnew/v1.5.0.rst there should be a section for Indexing bugs. Add an entry describing the bug this fixes. (use the existing entries to get an idea for how much to write)

Cool. I have made the changes. Thanks for the help!

thanks @Shashank-Shet