Fixed regression of Multi index with NaN
State: closed

- [x] closes #22247
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry
As pull #19074 changed compute of hashtable from `self.values` to `self.codes` and `self.levels`, NaN values doesn't cover.
I just simply fixed this regression the special case, by check return value from the hashtable.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Regression
description: Functionality that used to work in a prior pandas version
name: MultiIndex

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25424?src=pr&el=h1) Report
> Merging [#25424](https://codecov.io/gh/pandas-dev/pandas/pull/25424?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/9eec9b86ddba46075b63aa776f6284188de4f1ce?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `84.21%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25424/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25424?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25424      +/-   ##
==========================================
- Coverage   91.25%   91.24%   -0.01%     
==========================================
  Files         172      172              
  Lines       52973    53007      +34     
==========================================
+ Hits        48338    48366      +28     
- Misses       4635     4641       +6
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.82% <84.21%> (-0.01%)` | :arrow_down: |
| #single | `41.76% <55.26%> (+0.01%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25424?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/25424/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `95.26% <84.21%> (-0.36%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25424?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25424?src=pr&el=footer). Last update [9eec9b8...7f7a12a](https://codecov.io/gh/pandas-dev/pandas/pull/25424?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25424?src=pr&el=h1) Report
> Merging [#25424](https://codecov.io/gh/pandas-dev/pandas/pull/25424?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/3855a27be4f04d15e7ba7aee12f0220c93148d3d?src=pr&el=desc) will **decrease** coverage by `50.04%`.
> The diff coverage is `45%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25424/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25424?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #25424       +/-   ##
===========================================
- Coverage   91.73%   41.69%   -50.05%     
===========================================
  Files         173      173               
  Lines       52856    52876       +20     
===========================================
- Hits        48490    22048    -26442     
- Misses       4366    30828    +26462
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `41.69% <45%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25424?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/25424/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `34.23% <45%> (-61.39%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/25424/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25424/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/25424/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/25424/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/25424/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/25424/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.35%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25424/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-95.46%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/25424/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.17%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/25424/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.15%)` | :arrow_down: |
| ... and [131 more](https://codecov.io/gh/pandas-dev/pandas/pull/25424/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25424?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25424?src=pr&el=footer). Last update [3855a27...b2879de](https://codecov.io/gh/pandas-dev/pandas/pull/25424?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jreback 
I want to check `NaN` in MultiIndex, but function `isna()` isn't implemented in MultiIndex.
How can I change?
and I change from `np.where()` to use `pd.core.algorithms.isin()` but can not pass the test cases.
Should I use directly `np.isin()` or modify `algorithms.isin()`?
Hello @hksonngan! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-03-14 21:47:12 UTC
@hksonngan you are adding an amazing amount of complexity here. Please step thru the setting code; This is likely a very small change.
@jreback I think one problem remains when set value multi-index with NaN, get_loc, will crash. I will investigate this.
@jreback I add code to pass of the case when setting value with NaN index, but I don't know why I get a lot of linting errors when I rebase. I don't change these files. Do you know why?
@jreback ping
@hksonngan this is a massive amount of code.  I am not even sure what you are trying to do your fix is VERY complicated. Please point out exactly where the issue is.
> @hksonngan this is a massive amount of code. I am not even sure what you are trying to do your fix is VERY complicated. Please point out exactly where the issue is.

@jreback if we have NaN in MultiIndex as:
```python
df = pd.DataFrame(
    [
        ['A', np.nan, 1.23, 4.56],
        ['A', 'G', 1.32, 4.65],
        ['A', 'D', 9.87, 10.54],
    ],
    columns=['pivot_0', 'pivot_1', 'col_1', 'col_2'],
)
df.set_index(['pivot_0', 'pivot_1'], inplace=True)
```
```
                                 col_1   col_2
pivot_0  pivot_1
A            NaN           1.23     4.56
              G                1.32     4.65
              D                9.87    10.54
```
Now I set new `df.at[('A', 'F'), 'col_2'] = 0.0` I get wrong result in `col_1` as:
```
                                 col_1   col_2
pivot_0  pivot_1
A            NaN           1.23      4.56
              G                1.32      4.65
              D                9.87    10.54
              F                 1.23      0.0
```
The right result is must `              F                 NaN      0.0`
This is because [`get_indexer`](https://github.com/hksonngan/pandas/blob/7f7a12a0fe18c89a22b6f4bf3f9055e5d00b1b6e/pandas/_libs/index.pyx#L660) `indexer = self._base.get_indexer(self, lab_ints)` return from hash with duplicate value for index `NaN` and new as `F` here.
For above example `indexer = [-1 4 6 -1]`
Or I add new MultiIndex with NaN, I get exception like this code:
```python



df = pd.DataFrame(
    [
        ['A', 'G', 1.32, 4.65],
        ['A', 'D', 9.87, 10.54],
    ],
    columns=['pivot_0', 'pivot_1', 'col_1', 'col_2'],
)
df.set_index(['pivot_0', 'pivot_1'], inplace=True)
df.at[('A', np.nan), 'col_2'] = 0.0 # Get exception in here
```
* So I fixed simple by check 'NaN' exist or not in MultiIndex with function  [`_hasnans`](https://github.com/hksonngan/pandas/blob/7f7a12a0fe18c89a22b6f4bf3f9055e5d00b1b6e/pandas/core/indexes/multi.py#L754), because when add new value to DataFrame, the [get_indexer](https://github.com/hksonngan/pandas/blob/7f7a12a0fe18c89a22b6f4bf3f9055e5d00b1b6e/pandas/_libs/index.pyx#L660) return from hash with dulicate value -1, as NaN value and new value is same result from hashtable.
* Adding new value when have NaN in index will get the exception at [line 684 index.pyx `lab_int`](https://github.com/hksonngan/pandas/blob/7f7a12a0fe18c89a22b6f4bf3f9055e5d00b1b6e/pandas/_libs/index.pyx#L684) was not calculated for NaN value at class MultiIndex with function [`_engine()` line 1234 ](https://github.com/hksonngan/pandas/blob/7f7a12a0fe18c89a22b6f4bf3f9055e5d00b1b6e/pandas/core/indexes/multi.py#L1234). So I compensate with position of NaN in `levels` from line [688 index.pyx](https://github.com/hksonngan/pandas/blob/7f7a12a0fe18c89a22b6f4bf3f9055e5d00b1b6e/pandas/_libs/index.pyx#L688), I raise exception to call function `_hasnans`, and recalculate position of NaN in function [`_engine()` line 1234 ](https://github.com/hksonngan/pandas/blob/7f7a12a0fe18c89a22b6f4bf3f9055e5d00b1b6e/pandas/core/indexes/multi.py#L1234)
* When MultiIndex initialize, also call [`function _hasnans`](https://github.com/hksonngan/pandas/blob/7f7a12a0fe18c89a22b6f4bf3f9055e5d00b1b6e/pandas/core/indexes/multi.py#L752), because I want to cache this value. If the new index `NaN`, I can raise exception in [`get_loc()`](https://github.com/hksonngan/pandas/blob/7f7a12a0fe18c89a22b6f4bf3f9055e5d00b1b6e/pandas/_libs/index.pyx#L689) as above.
close as I will investigate another way to solve this