BUG: MultiIndex not dropping nan level and invalid code value
State: closed

- [x] closes #19387
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Please take note that another bug is also fixed in this PR:

When the MultiIndex is constructed with code value < -1, a segmentation fault would be resulted in subsequent operations e.g. putting the MultiIndex in a DataFrame
Input:
```python
x = pd.MultiIndex(levels=[['a'], ['x', np.nan]], labels=[[0,-2], [0,1]])
pd.DataFrame(index=x)
```
A segmentation fault can be resulted by these commands


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: MultiIndex

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26408?src=pr&el=h1) Report
> Merging [#26408](https://codecov.io/gh/pandas-dev/pandas/pull/26408?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/3b24fb678672ecb6ef24a1915c49f3cf815d5556?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26408/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26408?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26408      +/-   ##
==========================================
- Coverage   91.69%   91.68%   -0.01%     
==========================================
  Files         174      174              
  Lines       50743    50749       +6     
==========================================
+ Hits        46527    46530       +3     
- Misses       4216     4219       +3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.19% <100%> (ø)` | :arrow_up: |
| #single | `41.18% <83.33%> (-0.15%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26408?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `95.64% <100%> (+0.02%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.02% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.7% <0%> (+0.1%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26408?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26408?src=pr&el=footer). Last update [3b24fb6...e0d3bb4](https://codecov.io/gh/pandas-dev/pandas/pull/26408?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26408?src=pr&el=h1) Report
> Merging [#26408](https://codecov.io/gh/pandas-dev/pandas/pull/26408?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/3b24fb678672ecb6ef24a1915c49f3cf815d5556?src=pr&el=desc) will **increase** coverage by `0.14%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26408/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26408?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26408      +/-   ##
==========================================
+ Coverage   91.69%   91.84%   +0.14%     
==========================================
  Files         174      174              
  Lines       50743    50658      -85     
==========================================
- Hits        46527    46525       -2     
+ Misses       4216     4133      -83
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.38% <100%> (+0.18%)` | :arrow_up: |
| #single | `41.7% <55%> (+0.37%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26408?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `95.67% <100%> (+0.04%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/dtypes/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbW1vbi5weQ==) | `95.85% <0%> (-1.56%)` | :arrow_down: |
| [pandas/core/indexes/timedeltas.py](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy90aW1lZGVsdGFzLnB5) | `90.96% <0%> (-0.63%)` | :arrow_down: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `96.36% <0%> (-0.49%)` | :arrow_down: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `93.54% <0%> (-0.4%)` | :arrow_down: |
| [pandas/core/indexes/frozen.py](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9mcm96ZW4ucHk=) | `91.78% <0%> (-0.33%)` | :arrow_down: |
| [pandas/io/excel/\_base.py](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsL19iYXNlLnB5) | `91.81% <0%> (-0.33%)` | :arrow_down: |
| [pandas/plotting/\_style.py](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19zdHlsZS5weQ==) | `76.92% <0%> (-0.26%)` | :arrow_down: |
| [pandas/plotting/\_misc.py](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19taXNjLnB5) | `38.23% <0%> (-0.23%)` | :arrow_down: |
| ... and [53 more](https://codecov.io/gh/pandas-dev/pandas/pull/26408/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26408?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26408?src=pr&el=footer). Last update [3b24fb6...a141e40](https://codecov.io/gh/pandas-dev/pandas/pull/26408?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Should we reserve negative codes for use by pandas, like we do for Categorical?
> Should we reserve negative codes for use by pandas, like we do for Categorical?

yes
@jreback 
This is the asv report
      
 before           after         ratio
     [3b24fb67]       [af35fa32]
     <tmp~7>          <multi~1> 
+         413±2μs         916±50μs     2.22  multiindex_object.Duplicates.time_remove_unused_levels
+        440±30μs         903±20μs     2.05  indexing.MultiIndexing.time_frame_ix
+        407±20μs         830±10μs     2.04  indexing.MultiIndexing.time_series_ix
+         187±1ms          341±2ms     1.82  stat_ops.SeriesMultiIndexOps.time_op([0, 1], 'skew')
+         783±3ms       1.41±0.01s     1.81  stat_ops.FrameMultiIndexOps.time_op([0, 1], 'kurt')
+       199±0.6ms          353±2ms     1.77  stat_ops.SeriesMultiIndexOps.time_op([0, 1], 'kurt')
+      94.9±0.8ms          161±1ms     1.70  stat_ops.FrameMultiIndexOps.time_op(1, 'kurt')
+      24.6±0.5ms       40.9±0.3ms     1.66  stat_ops.SeriesMultiIndexOps.time_op(1, 'skew')
+      26.0±0.5ms       42.1±0.5ms     1.62  stat_ops.SeriesMultiIndexOps.time_op(1, 'kurt')
+      68.2±0.4ms          101±1ms     1.48  stat_ops.SeriesMultiIndexOps.time_op(1, 'mad')
+         622±1ms          917±4ms     1.48  stat_ops.SeriesMultiIndexOps.time_op([0, 1], 'mad')
+         321±2μs          457±2μs     1.43  multiindex_object.Values.time_datetime_level_values_sliced
+         379±7ms          534±2ms     1.41  stat_ops.FrameMultiIndexOps.time_op([0, 1], 'skew')
+        1.23±0ms      1.73±0.01ms     1.41  reshape.SparseIndex.time_unstack
+      24.5±0.2ms       34.1±0.2ms     1.39  stat_ops.FrameMultiIndexOps.time_op(0, 'kurt')
+      6.50±0.2ms       8.77±0.3ms     1.35  stat_ops.SeriesMultiIndexOps.time_op(0, 'skew')
+     6.96±0.08ms       9.39±0.1ms     1.35  stat_ops.SeriesMultiIndexOps.time_op(0, 'kurt')
+      51.4±0.6ms       68.6±0.4ms     1.33  stat_ops.FrameMultiIndexOps.time_op(1, 'skew')
+      13.0±0.5ms       17.1±0.2ms     1.32  stat_ops.SeriesMultiIndexOps.time_op(0, 'mad')
+         957±6μs      1.22±0.01ms     1.28  groupby.GroupByMethods.time_dtype_as_group('float', 'value_counts', 'transformation')
+         957±9μs      1.21±0.01ms     1.27  groupby.GroupByMethods.time_dtype_as_group('float', 'value_counts', 'direct')
+     1.08±0.01ms      1.36±0.01ms     1.25  groupby.GroupByMethods.time_dtype_as_field('float', 'value_counts', 'transformation')
+     3.76±0.03ms      4.71±0.05ms     1.25  period.DataFramePeriodColumn.time_set_index
+     1.09±0.01ms      1.35±0.01ms     1.25  groupby.GroupByMethods.time_dtype_as_field('float', 'value_counts', 'direct')
+         931±5μs         1.16±0ms     1.24  groupby.GroupByMethods.time_dtype_as_field('datetime', 'value_counts', 'direct')
+         211±6μs        262±0.9μs     1.24  indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.Int8Engine'>, <class 'numpy.int8'>), 'monotonic_incr')
+         940±2μs      1.16±0.01ms     1.23  groupby.GroupByMethods.time_dtype_as_field('datetime', 'value_counts', 'transformation')
+         207±3μs        253±0.9μs     1.22  reindex.Fillna.time_float_32('pad')
+         955±6μs      1.17±0.02ms     1.22  groupby.GroupByMethods.time_dtype_as_group('datetime', 'value_counts', 'direct')
+         957±9μs      1.17±0.02ms     1.22  groupby.GroupByMethods.time_dtype_as_group('datetime', 'value_counts', 'transformation')
+        1.51±0ms      1.82±0.02ms     1.20  sparse.FromCoo.time_sparse_series_from_coo
+     7.80±0.05ms      9.37±0.04ms     1.20  ctors.MultiIndexConstructor.time_multiindex_from_iterables
+         238±8μs          282±1μs     1.19  indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.Int16Engine'>, <class 'numpy.int16'>), 'monotonic_incr')
+         945±2μs      1.11±0.01ms     1.18  groupby.GroupByMethods.time_dtype_as_group('int', 'value_counts', 'direct')
+        756±20μs          888±5μs     1.17  groupby.GroupByMethods.time_dtype_as_group('object', 'value_counts', 'transformation')
+         267±3μs          313±3μs     1.17  reindex.Fillna.time_reindexed('backfill')
+         873±6μs         1.01±0ms     1.16  groupby.GroupByMethods.time_dtype_as_field('object', 'value_counts', 'transformation')
+         275±4μs          318±3μs     1.16  reindex.Fillna.time_reindexed('pad')
+         948±3μs      1.10±0.01ms     1.16  groupby.GroupByMethods.time_dtype_as_group('int', 'value_counts', 'transformation')
+        984±20μs         1.13±0ms     1.15  groupby.GroupByMethods.time_dtype_as_field('int', 'value_counts', 'direct')
+         289±1μs          331±4μs     1.14  indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.UInt32Engine'>, <class 'numpy.uint32'>), 'monotonic_incr')
+         424±5ms          483±2ms     1.14  groupby.Transform.time_transform_lambda_max
+      13.0±0.2ms      14.7±0.05ms     1.14  rolling.Pairwise.time_pairwise(1000, 'cov', True)
+         216±1μs         245±10μs     1.13  indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.UInt8Engine'>, <class 'numpy.uint8'>), 'monotonic_incr')
+      13.0±0.1ms      14.7±0.07ms     1.13  rolling.Pairwise.time_pairwise(10, 'cov', True)
+     6.78±0.04ms      7.64±0.04ms     1.13  groupby.Transform.time_transform_multi_key2
+        789±20μs          884±7μs     1.12  groupby.GroupByMethods.time_dtype_as_group('object', 'value_counts', 'direct')
+     5.75±0.02ms      6.42±0.02ms     1.12  multiindex_object.Integer.time_get_indexer
+     3.30±0.01ms      3.69±0.03ms     1.12  reshape.SimpleReshape.time_stack
+     1.81±0.01ms      2.02±0.01ms     1.12  reshape.SimpleReshape.time_unstack
+     14.6±0.09ms       16.2±0.1ms     1.11  rolling.Pairwise.time_pairwise(10, 'corr', True)

@jiangyue12392 on the asv's

so looks like somewhat slower, can you see if we are *always* checking codes? (I think we are checking twice sometims, and shouldn't check unless verify_integrity=True).

in the tests you may need to test with verfiy_integrity =True as well as the default (False)
how’s perf now? (you can just show some before / after with a sample benchmark that showed up before)
@jreback 
The new asv results on mutiindex related objects are better. The report said "BENCHMARKS NOT SIGNIFICANTLY CHANGED". For example, multiindex_object.Duplicates.time_remove_unused_levels that deteriorates in the previous run is now almost the same for this branch and the master


great will look soon
thanks @jiangyue12392 this was very nice! a non-trivial change and were very responsive! keep em coming!