na_position doesn't work for sort_index() with MultiIndex
State: closed

 - [X] closes #14784 
 - [X] tests added / passed
 - [X] passes ``git diff upstream/master --name-only -- '*.py' | flake8 --diff``
 - [ ] whatsnew entry


Labels:
name: Bug
name: MultiIndex

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=h1) Report
> Merging [#15845](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/de589c23e0ae79d9cae59674259dc2707513795f?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/15845/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #15845      +/-   ##
==========================================
- Coverage   90.97%   90.96%   -0.02%     
==========================================
  Files         143      143              
  Lines       49442    49446       +4     
==========================================
- Hits        44982    44977       -5     
- Misses       4460     4469       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.72% <ø> (ø)` | :arrow_up: |
| #single | `40.69% <ø> (-0.12%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.57% <ø> (-0.1%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=footer). Last update [de589c2...e56a524](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=h1) Report
> Merging [#15845](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/f114af045f68ed960cba02d234a959301ad97a79?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/15845/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #15845      +/-   ##
==========================================
- Coverage   90.79%   90.79%   -0.01%     
==========================================
  Files         156      156              
  Lines       50534    50537       +3     
==========================================
+ Hits        45883    45885       +2     
- Misses       4651     4652       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.56% <100%> (-0.01%)` | :arrow_down: |
| #single | `40.44% <20%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.97% <100%> (ø)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.65% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `96.71% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.33% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=footer). Last update [f114af0...66f809e](https://codecov.io/gh/pandas-dev/pandas/pull/15845?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@linebp I'd prefer if you did something like I illustrated here: https://github.com/pandas-dev/pandas/issues/14784#issuecomment-287362444

Put these tests (and on Series as well) in ``test_multilevel.py`` where these types of things live. Further Ideally we'd do a lower-level test in ``pandas/tests/indexes/tests/test_multi.py`` and/or ``pandas/tests/tests_sorting.py``
I am not sure what I am missing, I tried to do as suggested. These are my steps figuring out what to do:
* I looked for a function called `_lexsort_indexer` and didn't manage to locate it. 
* I figured what you meant was the function `lexsort_indexer`, which made sense to me, as this function is called while sorting the index.
* The `lexsort_indexer` is called directly from `sort_index`, no code in MultiIndex is called(?)
* I put the fix in `sort_index` as that is the function calling `lexsort_indexer`. This made good sense to me, as `lexsort_indexer` is not actually broken, but the data passed to it is. 

Where and how did I go wrong?

I'll move the tests to a more appropriate location! ... and test the Series sorting too.
so, add the `is_lexsorted` check to both series and frames `sort_index` right before call to `_lexsort_indexer`? What is the reason for not adding the check to the `_lexsort_indexer` function?

Panels seems to have a function similar to `sort_index` on frames and series, is that in the scope of this bug?

If you use the `level` option when calling `sort_index`, the `na_position` option is ignored.  Should this be fixed to?
@linebp this needs to be very targeted. don't touch Panel and don't worry about level.

instead of a``labels.labels``, call ``labels._get_labels_for_sorting()``

it is *very* easy to break things w/o very specific patches.
@linebp maybe I wasn't clear. These ALL the existing code, except for

``indexer = lexsort_indexer(index._get_labels_for_sorting(), orders=ascending)``

where ``_get_labels_for_sorting()`` is the routine that you added ``_lexsort_indexer``

so you are adding 1 function to MultiIndex, and changing 1 line in series & 1 line in frame.py thats it.
I am still a bit confused about what to do, but this adds 1 function and only changes 1 line in series and frame each. No comments on the tests, so I let those stay as is.
ok, pls rebase this as I pushed in a bunch of fixes for sorting (generally)
I made an effort to rebase, how is it looking now?
@linebp code looks good. pls add a whatsnew (bug fix). make sure you rebase on origin (there are still conflicts showing)

```
git rebase -i thisbranch origin/master
```

then push with ``-f`` to update.
It fails when `Categorical.from_codes()` is called on an index where some rows have been removed, for example with `dropna()`. The index is not relabelled and so the labels may be outside the -1 to len(categories) range. Example:
 
```
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'date': pd.to_datetime([
        '20121002', '20121007', '20130130', '20130202', '20130305',
        '20121002', '20121207', '20130130', '20130202', '20130305',
        '20130202', '20130305'
    ]),
    'user_id': [1, 1, 1, 1, 1, 3, 3, 3, 5, 5, 5, 5],
    'whole_cost': [1790, np.nan, 280, 259, np.nan, 623, 90, 312, np.nan, 301,
                   359, 801],
    'cost': [12, 15, 10, 24, 39, 1, 0, np.nan, 45, 34, 1, 12]
}).set_index(['date', 'user_id'])

df.dropna().sort_index()

``` 
@linebp I pushed a fix to your branch. can you add the above as an additional tests. ping on green.