ENH: support kind and na_position kwargs in Series.sort_index
State: closed

closes #13589
closes #13729

I'm waiting on adding the whatsnew entry.  I'm not sure if this is a BUG (the [documentation](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.sort_index.html) says these parameters exist) or an ENH (we're adding support for these parameters).  It depends on how you look at it!


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14445?src=pr) is 85.20% (diff: 100%)


> No coverage report found for **master** at 3d73d05.

> Powered by [Codecov](https://codecov.io?src=pr). Last update [3d73d05...b013103](https://codecov.io/gh/pandas-dev/pandas/compare/3d73d0501bacaeb2c3c1686c0e936cf3707887c2...b013103ee05398d164dc25c09455660ba711f375?src=pr)
pls add a whatsnew note, 0.19.1 is fine. (you can list both issues)

I think this only handles single Index for now, we should also deal with MultiIndex (see also the older PR for this https://github.com/pandas-dev/pandas/pull/13729/files)

@jorisvandenbossche that looks right. on merge let's make sure to close the other PR

ping on green.

For me it's good to have this merged, but note that it does not fully close the issue, as it's not yet solved for MultiIndex, so I would leave the issue open and clarify there. 
The DataFrame.sort_index can handle `na_position` for MultiIndex, so I suspect it should not be difficult to add this to Series.sort_index as well.

@jorisvandenbossche I overlooked that; thanks for pointing it out.  I've now added `na_position` to MultiIndex as well.  Thanks!

@jreback Moved to `other enhancements` in 0.20.0.

@brandonmburroughs thanks for the update. Last question: can you also add a test for `'na_position'` for the multi-index case
@jorisvandenbossche As I was adding the test, I realized that this doesn't seem to really use the `na_position` argument due to the way we sort the `MultiIndex`.  Whenever we create a `MultiIndex`, we store the labels as relative values.  For instance, if we have the following `MultiIndex`:  
```python
MultiIndex.from_tuples([[1, 1, 3], [1, 1, np.nan]], names=list('ABC'))
```
the values get stored as 
```python
MultiIndex(levels=[[1], [1], [3]],
           labels=[[0, 0], [0, 0], [0, -1]],
           names=[u'A', u'B', u'C'])
```

These label values are what get passed to the sorting algorithm for both [DataFrames](https://github.com/pandas-dev/pandas/blob/master/pandas/core/frame.py#L3334-L3337) and [Series](https://github.com/pandas-dev/pandas/blob/master/pandas/core/series.py#L1782).  Since the sorting only happens on the `labels`, it has no notion of the `NaN`.  Hence, even with the updated code, we get the following results for a Series:
```python
In [206]: mi = MultiIndex.from_tuples([[1, 1, 3], [1, 1, np.nan]], names=list('ABC'))

In [207]: s = Series([1, 2], mi)

In [208]: s.sort_index(na_position="last")
Out[208]: 
A  B  C  
1  1  NaN    2
      3      1
dtype: int64

In [209]: s.sort_index(na_position="first")
Out[209]: 
A  B  C  
1  1  NaN    2
      3      1
dtype: int64
```

As I did more digging, I realized the same thing happens for `DataFrame` and it would seem the `na_position` argument never really worked.

```python
In [1]: import numpy as np

In [2]: import pandas as pd

In [3]: pd.__version__
Out[3]: u'0.19.0'

In [4]: mi = pd.MultiIndex.from_tuples([[1, 1, 3], [1, 1, np.nan]], names=list('ABC'))

In [5]: df = pd.DataFrame([[1, 2], [3, 4]], mi)

In [6]: df.sort_index(na_position="first")
Out[6]: 
         0  1
A B C        
1 1 NaN  3  4
    3    1  2

In [7]: df.sort_index(na_position="last")
Out[7]: 
         0  1
A B C        
1 1 NaN  3  4
    3    1  2
```

I've done a bit of looking and I think it can be fixed by passing the values of the `MultiIndex` to sort rather than the `labels`.  Should I

1. Add the changes here to be reviewed?
2. Create a new pull request since it's kind of a separate problem?
3. Create an issue detailing this behavior and then create a pull request?
this has not been implemented afair for multi index; it's straightforward to do; you can create an issue about it

you will always sort labels; however (-1) is the nan indicator
@brandonmburroughs Let's just leave MultiIndex out of this PR then (so it can be merged), it indeed does not work for DataFrames as well (maybe you can add to the docstring of this keyword something like "Not implemented for MultiIndex").

>  I think it can be fixed by passing the values of the MultiIndex to sort rather than the labels

I would first open an issue about it (or look for an existing one, I would think we already have an issue about it) to discuss it, as this is not a simple change (maybe in code changes, but has implications for usage)


Regarding the MutliIndex sort, I don't directly find an issue about it explicitly, but see the discussion in https://github.com/pandas-dev/pandas/issues/14015, and an issue that followed from that discussion https://github.com/pandas-dev/pandas/issues/14672.
all of @jorisvandenbossche comments are good

multi index sorting is not well implemented except for lexsorting
let's open a new issue (maybe should instead be a master issue encompassing the other 2 mi sort issues)
Okay, thanks for all of the feedback @jreback and @jorisvandenbossche .  I created a new issue for `na_position` specifically, #14784 .  I also added a note in the `sort_index` docstring that `na_position` isn't implemented for `MultiIndex`.
@brandonmburroughs Thanks a lot!