ENH partial sorting for mi in sortlevel
State: closed

fixes #3984.

Still need to tweak the docstrings and stuff. But I think this is working.

cc @jtratner


Labels:
name: API Design
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Would it be more general to overload `level` using types rather then adding a keyword
in 3 places? Perhaps that way supporting sorting on a "slice" of levels rather then
introducing a kw to choose between "that one" and "that one and all those that follow"?

I was just concerned about keeping old (strange?) behavior, I don't follow how to do this with types, but definitely the kwargs are inelegant.

What I meant was  you can express the same operation and more
by accepting `slice(3,None)` or `slice(3,5)`, or correspondingly `(3,)` and `(3,5)`
as values for `level`. Using types as in, rather then integers, accept tuples/slices 
to express a range of levels to sort.

That's a nice idea. The issue is the current behaviour is to sort _all_ when you pass a single level e.g. sortlevel('A')... maybe this shouldn't be the default and we could do something more logical (like what you suggest)?

Actually this current behaviour is a little deranged... is sortlevels the only way to sort a mi (sort doesn't work!)...

You'll have to give examples, I'm not sure what you mean by deranged.
It doesn't look to me like it sorts all levels at all.

You probably tried the following already:

```
mi=pd.MultiIndex.from_product([['A0','A1','A2'],['B0','B1','B2'],['C0','C1','C2']])
list(mi.get_values())
list(mi.sortlevel(0)[0].get_values())
list(mi.sortlevel(1)[0].get_values())
list(mi.sortlevel(2)[0].get_values())
```

That sorts by one and only one level, and it looks stable wrt to the other levels doing so.
`mi.lexsort_depth` seems to be set to 0 when sorting on inner levels, which is right. 
all seems to be in order.

What are you after, preferably with explicit example?

"That sorts by one and only one level," here's an example, let's reverse:

```
In [8]: mi=pd.MultiIndex.from_product([['A2','A1','A0'],['B2','B1','B0'],['C2','C1','C0']])

In [10]: print mi.sortlevel(1)[0].get_values()
[('A0', 'B0', 'C0') ('A0', 'B0', 'C1') ('A0', 'B0', 'C2')
 ('A1', 'B0', 'C0') ('A1', 'B0', 'C1') ('A1', 'B0', 'C2')
 ('A2', 'B0', 'C0') ('A2', 'B0', 'C1') ('A2', 'B0', 'C2')
 ('A0', 'B1', 'C0') ('A0', 'B1', 'C1') ('A0', 'B1', 'C2')
 ('A1', 'B1', 'C0') ('A1', 'B1', 'C1') ('A1', 'B1', 'C2')
 ('A2', 'B1', 'C0') ('A2', 'B1', 'C1') ('A2', 'B1', 'C2')
 ('A0', 'B2', 'C0') ('A0', 'B2', 'C1') ('A0', 'B2', 'C2')
 ('A1', 'B2', 'C0') ('A1', 'B2', 'C1') ('A1', 'B2', 'C2')
 ('A2', 'B2', 'C0') ('A2', 'B2', 'C1') ('A2', 'B2', 'C2')]
```

now I see what you mean.

This is kindof documented in the DataFrame.sortlevel, but tbh this is more of an observation than a spec:

> Sort multilevel index by chosen axis and primary level. Data will be
> lexicographically sorted by the chosen level followed by the other
> levels (in order)

... :s

Yeah, it's a tossup on which docstring is more abstruse, `df.sortlevel` or `mi.sortlevel`.

You could have `level=(1,2)` can sort just level 1, `level=(1,3)` means sort levels [1,2) ,
and  `level=(1,)` can mean sort levels 1..end.

If you want to sort on multiple single levels, just do multiple invocations.

But now I understand why you chose to have the default True keyword. makes sense.

@hayd It's maybe also an option to fix this in `sort/sort_index` by providing there a level keyword? It does already work, but it just sorts all levels. And then you can do there the more 'logical' thing to only sort the level you specified?

@jorisvandenbossche I wasn't sure if this was part of @jtratner's work on refactoring index - and making the common methods.

+1 a level kwarg makes sense!

I was confused about what sort_index actually does, I thought it would sort by the index, but it appears to accept _columns_... i.e is just sort with an axis arg? 

@hayd related is #5190 

hows this coming?

@jreback Is the remaining thing here to add a level (and sort_remaining?) arg to sort and sort_index ?

I think this is fine...wouldn't really change sort/sort_index ATM.

ping me to look at this in a day or 2

ping

this was nice.... any other possibilites besides `sort_remaining`?

Not sure, happy to change, what's your issue with sort_remaining?

(sort_remaining_levels ?)

just sounds odd - though can't think of anything better

you are defaulting this to true right?

yeah, defaults to true so doesn't break backwards compat... (though perhaps we will later)

maybe I think it's a perf issue to completely sort 

@jreback shall we get this in?

looks fine

release note / v0.14.0 mention

merge when ready

I don't think docs are really necessary as the do string covers

looks fine....just check release notes / v0.14

:+1: from me as well. In addition to the minor suggestion on the docstring, should we add a warning about the performance hit from non-lexsorted `MultiIndex`? (either via `level` != `0` or `sort_remaining=False`)

ping!

ping

@hayd ping....need to get this in ASAP

ping!