pydata/pandas — pull request #5231 by unutbu (2013-10-15 17:15:17+00:00)
Title: EHN/FIX: Add na_last parameter to DataFrame.sort. Fixes GH3917

This is an attempt to fix the Nested Sort with NaN bug (https://github.com/pydata/pandas/issues/3917).

I've added tests to `test_frame.py` and `test_hashtable.py` to demonstrate the problem.

`hashtable.Factorizer.factorize` has been modified to map `nan` to `na_sentinel`. Before it was mapping `nan` to a label which was already being used. This, I believe is the origin of the bug.

My first idea was to mimick/reuse code from `Series.order`, since this method already handles `nan`s nicely, and allows the user to choose if nans should be placed at the beginning or the end of the sort via the `na_last` parameter. Although I found a solution using code from `Series.order`, I eventually abandoned this when I realized the problem this patches the problem at too high a level and that it could be handled more generally with a modification of `factorize`. I retained the idea that `df.sort` should have a `na_last` parameter, however.

To that end, `groupby._lexsort_indexer` has been modified to handle all possible combinations of `na_last` and `orders` settings. There are four tests (assertions) in `test_frame.py` to exercise the possibilities, one of which demonstrates that `df.sort(['A','B'])` now behaves correctly for the DataFrame shown in GH3917.