BUG/TST: test for groupby max nan subselection bug

closes #6346