BUG: groupby with a Float like index misbehaving when the index is non-monotonic (related GH5375)

related #5375