[Fix bug 980887] Speed up browsing large groups

Refactor code so we can always give the paginator a queryset without having to iterate through the whole thing first.