This doesn't work. I tested it out and you actually need to do this:
Group.objects.annotate(num_members=Count('members')).order_by(sort_by) and change the sort_choices to "-num_members" and "num_members" respectively.