Admin interface for user bans should use raw_id_fields instead of loading all the users.