with_group_user=False isn't respected when passed with attach_perms=True to get_users_with_perms()

If attach_perms=True, get_users_with_perms() returns users AND groups with permissions on the object, even if with_group_user=False