Remove group members from group properly