More security on add/remove

Make sure that the people being added/removed are actually in the group.