Check if contact is valid when removing from group