Fixed #175; remove user association with groups after declining invite.