Fix modify/remove contact-group

Without this groups only get added, but not removed.