libcontacts] Ensure contacts are not in name groups on cache removal