Ensure that relationships are properly updated for an object matching a uniqueness constraint

Previously, objects were added/updated, but were not not deleted if not
present in the model's relationship anymore.