Only set updated_at when setting or unsetting.