Don't delete tags unnecessarily when using the managers set method.