When an attribute is removed -> remove it also from AttributeSets