Fix potential UAF set up by RemoveAttribute.