Don't try to remove twice