Only remove labels when they already exist

Prevents nonsensical history entry