Don't try to remove if already removed