Only remove a tag from the server when it is deleted from its listing