From what I tested there is no need for a user to have `administer taxonomy` in order to delete a term.