As a surrogate, one may compare the cost of assigning tags with that of simply extracting words from text tokenization . A tagger should correctly tag text previously unseen by the system. It must accommodate previously unseen words, as unseen texts frequently contain unseen words. New grammatical constructions may also be encountered. Ideally a tagger will accommodate these too.
Texts vary in things as mundane as typographic conventions and as fundamental as natural languages.
Different tasks may require different tagsets, e.g., some may need to distinguish subject and object pronouns, while others may not. Transition probabilities are recorded for sequences of tags, usually pairs, and indicate the probability of that sequence occurring in text, e.g. the probability that a determiner is followed by a noun. The tag assignment with the highest such product is selected as most likely. A weakness with this approach is that symbol probabilities are difficult to estimate for words. A substantial portion of text is composed of low-frequency words.