Why Build Another Part-of-Speech Tagger ? A Minimalist Approach

We use a Dynamic Bayesian Network (DBN) to build a compact representation of the features relevant to Part-of-Speech (PoS) tagging (Word, Suffix, Prefix, Capitalization, Hyphen, Numeric and Previous Tag). The outcome is a flexible tagger (LegoTag) with state-of-the-art performance (3.6% error on a benchmark corpus). We explore the effect of radically reducing the size of feature vocabularies for Word and Suffix. We find that reducing the Suffix vocabulary to a linguistically motivated set, results in improved cross-corpora generalization. Furthermore, relying on function words alone is sufficient to achieve reasonable performance, but minimizing the vocabularies for both Word and Suffix results in degradation. 1 Part of Speech Tagging Many NLP applicationssss are faced with the dilemma whether to use statistically extracted or expert-selected features. There are good arguments in support of either view. Statistical feature selection does not require extensive use of human domain knowledge, while feature sets chosen by experts are more economical and generalize better to novel data. Most currently available PoS taggers perform with a high degree of accuracy. However, it appears that the success in performance can be overwhelmingly attributed to an acrossthe-board lexicalization of the task. Indeed, Charniak et al. [1993] notes that a simple strategy of picking the most likely tag for each word in a text leads to 90% accuracy. If so, it is not surprising that taggers using vocabulary lists, with number of entries ranging from 20k to 45k, perform well. Even though a unigram model achieves an overall accuracy of 90%, it relies heavily on lexical information and is next to useless on nonstandard texts that contain lots of domain-specific terminology. The lexicalization of the PoS tagging task comes at a price. Since word lists are assembled from the training corpus, they hamper generalization across corpora. In our experience, taggers trained on the Wall Street Journal (WSJ) perform poorly on novel text e.g. email or newsgroup messages (a.k.a. Netlingo). At the same time, alternative training data are scarce and expensive to create. This paper explores an alternative to lexicalization. Using linguistic knowledge, we construct a minimalist tagger with a small but efficient feature set, which maintains a reasonable performance across corpora. A look at the previous work on this task reveals that the unigram model is at the core of even the most sophisticated taggers. The bestknown rule-based tagger [Brill’94] works in two stages: it assigns the most likely tag to each word in the text; then, it applies transformation rules of the form “Replace tag X by tag Y in triggering environment Z”. The triggering environments span up to three sequential tokens in each direction and refer to words, tags or properties of words within the region. The Brill tagger achieves less than 3.5% error on the Wall Street Journal (WSJ) corpus. However, its performance depends on a comprehensive vocabulary (70k words) employed in the first stage. Statistical tagging is a classic application of Markov Models (MMs). Brants [2000] argues that second-order MMs can also achieve stateof-the-art accuracy, provided they are supplemented by smoothing techniques and mechanisms to handle unknown words. TnT handles unknown words by estimating the tag probability given the suffix of the unknown word and its capitalization. The reported 3.3% error for Trigrams 'n Tags (TnT) tagger on the WSJ (trained on 1 million words and tested on 10,000) appears to be a result of overfitting. Indeed, this is the maximum performance obtained by training TnT until only 2.9% of words are unknown in the test corpus. A simple examination of WSJ shows that such percentage of unknown words in the testing section (10% of WSJ corpus) requires simply building a unreasonably large lexicon of nearly all (about 44k) words seen in the training section (90% of WSJ), thus ignoring the danger of overfitting. Hidden MMs (HMMs) are trained on a dictionary with information about the possible PoS of words [Jelinek’85; Kupiec’92]. This means HMM taggers also rely heavily on lexical information. Obviously, PoS tags depend on a variety of sub-lexical features, as well as on the likelihood of tag/tag and tag/word sequences. In general, all existing taggers have incorporated such information to some degree. The Conditional Random Fields (CRF) model [Lafferty et al.’02] outperforms the HMM tagger on unknown words by extensively relying on orthographic and morphological features. It checks whether the first character of a word is capitalized or numeric; it also registers the presence of a hyphen and morphologically relevant suffixes (-ed, -ly, -s, -ion, -tion, -ity, -ies). The authors note that CRF-based taggers are potentially flexible because they can be combined with feature-induction algorithms. However, training is complex (AdaBoost + Forwardbackward) and slow (1000 iterations with optimized initial parameter vector; fails to converge with unbiased initial conditions). It is unclear what is the relative contribution of features in this model. The Maximum Entropy (MaxEnt) [Ratnaparkhi’96] tagger accounts for the joint distribution of PoS tags and features of a sentence with an exponential model. Its features are along the lines of the CRF model: • Does the token contain a capital letter; • Does the token contain a hyphen; • Does the token contain a number; • Frequent prefixes, up to 4 letters long; • Frequent suffixes, up to 4 letters long; In addition, Ratnaparkhi uses lexical information on frequent words in the context of five words. The sizes of the current word, prefix, and suffix lists were 6458, 3602 and 2925, respectively. These are supplemented by special Previous Word vocabularies. Features frequently observed in a training corpus are selected from a candidate feature pool. The parameters of the model are estimated using the computationally intensive procedure of Generalized Iterative Scaling to maximize the conditional probability of the training set given the model. MaxEnt tagger has 3.4% error rate. Our investigation examines to what extent the information carried by word lists can be subsumed under the information supplied by sublexical features. In order to address these issues we reuse the feature set of MaxEnt in a new model, which we subsequently minimize with the help of linguistically smart vocabularies. 2 PoS Tagging Bayesian Net This section presents our tagger, which combines the features suggested in the literature to date into a Dynamic Bayesian Network (DBN). We briefly introduce the essential aspects of DBNs here and refer the reader to a recent PhD thesis [Murphy’02] for an excellent survey. A DBN is a Bayesian network unwrapped in time, such that it can represent dependencies between variables at adjacent time slices. More formally, a DBN consists of two models B and B, where B defines the initial distribution over the variables at time 0, by specifying: • set of variables X1, . , Xn; • directed acyclic graph over the variables; • for each variable Xi , a table specifying the conditional probability of Xi given its parents in the graph Pr(XiPar{Xi}). The joint probability distribution over the initial state is ( ) { } ( ) ∏ = n 1 i i n 1 X Par X Pr X ,. , X Pr . The transition model B specifies the conditional probability distribution (CPD) over the state at time t given the state at time t-1 . B consists of: • a directed acyclic graph over the variables X1,. ,Xn and their predecessors