Raising the Bar: Stacked Conservative Error Correction Beyond Boosting

We introduce a conservative error correcting model, Stacked TBL, that is designed to improve the performance of even high-performing models like boosting, with little risk of accidentally degrading performance. Stacked TBL is particularly well suited for corpus-based natural language applications involving high-dimensional feature spaces, since it leverages the characteristics of the TBL paradigm that we appropriate. We consider here the task of automatically annonating named entities in text corpora. The task does pose a number of challenges for TBL, to which there are some simple yet effective solutions. We discuss the empirical behavior of Stacked TBL, and consider evidence that despite its simplicity, more complex and time-consuming variants are not generally required. 1. Setting the Bar: Introduction In this paper we develop a general stacking-based method called Stacked TBL (STBL) that error-corrects the output of a boosting model that is already highly tuned. We deploy TBL in an unconventional fashion, and discuss motivation and evidence to support its use. Several modifications to the traditional TBL procedure are required, but the revised procedure remains relatively simple. To demonstrate the applicability of STBL, we construct a base model trained using the AdaBoost boosting algorithm (Freund and Schapire, 1997). Boosting has acquired a superior reputation for error driven learning of ensemble models and, when used in corpus-based NLP systems, typically finds a place as the ultimate stage. Two of the best-performing three teams in the CoNLL-2002 Named Entity Recognition shared task evaluation used boosting as their base system (Carreras et al., 2002; Wu et al., 2002). However, we have found that, like all learning models, even boosting models can and do reach certain limits that other models are less susceptible to. This holds even after careful feature engineering to compensate is carried out. We are thus driven to investigate the problem of correcting errors after boosting has done its best. This establishes a high bar for any model stacked on the boosting base model, because it is difficult to correct the few remaining errors without also accidentally undoing correct classifications at the same time. In the following sections, we first define the particular stacking approach we will use. Subsequently we describe our model in detail, and analyze issues that arise from repurposing TBL for this task. We discuss the principles behind our proposed solutions, and demonstrate the The author would like to thank the Hong Kong Research Grants Council (RGC) for supporting this research in part through research grants RGC6083/99E, RGC6256/00E, and DAG03/04.EG09. The author would like to thank the Hong Kong Polytechnic University for supporting this research in part through research grants A-PE37 and 4-Z03S. method’s empirical behavior. Finally, we consider evidence that more complicated and time-consuming alternative variants of STBL are unnecessary in practice. 2. Piping versus Stacking Stacking has become widely used since its introduction a decade ago (Wolpert, 1992). However, a few words of clarification on stacking are in order since it is an extremely general concept, that is often confusingly used to lump together various approaches that are in fact methodologically quite different. The major division of stacking approaches is between (1) those employing multiple heterogenous base learning models, and (2) those employing a single base learning model. The former case is a general alternative to simple voting among heterogenous base models. To avoid confusion we will use the term piping for the latter case, i.e., stacking with a single base learning model. In this paper we will restrict our attention to piping, specifically using a boosting base model. Usage of piping has two common subcases. First, piping provides a kind of arcing framework (Breiman, 1998), and as such is an alternative to simple bagging models. As an arcing procedure, however, its use appears to have been eclipsed in recent years by boosting. Second, for some tasks, piping is effective as a sequentially chained error correcting ensemble. The investigation in this paper falls in this category. Piped classifiers differ from cascaded classifiers (e.g., Alpaydin (1998)) in several important respects, one of which is that in cascading, confidence scores are not assigned to the predictions of the earlier classifiers. A cascaded classifier only attempts to classify examples that earlier stages have voluntarily passed on; it does not identify where errors are likely to have occurred and therefore is not an error corrector per se. Piping can, however, be viewed as a combination of cascading and confidence prediction. In this view, the error correcting stage is responsible for both predicting the confidence on each example that was output by the previous stage, as well as performing corrections on low-confidence examples. In fact, to be more precise, it is not actually necessary for the error correcting stage to predict an absolute confidence score for the base model’s output. It merely needs to predict with high confidence when the error corrector’s confidence score is relatively higher than the base model’s. Piping, broadly interpreted, is widespread in corpusbased NLP. An example of piping in an NER application very similar to the experiments described later in this paper is Florian (2002), who used a TBL base model piped to a forward-backward decoder model. However, narrowly interpreted piping, which employs correct and sound stacking training procedures, is rather less common, perhaps especially in NLP work. 3. Raising the Bar: Repurposing TBL for Error Correction Certain kinds of models are better suited to error correction than others. This is particularly true when the performance of the base model is already high. The error corrector model must (1) have characteristics that vary sufficiently from the base model so that the corrector will make a significant difference, and (2) be excellent at “leaving well enough alone” so as not to miscorrect the already highly accurate predictions from the boosting model. We consider transformation-based learning to be a reasonable candidate for error correction, and we call this model Stacked TBL (STBL). The odd thing is that although it is inherently an error-correcting paradigm, TBL has not received much attention as an error-corrector for models that already achieve high performance. Instead, TBL is traditionally used by itself, trained by correcting the output of relatively “dumb” base models. Even less attention has been given to using TBL as an error corrector for highperformance models.1 Nevertheless, TBL appeared to fit the bill since (1) its sequential processing characteristics are very different from the boosting model’s, and (2) it can be modified to leave well enough alone, as we shall see in the subsequent section. Transformation-based learning was first introduced by Brill (1995) for part-of-speech tagging, and it has since been applied to a wide range of corpus-based NLP tasks, including parsing (Brill, 1996), noun phrase chunking (Ramshaw and Marcus, 1999), phrase chunking (Florian et al., 2000), and dialog act tagging (Samuel et al., 1998). It is a flexible model which is easily extensible to various tasks, and it has the advantage of being able to achieve state-ofthe-art performance with a small set of perspicious rules. In some ways, TBL is similar to boosting in that it is an iterative process in which each iteration targets the residual error from previous iterations. A traditional TBL system is trained using the following algorithm: 1. Create an initial assignment of classifi cations using simple