Scaling conditional random fields for natural language processing

This thesis deals with the use of Conditional Random Fields (CRFs; Lafferty et al. (2001)) for Natural Language Processing (NLP). CRFs are probabilistic models for sequence labelling which are particularly well suited to NLP. They have many compelling advantages over other popular models such as HiddenMarkovModels andMaximum Entropy Markov Models (Rabiner, 1990; McCallum et al., 2001), and have been applied to a number of NLP tasks with considerable success (e.g., Sha and Pereira (2003) and Smith et al. (2005)). Despite their apparent success, CRFs suffer from two main failings. Firstly, they often over-fit the training sample. This is a consequence of their considerable expressive power, and can be limited by a prior over the model parameters (Sha and Pereira, 2003; Peng and McCallum, 2004). Their second failing is that the standard methods for CRF training are often very slow, sometimes requiring weeks of processing time. This efficiency problem is largely ignored in current literature, although in practise the cost of training prevents the application of CRFs to many new more complex tasks, and also prevents the use of densely connected graphs, whichwould allow formuch richer feature sets. This thesis addresses the issue of training efficiency. Firstly, we demonstrate that the asymptotic time complexity of standard training for a linear chain CRF is quadratic in the size of the label set, linear in the number of features and almost quadratic in the size of the training sample. The cost of inference in cyclic graphs, such as lattice structured Dynamic CRFs (Sutton et al., 2004), is even greater. The complexity of training limits the application of CRFs to large and complex tasks. We compare the accuracy of a number of popular approximate training techniques, which can greatly reduce the training cost. However, for most tasks this saving is coupled with a substantial loss in accuracy. For this reason we propose two novel training methods, which both reduce the resource requirements and improve the scalability of training, such that CRFs can be applied to substantially larger tasks. The first method uses error-correcting output coding, a method originally devised for classifier combination (Freund and Schapire, 1996). This method decomposes the task of training a multiclass CRF (modelling a problem with three or more labels) into a series of simpler binary labelling tasks. Each of these sub-tasks are modelled with a CRF and are trained in the standard manner. Overall, these constituent models are considerably faster to train than a full multiclass CRF; critically, this can also reduce the complexity of training. Once trained these constituent models can be combined to decode unlabelled test instances and this results in similar accuracy to standard training. We introduce a second alternative training method which uses feature constraints to improve the time cost of inference. These constraints tie groups of features in the model which are exploited in sum-product and max-product belief propagation (Pearl, 1988). This leads to faster training and decoding. Overall, even simple tying strategies