Venue: EMNLP 2017

# Learning the Structure of Variable-Order CRFs: a Finite-State Perspective

Thomas Lavergne

Francois Yvon

LIMSI, CNRS, Univ. Paris-Sud, Universite Paris Saclay

Campus Universitaire, F-91 403 Orsay, France

{lavergne,yvon}@limsi.fr

###### Abstract

The computational complexity of linear-chain Conditional Random Fields (CRFs) makes it difficult to deal with very large label sets and long range dependencies. Such situations are not rare and arise when dealing with morphologically rich languages or joint labelling tasks. We extend here recent proposals to consider variable order CRFs. Using an effective finite-state representation of variable-length dependencies, we propose new ways to perform feature selection at large scale and report experimental results where we outperform strong baselines on a tagging task.

## 1 Introduction

Conditional Random Fields (CRFs) [8, 2] are a method of choice for many sequence labelling tasks such as Part of Speech (PoS) tagging, Text Chunking, or Named Entity Recognition. Linear-chain CRFs are easy to train by solving a convex optimization problem, can accomodate rich feature patterns, and enjoy polynomial exact inference procedures. They also deliver state-of-the-art performance for many tasks, sometimes surpassing seq2seq neural models [5].

A major issue with CRFs is the complexity of training and inference procedures, which are quadratic in the number of possible output labels for first order models and grow exponentially when higher order dependencies are considered. This is problematic for tasks such as precise PoS tagging for Morphologically Rich Languages (MRLs), where the number of morphosyntactic labels is in the thousands [6, 7]. Large label sets also naturally arise when _joint labelling tasks_ (eg. simultaneous PoS tagging and text chunking) are considered, For such tasks, processing first-order models is demanding, and full size higher-order models are out of the question. Attempts to overcome this difficulty are based on a greedy approach which starts with first-order dependencies between labels and iteratively increases the scope of dependency patterns under the constraint that a high-order dependency is selected only if it extends an existing lower order feature [7]. As a result, feature selection may only choose only few higher-order features, motivating the need for an effective variable-order CRF (voCRF) training procedure [10].1 The latest implementation of this idea [10] relies on (structured) sparsity promoting regularization [11] and on finite-state techniques, handling high-order features at a small extra cost (see SS 2). In this approach, the sparse set of label dependency patterns is represented in a finite-state automaton, which arises as the result of the feature selection process.

Footnote 1: This is reminiscent of _variable order HMMs_, introduced eg. in [5, 12].

In this paper, we somehow reverse the perspective and consider VoCRF training mostly as an automaton inference problem. This leads us to consider alternative techniques for learning the finite-state machine representing the dependency structure of sparse VoCRFs (see SS 3). Two lines of enquiries are explored: (a) to take into account the internal structure of large tag sets in order to learn better and/or leaner feature sets; (b) to detect unconditional structural dependencies in label sequences in order to speed-up the discovery of useful features. These ideas are implemented in \(6\) feature selection strategies, allowing us to explore a large set of dependency structures. Relying on lazy finite-state operations, we train VoCRFs up to order 5, and achieve PoS tagging performance that

surpass strong baselines for two MRLs (see SS 4).

## 2 Variable order CRFs

In this section, we recall the basics of CRFs and VoCRFs and introduce some notations.

### Basics

First-order CRFs use the following model:

\[p_{\theta}(\mathbf{y}|\mathbf{x})=Z_{\theta}(\mathbf{x})^{-1}\exp(\theta^{T}F( \mathbf{x},\mathbf{y})) \tag{1}\]

where \(\mathbf{x}=(x_{1},\ldots,x_{T})\) and \(\mathbf{y}=(y_{1},\ldots,y_{T})\) are the input (in \(\mathcal{X}^{T}\)) and output (in \(\mathcal{Y}^{T}\)) sequences and \(Z_{\theta}(\mathbf{x})\) is a normalizer. Each component \(F_{j}(\mathbf{x},\mathbf{y})\) of the global feature vector decomposes as a sum of local features \(\sum_{t=1}^{T}f_{j}(y_{t-1},y_{t},x_{t})\) and is associated to parameter \(\theta_{j}\). Local features typically use binary tests and take the form:

\[f_{u,g}(y_{t-1},y_{t},x,t)=\mathbb{I}(y_{t}=u\wedge g(x,t))\] \[f_{uv,g}(y_{t-1},y_{t},x,t)=\mathbb{I}(y_{t-1}y_{t}=uv\wedge g( x,t))\]

where \(\mathbb{I}()\) is an indicator function and \(g()\) tests a local property of \(\mathbf{x}\) around \(x_{t}\). In this setting, the number of parameters is \(|\mathcal{Y}|^{2}\times|\mathcal{X}|_{\text{train}}\), where \(|A|\) is the cardinality of \(A\) and \(|\mathcal{X}|_{\text{train}}\) is the number of values of \(g(x,t)\) observed in the training set. Even in moderate size applications, the parameter set can be very large and contain dozen of millions of features, due to the introduction of sequential dependencies in the model.

Given \(N\) i.i.d. sequences \(\{\mathbf{x}^{(i)},\mathbf{y}^{(i)}\}_{i=1}^{N}\), estimation is based on the minimization of the negated conditional log-likelihood \(l(\theta)\). Optimizing this objective requires to compute its gradient and to repeatedly evaluate the conditional expectation of the feature vector. This can be done using a forward-backward algorithm having a complexity that grows quadratically with \(|\mathcal{Y}|\). \(l(\theta)\) is usually complemented with a regularization term so as to avoid overfitting and stabilize the optimization. Common regularizers use the \(\ell_{1}\)- or the \(\ell_{2}\)-norm of the parameter vector, the former having the benefit to promote sparsity, thereby performing automatic feature selection (Tibshirani, 1996).

```
\(\mathcal{W}:\) list of patterns, \(A[\mathcal{W}]\) initially empty \(\mathcal{U}=\texttt{Pref}(\mathcal{W})\) foreach\(w\in\mathcal{W}\)do  \(\textsc{TrieInsert}(w,A[\mathcal{W}])\)// Add missing transitions foreach\(u=vy\in\mathcal{U}\)do    new FailureTrans\((u,\textsc{LqSuff}(v,\mathcal{U}))\)
```

**Algorithm 1**Building \(A[\mathcal{W}]\)

### Variable order CRFs (VoCRFs)

When the label set is large, many pairs of labels never occur in the training data and the sparsity of label ngrams quickly increases with the order \(p\) of the model. In the _variable order CRF_ model, it is assumed that only a small number of ngrams (out of \(|\mathcal{Y}|^{p}\)) are associated with a non-zero parameter value. Denoting \(\mathcal{W}\) the set of such ngrams and \(w\in\mathcal{W}\), a generic feature function is then \(f_{w,g}(w,x,t)=\mathbb{I}(y_{t-s}\dots y_{t}=w\wedge g(x,t))\).

In (order-\(p\)) VoCRFs, the computational cost of training and inference is proportional to the size of a finite-state automaton \(A[\mathcal{W}]\) encoding the patterns in \(\mathcal{W}\),2 which can be much less than \(|\mathcal{Y}|^{p}\). Our procedure for building \(A[\mathcal{W}]\) is sketched in Algorithm 1, where TrieInsert inserts a string in a trie, Pref\((\mathcal{W})\) computes the set of prefixes of the strings in \(\mathcal{W}\),3 LgSuff\((v,\mathcal{U})\) returns the longest suffix of \(v\) in \(\mathcal{U}\), and FailureTrans is a special \(\varepsilon\)-transition used only when no labelled transition exists (Allauzen et al., 2003).4 Each state (or pattern prefix) \(v\) in \(A[\mathcal{W}]\) is associated with a set of feature functions \(\{f_{u,g},\forall u\in\mathtt{Suff}(v),g\}\).5 The forward step of the gradient computation maintains one value \(\alpha(v,t)\) per state and time step, which is recursively accumulated over all paths ending in \(v\) at time \(t\).

Footnote 2: More precisely, Vieira et al. (2016) consider \(\overline{\mathcal{W}}\), the closure of \(\mathcal{W}\) under suffix and last character substitution, which factors as \(\widehat{\mathcal{W}}=\mathcal{H}\times\mathcal{Y}\). The complexity of training depends on the size of the finite-state automaton representing \(\widehat{\mathcal{W}}\).

Footnote 3: A trie has one state for each prefix.

Footnote 4: This was also suggested by Cotterell and Eisner (2015) as a way to build a more compact pattern automaton.

Footnote 5: Upon reaching a state \(v\), we need to access the features that fire for that pattern, and also for all its suffixes. Each state thus stores a set of pattern; each pattern is associated with a set of tests on the observation (cf. 2.1).

The next question is to identify \(\mathcal{W}\). The simplest method keeps all the ngrams viewed in training, additionally filtering rare patterns (Cuong et al., 2014). However, frequency based feature selection does not take interactions into account and is not the best solution. Ideally, one would like to train a complete order-\(p\) model with a sparsity promoting penalty, a technique that only works for small label sets.6 The greedy algorithm of 

Schmidt and Murphy (2010); Vieira et al. (2016) is more scalable: it starts with all unigram patterns and iteratively grows \({\cal W}\) by extending the ngrams that have been selected in the simpler model. At each round of training, feature selection is performed using a \(\ell_{1}\) penalty and identifies the patterns that will be further augmented.

## 3 Learning patterns

We introduce now several alternatives for learning \({\cal W}\). Our motivation for doing so is twofold: (a) to take the internal structure of large label sets into account; (b) to identify more abstract patterns in label sequences, possibly containing gaps or iterations, which could yield smaller \(A[{\cal W}]\). As discussed below, both motivations can be combined.

### Greedy \(\ell_{1}\)

The greedy strategy iteratively grows patterns up to order \(p\). Considering all possible unigram and bigram patterns, we train a sparse model to select a first set of useful bigrams. In subsequent iterations, each pattern \(w\) selected at order \(k\) is extended in all possible ways to specify the pattern set at order \(k+1\), which will be filtered during the next training round. This approach is close, yet simpler, than the group lasso approach of Vieira et al. (2016) and experimentally yields slightly smaller pattern sets (see Table 2). This is because we do not enforce closure under last-character replacement: once pattern \(w\) is pruned, longer patterns ending in \(w\) are never considered.7

Footnote 7: cf. the discussion in Vieira et al. (2016, § 4).

### Component-wise training

Large tag sets often occur in joint tasks, where multiple levels of information are encoded in one compound tag. For instance, the fine grain labels in the Tiger corpus Brants et al. (2002) combine PoS and morphological information in tags such as NN.Dat.Sg.Fem for a feminine singular dative noun. In the sequel, we refer to each piece of information as a tag _component_. We assume that all tags contain the same components, using a "non-applicable" value whenever needed. Using features that test arbitrary combinations of tag components would make feature selection much more difficult, as the number of possible patterns grows combinatorially with the number of components. We keep things simple by allowing features to only evaluate one single component at a time: this allows us to identify dependencies of different orders for each component.

Assuming that each tag \(y\) contains \(K\) components \(y=[z_{1},z_{2}\ldots,z_{K}]\), with \(z_{k}\in{\cal Y}_{k}\), \({\cal W}\) is then computed as in SS 3.1, except that we now consider one distinct set of patterns \({\cal W}_{k}\) for each component \(k\). At each training round, each set \({\cal W}_{k}\) is extended and pruned independently from the others. Note that all these automata are trained simultaneously using a common set of features. This process results in \(K\) automata, which are intersected on the fly8 using "lazy" composition. In our experiments, we also consider the case where we additionally combine the automaton representing complete tag sequences: this has the beneficial effect to restrict the combinations of subtags to values that actually exist in the data.

Footnote 8: Formally, each \(A[{\cal W}_{k}]\) has transitions labelled with elements of \({\cal J}_{k}\); lazy intersection operates on “generalized” translations, where each label \(z\) is replaced with \([?,\ldots,z,\ldots,?]\), where ? matches any symbol. \(A[{\cal W}]\) is the intersection \(\bigcap_{k}A[{\cal V}_{k}]\) and is labeled with completely specified tags.

### Pruned language models

Another approach for computing \({\cal W}\) assumes that useful dependencies between tags can be identified using an auxiliary language model (LM) trained _without paying any attention to observation sequences_. A pattern \(w\) will then be deemed useful for the labelling task only if \(w\) is a useful history in a LM of tag sequences. This strategy was implemented by first training a compact \(p\)-gram LM with entropy pruning9Stolcke (1998) and including all the surviving histories in \({\cal W}\). In a second step, we train the complete CRF as usual, with all observation features and \(\ell_{1}\) penalty to further prune the parameter set.

Footnote 9: Starting with a full back-off n-gram language model, this approach discards n-grams if their removal causes a sufficiently small drop in cross-entropy. We used the implementation of Stolcke (2002).

### Maximum entropy language models

Another technique, which combines the two previous ideas, relies on Maximum Entropy LMs

\begin{table}
\begin{tabular}{|l|c|c|} \hline  & cz & de \\ \hline train set & 38,727 & 40,474 \\ development set & 5,228 & 5,000 \\ test set & 4,213 & 5,000 \\ \hline \# PoS & 12 & 54 \\ \# attributes & 13 & 8 \\ \# full tags & 1,924 & 781 \\ \hline \end{tabular}
\end{table}
Table 1: Corpus description 

(MELMs) [12]. MELMs decompose the probabililty of a sequence \(y_{1}\ldots y_{T}\) using the chain rule, where each term \(p_{\lambda}(y_{t}|y_{<t})\) is a locally normalized exponential model including all possible ngram features up to order \(p\):

\[p(y_{t}|y_{<t};\lambda)=Z(\lambda)^{-1}\exp\lambda^{T}G(y_{1}\ldots y_{t})\]

In contrast to globally normalized models, the complexity of training remains linear wrt. \(|\mathcal{Y}|\), irrespective of \(p\). It it also straightforward both to (a) use a \(\ell_{1}\) penalty to perform feature selection; (b) include features that only test specific components of a complex tag. For an order \(p\) model, our feature functions evaluate all \(n\)-grams (for \(n\leq p\)) of complete tags or of one specific component:

\[G_{w}(y_{1},\dots,y_{t})= \mathbb{I}(y_{t-n+1}\ldots y_{t}=w)\] \[G_{u}(y_{1},\dots,y_{t})= \mathbb{I}(z_{k,t-n+1}\ldots z_{k,t}=u)\]

Once a first round of feature selection has been performed,10 we compute \(A[\mathcal{W}]\) as explained above. The last step of training reintroduces the observations and estimates the CRF paramaters. A variant of this approach adds extra _gappy_ features to the \(n\)-gram features. Gappy features at order \(p\) test whether some label \(u\) occurs in the remote past anywhere between position \(t-p+1\)11 and \(t-n\). They take the following form:

Footnote 10: As the LM building step only look at labels, we tune the regularization to optimize the perplexity of the LM on a development set.

Footnote 11: We use \(p=6\) in our experiments.

\[G_{w,u}(y_{1},\dots,y_{t})= \mathbb{I}(y_{t-n+1}\ldots y_{t}=w\wedge\] \[u\in\{y_{t-p+1}\ldots y_{t-n}\}),\]

and likewise for features testing components.

## 4 Experiments

### Training protocol

The following protocol is used throughout: (a) identify \(\mathcal{W}\) (SS3) - note that this may imply to tune a regularization parameter; (b) train a full model (including tests on the observations for each pattern in \(\mathcal{W}\)) using \(\ell_{1}\) regularization and a very small \(\ell_{2}\) term to stabilize convergence. The best regularization in (a) and (b) is selected on development data and targets either perplexity (for LMs) or label accuracy (for CRFs).

### Datasets and Features

Experiments are run on two MRLs: for Czech, we use the CoNLL 2009 data set [5] and for German, the Tiger Treebank with the split of Fraser et al. (2013)). Both datasets include rich morphological attributes (cf. Table 1).

All the patterns in \(\mathcal{W}\) are combined with _lexical_ features testing the current word \(x_{t}\), its prefixes and suffixes of length 1 to 4, its capitalization and the presence of digit or punctuation symbols. Additional contextual features also test words in a local window around position \(t\). These tests greatly increase the feature count and are not provided for all label patterns: for unigram patterns, we test the presence of all unigrams and bigrams of words in a window of 5 words; for bigrams patterns we only test for all unigrams in a window of 3 words. Contextual features are not used for larger patterns.

### Results

We consider several baselines: Maxent and MEMM models, neither of which considers label dependencies in training, a linear chain CRF12 and our own implementation of the group lasso of Vieira et al. (2016). For the latter, we contrast two setups: one where each pattern in \(\mathcal{W}\) gives rise to one single feature, and one where it is conjoined with tests on the observation.13 All scores in Table 2 are label accuracies on unseen test data.

Footnote 13: Using the implementation of Lavergne et al. (2010).

As expected, Maxent and MEMM are outperformed by almost all variants of CRFs, and their scores are only reported for completeness. Group lasso results demonstrate the effectiveness of using contextual information with high order features: the gain is \(\approx\) 0.7 points for both languages and all values of \(p\). Greedy\(\ell_{1}\) achieves accuracy results similar to group lasso, suggesting that \(\ell_{1}\) penalty alone is effective to select higher-order features. It also yields slighly smaller models and very comparable training time across the board: indeed, greedy parameter selection strategies imply multiple rounds of training which are overall quite costly, due to the size of the full label set. Testing individual subtags (SS 3.2) results in a slight improvement (\(\approx\)+0.3) in accuracy over Greedy\(\ell_{1}\). When using an additional automata for the full tag, we get a larger gain of \(\approx\) 0.6 points for Czech, slightly less for German: including a model for complete t 

ate invalid combinations of subtags. These models represent different tradeoffs between accuracy and training time: the 4-gram Component-wise experiment only took 14 hrs to complete on German data and outperforms the corresponding Greedy\(\ell_{1}\) setup while containing approximately 100 times less features. Component-wise+Full is more comparable in size and training time to Greedy\(\ell_{1}\), but yields a larger improvement in performance. The last sets of experiments with LMs yields even better operating points, as the first stage of pattern selection is performed with a cheap model. They are our best trade-off to date, yielding the best performance for all values of \(p\).

## 5 Conclusion

In this work, we have explored ways to take advantage of the flexibility offered by implementations of VoCRFs based on finite-state techniques. We have proposed strategies to include tests on subparts of complex tags, as well as to select useful label patterns with auxiliary unconditional LMs. Experiments with two MRLs with large tagsets yielded consistent improvements (\(\approx\) +0.8 points) over strong baselines. They offer new perspectives to perform feature selection in high order CRFs. In our future work, we intend to also explore how to complement \(\ell_{1}\) penalties with terms penalizing more explicitely the processing time; we also wish to study how these ideas can be used in combination with neural models.

## Acknowledgements

The authors wish to thank the reviewers for their useful comments and suggestions. This work has been partly funded by the European Union's Horizon 2020 research and innovation programme under grant agreement No. 645452 (QT21).

\begin{table}
\begin{tabular}{|l||c|c|c|c|c|c|c|} \multicolumn{7}{|c|}{**Cz**} & \multicolumn{3}{c|}{**de**} \\ \hline  & \(p=2\) & \(p=3\) & \(p=4\) & \(p=5\) & \(p=2\) & \(p=3\) & \(p=4\) & \(p=5\) \\ \hline \multirow{4}{*}{Maxent} & 90.01\% & 91.12\% & 91.17\% & 91.14\% & 85.62\% & 85.84\% & 85.96\% & 86.02\% \\  & 1924 & 1924 & 1924 & 1924 & 781 & 781 & 781 & 781 \\  & 191min & 219min & 286min & 349min & 142min & 193min & 252min & 297min \\ \cline{2-10}  & 90.96\% & 92.09\% & 92.13\% & 92.12\% & 86.48\% & 86.88\% & 87.13\% & 87.19\% \\ MEMM & 1924 & 1924 & 1924 & 1924 & 781 & 781 & 781 & 781 \\  & 191imin & 219omin & 286min & 349min & 142min & 193min & 252min & 297min \\  & 91.93\% & & & & 86.95\% & & & \\ Linear Chain CRF & 3.76\% & – & – & – & 6.1e5 & – & – & – \\  & 657min & & & & & & & \\  & 657min & & & & & & & \\ Group lasso & 9.19\% & 92.27\% & 92.41\% & & 86.92\% & 87.24\% & 87.48\% & \\  & 9.65 & 4.36\% & 1.2e8 & – & 1.8e5 & 9.26\% & 5.467 & – \\  & 421min & 1656min & 3067min & – & 305 min & 1134min & 2101min & — \\  & 92.51\% & 92.95\% & 93.03\% & & 87.48\% & 87.92\% & 87.96\% & – \\ Group lasso + ctx & 9.2ec & 4.1ef7 & 1.2e8 & – & 1.7e5 & 7.86\% & 5.367 & – \\  & 520min & 1632min & 3285min & – & 349min & 1218min & 239min & — \\ \hline \multirow{4}{*}{Greedy \(\ell_{1}\)} & 92.47\% & 92.94\% & 93.01\% & & 87.43\% & 87.87\% & 87.96\% & \\  & 8.4e5 & 4.1ef7 & 1.1e8 & – & 1.7e5 & 7.1e6 & 5.067 & – \\  & 462min & 1759min & 3300min & – & 340min & 1239min & 2357min & — \\ \hline \multirow{4}{*}{Component-wise + Full} & 92.76\% & 93.24\% & 93.36\% & 93.28\% & 87.47\% & 88.16\% & 88.26\% & 88.26\% \\  & 6.2et & 2.8e5 & 8.2e5 & 3.7e6 & 2.4e4 & 7.2e4 & 3.7e5 & 1.4e6 \\  & 247min & 370min & 1179min & 224min & 173min & 268min & 836min & 1483min \\  & 92.97\% & 93.41\% & 93.69\% & 93.65\% & 87.39\% & 88.36\% & 88.59\% & 88.60\% \\  & 8.7e5 & 2.8er & 8.3ef & 4.6ee8 & 1.4e5 & 5.2e6 & 2.1ef7 & 1.3e8 \\  & 463min & 1659min & 3162min & 4321min & 311min & 1097min & 2181min & 3249min \\ \hline \multirow{4}{*}{Pruned LM} & 92.98\% & 93.27\% & 93.51\% & 93.53\% & 87.43\% & 88.12\% & 88.25\% & 88.21\% \\  & 3.2e5 & 8.2ede & 1.1ef & 8.6ert & 1.3e5 & 8.9e5 & 9.16\% & 5.3e7 \\  & 233min & 487min & 1210min & 2519min & 163min & 372min & 890min & 1894min \\  & 93.02\% & 93.33\% & 93.81\% & 93.63\% & 87.41\% & 88.61\% & 88.76\% & 88.74\% \\ MELM & 4.6e5 & 1.7e7 & 2.3e7 & 1.4e8 & 1.4e5 & 2.9e6 & 1.4e7 & 9.8e7 \\  & 303min & 545min & 1478min & 2559min & 206min & 407min & 1063min & 1924min \\  & 93.52\% & 93.68\% & 93.79\% & & 88.38\% & 88.70\% & 88.78\% & \\  & 4.5e5 & 1.5ef & 1.9ef & – & 1.4e5 & 2.3e6 & 1.1ef & – \\  & 289min & 658min & 1751min & 217min & 439min & 1297min & — \\ \hline \end{tabular}
\end{table}
Table 2: Experimental results. Each cell reports accuracy, number of states in \(A[\mathcal{W}]\) and total training time. Group lasso is our reimplementation of Vieira et al. (2016) (+Ctx = +context features) ; Greedy \(\ell_{1}\) is described in section 3.1, Component-wise is the decomposition approach of § 3.2, PrunedLM and MELM (+Gaps) were described in § 3.3 and § 3.4.

 



## References

* Allauzen et al. (2003) Cyril Allauzen, Mehryar Mohri, and Brian Roark. 2003. Generalized algorithms for constructing statistical language models. In _Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Sapporo, Japan, pages 40-47. [https://doi.org/10.3115/1075096.1075102](https://doi.org/10.3115/1075096.1075102).
* Berger et al. (1996) Adam L. Berger, Vincent J. Della Pietra, and Stephen A. Della Pietra. 1996. A maximum entropy approach to natural language processing. _Comput. Linguist._ 22(1):39-71.
* Brants et al. (2002) Sabine Brants, Stefanie Dipper, Silvia Hansen, Wolfgang Lezius, and George Smith. 2002. The TIGER treebank. In _Proceedings of the workshop on treebanks and linguistic theories_. pages 24-41.
* Cotterell and Eisner (2015) Ryan Cotterell and Jason Eisner. 2015. Penalized expectation propagation for graphical models over strings. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, pages 932-942. [https://doi.org/10.3115/v1/N15-1094](https://doi.org/10.3115/v1/N15-1094).
* Cuong et al. (2014) Nguyen Viet Cuong, Nan Ye, Wee Sun Lee, and Hai Leong Chieu. 2014. Conditional Random Field with High-order Dependencies for Sequence Labeling and Segmentation. _Journal of Machine Learning Research_ 15:981-1009. [http://jmlr.org/papers/v15/cuong14a.html](http://jmlr.org/papers/v15/cuong14a.html).
* Fraser et al. (2013) Alexander Fraser, Helmut Schmid, Richard Farkas, Renjing Wang, and Hinrich Schutze. 2013. Knowledge sources for constituent parsing of german, a morphologically rich and less-configurational language. _CL_ 39(1):57-85.
* Hajic (2000) Jan Hajic. 2000. Morphological tagging: Data vs. dictionaries. In _Proceedings of the 1st North American chapter of the Association for Computational Linguistics conference_. Seattle, WA, pages 94-101.
* Hajic et al. (2009) Jan Hajic, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antonia Marti, Lluis Marquez, Adam Meyers, Joakim Nivre, Sebastian Pado, Jan Stepanek, Pavel Stra\(\hat{\text{n}}\)kiu Shirael Surdeanu, Nianwen Xue, and Yi Zhang. 2009. The conll-2009 shared task: Syntactic and semantic dependencies in multiple languages. In _Proceedings of the Thirteenth Conference on Computational Natural Language Learning: Shared Task_. CoNLL '09, pages 1-18.
* Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In _Proceedings of the 18th International Conference on Machine Learning_. Morgan Kaufmann, San Francisco, CA, Williamstown, MA, (ICML'01), pages 282-289.
* Lavergne et al. (2010) Thomas Lavergne, Olivier Cappe, and Francois Yvon. 2010. Practical very large scale CRFs. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_. Uppsala, Sweden, pages 504-513.
* Martins et al. (2011) Andre Martins, Noah Smith, Mario Figueiredo, and Pedro Aguiar. 2011. Structured sparsity in structured prediction. In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_. pages 1500-1511.
* Muller et al. (2015) Thomas Muller, Ryan Cotterell, Alexander Fraser, and Hinrich Schutze. 2015. Joint lemmatization and morphological tagging with lemming. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Lisbon, Portugal, EMNLP'15, pages 2268-2274.
* Muller et al. (2013) Thomas Muller, Helmut Schmid, and Hinrich Schutze. 2013. Efficient higher-order CRFs for morphological tagging. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Seattle, Washington, USA, EMNLP'15, pages 322-332.
* Pal et al. (2006) Chris Pal, Charles Sutton, and Andrew McCallum. 2006. Sparse forward-backward using minimum divergence beams for fast training of conditional random fields. In _2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings_. volume 5, pages V-V. [https://doi.org/10.1109/ICASSP.2006.1661342](https://doi.org/10.1109/ICASSP.2006.1661342).
* Ron et al. (1996) Dana Ron, Yoram Singer, and Naftali Tishby. 1996. The power of amnesia: Learning probabilistic automata with variable memory length. _Machine Learning_ 25(2-3):117-149.

* Schmidt and Murphy (2010) Mark W. Schmidt and Kevin P. Murphy. 2010. Convex structure learning in log-linear models: Beyond pairwise potentials. In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics,_. Chia Laguna Resort, Sardinia, Italy, AISTATS, pages 709-716.
* Schrooper et al. (2016) Carsten Schooper, Steffen Eger, Erik-Lan Do Dinh, and Iryna Gurevych. 2016. Still not there? comparing traditional sequence-to-sequence models to encoder-decoder neural networks on monotone string translation tasks. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_. The COLING 2016 Organizing Committee, Osaka, Japan, pages 1703-1714.
* Schutze and Singer (1994) Hinrich Schutze and Yoram Singer. 1994. Part-of-speech tagging using a variable memory Markov model. In _Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics_. Las Cruces, New Mexico, pages 181-187

Nobuyuki Shimizu and Andrew Haas. 2006. Exact decoding for jointly labeling and chunking sequences. In _Proceedings of COLING/ACL_. pages 763-770.
* Smith et al. (2005) Noah A. Smith, David A. Smith, and Roy W. Tromble. 2005. Context-based morphological disambiguation with random fields. In _Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing_. Vancouver, British Columbia, Canada, pages 475-482.
* Stolcke (1998) Andreas Stolcke. 1998. Entropy-based pruning of backoff language models. In _Proc. DARPA Broadcast News Transcription and Understanding Workshop_. Lansdowne, VA, pages 270-274.
* an extensible language modeling toolkit. In _Proceedings of the International Conference on Spoken Langage Processing (ICSLP)_. Denver, CO, volume 2, pages 901-904.
* Sutton and McCallum (2006) Charles Sutton and Andrew McCallum. 2006. An introduction to conditional random fields for relational learning. In Lise Getoor and Ben Taskar, editors, _Introduction to Statistical Relational Learning_. The MIT Press, Cambridge, MA.
* Tibshirani (1996) Robert Tibshirani. 1996. Regression shrinkage and selection via the lasso. _Journal of the Royal Statistical Society B_ 58(1):267-288.
* Totuanova and Cherry (2009) Kristina Totuanova and Colin Cherry. 2009. A global model for joint lemmatization and part-of-speech prediction. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_. Association for Computational Linguistics, pages 486-494. [http://aclweb.org/anthology/P09-1055](http://aclweb.org/anthology/P09-1055).
* Vieira et al. (2016) Tim Vieira, Ryan Cotterell, and Jason Eisner. 2016. Speed-accuracy tradeoffs in tagging with variable-order crfs and structured sparsity. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. EMNLP, pages 1973-1978.
* Ye et al. (2009) Nan Ye, Wee S. Lee, Hai L. Chieu, and Dan Wu. 2009. Conditional random fields with high-order features for sequence labelling. In Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta, editors, _Advances in Neural Information Processing Systems 22_. Curran Associates, Inc., pages 2196-2204. [http://papers.nips.cc/paper/3815-conditional-random-fields-with-high-order-features-for-sequence-labeling.pdf](http://papers.nips.cc/paper/3815-conditional-random-fields-with-high-order-features-for-sequence-labeling.pdf)