Venue: NAACL 2016

# Dynamic Feature Induction:

The Last Gist to the State-of-the-Art

 Jinho D. Choi

Department of Mathematics and Computer Science

Emory University

Atlanta, GA 30322, USA

jinho.choi@emory.edu

###### Abstract

We introduce a novel technique called dynamic feature induction that keeps inducing high dimensional features automatically until the feature space becomes 'more' linearly separable. Dynamic feature induction searches for the feature combinations that give strong clues for distinguishing certain label pairs, and generates joint features from these combinations. These induced features are trained along with the primitive low dimensional features. Our approach was evaluated on two core NLP tasks, part-of-speech tagging and named entity recognition, and showed the state-of-the-art results for both tasks, achieving the accuracy of \(97.64\) and the F1-score of \(91.00\) respectively, with about a 25% increase in the feature space.

## 1 Introduction

Feature engineering typically involves two processes: the process of discovering novel features with domain knowledge, and the process of optimizing combinations between existing features. Discovering novel features may require linguistic background as well as good understanding in machine learning such that it is often difficult to do. Optimizing feature combinations can be also difficult but usually requires less domain knowledge and more importantly, it can be as effective as discovering new features. It has been shown for many tasks that approaches using simple machine learning with extensive feature engineering outperform ones using more advanced machine learning with less intensive feature engineering Xue and Palmer (2004); Bengtsson and Roth (2008); Ratinov and Roth (2009); Zhang and Nivre (2011). Recently, people have tried to automate the second part of feature engineering, the optimization of feature combinations, through leading-edge models such as neural networks Collobert et al. (2011). Coupled with embedding approaches Mikolov et al. (2013); Le and Mikolov (2014); Pennington et al. (2014), neural networks can find the optimal feature combinations using techniques such as random weight initialization and back-propagation, and have established the new state-of-the-art for several tasks Socher et al. (2013); Devlin et al. (2014); Yu et al. (2014). However, neural networks are not as good at optimizing combinations between sparse features, which are still the most dominating factors in natural language processing.

This paper introduces a new technique called dynamic feature induction that automates the optimization of feature combinations (Section 3), and can be easily adapted to any NLP task using sparse features. Dynamic feature induction allows humans to focus on the first part of feature engineering, the discovery of novel features, while machines handle the second part. Our approach was experimented with two core NLP tasks, part-of-speech tagging (Section 4) and named entity recognition (Section 5) and showed the state-of-the-art results for both tasks.

## 2 Background

### Nonlinearity in NLP

Linear classification algorithms such as Perceptron, Winnow, or Support Vector Machines with a linear kernel have performed exceptionally well for various NLP tasks Collins (2002); Zhang and Johnson (2003); Pradhan et al. (2005). This is not because our feature space is linearly separable by nature, but sparse fea

very high dimensional vector space such that it is rather forced to be linearly separable. For example, NLP features for a word \(w_{i}\) typically involve the word forms of \(w_{i-1}\) and \(w_{i}\) (e.g., \(f_{i-1}\), \(f_{i}\)). If the feature space is not linearly separable with these features, a common trick is to introduce 'higher' dimension features by joining 'lower' dimension features together (e.g., \(f_{i-1}\)-\(f_{i}\)). The more joint features we introduce, the higher chance we get for the feature space being linearly separable although these joint features can be very overfitted.

Let us define low dimensional features as the primitive features such as \(f_{i-1}\) or \(f_{i}\), and high dimensional features as the joint features such as \(f_{i}\)-\(f_{i+1}\). 1 Low dimensional features are well explored for most NLP tasks; it is the high dimensional features that are quite sensitive to specific tasks. Finding high dimensional features can be a manual intensive work and this is what dynamic feature induction intends to take over.

Footnote 1: The joint features tend to yield a much higher dimensional feature space than the primitive features.

### Related Work

Kudo and Matsumoto (2003) introduced the polynomial kernel expansion that explicitly enumerated the feature combinations. Our approach is distinguished because they used a frequency-based PrefixSpan algorithm Pei et al. (2001) whereas we used the online learning weights for finding the feature combinations. Goldberg and Elhadad (2008) suggested an efficient algorithm for computing polynomial kernel SVMs by combining inverted indexing and kernel expansion. Their work is focused more on improving support vector machines whereas our work is generalized to any linear classification algorithm. Okanohara and Tsujii (2009) introduced an approach for generating feature combinations using \(\ell_{1}\) regularization and grafting Perkins et al. (2003). Although we share similar ideas, their grafting algorithm starts with an empty feature set whereas ours starts with low dimensional features, and their correlation parameters \(\alpha_{i,y}\) are pre-computed whereas ours are dynamically determined. Strubell et al. (2015) suggested an algorithm that dynamically selected strong features during decoding. Our work is distinguished because we do not run multiple training phases as they do for figuring our strong features.

## 3 Dynamic Feature Induction

The intuition behind dynamic feature induction is to keep populating high dimensional features by joining low dimensional features together until the feature space becomes 'more' linearly separable.2 Figure 1 shows how features are induced during training:

Footnote 2: The term ‘more’ is used because dynamic feature induction does not guarantee for the feature space to be linearly separable.

1. Given a training instance \((x_{1},y_{1})\), where \(x_{1}\) is a feature set consisting of 5 features and \(y_{1}\) is the gold label, the classifier predicts the label \(\hat{y}_{1}\).
2. Let us refer "strong features for \(y\) against \(\hat{y}\)" to features that give strong clues for distinguishing \(y\) from \(\hat{y}\). If \(\hat{y}_{1}\) is not equal to \(y_{1}\) (2.1), strong features for \(y_{1}\) against \(\hat{y}_{1}\) in \(x_{1}\) are selected (2.2), and combinations of these features are added to the induced feature set \(\mathcal{F}\) (2.3).
3. Given a new training instance \((x_{2},y_{2})\), combinations of features in \(x_{2}\) are checked by \(\mathcal{F}\) (3.1), and appended to \(x_{2}\) if allowed (3.2).

Figure 1: Overview of dynamic feature induction.

 

4. The extended feature set \(x_{2}\) is fed into the classifier. If \(\hat{y}_{2}\) is equal to \(y_{2}\), no feature combination is induced from \(x_{2}\).

Thus, high dimensional features in \(\mathcal{F}\) are incrementally induced and learned along with low dimensional features during training. During decoding, each feature set is extended by the induced features in \(\mathcal{F}\), and the prediction is made using the extended feature set. The size of \(\mathcal{F}\) can grow up to \(|\mathcal{X}|^{2}\), where \(|\mathcal{X}|\) is the size of low dimensional features. However, we found that \(|\mathcal{F}|\) is more like \(\nicefrac{{1}}{{4}}\cdot|\mathcal{X}|\) in practice.

The following sections explain our approach in details. Sections 3.1, 3.2, and 3.3 describe how features are induced and learned during training. Sections 3.4 and 3.5 describe how the induced features are stored and expanded during decoding.

### Feature Induction

Algorithm 1 shows an online learning algorithm that induces and learns high dimensional features during training. It takes the set of training instances \(D\) and the learning rate \(\eta\), and returns the weight vector **w** and the set of induced features \(\mathcal{F}\).

```
Input:\(D\): training set, \(\eta\): learning rate. Output:\(\mathbf{w}\): weight vector, \(\mathcal{F}\): induced feature set.
1:\(\mathbf{w}\leftarrow\textbf{g}\gets 0\)
2:\(\mathcal{F}\leftarrow\varnothing\)
3:until\(\max\) epoch is reached do
4:foreach\((x,y)\in D\)do
5:\(\hat{y}\leftarrow\arg\max_{y^{\prime}\in\mathcal{Y}}(\textbf{w}\cdot\phi(x, y^{\prime},\mathcal{F})-I_{y}(y^{\prime}))\)
6:if\(y\neq\hat{y}\)then
7:\(\partial\leftarrow\phi(x,y,\mathcal{F})-\phi(x,\hat{y},\mathcal{F})\)
8:\(\textbf{g}\leftarrow\textbf{g}+\partial\circ\partial\)
9:\(\textbf{w}\leftarrow\textbf{w}+(\nicefrac{{\eta(\rho+\sqrt{\hat{\textbf{w}}}) }}{{\bf b}})\cdot\partial\)
10:\(\textbf{v}\leftarrow[\textbf{w}\circ\phi(x,y,\varnothing)]_{y}-[\textbf{w} \circ\phi(x,\hat{y},\varnothing)]_{\hat{y}}\)
11:\(\mathcal{L}\leftarrow\arg k\max_{\forall i}\textbf{v}_{i}\)
12:for\(i=2\)to\(|\mathcal{L}|\)do
13:\(\mathcal{F}\leftarrow\mathcal{F}\cup\left\{(\mathcal{L}_{1},\mathcal{L}_{i})\right\}\)
14:return\(\textbf{w},\mathcal{F}\)
```

**Algorithm 1** Feature Induction

The algorithm begins by initializing the weight vector **w**, the diagonal vector **g**, and the induced feature set \(\mathcal{F}\) (lines 1-2). For each instance \((x,y)\in D\) where \(y\) is the gold-label for the feature set \(x\), it predicts \(\hat{y}\) maximizing \(\textbf{w}\cdot\phi(x,y^{\prime},\mathcal{F})-I_{y}(y^{\prime})\), where \(I\) is defined as follows (lines 4-5):

\[I_{y}(y^{\prime})\leftarrow\begin{cases}1,&\text{if $y=y^{\prime}$.}\\ 0,&\text{otherwise.}\end{cases}\]

The feature map \(\phi\) takes \((x,y,\mathcal{F})\), and returns a \(d\times l\)-dimensional vector, where \(d\) and \(l\) are the sizes of features and labels, respectively; each dimension contains the value for a particular feature and a label.3 If certain combinations between features in \(x\) exist in \(\mathcal{F}\), they are appended to the feature vector along with the low dimensional features (see Section 3.5 for more details). The indicator function \(I\) allows our algorithm to be optimized for the hinge loss for multiclass classification (Crammer and Singer, 2002):

Footnote 3: In most cases, these values are either \(0\) or \(1\).

\[\ell_{h}=\max[0,1+\textbf{w}\cdot(\phi(x,\hat{y},\mathcal{F})-\phi(x,y, \mathcal{F}))]\]

If \(y\) is not equal to \(\hat{y}\) (line 6), the partial vector \(\partial\) is measured (line 7), and **g** and **w** are updated (lines 8-9) by AdaGrad (Duchi et al., 2011), where the learning rate \(\eta\) is adjusted by **g** (in our case, \(\rho=\)1E-5). Once **w** is updated, the \(d\)-dimensional vector **v** is generated by subtracting \([\textbf{w}\!\circ\!\phi(x,\hat{y},\varnothing)]_{\hat{y}}\) from \([\textbf{w}\!\circ\!\phi(x,y,\varnothing)]_{y}\) (line 10), where \([\ldots]_{y}\) returns only the portion of the values relevant to \(y\) (Figure 2).

The \(i\)'th element in **v** represents the strength of the \(i\)'th feature for \(y\) against \(\hat{y}\); the greater \(\textbf{v}_{i}\) is, the stronger the \(i\)'th feature is. Next, indices of the top-\(k\) entries in **v** are collected in the ordered list \(\mathcal{L}\) (line 11), representing the strongest features for \(y\) against \(\hat{y}\).4 Finally, the pairs of the first index in \(\mathcal{L}\), representing the strongest feature, and the other indices in \(\mathcal{L}\) are added to the induced feature set \(\mathcal{F}\) (lines 12-13). For example, if \(\mathcal{L}=[i,j,k]\) such that \(\textbf{v}_{i}\geq\textbf{v}_{j}\geq\textbf{v}_{k}>0\), two pairs, \((i,j)\) and \((i,k)\), are added to \(\mathcal{F}\).

Footnote 4: “\(\arg k\max\)’ returns the ordered list of indices whose values in **v** are \(\nicefrac{{1}}{{k}}\)-largest and \(\nicefrac{{2}}{{2}}\)greater than 0.

For all our experiments, \(k=3\) is used; increasing \(k\) beyond this cutoff did not show much improvement. Notice that all induced features in \(\mathcal{F}\) are derived by joining only low dimensional features together. Our algorithm does not join a high dimensional feature with either a low dimensional feature or another high dimensional feature. This was done intentionally to prevent from the feature space being exploded; such features can be induced by replacing \(\varnothing\) with \(\mathcal{F}\) in the line 10 as follows:

\[\textbf{v}\leftarrow[\textbf{w}\circ\phi(x,y,\mathcal{F})]_{y}-[\textbf{w} \circ\phi(x,\hat{y},\mathcal{F})]_{\hat{y}}\] 

It is worth mentioning that we did not find it useful for joining intermediate features together (e.g., \((j,k)\) in the above example). It is possible to utilize these combinations by weighting them differently, which we will explore in the future. Additionally, we experimented with the combinations between strong and weak features (joining _i'_th and _j'_th features, where \({\bf v}_{i}>0\) and \({\bf v}_{j}<0\)), which again was not so useful. We are planning to evaluate our approach on more tasks and data, which will give us better understanding of what combinations are the most effective.

### Regularized Dual Averaging

Each high dimensional feature in \(\mathcal{F}\) is induced for making classification between two labels, \(y\) and \(\hat{y}\), but it may or may not be helpful for distinguishing labels other than those two. Our algorithm can be modified to learn the weights of the induced features only for their relevant labels by adding the label information to \(\mathcal{F}\), which would change the line 13 in Algorithm 1 as follows:

\[\mathcal{F}\leftarrow\mathcal{F}\cup\left\{(\mathcal{L}_{1},\mathcal{L}_{i}, y,\hat{y})\right\}\]

However, introducing features targeting specific label pairs potentially confuses the classifier, especially when they are trained with the low dimensional features targeting all labels. Instead, it is better to apply a feature selection technique such as \(\ell_{1}\) regularization so the induced features can be selectively learned for labels that find those features useful. We adapt regularized dual averaging Xiao (2010), which efficiently finds the convergence rates for online convex optimization, and works most effectively with sparse feature vectors. To apply regularized dual averaging, the line 1 in Algorithm 1 is changed to:

\[{\bf w}\leftarrow{\bf g}\leftarrow{\bf c}\gets 0;~{}~{}t\gets 1\]

**c** is a \(d\times l\)-dimensional vector consisting of accumulative penalties. \(t\) is the number of weight vectors generated during training. Although \({\bf w}\) is technically not updated when \(y=\hat{y}\), it is still considered a new vector. Thus, \(t\) is incremented for every training instance, so \(t\gets t+1\) is inserted after the line 5. **c** is updated by adding the partial vector \(\partial\) as follows (to be inserted after the line 7):

\[{\bf c}\leftarrow{\bf c}+\partial\]

Thus, each dimension in **c** represents the accumulative penalty (or reward) for a particular feature and a label. At last, the line 9 is changed to:

\[{\bf w}\leftarrow(\nicefrac{{\eta}}{{(\rho+\sqrt{\textbf{w}})}}) \cdot\ell_{1}({\bf c},t,\lambda)\] \[\ell_{1}({\bf c},t,\lambda)\leftarrow\begin{cases}{\bf c}_{i}-{ \rm sgn}({\bf c}_{i})\cdot\lambda\cdot t,&|{\bf c}_{\forall i}|>\lambda\cdot t .\\ 0,&\text{otherwise}.\end{cases}\]

The function \(\ell_{1}\) takes **c**, \(t\), and the regularizer parameter \(\lambda\) tuned during development. If the absolute value of the accumulative penalty \({\bf c}_{i}\) is greater than \(\lambda\cdot t\), the weight \({\bf w}_{i}\) is updated by \(\lambda\) and \(t\); otherwise, it is assigned to \(0\). For our experiments, RDA was able to throw out irrelevant features successfully, and showed improvement in accuracy; in fact, dynamic feature induction without RDA did not show as much improvement over low dimensional features.

 



### Locally Optimal Learning to Search

Features in most NLP tasks are extracted from structures (e.g., sequence, tree). For structured learning, we adapt "locally optimal learning to search" Chang et al. (2015b), that is a member of imitation learning similar to DAggerRoss et al. (2011). LOLS not only performs well relative to the reference policy, but also can improve upon the reference policy, showing very good results for tasks such as part-of-speech tagging and dependency parsing. We adapt LOLS by setting the reference policy as follows:

1. The reference policy \(\pi\) determines how often the gold label \(y\) is picked over the predicted label \(\hat{y}\) to build a structure. For all our experiments, \(\pi\) is initialized to \(0.95\).
2. For the first epoch, since \(\pi\) is \(0.95\), \(y\) is randomly picked over \(\hat{y}\) for 95% of the time.
3. After every epoch, \(\pi\) is multiplied by \(0.95\). This allows the next epoch to pick \(y\) less often than the previous epoch (e.g., \(\pi\) becomes \(0.95^{2}=0.9025\) for the 2nd epoch so \(y\) is picked about 90% of the time instead of 95%).

For our experiments, LOLS gave only marginal improvement, probably because the tasks we evaluated, part-of-speech tagging and named entity recognition, did not yield complex structures. However, we still included this in our framework because we wanted to evaluate our approach on more tasks such as dependency parsing where learning to search algorithms show a clear advantage Goldberg and Nivre (2012); Choi and McCallum (2013); Chang et al. (2015a).

### Feature Hashing

Feature hashing is a technique of converting string features to vectors Ganchev and Dredze (2008); Weinberger et al. (2009). Given a string feature \(f\) and a hash function \(h\), the index of \(f\) in the vector space is determined by taking the remainder of the hash code:

\[k\gets h_{\mathrm{string}\rightarrow\mathrm{int}}(f)\textbf{ mod }\delta\]

The divisor \(\delta\) is tuned during development. Feature hashing allows to convert string features into sparse vectors without reserving an extra space for a map whose keys and values are the string features and their indices. Given a feature index pair \((i,j)\) representing strong features for \(y\) against \(\hat{y}\) (Section 3.1), the index of the induced feature can be measured as follows:

\[k\gets h_{\mathrm{int}\rightarrow\mathrm{int}}(i\cdot|\mathcal{X}|+j) \textbf{ mod }\delta\]

For efficiency, feature hashing is adapted to our system such that the induced feature set \(\mathcal{F}\) is actually not a set but a \(\delta\)-dimensional boolean array, where each dimension represents the validity of the corresponding induced feature. Thus, the line 13 in Algorithm 1 is changed to:

\[k\gets h_{\mathrm{int}\rightarrow\mathrm{int}}( \mathcal{L}_{1}\cdot|\mathcal{X}|+\mathcal{L}_{i})\textbf{ mod }\delta\] \[\mathcal{F}_{k}\leftarrow\texttt{True}\]

For the choice of \(h\), xxHash is used, that is a fast non-cryptographic hash algorithm showing the perfect score on the Q.Score.6

Footnote 6: [https://github.com/Cyan4973/xxHash](https://github.com/Cyan4973/xxHash)

### Feature Expansion

Algorithm 2 describes how high dimensional features are expanded from low dimensional features during training and decoding. It takes the sparse vector \(\mathbf{x}^{l}\) containing only low dimensional features and returns a new sparse vector \(\mathbf{x}^{l+h}\) containing both low and high dimensional features.

```
Input:\(\mathbf{x}^{l}\): sparse feature vector containing only low dimensional features. Output:\(\mathbf{x}^{l+h}\): sparse feature vector containing both low and high dimensional features.
1:\(\mathbf{x}^{l+h}\leftarrow\operatorname*{copy}(\mathbf{x}^{l})\)
2:for\(i\gets 1\)to\(|\mathbf{x}^{l}|\)do
3:for\(j\gets i+1\)to\(|\mathbf{x}^{l}|\)do
4:\(k\gets h_{\mathrm{int}\rightarrow\mathrm{int}}(i\cdot|\mathcal{X}|+j) \textbf{ mod }\delta\)
5:if\(\mathcal{F}_{k}\)then\(\mathbf{x}^{l+h}.\mathrm{append}(k)\)
6:return\(\mathbf{x}^{l+h}\)
```

**Algorithm 2** Feature Expansion

The algorithm begins by copying \(\mathbf{x}^{l}\) to \(\mathbf{x}^{l+h}\) (line 1). For every combination \((i,j)\in\mathbf{x}^{l}\times\mathbf{x}^{l}\), where \(i\) and \(j\) represent the corresponding feature indices (lines 2-3), it first measures the index \(k\) of the feature combination (line 4), then checks if this combination is valid (Section 3.4). If the combination is valid, meaning that \((\mathcal{F}_{k}=\texttt{True})\), \(k\) is added to \(\mathbf{x}^{l+h}\) (line 5). Finally, \(\mathbf{x}^{l+h}\) is returned with the expanded high dimensional features.





## 4 Part-of-Speech Tagging

### Corpus

The Wall Street Journal corpus from the Penn Treebank III is used Marcus et al. (1993) with the standard split for part-of-speech tagging experiments.

### Tagging and Learning Algorithms

A one-pass, left-to-right tagging algorithm is used for our experiments. Such a simple algorithm is chosen because we want to see the performance gain purely from our approach, not by a more sophisticated tagging algorithm Toutanova et al. (2003); Shen et al. (2007), which may improve the performance further.

For learning, the final algorithm from Section 3 is used. Additionally, mini-batch is applied, where each batch consists of training instances from \(k\)-number of sentences, causing the sizes of these batches different. We found that grouping instances with respect to the sentence boundary was more effective than batching them across arbitrary sentences. For all our experiments, the learning rate \(\eta=0.02\) and the mini-batch boundary \(k=5\) were used without tuning.

### Ambiguity Classes

The ambiguity class of a word is the concatenation of all possible tags for that word. For example, if the word 'study' can be tagged by NN (common noun) or VB (base verb), its ambiguity class becomes NN_VB. Instead of building ambiguity classes only from the training dataset, we automatically tagged a mixture of large datasets, the English Wikipedia articles6 and the New York Times corpus,7 and pre-constructed ambiguity classes using the automatic tags before training. This was motivated by Moore Moore (2015), who showed extraordinary results on the out-of-vocabulary words by limiting the classification to the ambiguity classes collected from such large corpora.

Footnote 6: dumps.wikimedia.org/enwiki

Footnote 7: catalog.ldc.upenn.edu/LDCE2008T19

We used the ClearNLP POS tagger Choi and Palmer (2012) for tagging the data (about 141M words), threw away tags appearing less than a certain threshold, and created the ambiguity classes. For each word, tags appearing less than 20% of the time for that word were discarded. As the result, about 2M ambiguity classes were collected from these datasets.

### Feature Template

Table 2 shows the template for low dimensional features. Digits inside the curly brackets imply the context windows with respect to the word \(w_{i}\) to be tagged. For example, \(f_{\{0,\pm 1\}}\) represents the word-forms of \(w_{i}\), \(w_{i-1}\), and \(w_{i+1}\). No joint features (e.g., \(f_{0}\)-\(f_{1}\)) are included in this template; they should be automatically induced by dynamic feature induction.

Orthographic Gimenez and Marquez (2004) and word shape Finkel et al. (2005) features are adapted from the previous work. The positional features indicate whether \(w_{i}\) is the first or the last word in the sentence. Word clusters are trained on the same datasets in Section 4.3 using Brown et al. Brown et al. (1992).

### Development

The regularization parameter \(\lambda\) (Section 3.2) and the modulo divisor \(\delta\) (Section 3.4) are tuned during development through grid search on \(\lambda\in\) [1E-9, 1E-6] and \(\delta\in\) [1.5M, 5M]. Table 3 shows the accuracies achieved by our models on the development set.

\begin{table}
\begin{tabular}{l||r|r|r|r}
**Set** & **Sections** & **Sentences** & **ALL** & **OOV** \\ \hline \hline TRN & 0-18 & 38,219 & 912,344 & 0 \\ DEV & 19-21 & 5,527 & 131,768 & 4,467 \\ TST & 22-24 & 5,462 & 129,654 & 3,649 \\ \end{tabular}
\end{table}
Table 1: Distributions of the Wall Street Journal corpus. TRN: training, DEV: development, TST: evaluation, ALL: all words, OOV: out-of-vocabulary words.

\begin{table}
\begin{tabular}{l||r|r||r}
**Model** & **ALL** & **OOV** & **FEAT** \\ \hline \hline M\({}_{0}\): baseline & 97.09 & 86.14 & 365,400 \\ M\({}_{1}\): M\({}_{0}\) + ext. ambi. & 97.37 & 91.92 & 365,409 \\ M\({}_{2}\): M\({}_{1}\) + clusters & 97.45 & 91.96 & 372,181 \\ M\({}_{3}\): M\({}_{1}\) + dynamic & 97.42 & 92.10 & 468,378 \\ M\({}_{4}\): M\({}_{2}\) + dynamic & 97.48 & 92.21 & 473,134 \\ \end{tabular}
\end{table}
Table 3: Part-of-speech tagging accuracies on the development set. FEAT: the number of features generated by each model.



M\({}_{0}\) used the tagging and the learning algorithms in Section 4.2 and the feature template in Section 4.4, where the ambiguity classes were collected only from the training dataset; dynamic feature induction was not used for M\({}_{0}\). By applying the external ambiguity classes in Section 4.3, M\({}_{1}\) achieved about a 5.8% improvement on OOV. M\({}_{2}\) gained small improvements by adding word clusters. Coupled with dynamic feature induction, M\({}_{3}\) and M\({}_{4}\) gained about \(0.04\)% and \(0.2\)% improvements on average for ALL and OOV.

For both M\({}_{3}\) and M\({}_{4}\), about 100K more features were generated from M\({}_{1}\) and M\({}_{2}\), implying that about 25% of the features were automatically induced by dynamic feature induction. It is worth pointing out that improving upon M\({}_{1}\) was a difficult task because it was already reaching near the state-of-the-art. The external ambiguity classes by themselves were strong enough to make accurate predictions such that the induced features did not find a critical role in the classification.

### Evaluation

Table 4 shows the accuracies achieved by the models from Section 4.5 and the previous state-of-the-art approaches on the evaluation set.

The results on the evaluation set appear much more promising. Still, the biggest gain was made by M\({}_{1}\), but our final model M\({}_{4}\) was able to achieve a 0.8% improvement on OOV over M\({}_{2}\), and showed the state-of-the-art results on both ALL and OOV. Interestingly, M\({}_{2}\) showed a slightly lower accuracy on OOV than M\({}_{1}\) even with the additional word cluster features. On the other hand, M\({}_{2}\) did show a slightly higher accuracy on ALL, indicating that the model was probably too overfitted to the in-vocabulary words.8 However, M\({}_{4}\) was still able to achieve improvements over M\({}_{2}\) on both ALL and OOV, implying that dynamic feature induction facilitated the classifier to be trained more robustly.

Footnote 8: A similar trend is shown in Table 3 for M\({}_{1}\) and M\({}_{2}\).

## 5 Named Entity Recognition

### Corpus

The English corpus from the CoNLL'03 shared task is used [6] for named entity recognition experiments.

### Feature Template

Table 6 shows the feature template for NER, adapting the specifications in Table 2. Following the state-of-the-art approaches (Table 8), word clusters are trained on the Reuters Corpus Volume I [7] using Brown et al. (1992). Named entity gazetteers are collected from DBPedia.9 Word embeddings are trained on the datasets in Section 4.3 using Mikolov et al. (2013) and appended to the sparse feature vectors as dense vectors. Note that the word embedding features did not participate in dynamic feature induction; it was not intuitive how to combine sparse and dense features together so we left its as a future work.

\begin{table}
\begin{tabular}{l|c} \hline \(f_{\{0,\pm 1\}},f_{u:\{0,\pm 1,\pm 2\}},s_{\{0,\pm 1\}},l_{\{0\}}, c_{\{0,1,2\}},\\ e_{\{0,\pm 1,\pm 2,z+3,z+4\}},\pi_{1:\{0\}},\pi_{3:\{1\}},\sigma_{1:\{0\}}, \sigma_{3:\{-1,0\}},\\ p_{\{0,\pm 1,\pm 2\}},n_{\{-1,-2,-3\}},z_{\{\pm 1,0,2,3\}},\mathcal{O}_{\{0\}}, \mathcal{O}_{\{1\}}\}\) \\ \end{tabular}
\end{table}
Table 6: Feature template for named entity recognition. \(f\): wordform, \(f_{u}\): uncapitalized word-form, \(s\): word shape, \(l\): lemma, \(c\): word cluster, \(e\): word embedding, \(\pi_{k}\): \(k\)’th prefix, \(\sigma_{k}\): \(k\)’th suffix, \(p\): part-of-speech tag, \(n\): named entity tag, \(z\): named entity gazetteer, \(\mathcal{O}\): orthographic feature set.

\begin{table}
\begin{tabular}{l|c|c|c}
* & \\ Moore (2015) & 97.36 & 91.09 & ✓ \\ Spoustová et al. (2009) & 97.44 &
- & ✓ \\ Sogaard (2011) & 97.50 &
- & ✓ \\ Tsuboi (2014) & 97.51 & 91.64 & ✓ \\ \hline \hline This work: M\({}_{0}\) & 97.18 & 86.35 & \\ This work: M\({}_{1}\) & 97.37 & 91.34 & ✓ \\ This work: M\({}_{2}\) & 97.46 & 91.23 & ✓ \\ This work: M\({}_{3}\) & 97.52 & 91.53 & ✓ \\ This work: M\({}_{4}\) & **97.64** & **92.03** & ✓ \\ \end{tabular}
\end{table}
Table 4: Part-of-speech tagging accuracies on the evaluation set. EXT: whether or not the approach used external data.
 



### Development

The regularization parameter and the modulo divisor are tuned during development through the same grid search in Section 4.5. Table 7 shows the precisions and the recalls achieved by our models on the development set (the F1-scores are shown in Table 8).

M\({}_{0}\) used the tagging and the learning algorithms in Section 4.2 and the feature template in Section 5.2, excluding the gazetteer, cluster, and embedding features; dynamic feature induction was not applied to M\({}_{0}\). M\({}_{\{1,2,3\}}\) gained incremental improvements from the gazetteer, cluster, and embedding features, respectively. M\({}_{4}\) showed \(0.36\)% and \(0.67\)% improvements on precision and recall respectively, and generated about 40K more features compared to M\({}_{3}\). This is about 23% increase in features that is similar to the increase shown in Table 3.

### Evaluation

Table 8 shows the F1-scores achieved by our models and the previous state-of-the-art approaches.10

Footnote 10: Ratinov and Roth (2009) reported the F1-score of 90.80 on the evaluation set, but that model was trained on both the training and the development sets so not compared in this table.

All models showed improvements over their predecessors; the improvements made in TST were more dramatic than the ones made in DEV although they followed a very similar trend. Notice that M\({}_{3}\), not using dynamic feature induction, showed very similar scores to Ratinov and Roth (2009). This was not surprising because M\({}_{3}\) adapted many features suggested by them, except for the non-local features.11

Footnote 11: We transformed the original data into the BILOU notation, which was also suggested by Ratinov and Roth (2009).

M\({}_{4}\) achieved about 0.5% improvements over M\({}_{3}\), showing the state-of-the-art result on TST. Considering that M\({}_{3}\) was already near state-of-the-art, this improvement was meaningful. It was interesting that Suzuki and Isozaki (2008) achieved the state-of-the-art result on DEV although their score on TST was much lower than the other approaches. This might be because features extracted from the huge external data they used were overfitted to DEV, but more thorough analysis needs to be done. On the other hand, Passos et al. (2014) achieved the near state-of-the-art result on DEV while it also got a very high score on TST by utilizing phrase embeddings, which we will look into in the future.

## 6 Conclusion

In this paper, we introduced a novel technique called dynamic feature induction that automatically induces high dimensional features so the feature space can be more linearly separable. Our approach was evaluated on two NLP tasks, part-of-speech tagging and named entity recognition, and showed the state-of-the-art results on both tasks. The improvements achieved by dynamic feature induction might not be statistically significant, but important because they gave the last gist to the state-of-the-art; without this last gist, our system would have not reached the bar.

It is worth mentioning that we also experimented with several feature templates including many joint features without applying dynamic feature induction. The results we got from these manually induced features were not any better (often worse) than the ones achieved by dynamic feature induction, which was very encouraging. In the future, we will experiment our approach on more NLP tasks such as dependency parsing and conference resolution where induced features should play a more critical role.

\begin{table}
\begin{tabular}{l||c|c||c}
**Model** & **P** & **R** & **FEAT** \\ \hline \hline M\({}_{0}\): baseline & 90.87 & 89.15 & 164,440 \\ M\({}_{1}\): M\({}_{0}\) + gazetteers & 92.30 & 90.61 & 164,720 \\ M\({}_{2}\): M\({}_{1}\) + clusters & 93.66 & 91.79 & 169,232 \\ M\({}_{3}\): M\({}_{2}\) + embeddings & 94.14 & 92.43 & 169,682 \\ M\({}_{4}\): M\({}_{3}\) + dynamic & 94.50 & 93.10 & 208,860 \\ \end{tabular}
\end{table}
Table 7: Precision and recall on the development set for named entity recognition. P: precision, R: recall.

\begin{table}
\begin{tabular}{l|c|c}
* & 90.90 \\ Passos et al. (2014) & 94.46 & 90.90 \\ \hline \hline This work: M\({}_{0}\) & 90.00 & 84.44 \\ This work: M\({}_{1}\) & 91.45 & 86.85 \\ This work: M\({}_{2}\) & 92.72 & 89.64 \\ This work: M\({}_{3}\) & 93.27 & 90.57 \\ This work: M\({}_{4}\) & 93.79 & **91.00** \\ \end{tabular}
\end{table}
Table 8: F1-scores on the development and the evaluation sets for named entity recognition.


We concede that our approach is more empirically motivated than theoretically justified. For instance, the choice of \(k\) (line 11) or the combination configuration for \({\cal L}\) (line 13) in Algorithm 1 are rather empirically derived. All the parameters are automatically tuned by running grid searches on the development sets (Sections 4.5 and 5.3); it would be intellectually intriguing to find a more principled way of adjusting these hyper-parameters than just brute-force search.

The locally optimal learning to search is used to help structured learning although it gives a relatively smaller impact to the tasks involving sequence classification such as part-of-speech tagging and named entity recognition. This framework is used because we plan to apply our approach on more structurally oriented tasks such as dependency parsing and AMR parsing. Our work is also related to feature grouping, which has been shown to be beneficial in learning high-dimensional data [20, 26]. It will be interesting to compare our work to the previous work and see the strengths and weaknesses of our approach.

## Acknowledgments

We gratefully acknowledge the support of the Yahoo Academic Career Enhancement Award, the IPsoft Development Enhancement Grant, the University Research Committee Award, and the Infosys Research Enhancement Grant. Any contents expressed in this material are those of the authors and do not necessarily reflect the views of these awards and grants.

## References

* [Bengtson and Roth2008] Eric Bengtson and Dan Roth. 2008. Understanding the Value of Features for Coreference Resolution. In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_, EMNLP'08, pages 294-303.
* [Brown et al.1992] Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992. Class-based n-gram Models of Natural Language. _Computational Linguistics_, 18(4):467-480.
* [Chang et al.2015a] Kai-Wei Chang, He He, Hal Daume III, and John Langford. 2015a. Learning to Search for Dependencies. _arXiv:1503.05615_.
* [Chang et al.2015b] Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daume, and John Langford. 2015b. Learning to Search Better than Your Teacher. In _Proceedings of the 32nd International Conference on Machine Learning_, ICML'15, pages 2058-2066.
* [Choi and McCallum2013] Jinho D. Choi and Andrew McCallum. 2013. Transition-based Dependency Parsing with Selectional Branching. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics_, ACL'13, pages 1052-1062.
* [Choi and Palmer2012] Jinho D. Choi and Martha Palmer. 2012. Fast and Robust Part-of-Speech Tagging Using Dynamic Model Selection. In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics_, ACL'12, pages 363-367.
* [Collins2002] Michael Collins. 2002. Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms. In _Proceedings of the conference on Empirical methods in natural language processing_, EMNLP'02, pages 1-8.
* [Collobert et al.2011] Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural Language Processing (Almost) from Scratch. _Journal of Machine Learning Research_, 12:2493-2537.
* [Crammer and Singer2002] Koby Crammer and Yoram Singer. 2002. On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines. _Journal of Machine Learning Research_, 2:265-292.
* [Devlin et al.2014] Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and Robust Neural Network Joint Models for Statistical Machine Translation. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics_, ACL'14, pages 1370-1380.
* [Duchi et al.2011] John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. _The Journal of Machine Learning Research_, 12(39):2121-2159.
* [Finkel et al.2005] Jenny Finkel, Shipra Dingare, Christopher Manning, Malvina Nissim, and Beatrice Alex. 2005. Exploring the Boundaries: Gene and Protein Identification in Biomedical Text. _BMC Bioinformatics_, 6:S5.
* [Ganchev and Dredze2008] Kuzman Ganchev and Mark Dredze. 2008. Small Statistical Models by Random Feature Mixing. In _Proceedings of the ACL Workshop on Mobile NLP_, pages 604-613.
* [Gimenez and Marquez2004] Jesus Gimenez and Lluis Marquez. 2004. SVMTool: A general POS tagger generator based on Support Vector Machines. In _Proceedings of the 4th International Conference on Language Resources and Evaluation_, LREC'04.
* [Goldberg and Elhadad2008] Yoav Goldberg and Michael Elhadad. 2008. splitSVM: Fast, Space-Efficient, non-Heuristic, Polynomial Kernel Computation for NLP Applications. In _Proceedings of the Annual Conference of the Association for Computational Linguistics_, ACL:HLT'08, pages 237-240.
* [Goldberg and Nivre2012] Yoav Goldberg and Joakim Nivre. 2012. A Dynamic Oracle for Arc-Eager Dependency Parsing. In _Proceedings 

of the 24th International Conference on Computational Linguistics_, COLING'12.
* Kudo and Matsumoto (2003) Taku Kudo and Yuji Matsumoto. 2003. Fast Methods for Kernel-Based Text Analysis. In _Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics_, ACL'04, pages 24-31.
* Le and Mikolov (2014) Quoc V. Le and Tomas Mikolov. 2014. Distributed Representations of Sentences and Documents. In _Proceedings of the 31th International Conference on Machine Learning_, ICML'14, pages 1188-1196.
* Lewis et al. (2004) David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. 2004. RCV1: A New Benchmark Collection for Text Categorization Research. _Journal of Machine Learning Research_, 5:361-397.
* Lin and Wu (2009) Dekang Lin and Xiaoyun Wu. 2009. Phrase Clustering for Discriminative Learning. In _Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics_, ACL'09, pages 1030-1038.
* Manning (2011) Christopher D. Manning. 2011. Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics? In _Proceedings of the 12th international conference on Computational linguistics and intelligent text processing_, CICLE'11, pages 171-189.
* Marcus et al. (1993) Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. _Computational Linguistics_, 19(2):313-330.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeff Dean. 2013. Efficient Estimation of Word Representations in Vector Space. _arXiv:1301.3781_.
* Moore (2015) Robert Moore. 2015. An Improved Tag Dictionary for Faster Part-of-Speech Tagging. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, EMNLP'15, pages 1303-1308.
* Okanohara and Tsujii (2009) Daisuke Okanohara and Jun'ichi Tsujii. 2009. Learning Combination Features with L1 Regularization. In _Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers_, NAACL'09, pages 97-100.
* Passos et al. (2014) Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon Infused Phrase Embeddings for Named Entity Resolution. In _Proceedings of the 18th Conference on Computational Natural Language Learning_, CoNLL'14, pages 78-86.
* Pei et al. (2001) Jian Pei, Jiawei Han, Behzad Mortazavi-Asl, Helen Pinto, Qiming Chen, Umeshwar Dayal, and Meichun Hsu. 2001. PrefixSpan: Mining Sequential Patterns by Prefix-Projected Growth. In _Proceedings of the 17th International Conference on Data Engineering_, pages 215-224.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, EMNLP'14, pages 1532-1543.
* Perkins et al. (2003) Simon Perkins, Kevin Lacker, and James Theiler. 2003. Grafting: Fast, Incremental Feature Selection by Gradient Descent in Function Space. _Journal of Machine Learning Research_, 3:1333-1356.
* Pradhan et al. (2005) Sameer Pradhan, Kadri Hacioglu, Valerie Krugler, Wayne Ward, James H. Martin, and Daniel Jurafsky. 2005. Support Vector Learning for Semantic Argument Classification. _Machine Learning_, 60(1):11-39.
* Ratinov and Roth (2009) Lev Ratinov and Dan Roth. 2009. Design Challenges and Misconceptions in Named Entity Recognition. In _Proceedings of the Thirteenth Conference on Computational Natural Language Learning_, CoNLL'09, pages 147-155.
* Ross et al. (2011) Stephane Ross, Geoffrey J. Gordon, and Drew Bagnell. 2011. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. In _Proceedings of the Workshop on Artificial Intelligence and Statistics_, AI-STATS'11, pages 627-635.
* Shen et al. (2007) Libin Shen, Giorgio Satta, and Aravind Joshi. 2007. Guided Learning for Bidirectional Sequence Classification. In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_, ACL'07, pages 760-767.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, EMNLP'13, pages 1631-1642.
* Sogaard (2011) Anders Sogaard. 2011. Semi-supervised condensed nearest neighbor for part-of-speech tagging. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, ACL:HLT'11, pages 48-52.
* "Johanka" Spoustova, Jan Hajic, Jan Raab, and Miroslav Spousta. 2009. Semi-supervised Training for the Averaged Perceptron POS Tagger. In _Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics_, EACL'09, pages 763-771.
* Strubell et al. (2015) Emma Strubell, Luke Vilnis, Kate Silverstein, and Andrew McCallum. 2015. Learning Dynamic Feature Selection for Fast Sequential Prediction. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics_, ACL'16, pages 146-155.
* Sun (2014) Xu Sun. 2014. Structure Regularization for Structured Prediction. In _Proceedings of Advances in Neural Information Processing Systems._, NIPS'14, pages 2402--2410 

Jun Suzuki and Hideki Isozaki. 2008. Semi-Supervised Sequential Labeling and Segmentation Using Gigap-Word Scale Unlabeled Data. In _Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, ACL:HLT'08, pages 665-673.
* Suzuki and Nagata (2013) Jun Suzuki and Masaaki Nagata. 2013. Supervised Model Learning with Feature Grouping based on a Discrete Constraint. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 18-23.
* Volume 4_, CONLL '03, pages 142-147. Association for Computational Linguistics.
* Toutanova et al. (2003) Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network. In _Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology_, NAACL'03, pages 173-180.
* Tsuboi (2014) Yuta Tsuboi. 2014. Neural Networks Leverage Corpus-wide Information for Part-of-speech Tagging. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, EMNLP'14, pages 938-950.
* Turian et al. (2010) Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word Representations: A Simple and General Method for Semi-Supervised Learning. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_, ACL'10, pages 384-394.
* Weinberger et al. (2009) Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. 2009. Feature Hashing for Large Scale Multitask Learning. In _Proceedings of the 26th Annual International Conference on Machine Learning_, ICML'09, pages 1113-1120.
* Xiao (2010) Lin Xiao. 2010. Dual Averaging Methods for Regularized Stochastic Learning and Online Optimization. _Journal of Machine Learning Research_, 11:2543-2596.
* Xue and Palmer (2004) Nianwen Xue and Martha Palmer. 2004. Calibrating Features for Semantic Role Labeling. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, EMNLP'04, pages 88-94.
* Yu et al. (2014) Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. 2014. Deep Learning for Answer Sentence Selection. In _Proceedings of the NIPS Deep Learning Workshop_.
* Zhang and Johnson (2003) Tong Zhang and David Johnson. 2003. A Robust Risk Minimization Based Named Entity Recognition System. In _Proceedings of the 7th Conference on Natural Language Learning_, CONLL'03, pages 204-207.
* Zhang and Nivre (2011) Yue Zhang and Joakim Nivre. 2011. Transition-based Dependency Parsing with Rich Non-local Features. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, ACL'11, pages 188-193.
* Zhong and Kwok (2011) Wenliang Zhong and James Kwok. 2011. Efficient Sparse Modeling with Automatic Feature Grouping. In Lise Getoor and Tobias Scheffer, editors, _Proceedings of the 28th International Conference on Machine Learning (ICML-11)_, ICML '11, pages 9-16, New York, NY, USA, June. ACM.