Venue: EMNLP 2017

# Learning What's Easy: Fully Differentiable Neural Easy-First Taggers

Andre F. T. Martins

Unbabel

& Instituto de Telecomunicacoes

Lisbon, Portugal

andre.martins@unbabel.com &Julia Kreutzer

Computational Linguistics

Heidelberg University, Germany

kreutzer@cl.uni-heidelberg.de

This research was partially carried out during an internship at Unbabel.

###### Abstract

We introduce a novel neural easy-first decoder that learns to solve sequence tagging tasks in a flexible order. In contrast to previous easy-first decoders, our models are end-to-end differentiable. The decoder iteratively updates a "sketch" of the predictions over the sequence. At its core is an attention mechanism that controls which parts of the input are strategically the best to process next. We present a new constrained softmax transformation that ensures the same cumulative attention to every word, and show how to efficiently evaluate and backpropagate over it. Our models compare favourably to BILSTM taggers on three sequence tagging tasks.

## 1 Introduction

In the last years, neural models have led to major advances in several structured NLP problems, including sequence tagging (Plank et al., 2016; Lample et al., 2016), sequence-to-sequence prediction (Sutskever et al., 2014), and sequence-to-tree (Dyer et al., 2015). Part of the success comes from clever architectures such as (bidirectional) long-short term memories (BILSTMs; Hochreiter and Schmidhuber (1997); Graves et al. (2005)) and attention mechanisms (Bahdanau et al., 2015), which are able to select the pieces of context relevant for prediction.

A noticeable aspect about many of the systems above is that they typically decode from left to right, greedily or with a narrow beam. While this is computationally convenient and reminiscent of the way humans process spoken language, the combination of unidirectional decoding and greediness leads to error propagation and suboptimal classification performance. This can partly be mitigated by globally normalized models (Andor et al., 2016) and imitation learning (Daume et al., 2009; Ross et al., 2011; Bengio et al., 2015), however these techniques still have a left-to-right bias.

**Easy-first decoders**(Tsuruoka and Tsujii, 2005; Goldberg and Elhadad, 2010, SS2) are an interesting alternative: instead of a fixed decoding order, these methods schedule their own actions by prefering "easier" decisions over more difficult ones. A disadvantage is that these models are harder to learn, due to the factorial number of orderings leading to correct predictions. Usually, gradients are _not_ backpropagated over this combinatorial latent space (Kiperwasser and Goldberg, 2016a), or a separate model is used to determine the easiest next move (Clark and Manning, 2016).

In this paper, we develop novel, **fully differentiable**, **neural easy-first sequence taggers** (SS3). Instead of taking discrete actions, our decoders use an attention mechanism to decide (in a soft manner) which word to focus on for the next tagging decision. Our models are able to learn their own sense of "easiness": the words receiving focus may not be the ones the model is most confident about, but the best to avoid error propagation in the long run. To make sure that all words receive the same cumulative attention, we further contribute with a new **constrained softmax transformation** (SS4). This transformation extends the softmax by permitting upper bound constraints on the amount of probability a word can receive. We show how to evaluate this transformation and backpropagate its gradients.

We run experiments in three sequence tagging tasks: multilingual part-of-speech (POS) tagging, named entity recognition (NER), and word-level quality estimation (SS5). We complement our findings with a visual analysis of the attention distribu

tions produced by the decoder, to help understand what tagging decisions the model finds the easiest.

## 2 Easy-First Decoders

The idea behind easy-first decoding is to perform "easier" and less risky decisions before committing to more difficult ones Tsuruoka and Tsujii (2005); Goldberg and Elhadad (2010); Ma et al. (2013). Alg. 1 shows the overall procedure for a sequence tagging problem (the idea carries out to other structured problems). Let \(x_{1:L}\) be an input sequence (e.g. words in a sentence) and \(y_{1:L}\) be the corresponding tag sequence (e.g. their POS tags). The algorithm assigns tags one position \(i\) at the time, maintaining a set \(\mathcal{B}\) of **covered positions**. It also maintains a set \(\mathcal{S}\) of pairs \((i,\hat{y}_{i})\), storing the tags that have already been predicted at those positions. We can regard this set as a **sketch** of the output sequence, built incrementally while the algorithm is executed. At each time step, the model computes a score \(f(i,y_{i};x_{1:L},\mathcal{S})\) for each position \(i\notin\mathcal{B}\) and each candidate tag \(y_{i}\), taking into account the current "sketch" \(\mathcal{S}\), which provides useful contextual information. The "easiest" position and the corresponding tag are then jointly obtained by maximizing this score (line 6). The algorithm terminates when all positions are covered.

Previous work has trained easy-first systems with variants of the perceptron algorithm Goldberg and Elhadad (2010); Ma et al. (2013) or with a gradient-based method Kiperwasser and Goldberg (2016)--but without backpropagating information about the best ordering chosen by the algorithm (only tag mistakes). In fact, doing so directly would be hard, since the space of possible orderings is combinatorial--the argmax in line 6 is not continuous, let alone differentiable. In the next section, we introduce a fully differentiable easy-first system that sidesteps this problem by working with a "continuous" space of actions.

```
Input: input sequence \(x_{1:L}\) Output: tagged sequence \(\widehat{y}_{1:L}\)
1: initialize \(\mathcal{B}=\mathcal{S}=\varnothing\)
2:while\(\mathcal{B}\neq\left\{1,...,L\right\}\)do
3:for each non-covered position \(i\notin\mathcal{B}\)do
4: compute scores \(f(i,y_{i};x_{1:L},\mathcal{S}),~{}\forall y_{i}\)
5:endfor
6:\((j,\widehat{y}_{j})=\text{argmax}_{i,y_{j}}f(i,y_{i};x_{1:L},\mathcal{S})\)
7:\(\mathcal{S}\leftarrow\mathcal{S}\cup\{(j,\widehat{y}_{j})\}\), \(\mathcal{B}\gets\mathcal{B}\cup\{j\}\)
8:endwhile
```

**Algorithm 1** Easy-First Sequence Tagging

## 3 Neural Easy-First Sequence Taggers

Let \(\Delta^{L-1}:=\{\boldsymbol{\alpha}\in\mathbb{R}^{L}\,|\,\mathbf{1}^{\top} \boldsymbol{\alpha}=1,\boldsymbol{\alpha}\geq\mathbf{0}\}\) be the probability simplex. Our **neural easy-first decoders** depart from Alg. 1 in the following key points:

* Instead of picking the position with the largest score at each step (line 6 in Alg. 1), we compute a (continuous) attention distribution \(\boldsymbol{\alpha}\in\Delta^{L-1}\) over word positions.
* Instead of a set of covered positions \(\mathcal{B}\), we maintain a (continuous) **cumulative attention vector**\(\boldsymbol{\beta}\in\mathbb{R}^{L}\) (ideally in \([0,1]^{L}\)) over the \(L\) positions in the sequence.
* The sketch set \(\mathcal{S}\) is replaced by a **sketch matrix**\(\mathbf{S}\in\mathbb{R}^{D_{s}\times L}\), whose columns are \(D_{s}\)-dimensional vector representations of the output labels to be predicted.

The high-level procedure is shown in Figure 1. We describe two models that implement this procedure: a **single-state model** and a **full-state model**. They differ in the way they update the sketch matrix: the single-state model applies a rank-one update, while the full-state model does a full update.

### Single-State Model (NEF-S)

Let \(\mathsf{concat}(\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{K})\in\mathbb{R}^ {\sum_{k=1}^{K}D_{k}}\) be the concatenation of the vectors \(\boldsymbol{x}_{k}\in\mathbb{R}^{D_{k}}\). We use the shorthand \(\textsf{affine}(\boldsymbol{x}):=\mathbf{W}\boldsymbol{x}+\mathbf{b}\) to denote an affine transformation of \(\boldsymbol{x}\), where \(\mathbf{W}\) is a weight matrix and \(\mathbf{b}\) is a bias vector.

Alg. 2 shows the overall procedure. We start by encoding the input sequence \(x_{1:L}\) as a matrix

Figure 1: A neural easy-first system applied to a POS tagging problem. Given the current input/sketch representation, an attention mechanism decides where to focus (see bar plot) and is used to generate the next sketch. Right: A sequence of sketches \((\mathbf{S}^{n})_{n=1}^{N}\) generated along the way.

 

\({\bf H}=[{\mathbf{h}_{1},\ldots,\mathbf{h}_{L}}]\in\mathbb{R}^{D_{k}\times L}\) (line 2). Our model is completely agnostic about this encoding step. In our experiments, we compute \({\bf H}\) by composing a lookup embedding layer with a BILSTM Hochreiter and Schmidhuber (1997); Graves et al. (2005), as this strategy was successful in similar structured prediction tasks. However, other choices are possible, for example using convolutional layers.

As stated above, our algorithm maintains a cumulative attention vector \(\mathbf{\beta}\in\mathbb{R}^{L}\) and a sketch matrix \({\bf S}\in\mathbb{R}^{D_{s}\times L}\), both with all entries initialized to zero. It then performs \(N\) sketching steps, which progressively refine this sketch matrix, producing versions \({\bf S}^{1},\ldots,{\bf S}^{N}\). At the \(n\)th step, the following operations are performed:

Input-Sketch Contextual Representation.For each word \(i\in[L]\), we compute a state \(\mathbf{c}_{i}^{n}\) summarizing the surrounding local information about other words and sketches. We use a simple vector concatenation over a \(w\)-wide context window:

\[\mathbf{c}_{i}^{n}=\mathsf{concat}(\mathbf{h}_{i-w},\dots,\mathbf{h}_{i+w},\mathbf{s}_{i-w}^{ n-1},\dots,\mathbf{s}_{i+w}^{n-1}), \tag{1}\]

where we denote by \(\mathbf{s}_{j}^{n-1}\) the \(j\)th column of \({\bf S}^{n-1}\). The intuition is that the current sketches provide valuable information about the neighboring words' predictions that can influence the prediction for the \(i\)th word. In the vanilla easy-first algorithm, this was assumed in the score computation (line 4 of Alg. 1).

Attention Mechanism.We then use an attention mechanism to decide what is the "best" word to focus on next. This is done in a similar way as the feedforward attention proposed by Bahdanau et al. (2015). We first compute a score for each word \(i\in[L]\) based on its contextual representation,

\[z_{i}^{n}=\mathbf{v}^{\top}\mathsf{tanh}(\mathsf{affine}(\mathbf{c}_{i}^{n})), \tag{2}\]

where \(\mathbf{v}\in\mathbb{R}^{D_{x}}\) is a model parameter. Then, we aggregate these scores in a vector \(\mathbf{z}^{n}\in\mathbb{R}^{L}\) and apply a transformation \(\rho\) to map them to a probability distribution \(\mathbf{\alpha}^{n}\in\Delta^{L-1}\) (optionally taking into account the past cumulative attention \(\mathbf{\beta}^{n-1}\)):

\[\mathbf{\alpha}^{n}=\rho(\mathbf{z}^{n};\mathbf{\beta}^{n-1}). \tag{3}\]

The "standard" choice for \(\rho\) is the softmax transformation. However, in this work, we consider other possible transformations (to be described in SS4). After this, the cumulative attention is updated via \(\mathbf{\beta}^{n}=\mathbf{\beta}^{n-1}+\mathbf{\alpha}^{n}\).

Sketch Generation.Now that we have a distribution \(\mathbf{\alpha}^{n}\in\Delta^{L-1}\) over word positions, it remains to generate a sketch for those words. We first compute a **single-state** vector representation of the entire sentence \(\bar{\mathbf{c}}^{n}=\sum_{i=1}^{L}\alpha_{i}^{n}\mathbf{c}_{i}^{n}\) as the weighted average of the word states defined in Eq. 1. Then, we update each column of the sketch matrix as:1

Footnote 1: Note that this is a rank-one update, as it can be written in matrix notation as \({\bf S}^{n}={\bf S}^{n-1}+\text{tanh}(\mathsf{affine}(\bar{\mathbf{c}}^{n}))\cdot \mathbf{\alpha}^{n\top}\).

\[\mathbf{s}_{i}^{n}=\mathbf{s}_{i}^{n-1}+\alpha_{i}^{n}\cdot\mathsf{ tanh}(\mathsf{affine}(\bar{\mathbf{c}}^{n})),~{}\forall i\in[L]. \tag{4}\]

The intuition for this update is the following: in the extreme case where the attention distribution is peaked on a single word (say, the \(k\)th word, \(\mathbf{\alpha}^{n}=\mathbf{e}_{k}\)), we obtain \(\bar{\mathbf{c}}^{n}=\mathbf{c}_{k}^{n}\) and the sketch update only affects that word, i.e.,

\[\mathbf{s}_{i}^{n}=\left\{\begin{array}{ll}\mathbf{s}_{i}^{n-1}+\mathsf{ tanh}(\mathsf{affine}(\mathbf{c}_{k}^{n}))&\mbox{if $i=k$}\\ \mathbf{s}_{i}^{n-1}&\mbox{if $i\neq k$}.\end{array}\right. \tag{5}\]

This is similar to the sketch update in the original easy-first algorithm (line 7 in Alg. 1).

```
0: input sequence \(x_{1:L}\), sketch steps \(N\)
0: tagged sequence \(\widehat{y}_{1:L}\)
1: initialize \(\mathbf{\beta}^{n}=\mathbf{0}\) and \(\mathbf{s}_{i}^{n}=\mathbf{0},~{}\forall i\in[L]\)
2: encode sequence as \(|\mathbf{h}_{1},\dots,\mathbf{h}_{L}|\)
3:for\(n=1,2,\dots,N\)do
4:for each position \(i\in[L]\)do
5: compute \(\mathbf{c}_{i}^{n}\) and \(z_{i}^{n}\) (Eqs. 1-2)
6: endfor
7: compute attention \(\mathbf{\alpha}^{n}=\rho(\mathbf{z}^{n};\mathbf{\beta}^{n-1})\)
8:for each position \(i\in[L]\)do
9: refine sketch \(\mathbf{s}_{i}^{n}\) from \(\alpha_{i}^{n}\) and \(\mathbf{c}_{i}^{n}\), via Eq. 4 (single-stage) or Eq. 7 (full-state)
10:endfor
11:\(\mathbf{\beta}^{n}=\mathbf{\beta}^{n-1}+\mathbf{\alpha}^{n}\)
12:endfor
13:for each position \(i\in[L]\)do
14:\(\mathbf{p}_{i}=\mathsf{softmax}(\mathsf{affine}(\mathsf{concat}(\mathbf{h}_{i},\mathbf{ s}_{i}^{N})))\)
15: predict \(\widehat{y}_{i}=\mathsf{argmax}_{y_{i}}p_{i}(y_{i})\)
16:endfor
```

**Algorithm 2** Neural Easy-First Sequence Tagging

The three operations above are repeated \(N\) times (or "sketch steps"). The standard choice is \(N=L\) (one step per word), which mimics the vanilla easy-first algorithm. However, it is possible to have fewer steps (or more, if we want the decoder to be able to "self-correct"). After completing the \(N\) sketch steps, we obtain the final sketch matrix \({\bf S}^{N}=[\mathbf{s}_{1}^{N},\dots,\mathbf{s}_{L}^{N}]\). Then, we compute a tag probability for every word as follows:2

Footnote 2: We found the concatenation with \(\mathbf{h}_{i}\) in Eq. 6 beneficial transfer input information directly to the output layer, avoiding the need of flowing this information through the sketches.

\[\mathbf{p}_{i}=\mathsf{softmax}(\mathsf{affine}(\mathsf{concat}(\mathbf{h}_{i},\mathbf{s}_{i}^{ N}))).\] (6) 

In SS5, we compare this to a BILSTM tagger, which predicts according to \({\mathbf{p}_{i}=\mathsf{softmax}(\mathsf{affine}(\mathbf{h}_{i}))}\).

### Full-State Model (NEF-F)

The full-state model differs from the single-state model in SS3.1 by computing a full matrix for every sketch step, instead of a single vector. Namely, instead of Eq. 4, it does the following sketch update for every word \(i\in[L]\):

\[\mathbf{s}^{n}_{i}=\mathbf{s}^{n-1}_{i}+\alpha^{n}_{i}\cdot\mathsf{tanh}(\mathsf{ affine}(\mathbf{e}^{n}_{i})). \tag{7}\]

Note that the only difference is in replacing the single vector \(\bar{\mathbf{c}}^{n}\) by the word-specific vector \(\mathbf{c}^{n}_{i}\). As a result, this is no longer a rank-one update of the sketch matrix, but a full update. In the extreme case where the attention is peaked on a single word, the sketch update reduces to the same form as in the single-state model (Eq. 5). However, the full-state model is generally more flexible and allows processing words in parallel, since it allows different sketch updates for multiple words receiving attention, instead of trying to force those words to receive the same update. We will see in the experiments (SS5) that this flexibility can be important in practice.

### Computational Complexity

For both models, assuming that the \(\rho(\mathbf{z};\mathbf{\beta})\) transformation in Eq. 3 takes \(O(L)\) time to compute, the total runtime of Alg. 2 is \(O((N+K)L)\) (where \(K\) is the number of tags), which becomes \(O(L^{2})\) if \(K\leq N=L\). This is so because the input-sketch representation, the attention mechanism, and the sketch generation step all have \(O(L)\) complexity, and the final softmax layer requires \(O(KL)\) operations. This is the same runtime of the vanilla easy-first algorithm, though the latter can be reduced to \(O(KL\text{log}L)\) with caching and a heap, if the scores in line 4 depend only on local sketches Goldberg and Elhadad (2010). By comparison, a standard BILSTM tagger has runtime \(O(KL)\).

## 4 Constrained Softmax Attention

An important part of our models is their attention component (line 7 in Alg. 2). To keep the "easy-first" intuition, we would like the transformation \(\rho\) in Eq. 3 to have a couple of properties:

1. **Sparsity:** being able to generate sparse distributions \(\mathbf{\alpha}^{n}\) (ideally, peaked on a single word).
2. **Evenness:** over iterations, spreading attention evenly over the words. Ideally, the cumulative attention should satisfy \(\mathbf{\beta}^{n}\in[0,1]^{L}\) for every \(n\in[N]\) and \(\mathbf{\beta}^{N}=\mathbf{1}\). The standard choice for attention mechanisms is the **softmax** transformation, \(\mathbf{\alpha}^{n}=\mathsf{softmax}(\mathbf{z}^{n})\). However, the softmax does not satisfy either of the properties above. For the first requirement, we could incorporate a "temperature" parameter in the softmax to push for more peaked distributions. However, this does not guarantee sparsity (only "hard attention" in the limit) and we found it numerically unstable when plugged in Alg. 2. For the second one, we could add a penalty before the softmax transformation, \(\mathbf{\alpha}^{n}=\mathsf{softmax}(\mathbf{z}^{n}-\lambda\mathbf{\beta}^{n-1})\), where \(\lambda\geq 0\) is a tunable hyperparameter. This strategy was found effective to prevent a word to receive too much attention, but it made the model less accurate. An alternative is the **sparsemax** transformation Martins and Astudillo (2016):

\[\mathsf{sparsemax}(\mathbf{z})=\underset{\mathbf{\alpha}\in\Delta^{L-1}}{\operatorname{ argmin}}~{}\|\mathbf{\alpha}-\mathbf{z}\|^{2}. \tag{8}\]

The sparsemax maintains most of the appealing properties of the softmax (efficiency to evaluate and backpropagate), and it is able to generate truly sparse distributions. However, it still does not satisfy the "evenness" property. Instead, we propose a novel **constrained softmax** transformation that satisfies both requirements. It resembles the standard softmax, but it allows imposing hard constraints on the maximal probability assigned to each word. Let us start by writing the (standard) softmax in the following variational form (7):

\[\mathsf{softmax}(\mathbf{z}) = \underset{\mathbf{\alpha}\in\Delta^{L-1}}{\operatorname{argmin}}~{} \mathbb{KL}(\mathbf{\alpha}\|\mathsf{softmax}(\mathbf{z})) \tag{9}\] \[= \underset{\mathbf{\alpha}\in\Delta^{L-1}}{\operatorname{argmin}}- \mathbb{H}(\mathbf{\alpha})-\mathbf{z}^{\top}\mathbf{\alpha},\]

where \(\mathbb{KL}\) and \(\mathbb{H}\) denote the Kullback-Leibler divergence and the entropy, respectively. Based on this observation, we define the constrained softmax transformation as follows:

\[\mathsf{csoftmax}(\mathbf{z};\mathbf{u}) = \underset{\mathbf{\alpha}\in\Delta^{L-1}}{\operatorname{argmin}}~{}- \mathbb{H}(\mathbf{\alpha})-\mathbf{z}^{\top}\mathbf{\alpha}\] s.t. \[\mathbf{\alpha}\leq\mathbf{u}, \tag{10}\]

where \(\mathbf{u}\in\mathbb{R}^{L}\) is a vector of upper bounds. Note that, if \(\mathbf{u}\geq\mathbf{1}\), all constraints are loose and this 

reduces to the standard softmax; on the contrary, if \({\mathbf{u}}\in\Delta^{L-1}\), they are tight and we must have \({\mathbf{\alpha}}={\mathbf{u}}\) due to the normalization constraint. Thus, we propose the following for Eq. 3:

\[{\mathbf{\alpha}}^{n}=\mathsf{csoftmax}({\mathbf{z}}^{n};{\bf 1}-{\mathbf{\beta}}^{n-1}). \tag{11}\]

The constraints guarantee \({\mathbf{\beta}}^{n}={\mathbf{\beta}}^{n-1}+{\mathbf{\alpha}}^{n}\leq{\bf 1}\). Since \({\bf 1}^{\top}{\mathbf{\beta}}^{N}=\sum_{n=1}^{N}{\bf 1}^{\top}{\mathbf{\alpha}}^{ n}=N\), they also ensure that \({\mathbf{\beta}}^{N}={\bf 1}\), hence the "evenness" property is fully satisfied. Intuitively, each word gets a credit of one unit of attention that is consumed during the execution of the algorithm. When this credit expires, all subsequent attention weights for that word will be zero.

```
0:\({\mathbf{z}}\), \({\mathbf{u}}\)
0:\({\mathbf{\alpha}}=\mathsf{csoftmax}({\mathbf{z}};{\mathbf{u}})\)
1: initialize \(s:=0\), \(A:=[L],Z:=\sum_{i=1}^{K}\text{exp}(z_{i})\)
2: sort \(q_{i_{1}}:=\infty\geq q_{i_{L}}\), where \(q_{i}=\text{exp}(z_{i})/u_{i}\), \(\forall i\in[L]\)
3:for\(k=1\) to \(L\)do
4:\(\alpha_{i_{k}}:=\text{exp}(z_{i_{k}})(1-s)/Z\)
5:if\(\alpha_{i_{k}}>u_{i_{k}}\)then
6:\(Z:=Z-\text{exp}(z_{i_{k}})\)
7:\(\alpha_{i_{k}}:=u_{i_{k}},\quad s:=s+u_{i_{k}},\quad{\mathcal{A}}:={\mathcal{ A}}\setminus\{i_{k}\}\)
8:endif
9:endfor
```

**Algorithm 3** Constrained Softmax Backprop

The next proposition shows how to evaluate the constrained softmax and compute its gradients.

**Proposition 1**: _Let \({\mathbf{\alpha}}=\mathsf{csoftmax}({\mathbf{z}};{\mathbf{u}})\), and define the set \({\mathcal{A}}=\{i\in[L]~{}|~{}\alpha_{i}<u_{i}\}\) of the constraints in Eq. 10 that are met strictly. Then:_

* **Forward propagation.** _The solution of Eq._ 10 _can be written in closed form as_ \(\alpha_{i}=\min\{\text{exp}(z_{i})/Z,u_{i}\}\)_, where_ \(Z=\frac{\sum_{i\in{\mathcal{A}}}\text{exp}(z_{i})}{1-\sum_{i\notin{\mathcal{A} }}u_{i}}\)__
* **Gradient backpropagation.** _Let_ \(L({\mathbf{\theta}})\) _be a loss function,_ \(\mathrm{d}{\mathbf{\alpha}}=\nabla_{\mathbf{\alpha}}L({\mathbf{\theta}})\) _be the output gradient, and_ \(\mathrm{d}{\mathbf{z}}=\nabla_{\mathbf{z}}L({\mathbf{\theta}})\) _and_ \(\mathrm{d}{\mathbf{u}}=\nabla_{\mathbf{u}}L({\mathbf{\theta}})\) _be the input gradients. Then, we have:_ \[\mathrm{d}{z}_{i} = \mathds{1}(i\in{\mathcal{A}})\alpha_{i}(\mathrm{d}\alpha_{i}-m)\] (12) \[\mathrm{d}u_{i} = \mathds{1}(i\notin{\mathcal{A}})(\mathrm{d}\alpha_{i}-m),\] (13) _where_ \(m=(\sum_{i\in{\mathcal{A}}}\alpha_{i}\,\mathrm{d}\alpha_{i})/(1-\sum_{i\notin {\mathcal{A}}}u_{i})\)_._

See App. A (supplementary material). \(\blacksquare\)

Algs. 3-4 turn the results in Prop. 1 into concrete procedures for evaluating \(\mathsf{csoftmax}\) and for backpropagating its gradient. Their runtimes are respectively \(O(L\text{log}L)\) and \(O(L)\).

## 5 Experiments

We evaluate our neural easy-first models in three sequence tagging tasks: POS tagging, NER, and word quality estimation.

### Part-of-Speech Tagging

We ran POS tagging experiments in 12 languages from the Universal Dependencies project v1.4 Nivre et al. (2016), using the standard splits. The datasets contain 17 universal tags.3

Footnote 3: Our choice of languages covers 8 families: Romance (French, Portuguese, Spanish), Germanic (English, German), Slavic (Czech, Russian), Semitic (Arabic), Indo-Iranian (Hindì, Austronesian ( Indonesian), Sino-Tibettan (Chinese), and Japonic (Japanese). For all languages, we used the datasets that have the canonical language name, except for Russian, where we used the (much larger) SynTagRus corpus. We preferred large datasets over small ones, to reduce the impact of overfitting in our analysis.

We implemented Alg. 2 in DyNet Neubig et al. (2017), which we extended with the constrained softmax operator (Algs. 3-4).4 We used 64-dimensional word embeddings, initialized with pre-trained Polyglot vectors Al-Rfou et al. (2013). Apart from the words, we embedded prefix and suffix character \(n\)-grams with \(n\leq 4\). We set the affix embedding size to 50 and summed all these embeddings; in the end, we obtained a 164-dimensional representation for each word (words, prefixes, and suffixes). We then fed these embeddings into a BILSTM (with 50 hidden units in each direction) to obtain the encoder states \([{\mathbf{h}}_{1},\dots,{\mathbf{h}}_{L}]\in\mathbb{R}^{100\times L}\). The other hyperparameters were set as follows: we used a context size \(w=2\), set the pre-attention size \(D_{z}\) and the sketch size \(D_{s}\) to \(50\), and applied dropout with a probability of \(0.2\) after the embedding and BILSTM layers and before the final softmax output layer.5 We ran 20 epochs of Adagrad to minimize the cross-entropy loss, with a stepsize 

clipping of \(5\) (DyNet's default). We excluded from the training set sentences longer than 50 words.

Table 1 compares several variants of our neural easy-first system--the single-state model (NEF-S), the full-state model (NEF-F), and the latter with softmax, sparsemax, and csoftmax attention. We used as many sketch steps as the number of words, \(N=L\). As baselines, we used:

* A feature-based linear model [23].
* A BILSTM tagger identical to our system, but without sketch steps (\(N=0\)).
* A vanilla easy-first tagger (Alg. 1), using the argument of the softmax in Eq. 6 as the scoring function. This uses the same sketch representations as the neural easy-first systems, but replaces the attention mechanism by "hard" attention placed on the highest scored word.

For comparison, we also show the accuracies reported by Gillick et al. (2016) for their byte-to-span system (trained separately on each language) and by Plank et al. (2016) for their state-of-the-art multi-task BILSTM tagger (these results are not fully comparable though, due to different treebank versions).

Among the neural easy-first systems, we observe that NEF-F with csoftmax attention generally outperforms the others, but the differences are very slight (excluding the sparsemax attention system, which performed substancially worse). This system wins over the linear system for all languages but Spanish, and over the BILSTM baseline for 9 out of 12 languages (loses in Arabic and German, and ties in Japanese). Note, however, that the differences are small (95.47% against 95.39%, averaged across treebanks). We conjecture that this is due to the fact that the BILSTM already captures most of the relevant context in its encoder. Our NEF-F system with csoftmax also wins over the vanilla easy-first system for 10 out of 12 languages (arguably due to its ability to backpropagate the gradients through the soft attention mechanism), but the difference in the average score is again small (95.47% against 95.42%).

Figure 2 depicts some patterns learned by the NEF-F model with various attention types. With the csoftmax, the model learns to move left and right, and the main verb _"thought"_ is the most

\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c|c} \hline  & Ara. & Chi. & Cze. & Eng. & Fre. & Ger. & Hin. & Ind. & Jap. & Por. & Rus. & Spa. & **Avg.** \\ \hline Gillick et al. (2016)† & – & – & _88.44_ & _94.00_ & _95.17_ & _92.34_ & – & _91.03_ & – & – & – & _95.26_ & – \\ Plank et al. (2016)� & † & _98.91_ & – & _98.24_ & _95.16_ & _96.11_ & _93.38_ & _97.10_ & _93.41_ & – & _97.90_ & – & _95.74_ & – \\ \hline Linear (TurboTagger) & 95.18 & 91.38 & 98.07 & 94.43 & 96.48 & 91.92 & 95.98 & 93.12 & 89.35 & 96.63 & 97.32 & **95.44** & 94.88 \\ BILSTM & 95.60 & 92.73 & 98.46 & 94.94 & 96.37 & **92.97** & 96.80 & 93.79 & 92.72 & 97.01 & 97.83 & 95.27 & 95.39 \\ Vanilla Easy-First & 95.62 & 92.78 & 98.50 & 94.78 & 96.35 & 92.91 & 96.81 & 93.68 & 92.68 & **97.12** & 97.93 & 95.31 & 95.42 \\ \hline NEF-S, csoftmax & **95.63** & 92.87 & 98.50 & 94.91 & 96.41 & 92.86 & 96.79 & 93.65 & **92.93** & 96.57 & 97.91 & 95.42 & 95.42 \\ NEF-F, softmax & 95.61 & 92.92 & 98.47 & **95.15** & 96.52 & 92.96 & 96.75 & 93.67 & 92.57 & **97.12** & 97.92 & 95.41 & 95.43 \\ NEF-F, sparsemax & 95.14 & 92.86 & 97.75 & 93.97 & 91.79 & 90.27 & 89.40 & 93.48 & 92.61 & 95.27 & 96.32 & 95.36 & 93.88 \\ NEF-F, csoftmax & 95.53 & **92.99** & **98.53** & 95.01 & **96.70** & 92.93 & **96.98** & **93.81** & 92.72 & 97.04 & **97.94** & 95.42 & **95.47** \\ \hline \end{tabular}
\end{table}
Table 1: POS tagging accuracies. The average in the rightmost column is over the words of each treebank.

Figure 2: Attention visualization for English POS tagging. From the left: constrained softmax, softmax, sparsemax, vanilla. Rows correspond to attention vectors (high values in yellow, low ones in dark blue).

 

prominent candidate for the easiest decision. In fact, in 57% of the test sentences, the model focuses first on a verb. The "raindrop" appearance of the plot is due to the evenness property of csoftmax, which causes the attention over a word to increase gradually until the cumulative attention is exhausted. This constrasts with the softmax attention (less diverse and non-sparse) and the sparsemax (sparse, but not even). We show for comparison the (hard) decisions made by the vanilla easy-first decoder.

### Named Entity Recognition

Next, we applied our model to NER. We used the official datasets from the CoNLL 2002-3 shared tasks Sang et al. (2002); Sang and De Meulder (2003), which tag names, locations, and organizations using a BIO scheme, and cover four languages Dutch, English, German, and Spanish). We made two experiments: one using the exact same BILSTM and NEF models with a standard softmax output layer, as in SS5.1 (which does not guarantee valid segmentations), and another one replacing the output softmax layer by a sequential CRF layer, which requires learning \(O(K^{2})\) additional parameters for pairs of consecutive tags Huang et al. (2015); Lample et al. (2016). We used the same hyperparameters as in the POS tagging experiments, except the dropout probability, which was set to \(0.3\) (tuned on the validation set). For English, we used pre-trained 300-dimensional GloVe-840B embeddings Pennington et al. (2014); for Spanish and German, we used the 64-dimensional word embeddings from Lample et al. (2016); for Dutch we used the aforementioned Polyglot vectors. All embeddings are fine-tuned during training. Since many words are not entities, and those receive a default "outside" tag, we expect that fewer sketch steps are necessary to achieve top performance.

Table 2 shows the results, which confirm this hypothesis. We compare the same BILSTM baseline to our NEF-S and NEF-F models with csoftmax attention (with and without the CRF output layer), varying the maximum number of sketch steps. We also compare against the byte-to-span model of Gillick et al. (2016) and the state-of-the-art character-based LSTM-CRF system of Lample et al. (2016).6 We can see that, for all languages, the NEF-CRF-F model with 5 steps is consistently better than the BILSTM-CRF and, with the exception of English, the NEF-CRF-S model. The same holds for the BILSTM and NEF-F models without the CRF output layer. With the exception of German, increasing the number of steps did not make a big difference.

Footnote 6: The current state of the art on these datasets Gillick et al. (2016); Lample et al. (2016); Ma and Hovy (2016) is achieved by more sophisticated systems with more parameters, mixing character and word-based models, sharing a model across languages, or combining CRFs with convolutional and recurrent layers. We used simpler models in our experiments since our goal is to assess how much the neural easy-first systems can bring in addition to a BILSTM system, rather than building a state-of-the-art system.

Figure 3 shows the attention distributions over the sketch steps for an English sentence, for full-state models trained with \(N\in\{5,L\}\). The model with \(L\) sketch steps learned that it is easiest to focus on the beginning of a named entity, and then to move to the right to identify the full span. The model with only 5 sketch steps learns to go straight to the point, placing most attention on the entity words and ignoring most of the o-tokens.

### Word-Level Quality Estimation

Finally, we evaluate our model's performance on word-level translation quality estimation. The goal is to evaluate a translation system's quality without access to reference translations Blatz et al. (2004); Specia et al. (2013). Given a sentence pair (a source sentence and its machine translated sentence in a target language), a word-level system classifies each target word as ok or bad. We used the official English-German dataset from the WMT16 shared task Bojar et al. (2016).

This task differs from the previous ones in which its input is a sentence pair and not a single

\begin{table}
\begin{tabular}{l l l l l} \hline \hline  & Dut. & Eng. & Ger. & Spa. \\ \hline Gillick et al. (2016) & 78.08 & 84.57 & 72.08 & 81.83 \\ Lample et al. (2016) & **81.74** & **90.94** & **78.76** & **85.75** \\ \hline \hline BILSTM & 76.56 & 85.74 & 70.05 & 77.00 \\ \hline NEF-S, \(N=5\) & 77.37 & 86.40 & 72.27 & 75.80 \\ NEF-F, \(N=5\) & 77.96 & 86.11 & 72.60 & **79.22** \\ NEF-F, \(N=10\) & 77.52 & **87.11** & **72.96** & 78.99 \\ NEF-F, \(N=L\) & **78.46** & 86.36 & 72.35 & 78.87 \\ \hline \hline BILSTM-CRF & 79.00 & 86.96 & 72.98 & 80.44 \\ \hline NEF-CRF-S, \(N=5\) & 78.89 & **88.33** & 72.37 & 80.21 \\ NEF-CRF-F, \(N=5\) & 80.03 & 88.01 & 73.45 & **81.00** \\ NEF-CRF-F, \(N=10\) & 79.86 & 87.51 & 73.63 & 80.35 \\ NEF-CRF-F, \(N=L\) & **80.29** & 87.58 & **74.75** & 80.71 \\ \hline \hline \end{tabular}
\end{table}
Table 2: \(F_{1}\) scores for NER, computed by the CoNLL 2002 evaluation script.

 

sentence. We replaced the affix embeddings by the concatenation of the 64-dimensional embeddings of the target words with those of the aligned source words (we used the alignments provided in the shared task), yielding 128-dimensional representations. We used the same hyperparameters as in the POS tagging task, except the dropout probability, set to 0.1. We followed prior work Kreutzer et al. (2015) and upweighted the bad words in the loss function to make the model more pessimistic; we used a weight of 5 (tuned in the validation set).

Table 3 shows the results. We see that all our NEF-S and NEF-F models outperform the BILSTM, and that the NEF-F model with 5 sketch steps achieved the best results.7 Figure 4 illustrates the attention over the target words for 5 sketches. We observe that the attention focuses early in the areas predicted bad and moves left and right within these areas, not wasting attention on the ok part of the sentence. This block-wise focus makes sense for quality estimation, since often complete phrases are bad.

Footnote 7: Our best system would rank third in the shared task, out of 13 submissions. The winner system, which achieved 49.52\(F_{1}\)-mult, was considerably more complex than ours, using an ensemble of three neural networks with a linear system Martins et al. (2016).

### Ablation Study

To better understand our proposed model, we carried out an ablation study for NER on the English dataset. The following alternate configurations were tried and compared against the NEF-CRF-F model with csoftmax attention and 5 sketch steps:

* A NEF-CRF-F model for which the final concatenation in Eq. 6 was removed, being replaced by \(\mathbf{p}_{i}=\mathsf{softmax}(\mathsf{affine}(\mathbf{s}^{N}_{i}))\). The goal was to see if the sketches retain enough information about the input to make a final prediction without requiring the states \(\mathbf{h}_{i}\).
* A model for which the attention mechanism applied at each sketch step was replaced by a uniform distribution over the input words.
* A vanilla easy-first system (Alg. 1). Since this system can only focus on one word at the time (unlike the models with soft attention), we tried both \(N=5\) and \(N=L\) sketch steps.
* A left-to-right and right-to-left model, which replaces the attention mechanism by one of these two prescribed orders.

Table 4 shows the results. As expected, the neural easy-first system was the best performing one, although the difference with respect to the ablated

Figure 4: Example for word-level quality estimation. The source sentence is ‘_To open the Actions panel, from the main menu, choose Window > Actions.”_bad words are red (bold font), ok words are green.

Figure 3: Attention visualization for English NER, for 5 (top) and L (bottom) sketch steps. Words tagged as b–* are marked in light blue, those with i-* tags, in bold red, and with o-* tags, in green.

 

systems is relatively small. Removing the concatenation in Eq. 6 is harmful, which suggests that there is information about the input not retained in the sketches. The uniform attention performs surprisingly well, and so do the left-to-right and right-to-left models, but they are still about half a point behind. The vanilla easy-first system has the worst performance with \(N=5\). This is due to the fact that the vanilla model is uncapable of processing words "in parallel" in the same sketch step, a disadvantage with respect to the neural easy-first models, which have this capability due to their soft attention mechanisms (see the top image in Fig. 3).

## 6 Related Work

Vanilla easy-first decoders have been used in POS tagging Tsuruoka and Tsujii (2005); Ma et al. (2013), dependency parsing Goldberg and Elhadad (2010), and coreference resolution Stoyanov and Eisner (2012), being related to cyclic dependency networks and guided learning Toutanova et al. (2003); Shen et al. (2007). More recent works compute scores with a neural network Socher et al. (2011); Clark and Manning (2016); Kiperwasser and Goldberg (2016a), but they still operate in a discrete space to pick the easiest actions (the non-differentiable argmax in line 6 of Alg. 1). Generalizing this idea to "continuous" operations is at the very core of our paper, allowing gradients to be fully backpropagated. In a different context, building differentiable computation structures has also been addressed by Graves et al. (2014); Grefenstette et al. (2015).

An important contribution of our paper is the constrained softmax transformation. Others have proposed alternatives to softmax attention, including the sparsemax Martins and Astudillo (2016) and multi-focal attention Globerson et al. (2016). The latter computes a KL projection onto a budget polytope to focus on multiple words. Our constrained softmax also corresponds to a KL projection, but (i) it involves box constraints instead of a budget, (ii) it is normalized to 1, and (iii) we also backpropagate the gradient over the constraint variables. It also achieves sparsity (see the "raindrop" plots in Figures 2-4), and is suitable for sequentially computing attention distributions when diversity is desired (e.g. soft 1-to-1 alignments). Recently, Chorowski and Jaitly (2016) developed an heuristic with a threshold on the total attention as a "coverage criterion" (see their Eq. 11), however their heuristic is non-differentiable.

Our sketch generation step is similar in spirit to the "deep recurrent attentive writer" (DRAW, Gregor et al. (2015)) which generates images by iteratively refining sketches with a recurrent neural network (RNN). However, our goal is very different: instead of generating images, we generate vectors that lead to a final sequence tagging prediction.

Finally, the visualization provided in Figures 2-4 brings up the question how to understand and rationalize predictions by neural network systems, addressed by Lei et al. (2016). Their model, however, uses a form of stochastic attention and it does not perform any iterative refinement like ours.

## 7 Conclusions

We introduced novel fully-differentiable easy-first taggers that learn to make predictions over sequences in an order that is adapted to the task at hand. The decoder iteratively updates a sketch of the predictions by interacting with an attention mechanism. To spread attention evenly through all words, we introduced a new constrained softmax transformation, along with an algorithm to backpropagate its gradients. Our neural-easy first decoder consistently outperformed a BILSTM on a range of sequence tagging tasks.

A natural direction for future work is to go beyond sequence tagging (which we regard as a simple first step) toward other NLP structured prediction problems, such as sequence-to-sequence prediction. This requires replacing the sketch matrix in Alg. 2 by a dynamic memory structure.

## Acknowledgments

We thank the Unbabel research team and the reviewers for their comments. This work was supported by the Fundacao para a Ciencia e Tecnologia through contracts UID/EEA/50008/2013, PTDC/EEI-SII/7092/2014 (LearnBig), and CMUPERI/TIC/0046/2014 (GoLocal).

\begin{table}
\begin{tabular}{l l} \hline \hline NEF-CRF-F, \(N=5\) & **88.01** \\ \hline NEF-CRF-F w/out concat, \(N=5\) & 87.47 \\ Uniform Attention, \(N=5\) & 87.46 \\ Vanilla EF + CRF, \(N=5\) & 87.17 \\ Vanilla EF + CRF, \(N=L\) & 87.46 \\ Left-to-right + CRF, \(N=L\) & 87.57 \\ Right-to-left + CRF, \(N=L\) & 87.53 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Ablation experiments. Reported are \(F_{1}\) scores for NER in the English test set.

 



## References

* Al-Rfou et al. (2013) Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual nlp. _arXiv preprint arXiv:1307.1662_ .
* Andor et al. (2016) Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. In _Proc. of the Annual Meeting of the Association for Computational Linguistics_. pages 2442-2452.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In _International Conference on Learning Representations_.
* Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In _Advances in Neural Information Processing Systems_. pages 1171-1179.
* Blatz et al. (2004) John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2004. Confidence estimation for machine translation. In _Proc. of the International Conference on Computational Linguistics_. page 315.
* Bojar et al. (2016) Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In _Proceedings of the First Conference on Machine Translation_. pages 131-198.
* Chorowski and Jailty (2016) Jan Chorowski and Navdeep Jaitly. 2016. Towards better decoding and language model integration in sequence to sequence models. _arXiv preprint arXiv:1612.02695_ .
* Clark and Manning (2016) Kevin Clark and Christopher D Manning. 2016. Improving coreference resolution by learning entity-level distributed representations. In _Proc. of the Annual Meeting of the Association for Computational Linguistics_.
* Daume et al. (2009) H. Daume, J. Langford, and D. Marcu. 2009. Search-based structured prediction. _Machine learning_ 75(3):297-325.
* Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In _Proc. of the Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Beijing, China, pages 334-343.
* Gillick et al. (2016) Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2016. Multilingual language processing from bytes. In _Proc. of the Annual Meeting of the North-American Chapter of the Association for Computational Linguistics_.
* Globerson et al. (2016) Amir Globerson, Neverna Lazic, Soumen Chakrabarti, Amarnag Subramanya, Michael Ringgaard, and Fernando Pereira. 2016. Collective entity resolution with multi-focusal attention. In _Proc. of the Annual Meeting of the Association for Computational Linguistics_.
* Goldberg and Elhadad (2010) Y. Goldberg and M. Elhadad. 2010. An efficient algorithm for easy-first non-directional dependency parsing. In _Proc. of Annual Conference of the North American Chapter of the Association for Computational Linguistics_. pages 742-750.
* Graves et al. (2005) Alex Graves, Santiago Fernandez, and Jurgen Schmidhuber. 2005. Bidirectional lstm networks for improved phoneme classification and recognition. In _International Conference on Artificial Neural Networks_. Springer, pages 799-804.
* Graves et al. (2014) Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural Turing Machines. _arXiv preprint arXiv:1410.5401_ .
* Grefenstette et al. (2015) Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. 2015. Learning to Transduce with Unbounded Memory. In _Advances in Neural Information Processing Systems_. pages 1819-1827.
* Gregor et al. (2015) Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. 2015. Draw: A recurrent neural network for image generation. In _Proc. of the International Conference on Machine Learning_. pages 1462-1471.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_ 9(8):1735-1780.
* Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. _arXiv preprint arXiv:1508.01991_ .
* Kiperwasser and Goldberg (2016a) Eliyahu Kiperwasser and Yoav Goldberg. 2016a. Easy-first dependency parsing with hierarchical tree lstms. _arXiv preprint arXiv:1603.00375_ .
* Kiperwasser and Goldberg (2016b) Eliyahu Kiperwasser and Yoav Goldberg. 2016b. Simple and accurate dependency parsing using bidirectional lstm feature representations. _arXiv preprint arXiv:1603.04351_ .
* Kreutzer et al. (2015) Julia Kreutzer, Shigehiko Schamoni, and Stefan Riezler. 2015. Quality estimation from scratch (quetch): Deep learning for word-level translation quality estimation. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_. pages 316-322

Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In _Proc. of the Annual Meeting of the North-American Chapter of the Association for Computational Linguistics_.
* Lei et al. (2016) Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In _Proc. of Empirical Methods for Natural Language Processing_.
* Ma et al. (2013) Ji Ma, Tong Xiao, and Nan Yang. 2013. Easy-first pos tagging and dependency parsing with beam search. In _In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers_.
* Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In _Proc. of the Annual Meeting of the Association for Computational Linguistics_.
* Martins et al. (2013) Andre F. T Martins, Miguel B. Almeida, and Noah A. Smith. 2013. Turning on the turbo: Fast third-order non-projective turbo parsers. In _Proc. of the Annual Meeting of the Association for Computational Linguistics_.
* Martins and Astudillo (2016) Andre F. T. Martins and Ramon Astudillo. 2016. From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification. In _Proc. of the International Conference on Machine Learning_.
* Martins et al. (2016) Andre F. T. Martins, Ramon Astudillo, Chris Hokamp, and Fabio Kepler. 2016. Unbabel's participation in the wmt16 word-level translation quality estimation shared task. In _Proceedings of the First Conference on Machine Translation_.
* Neubig et al. (2017) Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017. Dynet: The dynamic neural network toolkit. _arXiv preprint arXiv:1701.03980_ .
* Nivre et al. (2014) Joakim Nivre, Zeljko Agic, Lars Ahrenberg, Maria Jesus Aranzabe, Masayuki Asahara, Aitziber Atutxa, Miguel Ballesteros, John Bauer, Kepa Bengoetxea, Yevgeni Berzak, Riyaz Ahmad Bhat, Eckhard Bick, Carl Borstell, Cristina Bosco, Gosse Bouma, Sam Bowman, Gulsen Cebirolu Eryitt, Giuseppe G. A. Celano, Fabricio Chalup, Car Coltekin, Miriam Connor, Elizabeth Davidson, Marie-Catherine de Marneffe, Arantza Diaz de Ilarraza, Kaja Dobrovoljc, Timothy Dozat, Kira Droganova, Puneet Dwivedi, Mariaba Eli, Tomaz Erjavec, Richard Farkas, Jennifer Foster, Claudia Freitas, Katarina Galdoscova, Daniel Galbraith, Marcos Garcia, Moa Gardenfors, Sebastian Garza, Filip Ginter, Iakes Goenaga, Koldo Gionola, Memduh Gokrmak, Yoav Goldberg, Xavier Gomez Guinovart, Berta Gonzales Saavedra, Matias Grioni, Normunds Gruzitis, Bruno Guillaume, Jan Hajic, Linh Ha M, Dag Haug, Barbara Hladka, Radu Ion, Elena Irima, Anders Johannsen, Fredrik Jorgensen, Huner Kaskara, Hiroshi Kanayama, Jenna Kaneva, Boris Katz, Jessica Kenney, Natalia Kostyba, Siemon Krek, Veronis Laippala, Lucia Lam, Phng Le Hong, Alessandro Lenci, Nikola Ljubesic, Olga Lyashevskaya, Teresa Lynn, Aibek Makazhanov, Christopher Manning, Catalina Maranduc, David Marecek, Hector Martinez Alonso, Andre Martins, Jan Masek, Yuji Matsumoto, Ryan McDonald, Anna Missila, Vergniac Mititelu, Yusuke Miyao, Simonetta Montemagnari, Keiko Sophie Mori, Shunsuke Mori, Bohdan Mosalevskyi, Kadri Muischchek, Nina Mustafina, Kaili Muurissep, Ling Nguyen Th, Huyn Nguyn Th Minh, Vitaly Nikolaev, Hanana Nurmi, Petya Osenova, Robert Ostling, Lilja Ovrelid, Valeria Paiva, Elena Pascual, Marco Passarotti, Cenel-Augusto Perez, Slav Petrov, Jussi Pitulainen, Barbara Plank, Martin Popel, Laura Pretkalnia, Prokopis Prokopidis, Tiina Puolakainen, Sampo Pyysalo, Alexandre Rademaker, Loganathan Ramasamy, Livy Real, Laura Rituma, Rudolf Rosa, Shadi Saleh, Baiba Saulite, Sebastian Schuster, Wolfgang Seeker, Moigaur Seraji, Lena Shakurova, Mo Shen, Natalia Silveira, Maria Simii, Radu Simionescu, Katalin Simko, Maria Simkova, Kiril Simov, Aaron Smith, Carolyn Spadine, Alane Suhr, Umut Suluback, Zsolt Szanto, Takaaki Tanaka, Reut Tsarfaty, Francis Tyvers, Sumire Uernatsu, Larraitz Uria, Gerjan van Noord, Viktor Varga, Veronika Vincze, Lars Wallin, Jing Xian Wang, Jonathan North Washington, Mats Wiren, Zdonek Zabokrtsky, Amir Zeldes, Daniel Zeman, and Hanzhi Zhu. 2016. Universal dependencies 1.4. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles University in Prague. [http://hal.handle.net/11234/1-1827](http://hal.handle.net/11234/1-1827).
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global Vectors for Word Representation. _Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014)_ 12:1532-1543.
* Plank et al. (2016) Barbara Plank, Anders Sogaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In _Proc. of the Annual Meeting of the Association for Computational Linguistics (short papers)_.
* Ross et al. (2011) Stephane Ross, Geoffrey J Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In _AISTATS_. volume 1, page 6.
* Sang (2002) E.F.T.K. Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In _Proc. of International Conference on Natural Language Learning_ 

Libin Shen, Giorgio Satta, and Aravind Joshi. 2007. Guided learning for bidirectional sequence classification. In _ACL_. volume 7, pages 760-767.
* Socher et al. (2011) Richard Socher, Cliff C Lin, Chris Manning, and Andrew Y Ng. 2011. Parsing natural scenes and natural language with recursive neural networks. In _Proceedings of the 28th international conference on machine learning (ICML-11)_. pages 129-136.
* a translation quality estimation framework. In _Proc. of the Annual Meeting of the Association for Computational Linguistics: System Demonstrations_. pages 79-84. [http://www.aclweb.org/anthology/P13-4014](http://www.aclweb.org/anthology/P13-4014).
* Stoyanov and Eisner (2012) Veselin Stoyanov and Jason Eisner. 2012. Easy-first coreference resolution. In _COLING_. pages 2519-2534.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in Neural Information Processing Systems_. pages 3104-3112.
* Toutanova et al. (2003) Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In _Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1_. Association for Computational Linguistics, pages 173-180.
* Tsuruoka and Tsujii (2005) Yoshimasa Tsuruoka and Jun'ichi Tsujii. 2005. Bidirectional inference with the easiest-first strategy for tagging sequence data. In _Proceedings of the conference on human language technology and empirical methods in natural language processing_. Association for Computational Linguistics, pages 467-474.
* Wainwright and Jordan (2008) M. Wainwright and M. Jordan. 2008. _Graphical Models, Exponential Families, and Variational Inference_. Now Publishers.

 

Proof of Proposition 1

We provide here a detailed proof of Proposition 1.

### Forward Propagation

The optimization problem is

\[{\sf csoftmax}(\mathbf{z},\mathbf{u})=\text{argmin} -H(\mathbf{\alpha})-\mathbf{z}^{\top}\mathbf{\alpha}\] s.t. \[\left\{\begin{array}{l}\mathbf{1}^{\top}\mathbf{\alpha}=1\\ \mathbf{0}\leq\mathbf{\alpha}\leq\mathbf{u}.\end{array}\right.\]

The Lagrangian function is:

\[\mathcal{L}(\mathbf{\alpha},\lambda,\mathbf{\mu},\mathbf{\nu}) = -H(\mathbf{\alpha})-\mathbf{z}^{\top}\mathbf{\alpha}+\lambda(\mathbf{1}^{\top }\mathbf{\alpha}-1) \tag{14}\] \[-\mathbf{\mu}^{\top}\mathbf{\alpha}+\mathbf{\nu}^{\top}(\mathbf{\alpha}-\mathbf{u}).\]

To obtain the solution, we invoke the Karush-Kuhn-Tucker conditions. From the stationarity condition, we have \(\mathbf{0}=\mathsf{log}(\mathbf{\alpha})+\mathbf{1}-\mathbf{z}+\lambda\mathbf{1}-\bm {\mu}+\mathbf{\nu}\), which due to the primal feasibility condition implies that the solution is of the form:

\[\mathbf{\alpha}=\text{exp}(\mathbf{z}+\mathbf{\mu}-\mathbf{\nu})/Z, \tag{15}\]

where \(Z\) is a normalization constant. From the complementarity slackness condition, we have that \(0<\alpha_{i}<u_{i}\) implies that \(\mu_{i}=\nu_{i}=0\) and therefore \(\alpha_{i}=\text{exp}(z_{i})/Z\). On the other hand, \(\nu_{i}>0\) implies \(\alpha_{i}=u_{i}\). Hence the solution can be written as \(\alpha_{i}=\text{min}\{\text{exp}(z_{i})/Z,u_{i}\}\), where \(Z\) is determined such that the distribution normalizes:

\[Z=\frac{\sum_{i\in\mathcal{A}}\text{exp}(z_{i})}{1-\sum_{i\notin\mathcal{A}} u_{i}}, \tag{16}\]

with \(\mathcal{A}=\{i\in[L]~{}|~{}\alpha_{i}<u_{i}\}\).

### Gradient Backpropagation

We now turn to the problem of backpropagating the gradients through the constrained softmax transformation. For that, we need to compute its Jacobian matrix, i.e., the derivatives \(\frac{\partial\alpha_{i}}{\partial z_{j}}\) and \(\frac{\partial\alpha_{i}}{\partial u_{j}}\) for \(i,j\in[L]\). Let us first express \(\mathbf{\alpha}\) as

\[\alpha_{i}=\left\{\begin{array}{ll}\frac{\text{exp}(z_{i})(1-s)}{\sum_{j\in \mathcal{A}}\text{exp}(z_{j})},&i\in\mathcal{A}\\ u_{i},&i\notin\mathcal{A},\end{array}\right. \tag{17}\]

where \(s=\sum_{j\notin\mathcal{A}}u_{j}\). Note that we have \(\partial s/\partial z_{j}=0,~{}\forall j\), and \(\partial s/\partial u_{j}=\mathds{1}(j\notin\mathcal{A})\). To compute the entries of the Jacobian matrix, we need to consider several cases.

Case 1:In this case, the evaluation of Eq. 17 goes through the first branch. Let us first compute the derivative with respect to \(u_{j}\). Two things can happen: if \(j\in\mathcal{A}\), then \(s\) does not depend on \(u_{j}\), hence \(\frac{\partial\alpha_{i}}{\partial u_{j}}=0\). Else, if \(j\notin\mathcal{A}\), we have

\[\frac{\partial\alpha_{i}}{\partial u_{j}}=\frac{-\text{exp}(z_{i})\frac{ \partial s}{\partial u_{j}}}{\sum_{k\in\mathcal{A}}\text{exp}(z_{k})}=-\alpha _{i}/(1-s).\]

Now let us compute the derivative with respect to \(z_{j}\). Three things can happen: if \(j\in\mathcal{A}\) and \(i\neq j\), we have

\[\frac{\partial\alpha_{i}}{\partial z_{j}} = \frac{-\text{exp}(z_{i})\text{exp}(z_{j})(1-s)}{\big{(}\sum_{k \in\mathcal{A}}\text{exp}(z_{k})\big{)}^{2}}\] (18) \[= -\alpha_{i}\alpha_{j}/(1-s).\] 

If \(j\in{\cal A}\) and \(i=j\), we have

\[\frac{\partial\alpha_{i}}{\partial z_{i}} = (1-s)\times \tag{19}\] \[= \alpha_{i}-\alpha_{i}^{2}/(1-s).\]

Finally, if \(j\notin{\cal A}\), we have \(\frac{\partial\alpha_{i}}{\partial z_{j}}=0\).

**Case 2:**\(i\notin{\cal A}\). In this case, the evaluation of Eq. 17 goes through the second branch, which means that \(\frac{\partial\alpha_{i}}{\partial z_{j}}=0\), always. Let us now compute the derivative with respect to \(u_{j}\). This derivative is always zero unless \(i=j\), in which case \(\frac{\partial\alpha_{i}}{\partial u_{j}}=1\).

To sum up, we have:

\[\frac{\partial\alpha_{i}}{\partial z_{j}}=\left\{\begin{array}{ll}\mathds{1 }(i=j)\alpha_{i}-\frac{\alpha_{i}\alpha_{j}}{1-s},&\mbox{if $i,j\in{\cal A}$}\\ 0,&\mbox{otherwise,}\end{array}\right. \tag{20}\]

and

\[\frac{\partial\alpha_{i}}{\partial u_{j}}=\left\{\begin{array}{ll}-\frac{ \alpha_{i}}{1-s},&\mbox{if $i\in{\cal A},j\notin{\cal A}$}\\ 1,&\mbox{if $i,j\notin{\cal A},i=j$}\\ 0,&\mbox{otherwise.}\end{array}\right. \tag{21}\]

Therefore, we obtain:

\[{\rm d}z_{j} = \sum_{i}\frac{\partial\alpha_{i}}{\partial z_{j}}{\rm d}\alpha_{ i} \tag{22}\] \[= \mathds{1}(j\in{\cal A})\left(\alpha_{j}{\rm d}\alpha_{j}-\frac{ \alpha_{j}\sum_{i\in{\cal A}}\alpha_{i}{\rm d}\alpha_{i}}{1-s}\right)\] \[= \mathds{1}(j\in{\cal A})\alpha_{j}({\rm d}\alpha_{j}-m),\]

and

\[{\rm d}u_{j} = \sum_{i}\frac{\partial\alpha_{i}}{\partial u_{j}}{\rm d}\alpha_{ i} \tag{23}\] \[= \mathds{1}(j\notin{\cal A})\left({\rm d}\alpha_{j}-\frac{\sum_{i \in{\cal A}}\alpha_{i}{\rm d}\alpha_{i}}{1-s}\right)\] \[= \mathds{1}(j\notin{\cal A})({\rm d}\alpha_{j}-m),\]

where \(m=\frac{\sum_{i\in{\cal A}}\alpha_{i}{\rm d}\alpha_{i}}{1-s}\).