Venue: NAACL 2016

# Expected F-Measure Training for Shift-Reduce Parsing with Recurrent Neural Networks

Wenduan Xu\({}^{\dagger}\) Michael Auli\({}^{\dagger}\) Stephen Clark\({}^{\dagger}\)

\({}^{\dagger}\)Computer Laboratory, Cambridge University

\({}^{1}\)Facebook AI Research

wx217@cam.ac.uk, michaelauli@fb.com, sc609@cam.ac.uk

###### Abstract

We present expected F-measure training for shift-reduce parsing with RNNs, which enables the learning of a global parsing model optimized for sentence-level F1. We apply the model to ccg parsing, where it improves over a strong greedy RNN baseline, by \(1.47\%\) F1, yielding state-of-the-art results for shift-reduce ccg parsing.

## 1 Introduction

Shift-reduce parsing is a popular parsing paradigm, one reason being the potential for fast parsers based on the linear number of parsing actions needed to analyze a sentence Nivre and Scholz (2004); Sagae and Lavie (2006); Zhang and Clark (2011); Goldberg et al. (2013); Zhu et al. (2013); Xu et al. (2014). Recent work has shown that by combining distributed representations and neural network models Chen and Manning (2014), accurate and efficient shift-reduce parsing models can be obtained with little feature engineering, largely alleviating the feature sparsity problem of linear models.

In practice, the most common objective for optimizing neural network shift-reduce parsing models is maximum likelihood. In the greedy search setting, the log-likelihood of each target action is maximized during training, and the most likely action is committed to at each step of the parsing process during inference Chen and Manning (2014); Dyer et al. (2015). In the beam search setting, Zhou et al. (2015) show that sentence-level likelihood, together with contrastive learning Hinton (2002), can be used to derive a global model which incorporates beam search at both training and inference time Zhang and Clark (2008), giving significant accuracy gains over a fully greedy model. However, despite the effectiveness of optimizing likelihood, it is often desirable to directly optimize for task-specific metrics, which often leads to higher accuracies for a variety of models and applications Goodman (1996); Och (2003); Smith and Eisner (2006); Rosti et al. (2010); Auli and Lopez (2011); He and Deng (2012); Auli et al. (2014); Auli and Gao (2014); Gao et al. (2014).

In this paper, we present a global neural network parsing model, optimized for a task-specific loss based on expected F-measure. The model naturally incorporates beam search during training, and is globally optimized, to learn shift-reduce action _sequences_ that lead to parses with high expected F-scores. In contrast to Auli and Lopez (2011), who optimize a ccg parser for F-measure via softmax-margin Gimpel and Smith (2010), we directly optimize an expected F-measure objective, derivable from only a set of shift-reduce action sequences and sentence-level F-scores. More generally, our method can be seen as an alternative approach for training a neural beam search parsing model Watanabe and Sumita (2015); Weiss et al. (2015); Zhou et al. (2015), combining the benefits of global learning and task-specific optimization.

We also introduce a simple recurrent neural network (RNN) model to shift-reduce parsing on which the greedy baseline and the global model is based. Compared with feed-forward networks, RNNs have the potential to capture and use an _unbounded_ history, and they have been used to learn explicit representations for parser states as well as actions

performed on the stack and queue in shift-reduce parsers (Dyer et al., 2015; Watanabe and Sumita, 2015), following Miikkulainen (1996) and Mayberry and Miikkulainen (1999). In comparison, our model is a natural extension of the feed-forward architecture in Chen and Manning (2014) using Elman RNNs (Elman, 1990).

We apply our models to ccg, and evaluate the resulting parsers on standard CCGBank data Hockenmaier and Steedman, 2007). More specifically, by combining the global RNN parsing model with a bidirectional RNN ccg supertagger that we have developed (SS4) -- building on the supertagger of Xu et al. (2015), we obtain accuracies higher than the shift-reduce ccg parsers of Zhang and Clark (2011) and Xu et al. (2014). Finally, although we choose to focus on shift-reduce parsing for ccg, we expect the methods to generalize to other shift-reduce parsers.

## 2 RNN Models

In this section, we start by describing the baseline model, which is also taken as the pretrained model to train the global model (SS2.4). We abstract away from the details of ccg and present the models in a canonical shift-reduce parsing framework (Aho and Ullman, 1972), which is henceforth assumed: partially constructed derivations are maintained on a _stack_, and a _queue_ stores remaining words from the input string; the initial parse item has an empty stack and no input has been consumed on the queue. Parsing proceeds by applying a sequence of shift-reduce actions to transform the input until the queue has been exhausted and no more actions can be applied.

### Model

Our recurrent neural network model is a standard Elman network (Elman, 1990) which is factored into an input layer, a hidden layer with recurrent connections, and an output layer. Similar to Chen and Manning (2014), the input layer \(x_{t}\) encodes stack and queue contexts of a parse item through concatenation of feature embeddings. The output layer \(y_{t}\) represents a probability distribution over possible parser actions for the current item.

The current state of the hidden layer is determined by the current input and the previous hidden layer state. The weights between the layers are represented by a number of matrices: matrix \(\mathbf{U}\) contains weights between the input and hidden layers, \(\mathbf{V}\) contains weights between the hidden and output layers, and \(\mathbf{W}\) contains weights between the previous hidden layer and the current hidden layer.

The hidden and output layers at time step \(t\) are computed via a series of vector-matrix products and non-linearities:

\[h_{t}=f(x_{t}\mathbf{U}+h_{t-1}\mathbf{W}),\] \[y_{t}=g(h_{t}\mathbf{V}),\]

where

\[f(z)=\frac{1}{1+e^{-z}},~{}~{}~{}g(z_{m})=\frac{e^{z_{m}}}{\sum_{ k}e^{z_{k}}}\]

are sigmoid1 and softmax functions, respectively.

Footnote 1: We also experimented with \(\tanh\), and found no difference in resulting performance.

### Feature Embeddings

Given a parse item, we first extract features using a set of predefined feature templates; each template belongs to a feature type \(f\) (such as word or pos tag), which has an associated look-up table, denoted as \(L_{f}\), to project a feature to its distributed representation; and \(L_{f}\in\mathbb{R}^{n_{f}\times d_{f}}\), where \(n_{f}\) is the vocabulary size of feature type \(f\) and \(d_{f}\) is its embedding dimension. The embedding for a concrete feature is obtained by retrieving the corresponding row from \(L_{f}\). At time step \(t\), the input layer \(x_{t}\) is:

\[x_{t}=[e_{f_{1,1}};\ldots;e_{f_{1,|f_{1}|}};\ldots;e_{f_{k,1}}; \ldots;e_{f_{k,|f_{k}|}}],\]

where "\(;\)" denotes concatenation, \(|f_{k}|\) is the number of feature templates for the \(k^{\text{th}}\) feature type and \(x_{t}\in\mathbb{R}^{1\times(d_{I_{1}}|f_{1}|+\cdots+d_{I_{k}}|f_{k}|)}\). For each feature type, a special embedding is used for unknown features.

### Greedy Training

To train a greedy model, we extract gold-standard actions from the training data and minimize cross-entropy loss with stochastic gradient descent (SGD) using backpropagation through time (BPTT; Rumelhart et al., 1988). Similar to Chen and Manning (2014), we compute the softmax over only feasible actions at each step.

Unfortunately, although we use an RNN, which keeps a representation of previous parse items in its 

hidden state and has the potential to capture long-term dependencies, the resulting model is still fully greedy: a locally optimal action is taken at each step given the current input \(x_{t}\) and the previous hidden state \(h_{t-1}\). Therefore, once a sub-optimal action has been committed to by the parser at any step, it has no means to recover and has to continue from that mistake. Such mistakes accumulate until the goal is reached, and they are referred to as _search errors_.

In order to enlarge the search space of the greedy model thereby alleviating some search errors, we experiment with applying beam search decoding during inference; and we observe some accuracy improvements by taking the highest scored action sequence as the output (Table 3). However, since the greedy model itself is only optimized locally, as expected, the improvements diminish after a certain beam size. Instead, we show below that by using the greedy model weights as a starting point, we can train a global model optimized for an expected F-measure loss, which gives further significant accuracy improvements (SS5).

### Expected F1 Training

The RNN we use to train the global model has the same Elman architecture as the greedy model. Given the greedy model, we summarize its weights as \(\theta=\{\mathbf{U},\mathbf{V},\mathbf{W}\}\) and initialize the weights of the global model to \(\theta\), and training proceeds as follows:

1. We use a beam-search decoder to parse a sentence \(x_{n}\) in the training data and let the decoder generate a _k_-best list2 of output parses using the _current_\(\theta\), denoted as \(\Lambda(x_{n})\). Similar to other structured training approaches that use inexact beam search Zhang and Clark (2008); Weiss et al. (2015); Watanabe and Sumita (2015); Zhou et al. (2015), \(\Lambda(x_{n})\) is as an approximation to the set of all possible parses of an input sentence. Footnote 2: We do not put a limit on \(k\), and whenever an item is finished, it is appended to the _k_-best list. We found the size of the _k_-best lists were on average twice the size of a given beam size.
2. Let \(y_{i}\) be the shift-reduce action sequence of a parse in the _k_-best list \(\Lambda(x_{n})\), and let \(|y_{i}|\) be its total number of actions and \(y_{ij}\) be the \(j^{\text{th}}\) action in \(y_{i}\), for \(1\leq j\leq|y_{i}|\). We compute the log-linear action sequence score of \(y_{i}\), \(\rho(y_{i})\), as a sum of individual action scores in that sequence: \(\rho(y_{i})=\sum_{j=1}^{|y_{i}|}\log s_{\theta}(y_{ij})\), where \(s_{\theta}(y_{ij})\) is the softmax action score of \(y_{ij}\) given by the RNN model. For each \(y_{i}\), we also compute its sentence-level F1 using the set of labeled, directed dependencies, denoted as \(\Delta\), associated with its parse item. (We assume F1 over labeled, directed dependencies is also the parser evaluation metric.)
3. We compute the negative expected F1 objective (-xF1, defined below) for \(x_{n}\) using the scores obtained in the above step and minimize this objective using SGD (maximizing the expected F1 for \(x_{n}\)). These three steps repeat for other sentences in the training data, updating \(\theta\) after processing each sentence, and training iterates in epochs until convergence.

We note that the above process is different from parse reranking Collins (2000); Charniak and Johnson (2005), in which \(\Lambda(x_{n})\) would stay the same for each \(x_{n}\) in the training data across all epochs, and a reranker is trained on all fixed \(\Lambda(x_{n})\); whereas the xF1 training procedure is on-line learning with parameters updated after processing each sentence and each \(\Lambda(x_{n})\) is generated with a new \(\theta\).

More formally, we define the loss \(J(\theta)\), which incorporates all action scores in each action sequence, and all action sequences in \(\Lambda(x_{n})\), for each \(x_{n}\) as

\[J(\theta) =-\text{xF1}(\theta)\] \[=-\sum_{y_{i}\in\Lambda(x_{n})}p(y_{i}|\theta)\text{F1}(\Delta_{ y_{i}},\Delta_{x_{n}}^{G}), \tag{1}\]

where \(\text{F1}(\Delta_{y_{i}},\Delta_{x_{n}}^{G})\) is the sentence level F1 of the parse derived by \(y_{i}\), with respect to the gold-standard dependency structure \(\Delta_{x_{n}}^{G}\) of \(x_{n}\); \(p(y_{i}|\theta)\) is the normalized probability score of the action sequence \(y_{i}\), computed as

\[p(y_{i}|\theta)=\frac{\text{exp}\{\rho(y_{i})\}}{\sum_{y\in\Lambda(x_{n})} \text{exp}\{\rho(y)\}}. \tag{2}\]

To apply SGD, we derive the error gradients used for backpropagation. First, by applying the chain 

rule to \(J(\theta)\), we have

\[\frac{\partial J(\theta)}{\partial\theta}=-\sum_{y_{i}\in\Lambda(x_{n })}\sum_{y_{ij}\in y_{i}}\frac{\partial J(\theta)}{\partial s_{\theta}(y_{ij})} \frac{\partial s_{\theta}(y_{ij})}{\partial\theta}\] \[=-\sum_{y_{i}\in\Lambda(x_{n})}\sum_{y_{ij}\in y_{i}}\delta_{y_{ ij}}\frac{\partial s_{\theta}(y_{ij})}{\partial\theta},\]

where \(\frac{\partial s_{\theta}(y_{ij})}{\partial\theta}\) is the standard softmax gradients. Next, to compute \(\delta_{y_{ij}}\), which are the error gradients propagated from the loss to the softmax layer, we rewrite the loss in (1) as

\[J(\theta) =-\text{xF1}=-\frac{G(\theta)}{Z(\theta)} \tag{3}\] \[=-\frac{\sum_{y_{i}\in\Lambda(x_{n})}\text{exp}\{\rho(y_{i})\} \text{F1}(\Delta_{y_{i}},\Delta^{G}_{x_{n}})}{\sum_{y_{i}\in\Lambda(x_{n})} \text{exp}\{\rho(y_{i})\}},\]

and by simplifying:

\[\frac{\partial G(\theta)}{\partial s_{\theta}(y_{ij})}=\frac{1}{s_{\theta}(y _{ij})}\text{exp}\{\rho(y_{i})\}\text{F1}(\Delta_{y_{i}},\Delta^{G}_{x_{n}}),\]

\[\frac{\partial Z(\theta)}{\partial s_{\theta}(y_{ij})}=\frac{1}{s_{\theta}(y _{ij})}\text{exp}\{\rho(y_{i})\},\]

since

\[\frac{\partial\rho(y_{i})}{\partial s_{\theta}(y_{ij})}=\frac{1}{s_{\theta}(y _{ij})}.\]

Finally, using (2) and (3) plus the above simplifications, the error term \(\delta_{y_{ij}}\) can be derived using the quotient rule:

\[\begin{split}\delta_{y_{ij}}&=-\frac{\partial\text {xF1}(\theta)}{\partial s_{\theta}(y_{ij})}\\ &=-\frac{\partial(G(\theta)/Z(\theta))}{\partial s_{\theta}(y_{ ij})}\\ &=\frac{G(\theta)Z^{\prime}(\theta)-G^{\prime}(\theta)Z(\theta)}{Z^{2}( \theta)}\\ &=\frac{\text{exp}\{\rho(y_{i})\}}{Z(\theta)}(\text{xF1}(\theta) -\text{FI}(\Delta_{y_{i}},\Delta^{G}_{x_{n}}))\frac{1}{s_{\theta}(y_{ij})}\\ &=p(y_{i}|\theta)(\text{xF1}(\theta)-\text{FI}(\Delta_{y_{i}}, \Delta^{G}_{x_{n}}))\frac{1}{s_{\theta}(y_{ij})},\end{split} \tag{4}\]

which has a simple closed form.

A naive implementation of the xF1 training procedure would backpropagate the error gradients individually for each \(y_{i}\) in \(\Lambda(x_{n})\). To make it efficient, we observe that the unfolded network in the beam containing all \(y_{i}\) becomes a DAG (with one hidden state leading to one or more resulting hidden states) and apply backpropagation through structure [69] to obtain the gradients.

## 3 Shift-Reduce CCG Parsing

We explain the application of the RNN models to ccg by first describing the CCG mechanisms used in our parser, followed by details of the shift-reduce transition system.

### Combinatory Categorial Grammar

A _lexicon_, together with a set of ccg_rules_, formally constitute a ccg. The former defines a mapping from words to sets of lexical categories representing syntactic types, and the latter gives schemas which dictate whether two categories can be combined. Given the lexicon and the rules, the syntactic types of complete constituents can be obtained by recursive combination of categories using the rules.

More generally, both lexical and non-lexical ccg categories can be either _atomic_ or _complex_: atomic categories are categories without any slashes, and complex categories are constructed recursively from atomic ones using forward \((/)\) and backward slashes \((\backslash)\) as two binary operators. As such, all categories can be represented as follows [50; 87]:

\[x\coloneqq\alpha|_{1}z_{1}|_{2}z_{2}\ldots|_{m}z_{m},\]

where \(m\geq 0\), \(\alpha\) is an atomic category, \(|_{1},\dots,|_{m}\in\{\backslash,/\}\) and \(z_{i}\) are meta-variables for categories.

ccg_rules_ have the following two schematic forms, each a generalized version of _functional composition_[50]:

\[\begin{array}{l}x/y\quad y|_{1}z_{1}\ldots|_{m}z_{m}\to x|_{1} z_{1}\ldots|_{m}z_{m},\\ y|_{1}z_{1}\ldots|_{m}z_{m}\quad x\backslash y\to x|_{1}z_{1}\ldots|_{m}z_{ m}.\end{array}\]

The first schematic form above instantiates into a forward application rule \((>)\) for \(m=0\), and forward composition rules (\(>_{\mathbf{B}}\)) for \(m>0\). Similarly, the second schematic form, which is symmetric to the first, instantiates into backward application \((<)\) and composition (\(<_{\mathbf{B}}\)) rules.

Fig.1 shows an example ccg derivation. All the _rule instances_ in this derivation are instantiated from 

forward rules; for example, \(N/N~{}N/N\to N/N\) is an instance of forward composition and \(N/N~{}N\to N\) is an instance of forward application.

Given CCGBank Hockenmaier and Steedman (2007), there are two approaches to extract a grammar from this data. The first is to treat all ccg derivations as phrase-structure trees, and a binary, context-free "cover" grammar, consisting of all ccg rule instances in the treebank, is extracted from local trees in all the derivations Fowler and Penn (2010); Zhang and Clark (2011). In contrast, one can extract the lexicon from the treebank and define only the rule schemas, without explicitly enumerating any rule instances Hockenmaier (2003). This is the approach taken in the c&c parser Clark and Curran (2007) and the one we use here. Moreover, following Zhang and Clark (2011), our ccg parsing model is also a normal-form model, which models action sequences of normal-form derivations in CCGBank.

### The Transition System

The transition system we use in this work is based on the ccg transition system of Zhang and Clark (2011). We denote parse items as \((j,\delta,\beta,\Delta)^{3}\), where \(\delta\) is the stack (with top element \(\delta|s_{0}\)), \(\beta\) is the queue (with top element \(x_{w_{j}}|\beta\)), \(j\) is the positional index of the word at the front of the queue, and \(\Delta\) is the set of ccg dependencies realized for the input consumed so far (needed to calculate the expected F-score). We also assume a set of lexical categories has been assigned to each word using a superdagger Bangalore and Joshi (1999); Clark and Curran (2004). The transition system is specified using three action types:

* Shift (sh) removes one of the lexical categories \(x_{w_{j}}\) of the front word \(w_{j}\) in the queue, and pushes it onto the stack; and removes \(w_{j}\) from the queue.
* Reduce (re) combines the top two subtrees \(s_{0}\) and \(s_{1}\) on the stack using a ccg rule (\(s_{1}s_{0}\to x\)) and replaces them with a subtree rooted in \(x\). It also appends the set of newly created dependencies on \(x\), denoted as \(\langle x\rangle\), to \(\Delta\).
* Unary (un) applies either a type-raising or type-changing rule (\(s_{0}\to x\)) to the stack-top element and replaces it with a unary subtree rooted in \(x\).

The deduction system (Fig. 2) of our shift-reduce parser follows from the transition system.4 Each parse item is associated with a step indicator \(\omega\), which denotes the number of actions used to build it. Given a sentence of length \(n\), a full derivation requires \(2n-1+\mu\) steps to terminate, where \(\mu\) is the total number of un actions applied. In Zhang and Clark (2011), a _finish_ action is used to indicate termination, which we do not use in our parser: an item finishes when no further action can be taken. Another difference between the transition systems is that Zhang and Clark (2011) omit the \(\Delta\) field in each parse item, due to their use of a context-free, phrase-structure cover, and dependencies are recovered at a post-processing step; in our system, we build dependencies as parsing proceeds.

Figure 1: An example ccg derivation.

Figure 2: The shift-reduce deduction system.

 



### RNN ccg Parsing

We use the same set of ccg rules as in Clark and Curran (2007) and the total number of output units in our RNN model is equal to the number of lexical categories (i.e., all possible sh actions), plus 10 units for re5 and 18 units for un actions.

Footnote 5: In principle, only 1 re unit is needed, but we use 9 additional units to handle non-standard ccg rules in the treebank.

All features in our model fall into three types: word, pos tag and ccg category. Table 1 shows the atomic feature templates and we have \(|f_{w}|=16\), \(|f_{p}|=16\) and \(|f_{c}|=8\) (all word-based features are generalized to pos features). Each template has two parts: the first part denotes parse item context and the second part denotes the feature type. \(s\) denotes stack contexts and \(q\) denotes queue contexts; e.g., \(s_{0}\) is the top subtree on the stack, and \(s_{o}\).l is its left child. W represents head words of constituents and w\({}_{0}\) is the right-most word of the input string that has been shifted onto the stack.

## 4 Bidirectional Supertagging

We extend the RNN supertagging model of Xu et al. (2015) by using a bidirectional RNN (BRNN). The BRNN processes an input in both directions with two separate hidden layers, which are then fed to one output layer to make predictions. At each time step \(t\), we compute the _forward_ hidden state \(h_{t}\) for \(t=(0,1,\ldots,n-1)\); the _backward_ hidden state \(h^{\prime}_{t}\) is computed similarly but from the reverse direction for \(t=(n-1,n-2,\ldots,0)\) as

\[h^{\prime}_{t}=f(x_{t}\mathbf{U}^{\prime}+h_{t+1}\mathbf{W}^{\prime}), \tag{5}\]

and the output layer, for \(t=(0,1,\ldots,n-1)\), is computed as

\[y_{t}=f([h_{t};h^{\prime}_{t}]\mathbf{V}^{\prime}). \tag{6}\]

The BRNN introduces two new parameter matrices \(\mathbf{U}^{\prime}\) and \(\mathbf{W}^{\prime}\) and replaces the old hidden-to-output matrix \(\mathbf{V}\) with \(\mathbf{V}^{\prime}\) to take two hidden layers as input. We use the same three feature embedding types as Xu et al. (2015), namely word, suffix and capitalization, and all features are extracted from a context window size of 7 surrounding the current word.

## 5 Experiments

Setup.All experiments were performed on CCGBank (Hockenmaier and Steedman, 2007) with the standard split.6 We used the c&c superdagger (Clark and Curran, 2007) and the RNN superstagger model of Xu et al. (2015) as two supertagger baselines. For the parsing experiments, the baselines were the shift-reduce ccg parsers of Zhang and Clark (2011) and Xu et al. (2014) and the c&c parser of (Clark and Curran, 2007).

Footnote 6: Training: Sections 02-21; development: Section 00; test Section 23.

To train the RNN parser, we used 10-fold cross validation for both pos tagging and supertagging. For both development and test parsing experiments, we used the c&c pos tagger and automatically assigned pos tags. The BRNN supertagging model was used as the supertagger by all RNN parsing models for both training and testing. F-score over directed, labeled ccg predicate-argument dependencies was used as the parser evaluation metric, obtained using the script from c&c.

Hyperparameters.For the BRNN supertagging model, we used identical hyperparameter settings as in Xu et al. (2015). For all RNN parsing models, the weights were uniformly initialized using the interval \([-2.0,2.0]\), and scaled by their fan-in (Bengio, 2012); the hidden layer size was \(220\), and 50-dimensional embeddings were used for all feature types and scaled Turian embeddings were used (Turian et al., 2010) for word embeddings. We also pretrained ccg lexci category and pos embeddings by using the gensim word2vec implementation.7 The data used for this was obtained by parsing a Wikipedia dump using the c&c parser and concatenating the output with CCGBank Sections 02-21. Embeddings for unknown words and ccg categories outside of the lexical category set were uniformly initialized (\([-2.0,2.0]\)) without scaling.

\begin{table}
\begin{tabular}{c c c c} \hline \(s_{0}\)**.W** & \(s_{1}\)**.W** & \(s_{2}\)**.W** & \(s_{3}\)**.W** \\ \(s\)**.W\({}_{0}\)** & \(s\)**.W\({}_{1}\)** & \(s\)**.W\({}_{2}\)** & \(s\)**.W\({}_{3}\)** \\ \(s_{0}\)**.l**.W** & \(s_{1}\)**.l**.W\({}_{s}\)** & \(s_{o}\)**.r**.W\({}_{s}\)** & \(s_{1}\)**.r**.W** \\ \(q_{0}\)**.W** & \(q_{1}\)**.W** & \(q_{2}\)**.W** & \(q_{3}\)**.W** \\ \hline \(s_{0}\)**.C** & \(s_{0}\)**.l**.C** & \(s_{0}\)**.f**.C** & \\ \(s_{1}\)**.C** & \(s_{1}\)**.l**.C** & \(s_{1}\)**.f**.C** & \\ \(s_{2}\)**.C** & \(s_{3}\)**.C** & & \\ \hline \end{tabular}
\end{table}
Table 1: Atomic feature templates.



To train all the models, we used a fixed learning rate of \(0.0025\) and did not truncate the gradients for BPTT, except for training the greedy RNN parsing model where we used a BPTT step size of 9. We applied dropout at the input layer Legrand and Collobert (2015), with a dropout rate of \(0.25\) for the supertagger and \(0.30\) for the parser.

### Supertagging Results

Table 2 shows 1-best supertagging results. The MaxEnt c&c supertagger uses pos tag features and a tag dictionary, neither of which are used by the RNN supertaggers. For all supertagers, the same set of 425 lexical categories is used Clark and Curran (2007). On the test set, our BRNN supertagger achieves a 1-best accuracy of \(93.52\%\), an absolute improvement of \(0.52\%\) over the RNN model, demonstrating the usefulness of contextual information from both input directions.

Fig. 3a shows multi-tagging accuracy comparison for the three supert aggersers by varying the variable-width beam probability cut-off value \(\beta\) for each supertagger. The \(\beta\) value determines the average number of supertags (ambiguity) assigned to each word by pruning supertags whose probabilities are not within \(\beta\) times the probability of the 1-best supertag; for this experiment we used \(\beta\) values ranging from \(0.09\) to \(2\times 10^{-4}\) and it can be seen that the BRNN supertagger consistently achieves better accuracies at similar ambiguity levels.

Finally, all shift-reduce ccg parsers mentioned in this paper take multi-tagging output obtained with a _fixed_\(\beta\) for training and testing; and in general, a smaller \(\beta\) value can be used by a shift-reduce ccg parser than by the c&c parser. This is because a \(\beta\) value too small may explode the dynamic program of the c&c parser, and it thus relies on an adaptive supertagging strategy Clark and Curran (2007), by starting from a large \(\beta\) value and backing off to smaller values if no spanning analysis can found with the current \(\beta\).

### Parsing Results

To pretrain the greedy model, we trained 10 cross-validated BRNN supertagging models to supply supertags for the parsing model, and used a supertagger \(\beta\) value of \(0.00025\) which gave on average \(5.02\) supertags per word. We ran SGD training for 60 epochs, observing no accuracy gains after that, and the best greedy model was obtained after the 52\({}^{\text{nd}}\) epoch (Fig. 3b).

Furthermore, we found that using a relatively smaller supertagger \(\beta\) value (higher ambiguity) for training, and a larger \(\beta\) value (lower ambiguity) for testing, resulted in more accurate models; and we chose the final \(\beta\) value used for the greedy model to be \(0.09\) using the dev set (Table 3). This observation was different from Zhang and Clark (2011) and Xu et al. (2014), which are two shift-reduce ccg parsers using the averaged perceptron and beam search Collins (2002); Collins and Roark (2004); Zhang and Clark (2008): they used the same \(\beta\) values for training and testing, which resulted in lower accuracy for our greedy model.

Table 3 also shows the effect on dev F1 by using different beam sizes at test time for the greedy model: with \(b=6\), we obtained an accuracy of \(85.02\%\), an improvement of \(0.41\%\) over \(b=1\) (with a \(\beta\) value of \(0.09\)); we saw accuracy gains up to \(b=8\) (with very minimal gains with \(b=16\) for \(\beta\) values \(0.06\) and \(0.07\)), after which the accuracy started to drop. F1 on dev with \(b=6\) across all training epochs are shown in Fig. 3b as well, and the best model was obtained after the 43\({}^{\text{rd}}\) epoch.

For the xF1 model, we used \(b=8\) and a supertagger \(\beta\) value of \(0.09\) for both training and testing. Fig.3c shows dev F1 versus the number of training epochs. The best dev F1 was obtained after the 54\({}^{\text{th}}\) epoch with an accuracy of \(85.73\%\), \(1.12\%\)

\begin{table}
\begin{tabular}{l|l l} \multicolumn{2}{l}{Supertagger} & Dev & Test \\ \hline C\&C (gold pos) & 92.60 & 93.32 \\ c\&c (auto pos) & 91.50 & 92.02 \\ RNN & 93.07 & 93.00 \\ BRNN & **93.49** & **93.52** \\ \end{tabular}
\end{table}
Table 2: 1-best supertagging accuracy comparison.

\begin{table}
\begin{tabular}{l|l l l l}  & \multicolumn{4}{c}{Supertagger \(\beta\)} \\  & 0.09 & 0.08 & 0.07 & 0.06 \\ \hline \(b=1\) & 84.61 & 84.58 & 84.55 & 84.50 \\ \(b=2\) & 84.94 & 84.86 & 84.86 & 84.81 \\ \(b=4\) & 85.01 & 84.95 & 84.92 & 84.92 \\ \(b=6\) & **85.02** & 84.96 & 84.94 & 84.93 \\ \(b=8\) & **85.02** & 84.99 & 84.96 & 84.95 \\ \(b=16\) & 85.01 & 84.95 & 84.97 & 84.98 \\ \end{tabular}
\end{table}
Table 3: The effect on dev F1 by varying the beam size and supertagger \(\beta\) value for the greedy RNN model.

 

higher than that of the greedy model with \(b=1\) and \(0.71\%\) higher than the greedy model with \(b\in\{6,8\}\). This result improves over shift-reduce ccg models of Zhang and Clark (2011) and Xu et al. (2014) by \(0.73\%\) and \(0.55\%\), respectively (Table 4).

Table 4 summarizes final results.9 RNN-xF1, the xF1 trained beam-search model, is currently the most accurate shift-reduce ccg parser, achieving a final F-score of \(86.42\%\), and gives an F-score improvement of \(1.47\%\) over the greedy RNN baseline. We show the results for the model of Xu et al. (2014) for reference only, since it uses a more sophisticated dependency, rather than normal-form derivation, model.

Footnote 9: We used \(b=8\) to do the precomputation.

At test time, we also used the precomputation trick of Devlin et al. (2014) to speed up the RNN models by caching the top 20K word embeddings and all pos embeddings,10 and this made the greedy RNN parser more than 3 times faster than the c&c parser (all speed experiments were measured on a workstation with an Intel Core i7 4.0GHz CPU).11

Footnote 10: The speed results for the c&c parser were obtained using the per-compiled c&c binary for Linux available from [http://svn.ask.it.usyd.edu.au/trac/](http://svn.ask.it.usyd.edu.au/trac/) candc/wiki/Download.

## 6 Related Work

Optimizing for Task-specific Metrics.Our training objective is largely inspired by task-specific optimization for parsing and MT. Goodman (1996) proposed algorithms for optimizing a parser for various constituent matching criteria, and it was one of the earliest work that we are aware of on optimizing a parser for evaluation metrics. Smith and Eisner (2006) proposed a framework for minimizing expected loss for log-linear models and applied it to dependency parsing by optimizing for labeled attachment scores, although they obtained little per

Figure 3: Experiment results on the dev set. (a) shows multi-tagging accuracy using the best tagging model. (b) shows F1 scores for the greedy RNN parsing models with beam size \(b\in\{1,6\}\). (c) shows F1 scores for the xF1 models with \(b=8\).

 

formance improvements. Auli and Lopez (2011) optimized the c&c parser for F-measure. However, they used the softmax-margin Gimpel and Smith (2010) objective, which required decomposing precision and recall statistics over parse forests. Instead, we directly optimize for an F-measure loss. In MT task-specific optimization has also received much attention (e.g., see Och (2003)). Closely related to our work, Gao and He (2013) proposed training a Markov random field translation model as an additional component in a log-linear phrase-based translation system using a \(k\)-best list based expected BLEU objective; using the same objective, Auli et al. (2014) and Auli and Gao (2014) trained a large scale phrase-based reordering model and a RNN language model respectively, all as additional components within a log-linear translation model. In contrast, our RNN parsing model is trained in an end-to-end fashion with an expected F-measure loss and all parameters of the model are optimized using backpropagation and SGD.

Parsing with RNNs.A line of work is devoted to parsing with RNN models, including using RNNs Miikkulainen (1996); Mayberry and Miikkulainen (1999); Legrand and Collobert (2015); Watanabe and Sumita (2015) and LSTM Hochreiter and Schmidhuber (1997) RNNs Vinyals et al. (2015); Ballesteros et al. (2015); Dyer et al. (2015); Kiperwasser and Goldberg (2016). Legrand and Collobert (2015) used RNNs to learn conditional distributions over syntactic rules; Vinyals et al. (2015) explored sequence-to-sequence learning Sutskever et al. (2014) for parsing; Ballesteros et al. (2015) utilized character-level representations and Kiperwasser and Goldberg (2016) built an easy-first dependency parser using tree-structured compositional LSTMs. However, all these parsers use greedy search and are trained using the maximum likelihood criterion except Kiperwasser and Goldberg (2016), who used a margin-based objective). For learning global models, Watanabe and Sumita (2015) used a margin-based objective, which was not optimized for the evaluation metric; although not using RNNs, Weiss et al. (2015) proposed a method using the averaged perceptron with beam search Collins (2002); Collins and Roark (2004); Zhang and Clark (2008), which required fixing the neural network representations, and thus their model parameters were not learned using end-to-end backpropagation.

Finally, a number of recent work Bengio et al. (2015); Vaswani and Sagae (2016) explored training neural network models for parsing and other tasks such that the network learns from the oracle as well as its own predictions, and are hence more robust to search errors during inference. In principle, these techniques are largely orthogonal to both global learning and task-based optimization, and we would expect further accuracy gains are possible by combining these techniques in a single model.

## 7 Conclusion

Neural network shift-reduce parsers are often trained by maximizing likelihood, which does not optimize towards the final evaluation metric. In this paper, we addressed this problem by developing expected F-measure training for an RNN shift-reduce parsing model. We have demonstrated the effectiveness of our method on shift-reduce parsing for ccg, achieving higher accuracies than all shift-reduce ccg parsers to date and the de facto c&c parser.11 We expect the general framework will be applicable to models using other types of neural networks such as feed-forward or LSTM nets, and to shift-reduce parsers for constituent and dependency parsing.

Footnote 11: Auli and Lopez (2011) present higher accuracies but on a different coverage to enable a comparison to Fowler and Penn (2010). Their results are thus not directly comparable to ours.

We thank the anonymous reviewers for their detailed comments. Xu acknowledges the Carnegie Trust for the Universities of Scotland and the Cambridge Trust's for funding. Clark is supported by ERC Starting Grant DisCoTex (306920) and EPSRC grant EP/I037512/1.

## References

* Alho and Ullman (1972) Alfred V Aho and Jeffrey D Ullman. 1972. _The theory of parsing, translation, and compiling_. Prentice-Hall.
* Auli and Gao (2014) Michael Auli and Jianfeng Gao. 2014. Decoder integration and expected BLEU training for recurrent neural network language models. In _Proc. of ACL (Volume 2)_.

 

Michael Auli and Adam Lopez. 2011. Training a log-linear parser with loss functions via softmax-margin. In _Proc. of EMNLP_.
* Auli et al. (2014) Michael Auli, Michel Galley, and Jianfeng Gao. 2014. Large-scale expected BLEU training of phrase-based reordering models. In _Proc. of EMNLP_.
* Ballesteros et al. (2015) Miguel Ballesteros, Chris Dyer, and Noah A Smith. 2015. Improved transition-based parsing by modeling characters instead of words with LSTMs. In _Proc. of EMNLP_.
* Bangalore and Joshi (1999) Srinivas Bangalore and Aravind K Joshi. 1999. Supertagging: An approach to almost parsing. In _Computational linguistics_. MIT Press.
* Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam M. Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In _Proc. of NIPS_.
* Bengio (2012) Yoshua Bengio. 2012. Practical recommendations for gradient-based training of deep architectures. In _Neural Networks: Tricks of the Trade_. Springer.
* Charniak and Johnson (2005) Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. In _Proc. of ACL_.
* Chen and Manning (2014) Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In _Proc. of EMNLP_.
* Clark and Curran (2004) Stephen Clark and James R Curran. 2004. The importance of supertagging for wide-coverage CCG parsing. In _Proc. of COLING_.
* Clark and Curran (2007) Stephen Clark and James R. Curran. 2007. Wide-coverage efficient statistical parsing with CCG and log-linear models. In _Computational Linguistics_. MIT Press.
* Collins and Roark (2004) Michael Collins and Brian Roark. 2004. Incremental parsing with the perceptron algorithm. In _Proc. of ACL_.
* Collins (2000) Michael Collins. 2000. Discriminative reranking for natural language parsing. In _Proc. of ICML_.
* Collins (2002) Michael Collins. 2002. Discriminative training methods for hidden Markov models: Theory and experiments with perceptron algorithms. In _Proc. of EMNLP_.
* Devlin et al. (2014) Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In _Proc. of ACL_.
* Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In _Proc. of ACL_.
* Elman (1990) Jeffrey L Elman. 1990. Finding structure in time. In _Cognitive science_. Elsevier.
* Fowler and Penn (2010) Timothy A.D. Fowler and Gerald Penn. 2010. Accurate context-free parsing with Combinatory Categorial Grammar. In _Proc. of ACL_.
* Gao and He (2013) Jianfeng Gao and Xiaodong He. 2013. Training MRF-based phrase translation models using gradient ascent. In _Proc. of NAACL_.
* Gao et al. (2014) Jianfeng Gao, Xiaodong He, Wen-tau Yih, and Li Deng. 2014. Learning continuous phrase representations for translation modeling. In _Proc. of ACL_.
* Gimpel and Smith (2010) Kevin Gimpel and Noah Smith. 2010. Softmax-margin CRFs: training log-linear models with cost functions. In _Proc. of NAACL_.
* Goldberg et al. (2013) Yoav Goldberg, Kai Zhao, and Liang Huang. 2013. Efficient implementation for beam search incremental parsers. In _Proc. of ACL (Volume 2)_.
* Golder and Kuchler (1996) Christoph Golfer and Andreas Kuchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. In _Proc. of IEEE International Conference on Neural Networks_.
* Goodman (1996) Joshua Goodman. 1996. Parsing algorithms and metrics. In _Proc. of ACL_.
* He and Deng (2012) Xiaodong He and Li Deng. 2012. Maximum expected BLEU training of phrase and lexicon translation models. In _Proc. of ACL_.
* Hinton (2002) Geoffrey E Hinton. 2002. Training products of experts by minimizing contrastive divergence. In _Neural computation_. MIT Press.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. In _Neural computation_. MIT Press.
* Hockenmaier and Steedman (2007) Julia Hockenmaier and Mark Steedman. 2007. CCGBank: A corpus of CCG derivations and dependency structures extracted from the Penn Treebank. In _Computational Linguistics_. MIT Press.
* Hockenmaier (2003) Julia Hockenmaier. 2003. _Data and Models for Statistical Parsing with Combinatory Categorial Grammar_. Ph.D. thesis, University of Edinburgh.
* Kiperwasser and Goldberg (2016) Eliyahu Kiperwasser and Yoav Goldberg. 2016. Easy-first dependency parsing with hierarchical tree LSTMs. _arXiv:1603.00375_.
* Kuhlmann and Satta (2014) Marco Kuhlmann and Giorgio Satta. 2014. A new parsing algorithm for combinatory categorial grammar. In _Transactions of the Association for Computational Linguistics_. ACL.
* Legrand and Collobert (2015) Joel Legrand and Ronan Collobert. 2015. Joint RNN-based greedy parsing and word composition. In _Proc. of ICLR_.
* Mayberry and Miikkulainen (1999) Marshall R. Mayberry and Risto Miikkulainen. 1999. Sardsm: A neural network shift-reduce parser. In _Proc. of IJCAI_.
* Miikkulainen (1996) Risto Miikkulainen. 1996. Subsymbolic case-role analysis of sentences with embedded clauses. In _Cognitive Science_. Elsevier.
* Mikolov (2012) Tomas Mikolov. 2012. _Statistical Language Models Based on Neural Networks_. Ph.D. thesis, Brno University of Technology.

 

J. Nivre and M Scholz. 2004. Deterministic dependency parsing of English text. In _Proc. of COLING_.
* [Nivre2008] Joakim Nivre. 2008. Algorithms for deterministic incremental dependency parsing. In _Computational Linguistics_. MIT Press.
* [Och.2003] Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In _Proc. of ACL_.
* [Rosti et al.2010] Antti-Veikko I Rosti, Bing Zhang, Spyros Matsoukas, and Richard Schwartz. 2010. BBN system description for WMT10 system combination task. In _Proc. of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR_.
* [ Rumelhart et al.1988] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1988. Learning representations by backpropagating errors. In _Nature_.
* [Sagae and Lavie2006] Kenji Sagae and Alon Lavie. 2006. A best-first probabilistic shift-reduce parser. In _Proc. of COLING/ACL_.
* [Smith and Eisner2006] David A. Smith and Jason Eisner. 2006. Minimum-risk annealing for training log-linear models. In _Proc. of COLING-ACL_.
* [Sutskever et al.2014] Ilya Sutskever, Oriol Vinyals, and Quoc Le. 2014. Sequence to sequence learning with neural networks. In _Proc. of NIPS_.
* [Turian et al.2010] Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In _Proc. of ACL_.
* [Vaswani and Sagea2016] Ashish Vaswani and Kenji Sagae. 2016. Efficient structured inference for transition-based parsing with neural networks and error states. In _Transactions of the Association for Computational Linguistics_. ACL.
* [Vijay-Shanker and Weir1993] Krishnamurti Vijay-Shanker and David J Weir. 1993. Parsing some constrained grammar formalisms. In _Computational Linguistics_. MIT Press.
* [Vinyals et al.2015] Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In _Proc. of NIPS_.
* [Watanabe and Sumita2015] Taro Watanabe and Eiichiro Sumita. 2015. Transition-based neural constituent parsing. In _Proc. of ACL_.
* [Weiss et al.2015] David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. 2015. Structured training for neural network transition-based parsing. In _Proc. of ACL_.
* [Xu et al.2014] Wenduan Xu, Stephen Clark, and Yue Zhang. 2014. Shift-Reduce CCG parsing with a dependency model. In _Proc. of ACL_.
* [Xu et al.2015] Wenduan Xu, Michael Auli, and Stephen Clark. 2015. CCG supertagging with a recurrent neural network. In _Proc. of ACL (Volume 2)_.
* [Zhang and Clark2008] Yue Zhang and Stephen Clark. 2008. A tale of two parsers: investigating and combining graph-based and transition-based dependency parsing using beamsearch. In _Proc. of EMNLP_.
* [Zhang and Clark2011] Yue Zhang and Stephen Clark. 2011. Shift-Reduce CCG parsing. In _Proc. of ACL_.
* [Zhou et al.2015] Hao Zhou, Yue Zhang, Shujian Huang, and Jiajun Chen. 2015. A neural probabilistic structured-prediction model for transition-based dependency parsing. In _Proc. of ACL_.
* [Zhu et al.2013] Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu. 2013. Fast and accurate shift-reduce constituent parsing. In _Proc. of ACL_.