Venue: NAACL 2016

# Transition-Based Syntactic Linearization with Lookahead Features

Ratish Puduppully \({}^{\dagger}\), Yue Zhang \(\ddagger\), Manish Shrivastava \(\dagger\)

+Kohli Center on Intelligent Systems (KCIS),

International Institute of Information Technology, Hyderabad (IIIT Hyderabad)

++Singapore University of Technology and Design

rathish.surendran@research.iiit.ac.in yue_zhang@sutd.edu.sg m.shrivastava@iiit.ac.in

Part of the work was done when the author was a visiting student at Singapore University of Technology and Design.

###### Abstract

It has been shown that transition-based methods can be used for syntactic word ordering and tree linearization, achieving significantly faster speed compared with traditional best-first methods. State-of-the-art transition-based models give competitive results on abstract word ordering and unlabeled tree linearization, but significantly worse results on labeled tree linearization. We demonstrate that the main cause for the performance bottleneck is the sparsity of Shift transition actions rather than heavy pruning. To address this issue, we propose a modification to the standard transition-based feature structure, which reduces feature sparsity and allows lookahead features at a small cost to decoding efficiency. Our model gives the best reported accuracies on all benchmarks, yet still being over 30 times faster compared with best-first-search.

## 1 Introduction

Word ordering is the abstract language modeling task of making a grammatical sentence by ordering a bag of words White (2004); Zhang and Clark (2015); De Gispert et al. (2014); Bohnet et al. (2010); Filippova and Strube (2007); He et al. (2009), which is practically relevant to text-to-text applications such as summarization Wann et al. (2009) and machine translation Blackwood et al. (2010). Zhang (2013) built a discriminative word ordering model, which takes a bag of words, together with optional POS and dependency arcs on a subset of input words, and yields a sentence together with its dependency parse tree that conforms to input syntactic constraints. The system is flexible with respect to input constraints, performing abstract word ordering when no constraints are given, but gives increasingly confined outputs when more POS and dependency relations are specified. It has been applied to syntactic linearization Song et al. (2014) and machine translation Zhang et al. (2014).

One limitation of Zhang (2013) is relatively low time efficiency, due to the use of time-constrained best-first-search White and Rajkumar (2009) for decoding. In practice, the system can take seconds to order a bag of words in order to obtain reasonable output quality. Recently, Liu et al. (2015) proposed a transition-based model to address this issue, which uses a sequence of state transitions to build the output. The system of Liu et al. (2015) achieves significant speed improvements without sacrificing accuracies when working with unlabeled dependency trees. With labeled dependency trees as input constraints, however, the system of Liu et al. (2015) gives much lower accuracies compared with Zhang (2013).

While the low accuracy can be attributed to heavy pruning, we show that it can be mitigated by modifying the feature structure of the standard transition-based framework, which scores the output transition sequence by summing the scores of each transition action. Transition actions are treated as an _atomic_ output component in each feature instance. This works effectively for most structured prediction tasks, including parsing Zhang and Clark (2011a). For word ordering, however, transition actions are significantly more complex and sparse compared

with parsing, which limits the power of the traditional feature model.

We instead break down complex actions into smaller components, merging some components into configuration features which reduces sparsity in the output action and allows flexible lookahead features to be defined according to the next action to be applied. On the other hand, this change in the feature structure prevents legitimate actions to be scored simultaneously for each configuration state, thereby reducing decoding efficiency. Experiments show that our method is slightly slower compared with Liu et al. (2015), but achieves significantly better accuracies. It gives the best results for all standard benchmarks, being over thirty times faster than Zhang (2013). The new feature structures can be applied to other transition-based systems also.

## 2 Transition-based linearization

Liu et al. (2015) uses a transition-based model for word ordering, building output sentences using a sequence of state transitions. Instead of scoring output syntax trees directly, it scores the transition action sequence for structural disambiguation. Liu et al.'s transition system extends from transition-based parsers (Nivre and Scholz, 2004; Chen and Manning, 2014), where a _state_ consists of a _stack_ to hold partially built outputs. Transition-based parsers use a _queue_ to maintain input word sequences. However, for word ordering, the input is a set without order. Accordingly, Liu et al. uses a _set_ to maintain the input. The transition actions are:

* Shift-_Word_-POS, which removes _Word_ from the _set_, assigns POS to it and pushes it onto the _stack_ as the top word \(S_{0}\);
* LeftArc-LABEL, which removes the second top of _stack_\(S_{1}\) and builds a dependency arc \(S_{1}\xleftarrow{LABEL}S_{0}\);
* RightArc-LABEL, which removes the top of _stack_\(S_{0}\) and builds a dependency arc \(S_{1}\xleftarrow{LABEL}S_{0}\).

Using the state transition system, the bag of words {_John, loves, Mary_} can be ordered by (Shift-_John_-NNP, Shift-_loves_-VBZ, LeftArc-SBJ, Shift-_Mary_-NNP, RightArc-OBJ).

Liu et al. (2015) use a discriminative perceptron model with beam search (Zhang and Clark, 2011a), designing decoding algorithms that accommodate flexible constraints. The features include word(\(w\)), pos(\(p\)) and dependency label(\(l\)) information of words on the stack (\(S_{0}\), \(S_{1}\), ... from the top). For example, the word on top of stack is \(S_{0}w\) and the POS of the stack top is \(S_{0}p\). The full set of feature templates can be found in Table 2 of Liu et al. (2015), reproduced here in Table 1. These templates are called _configuration features_. When instantiated, they are combined with each legal output _action_ to score the action. Therefore, actions are atomic in feature instances.

Formally, given a configuration \(C\), the score of a possible action \(a\) is calculated as:

\[Score(a)=\vec{\theta}\cdot\Phi(\vec{C},a),\]

where \(\vec{\theta}\) is the model parameter vector of the model and \(\Phi(\vec{C},a)\) denotes a sparse feature vector that consists of features with _configuration_ and _action_ components i.e \(\Phi(\vec{C},a)\) is sparse. \(\vec{\theta}\) has to be loaded for each \(a\).

For efficiency considerations and following transition-based models, Liu et al. (2015) scores all possible actions given a configuration simultaneously. This is effectively the same as formulating the

\begin{table}
\begin{tabular}{c} Unigrams \\ \hline \(S_{0}w\); \(S_{0}p\); \(S_{0,l}w\); \(S_{0,l}p\); \(S_{0,r}w\); \(S_{0,r}p\); \\ \(S_{0,l2}w\); \(S_{0,l2}p\); \(S_{0,r2}w\); \(S_{0,r2}p\); \\ \hline \(S_{1}w\); \(S_{1}p\); \(S_{1,r}w\); \(S_{1,l}p\); \(S_{1,r}w\); \(S_{1,r}p\); \\ \(S_{1,l2}w\); \(S_{1,r2}p\); \(S_{1,r2}w\); \(S_{1,r2}p\); \\ \end{tabular} 
\begin{tabular}{c} Diagram \\ \hline \(S_{0}w\);\(S_{0,l}w\); \(S_{0}w\);\(S_{0,l}p\); \(S_{0,p}S_{0,l}w\); \(S_{0}pS_{0,l}p\); \\ \(S_{0}w\);\(S_{0,w}Sw\);\(S_{0,p}p\); \(Sp_{0}S_{r,w}\); \(S_{0}pS_{0,r}p\); \\ \hline \(S_{1}w\);\(S_{1,r}\); \(S_{1}w\); \(S_{1,p}\); \(S_{1}ps_{1,l}w\); \(S_{1}p\); \\ \(S_{1}w\);\(S_{1,r}\); \(S_{1}ps_{1,r}\); \(S_{1}ps_{1,r}\); \\ \hline \(S_{0}ws_{1}w\); \(S_{0}WS_{1}p\); \(S_{0}pS_{1}w\); \(S_{0}pS_{1}p\) \\ \hline \hline Trigram \\ \hline \(S_{0}wS_{0}pS_{0,l}w\); \(S_{0}swS_{0,l}w\); \(S_{0}wS_{0,l}p\); \(S_{0}wS_{0}pS_{0,l}p\); \\ \(S_{0}pS_{0,l}w\);\(S_{0}spS_{0,l}p\); \(S_{0}wsS_{0}pS_{0,r}w\); \(S_{0}wS_{0,l}w\); \\ \(S_{0}wS_{0}pS_{0,r}p\); \(S_{0}pS_{0,r}w\); \(S_{0}pS_{0,r}p\); \\ \hline \(S_{1}wS_{1}p\);\(W\); \(S_{1}wS_{1,l}w\);\(S_{1,l}p\); \(S_{1}wS_{1,p}\); \(S_{1}wS_{1,p}\); \\ \(S_{1}ps_{1,l}w\);\(S_{1}ps_{1,r}p\); \(S_{1}wS_{1,l}w\);\(S_{1,r}p\); \\ \(S_{1}wS_{1}ps_{1,r}p\); \(S_{1}pS_{1,r}wS_{1,r}p\); \\ \hline \hline Linearization \\ \hline \(w_{0}\); \(p\); \(w_{-}|w_{0}\); \(p_{-1}w_{0}\); \(p_{-2}w_{-1}w_{0}\); \(p_{-2}p_{-1}p_{0}\); \\ \(S_{0,l}wS_{0,l2}w\); \(S_{0,l}pS_{0,l2}p\); \(S_{0,r2}wS_{0,r}w\); \(S_{0,r2}pS_{0,r}p\); \\ \(S_{1,l}wS_{1,l2}w\); \(S_{1,l}pS_{1,l2}p\); \(S_{1,r2}wS_{1,r}w\); \(S_{1,r2}pS_{1,r}p\); \\ \hline \end{tabular}
\end{table}
Table 1: Base feature templates.

 

score into

\[Score(a)={\vec{\theta_{a}}}\cdot\Phi(\widetilde{C}),a\in A.\]

Here \(A\) is the full set of actions and \(\Phi(\widetilde{C})\) is fixed, and \(\vec{\theta_{a}}\) for all \(a\) can be loaded simultaneously. In a hash-based parameter model, it significantly improves the time efficiency.

## 3 Feature structure modification

### Two limitations of the baseline model

There are two major limitations in the feature structure of Liu et al. (2015). First, the Shift actions, which consist of the word to shift and its POS, are highly sparse. Since the action is combined with all configuration features, there will be no active feature for disambiguating the shift actions for OOV words. This issue does not exist in transition-based parsers because words are not a part of their transition actions. Second, input constraints are not leveraged by the feature model. Although the dependency relations of the word to shift can be given as inputs, they are used only as constraints to the decoder, but not as features to guide the shift action. Such lookahead information on the to-be-shifted word can be highly useful for disambiguation.

For example, consider the bag of words {_John_, _loves_, _Mary_}. Without constraints, both '_John loves Mary_' and '_Mary loves John_' are valid word ordering results. However, given the constraint (_John_, SBJ, _loves_), the correct answer is reduced to the former. The first action to build the two examples are (Shift-_John_-NNP) and (Shift-_Mary_-NNP), respectively. According to Liu et al.'s feature model, there is no feature to disambiguate the first Shift action if both '_John_' and '_Mary_' are OOV words. The system has to maintain both hypotheses and rely on the search algorithm to disambiguate them after the dependency arcs (_John_, SBJ, _loves_) and (_Mary_, OBJ, _loves_) are built. However, given the syntactic constraint that '_John_' is the subject, the disambiguation can be done right when performing the first Shift action. This requires the dependency arc label to be extracted for the word to shift e.g.(_John, Mary_), which is a lookahead feature. In addition, the OOV word '_John_' must be excluded from the feature instance, which implies that the Shift-_John_-NNP action must be simplified.

As a second example, information about dependents can also be useful for disambiguating Shift actions. In the above case, the fact that the _subject_ has not been shifted onto the stack can be a useful indicator for not shifting the verb '_loves_' onto the stack in the beginning. Inspired by the above, we exploit a range of lookahead features from syntactic constraints.

### New feature structure for SHIFT actions

We modify the feature structure of Liu et al. (2015) by breaking down the Shift-_Word-POS_ action into three components, namely Shift, _Word_ and _POS_, using only the action type Shift as the output action component in feature instances, while combin

\begin{table}
\begin{tabular}{p{49.7pt}} \hline set of label and POS of child nodes of \(L\) \\ \hline \(L_{cls};L_{cnjs};L_{cps};L_{cps};S_{0}wL_{cls};S_{0}pL_{cls};S_{1}wL_{ cls};S_{1}pL_{cls};\\ S_{0}wL_{clns};S_{0}pL_{cls};S_{1}wL_{clns};S_{1}pL_{clns};\\ S_{0}wL_{cps};S_{0}pL_{cps};S_{1}wL_{cps};S_{1}pL_{cps};\\ S_{0}wL_{cps};S_{0}pL_{cps};S_{1}wL_{cps};S_{1}pL_{cps};\\ \end{tabular} \\ \hline \hline set of label and POS of siblings of \(L\) \\ \hline \(L_{sls};L_{slsns};L_{sps};L_{spns};\\ S_{0}wL_{sls};S_{0}pL_{sls};S_{1}wL_{sls};S_{1}pL_{sls};\\ S_{0}wL_{sls};S_{0}pL_{sls};S_{1}wL_{sls};S_{1}pL_{sls};\\ S_{0}wL_{sss};S_{0}pL_{sls};S_{1}wL_{sls};S_{1}pL_{sls};\\ S_{0}wL_{spns};S_{0}pL_{spns};S_{1}wL_{spns};S_{1}pL_{spns};\\ \end{tabular} \\ \hline \hline parent label, POS and Word of \(L\) \\ \hline \(L_{ps}L_{lp};L_{ps}L_{rp};L_{ps}L_{wp};\\ S_{0}wL_{ps}L_{lp};S_{0}pL_{ps}L_{lp};S_{1}wL_{ps}L_{lp};S_{1}pL_{ps}L_{ lp};\\ S_{0}wL_{ps}L_{pp};S_{0}pL_{ps}L_{np};S_{1}wL_{ps}L_{pp};S_{1}pL_{ps}L_{pp};\\ S_{0}wL_{psps};S_{0}pL_{ps}L_{wp};S_{1}wL_{psps}L_{wp};S_{1}pL_{ps}L_{wp};\\ \end{tabular} \\ \hline \hline set of label and POS of child nodes of \(S_{0}\) \\ \hline \(S_{cls};S_{clns};S_{cps};S_{cpsn};\\ S_{0}wS_{cls};S_{0}pL_{cls};S_{1}wS_{cls};S_{1}pS_{cls};\\ S_{0}wS_{cps};S_{0}pS_{cls};S_{1}wS_{clns};S_{1}pS_{cls};\\ S_{0}wS_{cps};S_{0}pS_{cls};S_{1}wS_{cps};S_{1}pS_{cps};\\ S_{0}wS_{cps};S_{0}pS_{cpsn};S_{1}Wsc_{cpsn};S_{1}pS_{cps};\\ \end{tabular} \\ \hline \hline set of label and POS of siblings of \(S_{0}\) \\ \hline \(S_{ls};S_{slns};S_{sps};S_{spns};\\ S_{0}wS_{sls};S_{0}pS_{sls};S_{1}wS_{sls};S_{1}pS_{sls};\\ S_{0}wS_{sls};S_{0}pS_{sls};S_{1}wS_{sls};S_{1}pS_{sls};\\ S_{0}wS_{sps};S_{0}pS_{ss};S_{1}wS_{sps};S_{1}pS_{sps};\\ S_{0}wS_{spns};S_{0}pS_{sps};S_{1}wS_{sps};S_{1}pS_{sps};\\ \end{tabular} \\ \hline \hline parent label and POS of \(S_{0}\) \\ \hline \(S_{ps}S_{lp};S_{ps}S_{pp};\\ S_{0}wS_{ps}S_{lp};S_{0}pS_{ps}Sl_{lp};S_{1}wS_{ps}Sl_{lp};S_{1}pS_{ps} Sl_{lp};\\ S_{0}wS_{ps}Sp;S_{0}pS_{sp};S_{1}wS_{ps}S_{pp};S_{1}pS_{sps} Sp_{pp};\) \\ \hline \end{tabular} \\ \end{tabular}
\end{table}
Table 2: Lookahead feature templates 

ing _Word_ and \(POS\) with other configuration features to form a set of lookahead features.

For example, consider the configuration feature \(S_{0}w\), which captures the word on the top of the stack. Under the feature structure of Liu et al., it is combined with each possible action to form features for scoring the action. As a result, for scoring the action Shift-\(Lw\)-\(Lp\), \(S_{0}w\) is instantiated into \(S_{0}w\)-Shift-\(Lw\)-\(Lp\), where \(Lw\) is the word to shift and \(Lp\) is its POS. Under our new feature structure, the action component is reduced to Shift only, while \(Lw\) and \(Lp\) should be used in lookahead features. Now a effectively equivalent configuration feature to Liu et al.'s \(S_{0}w\) is \(S_{0}w\)-\(Lw\)-\(Lp\), with the lookahead \(Lw\) and \(Lp\). It gives \(S_{0}w\)-\(Lw\)-\(Lp\)-Shift when combined with the action Shift.

This new feature structure reformulates the Shift action features only. The LeftArc/RightArc actions remain LeftArc/ RightArc-LABEL since they are not sparse. Note that the change is in the _action features_ rather than the _actions_ themselves. Given the bag of words {_John_, _loves_, _Mary_}, the action Shift-_John_-NNP is still different from the action Shift-_Mary_-NNP. However, the _action component_ of the features becomes Shift only, and the words _John_/ _Mary_ must be used as lookahead _configuration features_ for their disambiguation.

The new feature structure can reduce feature sparsity by allowing lookahead features without word information. For example, a configuration feature \(S_{0}w\)-\(Lp\), which contains only the stack top word and the POS of the lookahead word, can still fire even if the word to shift is OOV, thereby disambiguating OOV words of different POS. In addition, the lookahead \(Lw\) and \(Lp\) do not have to be combined with every other configuration feature, as with Liu et al. (2015), thereby allowing more flexible feature combination and a leaner model.

### The new features

The new feature structure includes two types of features. The first is the same feature set as Liu et al. (2015), but with the Shift action component not having _Word_ and _POS_ information. We call this type of features as _base features_. The second is a set of _lookahead features_, which are shown in Table 2. Here \(L_{cls}\) represents set of arc labels on child nodes (of the word \(L\) to shift) that have been shifted on to the stack, \(L_{clns}\) represents set of labels on child nodes that have not been shifted, \(L_{sls}\) the label set of shifted sibling nodes, \(L_{slns}\) the label set of unshifted sibling nodes, \(L_{cps}\) the POS set of shifted child nodes, \(L_{cpns}\) the POS set of unshifted child nodes, \(L_{sps}\) the POS set of shifted sibling nodes and \(L_{spn}\) the POS set of unshifted sibling nodes. \(L_{ps}\) is a binary feature indicating if the parent has been shifted. \(L_{lp}\) represents label on the parent, \(L_{pp}\) POS of parent and \(L_{wp}\) the parent word form. We define similar lookahead features for \(S_{0}\). These features are instantiated only for Shift actions.

The new feature structure prevents all possible actions from being scored simultaneously, because the lookahead _Word_ and _POS_ are now in configuration features, rather than output actions, making it necessary to score the shifting of different words or POS separately. This leads to reduced search speed. Nevertheless, our experiments show that they give a desirable tradeoff between efficiency and accuracy.

Note that the new features are much less than a full Cartesian product of lookahead features and the original features. This is a result of manual feature engineering, which allows similar accuracies to be achieved using a much smaller model, thereby increasing the time efficiency.

\begin{table}
\begin{tabular}{|l||r r|r r|r r|r r|r r|r r|r r|r r|} \hline  & no pos & \multicolumn{3}{c|}{50\% pos} & \multicolumn{3}{c|}{all pos} & \multicolumn{3}{c|}{no pos} & \multicolumn{3}{c|}{50\% pos} & \multicolumn{3}{c|}{all pos} & \multicolumn{3}{c|}{no pos} & \multicolumn{3}{c|}{50\% pos} & \multicolumn{3}{c|}{all pos} \\  & no dep & \multicolumn{3}{c|}{no dep} & \multicolumn{3}{c|}{no dep} & \multicolumn{3}{c|}{50\% dep} & \multicolumn{3}{c|}{50\% dep} & \multicolumn{3}{c|}{50\% dep} & \multicolumn{3}{c|}{all dep} & \multicolumn{3}{c|}{all dep} & \multicolumn{3}{c|}{all dep} & \multicolumn{3}{c|}{all dep} \\ \cline{2-19}  & BL & SP & BL & SP & BL & SP & BL & SP & BL & SP & BL & SP & BL & SP & BL & SP \\ \hline Z13 & 42.9 & 4872 & 43.4 & 4856 & 44.7 & 4826 & 50.5 & 4790 & 51.4 & 4737 & 52.2 & 4720 & 73.3 & 4600 & 74.7 & 4431 & 76.3 & 4218 \\ L15 & 47.5 & 155 & 47.9 & 119 & 48.8 & 74 & 54.8 & 132 & 55.2 & 91 & 56.2 & 41 & 77.8 & 40 & 79.1 & 28 & 81.1 & 22 \\ Ours & 48.0 & 175 & 49.0 & 156 & 51.5 & 148 & 59.0 & 144 & 62.0 & 160 & 67.1 & 171 & 82.8 & 62 & 86.2 & 68 & 89.9 & 70 \\ \hline \end{tabular}
\end{table}
Table 3: Development partial-tree linearization results. \(BL-\text{BLEU}\) score; \(SP-\text{number}\) of milliseconds per sentence. Z13 – best-first system of Zhang (2013) and L15 – transition-based system of Liu et al. (2015).

 



## 4 Experiments

Following previous work we conduct experiments on the Penn TreeBank (PTB), using Wall Street Journal sections 2-21 for training, 22 for development and 23 for testing. Gold-standard dependency trees are derived from bracketed sentences using Penn2Malt, and base noun phrases are treated as a single word. The BLEU score is used to evaluate the performance of linearization.

Table 4 shows a difference in scores between transition-based linearization system of Liu et al. (2015) (L15) and best-first system of Zhang (2013) (Z13). L15 performs better for word ordering with unlabeled dependency arcs, but poorly for the task of labeled syntactic linearization.

Table 3 shows a series of development experiments comparing our system with Z13 and L15. We vary the amount of input syntactic constraints by randomly sampling from POS and dependency labels of the development set. Our system gives consistently higher accuracies when compared with both Z13 and L15. Compared to L15, the increase in scores for unconstrained word ordering is due to the introduction of reduced feature sparsity. The improvements on tree linearization tasks involving partial to full dependency constraints are also due to lookahead features that leverage tree information to reduce ambiguity early. Though slower than L15, our system is over 30 times faster compared to Z13.

We compare final test scores with previous methods in the literature in Table 4. Our system improves upon the previous best scores by 8.7 BLEU points for the task of unlabeled syntactic linearization. For the task of labeled syntactic linearization, we achieve the score of 91.8 BLEU points, the highest results reported so far.

Table 5 contains examples of fully constrained output . In the first example '_will_' is the ROOT node with two child nodes '_also_' and '_compete_'. Lookahead feature for child dependency labels \(L_{cls},L_{clns}\) on the node '_will_' can help order the segment '_also will compete_' correctly in our system. Without such features, the system of L15 yields an output that starts with '_The spinoff with Fujitsu_' which is locally fluent, but leaving the words '_also_' and '_will_' difficult to handle. In the second example, '_Dr. Talcott_' is OOV. Hence system of L15 is not able to score it and thus order it correctly. Our system makes use of both POS and dependency label of '_Dr. Talcott_' to order it correctly.

## 5 Conclusion

We identified a feature sparsity issue in state-of-the-art transition-based word ordering, proposing a solution by redefining the feature structure and introducing lookahead features. The new method gives the best accuracies on a set of benchmarks, which show that transition-based methods are a fast and accurate choice for syntactic linearization. Future work include the testing of this model in a linearization shared task Belz et al. (2011) and investigating the integration of large scale training data Zhang et al. (2012); Liu and Zhang (2015).

We release our source code under GPL at [https://github.com/SUTDNLP/ZGen/releases/tag/v0.2](https://github.com/SUTDNLP/ZGen/releases/tag/v0.2).

## Acknowledgments

We thank Yijia Liu for helpful discussions and for sharing the Latex templates and the anonymous reviewers for their constructive comments. This work was supported by the Singapore Ministry of Education (MOE) AcRF Tier 2 grant T2MOE201301.

\begin{table}
\begin{tabular}{l||c c c|c} \hline  & \multicolumn{3}{c}{unlabeled} & \multicolumn{1}{c}{labeled} \\  & no pos & all pos & all pos & all pos \\  & no dep & no dep & all dep & all dep \\ \hline W09 & - & 33.7 & - & - \\ Z11 & - & 40.1 & - & - \\ Z13 & 44.7 & 46.8 & 76.2 & 89.3 \\ L15 & 49.4 & 50.8 & 82.3 & 82.9 \\ This paper & **50.5** & **53.0** & **91.0** & **91.8** \\ \hline \end{tabular}
\end{table}
Table 4: Final results. W09 – Wann et al. (2009), Z11 – Zhang and Clark (2011b)

\begin{table}
\begin{tabular}{l||p{142.3pt}} \hline  & Fully constrained output \\ \hline ref. & The spinoff also will compete with Fujitsu \\ L15 & The spinoff with Fujitsu compete also will \\ Ours & The spinoff also will compete with Fujitsu \\ \hline ref. & Dr. Talcott led a team of researchers from the National Cancer Institute . \\ L15 & a team of researchers from the National Cancer Institute led Dr. Talcott . \\ Ours & Dr. Talcott led a team of researchers from the National Cancer Institute . \\ \hline \end{tabular}
\end{table}
Table 5: Example outputs.





## References

* [Belz et al.2011] Anja Belz, Michael White, Dominic Espinosa, Eric Kow, Deirdre Hogan, and Amanda Stent. 2011. The first surface realisation shared task: Overview and evaluation results. In _Proceedings of the 13th European workshop on natural language generation_, pages 217-226. Association for Computational Linguistics.
* [Blackwood et al.2010] Graeme Blackwood, Adria De Gispert, and William Byrne. 2010. Fluency constraints for minimum bayes-risk decoding of statistical machine translation lattices. In _Proceedings of the 23rd International Conference on Computational Linguistics_, pages 71-79. Association for Computational Linguistics.
* [Bohnet et al.2010] Bernd Bohnet, Leo Wanner, Simon Mille, and Alicia Burga. 2010. Broad coverage multilingual deep sentence generation with a stochastic multi-level realizer. In _Proceedings of the 23rd International Conference on Computational Linguistics_, pages 98-106. Association for Computational Linguistics.
* [Chen and Manning2014] Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, 1:740-750.
* [De Gispert et al.2014] A De Gispert, M Tomalin, and W Byrne. 2014. Word ordering with phrase-based grammars. _14th Conference of the European Chapter of the Association for Computational Linguistics 2014, EACL 2014_, pages 259-268.
* [Filippova and Strube2007] Katja Filippova and Michael Strube. 2007. Generating constituent order in german clauses. In _Annual Meeting-Association for Computational Linguistics_, volume 45, page 320.
* [He et al.2009] Wei He, Haifeng Wang, Yuqing Guo, and Ting Liu. 2009. Dependency based chinese sentence realization. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2_, pages 809-816.
* [Liu and Zhang2015] Jiangming Liu and Yue Zhang. 2015. An empirical comparison between n-gram and syntactic language models for word ordering. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 369-378, Lisbon, Portugal, September. Association for Computational Linguistics.
* June 5, 2015_, pages 113-122.
* [Nivre and Scholz2004] Joakim Nivre and Mario Scholz. 2004. Deterministic dependency parsing of english text. In _Proceedings of the 20th international conference on Computational Linguistics_, page 64. Association for Computational Linguistics.
* [Song et al.2014] Linfeng Song, Yue Zhang, Kai Song, and Qun Liu. 2014. Joint morphological generation and syntactic linearization. _Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence_, pages 1522-1528.
* [Wann et al.2009] Stephen Wann, Mark Dras, Robert Dale, and Cecile Paris. 2009. Improving grammaticality in statistical sentence generation: Introducing a dependency spanning tree algorithm with an argument satisfaction model. In _Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics_, pages 852-860.
* [White and Rajkumar2009] Michael White and Rajakrishnan Rajkumar. 2009. Perceptron reranking for ccg realization. In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1_, pages 410-419. Association for Computational Linguistics.
* [White2004] Michael White. 2004. Reining in ccg chart realization. In _Natural Language Generation_, pages 182-191. Springer Berlin Heidelberg.
* [Zhang and Clark2011a] Yue Zhang and Stephen Clark. 2011a. Syntactic processing using the generalized perceptron and beam search. _Computational linguistics_, 37(1):105-151.
* [Zhang and Clark2011b] Yue Zhang and Stephen Clark. 2011b. Syntax-based grammaticality improvement using ccg and guided search. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 1147-1157. Association for Computational Linguistics.
* [Zhang and Clark2015] Yue Zhang and Stephen Clark. 2015. Discriminative syntax-based word ordering for text generation. _Computational Linguistics_, 41(3):503-538.
* [Zhang et al.2012] Yue Zhang, Graeme Blackwood, and Stephen Clark. 2012. Syntax-based word ordering incorporating a large-scale language model. In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_, pages 736-746. Association for Computational Linguistics.
* [Zhang et al.2014] Yue Zhang, Kai Song, Linfeng Song, Jingbo Zhu, and Qun Liu. 2014. Syntactic smt using a discriminative text generation model. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 177-182, Doha, Qatar, October. Association for Computational Linguistics.
* [Zhang2013] Yue Zhang. 2013. Partial-tree linearization: generalized word ordering for text synthesis. In _Proceedings of the Twenty-Third international joint conference on Artificial Intelligence_, pages 2232-2238. AAAI Press