SlowGuess/ACL_2020_OCR huggingface dataset:
Xiang Yu, Simon Tannert, Ngoc Thang Vu, Jonas Kuhn
Institut für Maschinelle Sprachverarbeitung
University of Stuttgart, Germany
firstname.lastname@ims.uni-stuttgart.de
We propose a graph-based method to tackle the dependency tree linearization task. We formulate the task as a Traveling Salesman Problem (TSP), and use a biaffine attention model to calculate the edge costs. We facilitate the decoding by solving the TSP for each subtree and combining the solution into a projective tree. We then design a transition system as post-processing, inspired by non-projective transition-based parsing, to obtain non-projective sentences. Our proposed method outperforms the state-of-the-art linearizer while being 10 times faster in training and decoding.
Surface realization is the task of generating a sentence from a syntactic or semantic representation. In several shared tasks (Belz et al., 2011; Mille et al., 2018, 2019), the input representations are unordered dependency trees. The state-of-the-art system (Yu et al., 2019a) in the Surface Realization Shared Task 2019 (SR'19) takes a pipeline approach, where the first step is linearization, namely ordering the tokens in the dependency tree. They use a Tree-LSTM to encode each token with awareness of the whole tree, then apply the divide-and-conquer strategy to split the full tree into subtrees and find the optimal order for each subtree using beam search. Finally, the linearized subtrees are combined into a full projective tree. The general strategy is adapted from Bohnet et al. (2010).
In this work, we tackle linearization decoding in a different way, by casting it as a Traveling Salesman Problem (TSP). Knight (1999) first formulated the word ordering of the target language in word-based machine translation as a TSP, where the words are the nodes to traverse, and the log probabilities of the bigrams are the edge costs.
Several works have followed this formulation. Among others, Zaslavskiy et al. (2009) formulate the word ordering in phrase-based machine translation as a TSP, and show that it achieves better performance and speed than beam search decoding with the same bigram language model. Horvat and Byrne (2014) explore higher-order  $n$ -gram language models for TSP-based word ordering, which transforms into a much larger TSP graph. All of the aforementioned works operate on a bag of words without syntax, which is a TSP graph of non-trivial size with little information about the internal structure. Much effort has been put into incorporating more powerful decoding algorithms such as Integer Programming (Germann et al., 2001) and Dynamic Programming (Tillmann and Ney, 2003).
Our work differs from the previous work on TSP-based word ordering in several aspects. (1) Linearization is a special case of word ordering with syntax, where we can use a tree-structured encoder to provide better representation of the tokens. (2) We adopt the divide-and-conquer strategy to break down the full tree into subtrees and order each subtree separately, which is faster and more reliable with an approximate decoder. (3) We apply deep biaffine attention (Dozat and Manning, 2016), which has yielded great improvements in dependency parsing, and reinterpret it as a bigram language model to compute edge costs for the TSP.
In this paper, we solve the dependency tree linearization task as a TSP. With the help of TreeLSTM to encode the tree and biaffine attention as a bigram language model, we can use a greedy TSP solver to linearize the tree effectively. Furthermore, the divide-and-conquer strategy greatly reduces the search space but introduces the projectivity restriction, which we remedy with a transition-based reordering system. As a result, the proposed linearizer outperforms the previous state-of-the-art model both in quality and speed.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1451-1462 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
2 Graph-based Linearization
2.1 System Overview
We follow the idea in Knight (1999) to treat linearization as a TSP. Under the TSP formulation, we need to calculate the cost from every node  $i$  to every other node  $j$ , which can be interpreted as the log likelihood of the bigram  $(i,j)$ . We use the bi-affine attention model (Dozat and Manning, 2016) to obtain the costs, and use an off-the-shelf TSP solver, OR-Tools<sup>1</sup>, to decode the TSP.
To facilitate the approximate decoding of this NP-hard problem, we follow the divide-and-conquer strategy in Bohnet et al. (2010) of splitting the tree into subtrees, and decoding each subtree separately. There are pros and cons of this approach: on the one hand, the search space is much smaller so that a greedy TSP solver can find good solutions in reasonable time; on the other hand, it restricts the output to be projective, i.e., non-projective sentences can never be produced.
To remedy the projectivity restriction, we introduce a post-processing step using a simple transition system with only two transitions, swap and shift, to sort the linearized projective trees into potentially non-projective ones.
This system is an extension of our previous work (Yu et al., 2019a). We use the same encoder and hyperparameters (see Appendix A) and only experiment with the decoders. The code is available at the first author's web page.
As an overview, Figure 1 illustrates our pipeline for the linearization task, with an unordered dependency tree as input, and a linearized sentence as output, which is potentially non-projective, i.e., with crossing dependency arcs.
To solve the task, we (1) divide the tree into subtrees, (2) linearize each subtree by solving a TSP, (3) combine the linearized subtrees into an projective tree, and (4) use the swap system to obtain a non-projective tree.
2.2 TSP Solver
To formulate the linearization task as a TSP, we use a node to represent each token in the tree, and an extra node with index 0 as both the origin and destination, which is interpreted as the boundaries
Figure 1: An overview of the linearization system.
(a) Cost matrix
(b) TSP graph
Figure 2: The graph and matrix views of a TSP.
in the output sequence. Figure 2 demonstrates a decoded TSP graph from its edge cost matrix, where the output sequence is “
 let us get 
”.
We use the routing solver of OR-Tools to solve the TSP given the edge costs. It is a generic optimization library, unlike the more specialized and optimized TSP solvers such as Concorde (Applegate et al., 2006), but it enables imposing extra word order constraints described in §2.6, and can be easily extended to other constraints.
Among all the first solution strategies provided in OR-Tools, we found GLOBAL_CHEAPEST_ARC to perform the best, which selects a valid edge with the lowest cost at every step until a full path is found. For the sake of efficiency, we use the greedy metaheuristic GREEDY_desCENT to refine the first solutions, which converges to local optima in very short time. In practice, it works extremely well in combination with the greedy training described in §2.4.
1https://developers.google.com/ optimization 2https://www.ims.uni-stuttgart.de/en/ institute/team/Yu-00010/
More advanced metaheuristics such as GUIDED_LOCAL_SEARCH (Voudouris and Tsang, 1999) could find better solutions, but also require much more decoding time, it is thus less practical for real-time generation tasks. We do not use it in the default setting, but include it in the analysis to demonstrate the effectiveness of the training.
2.3 Scoring Model
We use the biaffine attention model (Dozat and Manning, 2016) to calculate the TSP edge costs. First, we obtain the representation for each token by concatenating the embeddings of the features, then encode the tree information with a bidirectional Tree-LSTM, as described in Yu et al. (2019b):
$$
\mathbf {v} _ {i} ^ {\circ} = \mathbf {v} _ {i} ^ {(l e m)} \oplus \mathbf {v} _ {i} ^ {(p o s)} \oplus \mathbf {v} _ {i} ^ {(d e p)} \oplus \mathbf {v} _ {i} ^ {(m o r)} \tag {1}
$$
$$
\mathbf {x} _ {i} = \operatorname {T r e e - L S T M} \left(\mathbf {v} _ {0} ^ {\circ} \dots \mathbf {v} _ {n} ^ {\circ}\right) _ {i} \tag {2}
$$
The parameters of the decoder consist of two Multi-Layer Perceptrons  $(\mathrm{MLP}^{(fr)}$  and  $\mathrm{MLP}^{(to)})$  and a biaffine matrix (W). We use the MLPs to obtain different views of the token representation as the first and second token in the bigram:
$$
\mathbf {h} _ {i} ^ {(f r)} = \operatorname {M L P} ^ {(f r)} (\mathbf {x} _ {i}) \tag {3}
$$
$$
\mathbf {h} _ {i} ^ {(t o)} = \operatorname {M L P} ^ {(t o)} (\mathbf {x} _ {i}) \tag {4}
$$
We then apply the biaffine transformation on the vectors of the first word  $\mathbf{h}_i^{(fr)}$  and the second word  $\mathbf{h}_j^{(to)}$  to compute the score  $s_{i,j}$  of each bigram  $(i,j)$ , where  $\mathbf{W}$  is the weight matrix of size  $(k + 1) \times (k + 1)$ , and  $k$  is the size of  $\mathbf{h}_i^{fr}$  and  $\mathbf{h}_j^{to}$ :
$$
s _ {i, j} = \left(\mathbf {h} _ {i} ^ {(f r)} \oplus 1\right) \times \mathbf {W} \times \left(\mathbf {h} _ {j} ^ {(t o)} \oplus 1\right) ^ {\top} \tag {5}
$$
In the actual computation, we parallelize Equation 5, where  $\mathbf{S}$  is the output score matrix of size  $n\times n$ , and  $n$  is the number of tokens:
$$
\mathbf {S} = \left(\mathbf {H} ^ {(f r)} \oplus \mathbf {1}\right) \times \mathbf {W} \times \left(\mathbf {H} ^ {(t o)} \oplus \mathbf {1}\right) ^ {\top} \tag {6}
$$
Finally, we turn the score matrix into a non-negative cost matrix for the TSP solver:
$$
\mathbf {C} = \max  (\mathbf {S}) - \mathbf {S} \tag {7}
$$
Our model is inspired by the biaffine dependency parser of Dozat and Manning (2016), but stands in contrast in many aspects. They use a bidirectional LSTM to encode the sequential information of the tokens, and the biaffine attention itself does not
model the sequence. Each cell  $s_{i,j}$  in their output matrix  $\mathbf{S}$  is interpreted as the score of a dependency arc  $(i,j)$ . They use a Maximal Spanning Tree algorithm to obtain a tree that maximizes the total score of the arcs in the tree.
In the case of linearization, our input and output are the opposite to theirs. The input has no sequential but syntactic information, encoded by the bidirectional Tree-LSTM. Each cell  $s_{i,j}$  in the output matrix  $\mathbf{S}$  is interpreted as the score of the bigram  $(i,j)$ . We use a TSP solver to obtain a traversal of the tokens by minimizing the total edge costs, i.e., maximizing the total bigram scores.
We use a greedy training objective to train the bi-affine scoring model, namely we enforce the score of each bigram  $(i,j)$  in the correct sequence  $z$  to be higher than any other bigrams in the same row or in the same column in the matrix by a margin:
$$
\begin{array}{l} L = \sum_ {(i, j) \in z} (\sum_ {j ^ {\prime} \neq j} \max (0, 1 + s _ {i, j ^ {\prime}} - s _ {i, j}) \\ + \sum_ {i ^ {\prime} \neq i} \max  \left(0, 1 + s _ {i ^ {\prime}, j} - s _ {i, j}\right) \tag {8} \\ \end{array}
$$
This objective aims to maximizing the score of each correct bigram  $(i,j)$  in both directions, essentially  $\log P(j|i)$  and  $\log P(i|j)$ , where the cells in the same row correspond to all possible tokens following  $i$ , and the cells in the column correspond to all possible tokens preceding  $j$ .
The objective is greedy in the sense that it updates more than "necessary" to decode the correct path. We contrast it to the structured loss in most graph-based dependency parsers (McDonald et al., 2005; Kiperwasser and Goldberg, 2016), which updates the scores of the correct path  $z$  against the highest scoring incorrect path  $z'$ :
$$
L ^ {\prime} = \max  (0, 1 + \max  _ {z ^ {\prime} \neq z} \sum_ {(i ^ {\prime}, j ^ {\prime}) \in z ^ {\prime}} s _ {i ^ {\prime}, j ^ {\prime}} - \sum_ {(i, j) \in z} s _ {i, j}) \tag {9}
$$
The greedy objective for the TSP has two main advantages: (1) it does not require decoding during training, which saves training time; (2) it pushes the scores of each correct bigram to be the highest in the row and the column, which facilitates the greedy solver (GLOBAL_CHEAPEST_ARC) to find a good initial solution. In fact, if the objective
reaches 0, the greedy solver is guaranteed to find the optimal solution, since at each step, the cheapest arc is always a correct bigram instead of any other bigram in the same row or column.
2.5 Generating Non-Projective Trees
If we directly linearize the full tree, the output is naturally unrestricted, i.e., possibly non-projective. However, when we linearize each subtree separately in order to reduce the search space, as in the proposed method, the reconstructed output is restricted to be projective (Bohnet et al., 2010).
To relax the projectivity restriction, we design a transition system to reorder projective trees into non-projective trees as a post-processing step, inspired by Nivre (2009) but working in the opposite way. It is essentially a reduced version of their transition system, removing the attachment transitions and keeping only swap and shift.
In the transition system (as shown in Table 1), a configuration consists of a stack  $\sigma$ , which is initially empty, and a buffer  $\beta$ , which initially holds all input tokens. The shift transition moves the front of the buffer to the top of the stack, and the swap transition moves the top of the stack back to the second place in the buffer. When all tokens are moved from the buffer to the stack, the procedure terminates. To prevent the model predicting infinite shift-swap loops, we only allow swap if the initial index of the top of the stack is smaller than the front of the buffer. The worst-case complexity of the sorting is quadratic to the number of tokens, however, since trees in natural language mostly only contain very few non-projective arcs, the transition system works in expected linear time, as shown in Nivre (2009).
We then implement a model to predict the transitions given the configurations. We use two LSTMs to dynamically encode the stack from left to right  $(\mathrm{LSTM}_{\sigma})$  and the buffer from right to left  $(\mathrm{LSTM}_{\beta})$ . We then concatenate the two outputs and use a MLP to predict the next transition.
When a shift is performed, we update  $\mathrm{LSTM}_{\sigma}$  state with the vector of the shifted token as the new stack representation, and the new buffer representation is the  $\mathrm{LSTM}_{\beta}$  output of the new front token; when a swap is performed, the new stack representation is the  $\mathrm{LSTM}_{\sigma}$  output of the new top token, and the new buffer representation is recalculated by feeding the now second and first token in the buffer to the  $\mathrm{LSTM}_{\beta}$  state of the third token.
Transition	Before	After
shift	(σ, [i|β]) → ([σ|i], β)
swap	([σ|i], [j|β]) → (σ, [j|i|β])
Table 1: The shift-swap transition system.
Figure 3: An illustration of the swap model. (1) shows a configuration [1 2 | 3 4 5 6], where the stack contains 1 and 2, and the buffer contains 3, 4, 5 and 6, and the model predicts shift to move 3 from the buffer to the stack; (2) is the resulted configuration, and the model predicts swap, which moves 3 back to the buffer behind 4, as shown in (3). The solid black arrows represent the computation already done in the previous steps, and the dashed red arrows represent the new computation needed for the current step.
Figure 3 illustrates the model under the transition system, where the arrows to the right represent  $\mathrm{LSTM}_{\sigma}$ , the arrows to the left represent  $\mathrm{LSTM}_{\beta}$  and the arrows between the stack and the buffer represent the MLP. After each transition, little computation is needed to represent the new stack and buffer, marked with the red dashed line. The example illustrates the steps to modify the configuration [12 | 3456] into [12 | 4356].
Note that the transition system is sound and complete, which means there is always a sequence of transitions to sort any sequence into any reordering. In other words, the transition system on its own could also linearize the tree by taking a random permutation as input. However, due to the noisy input order, it is very difficult for the LSTM model to learn good representations for the stack and buffer and predict correct transitions (cf. Vinyals et al. (2015) for the discussion on encoding a set with an LSTM). In contrast, when we only use this sys
tem to reorder a linearized projective tree as postprocessing, where input sequence is meaningful and consistent, it is much easier to learn.
Using the swap system as a post-processing step stands in contrast to Bohnet et al. (2012), where they pre-process the tree by lifting the arcs so that the correct word order could form a projective tree. These two approaches draw inspiration from the non-projective parsing in Nivre (2009) and the pseudo-projective parsing in Nivre and Nilsson (2005) respectively. We argue that our post-processing approach is more convenient since there is no need to change the syntactic annotation in the original tree, and it is much easier to evaluate the effectiveness of the sorting model.
2.6 Relative Word Order Constraints
In the SR'19 dataset, some relative word order information is given, which indicates e.g. the order of the conjuncts in the coordination. Since the order in a coordination is generally arbitrary (at least syntactically), it will thus introduce randomness in the single reference evaluation. We believe that using such information leads to more accurate evaluation, and therefore by default always use these constraints in the comparison.
The constraints does not specify direct adjacency, but only general precedence relations. For example, to order the nodes  $\{1,2,3,4,5\}$  with the constraint  $2\prec 3\prec 1$  and  $4\prec 5$ , a valid sequence could be  $[2,4,3,5,1]$ , while  $[4,5,2,1,3]$  is invalid.
To incorporate such constraints in the solver, we introduce an additional variable associated with each node in the routing problem, where the value is incremented by 1 after each step. In other words, if a node is visited in the  $n$ -th step, then the associated variable will have the value  $n$ . Then we add inequality constraints about those variables that are specified in the word order information into the routing problem and let the solver find the path that satisfies the constraints.
In practice, the solver can always find a solution to linearize the subtrees with the constraints. However, it sometimes cannot find any solution to directly linearize the full tree within the time limit (1-10% of the cases depending on the treebank), because there are more nodes and more constraints in the full tree. In this case, we simply remove the constraints and rerun the solver.
3.1 Data and Baselines
We use the datasets from the Surface Realization 2019 Shared Task (Mille et al., 2019) in our experiments, which includes 11 languages in 20 treebanks from the Universal Dependencies (Nivre et al., 2016). We experiment on the shallow track, i.e., all tokens in the output are present in the input tree. We only report the BLEU score (Papineni et al., 2002) as the evaluation metric, since we mostly evaluate on the lemma level, where the metrics involving word forms are irrelevant.
As baselines for the final evaluation, we use several available linearizers by Bohnet et al. (2010) (B10), Puduppully et al. (2016) (P16) and Yu et al. (2019a) (Y19). B10, P16 and our linearizer all use the same inflection and contraction models, trained with the same hyperparameters as in Y19, and we compare to the reported shared task results of Y19.
Table 2 shows the performance of different linearizers, where beam is the baseline beam-search linearizer as in Yu et al. (2019b) with default hyperparameters, full is the TSP decoder on the full tree level, sub is the TSP decoder on the subtree level, and +swap is sub post-processed with reordering. We test the decoders under two conditions: without word order constraints (-constraints) and with word order constraints (+constraints). Columns 2-9 show the BLEU scores on lemmata on the development set, and in the last 4 columns are the BLEU scores on inflected and contracted word forms on the test sets with the official evaluation script of SR'19.
While both only generating projective sentences, the sub decoder outperforms the baseline beam decoder by 0.6 BLEU points without word order constraints and 0.3 BLEU points with constraints. Note that the beam search decoder uses an LSTM to score the sequences, which is essentially an unlimited language model, while the TSP decoders only uses a bigram language model.
While comparing the two TSP decoders, sub performs on average higher than full, while full performs better on treebanks with more nonprojective sentences, since it is not restricted. Without word order constraints, full even slightly outperforms sub. The reason that full performs relatively worse with constraints is that it sometimes has to remove the constraints to find a solution.
%NP	-constraints	+constraints	Final Test
beam	full	sub	+swap	beam	full	sub	+swap	B10	P16	Y19	Ours
ar_padt	0.48	85.09	85.00	85.67	85.67	86.74	85.54	86.74	86.74	56.62	56.17	64.90	67.02
en_ewt	0.62	85.19	85.30	85.99	85.99	88.38	87.40	88.01	88.03	72.97	74.53	82.98	84.08
en_gum	1.00	84.75	85.20	85.86	86.00	86.96	86.53	87.43	87.55	69.94	70.88	83.84	84.72
enlines	3.81	79.23	81.60	80.04	81.63	81.71	82.92	81.97	83.51	63.15	67.67	81.00	81.55
en_partut	0.34	87.05	87.90	86.73	86.73	88.13	87.03	88.77	88.77	80.64	70.97	87.25	85.52
es_ancora	0.88	83.68	85.00	84.21	84.70	84.88	85.29	85.55	86.09	80.80	69.73	83.70	85.34
es_gsd	0.51	83.68	84.70	84.26	84.31	85.70	85.62	86.29	86.34	79.18	70.34	82.98	82.78
fr_gsd	0.61	87.13	88.00	87.58	87.60	89.41	88.88	89.68	89.77	79.34	72.25	83.95	83.34
fr_partut	0.46	87.56	88.40	90.38	90.38	90.07	89.76	90.58	90.58	75.13	64.20	83.38	83.21
fr_sequoia	0.27	87.20	85.50	87.13	87.29	89.74	86.85	89.44	89.60	77.48	62.67	84.52	83.81
hi_hdtb	1.20	83.14	85.10	83.82	85.37	85.04	86.32	85.50	87.06	77.89	74.70	80.56	82.54
id_gsd	0.57	81.65	81.80	81.90	82.06	85.56	82.45	86.30	86.44	77.90	76.51	85.34	85.57
ja_gsd	0.14	90.66	90.10	90.50	90.50	92.83	91.65	92.83	92.83	83.67	81.21	87.69	87.87
ko_gsd	3.10	76.29	76.60	75.73	77.21	79.26	79.39	79.47	80.87	61.76	65.89	74.19	75.12
ko_kaist	3.59	79.24	84.00	80.52	83.18	80.29	84.50	80.84	83.53	63.48	71.41	73.93	77.50
pt_bosque	2.95	82.57	84.40	83.04	84.02	83.97	85.46	84.40	85.19	75.41	67.91	77.75	79.15
pt_gsd	0.44	87.45	88.40	87.91	88.15	88.96	89.12	89.83	90.06	74.44	68.11	75.93	77.00
ru_gsd	0.84	74.50	73.80	74.99	75.06	79.07	75.37	78.86	78.87	63.32	62.93	71.23	71.27
ru_syntagrus	1.14	77.95	78.90	79.23	79.39	81.00	80.77	81.59	81.71	74.28	71.50	76.94	78.39
zh_gsd	0.06	81.83	80.20	82.25	82.25	83.29	79.65	82.93	82.93	77.88	70.55	83.85	84.76
AVG	1.15	83.29	84.04	83.89	84.34	85.55	85.03	85.85	86.32	73.26	69.51	80.30	81.03
Table 2: Percentage of non-projective arcs (column 1); BLEU scores on lemmata on the development set for different linearization decoders, with linear order constraints (column 2-5) and without linear order constraints (column 6-9); and BLEU scores on inflected words on the test set compared with several baseline systems (column 10-13), where B10 denotes Bohnet et al. (2010), P16 denotes Puduppully et al. (2016), and Y19 denotes Yu et al. (2019a). The best result in each group is marked with bold face.
The sub+swap decoder eliminates the projectivity restriction, closing the performance gap to full for non-projective treebanks, and it does not hurt the performance on the projective treebanks.