SlowGuess/ACL_2020_OCR huggingface dataset:
In the last four columns we compare our sub+swap linearizer on the test set for the full pipeline with three external baselines, including the best system in the SR'19 shared task (Y19). Our system outperforms B10 and P16 by a large margin of 7 and 11 BLEU points. Note that their off-the-shelf systems are not designed to use word order constraints and morphological tags, which would account for a difference of about 3 points (see the effect of constraints in Table 2 and feature ablation in §3.7). Under the same condition, our system outperforms Y19 on most of the treebanks and on average by 0.7, because of (1) a better projective decoder and (2) the non-projective post-processing step. Furthermore, our system is much faster than Y19, see the comparison in §3.8.
3.3 Error Analysis
To illustrate the characteristics of different TSP decoders, we analyze their performance on sentences with different lengths and percentages of non-projective arcs.
Figure 4a shows the BLEU score of different TSP decoders with respect to the sentence length, averaged over all sentences in the development sets. The sub model performs quite stably across the sentences with different lengths, while the full model performs much worse on longer sentences. This confirms our hypothesis that the divide-and-conquer strategy of the subtree decoder can reduce search errors for large TSP problems. Post-processing with the swap system (tsp+swap) consistently improves tsp across all sentence lengths.
Note that the very short sentences have even lower BLEU score, this is caused by the smoothing function in the BLEU evaluation, which gives a low score even for exact match.
(a) BLEU vs.Sentence lengths.
(b) BLEU vs.non-projective arc rate.
Figure 4: BLEU score with respect to the sentence length and the percentage of non-projective arcs.
Figure 4b shows the BLEU score with respect to the percentage of non-projective arcs in the gold tree, averaged over all sentences in the development sets. Clearly, sub performs lower than full for sentences with more non-projective arcs due to the projectivity restriction, while the overall BLEU score of sub is higher, since  $99\%$  of the arcs and  $90\%$  of the sentences are projective. With the help of the swap system, sub+swap closes the gap to full on the non-projective sentences.
In sum, the sub+swap model shows clear advantages over the other models since it is less prone to search error due to the reduced TSP size and free from the projectivity restriction, it is thus the best of both worlds.
As described in §2.4, we use a greedy training objective to train the biaffine model, namely we calculate a hinge loss of the correct bigram against
row+col	row	col	path
sub	85.85	85.56	85.60	85.13
full	85.03	84.36	84.38	80.96
Table 3: BLEU scores of different decoders with different training objectives.
BLEU	ΔBLEU
gold	pred	rand	gold	pred	rand
gold	98.50	86.32	2.63	+0.67	+0.47	+0.13
pred	97.52	85.78	3.61	-0.32	-0.07	+1.11
rand	87.83	81.97	71.36	-10.01	-3.88	+68.86
Table 4: The change in BLEU scores after applying the transition-based sorting models that are trained (row) and tested (column) under different conditions.
all other bigrams in the same row and in the same column. This is in contrast to the structured loss, which is calculated between the gold sequence and the predicted sequence.
This contrast is similar to the two different training objective in Dozat and Manning (2016) against Kiperwasser and Goldberg (2016) for graph-based dependency parsing. We experiment with the structured loss, following Kiperwasser and Goldberg (2016), where we also apply loss augmented inference (Taskar et al., 2005), i.e., adding a constant for all the bigrams that are not in the gold sequence.
We also experiment with only updating against the row or the column, which could be thought of as the bigram language model only in one direction, while updating against both is training a bidirectional language model.
Table 3 shows the results, where we train the sub and full models with different objectives: row+col is the default one, row and col only update against the row or the column, and path updates the gold path against the predicted path.
The results are clear: for both sub and full models, training on both directions is better than training on one direction, and the greedy objective is better than the structured objective. The gap between the bidirectional greedy objective and others is larger in full than in sub, since full solves a larger TSP, where the greedy training is even more important for effective greedy decoding.
3.5 Transition-based Sorting
As discussed in §2.5, we use the transition system only for post-processing the linearized projective sentences, although the transition system itself is theoretically able to sort a random sequence. The question is whether the model is able to learn to handle the random input.
We experiment with different training and testing scenarios of the sorting models. They are trained in three scenarios, namely to sort (1) gold projective sentences into correct (potentially non-projective) sentences, noted as gold; (2) predicted projective sentences, where the sentences are obtained by 5-fold jackknifing on the training set using the sub model, noted as pred; and (3) random sequences, where the input is always shuffled during training, noted as rand. The models are then applied to sort (1) gold projective sentences (gold); (2) predicted projective sentences from the sub model (pred); and (3) random permutation of the tokens (rand). In the main experiment, the way we use the transition system corresponds to the gold-pred scenario.
Table 4 shows the BLEU scores on the development set averaged over all treebanks. We also show the change of BLEU scores from the input to the output  $(\Delta_{\mathrm{BLEU}})$  in different scenarios.
First, the gold model improves the input in all scenarios, especially the gold-pred scenario used in the main experiment brings 0.47 BLEU points improvement. Interestingly, the pred model from jackknifing does not improve the performance, while usually training on the data with erroneous prediction should prevent overfitting to the gold data. We conjecture the reason could be that the model is overfitting to fixing the particular errors in the predicted training data instead of learning to produce non-projective sentences.
Purely using the transition system for linearization (rand-rand) works to some extent, but performs lower than the baseline by a large gap for several reasons. First, it imposes an arbitrary order in the input which is a suboptimal way to represent a bag of word. Second, learning to sort random permutation requires a lot more training instances to generalize. Finally, it takes on average  $\mathcal{O}(n^2)$  steps, which also increases the chance of error propagation. In contrast, sorting a projective tree does not have any of these disadvantages.
Generally, when the training and testing scenarios are not aligned, the performance is always
beam	sub	full	swap
+tree	85.55	85.85	85.03	71.36
-tree	78.31	74.17	35.82	19.74
Δ	-7.24	-11.68	-49.21	-51.62
Table 5: Comparing the decoders with and without the Tree-LSTM encoder.
worse due to the mismatched bias of transitions. For example, gold-rand barely changes the random input since it mostly predicts shift, and rand-gold predicts swap too often such that the outcome is even worse than the input sentence.
3.6 Syntax Ablation
The success of the simple bigram language model and greedy TSP decoding relies heavily on the TreeLSTM encoding. To demonstrate its importance, we remove the tree encoding for each linearizer, i.e., they only receive the token level features as the representation. We experiment with four linearizers: apart from beam, sub and full as in the main experiments, we also include the swap linearizer that is trained to sort random input sequences. The condition  $+tree$  is the default case, while in -tree we do not use the tree encoding. Note that in the latter case, beam and sub still use the tree information to split the tree into subtrees, while full and swap do not use the tree information in any way.
The results are shown in Table 5. Without the tree encoder, the performance drop in sub is larger than beam, which suggests that sub is more dependent on the good representation of the TreeLSTM encoder, since its scoring function is essentially a bigram language model, which would be much less expressive than the LSTM in beam if syntax is absent. This result draws an interesting analogy to the fact that first-order graph-based dependency parsers (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2016) also outperform the transition-based counterparts with a simpler scoring model but without error propagation.
The much larger drop in full and swap emphasizes the importance of the inductive bias introduced by the divide-and-conquer strategy, since natural languages are predominantly projective.
Generally, the syntax ablation experiment highlights the crucial difference between our work and the original idea by Knight (1999), namely we use contextualized bigrams in our TSP model, which
is much more expressive than the vanilla version. Consider the subtree with the words "this" and "with" in Figure 1, a vanilla bigram model would calculate a much higher score for "with this" than "this with", while a contextualized bigram model could be aware that it is part of a rather special syntactic construction in English.
3.7 Feature Ablation
To understand how much each feature contributes to the linearization task, we perform ablation experiments on the selection of features. In the default setting of our models, we use the lemma, UPOS, dependency relation, and morphological tags to encode each token. We experiment with turning off each feature for the sub linearizer, as well as only using one feature, and the results are in Table 6. The results suggest that the UPOS tags and morphological tags do not provide much additional information and could be dropped if simplicity is desired. In contrast, the lemmata and dependency relations are crucial to determine the word order, since the performance drops considerably without them.
none	lemma	dep	upos	morph
without	85.85	83.12	82.49	85.56	85.58
with only	-	79.51	81.11	76.14	79.31
Table 6: Feature ablation experiments, where we test removing one feature (first row) and using only one feature (second row).
3.8 Performance vs. Speed
By default, we use a greedy TSP solver, which already yields satisfactory performance. We then make additional experiments with a more optimized metaheuristic (guided local search) to see if better performance can be gained in exchange for more decoding time. With the guided local search, we set the search limit to 1 second or 100 solutions for each subtree, and 10 seconds or 1000 solutions for the full tree. We also compare to the beam search linearizer with varying beam sizes from 1 to 64. The results are shown in Figure 5, where the decoding time is measured on a single CPU core.
Generally, all greedy TSP solvers outperform the Pareto front of the beam search decoders. The greedy solver performs almost as well as the optimized solver for the subtree TSP (85.85 vs. 85.91), while it performs clearly worse for the full tree
Figure 5: The speed (on a log scale) and BLEU score for different decoders, where the dots are projective decoders, and the crosses are non-projective decoders.
TSP (85.03 vs. 85.85). This contrast again demonstrates that the divide-and-conquer strategy indeed greatly simplifies the problem for the greedy solver. Post-processing with the swap system only slightly increases the decoding time (in total 50ms per sentence), but considerably improves the performance.
In this paper, we revisit the idea of treating word ordering as a TSP, but unlike the common bag-of-words scenario, the words have an underlying syntactic structure. We demonstrate that with the Tree-LSTM encoder, the biaffine scoring model, the divide-and-conquer strategy, and a transition-based sorting system, we can linearize a dependency tree with high speed and quality and without the projectivity restriction. We show with various ablation experiments that all of the components are crucial for the success of the TSP-based linearizer.
Our work emphasizes the importance of syntax in the word ordering task. We discussed many connections and similarities between linearization and parsing. We believe that quite generally, systems for solving one task can benefit from the other task's view on syntactic structure. One possibility to capitalize on these synergies is to explore data augmentation methods to select beneficial extra training data in an unsupervised fashion.
This work was in part supported by funding from the Ministry of Science, Research and the Arts of the State of Baden-Württemberg (MWK), within the CLARIN-D research project.
David Applegate, Ribert Bixby, Vasek Chvatal, and William Cook. 2006. Concorde TSP Solver.
Anja Belz, Mike White, Dominic Espinosa, Eric Kow, Deirdre Hogan, and Amanda Stent. 2011. The First Surface Realisation Shared Task: Overview and Evaluation Results. In Proceedings of the 13th European Workshop on Natural Language Generation, pages 217-226, Nancy, France. Association for Computational Linguistics.
Bernd Bohnet, Anders Björkelund, Jonas Kuhn, Wolfgang Seeker, and Sina Zarriess. 2012. Generating Non-Projective Word Order in Statistical Linearization. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 928–939, Jeju Island, Korea. Association for Computational Linguistics.
Bernd Bohnet, Leo Wanner, Simon Mille, and Alicia Burga. 2010. Broad Coverage Multilingual Deep Sentence Generation with a Stochastic Multi-Level Realizer. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 98-106. Association for Computational Linguistics.
Timothy Dozat and Christopher D. Manning. 2016. Deep Biaffine Attention for Neural Dependency Parsing. ArXiv, abs/1611.01734.
Ulrich Germann, Michael Jahr, Kevin Knight, Daniel Marcu, and Kenji Yamada. 2001. Fast Decoding and Optimal Decoding for Machine Translation. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 228-235, Toulouse, France. Association for Computational Linguistics.
Matic Horvat and William Byrne. 2014. A Graph-Based Approach to String Regeneration. In Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 85-95, Gothenburg, Sweden. Association for Computational Linguistics.
Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations. Transactions of the Association for Computational Linguistics, 4:313-327.
Kevin Knight. 1999. Decoding Complexity in Word-Replacement Translation Models. Computational Linguistics, 25(4):607-615.
Ryan McDonald, Fernando Pereira, Kiril Rbarov, and Jan Hajic. 2005. Non-Projective Dependency Parsing using Spanning Tree Algorithms. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 523-530, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
Simon Mille, Anja Belz, Bernd Bohnet, Yvette Graham, Emily Pitler, and Leo Wanner. 2018. The First Multilingual Surface Realisation Shared Task (SR'18): Overview and Evaluation Results. In Proceedings of the First Workshop on Multilingual Surface Realisation, pages 1-12, Melbourne, Australia. Association for Computational Linguistics.
Simon Mille, Anja Belz, Bernd Bohnet, Yvette Graham, and Leo Wanner. 2019. The Second Multilingual Surface Realisation Shared Task (SR'19): Overview and Evaluation Results. In Proceedings of the 2nd Workshop on Multilingual Surface Realisation (MSR 2019), pages 1-17, Hong Kong, China. Association for Computational Linguistics.
Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, and Pengcheng Yin. 2017. DyNet: The Dynamic Neural Network Toolkit. arXiv preprint arXiv:1701.03980.
Joakim Nivre. 2009. Non-Projective Dependency Parsing in Expected Linear Time. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 351-359, Suntec, Singapore. Association for Computational Linguistics.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A Multilingual Treebank Collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1659-1666, Portoroz, Slovenia. European Language Resources Association (ELRA).
Joakim Nivre and Jens Nilsson. 2005. Pseudo-Projective Dependency Parsing. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 99-106, Ann Arbor, Michigan. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Ratish Puduppully, Yue Zhang, and Manish Shrivastava. 2016. Transition-Based Syntactic Linearization with Lookahead Features. In Proceedings of
the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 488-493, San Diego, California. Association for Computational Linguistics.
Ben Taskar, Vassil Chatalbashev, Daphne Koller, and Carlos Guestrin. 2005. Learning structured prediction models: A large margin approach. In Proceedings of the 22nd international conference on Machine learning, pages 896-903. ACM.
Christoph Tillmann and Hermann Ney. 2003. Word Reordering and a Dynamic Programming Beam Search Algorithm for Statistical Machine Translation. Computational Linguistics, 29(1):97-133.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. 2015. Order Matters: Sequence to Sequence for Sets. arXiv preprint arXiv:1511.06391.
Christos Voudouris and Edward Tsang. 1999. Guided Local Search and its Application to the Traveling Salesman Problem. European journal of operational research, 113(2):469-499.
Xiang Yu, Agnieszka Falenska, Marina Haid, Ngoc Thang Vu, and Jonas Kuhn. 2019a. IM-SurReal: IMS at the Surface Realization Shared Task 2019. In Proceedings of the 2nd Workshop on Multilingual Surface Realisation (MSR 2019), pages 50-58, Hong Kong, China. Association for Computational Linguistics.
Xiang Yu, Agnieszka Falenska, Ngoc Thang Vu, and Jonas Kuhn. 2019b. Head-First Linearization with Tree-Structured Representation. In Proceedings of the 12th International Conference on Natural Language Generation, pages 279–289, Tokyo, Japan. Association for Computational Linguistics.
Mikhail Zaslavskiy, Marc Dymetman, and Nicola Cancedda. 2009. Phrase-Based Statistical Machine Translation as a Traveling Salesman Problem. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 333-341, Suntec, Singapore. Association for Computational Linguistics.
A Model Details and Hyperparameters
Our system is a modification on Yu et al. (2019a), where the encoder architecture and hyperparameters are identical to theirs, and we only change the decoder. The system is implemented with the DyNet library (Neubig et al., 2017). Training and testing are conducted on a single CPU core, where the average training time is under 1 hour and the average decoding speed is 50ms per sentence by the proposed model (Tree-LSTM encoder + subtree TSP decoder + swap post-processing).
Hyperparameter	Value
lemma dim	64
UPOS dim	32
morphological feature dim	32
dependency label dim	32
all other hidden dims	128
all LSTM layers	1
beam size	32
avg. token feature params	1.0 × 106
Tree-LSTM params	7.4 × 105
beam decoder params	1.6 × 106
TSP decoder params	6.6 × 104
swap decoder params	6.4 × 105
batch size	1
dropout	none
max training step	1 × 106
optimizer	Adam
Adam α	0.001
Adam β1	0.9
Adam β2	0.999