SlowGuess/NAACL_2021_OCR huggingface dataset:
For all neural models the embedding dimension was 50 and the hidden dimension 300. The bidirectional LSTM used for representing the graph nodes has a single layer, and the recurrent step in the GCRN is  $2(K = 2)$ . We applied the Adam optimizer (Kingma and Ba, 2014). We use accuracy to measure the percentage of absolute orders which are predicted correctly and Kendall's  $\tau$  coefficient to measure the relationship between two lists of ordered items;  $\tau$  ranges from  $-1$  to 1, where  $-1$  means perfect inversion and 1 means perfect agreement.
Results Table 2 summarizes our results. SiGraph performs better than NoEdges  $(+14.83\%$  accuracy), showing that edge information is helpful for the representation of nodes which are used to order conditions. FullEdges performs worse than SiGraph  $(-13.68\%$  accuracy), underlying the fact that graph structure matters (i.e., edges are helpful when connecting certain pairs of nodes). BiGraph achieves the best ordering performance by a large margin compared to SiGraph  $(+9.63\%$  accuracy). One possible reason is that bidirectionality ensures all nodes have incoming edges, which can be used to update the node representations.
Models	Acc (%)	τ	Parameters
Counting	14.38	0.57	—
NoEdges	44.64	0.75	6.1M
FullEdges	45.79	0.75	6.1M
SiGraph	59.47	0.85	6.1M
BiGraph	69.10	0.89	6.1M
Table 2: Results for condition ordering (dev set).
Models	BLEU	Parameters
Seq	71.79	47.4M
ChildSum	72.98 (+1.19)	47.1M
Nary	73.24 (+1.45)	49.2M
Sibling	74.22 (+2.43)	49.2M
3.2 Ideal-World Generation
Models and Settings We first examine generation performance in an ideal setting where (gold standard) condition orders are given and the indices of variables are fixed.
We compared the proposed treeLSTM against Seq, a baseline sequence-to-sequence model which adopts a bidirectional LSTM as its encoder. Trees were linearized in a top-down and left-to-right fashion,  $X = [x_0, x_1, \ldots, x_{n-1}]$ , where  $n$  is the tree length. We obtained hidden representations  $H = [h_0, h_1, \ldots, h_{n-1}]$  of the input with:
$$
[ h _ {0}, h _ {1}, \dots , h _ {n - 1} ] = \operatorname {B i L S T M} ([ x _ {0}, x _ {1}, \dots , x _ {n} ])
$$
In addition, we included various models with tree-based encoders: ChildSum, is the bidirectional childsum-treeLSTM encoder of Tai et al. (2015); it operates over right-branch binarized trees; Nary, is the bidirectional Nary-TreeLSTM of Tai et al. (2015), again over right-branch binarized trees; and Sibling is our bidirectional sibling-TreeLSTM. All models were equipped with the same LSTM decoder, global attention (Bahdanau et al., 2015), and the copy strategy of See et al. (2017).
The embedding dimension was 300 and the hidden dimension 512. All encoders and decoders have 2 layers. The detailed settings are shown in
Table 3: Ideal-world generation (dev set); improvements compared to Seq shown in parentheses.
Models	BLEU
Seq+Naive	4.61
Seq+Random	24.34 (16.77)
Seq+Counting	45.17
Seq+GraphOrder	55.57
Sibling+Naive	6.98
Sibling+Random	43.43 (0.26)
Sibling+Counting	49.54
Sibling+GraphOrder	58.73
Table 4: Real-world generation (dev set). For Random, we report average results after shuffling 5 times (variance shown in parentheses).
Models	BLEU	Parameters
Graph	45.72	30.1M
Seq+GraphOrder	55.28	32.4M + 6.1M
Sibling+GraphOrder	59.26	34.5M + 6.1M
Table 5: Real-world generation (test set).
the Appendix. We measure generation quality with case-insensitive BLEU (Papineni et al., 2002).
Results Table 3 shows our results on the development dataset. Overall, treeLSTM models performs better (average  $+1.69$  BLEU) than sequence models. Nary performs better  $(+0.26$  BLEU) than ChildSum because the latter cannot model the order of children. Sibling performs best (74.22 BLEU), because it not only encodes the tree structure but also keeps track of sequential information.
3.3 Real-World Generation
Models and Settings We finally, present our results in a more realistic setting where both problems of condition ordering and variable naming must be addressed. We recover condition order using four approaches: a Naive method which has no special-purpose ordering mechanism; the order of conditions is random in the development/test sets and fixed in the training set; Random, the order of conditions is random in the training, development, and test sets; Counting, the order of conditions is recovered by the Counting method; GraphOrder recovers the order of conditions with BiGraph. All comparison systems employ variable renaming as introduced in Section 2.2. We report experiments with a sequence-to-sequence generator and our sibling-TreeLSTM.
4The length of the input tokens can be around 4,000.
<sup>5</sup>We experimented with  $n$ -ary ( $n > 2$ ) trees, but found that binary trees perform best. Right-branch binary trees are also empirically better than left-branch ones.
Figure 7: BLEU score against DRS size (test set).
Results Table 4 summarizes our results on the development set. Naive performs poorly, indicating that both Seq and Sibling models are sensitive to the order of conditions. Random, has higher variance with Seq (+16.51) compared to Sibling. Hidden representations for each timestep in Seq are heavily influenced by all previous steps, which are sequentially encoded; subtrees are encoded as a unit in Sibling, which is a more global representation for capturing patterns. Overall, we observe that the order of conditions plays a key role in the generation: both Seq and Sibling models improve when ordering of conditions is explicitly incorporated (either with Counting or GraphOrder). We observe that the combination of Sibling with GraphOrder achieves the best results (58.73 BLEU).
Table 5 presents our results on the test set. We compare our Sibling encoder against a sequential one. Both models are interfaced with GraphOrder. We also compare to a previous graph-to-text model (Song et al., 2018; Damonte and Cohen, 2019) which has been used for generating from AMRs. We converted DRSs to graphs following the method of Liu et al. (2020); graphs were encoded with a GCRN (Seo et al., 2018) and decoded with an LSTM. As can be seen, Sibling+GraphOrder outperforms all comparison systems achieving a BLEU of 59.26. However, compared to ideal-world generation (see Table 3) there is still considerable room for improvement.
Figure 7 shows model performance on test set against DRS size (i.e., the number of nodes in a DRS tree). Perhaps unsurprisingly, we see that generation quality deteriorates with bigger DRSs (i.e., with  $>1,600$  nodes).
While BLEU is frequently adopted as an automatic evaluation metric for generation tasks, it is
somewhat problematic in our case as it merely calculates word overlap between generated and gold-standard text without assessing whether model output is faithful to the semantics of the input (i.e., the DRS meaning representations). To this effect, we present examples of text generated by our model, demonstrating how the DRS input constrains and affects the output text.
Figure 8 shows examples of text generation from the test set. In the first example, the model generates the word because from the rhetorical relation, BECAUSE  $(b_{10}, b_{12})$ . Temporal information (highlighted in blue in the figure) is also accurately reflected in the generated text (sell is inflected to its present tense form). In addition, the model tends to over-generate (e.g., the word dollar is mentioned twice) and sometimes misses out on important determiners (e.g., some). In the second example, the model generates the word themselves referring to the entities mentioned before, e.g.,  $x_{29}$  equals to  $x_{27}$  which refers to inmates, resolving the coreference. In the third example, the model generates the modal verb must in accordance with the scope operator NEC (a shorthand for Necessity,  $\square$ ). Also, the model generates all for food and goods corresponding to the Implication (IMP) condition (i.e.,  $\forall x(P(x) \rightarrow Q(x)))$ .
Much previous work has focused on text generation from formal representations of meaning focusing exclusively on isolated sentences or queries. The literature offers a collection of approaches to generating from AMRs most of which employ neural models and structured encoders (Song et al., 2018; Beck et al., 2018; Damonte and Cohen, 2019; Ribeiro et al., 2019; Zhu et al., 2019; Cai and Lam, 2020; Wang et al., 2020). Other work generates text from structured query language (SQL) adopting either sequence-to-sequence (Iyer et al., 2016) or graph-to-sequence models (Xu et al., 2018).
Basile (2015) was the first to attempt generation from DRT-based meaning representations. He proposes a pipeline system which operates over graphs and consists of three components: an alignment module learns the correspondence between surface text and DRS structure, an ordering module determines the relative position of words and phrases in the surface form and a realizer generates the final text. Narayan and Gardent (2014) simplify complex sentences with a two-stage model which
DRS	... DRS( b10 Pred( b10 x24 “speculator.n.01”) Pred( b10 x25 “dollar.n.01”) Pred( b10 e8 “sell.v.01”) Agent( b10 e8 x24 ) Theme( b10 e8 x25 ) Pred( b4 t1 “now.r.01”) Equ( b10 X26 t1 ) temp includes( b10 t5 x26 ) temp_overlap( b10 e8 t5 ) ) SDRS( b12 DRS( b11 Pred( b3 x5 “thing.n.12”) Pred( b11 e9 “expect.v.01”) Agent( b11 e9 x5 ) ... BECAUSE( b10 b12 ))
Gold	... some speculators are selling dollars because they expect ...
Ours	... , the dollar . speculators are selling dollars because they expect ...
DRS	... Pred( b16 e11 “be.v.00”) Agent( b16 e11 x26 ) Ref( b16 x27 ) Card( b16 x27 7 ) Ref( b16 x28 ) Pred( b16 x27 “innate n.01”) Ref( b16 x29 ) Equ( b16 x27 x29 ) Ref( b17 x30 ) Pred( b17 x30 “group.n.01”) Ref( b16 e12 ) Pred( b16 e12 “disguise.v.01”) Theme( b16 e12 x29 ) ...
Gold	... were among seven inmates who disguised themselves ...
Ours	... were among seven inmates disguised themselves ...
DRS	... DRS( b4 NEC( b4 DRS( b5 IMP( b5 DRS( b6 Ref( b6 x11 ) Ref( b6 x10 ) subset_of( b6 x11 x10 ) Ref( b6 x12 ) subset_of( b6 x12 x10 ) Ref( b6 x13 ) Pred( b6 x13 “food.n.01” ) In( b6 x11 x13 ) Pred( b6 x11 “goods.n.01”) Ref( b6 s2 ) Topic( b6 s2 x12 ) Pred( b6 s2 “manufactured.a.01”) Pred( b6 x12 goods n.01 ) ) DRS( b7 Ref( b7 e4 ) Pred( b7 e4 “import.v.01”) Theme( b7 e4 x10 ) Pred( b3 t1 “now.r.01”) Ref( b7 t3 ) temp_included( b7 e4 t3 ) temp_before( b7 t1 t3 )) )IMP ) )NEC ) ...
Gold	... all food and manufactured goods must be imported ...
Ours	... all food and manufactured goods must be imported ...
Figure 8: DRS example from test set with gold and automatically generated text by ours (Sibling+GraphOrder). Temporal information marked in blue, rhetorical relations marked in red, co-reference marked as green, and scope marked in brown.
first performs sentence splitting and deletion operations over DRSs and then uses a phrase-based machine translation model for surface realization.
Our work is closest to Basile (2015); we share the same goal of generating from DRSs, however, our model is trained end-to-end and can perform long-form generation for documents and sentences alike. We also adopt an ordering component, but we order DRS conditions rather than lexical items, and propose a model capable of inferring a global order. There has been long-standing interest in information ordering within NLP (Lapata, 2003; Abend et al., 2015; Chen et al., 2016; Gong et al., 2016; Logeswaran et al., 2018; Cui et al., 2018; Yin et al., 2019; Honovich et al., 2020). Our innovation lies in conceptualizing ordering as a graph scoring task which can be further realized with graph neural network models (Wu et al., 2020).
In this paper, we have focused on document-level generation from formal meaning representations. We have adopted DRT as our formalism of choice and highlighted various challenges associated with
the generation task. We have introduced a novel sibling treeLSTM for encoding DRSs rendered as trees and shown it is particularly suited to trees with wide branches. We have experimentally demonstrated that our encoder coupled with a graph-based condition ordering model outperforms strong comparison systems. In the future, we would like to embed our generator in practical applications such as summarization and question answering.
Omri Abend, Shay B. Cohen, and Mark Steedman. 2015. Lexical event ordering with an edge-factored model. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech
nologies (NAACL-HLT), pages 1161-1171, Denver, Colorado, USA.
Nicholas Asher and Alex Lascarides. 2003. *Logics of conversation*. Cambridge University Press.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 4th International Conference on Learning Representations (ICLR), San Diego, California.
Valerio Basile. 2015. From Logic to Language: Natural Language Generation from Logical Forms. Ph.D. thesis, University of Groningen, Netherlands.
Valerio Basile and Johan Bos. 2011. Towards generating text from discourse representation structures. In Proceedings of the 13th European Workshop on Natural Language Generation, pages 145-150, Nancy, France.
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. 2018. Graph-to-sequence learning using gated graph neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 273-283, Melbourne, Australia.
Johan Bos, Valerio Basile, Kilian Evang, Noortje J Venhuizen, and Johannes Bjerva. 2017. The Groningen meaning bank. In Handbook of Linguistic Annotation, pages 463-496. Springer.
Deng Cai and Wai Lam. 2020. Graph transformer for graph-to-sequence learning. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI), pages 7464-7471, New York, USA.
John Carroll and Stephan Oepen. 2005. High-efficiency realization for a wide-coverage unification grammar. In Robert Dale and Kam-Fai Wong, editors, Proceedings of the 2nd International Joint Conference on Natural Language Processing (IJCNLP), volume 3651, pages 165-176. Springer, Jeju, Korea.
Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. 2016. Neural sentence ordering. arXiv preprint arXiv:1607.06952.
Stephen Clark and James Curran. 2007. Wide-coverage efficient statistical parsing with CCG and log-linear models. Computational Linguistics, 33(4):493-552.
Baiyun Cui, Yingming Li, Ming Chen, and Zhongfei Zhang. 2018. Deep attentive sentence ordering network. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4340-4349, Brussels, Belgium.
Marco Damonte and Shay B. Cohen. 2019. Structural neural encoders for AMR-to-text generation. In Proceedings of the 2019 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 3649-3658, Minneapolis, USA.
Federico Fancellu, Sorcha Gilroy, Adam Lopez, and Mirella Lapata. 2019. Semantic graph parsing with recurrent neural network DAG grammars. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2769-2778, Hong Kong, China.
Christiane Fellbaum, editor. 1998. WordNet: An Electronic Database. MIT Press, Cambridge, MA.
Jeffrey Flanigan, Chris Dyer, Noah A. Smith, and Jaime G. Carbonell. 2016. Generation from abstract meaning representation using tree transducers. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 731-739, San Diego, California, USA.
Jingjing Gong, Xinchi Chen, Xipeng Qiu, and Xu- anjing Huang. 2016. End-to-end neural sentence ordering using pointer network. arXiv preprint arXiv:1611.04953.
Valerie Hajduk, Jan Buys, Michael Wayne Goodman, and Emily M. Bender. 2019. Neural text generation from rich semantic representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2259-2266, Minneapolis, Minnesota.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Or Honovich, Lucas Torroba Hennigen, Omri Abend, and Shay B. Cohen. 2020. Machine reading of historical events. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7486-7497, Online. Association for Computational Linguistics.
Matic Horvat, Ann Copestake, and Bill Byrne. 2015. Hierarchical statistical semantic realization for Minimal Recursion Semantics. In Proceedings of the 11th International Conference on Computational Semantics (IWCS), pages 107-117, London, UK. Association for Computational Linguistics.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2073-2083, Berlin, Germany.
Hans Kamp. 1981. A theory of truth and semantic representation. In J. A. G. Groenendijk, T. M. V. Janssen, and M. B. J. Stokhof, editors, Formal Methods in the Study of Language, volume 1, pages 277-322. Mathematisch Centrum, Amsterdam.
Hans Kamp and Uwe Reyle. 1993. From Discourse to Logic; An Introduction to Modeltheoretic Semantics of Natural Language, Formal Logic and Discourse Representation Theory. Kluwer, Dordrecht.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), Banff, Canada.
Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: Sequence-to-sequence models for parsing and generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 146-157, Vancouver, Canada.
Mirella Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics (ACL), pages 545-552, Sapporo, Japan.
Jiangming Liu, Shay B. Cohen, and Mirella Lapata. 2018. Discourse representation structure parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 429-439, Melbourne, Australia.
Jiangming Liu, Shay B. Cohen, and Mirella Lapata. 2019a. Discourse representation parsing for sentences and documents. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 6248-6262, Florence, Italy.
Jiangming Liu, Shay B. Cohen, and Mirella Lapata. 2019b. Discourse representation structure parsing with recurrent neural networks and the transformer model. In Proceedings of the IWCS Shared Task on Semantic Parsing, Gothenburg, Sweden. Association for Computational Linguistics.
Jiangming Liu, Shay B. Cohen, and Mirella Lapata. 2020. Dscorer: A fast evaluation metric for discourse representation structure parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4547-4554.
Lajanugen Logeswaran, Honglak Lee, and Dragomir Radev. 2018. Sentence ordering and coherence modeling using recurrent neural networks. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pages 5285-5292, New Orleans, Louisiana, USA.
Wei Lu and Hwee Tou Ng. 2011. A probabilistic forest-to-string model for language generation from typed lambda calculus expressions. In Proceedings of the
2011 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1611-1622, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Shashi Narayan and Claire Gardent. 2014. Hybrid simplification using deep semantics and machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 435-445, Baltimore, Maryland. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL), pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Leonardo F.R. Ribeiro, Claire Gardent, and Iryna Gurevych. 2019. Enhancing AMR-to-text generation with dual graph representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3174-3185, Hong Kong, China.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1073-1083, Vancouver, Canada.
Youngjoo Seo, Michael Defferrard, Pierre Vandergheynst, and Xavier Bresson. 2018. Structured sequence modeling with graph convolutional recurrent networks. In Neural Information Processing Systems (NeurIPS), pages 362-373, Montreal, Canada. Springer.
Stuart M. Shieber, Gertjan van Noord, Fernando C. N. Pereira, and Robert C. Moore. 1990. Semantic-head-driven generation. Computational Linguistics, 16(1):30-42.
Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A graph-to-sequence model for AMR-to-text generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1616-1626, Melbourne, Australia.
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP), pages 1556-1566, Beijing, China.