SlowGuess/NAACL_2021_OCR huggingface dataset:
MRP 2020: Text-to-graph-notation transducer. In Proceedings of the CoNLL 2020 Shared Task: Cross-Framework Meaning Representation Parsing, pages 40-52, Online. Association for Computational Linguistics.
Janaki Sheth, Youngsuk Lee, Ramón Fernández Astudillo, Tahira Naseem, Radu Florian, Salim Roukos, and Todd Ward. 2021. Bootstrapping multilingual AMR with contextual word alignments. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL).
Linfeng Song, Daniel Gildea, Yue Zhang, Zhiguo Wang, and Jinsong Su. 2019. Semantic neural machine translation using AMR. Transactions of the Association for Computational Linguistics, 7:19-31.
Elior Sulem, Omri Abend, and Ari Rappoport. 2018. Semantic structural evaluation for text simplification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 685-696, New Orleans, Louisiana. Association for Computational Linguistics.
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2021. Multilingual translation with extensible multilingual pretraining and finetuning. ArXiv, abs/2008.00401.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 2214-2218, Istanbul, Turkey. European Language Resources Association (ELRA).
Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT - building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479-480, Lisboa, Portugal. European Association for Machine Translation.
Rik van Noord and Johan Bos. 2017. Neural semantic parsing by character-based translation: Experiments with abstract meaning representations. Computational Linguistics in the Netherlands Journal, 7:93-108.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998-6008. Curran Associates, Inc.
Thomas Wolf, Julien Chaumont, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, Remi Louf, Patrick von Platen, Tim Rault,
Yacine Jernite, Teven Le Scao, Sylvain Gugger, Julien Plu, Clara Ma, Canwei Shen, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Dongqin Xu, Junhui Li, Muhua Zhu, Min Zhang, and Guodong Zhou. 2020a. Improving AMR parsing with sequence-to-sequence pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2501-2511, Online. Association for Computational Linguistics.
Jin Xu, Yinuo Guo, and Junfeng Hu. 2020b. Incorporate semantic structures into machine translation evaluation via UCCA. In Proceedings of the Fifth Conference on Machine Translation, pages 934-939, Online. Association for Computational Linguistics.
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019a. AMR parsing as sequence-to-graph transduction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 80-94, Florence, Italy. Association for Computational Linguistics.
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019b. Broad-coverage semantic parsing as transduction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3786-3798, Hong Kong, China. Association for Computational Linguistics.
Qiji Zhou, Yue Zhang, Donghong Ji, and Hao Tang. 2020. AMR parsing with latent structural information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4306-4319, Online. Association for Computational Linguistics.
A Model and Training Details
Cross For Cross, we leverage the implementation available in OpenNMT-py $^{20}$  (Klein et al., 2017) and define a Transformer model closely following Vaswani et al. (2017), except for the embedding modification we described. We use 128 as the embedding size and 512 as the Transformer hidden size when training in single-task settings; when scaling to the multilingual framework, we augment the hidden size to 768 so as to increase the model capacity. The number of layers in both the encoder and the decoder is set to 6, while the number of attention heads to 8; therefore,  $Cross_{st}$  contains
$^{20}$ https://github.com/OpenNMT/OpenNMT-py
$46M$  trainable parameters, while  $Cross_{mt}$ $105M$ . We optimize the models parameters using Adam (Kingma and Ba, 2015) and the original scheduler of Vaswani et al. (2017). We train with an effective batch size of 8000 tokens and for a maximum of  $300k$  steps on a NVIDIA GeForce RTX 2080 Ti, using semantic parsing accuracy as the early stopping criterion with  $25k$  steps of patience; training lasted 1 day for  $Cross_{st}$  and roughly 4 days for  $Cross_{mt}$ . We did not perform any significant tuning of decoding time parameters: we use 5 beams and, following Ge et al. (2019), we set the GNMT length penalty parameter alpha to 1.0.
MBart We use the mbart-large-cc25 model provided by the Huggingface's transformers library (Wolf et al., 2020) and follow the specifications given in the original paper for the training configuration: we use 2500 warm-up steps,  $3e - 5$  maximum learning rate, Adam as the optimizer and 0.3 dropout. We used 8000 effective token batch size and carried out training on a cloud-provided NVIDIA GeForce 3090, using semantic parsing accuracy as the early stopping criterion with  $25k$  steps of patience; training lasted slightly less than 2 days for  $mBART_{st}$  and around 4 days for  $mBART_{mt}$ . As we fine-tune the original model, the number of parameters is unchanged (but for the two additional vectors for UCCA and AMR in the embedding matrix), with both  $mBART_{st}$  and  $mBART_{mt}$  featuring  $610M$  trainable parameters.
B Training Task Scheduler
When training  $Cross_{mt}$  and  $mBART_{mt}$ , we oversample semantic parsing instances by sampling from the concatenation of AMR and UCCA with probability  $p_1 = 0.15$  and  $p_1 = 0.4$  respectively; the remaining probability  $p_2$  is uniformly divided among the MT corpora. When considering  $Cross_{mt}^{ft}$  and  $mBART_{mt}^{ft}$ , we bias the schedule towards the semantic formalism under consideration by bringing its probability to  $p_3 = 0.8$ ; we assign the remaining  $p_4 = 0.2$  uniformly to the MT corpora.
C Filtering over Parallel Corpora
We perform a mild filtering over parallel sentences, enforcing reasonable minimum, maximum and relative lengths; in particular, we discard sentences:
- that are shorter than 25 characters;
- that are longer than 1000 characters;
- where the relative character ratio between source and target is bigger than 3.0.
We then take the first  $5M$  out of these. This process results in  $5M$  parallel sentences for all four translation paths except EN  $\rightarrow$  IT, where, owing to the smaller size of Europarl, it results in only  $1.6M$  sentence pairs.