SlowGuess/NAACL_2021_OCR huggingface dataset:
12We use the hanziconv library (https://github. com/berniey/hanziconv).
13https://github.com/snowblink14/smatch
$^{14}$ https://github.com/mdtux89/amr-evaluation
15https://github.com/cfmrp/mtool
does indeed result in underfitting for AMR and that further training is beneficial; this fine-tuned alternative achieves 79.5 Smatch score, less than one point behind Xu et al. (2020a). Considering the similarity between the two approaches, this difference is likely caused by the increased number of tasks our model is asked to handle.
Once we replace Cross with  $mBART$ , all performances rise significantly. In particular, even  $mBART_{st}$ , a single-task variant with no additional data, outperforms all its alternatives except for SPRING and SPRING $_{bart}$  (Bevilacqua et al., 2021), highlighting the potential of fully pre-trained Transformer language models for translation-based approaches.  $mBART_{mt}$  and  $mBART_{mt}^{ft}$  push performances further up, showing that the MT data are beneficial even in this pretrained setting and that the multi-task training set, which enables a single shared model to scale across formalisms and languages, is not detrimental to English AMR parsing.
However, arguably more interesting is the comparison between the performances of  $mbART$  models and SPRING, which, in contrast, builds upon the English-only  $BART$  (Lewis et al., 2020). In particular, as  $\mathrm{SPRING}_{bart}$  outperforms even  $mbART_{mt}^{ft}$ , this finding suggests that, as expected,  $BART$  is more suitable than  $mbART$  when dealing with English AMR. However, as we show in §5.2, our choice is beneficial for cross-lingual AMR parsing and results in an interesting trade-off.
Finally, we also evaluate SGL on AMR-3.0 and report the results of  $Cross_{mt}^{ft}$ ,  $mBART_{st}$  and  $mBART_{mt}^{ft}$  when trained on this dataset (Figure 1 bottom). Overall, we witness a similar trend compared to AMR-2.0.
We now show the performances of SGL on crosslingual AMR parsing in terms of Smatch score over Chinese (ZH), German (DE), Italian (IT) and Spanish (ES). For comparison, we report the results of the systems proposed by Damonte and Cohen (2018, AMREAGER), Blloshmi et al. (2020, XL-AMR) and Sheth et al. (2021); along with their best systems, we also show the strongest MT baseline reported in Damonte and Cohen (2018, AMREAGERMT) and the zero-shot configuration explored in Blloshmi et al. (2020, XL-AMR).
Table 2 (top) shows a very interesting trend. First of all,  $Cross_{mt}^{ft}$  achieves competitive performances,
Model	DE	ES	IT	ZH
HT	AMREAGER	39.0	42.0	43.0	35.0
AMREAGERMT	57.0	60.0	58.0	50.0
XL-AMR∅	32.7	39.1	37.1	25.9
XL-AMR	53.0	58.0	58.1	41.5
Sheth et al. (2021)	62.7	67.9	67.4	-
Crossmt	60.8	62.9	63.2	51.8
Crossftmt	61.8	63.7	64.1	52.6
mBARTst	54.8	60.4	63.6	47.8
mBARTmt	66.3	69.0	69.8	55.4
mBARTftmt	65.8	69.2	69.6	54.8
mBARTftmt + AP	69.8	72.4	72.3	58.0
MT	Sheth et al. (2021)	66.9	69.6	71.0	-
mBARTftmt + AP	73.3	73.9	73.4	64.9
Table 2: Smatch scores on cross-lingual AMR parsing for both human (top, HT) and machine (bottom, MT) translations of the test set.
falling short only when compared to the recent work of Sheth et al. (2021); in particular, it surpasses the strong AMREAGER  $MT$  baseline. The most interesting aspect of this result is that  $Cross_{mt}^{ft}$  attains these performances without ever seeing at training time any  $\mathrm{X} \rightarrow \mathrm{AMR}$  translation path; this is in marked contrast with all previous literature and with the systems we report in Table 2. This finding clearly highlights the effectiveness of transfer learning and, by extension, of our proposed framework in this setting.
Secondly, the performances  $mBART_{st}$  achieve are astounding under multiple perspectives. First, to the best of our knowledge, it is the first reported result of AMR systems achieving competitive performances on cross-lingual AMR parsing in a fully zero-shot configuration:  $mBART_{st}$  is fine-tuned solely on EN  $\rightarrow$  AMR and then applied directly to X  $\rightarrow$  AMR translation; especially when compared to XL-AMR  $\emptyset$ , the only similar approach we are aware of, the gap is significant. Second, among the languages we consider, the case of Chinese is especially interesting as it appears to require constrained decoding in order to work properly: in particular, we restrict the model to generate only subwords whose characters belong to the English alphabet.[16] If we were to perform ZH  $\rightarrow$  AMR parsing with no additional decoding machinery, as for the other languages, performances would be significantly lower, with  $mBART_{st}$  attaining only 31.9. This performance drop is caused by
16The reported results on Chinese of all mBART models have been computed using this form of decoding.
the model leaving some nodes of the graph untranslated, i.e. named entities left written in Chinese (奥巴马 rather than Obama), which disrupts the auto-regressive nature of the decoding procedure and, besides, eventually results in a penalized Smatch score. Finally, despite the larger amount of pre-training  $mBART$  has been exposed to, its bigger capacity and better Smatch score on English,  $mBART_{st}$  still falls short when compared to  $Cross_{mt}^{ft}$ , highlighting the benefits of seeing related translation directions at training time.
$mBART_{mt}$  pushes the bar further up, with performances on German, Spanish and Italian that are now only roughly 10 points behind their English counterparts. As  $mBART_{mt}$  significantly outperforms  $mBART_{st}$ , this result shows that, despite the massive pretraining, parallel data are still beneficial for cross-lingual AMR. Moreover, differently from English AMR,  $mBART_{mt}^{ft}$  does not yield an improvement and, in fact, performances slightly drop on average.
While the scores  $mBART_{mt}$  attains are already unprecedented, it is natural to wonder whether annotation projection (AP) might yield a further beneficial effect. To this end, similarly to Blloshmi et al. (2020), we translate the input sentences of AMR-2.0 into the four languages under consideration[17] and build a training set for each language by pairing the translated sentence with the original AMR graph. We further fine-tune  $mBART_{mt}^{ft}$ , including also these new datasets among the training data. This model, which we denote by  $mBART_{mt}^{ft} + \mathrm{AP}$ , surpasses further  $mBART_{mt}$ , clearly underlining the beneficial effect of this technique.
Finally, following Sheth et al. (2021), we also report the results of SGL when evaluated on the machine-translated test set, $^{18}$  similarly to their findings, we observe that, as the mismatch between the training and test set is reduced, our parser performs better in this setting than on the human-translated one.
5.3 UCCA Parsing
We report in Table 3 the performance of SGL on UCCA parsing. We compare our approach with the original multi-task baseline (Oepen et al., 2019) and 3 transition-based parsers that participated; in
Model	Type	Score
Oepen et al. (2019)	multi-task	41.0
Hershcovich and Arviv (2019)	single-task	82.1
Hershcovich and Arviv (2019)	multi-task	73.1
Che et al. (2019)	multi-task	82.6
Crossst	single-task	55.7
Crossmt	multi-task	72.0
Crossft	multi-task	75.1
mBARTst	single-task	77.0
mBARTmt	multi-task	79.9
mBARTft	multi-task	76.9
Table 3: UCCA results on The Little Prince.
particular, we report the score of Che et al. (2019), the system that ranked first in both all-framework and UCCA parsing.
First of all, we note the result of  $Cross_{st}$ ; while its performance is far below the score Che et al. (2019) achieve, it still outperforms the original proposed baseline by more than 10 points. Furthermore, to the best of our knowledge, apart from the recent works proposed in the latest shared task of Oepen et al. (2020), this is the first reported result of translation-based approaches on UCCA parsing.
Once plugged into our multilingual framework, UCCA benefits from transfer learning to an even greater extent than AMR parsing, likely owing to the smaller amount of training data:  $Cross_{mt}$  and especially  $Cross_{mt}^{ft}$  significantly reduce the gap between SGL and Che et al. (2019), with  $Cross_{mt}^{ft}$  outperforming the multi-task transition-based approach of Hershcovich and Arviv (2019). The usage of  $mBART$  pushes up the system's performance further, with  $mBART_{st}$  achieving 77.0 and  $mBART_{mt}$  79.9; differently from AMR,  $mBART_{mt}^{ft}$  suffers from overfitting and its performance is actually lower than that of  $mBART_{mt}$ . Even though these scores are lower than those of Che et al. (2019), we argue that such results are still incredibly promising as they demonstrate the effectiveness of SGL in tackling cross-framework semantic parsing. Indeed, these results show that multilingual translation-based approaches allow for a single model to jointly accommodate different formalisms, each potentially linearized according to a different linearization scheme. Furthermore, we believe there is a significant margin for improvement on both the linearization used and the model; for instance, we did not consider node ids such as  $<root\_0>$  as special tokens, but instead had the unigram tokenizer handle them as if they were normal
<sup>17</sup>We use the MarianMT models (Tiedemann and Thottin-gal, 2020) available in the HuggingFace Transformers library (Wolf et al., 2020).
18We use the same MT models we utilized for annotation projection.
Model	EN	DE	AMR ES	IT	ZH	UCCA EN
Crossst	70.3	-	-	-	-	55.7
Crossmt	78.1	60.8	62.9	63.2	51.8	72.0
CrossNmt	74.5	22.0	22.0	29.4	12.8	68.7
Table 4: Results on AMR, cross-lingual AMR and UCCA parsing of  $Cross_{st}$ ,  $Cross_{mt}$  and  $Cross_{mt}^{N}$ .
Finally, we wish to point out that direct comparability between our system and those reported is hindered by the fact that our training setting is significantly different from theirs; in particular, we limit ourselves to two frameworks only and leverage resources (the parallel corpora from MT) whose usage was forbidden to the shared task participants.[19] Nevertheless, we believe that their results are needed here to better contextualize the performances SGL obtains.
6 Analysis: is MT the one helping?
Although the performances of  $Cross_{mt}$  are remarkable,  $mBART_{st}$  achieves competitive results on cross-lingual parsing and fares even better on English parsing. While  $mBART_{st}$  admittedly features a massive amount of pre-training, this pre-training is over monolingual corpora and, as such, the model has never seen any parallel data. We therefore wonder to what extent the parallel nature of the additional MT data we use is crucial for  $Cross_{mt}$ .
To answer this question, we treat our MT corpora as monolingual data by sampling, for each instance, either the source or target side and converting the translation task into a denoising one: given an instance EN  $\rightarrow$  IT, we sample either EN or IT with equal probability, denoting the result by  $Z$ , and convert the instance into  $g(Z)\to Z$ , where  $g$  is a noisig function that corrupts the input text. We follow Lewis et al. (2020) and choose a noisig function that masks  $35\%$  of the words by random sampling a span length from a Poisson distribution  $(\lambda = 3.5)$ . Applying this noisification scheme to the MT data, we train a model identical to  $Cross_{mt}$  and denote it by  $Cross_{mt}^{N}$ .
As shown in Table 4, in this data regime, the parallel nature is crucial both for English and, especially, for cross-lingual parsing. While  $Cross_{mt}^{N}$  does yield a significant boost over  $Cross_{st}$ , when
compared instead to  $Cross_{mt}$ , it is 4 points behind on UCCA parsing and only half way on AMR parsing. The difference is even more marked in the cross-lingual setting, where  $Cross_{mt}^{N}$  simply does not work.
In this work, we presented SGL, a novel framing of semantic parsing towards multiple formalisms as Multilingual Neural Machine Translation. That is to say, given a sentence and the desired output formalism, a many-to-many neural model has to learn to translate from the input sentence to the corresponding linearized graph. Within this framework, we show that we can address the paucity of annotated data that afflicts semantic parsing effectively by performing the learning procedure jointly on large parallel corpora coming from MT, and leveraging the power of pre-trained Transformer language models.
Using AMR and UCCA as our cases in point, we report competitive performances on their parsing, especially once pre-trained models enter the picture. Furthermore, we find that the benefit MT data provide goes beyond merely improving English-centric parsing, yielding astonishing performances on cross-lingual AMR parsing as well, and allowing SGL to outperform all existing approaches by a large margin. Most interestingly, differently from all previous literature, this result is attained without ever explicitly seeing at training time the translation paths the model is tested upon. Once we use annotation projection and include these data as well, performances rise even further, attaining unprecedented results.
As future work, thanks to the nimbleness with which we can add new languages, we plan to assess the scalability of this framework as more formalisms are taken into account.
This work was partially supported by the MIUR under the grant "Dipartimento di eccellenza 2018-2022" of the Department of Computer Science of Sapienza University.
19Allowed resources are specified at: http://svn.nlpl.eu/mrp/2019/public/resources.txt
Omri Abend and Ari Rappoport. 2013. Universal Conceptual Cognitive Annotation (UCCA). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 228-238, Sofia, Bulgaria. Association for Computational Linguistics.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021. One SPRING to rule them both: Symmetric AMR semantic parsing and generation without a complex pipeline. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence.
Rexhina Blloshmi, Rocco Tripodi, and Roberto Navigli. 2020. XL-AMR: Enabling cross-lingual AMR parsing with transfer learning techniques. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2487-2500, Online. Association for Computational Linguistics.
Claire Bonial, Lucia Donatelli, Mitchell Abrams, Stephanie M. Lukin, Stephen Tratz, Matthew Marge, Ron Artstein, David Traum, and Clare Voss. 2020. Dialogue-AMR: Abstract Meaning Representation for dialogue. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 684-695, Marseille, France. European Language Resources Association.
Deng Cai and Wai Lam. 2020. AMR parsing via graph- sequence iterative inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1290-1301, Online. Association for Computational Linguistics.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748-752, Sofia, Bulgaria. Association for Computational Linguistics.
Wanxiang Che, Longxu Dou, Yang Xu, Yuxuan Wang, Yijia Liu, and Ting Liu. 2019. HIT-SCIR at MRP 2019: A unified pipeline for meaning representation parsing via efficient training and effective encoding. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pages 76–85, Hong Kong. Association for Computational Linguistics.
Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N. Mendes. 2013. Improving efficiency and accuracy in multilingual entity extraction. In Proceedings of the 9th International Conference on Semantic Systems (I-Semantics), page 121-124. Association for Computing Machinery.
Marco Damonte and Shay Cohen. 2020. Abstract Meaning Representation 2.0 - Four Translations LDC2020T07. Web Download, Philadelphia: Linguistic Data Consortium.
Marco Damonte and Shay B. Cohen. 2018. Crosslingual Abstract Meaning Representation parsing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1146-1155, New Orleans, Louisiana. Association for Computational Linguistics.
Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for Abstract Meaning Representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 536-546, Valencia, Spain. Association for Computational Linguistics.
Miquel Esplà, Mikel Forcada, Gema Ramírez-Sánchez, and Hieu Hoang. 2019. ParaCrawl: Web-scale parallel corpora for the languages of the EU. In Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks, pages 118-119, Dublin, Ireland. European Association for Machine Translation.
DongLai Ge, Junhui Li, Muhua Zhu, and Shoushan Li. 2019. Modeling source syntax and semantics for neural AMR parsing. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 4975-4981.
Michael Wayne Goodman. 2020. Penman: An open-source library and tool for AMR graphs. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 312-319, Online. Association for Computational Linguistics.
Jan Hajic, Eva Hajicova, Jarmila Panevova, Petr Sgall, Ondrej Bojar, Silvie Cinkova, Eva Fučíková, Marie Mikulová, Petr Pajas, Jan Popelka, Jiří Semecký, Jana Šindlerová, Jan Štepanek, Josef Toman, Zdenka Urešová, and Zdeněk Žabokrtský. 2012. Announcing Prague Czech-English Dependency Treebank 2.0. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 3153-3160, Istanbul, Turkey. European Language Resources Association (ELRA).
Hardy Hardy and Andreas Vlachos. 2018. Guided neural language generation for abstractive summarization using Abstract Meaning Representation. In Pro
ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 768-773, Brussels, Belgium. Association for Computational Linguistics.
Daniel Hershcovich, Omri Abend, and Ari Rappoport. 2018. Multitask parsing across semantic representations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 373-385, Melbourne, Australia. Association for Computational Linguistics.
Daniel Hershcovich, Zohar Aizenbud, Leshem Choshen, Elior Sulem, Ari Rappoport, and Omri Abend. 2019. SemEval-2019 task 1: Cross-lingual semantic parsing with UCCA. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 1-10, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Daniel Hershcovich and Ofir Ariviv. 2019. TUPA at MRP 2019: A multi-task baseline system. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pages 28-39, Hong Kong. Association for Computational Linguistics.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67-72, Vancouver, Canada. Association for Computational Linguistics.
Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: Sequence-to-sequence models for parsing and generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 146-157, Vancouver, Canada. Association for Computational Linguistics.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Chunchuan Lyu and Ivan Titov. 2018. AMR parsing as graph prediction with latent alignment. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 397-407, Melbourne, Australia. Association for Computational Linguistics.
Roberto Navigli. 2018. Natural Language Understanding: Instructions for (present and future) use. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI), pages 5697-5702.
Stephan Oepen, Omri Abend, Lasha Abzianidze, Johan Bos, Jan Hajic, Daniel Hershcovich, Bin Li, Tim O'Gorman, Nianwen Xue, and Daniel Zeman. 2020. MRP 2020: The second shared task on cross-framework and cross-lingual meaning representation parsing. In Proceedings of the CoNLL 2020 Shared Task: Cross-Framework Meaning Representation Parsing, pages 1-22, Online. Association for Computational Linguistics.
Stephan Oepen, Omri Abend, Jan Hajic, Daniel Herschovich, Marco Kuhlmann, Tim O'Gorman, Nianwen Xue, Jayeol Chun, Milan Straka, and Zdenka Uresova. 2019. MRP 2019: Cross-framework meaning representation parsing. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pages 1-27, Hong Kong. Association for Computational Linguistics.
Stephan Oepen and Jan Tore Lønning. 2006. Discriminant-based MRS banking. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06), Genoa, Italy. European Language Resources Association (ELRA).
Hiroaki Ozaki, Gaku Morio, Yuta Koreeda, Terufumi Morishita, and Toshinori Miyoshi. 2020. Hitachi at