SlowGuess/NAACL_2022_OCR huggingface dataset:
Models	LM	DE	ES	IT	ZH
Translate and Parse Pipelines
Uhrig et al. (2021)	67.6	72.3	70.7	59.1
WLT+Structured-BART+MBSE-G silver1	BART	73.9	76.5	76.1	63.7
WLT+Structured-BART+MBSE-A silver1+2+3	BART	74.6	77.1	76.7	64.0
Cross-lingual Parsers
Blloshmi et al. (2020)	53.0	58.0	58.1	43.1
Sheth et al. (2021) (85K silver AMR)	XLMR	62.7	67.9	67.4	-
Procopio et al. (2021) (5M parallel corpus)	mBATmt	69.8	72.4	72.3	58.0
Cai et al. (2021b)	64.0	67.3	65.4	53.7
Xu et al. (2021)	70.5	71.8	70.8	-
Cai et al. (2021a) (320K silver AMR)	mBATmmt	73.1	75.9	75.4	61.9
Ours below (with Structured-mBART)
Structured-mBART-baseline	mBART	69.9±0.0	74.4±0.3	73.3±0.2	59.9±0.0
Ensemble-4 distillation (APT + Structured-BART + SPRING1 + SPRING2)
+ MBSE-G silver1	mBART	72.5±0.1	76.5±0.2	75.4±0.0	62.2±0.1
+ MBSE-G silver1+AMR2Text	mBART	72.9±0.1	76.6±0.0	75.6±0.0	62.3±0.0
+ MBSE-G silver1+AMR2Text + ens. dec.	mBART	73.2	76.9	75.7	62.7
Ensemble-5 distillation (APT + Structured-BART + SPRING1 + SPRING2 + AMRBART)
+ MBSE-A silver1+2+3	mBART	73.5±0.1	77.1±0.2	76.0±0.1	62.7±0.1
+ MBSE-A silver1+2+3 + ens. dec.	mBART	73.7	77.0	76.1	63.0
Table 4: Cross-lingual parser Smatch scores on AMR2.0 human translated test sets.  $\mathsf{mBART}_{mt}$  of Procopio et al. (2021) indicates the mBART model fine-tuned on both semantic parsing tasks and the MT data. mBARTmmt of Cai et al. (2021a) indicates an NMT model by (Tang et al., 2020), trained from mBART covering 50 languages. Shortnames: MBSE-G (greedy-selection), MBSE-A (average-Smatch) 'ens. dec.', ensemble decoding.
parsers are trained with Structured-mBART, always using separate vocabulary (sep-voc). Input sentences of the English training data are machine translated into the target languages with WLT<sup>13</sup> to generate cross-lingual parser training data.
Table 4 shows the results on the human translated AMR2.0 test set, following standard practices. We provide results for recently published cross-lingual AMR parsers and different silver training versions of Structured-mBART. Structured-mBART with 4 model ensemble (Ensemble-4) distillation with just silver<sup>1</sup> improves the Smatch score by 2.1 to 2.6 over the Structured-mBART baselines, outperforming very strong previous SoTA from (Cai et al., 2021a) on Chinese and Spanish and tied on Italian. Increasing the input sentence diversity via AMR-to-text generation and ensemble decoding further improve the system performances, attaining new cross-lingual SoTA on all four languages. Increasing the silver training data size to silver<sup>1+2+3</sup> and using 5 model ensemble (Ensemble-5) push the numbers higher by 0.2-0.5 Smatch points.
(Uhrig et al., 2021) report that translate-and-parse pipelines outperform the conventional cross
lingual parsers, we thus include translate-and-parse from the combination of WLT and Structured-BART + MBSE distillation. This out-performs the cross-lingual parsers by 0.6-1.0 Smatch on all languages except for Spanish, when trained with the same MBSE avg.-Smatch silver $^{1+2+3}$  data.
Comparing the fine-grained F1 scores for cross-lingual parsers with those for English, as shown in Table 5, we observe that cross-lingual parsers are particularly worse than English for negation. For instance, German negations are often realized as a compound, as in nichtarifäre (non - tariff), which is aligned to the non-negated stem portion of the concept tariff, losing its negation meaning. We observe similar issues in English with prefixed negations such as unhappy, inadequate, atypical.
5.3 Domain Adaptation
We use the AMR2.0 version of BioAMR (medical domain) as this has clearly defined partitions $^{14}$  and was used in Bevilacqua et al. (2021). We also use QALD-9-AMR, constructed from QALD-9 data $^{15}$  (Usbeck et al., 2018), a corpus of natural language
$^{14}$ amr.isi.edu/download/2016-03-14/amr-release-training-bio.txt, amr-release-dev-bio.txt, amr-release-test-bio.txt
15 https://github.com/ag-sc/QALD
<sup>13</sup>https://www.ibm.com/cloud/watson-language-Translator
Languages	Smatch	Unlabeled	NoWSD	Concepts	NER	Neg.	Wiki	Reentrant	SRL
EN-mono	85.9	89.0	86.3	92	93	75	81	78	85
DE-cross	73.7	77.9	73.8	75	89	48	79	61	68
ES-cross	77.1	81.4	77.5	81	89	62	79	67	74
IT-cross	76.1	80.4	76.3	79	90	56	78	65	73
ZH-cross	63.0	67.9	63.1	65	85	35	70	51	58
DE-pipeline	74.6	78.9	74.8	75	91	51	80	62	68
ES-pipeline	77.1	81.1	77.3	80	91	61	79	66	74
IT-pipeline	76.7	80.9	76.9	79	91	58	80	65	73
ZH-pipeline	64.0	68.9	64.0	66	86	40	74	51	59
questions for executable semantic parsing (Kapani-pathi et al., 2021). Corpus statistics of the domain adaptation data is summarized in Table 1.
Table 6 shows the experimental results. Results for SPRING are taken from Bevilacqua et al. (2021). For each test set, we report the results under three different training conditions, all of which include either AMR2.0 or AMR3.0 treebank in the training data: (1) use only silver data with MBSE distillation, (2) use only domain gold sentences, (3) use both silver data and domain gold sentences. Since BioAMR is annotated in AMR2.0 style and QALD-9-AMR in AMR3.0 style, we use the corresponding Structured-BART models as indicated in the table.
As for BioAMR data, MBSE distillation (with both graphene-Smatch and greedy-select) on silver $^{b}$  - comprising PubMed (LDC2008T20, LDC2008T21), BioNLP-ST-2011 and CRAFT - improves over the Structured-BART baseline by 6.5 Smatch point (60.4 vs. 66.9). However, adding just 201 domain gold sentences to AMR2.0 treebank results in 11.9 Smatch point improvement over the baseline (60.4 vs. 72.3). A close inspection shows that this is largely due to the NER score improvement, as shown in the column under NER, i.e. NER score 27.0 in Structured-BART (AMR2.0) vs. 68.0 after adding 201 domain gold sentences. The dramatic impact of NE coverage no longer holds when we double the domain gold sentences from 201 to 403. In fact, MBSE greedy-select silver $^{b}$  + 201 domain gold sentences (75.8) is more effective than doubling the domain gold sentences (74.3). Finally, by combining MBSE distillation on silver $^{b}$  with 5K domain gold sentences, the system achieves 81.3 Smatch, outperforming the previous
SoTA by 1.4.
Regarding QALD-9-AMR data, MBSE distillation on silver $^{q}$ , i.e. SQuAD-Q(uestion) sentences,
Table 5: Fine-grained F1 scores on the AMR2.0 test set for EN (English), DE (German), ES (Spanish), IT (Italian) and ZH (Chinese). EN-mono denotes English mono-lingual parser, {DE,ES,IT,ZH}-cross, cross-lingual parsers and {DE,ES,IT,ZH}-pipeline, translate-and-parse pipeline.
Training Data	Smatch	NER
BioAMR Evaluations
SPRINGDFS	59.7
SPRINGDFS+ silver	59.5
SPRINGDFS (In domain)	79.9
Ours
Struct-BART (AMR2.0)	60.4	27.0
+MBSE-G silver1	63.2	31.0
+MBSE-G silverb	66.9±0.2	30.0
+MBSE-P silverb	66.9±0.2	31.0
+201 domain gold sent.	72.3±0.2	68.0
+403 domain gold sent.	74.3±0.2	70.0
+5K domain gold sent.	79.8±0.2	80.0
+MBSE-G silv.b+201 gold	75.8±0.3	70.0
+MBSE-G silv.b+5K gold	81.3±0.2	81.0
QALD-9-AMR Evaluations
Ours
Struct-BART (AMR3.0)	87.2	84.0
+MBSE-G silver1	88.0	88.0
+MBSE-G silverq	89.5±0.1	85.0
+MBSE-P silverq	89.3±0.2	87.0
+200 domain gold sent.	88.5±0.5	84.0
+408 domain gold sent.	89.8±0.1	86.0
+MBSE-G silv.q+200 gold	90.0±0.3	87.0
+MBSE-G silv.q+408 gold	90.1±0.1	87.0
Table 6: Smatch scores on BioAMR and QALD-9 test sets with varying sizes of human annotated (gold) domain sentences and silver data. MBSE-G (greedy-select) and MBSE-P (Graphene-Smatch respectively). MBSE distillations are all with Ensemble-4 (APT + Structured-BART + SPRING $_1$  + SPRING $_2$ ).
Models	BioAMR	Q9AMR
Structured-BART	8.7%	0.7%
+MBSE silver	3.7%	0.7%
+200 domain gold sents	0.6%	0.7%
Table 7: Named entity (NE) type out-of-vocabulary ratio w.r.t the target vocabulary of various models. BioAMR and QALD-9 test sets include 1691 and 150 occurrences of named entities, respectively.
is almost as effective as 408 domain gold sentences (89.8) for both graphene-Smatch (89.3) and greedy-select (89.5) algorithms. Combining 408 domain gold sentences with MBSE greedy-select silver $^q$  adds less than 1 Smatch point to 90.1.
Since MBSE distillation on silver $^{b}$  lags behind the performance of 201 human annotated AMR for BioAMR, mostly due to low NER scores, we further analyze the target vocabulary coverage of named entity (NE) types occurring in the test sets. The analysis is shown in Table 7. NE types are equally well covered in all models for Q9AMR (QALD-9-AMR).  $0.7\%$  out-of-vocabulary (OOV) ratio is caused by a typo in human annotation of the test set, i.e. country misspelled as country. For BioAMR, however, NE type OOV ratio of MBSE silver model is  $3.7\%$ , e.g. protein-segment, macromolecular-complex, substantially higher than  $0.6\%$  of the model trained with 201 domain gold sentences. When the NE type is OOV, there is no chance for the system to produce the missing NE type, let alone predicting it correctly, underscoring the challenges posed by domain specific concepts unavailable elsewhere.
There have been numerous works applying ensemble/knowledge distillation (Hinton et al., 2015) to machine translation (Kim and Rush, 2016; Freitag et al., 2017; Nguyen et al., 2020; Wang et al., 2020, 2021), dependency parsing (Kuncoro et al., 2016) and question answering (Mun et al., 2018; Ze et al., 2020; You et al., 2021; Chen et al., 2012). Regarding ensembling AMR graphs, Barzdins and Gosko (2016) propose choosing the AMR with highest average sentence Smatch to all other AMRs. Hoang et al. (2021) proposed a more complex technique capable of building new AMRs by exploiting Smatch's hill climbing algorithm. Our work brings together ensemble distillation and Smatch-based ensembling and shows that it can provide substan
tial gains over the standard self-training.
Damonte and Cohen (2018) show that it may be possible to use the original AMR annotations devised for English as representations of equivalent sentences in other languages. Damonte and Cohen (2018); Sheth et al. (2021) propose annotation projection of English AMR graphs to target languages to train cross-lingual parsers, using word alignments. Błoshmi et al. (2020) show that one may not need alignment-based parsers for cross-lingual AMR, and model concept identification as a seq2seq problem. Procopio et al. (2021) reframe semantic parsing as multilingual machine translation (MNMT) and propose a seq2seq architecture finetuned on pretrained-mBART with an MNMT objective. Cai et al. (2021b) propose to use bilingual input to enable a model to predict more accurate AMR concepts. Xu et al. (2021) propose a cross-lingual pretraining approach via multitask learning for AMR parsing. Cai et al. (2021a) propose to use noisy knowledge distillation for multilingual AMR parsing. We introduce Structured-mBART and attain new SoTA in Chinese, German, Italian and Spanish cross-lingual parsing by applying MBSE distillation and AMR-to-text.
We subsume domain adaptation under data augmentation with MBSE distillation, where the only difference between the two lies in the properties of the unlabeled data. The unlabeled data is drawn from the target domain for the purpose of domain adaptation rather than those similar to the source training data for data augmentation in general.
We proposed a technique called Maximum Bayes Smatch Ensemble (MBSE) distillation, which brings together Smatch-based model ensembling Barzdins and Gosko (2016); Hoang et al. (2021) and ensemble distillation Hinton et al. (2015) of heterogeneous parsers, to significantly improve AMR parsing. The technique generalizes well across various tasks and is highly effective, leading to a new single system SoTA in English and cross-lingual AMR parsing and achieving the performance comparable to human annotated training data in domain adaptation of QALD-9-AMR corpus. Remaining technical challenges include tokenization and alignment of an input token corresponding to more than one concept for AMR parsing and identification of unknown named entities and their types for domain adaptation.
Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022. Graph pre-training for AMR parsing and generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
Guntis Barzdins and Didzis Gosko. 2016. Riga at semeval-2016 task 8: Impact of smatch extensions and character-level neural machine translation on amr parsing accuracy. In Proceedings of SemEval-2016, pages 1143-1147.
Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021. One spring to rule them both: Symmetric amr semantic parsing and generation without a complex pipeline. In AAAI Technical Track on Speech and Natural Language Processing I, Vol. 35 No. 14, pages 12564-12573.
Rexhina Blloshmi, Rocco Tripodi, and Roberto Navigli. 2020. XL-AMR: Enabling cross-lingual AMR parsing with transfer learning techniques. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2487-2500, Online. Association for Computational Linguistics.
Deng Cai and Wai Lam. 2020. Amr parsing via graph sequence iterative inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1290-1301.
Deng Cai, Xin Li, Jackie Chun-Sing Ho, Lidong Bing, and Wai Lam. 2021a. Multilingual amr parsing with noisy knowledge distillation. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 2778-2789.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748-752, Sofia, Bulgaria. Association for Computational Linguistics.
Yitao Cai, Zhe Lin, and Xiaojun Wan. 2021b. Making better use of bilingual information for cross-lingual amr parsing. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 1537–1547.
Cen Chen, Chengyu Wang, Minghui Qiu, Dehong Gao, Linbo Jin, and Wang Li. 2012. Cross-domain knowledge distillation for retrieval-based question answering systems. In Proceedings of the World Wide Web Conference 2021.
K. B. Cohen, K. Verspoor, K. Fort, C. Funk, M. Bada, M. Palmer, and L. E. Hunter. 2017. The colorado richly annotated full text (craft) corpus: multi-model annotation in the biomedical domain. *Handbook of Linguistic Annotation*, pages 1379-1394.
Marco Damonte and Shay B. Cohen. 2018. Crosslingual Abstract Meaning Representation parsing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1146-1155, New Orleans, Louisiana. Association for Computational Linguistics.
Ramon Fernandez Astudillo, Miguel Ballesteros, Tahira Naseem, Austin Blodgett, and Radu Florian. 2020. Transition-based parsing with stack-transformers. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1001-1007.
Markus Freitag, Yaser Al-Onaizan, and Baskaran Sankaran. 2017. Ensemble distillation for neural machine translation. In arXiv:1702.01802.
Vaibhava Goel and William J Byrne. 2000. Minimum bayes-risk automatic speech recognition. Computer Speech & Language, 14(2):115-135.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Thanh Lam Hoang, Gabriele Picco, Yufang Hou, Young-Suk Lee, Lam M. Nguyen, Dzung T. Phan, Vanessa López, and Ramon Fernandez Astudillo. 2021. Ensemble graph prediction for amr parsing. arXiv preprint arXiv:2110.09131.
Pavan Kapanipathi, Ibrahim Abdelaziz, Srinivas Ravishankar, Salim Roukos, Alexander Gray, Ramón Fernández Astudillo, Maria Chang, Cristina Cornelio, Saswati Dana, Achille Fokoue, Dinesh Garg, Alfio Gliozzo, Sairam Gurajada, Hima Karanam, Naweed Khan, Dinesh Khandelwal, Young-Suk Lee, Yunyao Li, Francois Luus, Ndivhuwo Makondo, Nandana Mihindukulasooriya, Tahira Naseem, Sumit Neelam, Lucian Popa, Revanth Gangi Reddy, Ryan Riegel, Gaetano Rossiello, Udit Sharma, G P Shrivatsa Bhargav, and Mo Yu. 2021. Leveraging Abstract Meaning Representation for knowledge base question answering. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3884-3894, Online. Association for Computational Linguistics.
Jin-Dong Kim, Sampo Pyysalo, Tomoko Ohta, Robert Bossy, Ngan Nguyen, and Junichi Tsujii. 2011. Overview of bionlp shared task 2011. In Proceedings of BioNLP Shared Task 2011 Workshop, pages 1-6.
Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327.
Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: Sequence-to-sequence models for parsing and generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 146-157, Vancouver, Canada. Association for Computational Linguistics.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Long Papers), pages 66-75.
Shankar Kumar and William Byrne. 2004. Minimum bayes-risk decoding for statistical machine translation. Technical report, JOHNS HOPKINS UNIV BALTimore MD CENTER FOR LANGUAGE AND SPEECH PROCESSING (CLSP).
Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, and Noah A. Smith. 2016. Distilling an ensemble of greedy dependency parsers into one MST parser. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1744-1753.
Young-Suk Lee, Ramon Fernandez Astudillo, Tahira Naseem, Revanth Gangi Reddy, Radu Florian, and Salim Roukos. 2020. Pushing the limits of amr parsing with self-learning. In *Findings of the EMNLP* 2020, pages 3208-3214.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. In arXiv:2001.08210.
Chunchuan Lyu, Shay B Cohen, and Ivan Titov. 2020. A differentiable relaxation of graph segmentation and alignment for amr parsing. arXiv preprint arXiv:2010.12676.
Manuel Mager, Ramón Fernandez Astudillo, Tahira Naseem, Md Arafat Sultan, Young-Suk Lee, Radu Florian, and Salim Roukos. 2020. GPT-too: A language-model-first approach for AMR-to-text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1846-1852, Online. Association for Computational Linguistics.
Jonghwan Mun, Kimin Lee, Jinwoo Shin, and Bohyung Han. 2018. Learning to specialize with knowledge distillation for visual question answering. In 32nd
Conference on Neural Information Processing Systems (NeurIPS 2018).
Tahira Naseem, Abhishek Shah, Hui Wan, Radu Florian, Salim Roukos, and Miguel Ballesteros. 2019. Rewarding smatch: Transition-based amr parsing with reinforcement learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4586-4592.
Xuan-Phi Nguyen, Shafiq Joty, Wu Kui, and Ai Ti Aw. 2020. Data diversification: A simple strategy for neural machine translation. In 34th Conference on Neural Information Processing Systems (NeurIPS 2020).
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. Fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, pages 48-53.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311-318.
Luigi Procopio, Rocco Tripodi, and Roberto Navigli. 2021. Sgl: Speaking the graph languages of semantic parsing via multilingual translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 325-337.
Leonardo FR Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2021. Investigating pretrained language models for graph-to-text generation. In Proceedings of the Third Workshop on Natural Language Processing for Conversational AI, pages 211-227.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725.
Janaki Sheth, Young-Suk Lee, Ramon Fernandez Astudillo, Tahira Naseem, Radu Florian, Salim Roukos, and Todd Ward. 2021. Bootstrapping multilingual amr with contextual word alignments. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 394-404.
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.00401.
Sarah Uhrig, Yoalli Rezepka García, Juri Opitz, and Anette Frank. 2021. Translate, then parse! a strong baseline for cross-lingual amr parsing. In Proceedings of the 17th International Conference on Parsing Technologies (IWPT 2021), pages 58–64.