Venue: NAACL 2022

# Maximum Bayes Smatch Ensemble Distillation for AMR Parsing

Young-Suk Lee+, Ramon Fernandez Astudillo+, Thanh Lam Hoang++,

###### Abstract

AMR parsing has experienced an unprecendented increase in performance in the last three years, due to a mixture of effects including architecture improvements and transfer learning. Self-learning techniques have also played a role in pushing performance forward. However, for most recent high performant parsers, the effect of self-learning and silver data augmentation seems to be fading. In this paper we propose to overcome this diminishing returns of silver data by combining Smatch-based ensembling techniques with ensemble distillation. In an extensive experimental setup, we push single model English parser performance to a new state-of-the-art, 85.9 (AMR2.0) and 84.3 (AMR3.0), and return to substantial gains from silver data augmentation. We also attain a new state-of-the-art for cross-lingual AMR parsing for Chinese, German, Italian and Spanish. Finally we explore the impact of the proposed technique on domain adaptation, and show that it can produce gains rivaling those of human annotated data for QALD-9 and achieve a new state-of-the-art for BioAMR.

## 1 Introduction

Adoption of the Transformer architecture Vaswani et al. (2017) for Abstract Meaning Representation (AMR) parsing Cai and Lam (2020); Fernandez Astudillo et al. (2020) as well as pretrained language models Bevilacqua et al. (2021); Zhou et al. (2021b); Bai et al. (2022) have enabled an improvement of above \(10\) Smatch points Cai and Knight (2013), the standard metric, in the last two years.

Data augmentation techniques have also shown great success in pushing the state-of-the-art of AMR parsing forward. These include generating silver AMR annotations with a trained parser Konstas et al. (2017); van Noord and Bos (2017), with multitask pre-training and fine-tuning Xu et al. (2020) as well as combining AMR to source text and silver AMR generation Lee et al. (2020) and stacked pre-training of silver data from different models - from low performance to high performance silver data Xia et al. (2021). However, the latest BART-based state-of-the-art parsers, have shown diminishing returns for data augmentation. Both SPRING Bevilacqua et al. (2021) and Structured-BART Zhou et al. (2021b) gain a mere \(0.5\)Smatch from self-learning, compared with over \(1\) point gains of the previous, less performant, models. Since performance scores are already above where inter annotator agreement (IAA) is assumed to be, i.e. 83 for newswire and 79 for web text reported in Banarescu et al. (2013), one possible explanation is that we are reaching some unavoidable performance plateau.

In this work we show that we can achieve significant performance gains close to 2 Smatch point with the newly proposed data augmentation technique, contrary to the results from the previous state-of-the-art systems. The main contributions of this paper are as follows:

* We propose to combine Smatch-based model ensembling Barzdins and Gosko (2016); Hoang et al. (2021) and ensemble distillation Hinton et al. (2015) of heterogeneous parsers to produce high quality silver data.
* We offer a Bayesian ensemble interpretation of this technique as alternative to views such as Minimum Bayes Risk decoding Goel and Byrne (2000) and name the technique Maximum Bayes Smatch Ensemble (MBSE).
* Applied to English monolingual parsing, MBSE distillation yields a new single system state-of-the-art (SoTA) on AMR2.0 (85.9) and AMR3.0 (84.3) test sets.
* Trained with Structured-mBART1, it yields new SoTA for Chinese (63.0), German (73.7),

Italian (76.1) and Spanish (77.1) cross-lingual parsing.
* Applied to domain adaptation, MBSE distillation achieves the performance comparable to human annotations of QALD-9 training data and achieves new SoTA on BioAMR test set.
* We release QALD-9-AMR treebank2 at, which comprises 408 training and 150 test sentences. Footnote 2: [https://github.com/IBM/AMR-annotations](https://github.com/IBM/AMR-annotations)

## 2 Maximum Bayes Smatch Ensemble

Ensemble distillation Hinton et al. (2015) integrates knowledge of different teacher models into a student model. For sequence to sequence models, e.g. machine translation, it is possible to ensemble models by combining probabilities of words given context at each time step Kim and Rush (2016); Freitag et al. (2017). Syntactic and semantic parsers model a distribution over graphs that is harder to integrate across teacher models in an optimal way. For particular cases like dependency parsing, it is possible to ensemble teachers based on the notion of edge attachment Kuncoro et al. (2016), which is related to the usual evaluation metric, Label Attachment Score (LAS). However, AMR graphs are quite complex and not explicitly aligned to words. The standard Smatch Cai and Knight (2013) metric approximates the NP-Complete problem of aligning nodes across graphs with a hill climbing algorithm. This illustrates the difficulty of achieving consensus across teachers for AMR ensembling.

Prior work ensembling AMR graphs has leveraged Smatch directly or its hill climbing strategy for ensembling. The ensemble in Barzdins and Gosko (2016) selects, among a number of candidate AMRs, the one that has the largest average Smatch with respect to all sampled AMRs. The ensemble in Hoang et al. (2021), modifies the candidate AMRs to increase consensus as measured by coverage. Then it selects from the union of original and modified graphs for the one with highest coverage or largest average Smatch. One possible intepretation of both techniques is that of Minimum Bayes Risk (MBR) decoding, a well established method in Automatic Speech Recognition (ASR) Goel and Byrne (2000) and Machine Translation (MT) Kumar and Byrne (2004). Assuming that we have a model predicting a graph from an input sentence \(p(g~{}|~{}w)\), normal decoding entails searching among model outputs \(g\) for the one that has the highest likelihood according to the model \(p(g~{}|~{}w)\). MBR searches instead for the model output that minimizes the risk with respect to the distribution of possible human (gold) outputs for a given input

Footnote 2: [https://github.com/IBM/AMR-annotations](https://github.com/IBM/AMR-annotations)

\[\hat{g}=\arg\min_{g}\{E_{p(g^{h}|w)}\{R(g,g^{h})\}\}\]

where \(p(g^{h}~{}|~{}w)\) is the distribution of correct human outputs, e.g. given by multiple annotators, and \(R\) is a risk function that measures how severe deviations from \(g^{h}\) are. In this case risk would be minus Smatch. Since in practice \(p(g^{h}~{}|~{}w)\) is not available, MBR takes often the strong assumption of replacing \(p(g^{h}~{}|~{}w)\) by the model distribution itself \(p(g~{}|~{}w)\).

Here we suggest another Bayesian interpretation, that requires less strong assumptions than MBR, a Bayesian model ensemble Wilson and Izmailov (2020). Indeed techniques above can be seen as solving

\[\hat{g}=\arg\max_{g\in\mathcal{G}}\{E_{p(\mathcal{M}|\mathcal{D})}\{\mathrm{ Smatch}(g,\tilde{g}_{\mathcal{M}})\}\}\]

where \(p(\mathcal{M}~{}|~{}\mathcal{D})\) is the distribution of models \(\mathcal{M}\) given training data \(\mathcal{D}\), approximated by a sample average of models of different architectures or different random seeds, and

is the output of a conventional decoding process for each parser prediction distribution \(p_{\mathcal{M}}\), including post-processing \(\mathrm{post}()\). This process differs across models indexed by \(\mathcal{M}\), for example \(y\) can be transition actions or linearized graphs and \(\mathrm{post}()\) running the state-machine or linearized graph post-processing3. \(\mathcal{G}\) is the space of candidate graphs, which in Barzdins and Gosko (2016) are the AMRs resulting from decoding each sample from \(p(\mathcal{M}~{}|~{}\mathcal{D})\) and in Hoang et al. (2021) are those same graphs plus the modified pivot graphs. There is in principle no restriction on how to build the set \(\mathcal{G}\). _Decoding_ a graph \(g\in\mathcal{G}\) means here selecting the member of that set maximizing the expected Smatch and is different from each parser's decoding process.

 

If we replace \({\rm Smatch}()\) by an indicator function on the decoding outputs \(1_{g=\hat{g}_{\mathcal{M}}}\), then

recovers majority voting of AMR graphs. Since the space of graphs is exponentially large on the input size, this would be too sparse to attain meaningful vote counts. The propagation of the uncertainty in through the transformation both solves the sparsity problem, and allows optimization on a space that is better related to the target metric. The method will be henceforth described here as Maximum Bayes Smatch Ensemble distillation (MBSE distillation).

In what follows, we will consider three versions for ensembling, the Smatch version of Hoang et al. (2021) (graphene-Smatch), the average-Smatch selection of Barzdins and Gosko (2016), and a greedy version of Barzdins and Gosko (2016) where we select the two highest Smatch AMRs and from that pair, keep the graph with the highest Smatch with respect to the remaining graphs (greedy-select). The greedy-select algorithm is given in Algorithm 1 of Appendix A and performs similarly to the average-Smatch of Barzdins and Gosko (2016).

## 3 Silver Training Strategy

We now describe the AMR silver training strategy proposed in this work. This strategy creates high quality English and cross-lingual AMR annotations for unlabeled data with MBSE and alternative input sentences of gold AMRs via AMR-to-text.

As depicted in Fig. 1, we start with 1) a set of gold-labeled (English sentence, AMR) pairs, 2) a set of unlabeled English sentences and 3) pre-trained English-to-foreign language Machine Translation systems. Assuming off-the-shelf AMR parsers, we train each of the \(N\) parsers using the gold data with their respective training procedure. More than one random seed may be trained for some parsers, leading to more than \(N\) AMR parses for each input sentence.

After the parsers have been trained, we use them to parse the unlabeled English text as in Konstas et al. (2017). Interpreting the set of trained models as samples of the model distribution, we apply the MBSE distillation methods described in Section. 2. We apply all variations of the MBSE algorithms including graphene-Smatch, greedy-select and average-Smatch algorithms.

For English parsers, the MBSE distilled AMR annotations are added to the human-annotated gold treebanks for enhanced model training. For cross-lingual parsers, we translate all English input sentences to the target foreign languages and train respective cross-lingual parsers with pairs of (Foreign language input sentences, AMR graphs in English), following Damonte and Cohen (2018).

Following Lee et al. (2020), we also apply an AMR-to-text model (Mager et al., 2020; Ribeiro et al., 2021; Bevilacqua et al., 2021) to generate additional sentences for human-annotated AMR. We filter out the generated texts if they are too similar (BLEU > 0.9) or too dissimilar (BLEU < 0.1) to the original input texts, as measured by BLEU Papineni et al. (2002). AMR-to-text generation4 is used for cross-lingual AMR parser training only.

Footnote 4: We use [https://github.com/SapienzaNLP/spring](https://github.com/SapienzaNLP/spring).

Figure 1: Data augmentation framework: Given a labeled example in English \((\mathbf{e}_{1},\mathbf{g}_{1})\), we use an AMR-to-text generation system to generate an alternative input text \(\hat{\mathbf{e}}_{2}\) for \(\mathbf{g}_{1}\) following (Lee et al., 2020). Given a sentence \(\mathbf{e}_{3}\), and various state-of-the-art off-the-shelf parser outputs (\(A\), \(B\), \(C\)), Maximum Bayes Smatch Ensemble (MBSE) produces a single annotation for each input sentence by selecting from existing AMRs or their modified versions. MBSE is only applied to unlabeled English sentences to produce \(\hat{\mathbf{g}}_{3}\). Following (Damonte and Cohen, 2018), we translate the English sentences to e.g. Spanish, to yield new training samples to train a Spanish cross-lingual parser. We use the English pairs \((\mathbf{e}_{1},\mathbf{g}_{1}),(\hat{\mathbf{e}}_{2},\mathbf{g}_{1}),( \mathbf{e}_{3},\hat{\mathbf{g}}_{3})\) to train an English parser.

 



## 4 Experimental Setup

### Corpus Statistics and QALD-9-AMR

Table 1 details the corpora considered for the standard benchmark experiments on AMR2.0 and AMR3.0 test sets (lef) and out-of-domain data used for domain adaptation experiments (right). Silver indicates the unlabeled data for silver AMR acquisition. SQuAD2.0-Q(uestions) are for QALD-9 (silver\({}^{q}\)) and PubMed, BioNLP-2011 Kim et al. (2011) and CRAFT Cohen et al. (2017) for BioAMR (silver\({}^{b}\)).

Since there were no human annotations of QALD-9 corpus, we created QALD-9-AMR treebank. QALD-9 training/test data have been annotated by 3 skilled resident human annotators with experience in AMR annotations over a year. Each of the annotators annotated both the train and test data sets, followed by cross validation by each other. The final annotations were adjudicated by the most experienced annotator. Inter-annotator agreement (IAA) rate on a subset of 158 training sentences is over 95% in Smatch. The data is made publicly available under an Apache2 license.

### Parsing Models

We use 4 off-the-shelf AMR parsers to parse unannotated raw texts. We train the parsers following their standard configurations.

**APT**Zhou et al. (2021)5 is a transition-based parser that combines hard attention over sentences with a target side action pointer mechanism to decouple source tokens from node representations and address alignments. Cross-attention of all decoder layers is used for action-source alignment.

Footnote 5: github.com/IBM/transition-amr-parser/tree/action-pointer, Apache2 License

**SPRING**Bevilacqua et al. (2021)6 fine-tunes BART Lewis et al. (2020) to predict linearized AMR graphs, avoiding complex pipelines.

Footnote 6: github.com/SapienzaNLP/spring, CC BY-NC-SA 4.0

**Structured-BART**Zhou et al. (2021)7 models the transition-based parser state within a pre-trained BART architecture, outperforming SPRING. This is the main parser for our work.

Footnote 7: github.com/IBM/transition-amr-parser, Apache2 License

**AMRBART**Bai et al. (2022)8 improves the structure awareness of pre-trained BART over AMR graphs by introducing node/edge denoising and sub-graph denoising tasks, for graph-to-graph pre-training, achieving significant improvement over previous BART-based systems.

Footnote 8: [https://github.com/muyeyby/AMRBART](https://github.com/muyeyby/AMRBART), MIT License

### Structured-mBART

For cross-lingual AMR parsing, we adapt Structured-BART by replacing the pretrained BART with mBART of Liu et al. (2020), henceforth Structured-mBART. The codebase is made publicly available under an Apache2 license. Structured-mBART diverges from Structured-BART mainly in input processing and vocabulary:

* For task vocabulary, Structured-mBART includes ~250K sentencepiece tokens of Kudo (2018) including 25 language tags, e.g. es_XX, whereas Structured-BART includes ~50K BPE tokens of Sennrich et al. (2016).
* We append the source language tag to the end

\begin{table}
\begin{tabular}{l|c c r||c|c|c c} \hline \multicolumn{6}{c}{**For Standard Experiments**} & \multicolumn{4}{c}{**For Domain Adaptation**} \\ \hline
**Dataset** & **Split** & **Sents** & **Tokens** & **Dataset** & **Split** & **Sents** & **Tokens** \\ \hline AMR2.0 & Train & 36,521 & 653K & QALD-9-AMR (_new_) & Train & 408 & 3,475 \\  & Test & 1,371 & 30K & & Test & 150 & 1,441 \\  & Dev. & 1,368 & 29K & & & & \\ \hline AMR3.0 & Train & 55,635 & 1M & Bio AMR & Train & 5,452 & 231K \\  & Test & 1,898 & 39K & & Test & 500 & 22K \\  & Dev. & 1,722 & 37K & & & & \\ \hline  & & & & LP & Test & 1,562 & 21K \\ \hline PropBank & silver\({}^{1}\) & 20K & 386K & SQuAD2.0-Q & silver\({}^{d}\) & 135K & 1.5M \\ SQuAD2.0-C & silver\({}^{1}\) & 70K & 2M & BioNLP-ST-2011 & silver\({}^{b}\) & 15K & 460K \\ Ontonotes5.0 & silver\({}^{2}\) & 59K & 1.1M & CRAFT & silver\({}^{b}\) & 27K & 740K \\ WikiText-103 & silver\({}^{3}\) & 70K & 2M & PubMed & silver\({}^{b}\) & 26K & 750K \\ \hline \end{tabular}
\end{table}
Table 1: Corpus statistics for the standard benchmark experiments on AMR2.0 and AMR3.0 test sets (left) and domain adaptation experiments (right). Silver indicates the unlabeled data for silver training.



of each input sentence without specifying the target language tag for Structured-mBART.
* For Structured-mBART, we set the learning rate to \(3{\rm e}{-5}\), cf. \(1{\rm e}{-4}\) of Structured-BART, and move the layer normalization to the beginning of each transformer block.

We obtain contextualized embeddings from the pre-trained mBART for multilingual input sentence representations. For target action sequences, we map the sentencepiece tokens to the corresponding target token, by averaging all values from the sentencepiece tokens corresponding to the target token. For German, Italian and Spanish input texts, we apply the tokenizer from JAMR parser9 before sentencepiece tokenization. For Chinese, we directly apply the sentencepiece tokenizer.

Footnote 9: [https://github.com/jflanigan/jamr](https://github.com/jflanigan/jamr)

## 5 Results

To explore the effect of the proposed MBSE distillation and training strategy, we consider an extensive experimental setup including standard English benchmarks (Section 5.1), cross-lingual benchmarks (Section 5.2) and out of domain data sets (Section 5.3).10 For model training and selection details, see Appendix B and Appendix C.

Footnote 10: We also applied the technique to APT, observing similar performance gains when using MBSE distillation.

We first provide the performance evaluation of each ensembling technique used in MBSE in Table 2 to demonstrate the effectiveness of the ensemble techniques by themselves. We test the algorithm on the standard test data sets from AMR2.0 and AMR3.0 and three out-of-domain data sets, Q9AMR (QALD-9-AMR), LP (Little Prince) and BioAMR in Table 1. We consider here only standard English AMR parsing. As expected, all MBSE algorithms, average-Smatch, graphene-Smatch and greedy-select, improve individual models by large margins. Note that while the ensembles outperform single model state-of-the-art by a large margin, the use of heterogeneous ensembles of models is computationally prohibitive in practice, both due to the cost of running different models but also the ensembling techniques.

### English AMR Parsing

As displayed in Table 1, we consider the standard AMR2.0 (LDC2017T10) and AMR3.0 (LDC2020TO2) treebank as gold data. For ensemble distillation, we use the data sets denoted by silver1 for comparison with previous work, and silver2 and silver3 to investigate the impact of unlabeled corpus size on model performance. For silver1, we use all sentence examples in PropBank (LDC2004T14). From SQuAD2.0-C(ontexts)11 we filter out the ~92K sentences, removing bad utf8 encoding (~7K) and ill-formed disconnected graphs produced by APT (~15K). Silver\({}^{2}\) comprises Ontontotes5.0 (LDC2013T19) and silver3 WikiText-10312 Footnote 11: [https://raipurkar.github.io/SQuAD-explorer/](https://raipurkar.github.io/SQuAD-explorer/)

Footnote 12: [https://www.salesforce.com/products/einstein/ai-research/the-wikitext-dependency-language-modeling-dataset/](https://www.salesforce.com/products/einstein/ai-research/the-wikitext-dependency-language-modeling-dataset/)

The results are shown in Table 3. The lower part of the table (denoted by **Ours**) compares the performances of Structured-BART in various silver data augmentation setups including our proposed MBSE distillation. With the same unlabeled corpus silver1, greedy-select distillation improves 1.0 Smatch point on AMR2.0 (84.2 vs. 85.2) and 1.5 Smatch point on AMR3.0 (82.0 vs. 83.5) over the

\begin{table}
\begin{tabular}{l|c|c|c|c|c} Models & AMR2.0 & AMR3.0 & Q9AMR & LP & BioAMR \\ \hline APT Zhou et al. (2021a) & 83.0 & 81.1 & 83.7 & 79.0 & 55.2 \\ Structured-BART Zhou et al. (2021b) & 84.6 & 83.1 & 87.7 & 81.0 & 62.4 \\ SPRING\({}_{1}\) Bevilacqua et al. (2021) & 84.2 & 83.2 & 87.7 & 81.3 & 61.6 \\ SPRING\({}_{2}\) Bevilacqua et al. (2021) & 83.8 & 82.9 & 86.4 & 81.0 & 60.5 \\ AMRBART Bai et al. (2022) & 85.4 & 84.2 & 88.0 & 82.3 & 63.4 \\ \hline aver.-Smatch A Barzdins and Gosko (2016) & 86.2 & 84.9 & 89.0 & 82.9 & 64.1 \\ graphene-Smatch P Hoang et al. (2021) & **86.7** & **85.4** & **89.3** & **83.1** & **65.8** \\ greedy-select G) & 85.9 & 84.8 & 88.8 & 82.8 & 63.9 \\ \end{tabular}
\end{table}
Table 2: English parsing performance in Smatch in general domain and domain adaptation for recent state-of-the-art systems (top). Performance in Smatch for the ensemble of all systems using different Stmatch-based ensembling techniques (bottom). SPRING\({}_{1}\) and SPRING\({}_{2}\) are \(2\) random seeds of the same model. Highest scores are **boldfaced**.

 

Structured-BART baselines. Graphene-Smatch distillation performs similarly to greedy-select one.

To isolate the effect of ensembling, we provide two additional baselines: 1) silver obtained from SPRING, which is expected to have complementary information to self-trained silver, and 2) an equal mixture of SPRING and Structured-BART (random 50:50), which tests if the MBSE selection strategy bears any effect. MBSE distillation outperforms these two baselines by between \(0.2\) and \(0.5\) Smatch point, depending on the scenario, proving that MBSE selection has a clear positive effect.

We also investigate the impact of unlabeled corpus size on model performance by adding silver\({}^{2}\) and silver\({}^{3}\) to silver\({}^{1}\), i.e. silver\({}^{1+2}\) and silver\({}^{1+2+3}\). We observe additional 0.3-0.4 improvement, complementary to the one obtainable with conventional ensemble decoding. This pushes the numbers to \(85.7\) and \(84.2\), setting a new SoTA for single system with 4 model ensemble (Ensemble-4) distillation. Note that using 5 model ensemble (Ensemble-5) distillation moves the Smatch scores even higher to 85.9 for AMR2.0 and 84.3 for AMR3.0.

### Cross-lingual AMR Parsing

For cross-lingual AMR parsing, we consider the well known cross-lingual extension of AMR2.0 Damonte and Cohen [2018]. Our cross-lingual

\begin{table}
\begin{tabular}{l|c|c|c c} \hline
* \\ Zhang et al. (2019a) & & 76.3\(\pm\)0.1 &
- \\ Zhang et al. (2019b) & & 77.0\(\pm\)0.1 &
- \\ Cai and Lam (2020) & & 80.2 &
- \\ Fernandez Astudillo et al. (2020) & & 80.2\(\pm\)0.0 &
- \\ Lyu et al. (2020) & &
- & 75.8 \\ Lee et al. (2020) & 85K & 81.3\(\pm\)0.0 &
- \\ Xu et al. (2020) & 14M & 81.4 &
- \\ Bevilacqua et al. (2021) & 200K & 84.5 & 83.0 \\ Zhou et al. (2021a) & 70K & 82.6\(\pm\)0.1 & 80.3 \\ Xia et al. (2021) & 1.8M & 84.2 &
- \\ Bai et al. (2022) & 200K & **85.4** & **84.2** \\ \hline Zhou et al. (2021b) & & sep-voc & joint-voc & sep-voc & joint-voc \\ Structured-BART-baseline & & 84.0\(\pm\)0.1 & 84.2\(\pm\)0.1 & 82.3\(\pm\)0.0 & 82.0\(\pm\)0.0 \\ + self-trained silver\({}^{1}\) & 90K &
- & 84.7\(\pm\)0.1 & 82.7\(\pm\)0.1 & 82.6\(\pm\)0.0 \\ + self-trained silver\({}^{1}\) + ensemble dec. & 90K &
- & 84.9 & 83.1 &
- \\ \hline
**Ours below** (Struct-BART) & & sep-voc & joint-voc & sep-voc & joint-voc \\ + SPRING silver\({}^{1}\) & 90K & 84.8\(\pm\)0.1 & 84.8\(\pm\)0.0 & 83.0\(\pm\)0.0 & 83.2\(\pm\)0.1 \\ + SPRING + self-trained silver\({}^{1}\) (50:50) & 90K & 84.8\(\pm\)0.1 & 84.7\(\pm\)0.0 & 83.0\(\pm\)0.0 & 83.2\(\pm\)0.1 \\ \hline Ensemble-4 distillation (APT + Structured-BART + SPRING\({}_{1}\) + SPRING\({}_{2}\)) & & & & \\ \hline + MBSE-P silver\({}^{1}\) & 90K & 85.1\(\pm\)0.1 & 85.1\(\pm\)0.1 & 83.2\(\pm\)0.1 & 83.5\(\pm\)0.1 \\ + MBSE-G silver\({}^{1}\) & 90K & 85.0\(\pm\)0.0 & 85.2\(\pm\)0.1 & 83.4\(\pm\)0.0 & 83.5\(\pm\)0.0 \\ + MBSE-G silver\({}^{1+2}\) & 149K & 85.3\(\pm\)0.1 & 85.4\(\pm\)0.1 & 83.6\(\pm\)0.1 & 83.7\(\pm\)0.1 \\ + MBSE-G silver\({}^{1+2+3}\) & 219K & 85.3\(\pm\)0.1 & 85.5\(\pm\)0.1 & 83.7\(\pm\)0.0 & 83.9\(\pm\)0.0 \\ + MBSE-G silver\({}^{1+2+3}\) + ensemble dec. & 219K & **85.6** & **85.7** & **84.0** & **84.2** \\ \hline Ensemble-5 distillation (APT + Structured-BART + SPRING\({}_{1}\) + SPRING\({}_{2}\) + AMRBART) & & & & \\ \hline + MBSE-A silver\({}^{1}\) & 90K & 85.3\(\pm\)0.1 & 83.6\(\pm\)0.1 \\ + MBSE-A silver\({}^{1+2}\) & 149K & 85.5\(\pm\)0.0 & 84.0\(\pm\)0.0 \\ + MBSE-A silver\({}^{1+2+3}\) & 219K & 85.7\(\pm\)0.0 & 84.1\(\pm\)0.0 \\ + MBSE-A silver\({}^{1+2+3}\) + ensemble dec. & 219K & **85.9** & **84.3** \\ \hline \end{tabular}
\end{table}
Table 3: Smatch scores on AMR2.0 and AMR3.0 test data. Upper rows display the performances of recent published works. Structured-BART results in Zhou et al. [2021b] are shown in middle rows. Lower rows show Structured-BART performances with various silver data augmentations. sep-voc denotes separate vocabulary and joint-voc, joint vocabulary. The numbers prefixed by \(\pm\) indicate the standard deviation of Smatch scores across 3 seeds.

 

parsers are trained with Structured-mBART, always using separate vocabulary (sep-voc). Input sentences of the English training data are machine translated into the target languages with WLT13 to generate cross-lingual parser training data.

Footnote 12: [https://www.ibm.com/cloud/watson-language-translator](https://www.ibm.com/cloud/watson-language-translator)

Table 4 shows the results on the human translated AMR2.0 test set, following standard practices. We provide results for recently published cross-lingual AMR parsers and different silver training versions of Structured-mBART. Structured-mBART with 4 model ensemble (Ensemble-4) distillation with just silver1 improves the Smatch score by 2.1 to 2.6 over the Structured-mBART baselines, outperforming very strong previous SoTA from Cai et al. (2021) on Chinese and Spanish and tied on Italian. Increasing the input sentence diversity via AMR-to-text generation and ensemble decoding further improve the system performances, attaining new cross-lingual SoTA on all four languages. Increasing the silver training data size to silver\({}^{1+2+3}\) and using 5 model ensemble (Ensemble-5) push the numbers higher by 0.2-0.5 Smatch points.

Footnote 13: [https://www.ibm.com/cloud/watson-language-translator](https://www.ibm.com/cloud/watson-language-translator)

(Uhrig et al., 2021) report that translate-and-parse pipelines outperform the conventional cross-lingual parsers, we thus include translate-and- parse from the combination of WLT and Structured-BART + MBSE distillation. This out-performs the cross-lingual parsers by 0.6-1.0 Smatch on all languages except for Spanish, when trained with the same MBSE avg.-Smatch silver\({}^{1+2+3}\) data.

Comparing the fine-grained F1 scores for cross-lingual parsers with those for English, as shown in Table 5, we observe that cross-lingual parsers are particularly worse than English for negation. For instance, German negations are often realized as a compound, as in _nichttarifare_ (non - tariff), which is aligned to the non-negated stem portion of the concept _tariff_, losing its negation meaning. We observe similar issues in English with prefixed negations such as _unhnappy_, _inadequate_, _atypical_.

### Domain Adaptation

We use the AMR2.0 version of BioAMR (medical domain) as this has clearly defined partitions14 and was used in Bevilacqua et al. (2021). We also use QALD-9-AMR, constructed from QALD-9 data15Usbeck et al. (2018), a corpus of natural language

\begin{table}
\begin{tabular}{p{71.1pt}|c c c c c} \hline
**Models** & **LM** & **DE** & **ES** & **IT** & **ZH** \\ \hline \multicolumn{6}{p{71.1pt}}{**Translate and Parse Pipelines**} \\ \hline Uhrig et al. (2021) & \multirow{3}{*}{BART} & 67.6 & 72.3 & 70.7 & 59.1 \\ WLT+Structured-BART+MBSE-G silver1  & \multirow{3}{*}{BART} & 73.9 & 76.5 & 76.1 & 63.7 \\ WLT+Structured-BART+MBSE-A silver\({}^{1+2+3}\) & & **74.6** & **77.1** & **76.7** & **64.0** \\ \hline \multicolumn{6}{p{71.1pt}}{**Cross-lingual Parsers**} \\ \hline Blloshmi et al. (2020) & \multirow{3}{*}{XLMR} & 53.0 & 58.0 & 58.1 & 43.1 \\ Sheth et al. (2021) (85K silver AMR) & & 62.7 & 67.9 & 67.4 & – \\ Procopio et al. (2021) (5M parallel corpus) & mBART\({}_{mt}\) & 69.8 & 72.4 & 72.3 & 58.0 \\ Cai et al. (2021b) & \multirow{3}{*}{mBARTmmt} & 64.0 & 67.3 & 65.4 & 53.7 \\ Xu et al. (2021) & & 70.5 & 71.8 & 70.8 & – \\ Cai et al. (2021a) (320K silver AMR) & mBARTmmt & **73.1** & **75.9** & **75.4** & **61.9** \\ \hline \multicolumn{6}{p{71.1pt}}{**Ours below** (with Structured-mBART)} \\ Structured-mBART-baseline & mBART & 69.9\(\pm\)0.0 & 74.4\(\pm\)0.3 & 73.3\(\pm\)0.2 & 59.9\(\pm\)0.0 \\ \hline Ensemble-4 distillation (APT + Structured-BART + SPRING\({}_{1}\) + SPRING\({}_{2}\)) & \multirow{3}{*}{75.4\(\pm\)0.0} & \multirow{3}{*}{62.2\(\pm\)0.1} \\ \cline{1-1} + MBSE-G silver1  & mBART & 72.5\(\pm\)0.1 & 76.5\(\pm\)0.2 & 75.4\(\pm\)0.0 & 62.2\(\pm\)0.1 \\ + MBSE-G silver1+AMR2Text + ens. dec. & mBART & 72.9\(\pm\)0.1 & 76.6\(\pm\)0.0 & 75.6\(\pm\)0.0 & 62.3\(\pm\)0.0 \\ + MBSE-G silver1+AMR2Text + ens. dec. & mBART & 73.2 & 76.9 & 75.7 & 62.7 \\ \hline Ensemble-5 distillation (APT + Structured-BART + SPRING\({}_{1}\) + SPRING\({}_{2}\) + AMRBERT) & \multirow{3}{*}{75.3\(\pm\)0.1} & \multirow{3}{*}{**77.1\(\pm\)**0.2} & \multirow{3}{*}{76.0\(\pm\)0.1} & \multirow{3}{*}{62.7\(\pm\)0.1} \\ \cline{1-1} + MBSE-A silver1+2+3 & & & & & \\ + MBSE-A silver1+2+3 + ens. dec. & mBART & **73.7** & 77.0 & **76.1** & **63.0** \\ \hline \end{tabular}
\end{table}
Table 4: Cross-lingual parser Smatch scores on AMR2.0 human translated test sets. mBART\({}_{mt}\) of Procopio et al. (2021) indicates the mBART model fine-tuned on both semantic parsing tasks and the MT data. mBARTnmt of Cai et al. (2021) indicates an NMT model by Tang et al. (2020), trained from mBART covering 50 languages. Shortnames: MBSE-G (greedy-selection), MBSE-A (average-Smatch) ’ens. dec.’, ensemble decoding.

 

questions for executable semantic parsing (Kapanipathi et al., 2021). Corpus statistics of the domain adaptation data is summarized in Table 1.

Table 6 shows the experimental results. Results for SPRING are taken from Bevilacqua et al. (2021). For each test set, we report the results under three different training conditions, all of which include either AMR2.0 or AMR3.0 treebank in the training data: (1) use only silver data with MBSE distillation, (2) use only domain gold sentences, (3) use both silver data and domain gold sentences. Since BioAMR is annotated in AMR2.0 style and QALD-9-AMR in AMR3.0 style, we use the corresponding Structured-BART models as indicated in the table.

As for BioAMR data, MBSE distillation (with both graphene-Smatch and greedy-select) on silver\({}^{b}\) - comprising PubMed (LDC2008T20, LDC2008T21), BioNLP-ST-2011 and CRAFT - improves over the Structured-BART baseline by 6.5 Smatch point (60.4 vs. 66.9). However, adding just 201 domain gold sentences to AMR2.0 treebank results in 11.9 Smatch point improvement over the baseline (60.4 vs. 72.3). A close inspection shows that this is largely due to the NER score improvement, as shown in the column under **NER**, i.e. NER score 27.0 in Structured-BART (AMR2.0) vs. 68.0 after adding 201 domain gold sentences. The dramatic impact of NE coverage no longer holds when we double the domain gold sentences from 201 to 403. In fact, MBSE greedy-select silver\({}^{b}\) + 201 domain gold sentences (75.8) is more effective than doubling the domain gold sentences (74.3). Finally, by combining MBSE distillation on silver\({}^{b}\) with 5K domain gold sentences, the system achieves 81.3 Smatch, outperforming the previous SoTA by 1.4.

Regarding QALD-9-AMR data, MBSE distillation on silver\({}^{q}\), i.e. SQuAD-Q(uestion) sentences,

\begin{table}
\begin{tabular}{l|c|c} \hline
**Training Data** & **Smatch** & **NER** \\ \hline \hline \multicolumn{3}{c}{BioAMR Evaluations} \\ \hline SPRING\({}^{\text{DFS}}_{\text{+ silver}}\) & 59.7 & \\ SPRING\({}^{\text{DFS}}_{\text{+ silver}}\) & 59.5 & \\ SPRING\({}^{\text{DFS}}\) (In domain) & 79.9 & \\ \hline
**Ours** & \multirow{4}{*}{72.0} & \multirow{4}{*}{60.4} & \multirow{4}{*}{27.0} \\ Struct-BART (AMR2.0) & 60.4 & & \\ +MBSE-G silver\({}^{1}\) & 63.2 & & 31.0 \\ +MBSE-G silver\({}^{b}\) & 66.9\(\pm 0.2\) & & 30.0 \\ +MBSE-P silver\({}^{b}\) & 66.9\(\pm 0.2\) & & 31.0 \\ \cline{2-3} +201 domain gold sent. & 72.3\(\pm 0.2\) & & 68.0 \\ +403 domain gold sent. & 74.3\(\pm 0.2\) & & 70.0 \\ +5K domain gold sent. & 79.8\(\pm 0.2\) & & 80.0 \\ \hline +MBSE-G silv.\({}^{b}\)+201 gold & 75.8\(\pm 0.3\) & & 70.0 \\ +MBSE-G silv.\({}^{b}\)+5K gold & **81.3\(\pm 0.2\)** & & **81.0** \\ \hline \multicolumn{3}{c}{**QALD-9-AMR Evaluations**} \\ \hline
**Ours** & \multirow{4}{*}{87.2} & \multirow{4}{*}{84.0} \\ Struct-BART (AMR3.0) & & & \\ +MBSE-G silver\({}^{1}\) & & & 88.0 \\ +MBSE-G silver\({}^{q}\) & & 89.5\(\pm 0.1\) & & 85.0 \\ +MBSE-P silver\({}^{q}\) & & 89.3\(\pm 0.2\) & & 87.0 \\ \cline{2-3} +200 domain gold sent. & 88.5\(\pm 0.5\) & & 84.0 \\ +408 domain gold sent. & 89.8\(\pm 0.1\) & & 86.0 \\ \hline +MBSE-G silv.\({}^{q}\)+200 gold & 90.0\(\pm 0.3\) & & 87.0 \\ +MBSE-G silv.\({}^{q}\)+408 gold & **90.1\(\pm 0.1\)** & & 87.0 \\ \hline \end{tabular}
\end{table}
Table 6: Smatch scores on BioAMR and QALD-9 test sets with varying sizes of human annotated (gold) domain sentences and silver data. MBSE-G (greedy-select) and MBSE-P (Graphene-Smatch respectively). MBSE distillations are all with Ensemble-4 (APT + Structured-BART + SPRING\({}_{1}\) + SPRING\({}_{2}\)).

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c|c|c} \hline Languages & Smatch & Unlabeled & NoWSD & Concepts & NER & Neg. & Wiki & Reentrant & SRL \\ \hline EN-mono & **85.9** & **89.0** & **86.3** & **92** & **93** & **75** & **81** & **78** & **85** \\ \hline DE-cross & 73.7 & 77.9 & 73.8 & 75 & 89 & _48_ & 79 & 61 & 68 \\ ES-cross & 77.1 & 81.4 & 77.5 & 81 & 89 & _62_ & 79 & 67 & 74 \\ IT-cross & 76.1 & 80.4 & 76.3 & 79 & 90 & _56_ & 78 & 65 & 73 \\ ZH-cross & 63.0 & 67.9 & 63.1 & 65 & 85 & _35_ & 70 & 51 & 58 \\ \hline DE-pipeline & 74.6 & 78.9 & 74.8 & 75 & 91 & _51_ & 80 & 62 & 68 \\ ES-pipeline & 77.1 & 81.1 & 77.3 & 80 & 91 & _61_ & 79 & 66 & 74 \\ IT-pipeline & 76.7 & 80.9 & 76.9 & 79 & 91 & _58_ & 80 & 65 & 73 \\ ZH-pipeline & 64.0 & 68.9 & 64.0 & 66 & 86 & _40_ & 74 & 51 & 59 \\ \hline \end{tabular}
\end{table}
Table 5: Fine-grained F1 scores on the AMR2.0 test set for EN (English), DE (German), ES (Spanish), IT (Italian) and ZH (Chinese). EN-mono denotes English mono-lingual parser, {DE,ES,IT,ZH}-cross, cross-lingual parsers and (DE,ES,IT,ZH)-pipeline, translate-and-parse pipeline.

 

is almost as effective as 408 domain gold sentences (89.8) for both graphene-Smatch (89.3) and greedy-select (89.5) algorithms. Combining 408 domain gold sentences with MBSE greedy-select silver\({}^{q}\) adds less than 1 Smatch point to 90.1.

Since MBSE distillation on silver\({}^{b}\) lags behind the performance of 201 human annotated AMR for BioAMR, mostly due to low NER scores, we further analyze the target vocabulary coverage of named entity (NE) types occurring in the test sets. The analysis is shown in Table 7. NE types are equally well covered in all models for Q9AMR (QALD-9-AMR). 0.7% out-of-vocabulary (OOV) ratio is caused by a typo in human annotation of the test set, i.e. _country_ misspelled as _country_. For BioAMR, however, NE type OOV ratio of MBSE silver model is 3.7%, e.g. _protein-segment_, _macromolecular-complex_, substantially higher than 0.6% of the model trained with 201 domain gold sentences. When the NE type is OOV, there is no chance for the system to produce the missing NE type, let alone predicting it correctly, underscoring the challenges posed by domain specific concepts unavailable elsewhere.

## 6 Related Work

There have been numerous works applying ensemble/knowledge distillation Hinton et al. (2015) to machine translation Kim and Rush (2016); Freitag et al. (2017); Nguyen et al. (2020); Wang et al. (2020), dependency parsing Kuncoro et al. (2016) and question answering Mun et al. (2018); Ze et al. (2020); You et al. (2021); Chen et al. (2012). Regarding ensembling AMR graphs, Barzdins and Gosko (2016) propose choosing the AMR with highest average sentence Smatch to all other AMRs. Hoang et al. (2021) proposed a more complex technique capable of building new AMRs by exploiting Smatch's hill climbing algorithm. Our work brings together ensemble distillation and Smatch-based ensembling and shows that it can provide substantial gains over the standard self-training.

Footnote 1: [https://github.com/faceface/](https://github.com/faceface/) 



## References

* Bai et al. (2022) Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022. Graph pre-training for AMR parsing and generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Cited by: SS3.4.
* D. Cai and K. Knight (2013)match: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Sofia, Bulgaria, pp. 748-752. External Links: Link, Document Cited by: SS1, SS2.
* C. Chen, C. Wang, M. Qiu, Dehong Gao, Linbo Jin, and W. Li (2012)Cross-domain knowledge distillation for retrieval-based question answering systems. In Proceedings of the World Wide Web Conference 2021, Cited by: SS1, SS2.
* K. B. Cohen, K. Verspoor, K. Fort, C. Funk, M. Bada, M. Palmer, and L. E. Hunter (2017)The colorado richly annotated full text (craft) corpus: multi-model annotation in the biomedical domain. Handbook of Linguistic Annotation, pp. 1379-1394. Cited by: SS1, SS2.
* M. Damonte and S. B. Cohen (2018)Cross-lingual abstract meaning representation parsing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 1146-1155. External Links: Link, Document Cited by: SS1, SS2.
* R. Fernandez Astudillo, M. Ballesteros, T. Naseem, A. Blodgett, and R. Florian (2020)Transition-based parsing with stack-transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, Vol. , pp. 1001-1007. External Links: Link, Document Cited by: SS1, SS2.
* M. Freitag, Y. Al-Onaizan, and B. Sankaran (2017)Ensemble distillation for neural machine translation. In arXiv:1702.01802, Cited by: SS1, SS2.
* V. Goel and W. J. Byrne (2000)Minimum bayes-risk automatic speech recognition. Computer Speech & Language14 (2), pp. 115-135. External Links: ISSN 0018-921X, Document, Link Cited by: SS1, SS2.
* G. Hinton, O. Vinyals, and J. Dean (2015)Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: SS1, SS2.
* T. Lam Hoang, G. Picco, Y. Hou, Y. Lee, L. M. Nguyen, D. T. Phan, V. Lopez, and R. Fernandez Astudillo (2021)Enseembling graph prediction for amr parsing. arXiv preprint arXiv:2110.09131. Cited by: SS1, SS2.
* P. Kapanipathi, I. Abdelazir, S. Ravishankar, S. Roukos, A. Gray, R. Fernandez Astudillo, M. Chang, C. Cornelio, S. Dana, A. Fokoue, D. Garg, A. Gliozzo, S. Gurajada, H. Karaman, X. Khan, D. Khandelwal, Y. Lee, Y. Li, F. Luus, M. Makondo, N. Mihindukuasooriya, T. Naseen, S. Nealam, L. Popa, R. Gandigi Reddy, R. Riegel, G. Rossiello, U. Sharma, G. P. Shrivastava Bhargav, and M. Yu (2021)Leveraging abstract meaning representation for knowledge base question answering. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Online, pp. 3884-3894. External Links: Link, Document Cited by: SS1, SS2.
* J. Kim, S. Pyysalo, T. Ohta, R. Bossy, N. Nguyen, and J. Tsujii (2011)Overview of bionlp shared task 2011. In Proceedings of BioNLP Shared Task 2011 Workshop, pp. 1-6. Cited by: SS1, SS2.
* Y. Kim and A. M. Rush (2016)Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317-1327.



Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: Sequence-to-sequence models for parsing and generation. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 146-157, Vancouver, Canada. Association for Computational Linguistics.
* Kudo (2018) Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Long Papers)_, pages 66-75.
* Kumar and Byrne (2004) Shankar Kumar and William Byrne. 2004. Minimum bayes-risk decoding for statistical machine translation. Technical report, JOHNS HOPKINS UNIV BALTIMORE MD CENTER FOR LANGUAGE AND SPEECH PROCESSING (CLSP).
* Kuncoro et al. (2016) Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, and Noah A. Smith. 2016. Distilling an ensemble of greedy dependency parsers into one mst parser. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1744-1753.
* Lee et al. (2020) Young-Suk Lee, Ramon Fernandez Astudillo, Tahira Naseem, Revanth Gangi Reddy, Radu Florian, and Salim Roukos. 2020. Pushing the limits of amr parsing with self-learning. In _Findings of the EMNLP2020_, pages 3208-3214.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceresigns of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880.
* Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. In _arXiv:2001.08210_.
* Lyu et al. (2020) Chunchuan Lyu, Shay B Cohen, and Ivan Titov. 2020. A differentiable relaxation of graph segmentation and alignment for amr parsing. _arXiv preprint arXiv:2010.12676_.
* Mager et al. (2020) Manuel Mager, Ramon Fernandez Astudillo, Tahira Naseem, Md Arafat Sultan, Young-Suk Lee, Radu Florian, and Salim Roukos. 2020. GPT-too: A language-model-first approach for AMR-to-text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1846-1852, Online. Association for Computational Linguistics.
* Mun et al. (2018) Jonghwan Mun, Kimin Lee, Jinwoo Shin, and Bohyoung Han. 2018. Learning to specialize with knowledge distillation for visual question answering. In _32nd Conference on Neural Information Processing Systems (NeurIPS 2018)_.
* Naseem et al. (2019) Tahira Naseem, Abhishek Shah, Hui Wan, Radu Florian, Salim Roukos, and Miguel Ballesteros. 2019. Rewarding smatch: Transition-based amr parsing with reinforcement learning. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4586-4592.
* Nguyen et al. (2020) Xuan-Phi Nguyen, Shafiq Joty, Wu Kui, and Ai Ti Aw. 2020. Data diversification: A simple strategy for neural machine translation. In _34th Conference on Neural Information Processing Systems (NeurlPS 2020)_.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auil. 2019. Fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of NAACL-HLT 2019: Demonstrations_, pages 48-53.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 311-318.
* Procopio et al. (2021) Luigi Procopio, Rocco Tripodi, and Roberto Navigli. 2021. Sgl: Speaking the graph languages of semantic parsing via multilingual translation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langaging Technologies_, pages 325-337.
* Ribeiro et al. (2021) Leonardo FR Ribeiro, Martin Schmitt, Hinrich Schutze, and Iryna Gurevych. 2021. Investigating pretrained language models for graph-to-text generation. In _Proceedings of the Third Workshop on Natural Language Processing for Conversational AI_, pages 211-227.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1715-1725.
* Sheth et al. (2021) Janaki Sheth, Young-Suk Lee, Ramon Fernandez Astudillo, Tahira Naseem, Radu Florian, Salim Roukos, and Todd Ward. 2021. Bootstrapping multilingual amr with contextual word alignments. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics_, pages 394-404.
* Tang et al. (2020) Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. _arXiv preprint arXiv:2008.00401_.
* Uhrig et al. (2021) Sarah Uhrig, Yoalli Rezepka Garcia, Juri Opitz, and Anette Frank. 2021. Translate, then parse! a strong baseline for cross-lingual amr parsing. In _Proceedings of the 17th International Conference on Parsing Technologies (IWPT 2021)_, pages 58-64 

Ricardo Usbeck, Ria Hari Gusmita, Muhammad Saleem, and Axel-Cyrille Nognga Ngomo. 2018. 9th challenge on question answering over linked data (qald-9). In _Joint proceedings of the 4th Workshop on Semantic Deep Learning (SemDeep-4) and NLIWoD4, Vol-2241_, pages 58-64.
* van Noord and Bos [2017] Rik van Noord and Johan Bos. 2017. Neural semantic parsing by character-based translation: Experiments with abstract meaning representations. _arXiv preprint arXiv:1705.09980_.
* Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _31st Conference on Neural Information Processing Systems (NIPS 2017)_.
* Wang et al. [2021] Fusheng Wang, Jianhao Yan, Fandong Meng, and Jie Zhou. 2021. Selective knowledge distillation for neural machine translation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_, pages 6456-6466.
* Wang et al. [2020] Yiren Wang, Lijun Wu, Yingce Xia, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2020. Transductive ensemble learning for neural machine translation. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)_, pages 6291-6298.
* Wilson and Izmailov [2020] Andrew Gordon Wilson and Pavel Izmailov. 2020. Bayesian deep learning and a probabilistic perspective of generalization. _arXiv preprint arXiv:2002.08791_.
* Xia et al. [2021] Qingrong Xia, Zhenghua Li, Rui Wang, and Min Zhang. 2021. Stacked amr parsing with silver data. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 4729-4738.
* Xu et al. [2020] Dongqiu Xu, Junhui Li, Muhua Zhu, Min Zhang, and Guodong Zhou. 2020. Improving amr parsing with sequence-to-sequence pre-training. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_, pages 2501-2511.
* Xu et al. [2021] Dongqin Xu, Junhui Li, Muhua Zhu, Min Zhang, and Guodong Zhou. 2021. Xlpt-amr: Cross-lingual pre-training via multi-task learning for zero-shot amr parsing and text generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_, pages 896-907.
* You et al. [2021] Chenyu You, Nuo Chen, and Yuexian Zou. 2021. Knowledge distillation for improved accuracy in spoken question answering. In _ICASSP 2021_.
* Ze et al. [2020] Yang Ze, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. 2020. Model compression with two-stage multi-teacher knowledge distillation for web question answering system. In _The Thirteenth ACM International Conference on Web Search and Data Mining_.
* Zhang et al. [2019a] Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019a. Amr parsing as sequence-to-graph transduction. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 80-94.
* Zhang et al. [2019b] Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019b. Broad-coverage semantic parsing as transduction. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP)_, pages 3784-3796.
* Zhou et al. [2021a] Jiawei Zhou, Tahira Naseem, Ramon Fernandez Astudillo, and Radu Florian. 2021a. Amr parsing with action-pointer transformer. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5585-5598.
* Zhou et al. [2021b] Jiawei Zhou, Tahira Naseem, Ramon Fernandez Astudillo, Young-Suk Lee, Radu Florian, and Salim Roukos. 2021b. Structure-aware fine-tuning of sequence-to-sequence transformers for transition-based amr parsing. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 6279-6290.

 



## Appendix A Greedy-Select Ensemble Algorithm

```
0: AMR\({}_{1}\)...AMR\({}_{n}\) parses from \(n\) AMR parsing models, where \(n\geq 3\) Optionally Require: Smatch score threshold = \(\theta\) Output: One-best AMR parse
1: Let bestAMR = \(NULL\)
2:for\(\forall_{i,j}\) in \(1\leq i,j\leq n\) and \(i\neq j\)do
3: Compute sentence Smatch score \(smatch\text{(AMR${}_{i}$, AMR${}_{j}$)}\), total \(n(n-1)/2\) scores.
4: Pick the highest \(smatch\text{(AMR${}_{i}$, AMR${}_{j}$)}\).
5:for Each AMR\({}_{a}\), where \(a=i\) or \(a=j\)do
6: Pick the highest \(smatch\text{(AMR${}_{a}$, AMR${}_{b}$)}\)
7:if\(a=i\)then
8: bestAMR = AMR\({}_{i}\)
9:else
10: bestAMR = AMR\({}_{j}\)
11:endif
12:if\(smatch\text{(AMR${}_{a}$,AMR${}_{b}$)}<\theta\)then
13: bestAMR = \(NULL\)
14: {no AMR to be used from this sentence}
15:endif
16:endfor
17:endfor
18:return bestAMR
```

**Algorithm 1**Greedy-Select MBSE Algorithm and Corpus Selection

We start with \(n\) parses from \(n\) heterogeneous parsing models, where the minimum number of parses is \(3\). For each input sentence, we compute sentence-level Smatch scores between any two parses across all \(n\) parses, for a total of \(n(n-1)/2\) Smatch scores (lines 2-3). Subsequently, we pick the two parses AMR\({}_{i}\) and AMR\({}_{j}\) with the highest Samatch score, where AMR\({}_{i}\) denotes the AMR parse from the system \(i\) (line 4) For each of the two parses, AMR\({}_{i}\) and AMR\({}_{j}\), we choose the parse with the higher Smatch score against the rest of the parses as the best parse (lines 5-11). When the scores are tied, we select the first parse output (equivalent to a random choice of fixed seed).

We incorporate an optional parse selection criterion into Algorithm 1, indicated as **Optionally Require** and specified in lines 12-15. The bestAMR for input sentence is selected for data augmentation if the Smatch score \(smatch\text{(AMR${}_{a}$, AMR${}_{b}$)}\) is greater than or equal to the pre-specified value \(\theta\).

## Appendix B Model Structures and Parameter Size

Pre-trained BART and mBART share the same model configurations except for the vocabulary size. There are 12 encoder/decoder layers, 16 heads per layer, 1024 model dimension and 4096 feed forward network (FFN) size. BART includes ~50K and mBART, ~250K task vocabulary.

When using separate vocabulary (sep-voc), Structured-BART and Structured-mBART use the same vocabulary as BART and mBART, respectively, for the source. For the target, they create embedding vectors for action symbols and the target vocabulary size vary according to the training data. When using joint vocabulary (joint-voc), Structured-BART shares the same vocabulary between the source and the target, a combination of BART vocabulary and the additional embedding vectors for some action symbols.

Vocabulary and parameter sizes for Structured-BART and Structured-mBART trained with MBSE distillation are shown in Table 9 and Table 10, respectively.

## Appendix C Implementation Details

Our models are implemented with FAIRSEQ toolkit Ott et al. (2019), trained and tested on a single NVIDIA Tesla A100/V100 GPU with 40-80GB memory. We use fp16 mixed precision training and all models are trained on 1 GPU.

For all English AMR parsing models with silver data, we use the Adam optimizer with \(\beta_{1}=0.9\)

\begin{table}
\begin{tabular}{l|c|r} \hline Models & AMR2.0 & AMR3.0 \\ \hline Structured-BART & 34,156 & 33,200 \\ SPRING\({}_{1}\) & 25,129 & 29,407 \\ SPRING\({}_{2}\) & 17,866 & 17,830 \\ APT & 10,235 & 6,949 \\ Total & 87,386 & 87,386 \\ \hline \end{tabular}
\end{table}
Table 8: Distribution of individual model parses from MBSE greedy-select distillation with silver\({}^{1}\) dataset in Table 1

\begin{table}
\begin{tabular}{l|c|c|c} \hline Model & Param & AMR2.0 & AMR3.0 \\ \hline  & src voc size & 50,265 & 50,265 \\ sep-voc & tgt voc size & 42,344 & 42,784 \\  & \# param & 493,011,968 & 493,913,088 \\ \hline joint-voc & joint voc size & 57,912 & 58,673 \\  & \# param & 414,121,984 & 414,901,248 \\ \hline \end{tabular}
\end{table}
Table 9: Vocabulary and parameter sizes of Structured-BART with MBSE distillation on silver\({}^{1+2+3}\) dataset from Table 1

and \(\beta_{2}=0.98\). Batch size is set to 1024 maximum number of tokens with gradient accumulation over 8 steps. Learning rate schedule is the same as Vaswani et al. (2017) with 4000 warm-up steps and \(1\mathrm{e}{-7}\) warm-up initial learning rate and the maximum learning rate \(1\mathrm{e}{-4}\). Dropout rate is 0.2 and label smoothing rate is 0.01. These hyper parameters are fixed and not tuned for different models and datasets. All models are trained for 10 epochs and the best 5 checkpoints are selected based on the development set Smatch from greedy decoding. Model parameters are averaged over the top 3 and top 5 models. The model that produces the highest development set score, after beam search decoding with beam size = 1, 5 and 10, is selected as the final model. Training with MBSE greedy-select silver\({}^{1+2+3}\) takes 48-72 hours, and all other models with less silver data take less time to train.

For cross-lingual AMR parsing, maximum learning rate is always set to \(3\mathrm{e}{-5}\). Baseline models trained only on AMR2.0 corpus are trained up to 80 epochs whereas models with silver\({}^{1}\) (and AMR-to-text) is trained up to 30 epochs and models with silver\({}^{1+2+3}\), up to 15 epochs. Model parameters are updated after gradient is accumulated for 8192 tokens. Dropout rate, label smoothing rate and model selection criteria are the same as the English parsers. Training baseline models takes about 10 hours. Training with silver\({}^{1}\) takes about 24 hours. Training with silver\({}^{1+2+3}\) takes about 96-120 hours. In order to reduce the vocabulary size, which subsequently reduces the model parameter size and memory requirement, we prune out singleton target vocabulary for training with silver data.

Inference time for AMR2.0 benchmark test set is shown in Table 11, where beam size=10 and batch size=64 for all languages. EN is decoded on NVIDIA Tesla A100 and all other languages, on NVIDIA Tesla V100.

## Appendix C Additional Results

### Experimental results

We report the performance of our proposed method using a single-layer architecture. We use the standard ResNet architecture to generate the best performance of our network. Our experiments show that our approach can be used to predict the performance of our network.

#### c.1.1 Ablation Study

The ablation study shows that our proposed method outperforms the state-of-the-art baselines by a factor of two. For each epoch, we compare the performances of our networks with those obtained by our network.

#### c.1.2 Ablation Study

To evaluate our network's performance, we conduct three ablation studies: 1) the ablation study on the validation dataset, 2) the ablation study on the validation dataset, and 3) the ablation study on the validation dataset.

#### c.1.3 Ablation Study

We also compare the performance of our network against our network. As mentioned earlier, we observe that our network achieves better performance than our network. This result demonstrates that our network has better performance than our network.

#### c.1.4 Ablation Study

We compare the performance of our network against our network. We see that our network performs better than our network.

#### c.1.5 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.6 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.7 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.8 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.9 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.1 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.1 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.1 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.2 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.3 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.4 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.5 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.6 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.7 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.8 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.9 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.1 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.1 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.1 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.1 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.2 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.3 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.4 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.5 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.6 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.7 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.8 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.9 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.1 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.1 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.1 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.1 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.2 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.3 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.4 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.5 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.6 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.7 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.8 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.9 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.1 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.

#### c.1.1 Ablation Study

We compare the performance of our network against our network. We observe that our network achieves better performance than our network.