AMRs Assemble! Learning to Ensemble with Autoregressive Models for AMR Parsing
Year: 2023

Abstract:
In this paper, we examine the current state-ofthe-art in AMR parsing, which relies on ensemble strategies by merging multiple graph predictions. Our analysis reveals that the present models often violate AMR structural constraints. To address this issue, we develop a validation method, and show how ensemble models can exploit SMATCH metric weaknesses to obtain higher scores, but sometimes result in corrupted graphs. Additionally, we highlight the demanding need to compute the SMATCH score among all possible predictions. To overcome these challenges, we propose two novel ensemble strategies based on Transformer models, improving robustness to structural constraints, while also reducing the computational time. Our methods provide new insights for enhancing AMR parsers and metrics. Our code is available at github.com/babelscape/AMRsAssemble.

heading: 1 Introduction
text: Semantic Parsing is the subfield of Natural Language Understanding (Navigli, 2018) that aims to encode the meaning of a sentence in a machineinterpretable structure. One of the formalisms that has gained more attention is the Abstract Meaning Representation (Banarescu et al., 2013, AMR), which embeds the semantics of a sentence in a directed acyclic graph. In AMR, concepts are represented by nodes, and semantic relations between concepts by edges (see Figure 1). AMR parsing has been applied to various areas of NLP, including Question Answering (Lim et al., 2020; Bonial et al., 2020; Kapanipathi et al., 2021), Text Summarization (Hardy and Vlachos, 2018; Liao et al., 2018), Information Extraction (Rao et al., 2017), and Machine Translation (Song et al., 2019), and has been extended to non-English languages (Anchiêta and Pardo, 2020; Blloshmi et al., 2020; Oral and Ery-
∗ Equal contributions.
iğit, 2022; Navigli et al., 2022; Martínez Lorenzo et al., 2022).
Current AMR parsing approaches are based on Transformer sequence-to-sequence (seq2seq) models (Bevilacqua et al., 2021, SPRING), which translate text into a linearized representation of the AMR graph. Recently, there have been some improvements through techniques such as pre-training on structural graph information (Bai et al., 2022), incorporating shallow semantic information (Chen et al., 2022), modifying ancestor information during decoding (Yu and Gildea, 2022), and adding a structural graph prediction task during training (Cheng et al., 2022). Nevertheless, in an attempt to push SMATCH (Cai and Knight, 2013) performance, there has been a recent trend towards ensemble models, which merge AMR graph predictions from multiple systems. Some examples include Graphene (Lam et al., 2021), a graph mining algorithm that searches for the largest common structure among the graph predictions, or the Maximum Bayes SMATCH Ensemble (Lee et al., 2022), which introduces a Bayesian ensemble approach in order to create high-quality silver data. However, notwithstanding their higher performance, ensem-
1595
ble models are potentially more vulnerable to producing corrupted AMR graphs. For instance, Opitz and Frank (2022) highlighted that better SMATCH scores do not always correlate with better parsing.
In this study, we conduct an investigation into the reasons why ensemble models improve their performance and in which cases they do so despite producing corrupted output. Our analysis reveals three significant drawbacks in these approaches: i) ensemble systems do not consider structural constraints in AMR, treating AMR graphs as regular sets of triplets, ii) they rely on SMATCH, which does not impose AMR constraints, exacerbating the problem of corrupted AMR graphs produced by ensemble methods that prioritize a higher score over adherence to structural constraints, as is the case with Graphene, and iii) they are computationally expensive. Our findings highlight the need for more robust evaluation metrics that hold to the structural constraints of AMR.
In this paper, we present two novel ensemble strategies that address the above limitations of current approaches. In the first strategy, we follow previous merging methods, showing how to train a seq2seq model to combine different predictions by taking into account both the original sentence and predictions from multiple models. In our second approach, we propose using selection as the ensembling strategy, where we select the best graph instead of merging. Specifically, we base our method on the perplexity score of the model. Additionally, we propose a graph algorithm that checks the structural constraints in AMR graphs. Through these contributions, we aim to provide more robust and efficient solutions for ensembling AMRs.

heading: 2 AMRs Assemble!
text: The task of AMR parsing can be framed as a seq2seq task, where the input t = [x1, x2, ..., xm] is a sequence of m tokens and the output g = [g1, g2, ..., gn] is a linearized graph with n tokens. To illustrate, the linearized representation of the AMR graph in Figure 1 is as follows: (z0 / schedule -01
:ARG0 (z1 / person :name (z2 / name
:op1 "Antonio" :op2 "Banderas "))
:ARG1 (z3 / premiere -01 :ARG0 z1 :ARG1 (z4 / movie :poss z1) :time (z5 / date -entity
:time "15:00")))
The goal of seq2seq AMR parsing task is to learn a function that models the conditional probability:
p(g|x) = n∏
t=1
p(et|e<t, x), (1)
where e<t are the tokens of the linearized graph g before step t.
In this work, we use LongT5 (Guo et al., 2022) as the seq2seq model, which is specialized for long sequences, making it feasible to provide sentences and linearized graphs as input.

heading: 2.1 Pre-training
text: To enhance the structure awareness of the language model in relation to AMR graphs and ensembling techniques, we extend the graph self-supervised pre-training method proposed by Bai et al. (2022, AMRBart). Formally, we denote a sentence as t = [x1, x2, ..., xm], a graph as g = [g1, g2, ..., gn], and a prediction by system s as ps = [ps1, p s 2, ..., p s ls ]. We follow AMRBart noise function with a dynamic masking rate and denote the noisy text and graph as t̂ and ĝ, respectively. Moreover, let t, g, and p be the empty text, graph and prediction, respectively. As shown in Table 1, our pre-training procedure includes tasks presented by AMRBart, such as: i) empty text graph denoising (tĝ2g), ii) text augmented graph denoising (tĝ2g), and iii) noisy text augmented graph denoising (t̂ĝ2g). Additionally, we introduce: iv) empty text multiple graph denoising tĝ1...ĝk2g, where the target graph is generated using different graphs’ masked versions, and v) noisy text augmented multiple graph denoising t̂ĝ1...ĝk2g, where we also include the masked sentence.

heading: 2.2 Fine-tuning
text: Prediction Corpus To fine-tune ensemble systems we create a corpus of multiple predictions, starting from AMR 3.0 (LDC2020T02), which consists of 59,255 human-annotated sentence-graph pairs. We create five distinct train-test splits of this dataset in such a way that each test set is one fifth of the data and mutually exclusive. We train five separate models, based on Blloshmi et al. (2021), on the corresponding training sets and use each model to generate predictions for its respective test set. By combining all of the predicted test sets, we obtain a corpus of AMR predictions. However, to train an ensemble model, it is necessary to merge predictions from multiple models. Therefore, we
generate five distinct prediction corpora by repeating this process four additional times with different train-test split sets.
Strategy Having obtained a corpus comprising multiple AMR predictions, we design a set of tasks that fine-tune the model for ensembling. The first task is AMR parsing (tp1...k2g), i.e., an AMR graph g is generated by using only a sentence t as input. In the second task, ensemble AMR predictions (tp1...k2g), the model is provided with a random set of AMR predictions p without the corresponding sentence, so it is forced to use just graph information to ensemble. In the last task, ensemble AMR predictions using the sentence (tp1...k2g), the model is provided with both a random set of AMR predictions p and the original sentence t. To ensure that the model is able to learn to merge a variety of predictions, we randomly modify the samples by changing the order and number of predictions in each epoch. As a result of this process, we obtain a model that is able to effectively integrate information from multiple sources to generate high-quality AMR graphs, without relying on the expensive SMATCH metric as has been the case for previous ensemblers.

heading: 2.3 Assemble! zero & avg
text: Nevertheless, using large autoregressive models to generate AMR graphs can be computationally expensive. Therefore, we propose an alternative approach that is more effective than previous merging strategies. Our method selects the best graph from a set of predictions. To achieve this, we introduce two novel scoring functions, in which we provide each predicted graph to the decoder of a model and extract their perplexity score, which can be done with a single forward pass. In the first method (Assemble!zero), we leverage our trained ensemble model by providing the sentence and all the predictions in order to extract their perplexities and
select the smallest one, i.e., we select prediction ps′ , where:
s′ = argmin s∈{1,...,l} perplexity(tp1...l2ps).
In the second method (Assemble!avg), instead of using our ensembler, we use each model that generated the predictions to extract the perplexity for all the candidates. The final output is the graph ps′ with the lowest average perplexity score, where:
s′ = argmin s∈{1,...,l}
1
l
∑
j∈{1,...,l} perplexityj(t2ps).

heading: 3 Experiments

heading: 3.1 Setup
text: Dataset We evaluate our model using AMR 3.0. For pre-training, we use the same 200k silver data parsed by Bevilacqua et al. (2021, SPRING) from the Gigaword (LDC2011T07) corpus. For finetuning, we use the corpus described in Section 2.2.
Metric To evaluate our results, we employ the SMATCH metric, which quantifies the similarity between graphs by measuring the degree of overlap between their triplets, and SMATCH’s breakdown metrics (see Appendix D). In addition, we validate our results using two novel AMR metrics: S2MATCH (Opitz et al., 2020) and WWLK (Opitz et al., 2021), in its WWLK-k3e2n version introduced in Opitz et al. (2021).
Ensemble Baselines For our selection strategy, we use the system of Barzdins and Gosko (2016) as a baseline, which calculates the average SMATCH score for a given graph in comparison to all the other candidates and selects the one with the highest score.
Our baseline for merging is Graphene (Lam et al., 2021), an algorithm that identifies the graph with the most nodes and edges in common among
different graphs. Specifically, given a pivot graph gi (where i = 1, 2, ..., k), Graphene collects votes from the other graphs for every existing vertex and existing/non-existing edges to correct gi. We use two variants of Graphene, i) Graphenebase, where every input graph is chosen as a pivot graph once, and the best among the modified pivot graphs is chosen as the final prediction based on average support; and ii) Graphenesmatch, which is similar to Graphenebase but chooses the best modified pivot graph based on average SMATCH score, similar to Barzdins and Gosko (2016).
We do not compare our approach using Maximum Bayes SMATCH Ensemble (Lee et al., 2022), as it is a technique for producing high-quality silver data by combining SMATCH-based ensembling techniques with ensemble distillation, and its code and data are not publicly available.
Our Models We simulate an ensemble of five models obtained by training SPRING on five different seeds, and apply these models to the test split of AMR 3.0 using each of them. Assemble! and Assemble!zero rely on LongT5 (Guo et al., 2022) and are trained as explained in Section 2.

heading: 3.2 Results
text: We present our results in Table 2. The Predictions block shows the performance of each individual system used for ensembling, which have an average SMATCH score of 82.8. The Bestgraphs row portrays the upper bound of the selection strategy, where the SMATCH score is calculated with an oracle that selects the graph with the highest SMATCH. This score is 3.4 points higher than the best predictions. The Mergers block presents the results of the ensembling strategies that combine predictions, where we observe that our model performs com-
parably to Graphenesmatch but is 10 times faster. Furthermore, the Selector block presents the results of the three different selection strategies, where the best graph is chosen out of a set of predictions. Our strategy outperforms SMATCHavg by 0.4 points while having a similar computation time. These results demonstrate the effectiveness of our proposed ensembling approaches and suggest that they may be an alternative to traditional merging methods.

heading: 3.3 Analysis
text: While our model is able to effectively ensemble graphs or select the most accurate one from a set of predictions in an efficient and competitive manner, it is important to note that a higher SMATCH score does not always equate to the best graph if the graph has structural issues. This is because the SMATCH metric simply views the graph as a set of triplets. For example, the AMR graph illustrated in Figure 2(a) is treated as the following triplets: (empty , :root , z0) ^ (z0 , :instance , schedule -01) ^ (z0 , :ARG0 , z1) ^ (z1 , :instance , person) ^ (z1 , :name , z2) ^ (z2 , :instance , name) ^ (z2 , :op1 , "Antonio ") ^ (z2 , :op2 , "Banderas ") ^ (z0 , :ARG1 , z3) ^ (z3 , :instance , premiere -01) ^ (z3 , :ARG0 , z1) ^ (z3 , :ARG1 , z4) ^ (z4 , :instance , movie) ^ (z4 , :poss , z1) ^ (z0 , :ARG3 , z5) ^ (z5 , :instance , date -entity) ^ (z5 , :time , "15:00")
SMATCH calculates the degree of overlapping between two sets of triplets, but it does not consider the implicit AMR constraints. To address this problem, we develop an algorithm that checks some AMR violations in graphs: i) non-predicate nodes with :ARG relations, ii) predicate nodes with :op or :snt relations, iii) compositional issues in entity
structures, and iv) compositional issues in connector structures. The Corrupt. column in Table 2 shows the number of graphs with structural problems out of 1898 graphs. This highlights the limitation of previous ensemblers, such as Graphene, which do not consider these structural constraints.
Ensembling As demonstrated in the Corrupt. column of Table 2, our ensemble method has a significantly lower number of graphs with structural issues (0.3%) as compared to Graphenebase and Graphenesmatch (13.7-19.7%). This is because previous ensemble models are only focused on achieving a higher SMATCH metric, interpreting the graphs just as a set of triplets. This leads to ensembled graphs with violations of AMR guidelines and semantic inconsistencies. Figure 2(d) shows the Graphenesmatch generated graph, and Figures 2(b) and 2(c) the two predictions used for ensembling. The Graphenesmatch graph presents multiple AMR violations that are not in its predictions, e.g., the premiere node is connected to the movie node with two different relations because Graphene cannot decide which is the correct edge (both relations have the same probability), and one of the relations is an argument relation (i.e., :ARG), which cannot be used with non-predicate nodes since their meaning is encoded in PropBank frames.
SMATCH Graphene results in Table 2 are competitive despite having a higher percentage of structural issues in the ensembled graphs. This discrepancy can be attributed to the inherent properties of the SMATCH metric, which penalizes missing triplets more than wrong triplets. For example, the ensembled graph of Figure 2(d) obtains a higher SMATCH score than the prediction of Figure 2(b), since, in case of doubt, selecting both triplets (relations ARG1 and mod) from node premiere to node movie results in a higher score than selecting
only the wrong triplet. This illustrates how current ensemble models exploit SMATCH weaknesses to attain higher scores. In contrast, our approaches provide competitive results while also being more robust to AMR constraints.
Furthermore, as highlighted in Opitz and Frank (2022), the current scores of AMR parsers and ensemblers (around 0.83 and 0.84, respectively) are higher than the average annotator vs. consensus inter-annotator agreement reported in Banarescu et al. (2013) (0.83 and 0.79 in newswire and web text, respectively). Additionally, WWLK results in Table 2 show how SPRING3 predictions achieve comparable results to all ensemble models. Therefore, given the issues discussed above, the suitability of SMATCH for evaluating the model’s performance beyond 0.83 has to be called into question.

heading: 4 Conclusion
text: In this paper, we leveraged self-supervised pretraining and a denoising autoencoder architecture to achieve strong results in merging AMR graph predictions. We also introduced two novel approaches for ensembling that select the best prediction from a set of candidates using simple and efficient perplexity score functions. These results suggest that the selection strategy is a promising alternative for ensembling, since it achieves competitive performance while being less expensive.
Furthermore, we developed an algorithm that checks the structural AMR constraints in parsing outputs. This allowed us to perform an analysis that revealed how previous ensemble models produce higher score graphs but exploit SMATCH weaknesses that lead to increased structural issues. Overall, our findings highlight the need for more robust evaluation metrics and ensemble models that are designed to adhere to the structural constraints.

heading: 5 Limitations
text: Our proposed ensemble approach for training the Transformer architecture has demonstrated promising results for the task of AMR ensembling. However, there are limitations that warrant further investigation in future research.
Our first limitation is the lack of generalization, as the approach was only evaluated on AMR parsing. Therefore, the application of an autoregressive ensembling model has not yet been tested on other Natural Language Processing tasks.
Moreover, in order to properly compare each ensemble system under the same conditions, we base all our experiments using the same underlying architecture, i.e. SPRING. There needs to be an exploration of these approaches using more recent, better performing parsers. However, this will require access to such systems.
Furthermore, the computational cost is also a limitation, as even though our proposed merger method, Assemble!, is more efficient than previous ensemblers, it is still computationally expensive, and particularly when we have to ensemble long graphs from multiple predictions. Moreover, as our Assemble! model is based on LongT5, it might be challenged when working with large datasets or when running experiments on resource-constrained systems. Therefore, we encourage the use of ensembling strategies focused on selecting the best graphs instead of merging.
Lastly, as our ensemble approach is based on Transformer, results can be difficult to interpret, as it can be challenging to understand how the generated graph has been ensembled by different predictions, leading to a lack of interpretability.
In summary, the proposed ensemble approach for training the Transformer architecture has shown promising results for the task of AMR ensembling and has the potential to be applied to other tasks, however, further research is necessary to address its limitations and improve performance.

heading: 6 Ethics Statement
text: Regarding the ethical and social implications of our approach for AMR ensembling, we do not believe it could have a negative impact. However, since ethical considerations are an important aspect of any research and development project, we will discuss a few ethical considerations here.
First, one potential concern is the use of Transformer-based models, which have been shown
to perpetuate societal biases present in the data used for training. Our approach relies on the use of these models, and it is crucial to ensure that the data used for training is diverse and unbiased.
Second, it is important to consider the potential impact of the proposed ensemble strategies on marginalized communities. It is possible that these strategies may inadvertently perpetuate or amplify existing biases in the data used to train and test these systems. Therefore, it is important to ensure that the proposed ensemble strategies are tested on a diverse set of data and that any biases are identified and addressed.
In conclusion, the proposed ensemble strategies in this paper can potentially have positive impact on the field of AMR parsing, however, it is important to consider the ethical implications of this research and take steps to mitigate any potential negative consequences.

heading: Acknowledgments
text: The authors gratefully acknowledge the support of the European Union’s Horizon 2020 research project Knowledge Graphs at Scale (KnowGraphs) under the Marie Marie Skłodowska-Curie grant agreement No 860801.
The last author gratefully acknowledges the support of the PNRR MUR project PE0000013-FAIR. The authors sincerely thank Lorenzo Proietti and Stefano Perrella for their contribution to this project.

heading: A Model Hyper-Parameters
text: Table 3 lists hyperparameters and search space for the experiments with SPRING models and our Assemble!. The masking probabilities of the pretraining task are: i) tĝ2g – 0.35%, ii) tĝ2g – 0.35%, iii) t̂ĝ2g – from 0.15% to 0.85% incrementing by epoch, iv) tĝ1...ĝk2g – 0.55%, and v) t̂ĝ1...ĝk2g – 0.55%.

heading: B Hardware and size of the model
text: We performed experiments on a single NVIDIA 3090 GPU with 64GB of RAM and Intel® Core™ i9-10900KF CPU. The total number of trainable parameters of SKD is 434,883,596. The pre-training phase on the silver data requires 168 hours, whereas fine-tuning requires 216 hours.

heading: C BLINK
text: All systems from Table 2 use BLINK (Wu et al., 2020) for wikification. For this purpose, we used the blinkify.py script from the SPRING repository.

heading: D Metric
text: To evaluate the predictions, we use the SMATCH metric and the extra scores of Damonte et al. (2017): i) Unlabel, compute on the predicted graphs after removing all edge labels, ii) No WSD, compute while ignoring Propbank senses (e.g., duck-01 vs duck-02), iii) Wikification, F-score on the wikification (:wiki roles), iv) NER, F-score on the named entity recognition (:name roles), v) Negations, F-score on the negation detection (:polarity roles), vi) Concepts, F-score on the concept identification task, vii) Reentrancy, computed on reentrant edges only, viii) Semantic Role Labeling (SRL), computed on :ARG-i roles only.

heading: E Data
text: The AMR 3.0 data used in this paper is licensed under the LDC User Agreement for Non-Members for LDC subscribers, which can be found here.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 5

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 6

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: In the abstract and in the introduction section.

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: Grammarly, we use to check the use of English of our paper
B 3 Did you use or create scientific artifacts? Section 2 and 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Yes, Section 1 and 3

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The licenses are self-explanatory and discussed in the Appendix.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The dataset has been widely used, and was already scrutinised for personal information before our use.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 2
C 3 Did you run computational experiments? Section 3

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We compare with previous approaches and use their implementation (Section 3)
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: However, there are limitations that warrant further investigation in future research. Our first limitation is the lack of generalization, as the approach was only evaluated on AMR parsing. Therefore, the application of an autoregressive ensembling model has not yet been tested on other Natural Language Processing tasks. Moreover, in order to properly compare each ensemble system under the same conditions, we base all our experiments using the same underlying architecture, i. e. SPRING. There needs to be an exploration of these approaches using more recent, better performing parsers. However, this will require access to such systems. Furthermore, the computational cost is also a limitation, as even though our proposed merger method, Assemble!, is more efficient than previous ensemblers, it is still computationally expensive, and particularly when we have to ensemble long graphs from multiple predictions. Moreover, as our Assemble! model is based on LongT5, it might be challenged when working with large datasets or when running experiments on resource-constrained systems. Therefore, we encourage the use of ensembling strategies focused on selecting the best graphs instead of merging. Lastly, as our ensemble approach is based on Transformer, results can be difficult to interpret, as it can be challenging to understand how the generated graph has been ensembled by different predictions, leading to a lack of interpretability. In summary, the proposed ensemble approach for training the Transformer architecture has shown promising results for the task of AMR ensembling and has the potential to be applied to other tasks, however, further research is necessary to address its limitations and improve performance