Ensembling Graph Predictions for AMR Parsing
Authors: Thanh Lam Hoang
Gabriele Picco
Yufang Hou
Young-Suk Lee
Lam M. Nguyen
Dzung T. Phan
Vanessa López
Ramon Fernandez Astudillo
Venue: NeurIPS 2021 Poster (2021)
Keywords: Abstract Meaning Representation
Graph ensemble
Deep Learning
Natural Language Processing

Abstract: In many machine learning tasks, models are trained to predict structure data such as graphs. For example, in natural language processing, it is very common to parse texts into dependency trees or abstract meaning representation (AMR) graphs. On the other hand, ensemble methods combine predictions from multiple models to create a new one that is more robust and accurate than individual predictions. In the literature, there are many ensembling techniques proposed for classification or regression problems, however, ensemble graph prediction has not been studied thoroughly. In this work, we formalize this problem as mining the largest graph that is the most supported by a collection of graph predictions. As the problem is NP-Hard, we propose an efficient heuristic algorithm to approximate the optimal solution. To validate our approach, we carried out experiments in AMR parsing problems. The experimental results demonstrate that the proposed approach can combine the strength of state-of-the-art AMR parsers to create new predictions that are more accurate than any individual models in five standard benchmark datasets. 

Ensembling Graph Predictions for AMR Parsing
Hoang Thanh Lam', Gabriele Picco', Yufang Hou', Young-Suk Lee²,
Lam M. Nguyen". Dzung T. Phan?, Vanessa López', Ramon Fernandez Astudillo?
IBM Research, Dublin, Ireland
IBM Research, Thomas J. Watson Research Center, Yorktown Heights, USA t. 1. w....honng@ii... ibm. com, gabriele wabrieee.piccoo@eeeeeeeeeeeeeeee piccoSibm. com, yhou@ie. ibm com, ysukleedus. imm com, LamNguyen. MLTD@ibm com, phandu@us iim. vanlopez@ie. imm. com, ramon astudillo@ibm com
Abstract
In many machine learning tasks, models trained to predict structure data such as graphs. For example, in natural language processing, it is very common to parse texts into dependency trees abstract meaning representation (AMR) graphs. On the other hand, ensemble methods combine predictions from multiple models to create
that more robust and accurate than individual predictions. In the literature, there are many ensembling techniques proposed for classification regression problems, however, ensemble graph prediction has not been studied thoroughly. In this work, we formalize this problem as mining the largest graph that is the most supported by a collection of graph predictions. As the problem is NP-Hard, we propose an efficient heuristic algorithm to approximate the optimal solution. To validate our approach, we carried out experiments AMR parsing problems. The experimental results demonstrate that the proposed approach can combine the strength of state-of-the-art AMR parsers to create new predictions that are more accurate than any individual models in five standard benchmark datasets.
Introduction
Ensemble learning is popular machine learning practice, which predictions from multiple models are blended to create new one that is usually more robust and accurate. Indeed, ensemble methods like XGBOOST are the winning solution in many machine learning and data science competitions [Chen and Guestrin 2016). A key reason behind the successes of the ensemble methods is that they can combine the strength of different models to reduce the variance and bias in the final prediction [Domingos] 2000 Valentini and Dietterich 2004] Research in ensemble methods mostly focuses on regression or classification problems [Dong et al. 2020). Recently, in many machine learning tasks prediction outputs are provided in form of graphs. For instance, in Abstract Meaning Representation (AMR) parsing [Banarescu et al. 2013), the input is a fragment text and the output is a rooted, labeled, directed, acyclic graph (DAG). It abstracts away from syntactic representations, in the sense that sentences with similar meaning should have the same AMR. Figure I shows an AMR graph for the sentence You told
wash the dog where nodes are concepts and edges relations.
AMR parsing is an important problem in natural language processing (NLP) research and it has a broad application in downstream tasks such question answering |Kapanipathi et al. 2020 and common sense reasoning [Lim et al.) 2020). Recent approaches for AMR parsing leverage the
advances from pretrained language models [Bevilaquua et al. 2021 and numerous deep neural
network architectures [Cai and Lam 2020a Zhou et al.) 2021].
Unlike methods for ensembling numerical or categorical values for regression or classification problems where the mean value or majority votes are used respectively, the problem of graph ensemble is more complicated. For instance, Figure 2 show three graphs 91, f2, g3 with different
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

structures, having varied number of edges and vertices with different labels. In this work, we formulate the ensemble graph prediction as graph mining problem where we look for the largest common structure among the graph predictions. In general, finding the largest common subgraph a well-known computationally intractable problem in graph theory. However, for AMR parsing problems where the AMR graphs have labels and simple tree-alike structure, we propose an efficient heuristic algorithm (Graphene) to approximate the solution of the given problem well.
tell-01
ARG1
ARGO
ARG2
wash-01
ARGO
ARG1
you
dog
Figure 1: An example AMR graph for the sentence You told me to wash the dog.
To validate our approach, we collect the predictions from four sfate-of-the-art AMR parsers and create new predictions using the proposed graph ensemble algorithm. The chosen AMR parsers are the recent tott--of-thh-attt AMR parsers including a seg2seq-based method using BART [Bevilacqua] approach proposed (Zhou et al.) 202T) and a graph-bacc approach 2020a]. In addition to those models, we also trained new seq2seq model based on T5 Raffel et al. 2020 to leverage the strength of this pretrained language model.
2021] a transition- ct al. proposed in [Cai and Lam
The experimental results show that in five standard benchmark datasets, our proposed ensemble approach outperforms the previous state-of-the-art models and achieves new state-of-the-art results in all datasets. For example, our approach achieves new state-of-the-arr results with 1.7, 1.5, and 1.3 points better than prior arts in the BIO (under out-of-distribttion evaluation), AMR 2.0, and AMR 3,0 datasets respectively. This result demonstrates the strength of our ensemble method in leveraging the model diversity to achieve better performance. An interesting property of our solution is that therefore it can be used to make an ensemble of existing model predictions without the requirement to have an access to model training. Source code is open- open-sourced!
Our paper is organized as follows: Section discusses a formal problem definition and study on the computational intractability of the formulated problem. The graph ensemble algorithm is described in Section 3 Experimental results are reported in Section while Section (asioon))iisccce discusss related works. The conclusion and future work are discussed in Section 6
2 Problem formulation
Denote (E, V) as graph with the set of edges E and the set of vertices V. Each vertex v E V and edge e E associated with a label denoted as I(v) and ((e) respectively, where 1(.) is labelling function. Given two graphs 9I (Ei, V1) and 92 (E2, V2), vertex matching Φ is bijective function that maps a vertex v E Vi to a vertex ((v) E V2.
Example 1. In Figure 2 between 91 and 92 there are many possible vertex matches, where 2,3 1] is of them (which can be read as the first vertex of g1 is mapped to the third vertex of 92 and so forth). Notice that not all vertices u E Vi has a match in Vz and vice versa. Indeed, in this example, the fourth vertex in g2 does not have a matched vertex in 9i-
Φ(91. 92) = [1 + 3,
Given two graphs It, 92 and vertex match (2)1.92). support of vertex with respect to the matching d, denoted as (v),, equal to if ((v) i(o(v)) and otherwise. Given an edge (U1, 0z) the support of with respect to the vertex match o, denoted as so(e), is equal to if I(e) = I((((v1), ((2))) and 0 otherwise.
ewwembeerapppeeeeeeeeeeeo learning

---

X
Z
Z
2
2
T
81
R2
Pivot graph
(2
(1 3,2
0 0= 2
Nodes
Nodes
Labels (votes)
11)
53
[3)
(1)
(2)
2
B (2)
9
(1)
Btt))
Laballs
Edines
Labelts (votes)
Ednes
Labels (votes)
2
X(1)
2
2 (1)
2
x (1)))22 212)
2
X+1) 422
13
(1)
13
(2)
T
(2)
23
(
23
(()
Stop 1: initialise vole
statistics
Step 2: update votes
Step 3t up uppaate votes
Step 4: filter by threshold
Figure 2: A graph ensemble example. Each node and edge of g occurs in at least two of three graphs 91, 92, 93- Therefore, g is O-supported where 0 = 2 by the given set of graphs. Graph is also the graph with the largest sum of supports among all 0-supported graphs. The tables show the node and edge support (votes) are updated in each step of the Graphene algorithm when 91 is a pivot graph.
In Figure 2 for the vertex match in g1 and the third vertex in 92 shares the same label A, therefore the support of the given vertex is equal I. On the other hand, the third vertex in 9n and the first vertex in 92 does not have the same label so their support is equal to O.
Example 2.
11, the first vertex
Between two graphs. there are many possible vertex matches, the best vertex match is defined as the one that has the maximal total vertex support and edge support. In our discussion, when we mention vertex match we always refer to the best vertex match.
Denote G {91 = E Vi). 92 = (Ez, V2).... 9m = (Em- Vm)) as set of m graphs. Given any graph = (E, V). for every denote (g. g.) as the best vertex match between and g,. The total support of a vertex E V or an edge e defined as follows:
support(e) = Em (e)
* support(v) (v) = EE= $6, (v)
Given support threshold 8, graph is called 0-supported by G if for any node V or any edge e E E, support(v) 2 0 and support(e) 2
Example 3. In Figure graph g O-supported by G [g (2, 933 where 2.
Intuitively, an ensemble graph should have as many common edges and vertices with all the graph predictions as possible. Therefore, define the graph ensemble problem follows:
Problem 1 (Graph ensemble). Given a support threshold 0 and a collection of graphs G. find the graph g that is O-supported by G and has the largest sum of vertex and edge supports.
Theorem 1. Finding the optimal O-supported graph with the largest total of support is NP-Hard.
Proof. We prove the HP-Haadnesss by reduction to the Maximum Common Edge Subgraph (MCES) problem, which is known to be an NP-Complete problem [Bahiense et al. 2012). Given two graphs gi and g2, the MCES problem finds a graph y that is common subgraph of 9i and f2 and the number of edges in is the largest. Consider the following instance of the Graph Ensemble problem with 0 2, and
{91, } created from the graphs in the MCES problem. Assume that all vertices and all edges of 91 and 92 have the same label A.
3

---

Since 0 = 2, d-supported graph is also common subgraph between 91 and 92 and vice versa. Denote 9s and Ir. as the common subgraph between gi and 92 with the largest support and the largest common edge, respectively. We can show that Os has as many edges Jc- In fact, since Is is the largest supported common subgraph there no vertex E ge such that E 9s because otherwise we can add t to Is to create larger supported graph. For any edge = (#1: 02) E De: since both vertices in and v22 also appear in Js. the edge e = (v1, U2) must also be part of gs otherwise we can add this edge to 9s to create subgraph with larger support. Therefore, Is has many edges as ge, which is also a solution to the MCES problem.
3 Graph ensemble algorithm
In this section, we discuss heuristic algorithm based the strategy "Please correct me if I am wrong!" to solve Problem [ The main idea is to improve pivot graph based on other graphs. Specifically, starting with pivot graph 9: (i 1, m.), we collect the votes from the other graphs for every existing vertex and algorithm Graphene which stands for Graph Ensemble algorithm. The key steps of the algorithm are provided in the pseudo-code in Algorithm!]
ceating/non-xxisstinnn edges to correct gs- We call the proposed
Algorithm 1: Graph ensemble with the Graphene algorithm.
Input: set of graphs G [91,92,""" 9m and the support threshold A Output: an ensemble graph gf
Algorithm: Graphene(G,,
for i 4-11 1 to m do
Spivol f- Fi
V - Initialise(gpivva/ m do
for j
j * then
V V
getVote(@@( 9j))
end
of t Filter((V,))
end
go t the graph with the largest support among gi. ". 9
Return g
For example, in Figure 2, the algorithm starts with the first graph gi and considers it as pivot graph Gpivat In the first step, it creates a table to keep voting statisties V initialized with the vote counts for every existing vertex and edge in Spivo-- To draw additional votes from the other graphs, it performs the following subsequent steps:
* Call the function Φ(g1, 9:) (a = 2,3, m) to get the best bijective mapping ø between the vertices of two graphs 91 and gi (with little bit abuse of notation we drop the index from Φ; when gi and Spivor are given in the context). For instance, the best vertex match between 9i and f2 is Φ + 3, 2 + 2, + because that vertex match has the largest number of common labeled edges and vertices.
Enumerate the matching vertices and edges to update the voting statistics accordingly. For instance, since the vertex 3 in 91 with label B is mapped to the vertex 1 in with label C, a new candidate label C' is added to the table for the given vertex. For the same reason, we add new candidate label Z for the edge (1, 2). For all the other edges and vertices where the labels are matched the votes are updated accordingly.
Once the complete voting statistics V is available, the algorithm filters the candidate labels of edges and vertices using the provided support threshold # by calling the function Filter(V, 8) to obtain an ensemble graph of. For special cases, when disconnected graphs are not considered as valid output, we keep all edges of the pivot graph even its support is below the threshold. On the other hand, for the graph prediction problem, where a graph is only considered valid graph if it does not have multiple edges between two vertices and multiple labels for any vertex, we remove all candidate labels for vertices and edges except the one with the highest number of votes.

---

Assume that the resulting ensemble graph that is created by using 9i as the pivot graph is denoted as g.. The final ensemble graph g chosen among the set of graphs 92 .2.-.. .......mmm as the one with the largest total support. Recall that ((pivoo It) finds the best vertex match between two graphs. In general, the given task computationally intractable. However, for labeled graphs like AMR a heuristic was proposed (Cai and Knight, 2013) to approximate the best match by hill-climbing algorithm. It first starts with the candidate with labels that are mostly matched. The initial match is modified iteratively to optimize the total number of matches with predefined number of iterations (default value set to 5). This algorithm is very efficient and effective, was used to calculate the Smatch score in [Cai and Knight 2013] so we reuse the same implementation to approximate (gpivoo 9s) (report on average running time can be found in the supplementary materials).
Experiments
We compare our Graphene algorithm against four previous state-of-the-art models on different benchmark datasets. Below we describe our experimental settings.
4.1
Experimental settings
4.1.1 Model settings
The SPRING model, presented in [Bevilacqua et al., 2021], tackles Text-to-AMR and AMR-to-Text as symmetric transduction task. The authors show that with a pretrained encoder- decoder model, it is possible to obtain state-of-the-art performances in both tasks using a simple seq2seq framework by predicting linearized graphs. In our experiments, we used the pretrained models provided in [Bevilacqua et al. 2022 [ In addition, we trained 3 more models using different random seeds following the same setup described (Bevilacqua et al.) [202T]. Blink (Li et al. 2020) was used to add wiki tags to the predicted AMR graphs as post-processing step.
SPRING
TS The T5 model, presented in [Raffel al. 2020 introduces unified framework that models wide range of NLP tasks as text-to-text problem. We follow the same idea proposed in [Xu et al. 2020) to train a model transfer a text to a lincarized AMR graph based on T5-Large. The data is preprocessed by linearization and removing wiki tags using the script provided in [amr] In addition to the main task, we added a new task that takes as input sentence and predicts the concatenation of word senses and arguments provided in the English Web Treebank dataset [goo). The model is trained with 30 epochs. We use ADAM optimization with learning rate of le-4 and a bini-batch size of Blink (Li et al. 2020
was used to add wiki tags to the predicted AMR graphs during post-processing.
APT Zhou et al. 2021 proposed a transition- based AMR parser based on Transformer Vaswani ctt al. 2017]. It combines hard-attentions over sentences with a target side action pointer mechanism to decouple source tokens from node representations, In our experiments, we use the setup described in [Zhou et al. [2211] and added 70K model- silver sentences to the training data, which was created from the 85K sentence set in [Lee et al. 2020 with self-learning described in the paper.
Cai&Lam The model proposed in CCai and Lam 2020b treats AMR parsing as series of dual decisions (i.e., which parts of the sequence to abstraci, and where in the graph construct) on the input sequence and constructs the AMR graph incrementally. Following (Cai and Lam| [0220b], we use Stanford CoreNLF for tokenization, lemmatization, part-of-speech tagging. and named entity recognition. We apply the pretrained model provided by the author to all testing datasets and follow the same pre-prooe pre-processing and post- pros-procssiig steps for graph re-categorization.
15)
Graphene (our algorithm) The only hyperparameter of the Graphene algorithm is the threshold 0. Following the majority voting strategy (Dong et al. 2020), we set the threshold 0 such that e > 0.5 (where is the number of models in the ensemble). In all experiments, we used a Tesla GPÜ V100 for model training and used 8 CPUs for making an ensemble.
AAaallable for download at
3 Availaale under https///githubb.om//BMM/ hwtp://///raaniiioooaamm-aiiaiie
4 Avaallable at paattp://ggihhee http://hhhhccccettttaaooronaaaaa
'The model "AMR2.0+ can be downloaded from https: /ttgtgttthuutu...e http://ttttuuuuotcco///////////t www/:cco//j/jaeeeec
5

---

4.1.2 Evaluation
We use the serip provided in [Damonte et al., 2017 to calculate the Smatch score (Cai and Knight 2013), the most relevant metric for measuring the similarity between the predictions and the gold AMR graphs. The overall Smatch score can be broken down into different sub-metrics:
Unlabeled (Unl.): Smatch score after removing all edge labels
No WSD (NWSD): Smatch score while ignoring Propbank senses.
NE: F-score on the named entity recognition (name roles)
Wikification (Wiki.): F-score on the wikification (zwiki roles)
Negations (Neg.): F-score on the negation detection (:polarity roles)
Concepts (Con.): F-score on the concept identification task
Reentrancy (Reen.): Smatch computed on reentrant edges only
SRL: Smatch computed on :ARG-i roles only
4.1.3 Datasets
Similarly to (Bevilacqua et al. 2021 our approach. train, development and testing sets and we use them for in-distribution evaluation in Section 442 Furthermore, the models trained on AMR 2.0 training data are used to evaluate out out-of of-distribution prediction on the BIO, the LP and the New3 dataset (See Section 4.3)
J. we use five standard benchmark datasets dat evaluate the statistics of the datasets. AMR 2.0 and AMR 3.0 are divided into
Table 1: Benchmark datasets. All instances of BIO, LP, and New3 are used to test models in out-of- distribution evaluation, For AMR 2.0 and 3.0, the models are trained on the training dataset, validated
on the development dataset. We report results testing sets in the in-distribution evaluation.
Datasets
AMR 2.0
AMR 3.0
BIO Little Prince (LP)
New3
Training
36,521
55,635
n/a
n/a
n/a
Dev
1,368
1,722
n/a
n/a
n/a
Test
1,371
1,898
6,952
1,562
527
4.2 In-distribution evaluation
In the same spirit of [Bevilacqua al. 2021 we evaluate the approaches when training and test data belong to the same domain. Table 2 shows the results of the models on the test split of the AMR 2.0 and AMR 3.0 datasets. The metrics reported for SPRING correspond to the model with the highest Smatch score among the 4 models(the checkpoint plus the 3 models with different random seeds).
For the ensemble approach, we report the result when Graphene is an ensemble of four SPRING checkpoints, denoted as Graphene 4S. The ensemble of all the models including four SPRING checkpoints, APT, T5, and Cai&Lam is denoted as Graphene All. For the AMR 3.0 dataset, the Cam& model is not available so the reported result corresponds to an ensemble of all six models.
We can see that Graphene successfully leverages the strength of all the models and provides better prediction both in terms of the overall Smatch score and sub-metrics. In both datasets, we achieve the state-of of- the-art results with performance gain of 1.6 and 1.2 Smatch points in AMR 2.0 and AMR 3.0 respectively. Table showws that by combining predictions from four checkpoints of the SPRING model, Graphene 4S provides better results than any individual models. The result is improved further when increasing the number of ensemble models, indeed Graphene All improves Graphene 4S further and outperforms the individual models in terms of the overall Smatch score.
4.3 Out-of-distribution evaluation
In contrast to in in-disriibttinn evaluation, we use the models trained with AMR 2.0 data collect AMR predictions for the testing datasets in the domains that differ from the AMR 2.0 dataset. The purpose of the experiment is to evaluate the ensemble approach under out-of distributti settings.

---

Table 2:
Results on the test splits of the AMR 2.0 and AMR 3.0 dataset.
Models
Smatch
Unl.
NWSD
Con.
NE
Neg.
Wiki.
Reen.
SRL
AMR 2.0
SPRING
84.22
87.38
84.72
89.98
90.77
72.65
82.76
74.30
82.89
APT
82.70
86.18
83.23
89.48
90.20
67.27
78.87
73.19
82.01
T5
82.98
86.17
83.43
89.85
90.65
73.43
77.99
72.44
82.02
Cai&Lam
80.15
83.60
80.66
87.39
82.25
78.09
85.36
66.46
77.35
Graphene 4S
84.78
87.96
85.29
90.64
92.19
75.22
83.88
71.42
83.46
Graphene All
85,85
88.68
86.35
91.23
92.30
77.01
84.63
74.49
84,41
AMR 3.0
SPRING
83.25
86.40
83.71
89.38
87.80
72.94
81.22
73.33
81.97
APT
80.57
83.96
81.07
88.38
86.82
68.69
76.88
70.78
80.17
T5
82.17
85.22
82.66
89.03
86.99
72.59
73.78
72.18
81.18
Graphene 4S
83.77
86.89
84.23
90.09
88.27
74.60
81.92
70.22
82.46
Graphene All
84.41
87.35
84.83
90.51
88.64
74.76
82.25
71.93
83.15
Table 3: Results of out-of-distribution evaluation on the BIO, New3, and Little Prince dataset.
Models
Smatch
Unl.
NWSD
Con.
NE
Neg.
Wiki.
Reen.
SRL
BIO
SPRING
60.52
65.33
61.42
67.76
33.92
65.68
3.80
51.19
62.86
APT
51.23
56.27
51.81
58.22
15.68
52.91
3.62
43.53
54.24
T5
58.89
63.86
59.69
66.63
30.42
65.11
2.46
48.56
61.47
Cai&Lam
42.22
49.78
42.85
47.10
5.19
51.42
7.32
39.23
51.00
Graphene 4S
61.51
66.22
62.28
68.48
33.02
68.24
4.46
50.40
63.70
Graphene All
62.29
66.89
63.07
68.64
32.62
69.48
4.54
52.06
64.21
New3
SPRING
74.66
78.99
75.21
82.38
67.52
67.48
67.20
66.47
75.65
APT
71.06
75.92
71.58
80.34
65.65
67.08
57.14
63.02
73.40
T5
73.04
77.30
73.68
82.65
68.24
64.20
56.42
64.65
75.03
Cai&Lam
60.81
66.00
61.29
72.79
45.60
59.57
46.39
57.70
68.87
Graphene 4S
74.84
79.23
75.30
82.56
69.98
69.51
68.34
63.53
76.31
Graphene All
75.60
79.64
76.14
83.08
68.40
69.62
67.98
67.16
76.88
Little Prince
SPRING
77.85
82.31
78.85
84.68
60.53
70.72
60.53
68.28
77.78
APT
75.21
80.07
76.12
85.29
65.15
67.92
69.70
63.28
75.31
T5
77.66
81.99
78.53
85.12
58.06
72.33
59.35
67.03
78.30
Cai&Lam
71.03
75.91
72.07
80.18
22.73
57.51
31.50
59.29
72.02
Graphene 4S
77.91
82.40
78.86
84.91
61.54
73.58
60.65
64.77
78.12
Graphene All
78.54
82.81
79.44
85.52
64.05
75.11
63.45
67.83
78.72
Table 3 shows the results of our experiments. Similar to the in-distribution experiments, the Graphene 4S algorithm achieves better results than other individual models, while the Graphene All approach improves the given results further. achieve the new state-of-the-an results in these benchmark datasets (under distribbutiin settings). This result has important practical implication because in practice is very common not to have labeled AMR data for domain speciffc texts. After all, the labeling task is very dime-demanding. Using the proposed ensemble methods achieve better results with domain- -specff data not included in the training sets.
4.4 How the ensemble algorithm works
explore a few examples to demonstrate the reason why the ensemble method works. Figure 3 shows sentence with a gold AMR in Penman format and list of AMRs corresponding to the
prediction of SPRING [Bevilacqua al., [022]. J, T5 [Raffel et al. 2020 APT (Zhou al. [021]
Cai and Lam Cai and Lam 2020b parser and the ensemble graph given by Graphene.

---

/ contrast-01
contrast-01 :ARG1 / want-01
JARG1 (w want-01
:ARGO (t/ they)
:ARGO (x2 they))
:ARG1 [m money))
:ARG1 (23 moneyl)
ARG2 (w2 want-O1
:ARG2 (z4 / want 01
-polarity-
:polarity- :ARGO
ARGDD
:ARG1 (f/ facell)
:ARG1 (25 facelll
Gold
Graphene
(:0 and
contrast-01
{c contrast-01 01
(cO / contrast-01
1op1 (z1 waant-01
:ARG1 (w/ want
:ARG1 (w want-01
:RGG (c1 want-01
:ARGO (z2 they)
:ARGO (t/ they)
:RGa (t / they)
:ARGO (c4 / /they)
:RRG1 (z3 money!)
|ARG1 (m / money])
ARG1 / money))
:ARG1 (c3 / moneyl)
:ARG2 (w2 want-81
:op2 (x4 want-01
JRG2 (w2 want-01
:AGG t
:RG2 (c2 want-01
spolarity-
:polarity-
:RGG (f / face)
:ARG
:ARGO x2
:ARGO
:polarity -})
face)))
:ARG1 (25 facelll
:ARG1 (f face)l)
SPRING
Figure 3:
The gold AMR and the ensemble AMR graph of SPRING, T5, APT and Cai&Lam using the Graphene algorithm for the sentence "They want money, not the face".
In this particular example, with the sentence "They want money, not the face", the AMR prediction from SPRING is inaccurate. Graphene corrects the prediction thanks to the votes given from the other models. In particular, the label and of the root node 2g of SPRING prediction was corrected to controst - 01 because T5, APT and Cai& parsers all vote for contrast 01. On the other hand, the labels opl and op2 of the edges (2) and (2u, 24) were modified to have the correct labels: ARGI and : ARG2 accordingly thanks to the votes from the other models. We can also see that even though the Cai&Lam method misses polarity prediction, since the other models prediet polarity correctly, the ensemble prediction does not inherit this mistake. Putting everything together, the prediction from Graphene perfectly matches with the gold AMR graph in this example.
Table 4: The average total support and Smatch score of SPRING, Graphene with SPRING as pivot and Graphene respectively. The support is highly correlated with Smatch score.
AMR 2.0
AMR 3.0
BIO
LP
New3
Sup.
Smat.
Sup.
Smat.
Sup.
Smat.
Sup.
Smat.
Sup.
Smat.
SPRING SPR. pivot
170.15
84.08
136.90
83.14
166.86
60.52
69.33
77.85
118.27
74.66
172.70
84.70
139.42
83.73
169.97
61.56
70.97
78.22
120.85
74.83
Graphene
175.73
85.85
142.07
84.43
179.38
62.29
72.64
78.54
123.62
75.60
The Graphene algorithm searches for the graph that has the largest support from all individual graphs. One question that arises from this is whether the support is correlated with the accuracy of AMR parsing. Table 4 showws the support and the Smatch score of three models in the standard benchmark datasets. The first model SPRING, while the second one denoted as SPR. pivor uses SPRING prediction as pivot. The last model corresponds to the Graphene algorithm. Since Graphene looks for the best pivot to have suppor-d ensemble graphs, the total supports of the Graphene predictions are larger than the SPR. pivot predictions. From the table, we can also see that the total support is highly correlated to the Smatch score. Namely, Graphene has higher support in all the benchmark datasets and higher Smatch score than SPR. pivot. This experiment suggests that by optimizing the total support we can obtain the ensemble graphs with higher Smatch score.
Related work
Ensemble learning. Ensemble learning is popular machine learning approach that combines predictions from different learners to make a more robust and more accurate prediction. Many ensembling approaches have been proposed, such as bagging [Breiman] 1996) or boosting (Schapire and Freund (2013), the winning solutions in many machine learning competitions [Chen and Guestrin 2016). These methods are proposed mainly for regression or classification problems. Recently,
8

---

structure prediction emerges as an important research problem, it is important to study ensemble methods for combining structure predictions.
Ensemble structure prediction. Previous studies have explored various ensemble learning ap- proaches for dependency and constituent parsing: [Sagae and Lavie, 2006] proposes reparsing framework that takes the output from different parsers and maximizes the number of votes for well-formed dependency or constituent structure; [Kuncoro et al. 2016) uses minimum Bayes risk inference to build consensus dependency parser from an ensemble of independently trained greedy LSTM transition-based parsers with different random initializations. Note that syntactic tree is special graph structure in which nodes for a sentence from different parsers are roughly the same. In contrast, we propose approach to ensemble graph predictions in which both graph nodes and edges can be different among base predictions.
Ensemble methods for AMR parsing. Parsing text to AMR is an important research problem. approaches in AMR parsing divided into three categories. Sequence sequence
models [Bevilaqqua et al. [202]] Konstas et al. 2017 Van Noord and Bos 2017 Xu et al. 2020)
consider the AMR parsing as machine translation problem that translates texts AMR graphs. The
transition-ben baase methods (Zhou et al., 2021] predicts a sequence of actions given the input text, and then the action sequence is turned into AMR graph using oracle decoder. Lastly, graph-based methods (Cai and Lam 2020b) directly construct the AMR graphs from textual data. All these each other and thus ensemble methods can leverage the strength of these methods to create a better prediction, as demonstrated in this paper. Ensemble of AMR predictions from single type of model is studied in [Zhou et al. 2021], by combining predictions from three different model's checkpoints it gains performance improvement in the final prediction. However, ensemble in sequential decoding requires that all predictions are from the same type of models. It is not applicable for cases when the predictions are from different types of models such as seq2seq, pansition-basse or graph-based models. In contrast to that approach, our algorithm model-agnostic, i.e. it can combine predictions from different models. In our experiments, we have demonstrated the benefit of combining predictions from different models, with additional gains in performance compared to the ensemble of predictions from single model's checkpoints.
methods are complementary
Comparison to Bazdins et al. Barzdins and Gosko 2016) proposed character-level based neural method for parsing texts into AMRs. To improve the robustness of the parser, an ensemble technique which selects among the prediction graphs the one that has the highest average SMATCH when it is compared against the other predictions was proposed. The key difference between Barzdins' and our approach is that while our solution modifies the predictions create new prediction candidates for ensemble prediction, Barzdins' approach only selects a prediction among existing predictions. For a detailed discussion with additional experimental results please refer to Lam al. 2021
6 Conclusions and future work
In this paper, we formulate the graph ensemble problem, study its computational intractability, and provide an algorithm for constructing graph ensemble predictions. We validate our approach with AMR parsing problems. The experimental results show that the proposed approach outperforms the previous statc-of-the-art AMR parsers and achieves new state-of-the-art results in five different benchmark datasets. We demonstrate that the proposed ensemble algorithm not only works well for in-distribution but also for out-of-distro disttriibbttion evaluations. This result has a significant practical impact, especially when applying the proposed method domain-specific texts where training data is not available. Moreover, our approach is model-agnostic, which means that it can be used to combine predictions from different models without the requirement of having an access to model training. In general, our approach provides the basis for graph ensemble, studying classical ensemble techniques such as bagging, boosting, or stacking for graph ensemble a promising future research direction that is worth considering to improve the results further.
We would like to thank Juri Opitz for pointing us to missing references as very useful feedback for our published work.

---

References
AMRLIB data preprocessing prepooessingghttss/gghhhhniinn http/////ggg.boooooooooommommmae
AMR benchmark datasets. hupp://amiissiccdddddvvvlood...
English Web Treebank dataset: https://eeaaogg.k.c.uuuoccdddddd URL htthttps://aaaalgg
1dc. upwnn aaraaneedu/LLii L.C2LLD20012113
Laura Bahiense, Gordana Manić, Breno Piva, and Cid C De Souza. The maximum common edge subgraph problem: A polyhedral investigation. Discrete Applied Mathematics, 160(18))2523-2541 2012.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Kochn, Martha Palmer, and Nathan Schneider. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186, 2013.
Guntis Barzdins and Didzis Gosko. Riga semeval-201e task 8: Impact of smatch extensions and character-level neural translation on amr parsing accuracy. arXiv preprint arXiv:/6044 0/278, 2016.
Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. One SPRING to rule them both: Symmetric AMR semantic parsing and generation without a complex pipeline. In Proceedings of AAAI, 2021.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-144 1996.
Deng Cai and Wai Lam. AMR parsing via rraph-sequence iterative inference. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1290-1301, Online, July 2020a. Association for Computational Linguisties. doi 10.
URL
https: www.............bb...... org/anthology/ 2020 acl main. 119
Deng Cai and Wai Lam. AMR parsing via graph- sequence iterative inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1290-1300 Online, July 2020b. Association for Computational Linguistics. doi 10. 1/2533//122222.cc-mmaiin https::/ htppp://wwwww //www. aclweb. org/anthology 2020. acl-main. 119
URL
Shu Cai and Kevin Knight. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748-752, Sofia, Bulgaria, August 2013. Association for Computational Linguisties. URL, httpt:://///ttt http://wwwwwwwwwbbbe.e aclweb. org/ anthologgy/P13221...2
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2016.
Marco Damonte, Shay B. Cohen, and Giorgio Satta. An incremental parser for Abstract Mean- ing Representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 536-546, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https: http:/wwwwlllllc//. aclweb. org/ anthology/E17-1100
Pedro Domingos. unified bias-variance decomposition. In Proceedings of 17th International Conference on Machine Learning, pages 231-238, 2000.
Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and Qianli Ma. A survey on ensemble learning. Frontiers of Computer Science, 14(2):241-255 2020.
Pavan Kapanipathi, Ibrahim Abdelaziz, Srinivas Ravishankar, Salim Roukos, Alexander Gray, Ra- mon Astudillo, Maria Chang, Cristina Cornelio, Saswati Dana, Achille Fokoue, et al. Question answering over knowledge bases by leveraging semantic parsing and neuro-symbolic reasoning. arXiv preprint arXiv:2012.0170 2020.
Joannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. Neural AMR: Sequence-to-sequeeeee models for parsing and generation. ACL, 2017.
10

---

Adhiguna Kuncoro. Miguel Ballesteros, Lingpeng Kong. Chris Dyer, and Noah A. Smith. Distilling an ensemble of greedy dependency parsers into one MST parser. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1744-1753, Austin, Texas, November 2016. Association for Computational Linguistics. doi 16011866655/11111-111880
URL https:: //tttps://ww..... aclweb. org/ a -180/
Hoang Thanh Lam, Gabriele Picco, Yufang Hou, Young-Suk Lee, Lam M. Nguyen. Dzung T. Phan, Vanessa López, and Ramon Fernandez Astudillo. Ensembling graph predictions for amr parsing, 2021. URL http:: /ttp:///axxiii org
Young-Suk Lee, Ramon Fernandez Astudillo, Tahira Naseem, Revanth Gangi Reddy, Radu Florian, and Salim Roukos. Pushing the limits of AMR parsing with self-learning. In Findings of EMNLP 2020, pages 3208-3214, 2020.
Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih, Efficient one-pass end-to-end entity linking for questions. In EMNLP, 2020.
Jungwoo Lim, Dongsuk Oh, Yoonna Jang, Kisu Yang, and Heuiseok Lim. I know what you asked: Graph path learning using AMR for commonsense reasoning. In Proceed- ings of the 28th International Conference Computational Linguistics, pages 2459-2471, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. main.222
doi
URL https: btopss//wwwee http://wwwwwlww...o////
aclweb. org/ anthology/h222
coling-mane
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
http://jmlr
org/ pttpp/ppppp////0.00/0// 074. html
Kenji Sagae and Alon Lavie. Parser combination by reparsing. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 129. 132, New York City, USA, June 2006. Association for Computational Linguistics. URL https
www..aclweb.co worclleeb.rrgarniie werg/annhollgyy/Wen 200-0loooyy//-2333333
Robert E Schapire and Yoav Freund. Boosting: Foundations and algorithms. Kybernetes, 2013.
Giorgio Valentini and Thomas G Dietterich. Bias-variance analysis of support vector machines for the development of svm-based ensemble methods. Journal of Machine Learning Research, 5(Jul): 725-775, 2004.
Rik Van Noord and Johan Bos. Neural semantic parsing by character-based translation: Experiments with abstract meaning representations. arXiv preprint arXiv: 1705:09955099 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is you need. Advances in Neural Information Processing Systems, pages 9098-6008. 2017.
Dongqin Xu, Junhui Li, Muhua Zhu, Min Zhang, and Guodong Zhou. Improving AMR parsing with sequence-tor pertennce-t--eqqueee so-seneence pre-trainn -raiirain EMNLP, 2020.
Jiawei Zhou, Tahira Naseem, Ramon Fernandez. Astudillo, and Radu Florian. AMR parsing with action-pointer transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies (NAACL 2021). pages 5585-5598, 2021.
11


Reviews:
[{"review_id": "tWbjcj-hC_B", "reviewer": "Reviewer_awK5", "summary": "The authors present a heuristic to combine several graphs, obtained by specialised linguistic parsers, to obtain a single more accurate parse graph. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main concern with this work lies in the benchmarking design. Specifically:  \n1. Size of the improvement:  the authors claim that since their proposal is deterministic they do not need to report the variance in the experimental results, however one should simply sample the training set to observe the variability of the proposed approach. All improvements reported in the result section are around the 1% mark, and is therefore difficult to exclude a random effect. \n2. No base line is offered: for example one could propose the \"median\" graph as a representative of the set, or one could select the best graph by formulating a re-ranking task (i.e. learn a preference score on the set of alternatives). The efficacy of the approach (i.e. deriving an adjusted graph from a set of graphs) is not compared against any approach or baseline that has in input a set of graphs and outputs a single graph. \n3. The algorithm (which is heavily based on prior work to solve the vertex matching problem) does not offer any guarantee of being able to preserve some characteristics of the graph set: for example if one is working on trees, the algorithm is not guaranteed to output a tree, if one was working on molecular graphs, the algorithm would not yield valid molecular graphs (the valence would not be guaranteed). However no mention of the necessity to check or compensate for the feasibility  of the resulting graph is mentioned in this work. It is not clear in this specific domain what characteristics of the graphs are responsible for the efficiency of the algorithm, it is also not clear if this approach could be applied effectively to other domains and hence be of broader interest for the community. \n4. Not self contained: the authors do not provide any description of what an Abstract Meaning Representation graph is (apart from a single example figure); the datasets are not described; even the evaluation metric uses terms that are domain specific and not introduced. \n\n###After the authors' reply\nMany thanks to the authors for the their response. Given the new experimental data provided by the authors I am inclined to improve the score. While the issues at points 1 and 2 regarding the effect size of the improvement achieved by their proposal and the baseline comparisons are somewhat addressed by the additional experiments, the concern regarding the generality of the method is not, as it is not sufficient to say that one can always introduce constraints to select the the non-violating examples (in fact in many important cases it can become exponentially hard to find viable candidates and hence a generate and test strategy would became infeasible). The generality problem could be easily addressed by specifying that the proposal is tailored for AMR graphs rather than for general graphs (given also that all experiments are in this very domain).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "plKC1qmvEOm", "reviewer": "Reviewer_7y9L", "summary": "The paper presents an ensemble approach for graph predictions.\nIt consists in an algorithm that finds the graph that have the most nodes and edges matching, among different graphs produced by different systems or different checkpoints of the same system.\nThe authors tested the system on AMR graphs produced by state of the art AMR parsers.\nEven if the problem is NP-hard the authors made it tractable using heuristics\n\nThe proposed ensemble models outperform the single-model approaches the authors compare with in two used AMR benchmarks in both in-distribution and out-of-distribution setting.\n\nUPDATE\nThanks for the complete response, I think the paper is much stronger now.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is easy to read and the approach is simple (which is good, it only has one hyperparameter) and effective. It will be likely used to boost performances of the next generation of AMR-parsers.\n\nI liked the explanation the authors give on how the algorithm work in Section 4.4.\n\nSince the authors only tested the model on AMR graphs, they should mention it in the title. The approach is clearly general and it would be interesting see its application on other graph prediction tasks.\n\nThe main thing missing in this paper in my opinion is an ensemble baseline.\nFor example, the transition base parser could be easily ensembled.\nAnother type of ensembling could be stacking in sequence to sequence, where the output of a model is used as input for another model.\n\nI understand that the way the authors proposed to merge graphs is novel, but I would also like to see a comparison with more \"standard\" ensembling approaches.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no discussion on the limitations of the work and societal impact, but I am not sure it can apply to this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "LnE2xito86n", "reviewer": "Reviewer_qSVh", "summary": "Ensemble learning is a popular machine learning practice, in which predictions from multiple models are blended to create a new one that is usually more robust and accurate. Lots of work show the benefits of ensemble learning on regression or classification problems but there is a lack of investigation on the problem of graph ensemble which is more complicated. The paper collects the predictions from four state-of-the-art AMR parsers and shows the proposed ensemble method's effectiveness on the standard benchmark datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The motivation of the paper to study ensemble learning over graphs (specifically on the AMR task) is quite clear and the tackling problem is very important and crucial. \n\n2. The proposed method which aims to figure out the largest supported common subgraph makes sense to me. \n\n3. The experimental results of this paper show massive numbers to show the effectiveness of the proposed method in ensembling graphs. The performance improvement from the proposed method is consistent and promising.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. I didn't see too many experiments on model analysis that could reveal the insights of the proposed model.\n\n2. Minor: The title of section 3, 'Please correct me if I am wrong', is quite interesting to me. It reflects the model's behavior and explicitly concludes the proposed method in a straightforward way. However, there is one more meaning that corrects you if you have something wrong in this section. I am not sure if you imply this meaning or not. (Although I really like this title, I suggest you use a conventional title to avoid any ambiguity)", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "LPtR2OFlMW-", "reviewer": "Reviewer_kDjg", "summary": "The authors proposed a new method for ensemble graph prediction. They formalized this problem as \"mining the largest subgraph that is the most supported by a collection of graph predictions\". Furthermore, they introduced an efficient heuristic algorithm to approximate the optimal solution given that the original problem is NP-Hard. They tested the algorithm in the AMR parsing problems. The experimental results demonstrate that the proposed approach can combine the strength of the state-of-the-art AMR parsers and the ensemble model outperforms individual models in five standard benchmark datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is innovative as previous work focuses on dealing with multiple predictions where nodes are the same but the current work deals with the situation where both graph nodes and edges can be different among base predictions. Furthermore, previous ensemble approaches require models to be of the same type such as sequence models, transition-based or graph-based models. But the current approach is model-agnostic in the sense that it can combine predictions from different models.\n\n The submission is relatively sound. The authors demonstrate that their ensemble approach outperforms individual models not only in the In-distribution setting but also in the Out-of-distribution setting. I do have two concerns. First, in section 4.4, the authors claim that \"From Table(4), the total support is highly correlated to the Smatch score. \" Without seeing the statistical test of the correlation coefficients, I'm not entirely convinced that the conclusion is true. Even in Table 4, there're cases in which the average support is high but the Smatch score is low (e.g., BIO in row 3). It would be nice if the author could report correlation coefficients and the p-values to support their claim. Also, the authors mention that they set the hyperparameter theta so that theta divided by the number of models is greater than 0.5 (Line 180). I was wondering how changing this hyperparameter would affect the results and whether tuning this hyperparameter would lead to any further improvement.\n\nThe submission is clearly written and well organized. The results are important in the sense that it provides a way to obtain the ensemble prediction from multiple models, even when the models being combined have different architectures. I was wondering if the authors have any intuitions on how to generalize their work to graphs more complicated than the AMR parsing trees. It seems that their heuristic algorithms only work for a subset of graphs. Therefore naming the paper as \"ensemble graph prediction\" seems to be a bit overselling.\n\n---------------\n\nUpdated: I've read the authors' responses to my review as well as the ones to the other reviewers'. I think their responses address my concerns, especially the additional details on the correlation analysis and the justification for choosing the hyper-parameter theta. I don't have a strong preference for adding \"AMR\" to the title, since I'd like to keep the paper appealing to a broad audience. I'm looking forward to seeing more work in the future which could verify that this ensemble approach is not limited to AMR graphs but can be applied to other graphs as well. Given the improvements and the limitation of the paper, I decided to keep my score unchanged as it reflects my best judgment of the work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There's no foreseeable negative societal impact of this work.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper details the problem of (labeled) graph ensembling, in particular for the task of semantic (AMR) parsing, and proposes a heuristic algorithm for solving it. 

This is a challenging problem of interest. The reviewers find the work well-motivated and well explained, and raised some issues about comparisons, which the authors satisfactorily addressed with additional experiments in the discussion period. One outstanding point would be to clarify early on the scope of graphs for which the method works, more clearly. (However, the reviewers do not feel strongly that title should mention AMR, as the work is indeed more general.) Indeed, in some settings like link prediction, averaging of arc weights can be much simpler, and it might not be clear for all readers right away why and when the problem gets harder, based on the readers' background.

The reviewers made plenty of editing suggestions that I strongly encourage the authors to implement. In addition, I would like to add a few:

 - while I agree with the reviewer that it's better to rename the title of Section 3, the idiom provides helpful context so I would encourage you to keep it in the body (if not in the section title.)
 - line 83 and on: `$support$` should be styled as `$\operatorname{support}$` or defined as a macro with `$\DeclareMathOperator{\support}{support}$`.
 - line 146, the subscript `$g_{pivot}$` should be `$g_\text{pivot}$` -- same everywhere else.
 - all tables: try to align numbers on their decimal point (e.g. by right-alinging with a tabular font.)
 - Check the capitalization and formatting in your references. (e.g. "Amr" should be "AMR", "Machine learning" should be "Machine Learning" etc.)
Decision: Accept (Poster)