Ensembling Graph Predictions for AMR Parsing

1 Introduction . Ensemble learning is a popular machine learning practice , in which predictions from multiple models are blended to create a new one that is usually more robust and accurate . Indeed , ensemble methods like XGBOOST are the winning solution in many machine learning and data science competitions [ Chen and Guestrin , 2016 ] . A key reason behind the successes of the ensemble methods is that they can combine the strength of different models to reduce the variance and bias in the final prediction [ Domingos , 2000 , Valentini and Dietterich , 2004 ] . Research in ensemble methods mostly focuses on regression or classification problems [ Dong et al. , 2020 ] . Recently , in many machine learning tasks prediction outputs are provided in a form of graphs . For instance , in Abstract Meaning Representation ( AMR ) parsing [ Banarescu et al. , 2013 ] , the input is a fragment of text and the output is a rooted , labeled , directed , acyclic graph ( DAG ) . It abstracts away from syntactic representations , in the sense that sentences with similar meaning should have the same AMR . Figure 1 shows an AMR graph for the sentence You told me to wash the dog where nodes are concepts and edges are relations . AMR parsing is an important problem in natural language processing ( NLP ) research and it has a broad application in downstream tasks such as question answering [ Kapanipathi et al. , 2020 ] and common sense reasoning [ Lim et al. , 2020 ] . Recent approaches for AMR parsing leverage the advances from pretrained language models [ Bevilacqua et al. , 2021 ] and numerous deep neural network architectures [ Cai and Lam , 2020a , Zhou et al. , 2021 ] . Unlike methods for ensembling numerical or categorical values for regression or classification problems where the mean value or majority votes are used respectively , the problem of graph ensemble is more complicated . For instance , Figure 2 show three graphs g1 , g2 , g3 with different 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . structures , having varied number of edges and vertices with different labels . In this work , we formulate the ensemble graph prediction as a graph mining problem where we look for the largest common structure among the graph predictions . In general , finding the largest common subgraph is a well-known computationally intractable problem in graph theory . However , for AMR parsing problems where the AMR graphs have labels and a simple tree-alike structure , we propose an efficient heuristic algorithm ( Graphene ) to approximate the solution of the given problem well . To validate our approach , we collect the predictions from four state-of-the-art AMR parsers and create new predictions using the proposed graph ensemble algorithm . The chosen AMR parsers are the recent state-of-the-art AMR parsers including a seq2seq-based method using BART [ Bevilacqua et al. , 2021 ] , a transition-based approach proposed in [ Zhou et al. , 2021 ] and a graph-based approach proposed in [ Cai and Lam , 2020a ] . In addition to those models , we also trained a new seq2seq model based on T5 [ Raffel et al. , 2020 ] to leverage the strength of this pretrained language model . The experimental results show that in five standard benchmark datasets , our proposed ensemble approach outperforms the previous state-of-the-art models and achieves new state-of-the-art results in all datasets . For example , our approach achieves new state-of-the-art results with 1.7 , 1.5 , and 1.3 points better than prior arts in the BIO ( under out-of-distribution evaluation ) , AMR 2.0 , and AMR 3.0 datasets respectively . This result demonstrates the strength of our ensemble method in leveraging the model diversity to achieve better performance . An interesting property of our solution is that it is model-agnostic , therefore it can be used to make an ensemble of existing model predictions without the requirement to have an access to model training . Source code is open-sourced1 . Our paper is organized as follows : Section 2 discusses a formal problem definition and a study on the computational intractability of the formulated problem . The graph ensemble algorithm is described in Section 3 . Experimental results are reported in Section 4 while Section 5 discusses related works . The conclusion and future work are discussed in Section 6 . 2 Problem formulation . Denote g = ( E , V ) as a graph with the set of edges E and the set of vertices V . Each vertex v ∈ V and edge e ∈ E is associated with a label denoted as l ( v ) and l ( e ) respectively , where l ( . ) is a labelling function . Given two graphs g1 = ( E1 , V1 ) and g2 = ( E2 , V2 ) , a vertex matching ϕ is a bijective function that maps a vertex v ∈ V1 to a vertex ϕ ( v ) ∈ V2 . Example 1 . In Figure 2 , between g1 and g2 there are many possible vertex matches , where ϕ ( g1 , g2 ) = [ 1 → 3 , 2 → 2 , 3 → 1 ] is one of them ( which can be read as the first vertex of g1 is mapped to the third vertex of g2 and so forth ) . Notice that not all vertices v ∈ V1 has a match in V2 and vice versa . Indeed , in this example , the fourth vertex in g2 does not have a matched vertex in g1 . Given two graphs g1 , g2 and a vertex match ϕ ( g1 , g2 ) , support of a vertex v with respect to the matching ϕ , denoted as sϕ ( v ) , is equal to 1 if l ( v ) = l ( ϕ ( v ) ) and 0 otherwise . Given an edge e = ( v1 , v2 ) the support of e with respect to the vertex match ϕ , denoted as sϕ ( e ) , is equal to 1 if l ( e ) = l ( ( ϕ ( v1 ) , ϕ ( v2 ) ) ) and 0 otherwise . 1https : //github.com/IBM/graph_ensemble_learning Example 2 . In Figure 2 , for the vertex match ϕ ( g1 , g2 ) = [ 1 → 3 , 2 → 2 , 3 → 1 ] , the first vertex in g1 and the third vertex in g2 shares the same label A , therefore the support of the given vertex is equal to 1 . On the other hand , the third vertex in g1 and the first vertex in g2 does not have the same label so their support is equal to 0 . Between two graphs , there are many possible vertex matches , the best vertex match is defined as the one that has the maximal total vertex support and edge support . In our discussion , when we mention a vertex match we always refer to the best vertex match . Denote G = { g1 = ( E1 , V1 ) , g2 = ( E2 , V2 ) , · · · , gm = ( Em , Vm ) } as a set of m graphs . Given any graph g = ( E , V ) , for every gi denote ϕi ( g , gi ) as the best vertex match between g and gi . The total support of a vertex v ∈ V or an edge e ∈ E is defined as follows : • support ( e ) = ∑m i=1 sϕi ( e ) • support ( v ) = ∑m i=1 sϕi ( v ) Given a support threshold θ , a graph g is called θ-supported by G if for any node v ∈ V or any edge e ∈ E , support ( v ) ≥ θ and support ( e ) ≥ θ . Example 3 . In Figure 2 , graph g is θ-supported by G = { g1 , g2 , g3 } where θ = 2 . Intuitively , an ensemble graph g should have as many common edges and vertices with all the graph predictions as possible . Therefore , we define the graph ensemble problem as follows : Problem 1 ( Graph ensemble ) . Given a support threshold θ and a collection of graphs G , find the graph g that is θ-supported by G and has the largest sum of vertex and edge supports . Theorem 1 . Finding the optimal θ-supported graph with the largest total of support is NP-Hard . Proof . We prove the NP-Hardness by reduction to the Maximum Common Edge Subgraph ( MCES ) problem , which is known to be an NP-Complete problem [ Bahiense et al. , 2012 ] . Given two graphs g1 and g2 , the MCES problem finds a graph g that is a common subgraph of g1 and g2 and the number of edges in g is the largest . Consider the following instance of the Graph Ensemble problem with θ = 2 , and G = { g1 , g2 } created from the graphs in the MCES problem . Assume that all vertices and all edges of g1 and g2 have the same label A . Since θ = 2 , a θ-supported graph is also a common subgraph between g1 and g2 and vice versa . Denote gs and ge as the common subgraph between g1 and g2 with the largest support and the largest common edge , respectively . We can show that gs has as many edges as ge . In fact , since gs is the largest supported common subgraph there is no vertex v ∈ ge such that v ̸∈ gs because otherwise we can add v to gs to create a larger supported graph . For any edge e = ( v1 , v2 ) ∈ ge , since both vertices v1 and v2 also appear in gs , the edge e = ( v1 , v2 ) must also be part of gs otherwise we can add this edge to gs to create a subgraph with a larger support . Therefore , gs has as many edges as ge , which is also a solution to the MCES problem . 3 Graph ensemble algorithm . In this section , we discuss a heuristic algorithm based on the strategy “ Please correct me if I am wrong ! '' to solve Problem 1 . The main idea is to improve a pivot graph based on other graphs . Specifically , starting with a pivot graph gi ( i = 1 , 2 , · · · , m ) , we collect the votes from the other graphs for every existing vertex and existing/non-existing edges to correct gi . We call the proposed algorithm Graphene which stands for Graph Ensemble algorithm . The key steps of the algorithm are provided in the pseudo-code in Algorithm 1 . Algorithm 1 : Graph ensemble with the Graphene algorithm . Input : a set of graphs G = { g1 , g2 , · · · , gm } and the support threshold θ Output : an ensemble graph ge Algorithm : Graphene ( G , θ ) for i← 1 to m do gpivot ← gi V ← Initialise ( gpivot ) for j ← 1 to m do if j ̸= i then V ← V ∪ getVote ( ϕ ( gpivot , gj ) ) end gei ← Filter ( V , θ ) end ge ← the graph with the largest support among ge1 , · · · , gem Return ge For example , in Figure 2 , the algorithm starts with the first graph g1 and considers it as a pivot graph gpivot . In the first step , it creates a table to keep voting statistics V initialized with the vote counts for every existing vertex and edge in gpivot . To draw additional votes from the other graphs , it performs the following subsequent steps : • Call the function ϕ ( g1 , gi ) ( i = 2 , 3 , · · · , m ) to get the best bijective mapping ϕ between the vertices of two graphs g1 and gi ( with a little bit abuse of notation we drop the index i from ϕi when gi and gpivot are given in the context ) . For instance , the best vertex match between g1 and g2 is ϕ = 1→ 3 , 2→ 2 , 3→ 1 because that vertex match has the largest number of common labeled edges and vertices . • Enumerate the matching vertices and edges to update the voting statistics accordingly . For instance , since the vertex 3 in g1 with label B is mapped to the vertex 1 in g2 with label C , a new candidate label C is added to the table for the given vertex . For the same reason , we add a new candidate label Z for the edge ( 1 , 2 ) . For all the other edges and vertices where the labels are matched the votes are updated accordingly . Once the complete voting statistics V is available , the algorithm filters the candidate labels of edges and vertices using the provided support threshold θ by calling the function Filter ( V , θ ) to obtain an ensemble graph gei . For special cases , when disconnected graphs are not considered as a valid output , we keep all edges of the pivot graph even its support is below the threshold . On the other hand , for the graph prediction problem , where a graph is only considered a valid graph if it does not have multiple edges between two vertices and multiple labels for any vertex , we remove all candidate labels for vertices and edges except the one with the highest number of votes . Assume that the resulting ensemble graph that is created by using gi as the pivot graph is denoted as gei . The final ensemble graph g e is chosen among the set of graphs ge1 , g e 2 , · · · , gem as the one with the largest total support . Recall that ϕ ( gpivot , gi ) finds the best vertex match between two graphs . In general , the given task is computationally intractable . However , for labeled graphs like AMR a heuristic was proposed [ Cai and Knight , 2013 ] to approximate the best match by a hill-climbing algorithm . It first starts with the candidate with labels that are mostly matched . The initial match is modified iteratively to optimize the total number of matches with a predefined number of iterations ( default value set to 5 ) . This algorithm is very efficient and effective , it was used to calculate the Smatch score in [ Cai and Knight , 2013 ] so we reuse the same implementation to approximate ϕ ( gpivot , gi ) ( report on average running time can be found in the supplementary materials ) .

Summary: Ensemble learning is a popular machine learning practice, in which predictions from multiple models are blended to create a new one that is usually more robust and accurate. Lots of work show the benefits of ensemble learning on regression or classification problems but there is a lack of investigation on the problem of graph ensemble which is more complicated. The paper collects the predictions from four state-of-the-art AMR parsers and shows the proposed ensemble method's effectiveness on the standard benchmark datasets.