Evaluate AMR Graph Similarity via Self-supervised Learning
Year: 2023

Abstract:
In work on AMR (Abstract Meaning Representation), similarity metrics are crucial as they are used to evaluate AMR systems such as AMR parsers. Current AMR metrics are all based on nodes or triples matching without considering the entire structures of AMR graphs. To address this problem, and inspired by learned similarity evaluation on plain text, we propose AMRSim, an automatic AMR graph similarity evaluation metric. To overcome the high cost of collecting human-annotated data, AMRSim automatically generates silver AMR graphs and utilizes self-supervised learning methods. We evaluated AMRSim on various datasets and found that AMRSim significantly improves the correlations with human semantic scores and remains robust under diverse challenges. We also discuss how AMRSim can be extended to multilingual cases.1

heading: 1 Introduction
text: An Abstract Meaning Representation (AMR; Banarescu et al., 2013) is a rooted, directed graph where nodes represent concepts and edges correspond to relations of concepts. The goal of an AMR metric is to evaluate the similarities of pairs of AMR graphs so that it can be used to evaluate the outputs of AMR generators such as parsers. Therefore, a good AMR metric is crucial for the design and evaluation of AMR parsers. However, the research on AMR metrics has so far lagged far behind the work on AMR parsing.
Current AMR metrics either transfer AMR graphs to triples and consider the one-to-one matching of variables (Cai and Knight, 2013) or linearize AMR graphs as sequences and calculate n-gram matching (Song and Gildea, 2019). These metrics fail to consider the entire AMR structure and lack flexibility, resulting in poor correlation with human annotations.
1The code and datasets can be found at https://github. com/zzshou/AMRSim.
Inspired by plain-text automatic similarity assessment methods that encode sentences into latent semantic representations to measure a similarity of the two representations (Reimers and Gurevych, 2019), we propose to learn the automatic assessment of AMR graph similarity through a similar pipeline. Our proposed metric, called AMRSim, adopts the pre-trained language model BERT as the backbone and incorporates GNN adapters to capture the structural information of AMR graphs. To overcome the high cost of collecting training data, we utilize self-supervised learning methods. The training objective is to maximize the dot product between positive embeddings and to minimize the dot product between different encodings. In contrast to one-to-one matching metrics, our AMRSim is alignment-free by computing the cosine of contextualized token embeddings. The prediction process can be considerably accelerated by leveraging GPUs.
We experiment with AMRSim on the transformed STSB (Baudiš et al., 2016) and SICK (Marelli et al., 2014) datasets, which contain pairs of AMR graphs and corresponding similarity scores. Our experiments demonstrate that AMRSim achieves prominent improvements in correlation with human annotations. In further analysis, AMRSim retains the highest performance under various challenges compared to previous metrics. We also explore the potential of extending our metric to multilingual cases, taking into account the generic nature of the transformer structure and the fact that the pipeline of AMRSim is not constrained to any particular language.
The remaining paper is structured as follows. Section 2 gives an overview of existing AMR metrics; Section 3 introduces our proposed metric, AMRSim; Section 4 includes experimental settings and main results; We analyze the robustness and the efficiency of our metrics in Section 5 and conclude in Section 6.
16112

heading: 2 Existing Metrics
text: AMR similarity metrics play a vital role in evaluating the performance of AMR parsers. However, computing the degree of similarity between AMR graphs is not trivial. In this section, we summarise the existing AMR metrics.
SMATCH SMATCH (Cai and Knight, 2013) evaluates the overlap of structures as the similarity score. Each AMR graph is viewed as a conjunction of triples. SMATCH tries to find a one-to-one matching of variables that maximizes the numbers of exact matches of triples through the hill climbing method in a greedy style. The alignment process, which has been proved as NP-hard, limits the efficiency of SMATCH, especially as the sizes of AMR graphs increase. Another weakness is that the structural matching is insufficient for meaning similarity assessment and fragile in concept synonym replacement and structure deformation (Blloshmi et al., 2020; Opitz et al., 2021).
S2MATCH To yield a better variable alignment, Opitz et al. (2020) propose S2MATCH by allowing soft semantic match instead of matching only identical triples in SMATCH. Accounting for cosine similarity of concepts helps to assess the similarity of graphs, however, computational limitations and structure deformation confusion in SMATCH have not been addressed.
SEMA SMATCH adds a TOP relation to the structure, but SEMA (Anchiêta et al., 2019) argues that this addition can potentially distort the evaluation. Therefore, they ignored triples identifying the graph top. Furthermore, instead of computing the maximum score like SMATCH, SEMA works as a breadth-first search and produces a deterministic result, making it faster than one-to-one matching of variable employed by SMATCH.
SEMBLEU BLEU (Papineni et al., 2002), which assesses text quality by comparing n-grams, is frequently adopted in machine translation evaluation. To extend BLEU for matching AMR graphs, SEMBLEU (Song and Gildea, 2019) linearizes AMR graphs through breadth-first traversal and extracts n-gram for comparison. The metric is alignmentfree and thus computationally efficient. Experimental results show that SEMBLEU achieved slightly higher consistency with human judgment than SMATCH.
WWLK Besides treating AMR graphs as triples and linearized grams, Weisfeiler-Leman AMR similarity metrics (Opitz et al., 2021) consider AMR graphs as high-dimensional objects. They first propagate node embeddings iteratively by incorporating contextualization and then employ Wasserstein Weisfeiler Leman kernel (WWLK) to calculate the minimum cost of transforming one graph to another. WWLK only considers node embeddings (GloVe embeddings) in AMR graphs, while edge labels have no corresponding embeddings. Therefore, WWLKθ is extended to learn AMR edge labels, which requires additional training data.
Considering embeddings of AMR edges improves the performance of metrics. However, supervised learning methods require additional efforts to collect human-labeled data. In contrast, our proposed learned AMR similarity metric adopts selfsupervised learning and achieves higher correlation performance.

heading: 3 The Proposed Approach
text: In this section, we introduce our proposed approach for learning AMR graphs similarity evaluation.

heading: 3.1 Problem Formulations
text: AMR graph similarity metrics compute the graded similarities of AMR graphs. Given a reference AMR graph G = {V,E,R} where V , E and R denote the node set, edge set, and relation set of AMR graph G respectively, and a candidate AMR graph Ĝ = {V̂ , Ê, R̂}, the primary motivation is to automatically learn the similarity sim(G, Ĝ) between the two AMR representations. To do that, we propose to use network structures to derive the contextual embeddings of the two AMR graphs and then compute their similarity score as the cosine similarity of the embeddings. More precisely, we use sim(G, Ĝ) = sim(e, ê) = e
⊤ê ∥e∥∥ê∥ , where
e and ê are the contextual embeddings of G and Ĝ, respectively. Hence the key of our approach is learning contextual embeddings.

heading: 3.2 AMRSim
text: The use of self-supervised methods is a wellestablished approach in semantic text similarity tasks (STS, Carlsson et al., 2021; Gao et al., 2021). To assess AMR graph similarity efficiently and well-correlated with human judgments, we propose migrating the self-supervised training process in STS to AMR similarity evaluation.

heading: 3.2.1 Self-supervised Training
text: In plain text applications, e.g., STS tasks and text generation tasks, many learned metrics are trained to optimize correlation with human annotations (Lowe et al., 2017; Reimers and Gurevych, 2019). However, AMR graph similarity data collection is more time-consuming because AMR evaluation has a learning cost of understanding the semantics of graphs, which are not as straightforward as plain text. Thus, self-supervised learning methods are an alternative solution. We adopt an efficient self-supervised approach Contrastive Tension (CT; Carlsson et al., 2021) in AMRSim metrics. The basic assumption is that AMR graphs with adjacent distributions have similar meanings.
In CT, two independent encoders are initialized identically. The training objective is to maximize the dot product between positive embeddings and to minimize the dot product between different encodings. CT constructs positive and negative pairs in each batch. For each randomly selected AMR graph G, copy G into an identical graph pair to construct the positive instance, and sample other K graphs to construct negative instances by pairing G. The K + 1 instances are included in the same batch. The training contrastive loss L is binary cross-entropy between the generated similarity scores and labels.
L(G, Ĝ) = { − log σ(e · ê), if G = Ĝ − log σ(1− e · ê), if G ̸= Ĝ
where σ refers to the Logistic function. Figure 1 demonstrates the pipeline of AMRSim. An instance containing two AMR graphs is input to graph encoders to generate contextual embeddings, then the dot product of the embeddings is used to compute the loss.

heading: 3.2.2 Network Structures
text: Compared to plain text, AMR graphs contain more structural information. We propose to incorporate graph neural networks into transformers to adapt to AMR graph structures and derive more descriptive contextual embeddings.
Transformers Transformer based neural networks have demonstrated exemplary performance in the fields of natural language processing but they only accept inputs as sequence data. Therefore, we first convert AMR graphs to sequences. In AMR graph G, the labeled edge (u, r, v) ∈ E, where u, v ∈ V and r ∈ R is a relation type, means that there is an edge labeled as r from node u to node v. Similar to Ribeiro et al. (2022), we convert each AMR graph G into an unlabeled graph by replacing each labeled edge (u, r, v) with two unlabeled edges {(u, r), (r, v)}. Unlabeled G′ = {V ′, E′} where V ′ include original nodes as well as additional nodes converted from relations. This preprocessing method facilitates BERT to learn word embeddings for edges.
The unlabeled AMR graph is then linearized. Position embeddings are crucial for modeling sequence order in transformers. The widely used position embedding takes the advantage of absolute positions of the input sequence. However, we argue that the linearization order should not affect AMR graph encoding, because of the inherent nature of AMR graphs that the relationships between nodes are mainly defined by the underlying semantic connections rather than their linear positioning. Therefore, instead of absolute position encoding, the shortest path lengths between all nodes and the root node are encoded as relative position embeddings.
In pre-trained language models like BERT, tokens may be split into smaller subwords due to
the fixed vocabulary. We refuse to split AMR relations and instead add them to the vocabulary as new words. However, concept nodes are tokenized as usual. The main reason is that relation words are artificial, while nodes are always concepts, closer to linguistic tokens in the vocabulary. For unlabeled edge (u, r), if node u is split into subword u1, . . . , uk in tokenization, we consider each subword as a new node and connect each subword to r, thus (u, r) is replaced by {(u1, r), . . . , (uk, r)}. Graph Neural Networks Adapter To generalize transformers to capture the structural information of AMR graphs, we incorporate graph neural networks as adapter into transformer layers. In particular, we refer to the idea of the k-dimensional Graph Neural Networks (k-GNN; Morris et al., 2019), which are based on the k-dimensional Weisfeiler and Leman algorithm (k-WL). For a given k, a k-set in [G]k, s = {Vs, Es} is a k-element subgraph over G, then the neighborhood of s is N(s) = {t ∈ [G]k||s ∩ t| = k − 1}. The GNN at layer t computes new features: htk(s) = σ(ht−1k (s) ·W t1 + ∑ u∈N(s) h t−1 k (u) ·W t2), where σ is the activation function, W t1 and W t 2 are layer parameters. Different from the graph adapter in Ribeiro et al. (2022), we employ an adapter module after the feed-forward sub-layer of the last layer of BERT. The experimental results comparison can be found in section 5.6. The normalization layer before the GNN layer and the projection after the GNN layer is kept, see figure 1. So given the hidden states hv for node v, the GNN adapter layer computes: zv = W · GNN_layer(LN(hv)) + hv, where W is the adapter parameters, LN(·) denotes layer normalization and GNN_layer represents the calculation process of GNN layers.

heading: 4 Experiments

heading: 4.1 Data Construction
text: A major advantage of the self-supervised method is that no human-annotated data is required for training. Following data preparation in AMR-DA (Shou et al., 2022), AMRSim utilized SPRING (Bevilacqua et al., 2021) to parse one-million sentences randomly sampled from English Wikipedia2 to AMR graphs. Generated silver AMR graphs were linearized by a depth-first traversal algorithm
2The one-million sentences sampled from Wikipedia is taken from datasets for SimCSE (Gao et al., 2021), which can be downloaded from https://huggingface.co/datasets/ princeton-nlp/datasets-for-simcse/tree/main.
(the choice of linearization method does not have impact on embeddings, refer to section 5.1), meanwhile, all edges were recorded as pairs (u, r). Computing relative positions of tokens was implemented by Dijkstra’s Method 3.

heading: 4.2 Experimental Setups
text: We implemented AMRSim with sentence transformers (Reimers and Gurevych, 2019). During training, we set the positive ratio to be 4/16. In a batch of 16, there were 4 positive graph pairs and 12 negative pairs. This indicates that we sampled 4 graphs and created one positive pair and three negative pairs for each graph. The transformer parameters were initialized from uncased BERT base model (Devlin et al., 2019), and parameters for graph adapters were initialized randomly. We carried out a search of sequence length ∈ {64, 128, 256} and determined the length of linearized AMR graphs as 128. Other hyperparameters were set as follows: learning rate as 1e-5, dropout rate as 0.1, and graph adapter size as 128. GNN embeddings from k layers were concatenated and input to a projection function to generate final embeddings. Our experiments were done using GeForce RTX 2080 Ti GPU. We trained our models for one epoch, which took approximately two and a half hours and reported in the table the average performance of our models over five repeated experiments with different seeds.

heading: 4.3 Main Results
text: We compared AMRSim with other AMR similarity metrics on evaluation datasets modified from semantic textual similarity datasets, STSB (Baudiš et al., 2016) and SICK (Marelli et al., 2014). Original datasets contain a set of pairs of sentences with a human-labeled similarity score. Opitz et al. (2021) utilized a strong parser to construct AMR graph pairs from sentence pairs and normalized similarity scores to the range [0, 1] to facilitate standardized evaluation. There are 1379 and 4927 test instances in the two datasets, respectively. Over 95% randomly selected data from generated AMR graphs were assessed as gold or with minor errors.
The best performance of baseline metrics on the test dataset was included in the comparison. For example, k = 2 achieved superior performance in STS and SICK than the default value k = 3
3The implementation code comes from networkx: https: //networkx.org.
in SEMBLEU, so we included SEMBLEUk=2 as SEMBLEU.
Table 1 shows the comparison of the evaluation results of various metrics on two test datasets. Baseline results are from Opitz et al.’s (2021) work. We conducted five experiments for each different kGNN setups and reported the mean and standard deviation of Pearson correlation scores. Our proposed AMRSim significantly outperformed previous AMR metrics and achieved the highest score on correlation with human annotation. Specifically, AMRSim5 with 5-GNN adapter improved the previous best Pearson score from 66.94% to 70.94% on STSB, and AMRSim4 with 4-GNN adapter improved the score from 67.64% to 73.10% on SICK dataset. When k increased from one to four, the average performance of AMRSim in both datasets increased, however, when increasing from four to five, the average performance of the model decreased. More parameters to optimize made the model harder to train without more improvement. Considering the average Pearson score and standard deviation, we took AMRSim4 as our final metric. It is worth mentioning that WWLKθ learned edge encodings through supervised learning methods. By contrast, our proposed AMRSim adopted self-supervised learning, and no human-labeled data was required.

heading: 5 Analysis
text: AMRSim shows a high correlation with human annotations. In this section, we further analyze the
robustness and efficiency of AMRSim.

heading: 5.1 Robustness Analysis
text: At first, we explore in depth how AMRSim performs under various challenges.
Reification Challenges Reification challenges import AMR rephrasing, which means changing the structure of graphs, but not its meaning (Banarescu et al., 2013; Goodman, 2019; Opitz et al., 2021). The reification rule is that edge (u, r, v) in the original graph induced (u, r1, insr) ∧ (insr, r2, v), where insr is r’s reification and rns are new generated edges. For example, in the following AMR graph:
(xv0 / brush-01 :ARG0 (xv1 / girl
:part xv2) :ARG1 (xv2 / hair))
Edge label :part has its reification :have-part-91, so for the above AMR graph, (girl, :part, hair) can be replaced by (girl, :ARG-of, have-part-91) ∧ (have-part-91, :ARG2, hair). The modified AMR graph keeps the meaning displayed as:
(xv0 / brush-01 :ARG0 (xv1 / girl
Reification challenges require that metrics have the capacity to handle structure variants. Table 2 shows the comparison of AMR metrics under reification challenges. AMRSim ranked first on both reified datasets. On reified STSB test set, AMRSim achieved the highest score of 70.54% and the lowest loss of 0.43% compared with performance
on STSB dataset. Similar results were presented on SICK dataset. AMRSim achieved the highest score of 73.42% and the second lowest loss of 0.31%. Encoding the entire AMR graph and comparing contextual embeddings is a contributing factor that improves robustness under reification challenges.
Synonym Challenges Another challenge is synonym challenge, which is conceptual synonym substitution while preserving meaning. AMR nodes are iterated over and replaced by their (near-) synonyms in PropBank, or their synset in WordNet (Opitz et al., 2021). The synonym transformation is not trivial for the reason that the single node may be replaced by a graph substructure. For example, instance(x, tofu) can be extended as (x, :mod, y) ∧ instance(x, curd) ∧ instance(y, bean) , so the AMR graph
(xv0 / cut-01 :ARG0 (xv2 / woman) :ARG1 (xv1 / tofu))
is transformed as:
(xv0 / cut-01 :ARG0 (xv2 / womanhood) :ARG1 (xv1 / curd
Table 3 shows the performance of various AMR metrics on synonym challenges. From the results, synonym challenges are more complex than reification challenges. Nevertheless, AMRSim still ranked first on both test datasets, with 66.50% and 67.73% respectively. SEMBLEU that matches ngram of linearized AMR graphs is the most vulnerable under this challenge. Besides, WWLKθ and
AMRSim suffer more loss than reification challenges. One possible reason is that both WWLKθ and AMRSim are learned metrics, and they have a common generalization problem to some extent. However, our proposed AMRSim outperformed WWLKθ by a large margin, but with a close score loss.
Linearization Challenges The basic assumption of AMRSim is that the similarity score is concerned with the entire AMR graph and will not be affected regardless of the linearization of the AMR graph. As the transformer model treats each token as independent, position embedding is added to retain the order information of tokens. To test AMRSim’s ability to handle the linearization challenge, we compared model outputs for different inputs: (i) A pair of AMR graphs that are isomorphic but have different linearization sequences; (ii) a pair of different AMR graphs with the same linearized sequence after adopting the original position embedding strategy in BERT; (iii) a pair of different AMR graphs with the same linearized sequence after adopting relative position embedding strategy in AMRSim. For example, four AMR graphs are listed below:
AMR A: (xv0 / play-01
:ARG0 (xv2 / group :consist-of (xv4 / boy))
:ARG1 (xv1 / soccer) :location (xv3 / beach))
AMR B: (xv0 / play-01
:location (xv3 / beach) :ARG0 (xv2 / group
:consist-of (xv4 / boy)) :ARG1 (xv1 / soccer))
AMR C: (xv0 / play-01
:ARG0 (xv2 / group :consist-of (xv4 / boy
:ARG1 (xv1 / soccer :location (xv3 / beach)))))
AMR D: (xv0 / play-01
:ARG0 (xv2 / group) :ARG1 (xv1 / soccer) :location (xv3 / beach
:consist-of (xv4 / boy)))
AMR graphs A and B are isomorphic but the position of :location beach differs in the linearized sequences. Graphs A and C are different, however, token positions ranked by the ascending algorithm in BERT are the same. Graphs A and D are also inconsistent, but the relative positions of all tokens are consistent. For example, the distance between :consist-of and the root play-01 is 4 on both graphs. To demonstrate the effectiveness of GNN adapters in AMRSim, we compare AMRSim with the original BERT and BERT with relative position embeddings (AMRSim without GNN). Table 4 con-
cluded similarity scores from BERT, BERT with relative position embedding, and AMRSim. BERT was confused by various linearizations of the same graph whereas BERTrelative has a shortage of capturing structure information. With GNN adapters, AMRSim can handle the linearization challenge. In terms of the other metrics, only SemBleu requires linearization, and it exhibits a high level of sensitivity towards the chosen linearization.

heading: 5.2 Scatter Plot Analysis
text: To visually analyze the distribution patterns and the relationship between human annotations and scores obtained from different metrics, we utilize scatter plots to show instances distributions and best fit lines to depict the trend. All scores are normalized to the range [0,1]. In figure 2, the y-axis of each subplots represents human annotated scores, while the x-axis represents the scores calculated using various metrics. By plotting individual examples on scatter plots, we observed that WWLK-related metrics and our AMRSim outperform other metrics significantly, with higher concentration around the fit and fewer outlier. However, they exhibit biases towards instances with distinct human scores. WWLK-related metrics tend to underrate examples
with higher human scores while AMRSim tends to overrate examples with lower human scores.

heading: 5.3 Case Study
text: Disagreements in preference for AMR graphs can affect the ranking of AMR parsers. Thus we employ a case study to study metrics’ preference for parsed results. Here is an example from Opitz et al.’s (2020) paper. CAMR and JAMR are two AMR parsers.
::snt: Legally, there are two remedies. Gold AMR: (t / thing :quant 2
:ARG2-of (r / remedy-01) :mod (l / law)
CAMR Parsed Result: (x6 / remedy-01
:quant 2)
JAMR Parsed Result: (l / legally
:manner-of (r / remedy-01 :quant 2))
The scores obtained from different metrics are as follows: SMATCH (0.2, 0.167); S2MATCH (0.2, 0.252); SEMA (0, 0); SEMBLEU (0.215, 0.215); WWLK (0.335, 0.352); WWLKθ (0.329, 0.345); AMRSim (0.81, 0.92). Among these metrics, SMATCH gives higher scores to CAMR parsed results. Conversely, SEMBLEU and SEMA assign the identical score to both parsed results. SEMA employs a more strict evaluation method as it assesses not only the individual node but also its dependencies. However, S2MATCH, WWLK, WWLKθ and AMRSim grant higher scores to the second parsed results. In this particular example, legally is indispensable information of semantics. Our preference leans towards the second parsed results.

heading: 5.4 Computational Time
text: In this section, we emphasize the importance of computational time as a crucial factor to consider for practical use. While the time complexity of AMRSim is O(n), it is necessary to consider that the processing with transformers possibly impact the overall time required. We provide an analysis of various metrics’ reference time based on SICK test dataset consisting of 4927 samples. The reference time is listed as follows: SMATCH: 5.24s, S2MATCH: 37.51s, SEMA: 1.132s, SEMBLEU: 0.67s, WWLK: 46.60s, WWLKθ: 48.13s. Our AMRSim differs slightly from other metrics as we employ neural networks to predict graph similarity, allowing us to utilize both CPU and GPU for the task. When using CPU, the reference time is approximately 67.66s. However, this is not the most optimal choice, as matrix computations can be accelerated by GPUs. By utilizing a single GPU (RTX2080ti in our experiments), for instance, with a batch size of 16, the reference time reduces to around 14.20s. Scaling up to a batch size of 256 further reduces the reference time to 8.04 seconds. Further increasing the batch size has limited benefits as the processing time for input and output becomes the dominant factor. In term of O(n) time complexity, the advantages of our AMRSim become more pronounced as the graph sizes increase.

heading: 5.5 Multilingual Metrics
text: The lack of multilingual AMR evaluation metrics becomes an obstacle to developing cross-lingual AMR parsers. It is common practice to translate
other languages into English or use multilingual embeddings for matching, which does not take into account that the source language may affect the structure of the AMR (Wein and Schneider, 2022). AMRSim has the potential to be extended to multilingual cases given the generic nature of our pipeline: encoding AMR graphs and computing the cosine similarity of graph embeddings, not limited to specific languages. However, annotating the similarity of a multilingual dataset is more challenging and subject due to varying levels of language proficiency. So we performed a preliminary test by collecting a small-scale test dataset comprising 20 Chinese-English AMR pairs, then adopting a well trained multilingual model4 to encode the corresponding text and calculating cosine similarity of text embeddings as the graph similarity score. To train our multilingual AMRSim, we employed the multilingual version of BERT base model and added a Chinese AMR dataset containing about 2,500 Chinese AMR graphs from Li et al. (2016) to English training data in Section 4.1. Re-
sults of metrics are concluded in table 5. SEMA lacks support for multilingual cases, resulting in it returning 0 for any multilingual inputs. Additionally, due to the absence of a multilingual dataset for WWLKθ supervised training, we exclude them from our comparison. A specific example is illustrated in Figure 3. The predicted similarity score for these two graphs needs to be close to 1 because they correspond to the same sentence in the Chinese and English versions of The Little Prince. We
used different metrics to evaluate the similarity of these two graphs and concluded the score in table 6. While AMRSim surpasses other metrics, its performance is poor compared to the English version of AMRSim. The possible explanation is the lim-
4The multilingual encoding model is downloaded from https://huggingface.co/sentence-transformers/ paraphrase-multilingual-MiniLM-L12-v2.
ited training data hinders effective self-supervised learning. A comprehensive study on multilingual applications is considered to be a future work.

heading: 5.6 Ablation Studies
text: We investigated the impact of GNN adapters’ position. Comparison of models’ performance is included in table 7. GNN represents that the model only contains GNN layers; GNN-Each means that one GNN adapter is included in each transformer layer, same as Ribeiro et al. (2022); GNN-Before denotes that GNN adapter is before transformer layers and AMRSim indicates that GNN adapters are incorporated after transformer layers. Models other than GNN-Each come with 4-GNN. GNN
achieved the worst correlation score because it is notoriously hard for GNN to train (Li et al., 2019). However, setting transformers as the backbone and incorporating GNN as adapters are more acceptable in our experimental settings. GNN-Each underperformed in the experiments, possibly because too many GNN parameters appear in each layer to be optimized for self-supervised training. AMRSim achieved superior performance, which is intuitive because BERT encoded sequence information, and then GNN captured structural information.

heading: 6 Conclusion
text: We have proposed a learning based AMR graph similarity metric called AMRSim. Its underlying network structure incorporates GNNs into a transformer network and employs self-supervised learning. Our proposed similarity metric addresses two common pitfalls of existing metrics: firstly, a low correlation with human annotations due to lack of considering the entire structures of AMR graphs. AMRSim effectively encodes AMR graphs, exhibiting the highest correlations with human evaluations and maintaining robust against various challenges. The second is computational inefficiency associated with matching or alignment of AMR graphs. AMRSim is alignment-free, allowing for a significant reduction in inference time when utilizing GPUs. We also show the potential of extending AMRSim to multilingual AMR similarity metrics.
Limitations
AMRSim has high prediction efficiency but the training process is time-consuming. In our experiments, one epoch training on one GeForce RTX2080Ti took about two and a half hours. Selfsupervised learning requires a large amount of training data. Parsing wiki sentences into graphs requires time, but the advantage is that it can be processed offline. Another issue is the length limit. Transformers can only handle limited sequence lengths due to the computational and memory complexity of attention calculation. Therefore, encoding large AMR graphs is challenging. Possible solutions include applying sliding window algorithm to split a large AMR graph into several subgraphs and merge the scores.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
3 A2. Did you discuss any potential risks of your work? Limitations
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Experiments
3 B1. Did you cite the creators of artifacts you used? Experiments
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Experiments
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Experiments
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Experiments
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Experiments
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Experiments
C 3 Did you run computational experiments? Experiments
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Experiments
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Experiments
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Experiments
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Experiments
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: A comprehensive study on multilingual applications is considered to be a future work