Incorporating Graph Information in Transformer-based AMR Parsing
Year: 2023

Abstract:
Abstract Meaning Representation (AMR) is a Semantic Parsing formalism that aims at providing a semantic graph abstraction representing a given text. Current approaches are based on autoregressive language models such as BART or T5, fine-tuned through Teacher Forcing to obtain a linearized version of the AMR graph from a sentence. In this paper, we present LeakDistill, a model and method that explores a modification to the Transformer architecture, using structural adapters to explicitly incorporate graph information into the learned representations and improve AMR parsing performance. Our experiments show how, by employing word-to-node alignment to embed graph structural information into the encoder at training time, we can obtain state-of-the-art AMR parsing through self-knowledge distillation, even without the use of additional data. We release the code at http://www.github.com/ sapienzanlp/LeakDistill.Meaning Representation (AMR) is a Semantic Parsing formalism that aims at providing a semantic graph abstraction representing a given text. Current approaches are based on autoregressive language models such as BART or T5, fine-tuned through Teacher Forcing to obtain a linearized version of the AMR graph from a sentence. In this paper, we present LeakDistill, a model and method that explores a modification to the Transformer architecture, using structural adapters to explicitly incorporate graph information into the learned representations and improve AMR parsing performance. Our experiments show how, by employing word-to-node alignment to embed graph structural information into the encoder at training time, we can obtain state-of-the-art AMR parsing through self-knowledge distillation, even without the use of additional data. We release the code at http://www.github.com/ sapienzanlp/LeakDistill.

heading: 1 Introduction
text: Creating a machine-interpretable representation of meaning lies at the core of Natural Language Understanding and has been framed as the Semantic Parsing task. Multiple formalisms have been proposed over the years, e.g., Prague Czech-English Dependency Treebank (Hajič et al., 2012), Universal Conceptual Cognitive Annotation (Abend and Rappoport, 2013), BabelNet Meaning Representation (Navigli et al., 2022; Martínez Lorenzo et al., 2022); however, Abstract Meaning Representation (Banarescu et al., 2013, AMR) has received more attention thanks to the large corpus available and a well-defined structure. AMR captures text semantics in the form of a directed acyclic graph (DAG), with nodes representing concepts and edges representing semantic relationships between them (see Figure 1). Currently, AMR is widely employed
∗ Equal contributions.
Figure 1: Top: sentence. Middle: AMR graph. Bottom: Linearized graph. Alignment is represented by colours.
in a plethora of NLP domains, such as Information Extraction (Rao et al., 2017), Text Summarization (Hardy and Vlachos, 2018; Liao et al., 2018), Question Answering (Lim et al., 2020; Bonial et al., 2020b; Kapanipathi et al., 2021), Human-Robot Interaction (Bonial et al., 2020a), and Machine Translation (Song et al., 2019), among others.
Until a short while ago, autoregressive models proved to be the best approach for semantic parsing because of their outstanding performance without relying on sophisticated ad-hoc architectures (Bevilacqua et al., 2021). Then, more recently, several approaches have emerged to increase performance by including structural information in the model (Chen et al., 2022), adding extra Semantic Role Labeling tasks (Bai et al., 2022) or by ensembling strategies (Lam et al., 2021; Lee et al., 2022).
In this paper, following the effort of strengthening the model’s learning phase by incorporating meaningful structural information, we investigate the use of structural adapters (Ribeiro et al., 2021a)
1995
that are basically Graph Neural Networks (GNNs) embedded in the encoder of a Transformer EncoderDecoder architecture. The structural information is derived from intrinsic concept-node alignments from which we build a word-based graph with a structure similar to the original AMR. Leveraging such a graph implies partial data leakage: the graph structure is revealed to a model during training. To overcome the lack of the leaked information at inference time, we explore Knowledge Distillation (KD), a technique that transfers knowledge from a teacher model to a student model (Hinton et al., 2015). The word-based graph is employed with the structural adapters to obtain soft targets (the teacher path), which are then used for selfdistillation, transferring the knowledge to the student, which only has access to the text.
Our main contributions are: i) exploring how to add structural information to the AMR parsing model using structural adapters and self-knowledge distillation, ii) state-of-the-art results in AMR parsing for AMR 2.0 and AMR 3.0 datasets, and iii) competitive base models for AMR parsing.

heading: 2 Related Work
text: Over the years, multiple trends have appeared to parse AMR graphs: using statistical methods (Flanigan et al., 2014, 2016; Wang et al., 2015), neural-transition based parsers (Ballesteros and Al-Onaizan, 2017; Liu et al., 2018; Fernandez Astudillo et al., 2020; Zhou et al., 2021) or bidirectional Transformers (Lyu and Titov, 2018; Zhang et al., 2019; Cai and Lam, 2020) based on BERT (Devlin et al., 2019).
Recently, autoregressive models based on BART (Lewis et al., 2020) have emerged as a dominant approach for AMR parsing, since they obtained state-of-the-art performance without complex pipelines. One notable example is SPRING (Bevilacqua et al., 2021), which frames AMR parsing as a neural machine translation task, where text is translated into a linearized version of the graph. Subsequently, several works extended SPRING using a variety of different strategies. Procopio et al. (2021) leverages multitask learning to improve cross-lingual AMR parsing results. ATP (Chen et al., 2022) expands the dataset with extra auxiliary tasks such as Semantic Role Labeling and Dependency Parsing, with pseudo-AMR graphs constructed based on a particular task. AMRBART (Bai et al., 2022) uses a pre-training strat-
egy based on Masked Language Modeling where both text and graph need to be denoised, using 200k graphs generated by SPRING. However, despite their efforts to enhance SPRING’s performance, all these systems rely on additional external data. Although Ancestor (Yu and Gildea, 2022), which modifies ancestor information during decoding, and BiBL (Cheng et al., 2022), that adds a secondary graph masking task while training, do not rely on extra data, their performance improvements remain relatively limited. Our proposed model effectively bridges the gap in performance between "with" and "without" extra data by integrating explicit structural information during the training phase.

heading: 3 Word-Aligned Graph
text: Our goal is to incorporate graph-structured information into the encoder of a Transformer-based parser. However, the model only has access to the input sentence at that stage, with no hidden representation of AMR-specific nodes and relations. Thus, we simplify the AMR structure to a wordbased graph by exploiting a pre-existing alignment between spans in text and semantic units in the corresponding AMR graph (see Figure 1).
First, starting with the source AMR graph, we replace the labels of the AMR nodes and relations with the words of the corresponding sentence as provided by the alignment (Figure 2, left). Next, we convert each edge into a node and connect it to its original endpoints (see Figure 2, center). Moreover, following what Ribeiro et al. (2021b) did for AMR graphs, we split each multi-token node (e.g., freedom in Figure 2) into a parent node represented by the first token and children nodes connected to it which contain the remaining tokens. We name the resulting graph representation the Word-Aligned Graph (WAG).
We will leverage WAGs to enrich the encoder’s hidden representations of words with the AMR graph’s structural information. Unfortunately, a problem arises with non-aligned nodes (e.g., the :location relation in Figure 2), since they will not have associated hidden states. Therefore, we have two alternatives: i) remove nodes for which we do not have hidden states (Contracted WAG), or ii) create new hidden states for them (Full WAG).
Contracted WAG As a first option, we remove non-aligned nodes from the graph. However, deleting the nodes from the original graph would pro-
duce a disconnected graph. To obtain a connected structure similar to the original graph, we contract nodes rather than removing them. A contracted WAG (CWAG) is a graph in which non-aligned nodes are merged with their closest parent node along with all their relations. Figure 2 (right) depicts a CWAG.
Full WAG Alternatively, we preserve the nodes without alignment (e.g., the node “location” in Figure 2 (center)). This type of graph is referred to as a Full WAG (FWAG), Figure 2 (center) shows an example of FWAG.

heading: 4 Structural Adapters for AMR parsing
text: In this section, we describe the main components of our structure-enhanced approach to AMR parsing.

heading: 4.1 Parsing with BART
text: AMR parsing can be defined as a sequence-tosequence (seq2seq) problem where the input x = (x1, ..., xn) is a sequence of n words (or subwords) and the output g = (e1, ..., em) is a linearized
graph with m elements. Our goal is to learn a function that models the conditional probability:
p(g|x) = m∏
t=1
p(et|e<t, x), (1)
where e<t are the tokens of the linearized graph g before step t.
Suppose we have a dataset D of size |D| which consists of pairs (xi, gi), with each gi having length mi. Our objective is then to minimize a negative log-likelihood loss function:
LDnll = Lnll(D) = − |D|∑
i=1
log p(gi|xi) =
= − |D|∑
i=1
mi∑
t=1
log p(eit|ei<t, xi) (2)
We use BART as our seq2seq model implementing the above formulation and, following Blloshmi et al. (2021, SPRING), add special tokens corresponding to i) AMR-related tokens, ii) variable names <R0>, <R1>, ... <Rn>, and iii) other tokens needed for the graph linearizations. Then, we fine-tune BART with the input x and the target g.

heading: 4.2 Structural adapters
text: To incorporate AMR structural information into the encoder, we embed the WAGs – obtained from AMR graphs as illustrated in Section 3 – into adapters that encode the graph structure imposed by them. Structural adapters, as introduced by Ribeiro et al. (2021b), are a modification of the Transformer architecture that improves pre-trained language models for modeling graph information. They consist of a Graph Convolutional (GraphConv) layer and a feed-forward layer, which are connected
through a residual connection. Moreover, we remove layer normalization and set GELU as an activation function (see Figure 3).
Structural adapters are inserted after each encoder’s layer (see Figure 4). For each hidden representation hlv ∈ Rb from the encoder layer l and the set of edges E in the WAG, we define the GraphConv operation as:
GraphConvl(h l v, E) =
∑
u∈N (v)
1√ dudv Wlgh l u (3)
where N (v) is the set of node v’s adjacent nodes in the WAG (including v itself), dv is the degree of v, and Wlg ∈ Rb×b is a parameter matrix. Then, the updated hidden states zlv are computed as:
glv = GraphConvl(h l v, E) zlv = W l aσ(g l v) + h l v,
(4)
where σ is the GELU activation function and Wla ∈ Rb×b is the feed-forward layer parameter matrix.

heading: 5 Our Models

heading: 5.1 Graph Leakage Model
text: We bring together the two main components described in Section 4 by incorporating structural
adapters in each layer of the encoder of a BARTbased AMR parsing model (see Figure 4 (left) and Algorithm 1). Here, a WAG, together with the hidden representations of tokens in the sentence, are input to the adapters. Since WAGs are constructed using gold AMR graphs, this constitutes a form of information leakage. We name this model the Graph Leakage Model (GLM), with the idea that it will serve as a study of the impact on performance when including WAGs (be they contracted or full, cf. Section 3).
To use FWAGs as input to the adapter, we need representations for non-aligned nodes that do not have an associated hidden state. Therefore, for nodes with labels corresponding to AMR special tokens (e.g., :location) we use their embedding. For other nodes, we tokenize the label and take the average embedding. Furthermore, these representations are concatenated after the hidden states in the first adapter layer. After each adapter block, we split representations into two groups: i) the updated hidden states for the original input tokens, which serve as inputs of the subsequent Transformer layer, ii) the updated hidden states for the non-aligned nodes, which are concatenated again in the next adapter block (see Algorithm 1).
Then, for both CWAG and FWAG, the input to each adapter layer l consists of a matrix of hidden
Algorithm 1 Modified BART Encoder
Input: E - set of WAG edges, S0 - states for non-aligned nodes, H0 - initial hidden states of the input sequence for l ∈ {1, ..., 12} do
H l ← BARTLayerl(Hl−1) if Leak Mode then
if Full WAG then Gl ← Concat(H l, Sl−1) else Gl ← H l end if G̃l ← StructAdaptl(Gl, E) if Full WAG then
[H̃ l;Sl]← Split(G̃l) else
H̃ l ← G̃l end if
else H̃ l ← H l end if H l ← H̃ l
end for
states H l and a set of edges E . Note that the set of edges E does not change through layers. Finally, the loss function for GLM is:
Lleak = Lnll(D̃) = − |D̃|∑
i=1
log q(gi|xi, wi), (5)
where D̃ is the updated dataset consisting of pairs ((xi, wi), gi), q is the probability for GLM, wi is the WAG.

heading: 5.2 Knowledge Distillation
text: GLM leverages the alignment information to improve the model’s understanding of the graph structure and enhance its (the model’s) performance in AMR parsing. Unfortunately, as discussed in the previous section, this constitutes a form of leakage at inference time. Therefore, following the idea of Knowledge Distillation (Hinton et al., 2015, KD), we set the fine-tuned GLM as a teacher model, which receives both the sentence and WAG as inputs, and our plain BART parser as the student (see Section 4.1). Then, the knowledge acquired by the teacher model is transferred to the student model, which only has access to the sentence. This enables the utilization of WAGs during training while avoiding their use during inference. Hence, our
objective is to achieve the following:
p(g|x) = q(g|x,w) (6)
where p and q are probabilities of the student and the teacher, respectively, and w is the WAG, used only at training time.
As is common in KD, we employ Kullback–Leibler divergence to match the student and the teacher probabilities:
LKL = KL(p, q) = C−1∑
k=0
pk log( pk qk ) (7)
where C is the number of classes, i.e. our token vocabulary. Usually, the loss LDnll for the original task is added to the total loss, thus becoming:
LKD = L D nll + αLKL =
= − |D|∑
i=1
mk∑
t=1
C−1∑
k=0
(δit(k) log p i t,k − αpit,k log( pit,k qit,k )),
pit,k = p(e i t=k | ei<t, xi), qit,k = q(e i t=k | ei<t, xi, wi) (8)
where δit(k) is 1 when k is a target class at step t and 0 otherwise; α is a hyperparameter.
There are only architectural differences between the teacher and the student model at the encoder, since the teacher additionally includes the structural adapters. Therefore, we copy the GLM decoder to the student model and freeze the decoder parameters.

heading: 5.3 LeakDistill
text: We anticipate that, in our experimentation, KD will have failed to properly transfer the structural information to the student model. Therefore, we propose a single model approach that can be trained by performing two forward passes at each training step, one with and one without the WAG structural information (see Figure 4 and Algorithm 2). We force the two passes to learn the same distribution by adding a Kullback–Leibler divergence loss to the output logits. As a result, the total loss becomes:
LLeakDistill = L D nll + βLleak + αLKL =
= − |D|∑
i=1
mk∑
t=1
C−1∑
k=0
(δit(k) log p i t,k + β δ i t(k) log q i t,k
−αpit,k log( pit,k qit,k )),
(9)
where Lleak is the loss for the first pass (basically, GLM), with leaked information, LDnll is the loss for the second pass (basically, BART), which is the original negative log-likelihood loss, and finally LKL is the above-described Kullback–Leibler divergence loss. α and β are hyperparameters to control each loss scale.
The above formulation implements what is called self-knowledge distillation (Hahn and Choi, 2019, SKD). Specifically, in our work we project the knowledge via leveraging data leakage in the first pass rather than computing soft target probabilities. Moreover, we calculate KL divergence for all classes to obtain more knowledge. Finally, based on the intuition that there is not enough information to distill at the beginning of training, we schedule a gradual decrease of Lleak’s multiplier β.

heading: 6 Experimental Setup
text: To demonstrate the benefits of incorporating structural information in AMR parsing, we devise a set of experiments to assess its performance in comparison to state-of-the-art models. Before delving into details, we provide information regarding the datasets (Section 6.1), the metrics (Section 6.2) and the model (Section 6.3) used in our experiments.

heading: 6.1 Datasets
text: We test on two AMR benchmark datasets: i) AMR 2.0, which has 36521, 1368, and 1371 sentenceAMR pairs in the training, validation, and test sets, respectively, and ii) AMR 3.0, which contains 55635, 1722, and 1898 sentence-AMR pairs in the training, validation, and test sets, respectively (see Appendix E). Furthermore, we test on The Little Prince (TLP) and the Bio AMR out-of-distribution datasets.
Alignment Our approach relies directly on the structural information extracted from the wordconcept alignment. There are several alignment
standards: first, Information Sciences Institute (ISI) provides extended AMR 2.0 and AMR 3.0 datasets with alignments of all the graph semantic units that are directly related to the sentences’ spans (Pourdamghani et al., 2014). Second, Linguistically Enriched AMR (Blodgett and Schneider, 2021, LEAMR) achieves full graph-alignment coverage by aligning all the graph semantic units to a corresponding span in the sentence.
Silver Data Following Bevilacqua et al. (2021), we explore the same strategy to generate a dataset with 140k silver sentence-graph pairs. The silver LEAMR alignments are generated using the approach of Huguet Cabot et al. (2022).

heading: 6.2 Metrics
text: We evaluate our models using the SMATCH metric (see Appendix D for more details). Additionally we also perform evaluation with two additional metrics: S2MATCH (Opitz et al., 2020) and WWLK (Opitz et al., 2021). For WWLK we use WWLKk3e2n introduced in Opitz et al. (2021).

heading: 6.3 Models
text: We use SPRING (Bevilacqua et al., 2021) as our baseline, and an auto-regressive model based on BART (Lewis et al., 2020) for predicting linearized versions of AMR graphs. Our models are built on top of this model, inheriting some hyperparameters (see Table 9).
In order to address the issue of overfitting, we implement a masking strategy which is used in conjunction with dropout and weight decay. For each batch, input tokens are masked with a varying probability pmask, which is uniformly sampled from the specified masking range (see Appendix A for details). The strategy is used for all models including SPRING (ours). In the following paragraphs, we explain the specific setup per each model.
Graph Leakage Model We explore two different settings for GLM: i) Contracted WAG, and ii) Full WAG (see Section 3).
Knowledge Distillation We test KD on the GLM with the highest SMATCH among CWAG and FWAG (see Table 1).
LeakDistill As done for GLM, we first examine the difference in performance between Contracted WAG and Full WAG. Then, we test Full WAG with i) β scheduling, ii) the silver data, iii) the combination of the silver data and the β scheduling. In the case of the scheduling of β, we start from β = 90 and decrease it linearly at each iteration for 21k iterations in total until it reaches 10. The hyperparameter α is set to 20. The value of β for the case i) and other hyperparameters are listed in Table 9.

heading: 7 Results
text: In this section, we provide our experimental findings. All tables show single-run results.
Graph Leakage Model Table 1 shows results for the Graph Leakage Model. While this setup relies on information being leaked from the final graph structure, it sets an upper bound on how encoding such information can improve performance. Here, we observe an increase of around five SMATCH points when using FWAG, whereas CWAG improvements are much smaller. While the model is certainly taking advantage of the leaked information, this is provided through the hidden states of the encoder. Therefore, we need to explore whether some of this performance gain can be kept implicitly without any information leak. Moreover, it is necessary to investigate the persistence of any performance disparity between CWAG and FWAG. This information is intriguing, as CWAG and FWAG differ in the context of additional information availability. CWAG only possesses a structure akin to the original graph, while FWAG not only exhibits a greater degree of structural similarity but also includes the original labels for nonaligned nodes.
KD and LeakDistill Table 2 compares the results between applying KD with GLM as the teacher versus the LeakDistill approach, explained in Section 5.3.We see how KD alone falls short of taking full advantage of the performance gains of GLM. On the other hand, LeakDistill, especially when including the KL loss, leads to about a 0.5 SMATCH point increase on the development set. Hence, we focus on LeakDistill as our main approach. Table 5 shows a breakdown of the experiments with LeakDistill, such as scheduling the KL loss or adding a silver data pretraining phase. It is evident that the performance difference between CWAG and FWAG remains, paving the way for more indepth research into the types of information that prove advantageous for LeakDistill. Additionally, the final row of Table 5 presents the outcome when the adaptors are active (the green path). It is noticeable that, despite the green path essentially being the GLM, it fails to match the performance level of 89.58.
Main results Tables 3 and 4 shows results for our proposed model, based on BART-large. Our system performs better than any previous single model parser, and, most notably, does so even without extra data, i.e. silver sentence-graph pairs. For AMR 2.0, we see up to 0.7 SMATCH increase over AMRBART and 0.4 on AMR 3.0. The use of extra data only leads to a small improvement, showing the efficiency of our approach, which is able to outperform previous state-of-the-art systems that relied on up to 200K extra samples. In the breakdown performance, we see how our system performs worse than ATP on Reentrancies, Negation and notably SRL. We believe this is due to the multitask nature of ATP, where SRL is explicitly included as a task. This opens the door to future work exploring the interaction between our approach and the inclusion of auxiliary tasks.
It is worth noting that our system relies on alignment information which is openly discussed at various stages in the paper. We do not consider this information as extra data since it is generated based on the existing data.
Out-of-distribution evaluation Table 6 shows the Out-of-Distribution of LeakDistill. We see a smaller improvement on TLP, 0.3 over AMRBART. On the harder BioAMR, performance increased by over a point, showing how the model is able to generalize well on different domains.
BART base Our state-of-the-art system relies on BART-large, which has 400M parameters. While it shows very strong performance, it has a big computational footprint, especially at inference time due to its auto-regressive generative nature. This makes the need for lighter, more compute efficient
models an important step towards better Semantic Parsers. Table 7 shows the performance of our approach when trained on top of BART-base, which has 140M parameters, achieving 83.5 SMATCH points on AMR 3.0, 1 point higher than AMRBART and, noticeably, surpassing SPRING-large performance by half a point. We believe it is crucial to have close to state-of-the-art performance base models, closing the gap from 2 points to 1 when compared to their large counterparts.
Other metrics Recent studies have shown that achieving a higher SMATCH score does not necessarily result in better performance of an AMR parser, as demonstrated by Opitz and Frank (2022). To address this issue, we use two additional evaluation metrics, namely S2MATCH and WWLKk3e2n (WWLK), which measure graded concept similarity and edge label importance, respectively. Our experiments reveal that S2MATCH correlates well with SMATCH, as expected for monolingual
parsers. Conversely, WWLK is specifically designed for monolingual AMR parsing and emphasizes edge labels. Interestingly, our findings suggest that ATP performs well, second only to our proposed system, LeakDistill. This may be due to the fact that both systems place greater emphasis on edges, with ATP leveraging semantic role labeling data and LeakDistill utilizing structural information such as edges in the FWAGs. In contrast, AMRBART and BiBL exhibit a significant drop in performance compared to the SPRING baseline, possibly due to their use of masking as an additional signal, as their masking strategies may not be beneficial for edge labels.

heading: 8 Performance Analysis
text: Seq2seq parsers show decreased performance for longer sentences since a single error at decoding time in an early step can lead to compound errors and suffer from exposure bias. We explore how this affects our model compared to SPRING, ATP and AMRBART. Figure 5 shows the performance
SPRING ATP AMRBART LeakDistill
on AMR 3.0 test set for buckets of 200 sentences split by the number of words. While performance is similar on shorter sentences, with AMRBART showing slightly better performance, in longer sentences of over 14 words LeakDistill fares better, especially compared to the baseline, which drops to 80 SMATCH points. This experiment also shows how performance is relatively stable for mediumlength sentences (10-30 words, oscillating around 85 points), while it starts deteriorating for longer ones. The high performance on short sentences is likely due to easy-to-parse structures, such as single date sentences.

heading: 9 Conclusion
text: We presented a new approach to training the Transformer architecture where partial information of the target sequence can be learned via self-knowledge distillation: the information can be leaked in the encoder implicitly through Transformer adapters which improve training but are switched off during inference. By employing this approach in AMR parsing, we achieved state-of-the-art results among non-ensemble methods. Moreover, we produced a lightweight AMR parser that outperforms SPRING while having four times fewer parameters. We also showed that, for all methods, performance degrades as the number of words increases.
Interestingly, our approach can potentially be used in other tasks, such as Relation Extraction, where alignments between input and target sequence elements exist, or structural information is unavailable at inference time.

heading: 10 Limitations
text: Our approach for training the Transformer architecture using self-knowledge distillation is promising, but there are still some limitations that need to be addressed in future work. One limitation is that our approach is only tested on the task of AMR parsing, and more evaluations are needed to see if it generalizes well to other tasks, such as Relation Extraction. Additionally, our approach, as is also the case for other current methods, exhibits performance degradation as the number of words in the sentence increases. This may be an indication of the current methods’ limitation or lack of robustness to longer sentences.
Another limitation is the added complexity and extra parameters required by the use of Transformer adapters, which increases the overall complexity of the architecture and training time. Even though our approach still achieves state-of-the-art results and it is as lightweight as previous systems at inference time, this fact should be considered by researchers if they should decide to adopt it for other tasks.
In summary, our approach presents an innovative way to train the Transformer architecture and achieve state-of-the-art results in AMR parsing. However, more work is needed to further improve the performance of the model and to apply it to other tasks as well.

heading: 11 Ethical considerations
text: In considering the ethical and social implications of our proposed approach to AMR parsing, we acknowledge that there are several important considerations to take into account.
One significant concern is the potential for bias in the training data and models, which can result in unfair or discriminatory outcomes for certain groups of individuals. Additionally, the training and test data may not be representative of the population that the model will be applied to, potentially leading to poor performance in specific domains.
Furthermore, our approach relies on the use of Transformer-based models, which have been shown to perpetuate societal biases present in the data used for training. It is, therefore, crucial to ensure that the data used for training is diverse and unbiased.
Moreover, the use of techniques such as selfknowledge distillation may lead to data leakage, where the model overfits the training data and performs poorly on new data, which could have negative impacts on the predictions.
In conclusion, even if we consider our approach does not have negative implications, it is important to note that bias and fairness are complex issues that require ongoing attention and improvement.

heading: Acknowledgments
text: The authors gratefully acknowledge the support of the European Union’s Horizon 2020 research project Knowledge Graphs at Scale (KnowGraphs) under the Marie Marie Skłodowska-Curie grant agreement No 860801.
The last author gratefully acknowledges the support of the PNRR MUR project PE0000013-FAIR.

heading: A Model Hyperparameters
text: Table 9 lists hyperparameters and search space for the experiments:
• LR sched. - learning rate scheduler
• KL temp. - Kullback–Leibler divergence temperature
• AMR 3 aligns. - type of alignments for AMR 3.0
• Mask. range - masking range. For each batch, we mask the input tokens with probability pmask, the value for which is sampled uniformly from the masking range. For instance, the [0; 0.15] range means pmask ∼ U(0, 0.15)
The LeakDistill experiments detailed in Table 2 were performed utilizing the final set of hyperparameters listed in Table 9. However, it should be noted that the experiment that did not involve KL loss did not necessitate the use of the variable α.

heading: B Hardware and size of the model
text: We performed experiments on a single NVIDIA 3090 GPU with 64GB of RAM and Intel® Core™ i9-10900KF CPU. The total number of trainable parameters of LeakDistill is 434,883,596. Training the model on the silver data took 33 hours, whereas further fine-tuning took 16 hours.

heading: C BLINK
text: All systems from Tables 3 and 4 use BLINK (Wu et al., 2020) for wikification. For this purpose, we used the blinkify.py script from the SPRING repository.

heading: D Metric
text: We evaluate AMR parsing using the SMATCH metric (Cai and Knight, 2013) and extra scores of Damonte et al. (2017): i) Unlabel, compute on the predicted graphs after removing all edge labels, ii) No WSD, compute while ignoring Propbank senses (e.g., duck-01 vs duck-02), iii) Wikification, F-score on the wikification (:wiki roles), iv) NER, F-score on the named entity recognition (:name roles), v) Negations, F-score on the negation detection (:polarity roles), vi) Concepts, F-score on the concept identification task, vii) Reentrancy, computed on reentrant edges only, viii) Semantic Role Labeling (SRL), computed on :ARG-i roles only.

heading: E Data
text: The AMR 3.0 (Kevin Knight, 2020) data used in this paper is licensed under the LDC User Agreement for Non-Members for LDC subscribers, which can be found here. The The Little Prince Corpus can be found here from the Information Science Institute of the University of Southern California.

heading: F Algorithms
text: Algorithm 2 shows one training step of the LeakDistill model.
Algorithm 2 One training step of the LeakDistill model
Input: X - batch of input sequences and WAGs, Y - batch of target graphs Set Model to Normal Mode LDnll, P robs1 ←Model(X,Y ) Set Model to Leak Mode Lleak, P robs2 ←Model(X,Y ) LKL ← KLDiv (Probs1, P robs2) L← αLKL + βLleak + LDnll Optimization step of L
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 10

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 10

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 6

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 2, 4, 5

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix E
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Self-evident given that AMR is widely used as a dataset for Semantic Parsing systems
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? our work is based on the AMR dataset from the LDC
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? out of scope of our work
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 6.1
C 3 Did you run computational experiments? Section 6

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix A

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 7, we run significance tests to compare the difference systems.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? no usage of such libs
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Our approach for training the Transformer architecture using self-knowledge distillation is promising, but there are still some limitations that need to be addressed in future work. One limitation is that our approach is only tested on the task of AMR parsing, and more evaluations are needed to see if it generalizes well to other tasks, such as Relation Extraction. Additionally, our approach, as is also the case for other current methods, exhibits performance degradation as the number of words in the sentence increases. This may be an indication of the current methods’ limitation or lack of robustness to longer sentences. Another limitation is the added complexity and extra parameters required by the use of Transformer adapters, which increases the overall complexity of the architecture and training time. Even though our approach still achieves state-of-the-art results and it is as lightweight as previous systems at inference time, this fact should be considered by researchers if they should decide to adopt it for other tasks. In summary, our approach presents an innovative way to train the Transformer architecture and achieve state-of-the-art results in AMR parsing. However, more work is needed to further improve the performance of the model and to apply it to other tasks as well