Cross-lingual AMR Aligner: Paying Attention to Cross-Attention
Year: 2023

Abstract:
This paper introduces a novel aligner for Abstract Meaning Representation (AMR) graphs that can scale cross-lingually, and is thus capable of aligning units and spans in sentences of different languages. Our approach leverages modern Transformer-based parsers, which inherently encode alignment information in their cross-attention weights, allowing us to extract this information during parsing. This eliminates the need for English-specific rules or the Expectation Maximization (EM) algorithm that have been used in previous approaches. In addition, we propose a guided supervised method using alignment to further enhance the performance of our aligner. We achieve state-of-the-art results in the benchmarks for AMR alignment and demonstrate our aligner’s ability to obtain them across multiple languages. Our code will be available at github.com/Babelscape/AMR-alignment.

heading: 1 Introduction
text: At the core of Natural Language Understanding lies the task of Semantic Parsing, aimed at translating natural language text into machine-interpretable representations. One of the most popular semantic formalisms is the Abstract Meaning Representation (Banarescu et al., 2013, AMR), which embeds the semantics of a sentence in a directed acyclic graph, where concepts are represented by nodes, such as time, semantic relations between concepts by edges, such as :beneficiary, and the co-references by reentrant nodes, such as r representing rose. In crosslingual AMR, the English AMR graph represents the sentence in different languages (see Figure 1). To date, AMR has been widely used in Machine Translation (Song et al., 2019), Question Answering (Lim et al., 2020; Kapanipathi et al., 2021), Human-Robot Interaction (Bonial et al., 2020), Text Summarization (Hardy and Vlachos, 2018;
∗ Equal contributions.
Liao et al., 2018) and Information Extraction (Rao et al., 2017), among other areas.
The alignment between spans in text and semantic units in AMR graphs is an essential requirement for a variety of purposes, including training AMR parsers (Zhou et al., 2021), cross-lingual AMR parsing (Blloshmi et al., 2020), downstream task application (Song et al., 2019), or the creation of new semantic parsing formalisms (Navigli et al., 2022; Martínez Lorenzo et al., 2022). Despite the emergence of various alignment generation approaches, such as rule-based methods (Liu et al., 2018) and statistical strategies utilizing Expectation Maximization (EM) (Pourdamghani et al., 2014; Blodgett and Schneider, 2021), these methods rely heavily on English-specific rules, making them incompatible with cross-lingual alignment. Furthermore, even though several attempts extend the alignment to non-English sentences and graphs (Damonte and Cohen, 2018; Uhrig et al., 2021), these efforts are inherently monolingual and therefore lack the connection to the richer AMR graph bank available in English, which can be exploited as a source of interlingual representations.
On the other hand, current state-of-the-art AMR parsers are auto-regressive neural models (Bevilacqua et al., 2021; Bai et al., 2022) that do not generate alignment when parsing the sentence to produce the graph. Therefore, to obtain both, one needs to i) predict the graph and then ii) generate the alignment using an aligner system that is based on language-specific rules.
This paper presents the first AMR aligner that can scale cross-lingually by leveraging the implicit information acquired in Transformer-based parsers (Bai et al., 2022). We propose an approach for extracting alignment information from crossattention, and a guided supervised method to enhance the performance of our aligner. We eliminate the need for language-specific rules and enable simultaneous generation of the AMR graph and
1726
alignment. Our approach is efficient and robust, and is suitable for cross-lingual alignment of AMR graphs.
Our main contributions are: (i) we explore how Transformer-based AMR parsers preserve implicit alignment knowledge and how we can extract it; (ii) we propose a supervised method using crossattention to enhance the performance of our aligner, (iii) we achieve state-of-the-art results along different alignment standards and demonstrate the effectiveness of our aligner across languages.

heading: 2 Related Work
text: AMR alignment Since the appearance of AMR as a Semantic Parsing formalism, several aligner systems have surfaced that provide a link between the sentence and graph units. JAMR (Flanigan et al., 2014) is a widely used aligner system that employs an ordered list of 14 criteria, including exact and fuzzy matching, to align spans to subgraphs. However, this approach has limitations as it is unable to resolve ambiguities or learn novel alignment patterns. TAMR (Liu et al., 2018) extends JAMR by incorporating an oracle parser that selects the alignment corresponding to the highest-scored candidate AMR graph. ISI (Pourdamghani et al., 2014) aligner utilizes an EM algorithm to establish alignment between words and graphs’ semantic units. First, the graph is linearized, and then the EM algorithm is employed with a symmetrized scoring function to establish alignments. This method leads to more diversity in terms of alignment patterns, but fails to align easy-to-recognize patterns that could be aligned using rules. LEAMR (Blodgett and Schneider, 2021) is another aligner system that combines rules and EM. This approach aligns all the subgraph structures to any span in the sentence. However, it is based on language-specific rules, making it unsuitable for cross-lingual settings. Moreover, despite several attempts to extend the alignment to non-English languages An-
chiêta and Pardo (2020); Oral and Eryiğit (2022), these efforts are still monolingual since they rely on language-specific strategies. Consequently, in this paper we present an approach that fills this gap.
Cross-attention Most state-of-the-art systems for AMR parsing are based on Encoder-Decoder Transformers, specifically on BART (Lewis et al., 2020). These models consist of two stacks of Transformer layers, which utilize self- and crossattention as their backbone. The popularity of Transformer models has led to increased interest in understanding how attention encodes information in text and relates to human intuition (Vashishth et al., 2019) and definitions of explainability (Bastings and Filippova, 2020; Bibal et al., 2022). Research has been conducted on how attention operates, relates to preconceived ideas, aggregates information, and explains model behavior for tasks such as natural language inference(Stacey et al., 2021), Translation (Yin et al., 2021; Zhang and Feng, 2021; Chen et al., 2021), Summarization (Xu et al., 2020; Manakul and Gales, 2021) or Sentiment Analysis (Wu et al., 2020). Furthermore, there have been attempts to guide attention to improve interpretability or performance in downstream tasks (Deshpande and Narasimhan, 2020; Sood et al., 2020). However, to the best of our knowledge, there has been no prior study on attention for AMR parsing. This paper fills this gap by investigating the role of attention in AMR parsing.

heading: 3 Method
text: Originally described by Vaswani et al. (2017) as “multi-head attention over the output of the Encoder”, and referred to as cross-attention in Lewis et al. (2020), it enables the Decoder to attend to the output of the Encoder stack, conditioning the hidden states of the autoregressive component on the input text. Self-attention and cross-attention
modules are defined as:
Attention(Q,K, V ) = att(Q,K)V
att(Q,K) = softmax( QKT√
dk )
CrossAtt(Q,K, V ) =
Concat(head1, ..., headH)W O
headh = Attention(QW Q h ,KW K h , V W V h )
where K,V = Eℓ ∈ Rne×dkH and Q = Dℓ ∈ Rnd×dkH are the Encoder and Decoder hidden states at layer ℓ, ne and nd are the input and output sequence lengths, H is the number of heads, WQh ,W K h and W V h ∈ RdkH×dk are learned weights that project the hidden states to the appropriate dimensions, dk, for each head and WO ∈ RdkH×dkH is a final learned linear projection. Therefore in each head h and layer ℓ we define the attention weights as attℓh = att(DℓWQh , E ℓWKh ) ∈ Rnd×ne .

heading: 3.1 Unguided Cross-Attention
text: We argue that there is an intuitive connection between cross-attention and alignments. Under the assumption the Decoder will attend to the parts in the input that are more relevant to predicting the next token, we infer that, when decoding the tokens for a certain node in the graph, attention should focus on related tokens in the input, and therefore the words that align to that node. We will use the crossattention matrices (attℓh) to compute an alignment between the input and the output.

heading: 3.2 Guided Cross-Attention
text: We also aim to explore whether cross-attention can be guided by the alignment between the words of the sentence and the nodes of the graph. To this end, we construct a sparse matrix align ∈ Rnd×ne from the automatically-generated alignments:
align(i, j) =    1 if xi ∼ yj
0 if xi ≁ yj
where ∼ indicates alignment between subword token xi and graph token yj .
However, even though there are sparse versions of attention (Martins and Astudillo, 2016), these did not produce successful alignments in our experiments. Hence we choose to alleviate the constraint
of imposing sparsity by employing the scalar mixing approach introduced in ELMo (Peters et al., 2018). We learn a weighted mix of each head and obtain a single attention matrix:
attℓ = γ H−1∑
h=0
sℓhatt ℓ h ∈ Rnd×ne (1)
where s = softmax(a) with scalar learnable parameters γ, a0, . . . , aH .
The model has the flexibility to learn how to distribute weights such that certain heads give sparser attention similar to alignment, while others can encode additional information that is not dependent on alignment. In our experiments, we use the implementation of Bevilacqua et al. (2021, SPRING) to train our parser but add an extra cross-entropy loss signal:
L = − nd∑
j=1
log pBART (yj | y<j , x)
− nd∑
j=1∑ i align(i,j)>0
ne∑
i=1
log   eatt ℓ(i,j)
nd∑ k=1 eattℓ(i,k) align(i, j) ne∑ k=1 align(k, j)
 

heading: 3.3 Saliency Methods
text: A theoretical alternative to our reasoning about cross-attention is the use of input saliency methods. These methods assign higher importance to the input tokens that correspond to a particular node in the graph or were more important in their prediction during decoding. To obtain these importance weights, we employ Captum (Kokhlikyan et al., 2020), an open-source library for model interpretability and understanding, which provides a variety of saliency methods, including gradientbased methods such as Integrated Gradients (IG), Saliency (Simonyan et al., 2014), and Input X Gradient (IxG), backpropagation-based methods such as Deeplift (Shrikumar et al., 2017) and Guided Backpropagation (GB) (Springenberg et al., 2015), and finally occlusion-based methods (Zeiler and Fergus, 2014).
We obtain a weight matrix sal ∈ Rnd×ne with the same size as the cross-attention matrix and use it to extract alignments in the same fashion as the unguided cross-attention method. This approach allows us to explore the input tokens that have a greater impact on the decoding process and can aid in understanding the reasoning behind the alignments made by the model.

heading: 3.4 Alignment Extraction
text: Our algorithm1 to extract and align the input-output spans is divided into six steps:
1. Alignment score matrix: we create a matrix M ∈ Rnd×ne , where ne is the number of tokens in the sentence and nd is the number of tokens in the linearized graph, using the crossattention weights (attℓh or att
ℓ) as described in Section 3.
2. Span segmentation: For each sentence word, we sum the scores of tokens that belong to the same word column-wise in M . Then, for LEAMR alignments (see Section 4.2), the sentence tokens are grouped into spans using their span segmentation (see Appendix A).
3. Graph segmentation: We sum the score of tokens that belong to the same graph’s semantic unit row-wise in M .
4. Sentence graph tokens map: We iterate over all the graph’s semantic units and map them to the sentence span with highest score in M .
5. Special graph structures: We revise the mapping by identifying subgraphs that represent literal or matching spans – e.g., named entities, dates, specific predicates, etc. – and align them accordingly.
6. Alignment formatting: We extract the final alignments to the appropriate format using the resulting mapping relating graph’s semantic units to sentence spans.

heading: 4 Experimental Setup

heading: 4.1 Graph inventory
text: AMR 3.0 (LDC2020T02) consists of 59,255 sentence-graph pairs that are manually annotated. However, it lacks alignment information between the nodes in the graphs and the concepts in the sentences. We use the train split for the guided approach and use the respective validation and test splits from the alignment systems. Additionally, to evaluate cross-lingual performance, we use the gold German, Italian, and Spanish sentences of “AMR 2.0 – Four Translation” (LDC2020T07) which are human parallel translations of the test set
1The pseudo-algorithm is described in the Appendix C.
in AMR 2.02, paired with their English graphs from the AMR 3.0 test set. Despite this, as the graph inventory does not contain alignment information, it becomes necessary to access other repositories in order to obtain the alignment.

heading: 4.2 Alignment Standards
text: We propose an approach that is agnostic to different alignment standards and we evaluate it on two standards that are commonly used: ISI and LEAMR.
ISI The ISI standard, as described in (Pourdamghani et al., 2014), aligns single spans in the sentence to graphs’ semantic units (nodes or relations), and aligns relations and reentrant nodes when they appear explicitly in the sentence. The alignments are split into two sets of 200 annotations each, which we use as validation and test sets, updated to the AMR 3.0 formalism. For the cross-lingual alignment setup, we project English ISI graph-sentence alignments to the sentences in other languages, using the machine translation aligner (Dou and Neubig, 2021). This involves connecting the nodes in the graph to the spans in non-English sentences using the projected machine translation alignments between the English spans and the corresponding non-English sentence spans. By leveraging this, we are able to generate a silver alignment for cross-lingual AMR, which enables us to validate the model’s performance in a cross-lingual setup and determine its scalability across-languages.
LEAMR The LEAMR standard differentiates among four different types of alignment: i) Subgraph Alignments, where all the subgraphs that explicitly appear in the sentence are aligned to a list of consecutive spans, ii) Duplicate Subgraph, where all the subgraphs that represent omit repeated concepts in the sentence are aligned, iii) Relation Alignments, where all the relations that were not part of a previous subgraph structure are aligned, and iv) Reentrancy Alignments, where all the reentrant nodes are aligned. In contrast to ISI, all the semantic units in the graph are aligned to some list of consecutive spans in the text. We use 150 alignments as the validation set and 200 as the test set, which includes sentence-graph pairs from The Little Prince Corpus (TLP) complemented with randomly sampled pairs from AMR 3.0.
2The sentences of AMR 2.0 are a subset of AMR 3.0.

heading: 4.3 Model
text: We use SPRING (Bevilacqua et al., 2021) as our parsing model based on the BART-large architecture (Lewis et al., 2020) for English and SPRING based on mBART for non-English languages mBART (Liu et al., 2020) for the multilingual setting. We extract all attℓh matrices from a model trained on AMR 3.0 as in Blloshmi et al. (2021) in order to perform our unguided crossattention analysis. For the guided approach we re-train using the same hyperparameters as the original implementation, but with an extra loss signal as described in Section 3.2 based on either LEAMR or ISI. When using LEAMR alignments, we restructure the training split in order to exclude any pair from their test and validation sets.

heading: 5 Experiments

heading: 5.1 Correlation
text: In this study, we investigate the correlation between cross-attention and alignment by computing the Pearson’s r correlation coefficient between the attℓh matrix and the LEAMR alignment matrix align. To do so, we first flatten the matrices and remove any special tokens that are not relevant for alignment. As shown in Figure 2, there is a clear positive correlation between the two.
While we do not have a clear explanation for
why certain heads have a higher correlation than others, it is evident that there is a connection between cross-attention and alignment. For example, head 6 in layer 3 (i.e., att36) has a correlation coefficient of 0.635, approximately the same as the sum of the entire layer.
With regard to the saliency methods described in Section 3.3, the two most highly correlated methods were Saliency and GB, with a correlation coefficient of 0.575. Despite this result, we observe that saliency methods tend to focus more on essential parts of the sentence, such as the subject or predicate. These parts are usually aligned to more nodes and relations, which explains the high correlation, but they lack nuance compared to cross-attention.
Our best results were obtained by supervising layer 3 during training with the approach outlined in Section 3.2, using Cross-Entropy Loss on half of the heads (i.e., 3, 4, 5, 6, 7, 11, 12, and 15) that were selected based on their correlation on the validation set. This did not affect the performance of parsing. When we looked at att3 using the learned weighted mix from Equation 1 with LEAMR alignments, the correlation reached 0.866, which is significantly higher than any other method. Figure 2 shows the impact of supervising half the heads on layer 3 and how it influences heads in other layers.
To gain a better understanding of these results, we present an example from the TLP corpus in Fig-
ure 3 to illustrate the different methods, including cross-attention and saliency methods. The left image shows the cross-attention values for att36. Despite not having seen any alignment information, the model is able to correctly match non-trivial concepts such as "merchant" and "person". The center-left image illustrates how saliency methods focus on essential parts of the sentence, but lack nuance compared to cross-attention. The center-right image shows that supervising learning on layer 3 results in more condensed attention, which is associated with the improvement in correlation. However, it is important to note that the model can reliably attend to incorrect positions, such as aligning "pointer" to "merchant" instead of "sold".

heading: 5.2 Results
text: LEAMR Table 1 shows the performances of our two approaches on the LEAMR gold alignments compared to previous systems. We use the same evaluation setup as Blodgett and Schneider (2021), where the partial match assigns a partial credit from Jaccard indices between nodes and tokens. In both guided and unguided methods, we extract the score matrix for Algorithm 1 from the sum of the crossattention in the first four layers. We use a Wilcoxon signed-rank test (Wilcoxon, 1945) on the alignment matches per graph to check for significant differences. Both our approaches are significantly different compared to LEAMR (p=0.031 and p=0.007 respectively). However, we find no statistical difference between our unguided and guided approaches (p=0.481).
Our guided attention approach performs best, improving upon LEAMR on Subgraph (+0.5) and Relation (+2.6). For Reentrancy, performance is relatively low, and we will explore the reasons for this in Section 7. Perhaps most interesting is the performance of the unguided system using raw cross-attention weights from SPRING. The system remains competitive against the guided model without having access to any alignment information. It outperforms LEAMR which, despite being unsupervised with respect to alignments, relies on a set of inductive biases and rules based on alignments. While we also draw on specific rules related to the graph structure in post-processing, we will need to investigate their impact in an ablation study.
Relations that are argument structures (i.e., :ARG and :ARG-of ) usually depend on the predictions for their parent or child nodes; hence their improvement would be expected to be tied to the Subgraph Alignment. The results in Table 2 reassure us that this intuition is correct. Notice how for Single Relations (such as :domain or :purpose in Figure 3) the performance by LEAMR was much lower, even worse than that of ISI: Blodgett and Schneider (2021) argued that this was due to the model being overeager to align to frequent prepositions such as to and of. On the other hand, our unguided method achieves 15 points over ISI and 20 over LEAMR, which hints at the implicit knowledge on alignment that cross-attention encodes. Our guided approach experiences a considerable drop for Single Relations since it was trained on data generated by LEAMR, replicating its faulty behavior albeit
being slightly more robust.
ISI When we test our systems against the ISI alignments, both our models achieve state-of-the-
art results, surpassing those of previous systems, including LEAMR. This highlights the flexibility of cross-attention as a standard-agnostic aligner (we provide additional information in Appendix B). Table 3 shows the performance of our systems and compares ones with the ISI alignment as a reference. We omit relations and Named Entities to focus solely on non-rule-based alignments and have a fair comparison between systems. Here, our aligner does not rely on any span-segmentation, hence nodes and spans are aligned solely based on which words and nodes share the highest crossattention values. Still, both our alignments outperform those of the comparison systems in English.
Moreover, only our approach achieves competitive results in Spanish, German and Italian – obtaining 40 points more on average above the second best model – while the other approaches are hampered by the use of English-specific rules. However, we found two reasons why non-English systems
perform worse than in English: 1) linguistic divergences (as explained in Wein and Schneider (2021)), and ii) the machine translation alignment error.

heading: 6 Ablation Study
text: Gold spans LEAMR relies on a span segmentation phase, with a set of multiword expressions and Stanza-based named entity recognition. We use the same system in order to have matching sentence spans. However, these sometimes differ from the gold spans, leading to errors. Table 4 (left) shows performance using an oracle that provides gold spans, demonstrating how our approach still outperforms LEAMR across all categories.
Rules All modern alignment systems depend on rules to some degree. For instance, we use the subgraph structure for Named Entities, certain relations are matched to their parent or child nodes, etc. (see Appendix A for more details). But what is the impact of such rules? As expected, both LEAMR and our unguided method see a considerable performance drop when we remove them. For Relation, LEAMR drops by almost 60 points, since it relies heavily on the predictions of parent and child nodes to provide candidates to the EM model. Our unguided approach also suffers from such dependency, losing 25 points. However, our guided model is resilient to rule removal, dropping by barely one point on Subgraph and 5 points on Relation.
Layers Figure 2 shows how alignment acts
differently across heads and layers. We explore this information flow in the Decoder by extracting the alignments from the sum of layers at different depths. The right of Table 4 shows this for both our unguided and guided models, as well as the Saliency method. [3] indicates the sum of heads in the supervised layer, while [3]* is the learned weighted mix. From our results early layers seem to align more explicitly, with performance dropping with depth. This corroborates the idea that Transformer models encode basic semantic information early (Tenney et al., 2019). While layers 7 and 8 did show high correlation values, the cross-attention becomes more disperse with depth, probably due to each token encoding more contextual information.

heading: 7 Error Analysis
text: We identify two main classes of error that undermine the extraction of alignments.
Consecutive spans Because each subgraph in LEAMR is aligned to a list of successive spans, the standard cannot deal correctly with transitive phrasal verbs. For example, for the verb "take off" the direct object might appear in-between ("He took his jacket off in Málaga"). Because these are not consecutive spans, we align just to "take" or "off".
Rules We have a few rules for recognizing subgraph structures, such as Named Entities, and align them to the same spans. However, Named Entity structures contain a placeholder node indicating the entity type; when the placeholder node appears explicitly in the sentence, the node should not be
part of the Named Entity subgraph. For example, when aligning ‘Málaga’, the city, the placeholder node should be aligned to city while our model aligned it to Málaga.

heading: 8 Cross-lingual Analysis: A Case Study
text: To investigate the potential causes of misalignment between English and non-English languages, we conduct a case study that qualitatively examines the differences in alignment generated by different systems and languages. Figure 4 illustrates the sentence, "why is it so hard to understand?" with its human translations in German, Spanish, and Italian, and its AMR. In the Italian translation, the subject of the verb is omitted, while in the Spanish translation the focus of the question is modified from asking the reason why something is difficult to understand to asking directly what is difficult to understand, making "qué" the subject. As a consequence, in both cases making it impossible to align "it" with any word in either the Italian or Spanish sentence by Machine Translation Alignment. Table 5 presents the alignments generated for the sentence in each language and with each model in ISI format. Although our model was able to align the node "it" by aligning it with the conjugated verb in the Italian sentence and with the word "qué" in the Spanish sentence, which serves as the subject, this resulted in an error in our evaluation since the alignment of "it" was not projected in either Italian or Spanish. In addition, we also observed the performance of jamr and tamr, which are rulebased systems, and found that they were only able to align the word "so" in the German translation, as it shares the same lemma in English. In contrast, LEAMR was able to detect more alignments
due to its requirement to align all nodes to a corresponding word in the target language. However, the alignments generated by LEAMR appeared to be almost entirely random.

heading: 9 Conclusion
text: In this paper, we have presented the first AMR aligner that can scale cross-lingually and demonstrated how cross-attention is closely tied to alignment in AMR Parsing. Our approach outperforms previous aligners in English, being the first to align cross-lingual AMR graphs. We leverage the cross-attention from current AMR parsers, without overhead computation or affecting parsing quality. Moreover, our approach is more resilient to the lack of handcrafted rules, highlighting its capability as a standard- and language-agnostic aligner, paving the way for further NLP tasks. As a future direction, we aim to conduct an analysis of the attention heads that are not correlated with the alignment information in order to identify the type of information they capture, such as predicate identification, semantic relations, and other factors. Additionally, we plan to investigate how the alignment information is captured across different NLP tasks and languages in the cross-attention mechanism of sequence-tosequence models. Such analysis can provide insights into the inner workings of the models and improve our understanding of how to enhance their performance in cross-lingual settings.

heading: 10 Limitations
text: Despite the promising results achieved by our proposed method, there are certain limitations that need to be noted. Firstly, our approach relies heavily on the use of Transformer models, which can be computationally expensive to train and run. Additionally, the lower performance of our aligner for languages other than English is still a substantial shortcoming, which is discussed in Section 5.2.
Furthermore, our method is not adaptable to nonTransformer architectures, as it relies on the specific properties of Transformer-based models to extract alignment information.
Lastly, our method is based on the assumption that the decoder will attend to those input tokens that are more relevant to predicting the next one. However, this assumption may not always hold true in practice, which could lead to suboptimal alignments.
In conclusion, while our proposed method presents a promising approach for cross-lingual AMR alignment, it is important to consider the aforementioned limitations when applying our method to real-world scenarios. Future research could focus on addressing these limitations and exploring ways to improve the performance of our aligner for languages other than English.

heading: 11 Ethics Statement
text: While our approach has shown itself to be effective in aligning units and spans in sentences of different languages, it is important to consider the ethical and social implications of our work.
One potential concern is the use of Transformerbased models, which have been shown to perpetuate societal biases present in the data used for training. Our approach relies on the use of these models, and it is therefore crucial to ensure that the data used for training is diverse and unbiased. Furthermore, the use of cross-attention in our approach could introduce new ways to supervise a model in order to produce harmful or unwanted model predictions. Therefore, it is crucial to consider the ethical implications of any guidance or supervision applied to models and to ensure that any training data used to guide the model is unbiased and does not perpetuate harmful stereotypes or discrimination.
Additionally, it is important to consider the potential impact of our work on under-resourced languages. While our approach has shown to be ef-
fective in aligning units and spans in sentences of different languages, it is important to note that the performance gap for languages other than English still exists. Further research is needed to ensure that our approach is accessible and beneficial for under-resourced languages.

heading: Acknowledgments
text: The authors gratefully acknowledge the support of the European Union’s Horizon 2020 research project Knowledge Graphs at Scale (KnowGraphs) under the Marie Marie Skłodowska-Curie grant agreement No 860801.
The last author gratefully acknowledges the support of the PNRR MUR project PE0000013-FAIR.

heading: A LEAMR Alignment Rules
text: The LEAMR standard has some predefined strategies for alignments that were followed during their annotation, as well as fixed in their alignment pipeline along EM. We kept a few of these strategies when extracting the alignment, just those related to the structure of the graph, but not those concerning token matching between the sentence and the graph.
A.1 Subgraph • Nodes have-org-role-91 and have-rel-role-91
follow a fixed structure related to a person ie. the sentence word enemy is represented as person→ have-rel-role-91→ enemy, therefore for such subgraphs we use the alignment from the child node.
• Similarly for Named Entities, we align the whole subgraph structure based on its child nodes which indicate its surfaceform. However this leads to some errors as described in Section 7.
• We align node amr-unknown to the question mark if it appears in the sentence.
A.2 Relations • For the relation :condition we align it to the
word if when it appears in the sentence.
• :purpose is aligned with to when in the sentence.
• :ARGX relations are aligned to the same span as the parent node, while :ARGX-of to that of the child, since they share the alignment of the predicate they are connected to.
• For :mod and :duration we use the alignment from the child node.
• For :domain and :opX we use the alignment from the parent node.

heading: B Extra Results
text: B.1 LEAMR Results We explore the variance with different seeds when guiding cross-attention. Table 1 reports on a single seed selected at random. Table 6 shows the results for five different seeds as well as the average and standard deviation. We observe some variance, especially for those alignment types with fewer
elements; however, average performance is always higher than previous approaches.

heading: C Alignment Extraction Algorithm
text: Algorithm 1 shows the procedure for extracting the alignment between spans in the sentence and the semantic units in the graphs, using a matrix that weights Encoder tokens with the Decoder tokens

heading: D AMR parsing
text: Since our guided approach was trained with a different loss than the SPRING model, it could influence the performance in the Semantic Parsing task.
Therefore, we also tested our model in the AMR parsing task using the test set of AMR 2.0 and AMR 3.0. Table 7 shows the result, where we can observe how our model preserves the performance on parsing.
Algorithm 1 Procedure for extracting the alignment between spans in the sentence and the semantic units in the graphs, using a matrix that weights Encoder tokens with the Decoder tokens.
1: function EXTRACTALIGNMENTS(encoderTokens,DecoderTokens, scoreMatrix) 2: alignmentMap← dict() 3: spansList← SPANS(encoderTokens) ▷ Extract sentence spans as in LEAMR 4: spanPosMap← TOK2SPAN(encoderTokens) ▷ Map input tokens to spans 5: graphPosMap← TOK2NODE(DecoderTokens) ▷ Map output tokens to graph unit 6: COMBINESUBWORDTOKENS(scoreMatrix) 7: for DecoderTokenPos,GraphUnit in graphPosMap do 8: encoderTokensScores← scoreMatrix[DecoderTokenPos] 9: maxScorePos← ARGMAX(encoderTokensScores) 10: alignmentMap[GraphUnit]← SELECTSPAN(spansList,maxScorePos) 11: end for 12: fixedMatches← GETFIXEDMATCHES(graphPosMap) ▷ Look for rule based matches 13: alignmentMap← APPLYFIXEDMATCHES(alignmentMap, fixedMatches) 14: alignments← FORMATALIGNMENT(alignmentMap) 15: return alignments 16: end function

heading: E Hardware
text: Experiments were performed using a single NVIDIA 3090 GPU with 64GB of RAM and Intel® Core™ i9-10900KF CPU. Training the model took 13 hours, 30 min per training epoch while evaluating on the validation set took 20 min at the end of each epoch. We selected the best performing epoch based on the SMATCH metric on the validation set.

heading: F Data
text: The AMR data used in this paper is licensed under the LDC User Agreement for Non-Members for LDC subscribers, which can be found here. The The Little Prince Corpus can be found here from the Information Science Institute of the University of Southern California.

heading: G Limitations
text: Even though our method is an excellent alternative to the current AMR aligner system, which is standard and task-agnostic, we notice some drawbacks when moving to other autoregressive models or languages:
Model In this work, we studied how Cross Attention layers retain alignment information between input and output tokens in auto-regressive models. In Section 5.1, we examined which layers in state-of-the-art AMR parser models based on BART-large best preserve this information. Unfortunately, we cannot guarantee that these layers are
optimal for other auto-regressive models, and so on. As a result, an examination of cross-attention across multiple models should be done before developing the cross-lingual application of this approach.
Sentence Segmentation It is necessary to apply LEAMR’s Spam Segmentation technique to produce the alignment in LEAMR format (Section 3.4). However, this segmentation method has several flaws: i) As stated in Section 7, this approach does not deal appropriately with phrasal verbs and consecutive segments; ii) the algorithm is Englishspecific; it is dependent on English grammar rules that we are unable to project to other languages. Therefore we cannot extract the LEAMR alignments in a cross-lingual AMR parsing because we lack a segmentation procedure. However, although LEAMR alignment has this constraint, ISI alignment does not require any initial sentence segmentation and may thus be utilized cross-lingually.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 9

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 10

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: In the Abstract and Introduction

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: Grammarly, we check the use of English of our paper
B 3 Did you use or create scientific artifacts? Section 4, 5, 6 and 7

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: In the Appendix
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We believe these are self-explained by the licences discussed for each artifact.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We base our work on widely used datasets which already performed these steps.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4 and Appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4 and Appendix.
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5 and Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Table 5 in Appendix

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Discussed through the paper when needed.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future research could focus on addressing these limitations and exploring ways to improve the performance of our aligner for languages other than English