Venue: NAACL 2021

# SGL: Speaking the Graph Languages of Semantic Parsing via

Multilingual Translation

Luigi Procopio  Rocco Tripodi  Roberto Navigli

Sapienza NLP Group

Department of Computer Science, Sapienza University of Rome

{procopio,tripodi,navigli}@di.uniroma1.it

###### Abstract

Graph-based semantic parsing aims to represent textual meaning through directed graphs. As one of the most promising general-purpose meaning representations, these structures and their parsing have gained a significant interest _momentum_ during recent years, with several diverse formalisms being proposed. Yet, owing to this very heterogeneity, most of the research effort has focused mainly on solutions specific to a given formalism. In this work, instead, we reframe semantic parsing towards multiple formalisms as Multilingual Neural Machine Translation (mnmt), and propose sgl, a many-to-many _seq2seq_ architecture trained with an mnmt objective. Backed by several experiments, we show that this framework is indeed effective once the learning procedure is enhanced with large parallel corpora coming from Machine Translation: we report competitive performances on amr and cuca parsing, especially once paired with pre-trained architectures. Furthermore, we find that models trained under this configuration scale remarkably well to tasks such as cross-lingual amr parsing: sgl outperforms all its competitors by a large margin without even explicitly seeing non-English to amr examples at training time and, once these examples are included as well, sets an unprecedented state of the art in this task. We release our code and our models for research purposes at [https://github.com/SapienzaNLP/sgl](https://github.com/SapienzaNLP/sgl).

caption=languageresource

In this work, we set ourselves to address these issues and, in particular, we propose _Speak the Graph Languages_ (sgl), a many-to-many _seq2seq_ architecture which we show to competitively scale across formalisms and across languages.1 The key idea is to train a _seq2seq_ model with a Multilingual Neural Machine Translation (mnmt) objective, where, given an input text and an identifier denoting the desired output formalism, a single shared model has to learn to translate towards the corresponding linearized graph. We use amr and cuca as our cases in point to show the effectiveness of this framework. In particular, we show that, once the learning procedure also considers large parallel corpora coming from Machine Translation (mt), this configuration becomes an effective approach for framework-independent parsing via a single model. Even more interestingly, this model scales surprisingly well to cross-lingual parsing and is capable of navigating through translation paths like it\({}\rightarrow\textsc{amr}\),2 which it has never seen during training. The contributions of this work are therefore as follows:

Footnote 1: By _across languages_, we mean that the model is capable of performing cross-lingual semantic parsing as defined for amr by Damonte and Cohen (2018). Unless otherwise specified, we will follow this perspective throughout the paper.

* We reframe semantic parsing towards multiple formalisms and from multiple languages as multilingual machine translation;
* On amr parsing, our framework achieves competitive performances, surpassing most of its current competitors once paired with a pre-trained Transformer;
* We outperform all current alternatives in cross-lingual amr parsing without ever seeing non-English to amr examples at training time and push the current state of the art even further once we include these examples;
* On cucca parsing, we reach competitive results, outperforming a strong BERT-powered baseline Hershcovich and Arviv (2019).

We release our code and our models for research purposes.

## 2 Related Work

Our work is mainly concerned with semantic parsing in cucca and amr, considering also the cross-lingual setting introduced by Damonte and Cohen (2018) for the latter.

Semantic ParsingArguably among the formalisms that have drawn the most interest, amr has seen the emergence of a rich yet dedicated literature, with recent approaches that can be roughly clustered into two groups. On the one hand, several graph-based solutions have been proposed Lyu and Titov (2018); Zhang et al. (2019, 2019); Zhou et al. (2020); Cam and Lam (2020); among these solutions, Zhou et al. (2020) show the effectiveness of enhancing an aligner-free parser with latent syntactic information, whereas Cai and Lam (2020) present an iterative method to build and refine amr graphs incrementally. On the other hand, translation-based approaches, where _seq2seq_ models are trained to translate from natural language text to linearized graphs, have been shown to reach competitive performances, despite the scarcity of training data Konstas et al. (2017); van Noord and Bos (2017); Ge et al. (2019). Continuing this latter direction and arguably closest to our work, Xu et al. (2020) and Bevilacqua et al. (2021) show that these models, once paired with adequate pre-training, can behave on par or better than dedicated and more sophisticated graph-based alternatives, surpassing the performances of Cai and Lam (2020). In particular, similarly to our work, Xu et al. (2020) leverage a multilingual framework inspired by Johnson et al. (2017) and explore the possibility of pre-training on a range of related tasks, including mt; however, their focus is limited to showing the effectiveness of transfer learning from related tasks to English amr parsing.

Conversely, here we show that the benefits of multilingual _seq2seq_ frameworks are not limited to English text-to-amr but, rather, that they enable astonishing performances on unseen translation paths such as it\({}\rightarrow\textsc{amr}\) and competitive results on other frameworks, using vacca as our case in point. In this sense, we continue the recent cross-framework trend formally started by the shared task of Oepen et al. (2019), exploring the possibility of using translation-based approaches for framework-independent parsing, as opposed to the transition-based parsers proposed in that seminal work. Our findings are in line with the recent results reported by Oepen et al. (2020) and, in particular, by Ozaki et al. (2020), where the authors cast semantic parsing in multiple formalisms as translation towards a novel Plain Graph Notation 

(pgn) they devise. However, whereas they train different independent models for each framework, we explore the possibility of using a single _multilingual_ model.

Cross-lingual amrWhile most of the research effort in the amr community has been focused on English only, the seminal work of Damonte and Cohen (2018) gave rise to an interesting new direction, i.e. exploring the extent to which amr can act as an interlingua. The authors introduced a new problem, cross-lingual amr parsing, and defined it as the task of recovering, given a sentence in any language, the amr graph corresponding to its English translation. Using an adapted version of the transition-based parser originally proposed by Damonte et al. (2017) and training it on silver data generated through annotation projection, they examined whether amr graphs could be recovered starting from non-English sentences. Even though their models fell short when compared to mt alternatives,3 their work showed promising results and suggested that, despite translation divergences, amr could act effectively as an interlingua.

Footnote 3: The input sentence is first translated towards English and, then, an English parser is used.

Annotation projection has been focal in subsequent work as well. Bloshmi et al. (2020) propose an aligner-free cross-lingual parser, thus disposing of the need for word alignments in the annotation projection pipeline; their parser manages to outperform mt alternatives when both annotation projection and these baselines have access to comparable amounts of data. Conversely, Sheth et al. (2021) leverage powerful contextualized word embeddings to improve the foreign-text-to-English-AMR alignments, surpassing all previous approaches and, most importantly, the yet-unbeaten mt baselines that have access to larger amounts of data.

We stand out from previous research and show that, as a matter of fact, annotation projection techniques are not needed to perform cross-lingual amr parsing. By jointly training on parallel corpora from mt and the en\({}\rightarrow\)sp data we have, we find that a multilingual model can navigate unseen translation paths such as it\({}\rightarrow\)amr effectively, outperforming all current approaches by a significant margin; yet, annotation projection is naturally beneficial and, when its training data are taken into account as well, sgl pushes performances even further.

## 3 Speak the Graph Languages (sgl)

In this section, we describe sgl, our proposed approach to graph-based semantic parsing. We first explain the graph linearizations we employ for amr and cuca, along with their delinearizations (SS3.1). We then describe the _seq2seq_ modelling approach we use (SS3.2) and, finally, we present our multilingual framework (SS3.3).

### Graph Linearizations

We now describe how we convert the considered meaning representations into translatable text sequences (_linearization_), along with their reverse process (_delinearization_).

For amr parsing, as in van Noord and Bos (2017), we first simplify amr graphs by removing variables and wiki links. We then convert these stripped amr graphs into trees by duplicating co-referring nodes. At this point, in order to obtain the final linearized version of a given amr, we concatenate all the lines of its PENMAN notation Goodman (2020) together, replacing newlines and multiple spaces with single spaces (Figure 0(a) and 0(b)). Conversely, delinearization is performed by assigning a variable to each predicted concept, per

Figure 1: amr and cuca graphs, along with their linearizations, for the sentence ”After graduation, John moved to Paris”. To ease readability, linearizations are shown with newlines and indentation; however, when fed to the neural model, they are in a single-line single-space format.

 

forming Wikification,4 restoring co-referring nodes and, where possible, repairing any syntactically malformed subgraph.5 For both phases, we use the scripts released by van Noord and Bos (2017).6

Footnote 4: We use DBpedia Spotlight API (Daiber et al., 2013).

Footnote 5: Although trained to generate syntactically correct graphs, the outputs _seq2seq_ models produce may contain syntactic errors, such as brackets that do not match.

Footnote 6: [https://github.com/RikVN/AMFR](https://github.com/RikVN/AMFR)

For usca parsing, we employ a Depth-First Search (dfs) approach: starting from the root, we navigate the graph, using square brackets to delimit subgraph boundaries and special variables to denote terminal and non-terminal nodes; remote edges are denoted by a special modifier appended to their labels, while re-entrancies, that is, edges whose target is a node already seen, are handled by simply entering the respective variable (Figure 1c and 1d). Similarly to amr, delinearization is performed by back-parsing this sequence into a cuca graph, repairing malformed subgraphs when possible;7 additionally, as terminal nodes are anchored in cuca, we remove those whose anchoring is impossible. The linearization and delinearization scripts for this schema are released along with the rest of our code.

Footnote 7: Should repairing fail, the faulty subgraph is discarded altogether.

### Sequence-to-sequence Modelling

We employ neural _seq2seq_ models based upon the _Transformer_ architecture (Vaswani et al., 2017). This architecture is essentially composed of two building blocks, namely, a _Transformer encoder_ and a _Transformer decoder_. The encoder is a stack of \(N\) identical layers, each made up of two sublayers: the first is a multi-head self-attention mechanism, while the second is a position-wise fully connected feed-forward network. The decoder follows a similar architecture, presenting, however, an additional sub-layer that performs multi-head attention over the output of the encoder.

Within this work, we use two different kinds of Transformer architecture, _Cross_ and _mBART_(Liu et al., 2020). _Cross_ is a randomly initialized Transformer closely following the architecture depicted by Vaswani et al. (2017), except for a significant difference: we leverage a factorized embedding parameterization Lan et al. (2020), that is, we decompose the large vocabulary embedding matrix into two smaller matrices. While the first of these represents the actual embedding matrix and projects one-hot vectors into an embedding space whose dimension is lower than the Transformer hidden size, the second one takes care of projecting these intermediate representations towards the actual Transformer hidden space. This technique significantly reduces the number of parameters and, within the context of our experiments, did not show any significant performance penalty.

On the other hand, _mBART_ is a multilingual Transformer pre-trained in many languages over large-scale monolingual corpora. As amr anducca are naturally not included among the supported languages in the vocabulary, we apply an architectural change to _mBART_ and increase its vocabulary with two new language ids. More specifically, we augment its embedding matrix by adding two additional vectors, which we randomly initialize as in Tang et al. (2021).

### Multilingual Framework

In order to empower our models to support translation from and towards multiple languages, we employ a data-driven approach: we replace the start token of the decoder with a special tag specifying the language the encoder representations should be unrolled towards. Figure 2 shows an example of this schema. It is worth pointing out that, while for _Cross_ we do not feed the source language to the encoder, when using the _mBART_ model we follow its input format and do provide it.

Once data have been tagged according to this schema, we train a many-to-many translation model on both the semantic parsing and English-centric parallel corpora.8 Considering that our focus is on semantic parsing, we perform oversampling on the amr anduca datasets. Furthermore, when considering the parallel corpora from mt, we flip the training direction with probability \(0.5\), hence allowing our model to see at training time both the x en and en \(\rightarrow\) x training directions; we argue that this stochastic flip benefits our models in multiple ways:

Footnote 8: Henceforth, without loss of generality, we will use English as the source language of the mt data and denote by x all the target-side languages.

* As en \(\rightarrow\) x shares the source language with both en \(\rightarrow\)amr and en \(\rightarrow\)ucca, this results in positive transfer;
* As amr, cuca and en are significantly related, x \(\rightarrow\)en also results in positive transfer (similar target language); 

* Finally, x \(\rightarrow\) en allows our model to navigate unseen translation paths (i.e. _zero-shot_) such as it \(\rightarrow\) amr and thus tackle tasks like cross-lingual amr parsing.

## 4 Experimental Setup

We assess the effectiveness of our proposed approach by evaluating its performance on all translation paths where the target language is a graph formalism, the only exception being x \(\rightarrow\) cucca, with x any language but English. This choice is motivated by the fact that, differently from amr where cross-lingual amr aims to produce English-based meaning representations Damonte and Cohen (2018), cucca builds graphs on top of its tokens which are, consequently, inherently in the same language as the input text Hershcovich et al. (2019); we leave exploring this direction to future work.

### Models

We choose to use both _Cross_, a randomly initialized Transformer, and _mBART_, a multilingual pretrained Transformer, to better grasp the effects of this joint multilingual framework in different regimes. In particular, we explore the following configurations:

* models trained only on a single semantic parsing task (amr or vacca parsing) and without considering any parallel data, denoted by \(\textit{Cross}_{st}\) and \(\textit{mBART}_{st}\);
* models trained on both semantic parsing tasks and the mt data, denoted by \(\textit{Cross}_{mt}\) and \(\textit{mBART}_{mt}\).

Furthermore, so as to explore whether the training schedules we use result in underfitting for amr and vacca, we also consider \(\textit{Cross}^{ft}_{mt}\) and \(\textit{mBART}^{ft}_{mt}\), that is, \(\textit{Cross}_{mt}\) and \(\textit{mBART}_{mt}\) fine-tuned with a training schedule biased towards the semantic parsing formalism that is being considered.9 Footnote 9: We report further details on schedules, models and the training procedure in Appendices A and B.

### Datasets and Preprocessing

amrFor amr parsing, we use AMR-2.0 (LDC2017T10) and its recently released expansion, AMR-3.0 (LDC2020T02), amounting, respectively, to \(39\,260\) and \(59\,255\) manually-created sentence-graph pairs.

Cross-Lingual amrWe use _Abstract Meaning Representation 2.0 - Four Translations_Damonte and Cohen (2020) to investigate the performance of sgl on cross-lingual amr parsing. This corpus contains translations of the sentences in the test set of AMR-2.0 in Chinese (zh), German (de), Italian (it) and Spanish (es).

uccaWe replicate the setting of the CoNLL 2019 Shared Task Oepen et al. (2019). We train our models using the freely available10ucca portion of the training data; this corpus amounts to \(6\,572\) sentence-graph pairs, drawn from the English Web Treebank (2012T13) and English Wikipedia articles on celebrities. As no official development set was included in the data release, following Hershcovich and Arviv (2019), we reserve \(500\) instances and use them as the validation set. To the best of our knowledge, the full evaluation data have not been released yet and, therefore, we compare with state-of-the-art alternatives and report results only on _The Little Prince_, a released subset consisting of \(100\) manually-tagged sentences sampled from the homonymous novel.

Figure 2: Our sgl multilingual translation framework. Best seen in color.

 

Parallel DataWe use English-centric parallel corpora in four languages, namely, Chinese, German, Italian and Spanish; we employ _MultiUN_(Tiedemann, 2012) for Chinese and Spanish, _ParaCrawl_(Espla et al., 2019) for German, and _Europarl_(Tiedemann, 2012) for Italian. We perform a mild filtering over all the available parallel sentences and then take the first \(5M\) out of these.11

Footnote 11: See Appendix C for further details.

PreprocessingWe do not perform any preprocessing or tokenization, except for the graph linearizations explained in SS3.1 and Chinese simplification.12 Instead, we directly apply subword tokenizer with a Unigram Model (Kudo, 2018). When working with _Cross_ in a single-task setting on amr or vacca, we follow Ge et al. (2019) and use a vocabulary size of \(20k\) subwords; instead, when working in the multilingual setting, we increase this value to \(50k\) so as to better accommodate the increased amount of languages. Conversely, when using _mBART_, we always use the original vocabulary consisting of \(250k\) subwords.

Footnote 12: We use the _hanzicconv_ library ([https://github.com/benriey/hanziconv](https://github.com/benriey/hanziconv)).

### Evaluation

We evaluate amr and cross-lingual amr parsing by using the Smatch score13(Cai and Knight, 2013), a metric that computes the overlap between two graphs. Furthermore, in order to have a better picture of the systems' performances, we also report the fine-grained scores as computed by the evaluation toolkit14 of Damonte et al. (2017). For usca parsing, we employ the official evaluation metric15 of the shared task, conceptually similar to the Smatch score.

Footnote 13: [https://github.com/snowblink14/smatch](https://github.com/snowblink14/smatch)

## 5 Results

We now report the results gsl achieves focusing on the following translation paths: i) en \(\rightarrow\)amr (SS5.1); ii) x \(\rightarrow\)amr, with x any language among Chinese, German, Italian and Spanish (SS5.2); iii) en \(\rightarrow\)ucca (SS5.3).

### armr Parsing

We report the Smatch and fine-grained scores that sgL and its current state-of-the-art alternatives attain on AMR-2.0 in Table 1 (top). Among the competing systems considered, for Bevilacqua et al. (2021) we report their _BART_-powered baseline (SPRING\({}_{bart}\)) and their best performing model (SPRING).

As a first result, we want to highlight the significant boost that jointly training within our proposed framework on mt data provides; \(\textit{Cross}_{mt}\) outperforms \(\textit{Cross}_{st}\) by more than \(7\) points and reaches competitive performances when compared with current state-of-the-art approaches. Furthermore, the gap of \(1.4\) points between \(\textit{Cross}_{mt}\) and \(\textit{Cross}^{ft}_{mt}\) shows that the training schedule we use for _Cross_

\begin{table}
\begin{tabular}{c l c c c c c c c c} \hline \multicolumn{1}{c}{Model} & \multicolumn{1}{c}{Smatch} & \multicolumn{1}{c}{Unlabeled} & No-WSD & Concepts & Wiki & NER & Reentrancies & Negations & SRL \\ \hline \multirow{6}{*}{**Gallings**} & Ge et al. (2019) & \(74.3\) & \(77.3\) & \(74.8\) & \(84.2\) & \(71.3\) & \(82.4\) & \(58.3\) & \(64.0\) & \(70.4\) \\  & Zhang et al. (2019) & \(77.0\) & \(80.0\) & \(78.0\) & \(86.0\) & \(86.0\) & \(79.0\) & \(61.0\) & \(77.0\) & \(71.0\) \\  & Zhou et al. (2020) & \(77.5\) & \(80.4\) & \(78.2\) & \(85.9\) & \(86.5\) & \(78.8\) & \(61.1\) & \(76.1\) & \(71.0\) \\  & Cai and Lam (2020) & \(80.2\) & \(82.8\) & \(80.0\) & \(88.1\) & \(86.3\) & \(81.1\) & \(64.6\) & \(78.9\) & \(74.2\) \\  & Xu et al. (2020) & \(80.2\) & \(83.7\) & \(80.8\) & \(87.4\) & \(75.1\) & \(85.4\) & \(66.5\) & \(71.5\) & \(78.9\) \\  & SPRING\({}_{part}\) & \(83.8\) & \(86.1\) & \(84.4\) & \(\mathbf{90.2}\) & \(84.3\) & \(\mathbf{90.6}\) & \(70.8\) & \(74.4\) & \(79.6\) \\  & SPRING & \(\mathbf{84.5}\) & \(\mathbf{86.7}\) & \(\mathbf{84.9}\) & \(89.6\) & \(\mathbf{87.3}\) & \(83.7\) & \(\mathbf{72.3}\) & \(\mathbf{79.9}\) & \(79.7\) \\ \cline{2-10}  & \(\textit{Cross}_{dt}\) & \(70.7\) & \(75.1\) & \(71.3\) & \(80.3\) & \(75.7\) & \(78.9\) & \(58.9\) & \(58.6\) & \(68.5\) \\  & \(\textit{Cross}_{mt}\) & \(78.1\) & \(82.1\) & \(78.7\) & \(85.1\) & \(80.6\) & \(85.0\) & \(66.6\) & \(71.5\) & \(75.2\) \\  & \(\textit{Cross}^{ft}_{mt}\) & \(79.5\) & \(83.2\) & \(80.2\) & \(86.5\) & \(80.7\) & \(85.9\) & \(68.4\) & \(71.8\) & \(77.1\) \\ \cline{2-10}  & \(\textit{mBART}_{st}\) & \(81.7\) & \(85.1\) & \(82.1\) & \(88.4\) & \(81.9\) & \(90.3\) & \(70.7\) & \(73.4\) & \(79.7\) \\  & \(\textit{mBART}_{mt}\) & \(81.9\) & \(85.3\) & \(82.3\) & \(88.5\) & \(81.0\) & \(89.4\) & \(71.0\) & \(75.3\) & \(80.0\) \\  & \(\textit{mBART}^{ft}_{mt}\) & \(82.3\) & \(85.7\) & \(82.8\) & \(88.9\) & \(82.3\) & \(89.3\) & \(71.5\) & \(73.7\) & \(\mathbf{80.4}\) \\ \hline \hline \multirow{6}{*}{**Fashion**} & SPRING & \(\mathbf{83.0}\) & \(\mathbf{85.4}\) & \(\mathbf{83.5}\) & \(\mathbf{89.8}\) & \(\mathbf{82.7}\) & \(\mathbf{87.2}\) & \(70.4\) & \(\mathbf{73.0}\) & \(78.9\) \\ \cline{2-10}  & \(\textit{Cross}^{ft}_{mt}\) & \(78.1\) & \(81.9\) & \(78.7\) & \(85.3\) & \(76.6\) & \(81.3\) & \(67.6\) & \(68.5\) & \(76.2\) \\ \cline{1-1}  & \(\textit{mBART}_{st}\) & \(80.0\) & \(83.2\) & \(80.5\) & \(86.6\) & \(77.2\) & \(86.3\) & \(70.0\) & \(68.5\) & \(78.4\) \\ \cline{1-1}  & \(\textit{mBART}^{ft}_{mt}\) & \(81.2\) & \(84.4\) & \(81.6\) & \(88.4\) & \(77.7\) & \(86.5\) & \(\mathbf{71.1}\) & \(69.7\) & \(\mathbf{79.7}\) \\ \hline \end{tabular}
\end{table}
Table 1: Smatch and fine-grained results on AMR-2.0 (top) and AMR-3.0 (bottom).

 

does indeed result in underfitting for amr and that further training is beneficial; this fine-tuned alternative achieves \(79.5\) Smatch score, less than one point behind Xu et al. (2020a). Considering the similarity between the two approaches, this difference is likely caused by the increased number of tasks our model is asked to handle.

Once we replace _Cross_ with _mBART_, all performances rise significantly. In particular, even _mBART\({}_{st}\)_, a single-task variant with no additional data, outperforms all its alternatives except for SPRING and SPRING\({}_{bart}\)Bevilacqua et al. (2021), highlighting the potential of fully pre-trained Transformer language models for translation-based approaches. _mBART\({}_{mt}\)_ and _mBART\({}^{ft}_{mt}\)_ push performances further up, showing that the MT data are beneficial even in this pretrained setting and that the multi-task training set, which enables a single shared model to scale across formalisms and languages, is not detrimental to English amr parsing.

However, arguably more interesting is the comparison between the performances of _mBART_ models and SPRING, which, in contrast, builds upon the English-only _BART_Lewis et al. (2020). In particular, as SPRING\({}_{bart}\) outperforms even _mBART\({}^{ft}_{mt}\)_, this finding suggests that, as expected, _BART_ is more suitable than _mBART_ when dealing with English amr. However, as we show in SS5.2, our choice is beneficial for cross-lingual amr parsing and results in an interesting trade-off.

Finally, we also evaluate sgl on AMR-3.0 and report the results of _Cross\({}^{ft}_{mt}\)_, _mBART\({}_{st}\)_ and _mBART\({}^{ft}_{mt}\)_ when trained on this dataset (Figure 1 bottom). Overall, we witness a similar trend compared to AMR-2.0.

### Cross-lingual amr Parsing

We now show the performances of sgl on cross-lingual amr parsing in terms of Smatch score over Chinese (zh), German (de), Italian (it) and Spanish (es). For comparison, we report the results of the systems proposed by Damonte and Cohen (2018, amreAger), Bloshmi et al. (2020, xl-amr) and Sheth et al. (2021); along with their best systems, we also show the strongest mt baseline reported in Damonte and Cohen (2018, amreAger\({}_{MT}\)) and the zero-shot configuration explored in Blloshmi et al. (2020, xl-amr\({}_{\emptyset}\)).

Table 2 (top) shows a very interesting trend. First of all, _Cross\({}^{ft}_{mt}\)_ achieves competitive performances, falling short only when compared to the recent work of Sheth et al. (2021); in particular, it surpasses the strong amreAger\({}_{MT}\) baseline. The most interesting aspect of this result is that _Cross\({}^{ft}_{mt}\)_ attains these performances without ever seeing at training time any X  + amr translation path; this is in marked contrast with all previous literature and with the systems we report in Table 2. This finding clearly highlights the effectiveness of transfer learning and, by extension, of our proposed framework in this setting.

Secondly, the performances _mBART\({}_{st}\)_ achieve are astounding under multiple perspectives. First, to the best of our knowledge, it is the first reported result of amr systems achieving competitive performances on cross-lingual amr parsing in a fully zero-shot configuration: _mBART\({}_{st}\)_ is fine-tuned solely on en  + amr and then applied directly to X  + amr translation; especially when compared to xl-amr\({}_{\emptyset}\), the only similar approach we are aware of, the gap is significant. Second, among the languages we consider, the case of Chinese is especially interesting as it appears to require constrained decoding in order to work properly: in particular, we restrict the model to generate only subwords whose characters belong to the English alphabet.16 If we were to perform zh  + amr parsing with no additional decoding machinery, as for the other languages, performances would be significantly lower, with _mBART\({}_{st}\)_ attaining only \(31.9\). This performance drop is caused by

\begin{table}
\begin{tabular}{c l c c c c} \hline \hline  & Model & DE & ES & IT & ZH \\ \hline \multirow{4}{*}{**mBART**} & amreAger & \(39.0\) & \(42.0\) & \(43.0\) & \(35.0\) \\  & amreAger\({}_{MT}\) & \(57.0\) & \(60.0\) & \(58.0\) & \(50.0\) \\ \cline{2-5}  & XL-AMr\({}_{\emptyset}\) & \(32.7\) & \(39.1\) & \(37.1\) & \(25.9\) \\  & XL-AMr & \(53.0\) & \(58.0\) & \(58.1\) & \(41.5\) \\  & Sheth et al. (2021) & \(62.7\) & \(67.9\) & \(67.4\) & – \\ \cline{2-5}  & _Cross\({}_{mt}\)_ & \(60.8\) & \(62.9\) & \(63.2\) & \(51.8\) \\  & _Cross\({}^{ft}_{mt}\)_ & \(61.8\) & \(63.7\) & \(64.1\) & \(52.6\) \\  & _mBART\({}_{st}\)_ & \(54.8\) & \(60.4\) & \(63.6\) & \(47.8\) \\  & _mBART\({}^{ft}_{mt}\)_ & \(66.3\) & \(69.0\) & \(69.8\) & \(55.4\) \\  & _mBART\({}^{ft}_{mt}\)_ & \(65.8\) & \(69.2\) & \(69.6\) & \(54.8\) \\  & _mBART\({}^{ft}_{mt}\)_ + AP & \(\mathbf{69.8}\) & \(\mathbf{72.4}\) & \(\mathbf{72.3}\) & \(\mathbf{58.0}\) \\ \hline \hline \multirow{4}{*}{**mBART**} & Sheth et al. (2021) & \(66.9\) & \(69.6\) & \(71.0\) & – \\  & _mBART\({}^{ft}_{mt}\)_ + AP & \(\mathbf{73.3}\) & \(\mathbf{73.9}\) & \(\mathbf{73.4}\) & \(\mathbf{64.9}\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Smatch scores on cross-lingual amr parsing for both human (top, HT) and machine (bottom, MT) translations of the test set.

 

the model leaving some nodes of the graph untranslated, i.e. named entities left written in Chinese (\(\text{\#L}\boxplus\text{\#J}\) rather than _Obama_), which disrupts the auto-regressive nature of the decoding procedure and, besides, eventually results in a penalized Smatch score. Finally, despite the larger amount of pre-training _mBART_ has been exposed to, its bigger capacity and better Smatch score on English, _mBART\({}_{st}\)_ still falls short when compared to _Cross\({}^{ft}_{mt}\)_, highlighting the benefits of seeing related translation directions at training time.

_mBART\({}_{mt}\)_ pushes the bar further up, with performances on German, Spanish and Italian that are now only roughly \(10\) points behind their English counterparts. As _mBART\({}_{mt}\)_ significantly outperforms _mBART\({}_{st}\)_, this result shows that, despite the massive pretraining, parallel data are still beneficial for cross-lingual amr. Moreover, differently from English amr, _mBART\({}^{ft}_{mt}\)_ does not yield an improvement and, in fact, performances slightly drop on average.

While the scores _mBART\({}_{mt}\)_ attains are already unprecedented, it is natural to wonder whether annotation projection (AP) might yield a further beneficial effect. To this end, similarly to Blloshmi et al. (2020), we translate the input sentences of AMR-2.0 into the four languages under consideration17 and build a training set for each language by pairing the translated sentence with the original amr graph. We further fine-tune _mBART\({}^{ft}_{mt}\)_, including also these new datasets among the training data. This model, which we denote by _mBART\({}^{ft}_{mt}+\text{AP}\)_, surpasses further _mBART\({}_{mt}\)_, clearly underlining the beneficial effect of this technique.

Footnote 17: We use the MarianMT models Tiedemann and Thottingal, 2020) available in the _HuggingFace Transformers_ library Wolf et al. (2020).

Finally, following Sheth et al. (2021), we also report the results of sgl when evaluated on the machine-translated test set;18 similarly to their findings, we observe that, as the mismatch between the training and test set is reduced, our parser performs better in this setting than on the human-translated one.

Footnote 18: We use the same mt models we utilized for annotation projection.

### cuca Parsing

We report in Table 3 the performance of sgl on cuca parsing. We compare our approach with the original multi-task baseline Oepen et al. (2019) and \(3\) transition-based parsers that participated; in particular, we report the score of Che et al. (2019), the system that ranked first in both _all-framework_ and vacca parsing.

First of all, we note the result of _Cross\({}_{st}\)_; while its performance is far below the score Che et al. (2019) achieve, it still outperforms the original proposed baseline by more than \(10\) points. Furthermore, to the best of our knowledge, apart from the recent works proposed in the latest shared task of Oepen et al. (2020), this is the first reported result of translation-based approaches on cucca parsing.

Once plugged into our multilingual framework, cucca benefits from transfer learning to an even greater extent than amr parsing, likely owing to the smaller amount of training data: _Cross\({}_{mt}\)_ and especially _Cross\({}^{ft}_{mt}\)_ significantly reduce the gap between sgl and Che et al. (2019), with _Cross\({}^{ft}_{mt}\)_ outperforming the multi-task transition-based approach of Hershcovich and Arviv (2019). The usage of _mBART_ pushes up the system's performance further, with _mBART\({}_{st}\)_ achieving \(77.0\) and _mBART\({}_{mt}\)_\(79.9\); differently from amr, _mBART\({}^{ft}_{mt}\)_ suffers from overfitting and its performance is actually lower than that of _mBART\({}_{mt}\)_. Even though these scores are lower than those of Che et al. (2019), we argue that such results are still incredibly promising as they demonstrate the effectiveness of sgl in tackling cross-framework semantic parsing. Indeed, these results show that multilingual translation-based approaches allow for a single model to jointly accommodate different formalisms, each potentially linearized according to a different linearization scheme. Furthermore, we believe there is a significant margin for improvement on both the linearization used and the model; for instance, we did not consider node ids such as <_root_0>_ as special tokens, but instead had the unigram tokenizer handle them as if they were normal

\begin{table}
\begin{tabular}{p{56.9pt} p{42.7pt} p{42.7pt}} \hline \hline Model & Type & Score \\ \hline Oepen et al. (2019) & multi-task & \(41.0\) \\ Hershcovich and Arrivy (2019) & single-task & \(82.1\) \\ Hershcovich and Arvić (2019) & multi-task & \(73.1\) \\ Che et al. (2019) & multi-task & \(\mathbf{82.6}\) \\ \hline _Cross\({}_{st}\)_ & single-task & \(55.7\) \\ _Cross\({}_{mt}\)_ & multi-task & \(72.0\) \\ _Cross\({}^{ft}_{mt}\)_ & multi-task & \(75.1\) \\ \hline _mBART\({}_{st}\)_ & single-task & \(77.0\) \\ _mBART\({}_{mt}\)_ & multi-task & \(79.9\) \\ _mBART\({}^{ft}_{mt}\)_ & multi-task & \(76.9\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: cucca results on _The Little Prince_.

 

words.

Finally, we wish to point out that direct comparability between our system and those reported is hindered by the fact that our training setting is significantly different from theirs; in particular, we limit ourselves to two frameworks only and leverage resources (the parallel corpora from mt) whose usage was forbidden to the shared task participants.19 Nevertheless, we believe that their results are needed here to better contextualize the performances sgl obtains.

Footnote 19: Allowed resources are specified at: [http://svn.nlpl.eu/mrp/2019/public/resources.txt](http://svn.nlpl.eu/mrp/2019/public/resources.txt)

## 6 Analysis: is Mt the one helping?

Although the performances of _Cross\({}_{mt}\)_ are remarkable, _mBART\({}_{st}\)_ achieves competitive results on cross-lingual parsing and fares even better on English parsing. While _mBART\({}_{st}\)_ admittedly features a massive amount of pre-training, this pre-training is over monolingual corpora and, as such, the model has never seen any parallel data. We therefore wonder to what extent the parallel nature of the additional mt data we use is crucial for _Cross\({}_{mt}\)_.

To answer this question, we treat our mt corpora as monolingual data by sampling, for each instance, either the source or target side and converting the translation task into a denoising one: given an instance en\(\rightarrow\textsc{it}\), we sample either en or it with equal probability, denoting the result by \(Z\), and convert the instance into \(g(Z)\to Z\), where \(g\) is a noising function that corrupts the input text. We follow Lewis et al. (2020) and choose a noising function that masks \(35\%\) of the words by random sampling a span length from a Poisson distribution (\(\lambda=3.5\)). Applying this noisification scheme to the mt data, we train a model identical to _Cross\({}_{mt}\)_ and denote it by _Cross\({}^{N}_{mt}\)_.

As shown in Table 4, in this data regime, the parallel nature is crucial both for English and, especially, for cross-lingual parsing. While _Cross\({}^{N}_{mt}\)_ does yield a significant boost over _Cross\({}_{st}\)_, when compared instead to _Cross\({}_{mt}\)_, it is \(4\) points behind on occa parsing and only half way on amr parsing. The difference is even more marked in the cross-lingual setting, where _Cross\({}^{N}_{mt}\)_ simply does not work.

## 7 Conclusion

In this work, we presented sgl, a novel framing of semantic parsing towards multiple formalisms as Multilingual Neural Machine Translation. That is to say, given a sentence and the desired output formalism, a many-to-many neural model has to learn to translate from the input sentence to the corresponding linearized graph. Within this framework, we show that we can address the paucity of annotated data that afflicts semantic parsing effectively by performing the learning procedure jointly on large parallel corpora coming from mt, and leveraging the power of pre-trained Transformer language models.

Using amr and occa as our cases in point, we report competitive performances on their parsing, especially once pre-trained models enter the picture. Furthermore, we find that the benefit mt data provide goes beyond merely improving English-centric parsing, yielding astonishing performances on cross-lingual amr parsing as well, and allowing sgl to outperform all existing approaches by a large margin. Most interestingly, differently from all previous literature, this result is attained without ever explicitly seeing at training time the translation paths the model is tested upon. Once we use annotation projection and include these data as well, performances rise even further, attaining unprecedented results.

As future work, thanks to the nimbleness with which we can add new _languages_, we plan to assess the scalability of this framework as more formalisms are taken into account.

## Acknowledgments

The authors gratefully acknowledge

* the support of the ERC Consolidator Grant MOUSSE No. 726487. This work was partially supported by the MIUR under the grant "Dipartimenti di eccellenza 2018-2022" of the Department of Computer Science of Sapienza University.

 



## References

* Ahend and Rappoport (2013) Omri Abend and Ari Rappoport. 2013. Universal Conceptual Cognitive Annotation (UCCA). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Sofia, Bulgaria. Association for Computational Linguistics.
* Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, Sofia, Bulgaria. Association for Computational Linguistics.
* Bevilacqua et al. (2021) Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021. One SPRING to rule them both: Symmetric AMR semantic parsing and generation without a complex pipeline. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence (Rec).
* Biloshmi et al. (2020) Rexhina Biloshmi, Rocco Tripodi, and Roberto Navigli. 2020. XL-AMR: Enabling cross-lingual AMR parsing with transfer learning techniques. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics.
* Bonial et al. (2020) Claire Bonial, Lucia Donatelli, Mitchell Abrams, Stephanie M. Lukin, Stephen Tratz, Matthew Marge, Ron Artstein, David Traum, and Clare Voss. 2020. Dialogue-AMR: Abstract Meaning Representation for dialogue. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 684-695, Marseille, France. European Language Resources Association.
* Cai and Lam (2020) Deng Cai and Wai Lam. 2020. AMR parsing via graph-sequence iterative inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computational Linguistics.
* Cai and Knight (2013) Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Sofia, Bulgaria. Association for Computational Linguistics.
* Che et al. (2019) Wanxiang Che, Longxu Dou, Yang Xu, Yuxuan Wang, Yijia Liu, and Ting Liu. 2019. HIT-SCIR at MRP 2019: A unified pipeline for meaning representation parsing via efficient training and effective encoding. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pps 76-85, Hong Kong. Association for Computational Linguistics.
* Daiber et al. (2013) Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N. Mendes. 2013. Improving efficiency and accuracy in multilingual entity extraction. In Proceedings of the 9th International Conference on Semantic Systems (1-Semantics), page 121-124. Association for Computing Machinery.
* Four Translations LDC2020T07_. Web Download, Philadelphia: Linguistic Data Consortium.
* Damonte and Cohen (2018) Marco Damonte and Shay B. Cohen. 2018. Cross-lingual Abstract Meaning Representation parsing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pps 1146-1155, New Orleans, Louisiana. Association for Computational Linguistics.
* Damonte et al. (2017) Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for Abstract Meaning Representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, Valencia, Spain. Association for Computational Linguistics.
* Espla et al. (2019)Miquel Espla, Mikel Forcada, Gema Ramirez-Sanchez, and Hieu Hoang. 2019. ParaCrawl: Web-scale parallel corpora for the languages of the EU. In Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks, pps 118-119, Dublin, Ireland. European Association for Machine Translation.
* Ge et al. (2019) DongLai Ge, Junhui Li, Muhua Zhu, and Shoushan Li. 2019. Modeling source syntax and semantics for neural AMR parsing. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pps 4975-4981.
* Goodman (2020) Michael Wayne Goodman. 2020. Penman: An open-source library and tool for AMR graphs. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Online. Association for Computational Linguistics.
* Hajic et al. (2012) Jan Hajic, Eva Hajicova, Jarmila Panevova, Petr Sgall, Ondrej Bojar, Silvie Cinkova, Eva Fucukova, Marie Mikulova, Petr Pajas, Jan Popelka, Jifi Semecky, Jana Sindlerova, Jan Stepanek, Josef Toman, Zdenika Uresova, and Zdenek Zabockratsky. 2012. Announcing Prague Czech-English Dependency Treebank 2.0. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pps 3153-3160, Istanbul, Turkey. European Language Resources Association (ELRA).
* Hardy and Vlachos (2018) Hardy Hardy and Andreas Vlachos. 2018. Guided neural language generation for abstractive summarization using Abstract Meaning Representation. In

ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 768-773, Brussels, Belgium. Association for Computational Linguistics.
* Hershcovich et al. (2018) Daniel Hershcovich, Omri Abend, and Ari Rappoport. 2018. Multitask parsing across semantic representations. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 373-385, Melbourne, Australia. Association for Computational Linguistics.
* Hershcovich et al. (2019) Daniel Hershcovich, Zohar Aizenbud, Leshem Choshen, Elior Sulem, Ari Rappoport, and Omri Abend. 2019. SemEval-2019 task 1: Cross-lingual semantic parsing with UCCA. In _Proceedings of the 13th International Workshop on Semantic Evaluation_, pages 1-10, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
* Hershcovich and Arviv (2019) Daniel Hershcovich and Ofir Arviv. 2019. TUPA at MRP 2019: A multi-task baseline system. In _Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning_, pages 28-39, Hong Kong. Association for Computational Linguistics.
* Johnson et al. (2017) Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viegas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. _Transactions of the Association for Computational Linguistics_, 5:339-351.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Opensource toolkit for neural machine translation. In _Proceedings of ACL 2017, System Demonstrations_, pages 67-72, Vancouver, Canada. Association for Computational Linguistics.
* Konstas et al. (2017) Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: Sequence-to-sequence models for parsing and generation. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 146-157, Vancouver, Canada. Association for Computational Linguistics.
* Kudo (2018) Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
* Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In _International Conference on Learning Representations_.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Online. Association for Computational Linguistics.
* Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. _Transactions of the Association for Computational Linguistics_, 8:726-742.
* Lyu and Titov (2018) Chunchuanu Lyn and Ivan Titov. 2018. AMR parsing as graph prediction with latent alignment. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 397-407, Melbourne, Australia. Association for Computational Linguistics.
* Navigli (2018) Roberto Navigli. 2018. Natural Language Understanding: Instructions for (present and future) use. In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI)_, pages 5697-5702.
* Oppen et al. (2020) Stephan Oppen, Omri Abend, Lasha Abzainidze, Johan Bos, Jan Hajic, Daniel Hershcovich, Bin Li, Tim O'Gorman, Nianwen Xue, and Daniel Zeman. 2020. MRP 2020: The second shared task on cross-framework and cross-lingual meaning representation parsing. In _Proceedings of the CoNLL 2020 Shared Task: Cross-Framework Meaning Representation Parsing_, pages 1-22, Online. Association for Computational Linguistics.
* Opgen et al. (2019) Stephan Opgen, Omri Abend, Jan Hajic, Daniel Hershcovich, Marco Kuhlmann, Tim O'Gorman, Nianwen Xue, Jayeol Chun, Milan Straka, and Zdenka Uresova. 2019. MRP 2019: Cross-framework meaning representation parsing. In _Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning_, pages 1-27, Hong Kong. Association for Computational Linguistics.
* Oepen and Lonning (2006) Stephan Oepen and Jan Torce Lonning. 2006. Discriminant-based MRS banking. In _Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)_, Genoa, Italy. European Language Resources Association (ELRA).
* Ozaki et al. (2020) Hiroaki Ozaki, Gaku Morio, Yuta Koreeda, Terufumi Morishita, and Toshinori Miyoshi. 2020. Hitachi at 

MRP 2020: Text-to-graph-notation transducer. In Proceedings of the CoNLL 2020 Shared Task: Cross-Framework Meaning Representation Parsing, Online, pp. 40-52. External Links: Link Cited by: SS1.
* J. Sheth, Y. Lee, R. Fernandez Astudillo, T. Naseem, R. Florian, S. Roukos, and T. Ward (2021)Bootstrapping multilingual AMR with contextual word alignments. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL), Cited by: SS3.1.
* L. Song, D. Gildea, Y. Zhang, Z. Wang, and J. Su (2019)Semantic neural machine translation using AMR. Transactions of the Association for Computational Linguistics7, pp. 19-31. External Links: Document, ISSN 0895-9608 Cited by: SS1.
* E. Sulem, O. Abend, and A. Rappoport (2018)Semantic structural evaluation for text simplification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Vol. , New Orleans, Louisiana, pp. 685-696. External Links: ISBN 978-3-319-1422-1, Link, Document Cited by: SS1.
* Y. Tang, C. Tran, X. Li, P. Chen, N. Goyal, V. Chaudhary, J. Gu, and A. Fan (2021)Multilingual translation with extensible multilingual pretraining and finetuning. ArXivabs/2008.00401. External Links: Link Cited by: SS1.
* J. Tiedemann (2012)Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), Istanbul, Turkey, pp. 2214-2218. External Links: Link Cited by: SS1.
* J. Tiedemann and S. Thottingal (2020)OPUS-MT -- building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, Lisboa, Portugal, Euro-pcan Association for Machine Translation, Cited by: SS1.
* R. van Noord and J. Bos (2017)Neural semantic parsing by character-based translation: experiments with abstract meaning representations. Computational Linguistics in the Netherlands Journal7, pp. 93-108. External Links: Link Cited by: SS1.
* A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and H. Polosukhin (2017)Attention is all you need. In Advances in Neural Information Processing Systems, M. Dy Dykman, K. Heilig, W. Teh, B. Larochelle, and F. Garnett (Eds.), pp. 5998-6008. External Links: Link Cited by: SS1.
* T. Wolf, J. Chaumond, L. Debut, V. Sanh, C. Delangue, A. Moi, P. Cistac, M. Funtowicz, J. Davison, S. Shleifer, R. Louf, P. von Platen, T. Rault, Y. Jernite, T. Le Scao, S. Gugger, J. Phu, C. Ma, C. Shen, M. Drame, Q. Hohest, and A. Rush (2020)Transformers: state-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online, pp. 38-45. External Links: Link Cited by: SS1.
* D. Xu, J. Li, M. Zhu, M. Zhang, and G. Zhou (2020)Improving AMR parsing with sequence-to-sequence pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 2501-2511. External Links: Link Cited by: SS1.
* J. Xu, Y. Guo, and J. Hu (2020)Incorporate semantic structures into machine translation evaluation via UCCA. In Proceedings of the Fifth Conference on Machine Translation, Online, pp. 934-939. External Links: Link Cited by: SS1.
* S. Zhang, X. Ma, K. Duh, and B. Van Durme (2019)AMP parsing as sequence-to-graph transduction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 80-94. External Links: Link Cited by: SS1.
* S. Zhang, X. Ma, K. Duh, and B. Van Durme (2019)Broad-coverage semantic parsing as transduction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3786-3798. External Links: Link Cited by: SS1.
* Q. Zhou, Y. Zhang, D. Ji, and H. Tang (2020)AMR parsing with latent structural information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 4306-4319. External Links: Link Cited by: SS1.

## Appendix A Model and Training Details

CrossFor _Cross_, we leverage the implementation available in _OpenNMT-py_20Klein et al. (2017) and define a Transformer model closely following vaswani et al. (2017)except for the embedding modification we described. We use \(128\) as the embedding size and \(512\) as the Transformer hidden size when training in single-task settings; when scaling to the multilingual framework, we augment the hidden size to \(768\) so as to increase the model capacity. The number of layers in both the encoder and the decoder is set to \(6\), while the number of attention heads to \(8\); therefore, _Cross\({}_{st}\)_ contains 

\(46M\) trainable parameters, while _Cross\({}_{mt}\)\(105M\)_. We optimize the models parameters using Adam Kingma and Ba (2015) and the original scheduler of Vaswani et al. (2017). We train with an effective batch size of \(8000\) tokens and for a maximum of \(300k\) steps on a NVIDIA GeForce RTX 2080 Ti, using semantic parsing accuracy as the early stopping criterion with \(25k\) steps of patience; training lasted \(1\) day for _Cross\({}_{st}\)_ and roughly \(4\) days for _Cross\({}_{mt}\)_. We did not perform any significant tuning of decoding time parameters: we use \(5\) beams and, following Ge et al. (2019), we set the \(GNMT\) length penalty parameter \(alpha\) to \(1.0\).

MBartWe use the _mhart-large-cc25_ model provided by the Huggingface's _transformers_ library Wolf et al. (2020) and follow the specifications given in the original paper for the training configuration: we use \(2500\) warm-up steps, \(3e-5\) maximum learning rate, Adam as the optimizer and \(0.3\) dropout. We used \(8000\) effective token batch size and carried out training on a cloud-provided NVIDIA GeForce 3090, using semantic parsing accuracy as the early stopping criterion with \(25k\) steps of patience; training lasted slightly less than \(2\) days for _mBART\({}_{st}\)_ and around \(4\) days for _mBART\({}_{mt}\)_. As we fine-tune the original model, the number of parameters is unchanged (but for the two additional vectors for cuca and amr in the embedding matrix), with both _mBART\({}_{st}\)_ and _mBART\({}_{mt}\)_ featuring \(610M\) trainable parameters.

## Appendix B Training Task Scheduler

When traning _Cross\({}_{mt}\)_ and _mBART\({}_{mt}\)_, we oversample semantic parsing instances by sampling from the concatenation of amr and vacca with probability \(p_{1}=0.15\) and \(p_{1}=0.4\) respectively; the remaining probability \(p_{2}\) is uniformly divided among the mt corpora. When considering _Cross\({}^{ft}_{mt}\)_ and _mBART\({}^{ft}_{mt}\)_, we bias the schedule towards the semantic formalism under consideration by bringing its probability to \(p_{3}=0.8\); we assign the remaining \(p_{4}=0.2\) uniformly to the mt corpora.

## Appendix C Filtering over Parallel Corpora

We perform a mild filtering over parallel sentences, enforcing reasonable minimum, maximum and relative lengths; in particular, we discard sentences:

* that are shorter than \(25\) characters;
* that are longer than \(1\,000\) characters;
* where the relative character ratio between source and target is bigger than \(3.0\).

We then take the first \(5M\) out of these. This process results in \(5M\) parallel sentences for all four translation paths except en\(\rightarrow\) it, where, owing to the smaller size of _Europarl_, it results in only \(1.6M\) sentence pairs.