SlowGuess/NAACL_2021_OCR huggingface dataset:
Luigi Procopio Rocco Tripodi Roberto Navigli
Department of Computer Science, Sapienza University of Rome
{procopio,tripodi,navigli}@di.uniromal.it
Graph-based semantic parsing aims to represent textual meaning through directed graphs. As one of the most promising general-purpose meaning representations, these structures and their parsing have gained a significant interest momentum during recent years, with several diverse formalisms being proposed. Yet, owing to this very heterogeneity, most of the research effort has focused mainly on solutions specific to a given formalism. In this work, instead, we reframe semantic parsing towards multiple formalisms as Multilingual Neural Machine Translation (MNMT), and propose SGL, a many-to-many seq2seq architecture trained with an MNMT objective. Backed by several experiments, we show that this framework is indeed effective once the learning procedure is enhanced with large parallel corpora coming from Machine Translation: we report competitive performances on AMR and UCCA parsing, especially once paired with pre-trained architectures. Furthermore, we find that models trained under this configuration scale remarkably well to tasks such as cross-lingual AMR parsing: SGL outperforms all its competitors by a large margin without even explicitly seeing non-English to AMR examples at training time and, once these examples are included as well, sets an unprecedented state of the art in this task. We release our code and our models for research purposes at https://github.com/SapienzaNLP/sgl.
Being able to associate natural language text with well-defined and machine-actionable meaning representations, i.e. the task of semantic parsing (SP), is one of the holy grails in Natural Language Processing (NLP) and Understanding (Navigli, 2018). Considering how a breakthrough in this direction would empower NLP systems to explicitly make sense of natural language, the ever-growing interest semantic parsing has been receiving really comes as no surprise. Graph-based
formalisms such as Abstract Meaning Representation (Banarescu et al., 2013, AMR), Elementary Dependency Structures (Oepen and Lonnning, 2006, EDS), Prague Tectogrammatical Graphs (Hajic et al., 2012, PTG), Universal Conceptual Cognitive Annotation (Abend and Rappoport, 2013, UCCA), inter alia, are emerging as the de facto standard for general-purpose meaning representations and have shown potential in Machine Translation (Song et al., 2019), Text Summarization (Hardy and Vlachos, 2018), Human-Robot Interaction (Bonial et al., 2020), and as evaluation metrics (Sulem et al., 2018; Xu et al., 2020b). These formalisms propose encoding meaning through directed graphs, however, each of them builds upon different linguistic assumptions, aims to target different objectives and, at a more practical level, assigns different functions to nodes and edges. For instance, while AMR uses nodes to encode concepts and edges to express the semantic relations between them, UCCA proposes using text tokens as terminal nodes and building graphs on top of them.
As a result of this heterogeneous landscape, often referred to as framework-specific balkanization (Oepen et al., 2020), graph-based semantic parsing has seen a proliferation of framework-specific solutions. However, approaches capable of competitively scaling across formalisms represent a natural desideratum, and recent works have started to explore this direction, examining the usage of multi-task learning in different architectures (Hershcovich et al., 2018; Oepen et al., 2019), or casting different formalisms under a unified framework where models can be trained to perform graph transduction (Zhang et al., 2019b). Nevertheless, despite achieving promising results, research in this direction has been hindered by the general lack of training data that afflicts semantic parsing. Indeed, due to the inherent complexity of this task, annotated corpora are still scarce and prohibitively expensive to expand.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 325-337 June 6-11, 2021. ©2021 Association for Computational Linguistics
In this work, we set ourselves to address these issues and, in particular, we propose Speak the Graph Languages (SGL), a many-to-many seq2seq architecture which we show to competitively scale across formalisms and across languages. The key idea is to train a seq2seq model with a Multilingual Neural Machine Translation (MNMT) objective, where, given an input text and an identifier denoting the desired output formalism, a single shared model has to learn to translate towards the corresponding linearized graph. We use AMR and UCCA as our cases in point to show the effectiveness of this framework. In particular, we show that, once the learning procedure also considers large parallel corpora coming from Machine Translation (MT), this configuration becomes an effective approach for framework-independent parsing via a single model. Even more interestingly, this model scales surprisingly well to cross-lingual parsing and is capable of navigating through translation paths like IT  $\rightarrow$  AMR, which it has never seen during training. The contributions of this work are therefore as follows:
- We reframe semantic parsing towards multiple formalisms and from multiple languages as multilingual machine translation;
- On AMR parsing, our framework achieves competitive performances, surpassing most of its current competitors once paired with a pre-trained Transformer;
- We outperform all current alternatives in crosslingual AMR parsing without ever seeing non-English to AMR examples at training time and push the current state of the art even further once we include these examples;
- On UCCA parsing, we reach competitive results, outperforming a strong BERT-powered baseline (Hershcovich and Arviv, 2019).
We release our code and our models for research purposes.
Our work is mainly concerned with semantic parsing in UCCA and AMR, considering also the cross
<sup>1</sup>By across languages, we mean that the model is capable of performing cross-lingual semantic parsing as defined for AMR by Damonte and Cohen (2018). Unless otherwise specified, we will follow this perspective throughout the paper. <sup>2</sup>IT stands for Italian.
lingual setting introduced by Damonte and Cohen (2018) for the latter.
Semantic Parsing Arguably among the formalisms that have drawn the most interest, AMR has seen the emergence of a rich yet dedicated literature, with recent approaches that can be roughly clustered into two groups. On the one hand, several graph-based solutions have been proposed (Lyu and Titov, 2018; Zhang et al., 2019a,b; Zhou et al., 2020; Cai and Lam, 2020); among these solutions, Zhou et al. (2020) show the effectiveness of enhancing an aligner-free parser with latent syntactic information, whereas Cai and Lam (2020) present an iterative method to build and refine AMR graphs incrementally. On the other hand, translation-based approaches, where seq2seq models are trained to translate from natural language text to linearized graphs, have been shown to reach competitive performances, despite the scarcity of training data (Konstas et al., 2017; van Noord and Bos, 2017; Ge et al., 2019). Continuing this latter direction and arguably closest to our work, Xu et al. (2020a) and Bevilacqua et al. (2021) show that these models, once paired with adequate pre-training, can behave on par or better than dedicated and more sophisticated graph-based alternatives, surpassing the performances of Cai and Lam (2020). In particular, similarly to our work, Xu et al. (2020a) leverage a multilingual framework inspired by Johnson et al. (2017) and explore the possibility of pre-training on a range of related tasks, including MT; however, their focus is limited to showing the effectiveness of transfer learning from related tasks to English AMR parsing.
Conversely, here we show that the benefits of multilingual seq2seq frameworks are not limited to English TEXT-to-AMR but, rather, that they enable astonishing performances on unseen translation paths such as IT  $\rightarrow$  AMR and competitive results on other frameworks, using UCCA as our case in point. In this sense, we continue the recent cross-framework trend formally started by the shared task of Oepen et al. (2019), exploring the possibility of using translation-based approaches for framework-independent parsing, as opposed to the transition-based parsers proposed in that seminal work. Our findings are in line with the recent results reported by Oepen et al. (2020) and, in particular, by Ozaki et al. (2020), where the authors cast semantic parsing in multiple formalisms as translation towards a novel Plain Graph Notation
(a) AMR graph
(b) AMR Linearization
(c) UCCA graph
Figure 1: AMR and UCCA graphs, along with their linearizations, for the sentence "After graduation, John moved to Paris". To ease readability, linearizations are shown with newlines and indentation; however, when fed to the neural model, they are in a single-line single-space format.
(d) UCCA Linearization
(PGN) they devise. However, whereas they train different independent models for each framework, we explore the possibility of using a single multilingual model.
Cross-lingual AMR While most of the research effort in the AMR community has been focused on English only, the seminal work of Damonte and Cohen (2018) gave rise to an interesting new direction, i.e. exploring the extent to which AMR can act as an interlingua. The authors introduced a new problem, cross-lingual AMR parsing, and defined it as the task of recovering, given a sentence in any language, the AMR graph corresponding to its English translation. Using an adapted version of the transition-based parser originally proposed by Damonte et al. (2017) and training it on silver data generated through annotation projection, they examined whether AMR graphs could be recovered starting from non-English sentences. Even though their models fell short when compared to MT alternatives, $^3$  their work showed promising results and suggested that, despite translation divergences, AMR could act effectively as an interlingua.
Annotation projection has been focal in subsequent work as well. Blloshmi et al. (2020) propose an aligner-free cross-lingual parser, thus disposing of the need for word alignments in the annotation projection pipeline; their parser manages to outperform MT alternatives when both annotation projection and these baselines have access to comparable amounts of data. Conversely, Sheth et al. (2021) leverage powerful contextualized word embeddings to improve the foreign-text-to-English-AMR alignments, surpassing all previous approaches and, most importantly, the yet-unbeaten MT baselines that have access to larger amounts of data.
We stand out from previous research and show that, as a matter of fact, annotation projection techniques are not needed to perform cross-lingual AMR parsing. By jointly training on parallel corpora from MT and the EN  $\rightarrow$  SP data we have, we find that a multilingual model can navigate unseen translation paths such as IT  $\rightarrow$  AMR effectively, outperforming all current approaches by a significant margin; yet, annotation projection is naturally beneficial and, when its training data are taken into account as well, SGL pushes performances even further.
3 Speak the Graph Languages (SGL)
In this section, we describe SGL, our proposed approach to graph-based semantic parsing. We first explain the graph linearizations we employ for AMR and UCCA, along with their delinearizations (§3.1). We then describe the seq2seq modelling approach we use (§3.2) and, finally, we present our multilingual framework (§3.3).
3.1 Graph Linearizations
We now describe how we convert the considered meaning representations into translatable text sequences (linearization), along with their reverse process (delinearization).
For AMR parsing, as in van Noord and Bos (2017), we first simplify AMR graphs by removing variables and wiki links. We then convert these stripped AMR graphs into trees by duplicating corefererring nodes. At this point, in order to obtain the final linearized version of a given AMR, we concatenate all the lines of its PENMAN notation (Goodman, 2020) together, replacing newlines and multiple spaces with single spaces (Figure 1a and 1b). Conversely, delinearization is performed by assigning a variable to each predicted concept, per
3The input sentence is first translated towards English and, then, an English parser is used.
forming Wikification, $^{4}$  restoring co-referring nodes and, where possible, repairing any syntactically malformed subgraph. $^{5}$  For both phases, we use the scripts released by van Noord and Bos (2017). $^{6}$
For UCCA parsing, we employ a Depth-First Search (DFS) approach: starting from the root, we navigate the graph, using square brackets to delimit subgraph boundaries and special variables to denote terminal and non-terminal nodes; remote edges are denoted by a special modifier appended to their labels, while re-entrancies, that is, edges whose target is a node already seen, are handled by simply entering the respective variable (Figure 1c and 1d). Similarly to AMR, delinearization is performed by back-parsing this sequence into a UCCA graph, repairing malformed subgraphs when possible; additionally, as terminal nodes are anchored in UCCA, we remove those whose anchoring is impossible. The linearization and delinearization scripts for this schema are released along with the rest of our code.
3.2 Sequence-to-sequence Modelling
We employ neural seq2seq models based upon the Transformer architecture (Vaswani et al., 2017). This architecture is essentially composed of two building blocks, namely, a Transformer encoder and a Transformer decoder. The encoder is a stack of  $N$  identical layers, each made up of two sublayers: the first is a multi-head self-attention mechanism, while the second is a position-wise fully connected feed-forward network. The decoder follows a similar architecture, presenting, however, an additional sub-layer that performs multi-head attention over the output of the encoder.
Within this work, we use two different kinds of Transformer architecture, Cross and mBART (Liu et al., 2020). Cross is a randomly initialized Transformer closely following the architecture depicted by Vaswani et al. (2017), except for a significant difference: we leverage a factorized embedding parameterization (Lan et al., 2020), that is, we decompose the large vocabulary embedding matrix into two smaller matrices. While the first of these represents the actual embedding matrix and projects one-hot vectors into an embedding space
whose dimension is lower than the Transformer hidden size, the second one takes care of projecting these intermediate representations towards the actual Transformer hidden space. This technique significantly reduces the number of parameters and, within the context of our experiments, did not show any significant performance penalty.
On the other hand,  $mBART$  is a multilingual Transformer pre-trained in many languages over large-scale monolingual corpora. As AMR and UCCA are naturally not included among the supported languages in the vocabulary, we apply an architectural change to  $mBART$  and increase its vocabulary with two new language ids. More specifically, we augment its embedding matrix by adding two additional vectors, which we randomly initialize as in Tang et al. (2021).
3.3 Multilingual Framework
In order to empower our models to support translation from and towards multiple languages, we employ a data-driven approach: we replace the start token of the decoder with a special tag specifying the language the encoder representations should be unrolled towards. Figure 2 shows an example of this schema. It is worth pointing out that, while for Cross we do not feed the source language to the encoder, when using the mBART model we follow its input format and do provide it.
Once data have been tagged according to this schema, we train a many-to-many translation model on both the semantic parsing and English-centric parallel corpora. Considering that our focus is on semantic parsing, we perform oversampling on the AMR and UCCA datasets. Furthermore, when considering the parallel corpora from MT, we flip the training direction with probability 0.5, hence allowing our model to see at training time both the  $\mathrm{X}\rightarrow \mathrm{EN}$  and  $\mathrm{EN}\rightarrow \mathrm{X}$  training directions; we argue that this stochastic flip benefits our models in multiple ways:
- As EN  $\rightarrow$  X shares the source language with both EN  $\rightarrow$  AMR and EN  $\rightarrow$  UCCA, this results in positive transfer;
- As AMR, UCCA and EN are significantly related,  $\mathrm{X} \rightarrow \mathrm{EN}$  also results in positive transfer (similar target language);
Henceforth, without loss of generality, we will use English as the source language of the MT data and denote by  $\mathbf{X}$  all the target-side languages.
4We use DBpedia Spotlight API (Daiber et al., 2013).
5 Although trained to generate syntactically correct graphs, the outputs seq2seq models produce may contain syntactic errors, such as brackets that do not match.
<sup>6</sup>https://github.com/RikVN/AMR
7Should repairing fail, the faulty subgraph is discarded altogether.
Figure 2: Our SGL multilingual translation framework. Best seen in color.
- Finally,  $\mathrm{X} \rightarrow \mathrm{EN}$  allows our model to navigate unseen translation paths (i.e. zero-shot) such as IT  $\rightarrow$  AMR and thus tackle tasks like cross-lingual AMR parsing.
We assess the effectiveness of our proposed approach by evaluating its performance on all translation paths where the target language is a graph formalism, the only exception being  $\mathrm{X} \rightarrow \mathrm{UCCA}$ , with  $\mathrm{X}$  any language but English. This choice is motivated by the fact that, differently from AMR where cross-lingual AMR aims to produce English-based meaning representations (Damonte and Cohen, 2018), UCCA builds graphs on top of its tokens which are, consequently, inherently in the same language as the input text (Hershcovich et al., 2019); we leave exploring this direction to future work.
We choose to use both Cross, a randomly initialized Transformer, and  $mBART$ , a multilingual pretrained Transformer, to better grasp the effects of this joint multilingual framework in different regimes. In particular, we explore the following configurations:
- models trained only on a single semantic parsing task (AMR or UCCA parsing) and without considering any parallel data, denoted by  $Cross_{st}$  and  $mBART_{st}$ ;
- models trained on both semantic parsing tasks and the MT data, denoted by  $Cross_{mt}$  and  $mBART_{mt}$ .
Furthermore, so as to explore whether the training schedules we use result in underfitting for AMR and
UCCA, we also consider  $Cross_{mt}^{ft}$  and  $mBART_{mt}^{ft}$ , that is,  $Cross_{mt}$  and  $mBART_{mt}$  fine-tuned with a training schedule biased towards the semantic parsing formalism that is being considered.
4.2 Datasets and Preprocessing
AMR For AMR parsing, we use AMR-2.0 (LDC2017T10) and its recently released expansion, AMR-3.0 (LDC2020T02), amounting, respectively, to 39260 and 59255 manually-created sentence-graph pairs.
Cross-Linguual AMR We use Abstract Meaning Representation 2.0 - Four Translations (Damonte and Cohen, 2020) to investigate the performance of SGL on cross-lingual AMR parsing. This corpus contains translations of the sentences in the test set of AMR-2.0 in Chinese (ZH), German (DE), Italian (IT) and Spanish (ES).
UCCA We replicate the setting of the CoNLL 2019 Shared Task (Oepen et al., 2019). We train our models using the freely available<sup>10</sup> UCCA portion of the training data; this corpus amounts to 6572 sentence-graph pairs, drawn from the English Web Treebank (2012T13) and English Wikipedia articles on celebrities. As no official development set was included in the data release, following Hershcovich and Arviv (2019), we reserve 500 instances and use them as the validation set. To the best of our knowledge, the full evaluation data have not been released yet and, therefore, we compare with state-of-the-art alternatives and report results only on The Little Prince, a released subset consisting of 100 manually-tagged sentences sampled from the homonymous novel.
<sup>9</sup>We report further details on schedules, models and the training procedure in Appendices A and B.
10Available on the UCCA website.
Model	Smatch	Unlabeled	No-WSD	Concepts	Wiki	NER	Reentrancies	Negations	SRL
AMR-2.0	Ge et al. (2019)	74.3	77.3	74.8	84.2	71.3	82.4	58.3	64.0	70.4
Zhang et al. (2019b)	77.0	80.0	78.0	86.0	86.0	79.0	61.0	77.0	71.0
Zhou et al. (2020)	77.5	80.4	78.2	85.9	86.5	78.8	61.1	76.1	71.0
Cai and Lam (2020)	80.2	82.8	80.0	88.1	86.3	81.1	64.6	78.9	74.2
Xu et al. (2020a)	80.2	83.7	80.8	87.4	75.1	85.4	66.5	71.5	78.9
SPRINGbart	83.8	86.1	84.4	90.2	84.3	90.6	70.8	74.4	79.6
SPRING	84.5	86.7	84.9	89.6	87.3	83.7	72.3	79.9	79.7
Crossst	70.7	75.1	71.3	80.3	75.7	78.9	58.9	58.6	68.5
Crossmt	78.1	82.1	78.7	85.1	80.6	85.0	66.6	71.5	75.2
Crossft	79.5	83.2	80.2	86.5	80.7	85.9	68.4	71.8	77.1
mBARTst	81.7	85.1	82.1	88.4	81.9	90.3	70.7	73.4	79.7
mBARTmt	81.9	85.3	82.3	88.5	81.0	89.4	71.0	75.3	80.0
mBARTftmt	82.3	85.7	82.8	88.9	82.3	89.3	71.5	73.7	80.4
AMR-3.0	SPRING	83.0	85.4	83.5	89.8	82.7	87.2	70.4	73.0	78.9
Crossftmt	78.1	81.9	78.7	85.3	76.6	81.3	67.6	68.5	76.2
mBARTst	80.0	83.2	80.5	86.6	77.2	86.3	70.0	68.5	78.4
mBARTftmt	81.2	84.4	81.6	88.4	77.7	86.5	71.1	69.7	79.7
Table 1: Smatch and fine-grained results on AMR-2.0 (top) and AMR-3.0 (bottom).
Parallel Data We use English-centric parallel corpora in four languages, namely, Chinese, German, Italian and Spanish; we employ MultiUN (Tiedemann, 2012) for Chinese and Spanish, ParaCrawl (Esplà et al., 2019) for German, and Europarl (Tiedemann, 2012) for Italian. We perform a mild filtering over all the available parallel sentences and then take the first  $5M$  out of these.[11]
Preprocessing We do not perform any preprocessing or tokenization, except for the graph linearizations explained in §3.1 and Chinese simplification. $^{12}$  Instead, we directly apply subword tokenization with a Unigram Model (Kudo, 2018). When working with Cross in a single-task setting on AMR or UCCA, we follow Ge et al. (2019) and use a vocabulary size of  $20k$  subwords; instead, when working in the multilingual setting, we increase this value to  $50k$  so as to better accommodate the increased amount of languages. Conversely, when using mBART, we always use the original vocabulary consisting of  $250k$  subwords.
We evaluate AMR and cross-lingual AMR parsing by using the Smatch score $^{13}$  (Cai and Knight, 2013), a metric that computes the overlap between two graphs. Furthermore, in order to have a better picture of the systems' performances, we also re
port the fine-grained scores as computed by the evaluation toolkit $^{14}$  of Damonte et al. (2017). For UCCA parsing, we employ the official evaluation metric $^{15}$  of the shared task, conceptually similar to the Smatch score.
We now report the results SGL achieves focusing on the following translation paths: i) EN  $\rightarrow$  AMR (§5.1); ii) X  $\rightarrow$  AMR, with X any language among Chinese, German, Italian and Spanish (§5.2); iii) EN  $\rightarrow$  UCCA (§5.3).
5.1 AMR Parsing
We report the Smatch and fine-grained scores that SGL and its current state-of-the-art alternatives attain on AMR-2.0 in Table 1 (top). Among the competing systems considered, for Bevilacqua et al. (2021) we report their BART-powered baseline  $(\mathrm{SPRING}_{bart})$  and their best performing model (SPRING).
As a first result, we want to highlight the significant boost that jointly training within our proposed framework on MT data provides;  $Cross_{mt}$  outperforms  $Cross_{st}$  by more than 7 points and reaches competitive performances when compared with current state-of-the-art approaches. Furthermore, the gap of 1.4 points between  $Cross_{mt}$  and  $Cross_{mt}^{ft}$  shows that the training schedule we use for  $Cross$
See Appendix C for further details.