Cross-lingual Abstract Meaning Representation Parsing

Abstract: Abstract Meaning Representation (AMR) annotation efforts have mostly focused on English. In order to train parsers on other languages, we propose a method based on annotation projection, which involves exploiting annotations in a source language and a parallel corpus of the source language and a target language. Using English as the source language, we show promising results for Italian, Spanish, German and Chinese as target languages. Besides evaluating the target parsers on non-gold datasets, we further propose an evaluation method that exploits the English gold annotations and does not require access to gold annotations for the target languages. This is achieved by inverting the projection process: a new English parser is learned from the target language parser and evaluated on the existing English gold standard.

Abstract Meaning Representation (AMR) parsing is the process of converting natural language sentences into their corresponding AMR representations BIBREF0 . An AMR is a graph with nodes representing the concepts of the sentence and edges representing the semantic relations between them. Most available AMR datasets large enough to train statistical models consist of pairs of English sentences and AMR graphs.
The cross-lingual properties of AMR across languages has been the subject of preliminary discussions. The AMR guidelines state that AMR is not an interlingua BIBREF0 and bojar2014comparing categorizes different kinds of divergences in the annotation between English AMRs and Czech AMRs. xue2014not show that structurally aligning English AMRs with Czech and Chinese AMRs is not always possible but that refined annotation guidelines suffice to resolve some of these cases. We extend this line of research by exploring whether divergences among languages can be overcome, i.e., we investigate whether it is possible to maintain the AMR annotated for English as a semantic representation for sentences written in other languages, as in Figure 1 .
We implement AMR parsers for Italian, Spanish, German and Chinese using annotation projection, where existing annotations are projected from a source language (English) to a target language through a parallel corpus BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 . By evaluating the parsers and manually analyzing their output, we show that the parsers are able to recover the AMR structures even when there exist structural differences between the languages, i.e., although AMR is not an interlingua it can act as one. This method also provides a quick way to prototype multilingual AMR parsers, assuming that Part-of-speech (POS) taggers, Named Entity Recognition (NER) taggers and dependency parsers are available for the target languages. We also propose an alternative approach, where Machine Translation (MT) is used to translate the input sentences into English so that an available English AMR parser can be employed. This method is an even quicker solution which only requires translation models between the target languages and English.
Due to the lack of gold standard in the target languages, we exploit the English data to evaluate the parsers for the target languages. (Henceforth, we will use the term target parser to indicate a parser for a target language.) We achieve this by first learning the target parser from the gold standard English parser, and then inverting this process to learn a new English parser from the target parser. We then evaluate the resulting English parser against the gold standard. We call this “full-cycle” evaluation.
Similarly to evangcross, we also directly evaluate the target parser on “silver” data, obtained by parsing the English side of a parallel corpus.
In order to assess the reliability of these evaluation methods, we collected gold standard datasets for Italian, Spanish, German and Chinese by acquiring professional translations of the AMR gold standard data to these languages. We hypothesize that the full-cycle score can be used as a more reliable proxy than the silver score for evaluating the target parser. We provide evidence to this claim by comparing the three evaluation procedures (silver, full-cycle, and gold) across languages and parsers.
Our main contributions are:
AMR is a semantic representation heavily biased towards English, where labels for nodes and edges are either English words or Propbank frames BIBREF5 . The goal of AMR is to abstract away from the syntactic realization of the original sentences while maintaining its underlying meaning. As a consequence, different phrasings of one sentence are expected to provide identical AMR representations. This canonicalization does not always hold across languages: two sentences that express the same meaning in two different languages are not guaranteed to produce identical AMR structures BIBREF6 , BIBREF7 . However, xue2014not show that in many cases the unlabeled AMRs are in fact shared across languages. We are encouraged by this finding and argue that it should be possible to develop algorithms that account for some of these differences when they arise. We therefore introduce a new problem, which we call cross-lingual AMR parsing: given a sentence in any language, the goal is to recover the AMR graph that was originally devised for its English translation. This task is harder than traditional AMR parsing as it requires to recover English labels as well as to deal with structural differences between languages, usually referred as translation divergence. We propose two initial solutions to this problem: by annotation projection and by machine translation.
AMR is not grounded in the input sentence, therefore there is no need to change the AMR annotation when projecting to another language. We think of English labels for the graph nodes as ones from an independent language, which incidentally looks similar to English. However, in order to train state-of-the-art AMR parsers, we also need to project the alignments between AMR nodes and words in the sentence (henceforth called AMR alignments). We use word alignments, similarly to other annotation projection work, to project the AMR alignments to the target languages.
Our approach depends on an underlying assumption that we make: if a source word is word-aligned to a target word and it is AMR aligned with an AMR node, then the target word is also aligned to that AMR node. More formally, let $S = s_1 \dots s_{\vert s \vert }$ be the source language sentence and $T = t_1 \dots t_{\vert t \vert }$ be the target language sentence; $A_s(\cdot )$ be the AMR alignment mapping word tokens in $S$ to the set of AMR nodes that are triggered by it; $A_t(\cdot )$ be the same function for $T$ ; $v$ be a node in the AMR graph; and finally, $W(\cdot )$ be an alignment that maps a word in $S$ to a subset of words in $T$ . Then, the AMR projection assumption is: $T = t_1 \dots t_{\vert t \vert }$0 
In the example of Figure 1 , Questa is word-aligned with This and therefore AMR-aligned with the node this, and the same logic applies to the other aligned words. The words is, the and of do not generate any AMR nodes, so we ignore their word alignments. We apply this method to project existing AMR annotations to other languages, which are then used to train the target parsers.
We invoke an MT system to translate the sentence into English so that we can use an available English parser to obtain its AMR graph. Naturally, the quality of the output graph depends on the quality of the translations. If the automatic translation is close to the reference translation, then the predicted AMR graph will be close to the reference AMR graph. It is therefore evident that this method is not informative in terms of the cross-lingual properties of AMR. However, its simplicity makes it a compelling engineering solution for parsing other languages.
We now turn to the problem of evaluation. Let us assume that we trained a parser for a target language, for example using the annotation projection method discussed in Section "Related Work" . In line with rapid development of new parsers, we assume that the only gold AMR dataset available is the one released for English.
We can generate a silver test set by running an automatic (English) AMR parser on the English side of a parallel corpus and use the output AMRs as references. However, the silver test set is affected by mistakes made by the English AMR parser, therefore it may not be reliable.
In order to perform the evaluation on a gold test set, we propose full-cycle evaluation: after learning the target parser from the English parser, we invert this process to learn a new English parser from the target parser, in the same way that we learned the target parser from the English parser. The resulting English parser is then evaluated against the (English) AMR gold standard. We hypothesize that the score of the new English parser can be used as a proxy to the score of the target parser.
To show whether the evaluation methods proposed can be used reliably, we also generated gold test AMR datasets for four target languages (Italian, Spanish, German and Chinese). In order to do so, we collected professional translations for the English sentences in the AMR test set. We were then able to create pairs of human-produced sentences with human-produced AMR graphs.
A diagram summarizing the different evaluation stages is shown in Figure 2 . In the case of MT-based systems, the full-cycle corresponds to first translating from English to the target language and then back to English (back-translation), and only then parsing the sentences with the English AMR parser. At the end of this process, a noisy version of the original sentence will be returned and its parsed graph will be a noisy version of the graph parsed from the original sentence.
We run experiments on four languages: Italian, Spanish, German and Chinese. We use Europarl BIBREF8 as the parallel corpus for Italian, Spanish and German, containing around 1.9M sentences for each language pair. For Chinese, we use the first 2M sentences from the United Nations Parallel Corpus BIBREF9 . For each target language we extract two parallel datasets of 20,000/2,000/2,000 (train/dev/test) sentences for the two step of the annotation projection (English $\rightarrow $ target and target $\rightarrow $ English). These are used to train the AMR parsers. The projection approach also requires training the word alignments, for which we use all the remaining sentences from the parallel corpora (Europarl for Spanish/German/Italian and UN Parallel Corpus for Chinese). These are also the sentences we use to train the MT models. The gold AMR dataset is LDC2015E86, containing 16,833 training sentences, 1,368 development sentences, and 1,371 testing sentences.
Word alignments were generated using fast_align BIBREF10 , while AMR alignments were generated with JAMR BIBREF11 . AMREager BIBREF12 was chosen as the pre-existing English AMR parser. AMREager is an open-source AMR parser that needs only minor modifications for re-use with other languages. Our multilingual adaptation of AMREager is available at http://www.github.com/mdtux89/amr-eager-multilingual. It requires tokenization, POS tagging, NER tagging and dependency parsing, which for English, German and Chinese are provided by CoreNLP BIBREF13 . We use Freeling BIBREF14 for Spanish, as CoreNLP does not provide dependency parsing for this language. Italian is not supported in CoreNLP: we use Tint BIBREF15 , a CoreNLP-compatible NLP pipeline for Italian.
In order to experiment with the approach of Section "Conclusions" , we experimented with translations from Google Translate. As Google Translate has access to a much larger training corpus, we also trained baseline MT models using Moses BIBREF16 and Nematus BIBREF17 , with the same training data we use for the projection method and default hyper-parameters.
Smatch BIBREF18 is used to evaluate AMR parsers. It looks for the best alignment between the predicted AMR and the reference AMR and it then computes precision, recall and $F_1$ of their edges. The original English parser achieves 65% Smatch score on the test split of LDC2015E86. Full-cycle and gold evaluations use the same dataset, while silver evaluation is performed on the split of the parallel corpora we reserved for testing. Results are shown in Table 1 . The Google Translate system outperforms all other systems, but is not directly comparable to them, as it has the unfair advantage of being trained on a much larger dataset. Due to noisy JAMR alignments and silver training data involved in the annotation projection approach, the MT-based systems give in general better parsing results. The BLEU scores of all translation systems are shown in Table 2 .
There are several sources of noise in the annotation projection method, which affect the parsing results: 1) the parsers are trained on silver data obtained by an automatic parser for English; 2) the projection uses noisy word alignments; 3) the AMR alignments on the source side are also noisy; 4) translation divergences exist between the languages, making it sometimes difficult to project the annotation without loss of information.
Figure 3 shows examples of output parses for all languages, including the AMR alignments by-product of the parsing process, that we use to discuss the mistakes made by the parsers.
In the Italian example, the only evident error is that Infine (Lastly) should be ignored. In the Spanish example, the word medida (measure) is wrongly ignored: it should be used to generate a child of the node impact-01. Some of the :ARG roles are also not correct. In the German example, meines (my) should reflect the fact that the speaker is talking about his own country. Finally, in the Chinese example, there are several mistakes including yet another concept identification mistake: intend-01 is erroneously triggered.
Most mistakes involve concept identification. In particular, relevant words are often erroneously ignored by the parser. This is directly related to the problem of noisy word alignments in annotation projection: the parser learns what words are likely to trigger a node (or a set of nodes) in the AMR by looking at their AMR alignments (which are induced by the word alignments). If an important word consistently remains unaligned, the parser will erroneously learn to discard it. More accurate alignments are therefore crucial in order to achieve better parsing results. We computed the percentage of words in the training data that are learned to be non-content-bearing in each parser and we found that the Chinese parser, which is our least accurate parser, is the one that most suffer from this, with 33% non-content-bearing words. On the other hand, in the German parser, which is the highest scoring, only 26% of the words are non-content-bearing, which is the lowest percentage amongst all parsers.
In order to investigate the hypothesis that AMR can be shared across these languages, we now look at translational divergence and discuss how it affects parsing, following the classification used in previous work BIBREF19 , BIBREF20 , which identifies classes of divergences for several languages. sulem2015conceptual also follow the same categorization for French.
Figure 4 shows six sentences displaying these divergences. The aim of this analysis is to assess how the parsers deal with the different kind of translational divergences, regardless of the overall quality of the output.
This divergence happens when two languages use different POS tags to express the same meaning. For example, the English sentence I am jealous of you is translated into Spanish as Tengo envidia de ti (I have jealousy of you). The English adjective jealous is translated in the Spanish noun envidia. In Figure 4 a we note that the categorical divergence does not create problems since the parsers correctly recognized that envidia (jealousy/envy) should be used as the predicate, regardless of its POS.
This divergence happens when verbs expressed in a language with a single word can be expressed with more words in another language. Two subtypes are distinguished: manner and light verb. Manner refers to a manner verb that is mapped to a motion verb plus a manner-bearing word. For example, We will answer is translated in the Italian sentence Noi daremo una riposta (We will give an answer), where to answer is translated as daremo una risposta (will give an answer). Figure 4 b shows that the Italian parser generates a sensible output for this sentence by creating a single node labeled answer-01 for the expression dare una riposta.
In a light verb conflational divergence, a verb is mapped to a light verb plus an additional meaning unit, such as when I fear is translated as Io ho paura (I have fear) in Italian: to fear is mapped to the light verb ho (have) plus the noun paura (fear). Figure 4 e shows that also this divergence is dealt properly by the Italian parser: ho paura correctly triggers the root fear-01.
This divergence happens when verb arguments result in different syntactic configurations, for example, due to an additional PP attachment. When translating He entered the house with Lui è entrato nella casa (He entered in the house), the Italian translation has an additional in preposition. Also this parsed graph, in Figure 4 c, is structurally correct. The missing node he is due to pronoun-dropping, which is frequent in Italian.
This divergence occurs when the direction of the dependency between two words is inverted. For example, I like eating, where like is head of eating, becomes Ich esse gern (I eat likingly) in German, where the dependency is inverted. Unlike all other examples, in this case, the German parser does not cope well with this divergence: it is unable to recognize like-01 as the main concept in the sentence, as shown in Figure 4 d.
Finally, the parse of Figure 4 f has to deal with a thematic divergence, which happens when the semantic roles of a predicate are inverted. In the sentence I like grapes, translated to Spanish as Me gustan uvas, I is the subject in English while Me is the object in Spanish. Even though we note an erroneous reentrant edge between grape and I, the thematic divergence does not create problems: the parser correctly recognizes the :ARG0 relationship between like-01 and I and the :ARG1 relationship between like-01 and grape. In this case, the edge labels are important, as this type of divergence is concerned with the semantic roles.
AMR parsing for languages other than English has made only a few steps forward. In previous work BIBREF22 , BIBREF7 , BIBREF6 , nodes of the target graph were labeled with either English words or with words in the target language. We instead use the AMR annotation used for English for the target language as well, without translating any word. To the best of our knowledge, the only previous work that attempts to automatically parse AMR graphs for non-English sentences is by vanderwende2015amr. Sentences in several languages (French, German, Spanish and Japanese) are parsed into a logical representation, which is then converted to AMR using a small set of rules. A comparison with this work is difficult, as the authors do not report results for the parsers (due to the lack of an annotated corpus) or release their code.
Besides AMR, other semantic parsing frameworks for non-English languages have been investigated BIBREF23 , BIBREF24 , BIBREF25 , BIBREF26 . evangcross is the most closely related to our work as it uses a projection mechanism similar to ours for CCG. A crucial difference is that, in order to project CCG parse trees to the target languages, they only make use of literal translation. Previous work has also focused on assessing the stability across languages of semantic frameworks such as AMR BIBREF7 , BIBREF6 , UCCA BIBREF27 and Propbank BIBREF28 .
Cross-lingual techniques can cope with the lack of labeled data on languages when this data is available in at least one language, usually English. The annotation projection method, which we follow in this work, is one way to address this problem. It was introduced for POS tagging, base noun phrase bracketing, NER tagging, and inflectional morphological analysis BIBREF29 but it has also been used for dependency parsing BIBREF30 , role labeling BIBREF31 , BIBREF32 and semantic parsing BIBREF26 . Another common thread of cross-lingual work is model transfer, where parameters are shared across languages BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36 , BIBREF37 .
We introduced the problem of parsing AMR structures, annotated for English, from sentences written in other languages as a way to test the cross-lingual properties of AMR. We provided evidence that AMR can be indeed shared across the languages tested and that it is possible to overcome translational divergences. We further proposed a novel way to evaluate the target parsers that does not require manual annotations of the target language. The full-cycle procedure is not limited to AMR parsing and could be used for other cross-lingual problems in NLP. The results of the projection-based AMR parsers indicate that there is a vast room for improvements, especially in terms of generating better alignments. We encourage further work in this direction by releasing professional translations of the AMR test set into four languages.
The authors would like to thank the three anonymous reviewers and Sameer Bansal, Gozde Gul Sahin, Sorcha Gilroy, Ida Szubert, Esma Balkir, Nikos Papasarantopoulos, Joana Ribeiro, Shashi Narayan, Toms Bergmanis, Clara Vania, Yang Liu and Adam Lopez for their helpful comments. This research was supported by a grant from Bloomberg and by the H2020 project SUMMA, under grant agreement 688139.

Figure and table captions:
Figure 1: AMR alignments for a English sentence and its Italian translation.
Figure 2: Description of SILVER, FULL-CYCLE and GOLD evaluations. e stands for English and f stands for the target (foreign) language. Dashed lines represent the process of transferring learning across languages (e.g. with annotation projection). SILVER uses a parsed parallel corpus as reference (“Ref”), FULL-CYCLE uses the English gold standard (Gold e) and GOLD uses the target language gold standard we collected (Silver f ).
Table 1: Silver, gold and full-cycle Smatch scores for projection-based and MT-based systems.
Table 2: BLEU scores for Moses, Nematus and Google Translate (GT) on the (out-of-domain) LDC2015E86 test set
Figure 3: Parsed AMR graph and alignments (dashed lines) for an Italian sentence, a Spanish sentence, a German sentences and a Chinese sentence.
Figure 4: Parsing examples in several languages involving common translational divergence phenomena: (a) contains a categorical divergence, (b) and (e) conflational divergences, (c) a structural divergence, (d) an head swapping and (f) a thematic divergence.
Figure 5: Linear regression lines for silver and fullcycle.

Questions about this paper:
How is annotation projection done when languages have different word order?

Answers:
Word alignments are generated for parallel text, and aligned words are assumed to also share AMR node alignments.