Introduction
Semantic parsing is a task of transducing natural language to meaning representations, which in turn can be expressed through many different semantic formalisms including lambda calculus BIBREF1, DCS BIBREF2, Discourse Representation Theory (DRT) BIBREF3, AMR BIBREF4 and so on. This availability of annotated data in English has translated into the development of a plethora of models, including encoder-decoders BIBREF5, BIBREF6 as well as tree or graph-structured decoders BIBREF5, BIBREF7, BIBREF0, BIBREF8.
Whereas the majority of semantic banks focus on English, recent effort has focussed on building multilingual representations, e.g. PMB BIBREF9, MRS BIBREF10 and FrameNetBIBREF11. However, manually annotating meaning representations in a new language is a painstaking process which explains why there are only a few datasets available for different formalisms in languages other than English. As a consequence, whereas the field has made great advances for English, little work has been done in other languages.
We ask: can we learn a semantic parser for English and test it where in another where annotations are not available? What would that require?
To answer this question, previous work have leveraged machine translation techniques to map the semantics from a language to another BIBREF12. However, these methods require parallel corpora to extract automatic alignments which are often noisy or not available at all.
In this paper we explore parameter-shared models instead, where a model is trained on English using language independent features and tested in a target language.
To show how this approach performs, we focus on the Parallel Meaning Bank BIBREF13 – a multilingual semantic bank, where sentences in English, German, Italian and Dutch have been annotated with their meaning representations. The annotations in the PMB are based on Discourse Representation Theory BIBREF3, a popular theory of meaning representation designed to account for intra and inter-sentential phenomena, like temporal expressions and anaphora. Figure 1 shows an example DRT for the sentence `I sat down and opened my laptop' in its canonical `box' representation. A DRS is a nested structure with the top part containing the discourse references and the bottom with unary and binary predicates, as well as semantic constants (e.g. `speaker'). DRS can be linked to each other via logic operator (e.g. $\lnot $, $\rightarrow $, $\diamond $) or, as in this case, discourse relations (e.g. CONTINUATION, RESULT, ELABORATION, etc.).
To test our approach we leverage the DRT parser of liu2018discourse, an encoder-decoder architecture where the meaning representation is reconstructed in three stages, coarse-to-fine, by first building the DRS skeleton (i.e. the `box' structures) and then fill each DRS with predicates and variables. Whereas the original parser utilizes a sequential Bi-LSTM encoder with monolingual lexical features, we experiment with language-independent features in the form of cross-lingual word-embeddings, universal PoS tags and universal dependencies. In particular, we also make use of tree encoders to assess whether modelling syntax can be beneficial in cross-lingual settings, as shown for other semantic tasks (e.g. negation scope detection BIBREF14).
Results show that language-independent features are a valid alternative to projection methods for cross-lingual semantic parsing. We show that adding dependency relation as features is beneficial, even when they are the only feature used during encoding. However, we also show that modeling the dependency structure directly via tree encoders does not outperform a sequential BiLSTM architecture for the three languages we have experimented with.
Methods ::: Model
In this section, we describe the modifications to the coarse-to-fine encoder-decoder architecture of BIBREF0; for more detail, we refer the reader to the original paper.
Methods ::: Model ::: Encoder
BiLSTM. We use BIBREF0's Bi-LSTM as baseline. However, whereas the original model represents each token in the input sentence as the concatenation of word ($e_{w_i}$) and lemma embeddings, we discard the latter and add a POS tag embedding ($e_{p_i}$) and dependency relation embedding ($e_{d_i}$) feature. These embeddings are concatenated to represent the input token. The final encoder representation is obtained by concatenating both final forward and backward hidden states.
TreeLSTM. To model the dependency structure directly, we use a child-sum tree-LSTM BIBREF15, where each word in the input sentence corresponds to a node in the dependency tree. In particular, summing across children is advantageous for cross-lingual tasks since languages might display different word orders. Computation follows Equation (1).
Po/treeLSTM. Completely discarding word order might hurt performance for related languages, where a soft notion of positioning can help. To this end, we add a positional embeddings $P_i$ BIBREF16 that helps the child-sum tree-LSTM discriminating between the left and right child of a parent node. This is computed following Equation (2) where $i$ is the position of the word, $j$ is the $jth$ dimension in total $d$ dimensions.
Bi/treeLSTM. Finally, similarly to chen2017improved, we combine tree-LSTM and Bi-LSTM, where a tree-LSTM come is initialized using the last layer of a Bi-LSTM, which encodes order information. Computation is shown in Equation (3).
Methods ::: Model ::: Decoder
The decoder of liu2018discourse reconstructs the DRS in three steps, by first predicting the overall structure (the `boxes'), then the predicates and finally the referents, with each subsequent step being conditioned on the output of the previous. During predicate prediction, the decoder uses a copying mechanism to predict those unary predicates that are also lemmas in the input sentence (e.g. `eat'). For the those that are not, soft attention is used instead. No modifications were done to the decoder; for more detail, we refer the reader to the original paper.
Methods ::: Data
We use the PMB v.2.1.0 for the experiments. The dataset consists of 4405 English sentences, 1173 German sentences, 633 Italian sentences and 583 Dutch sentences. We divide the English sentences into 3072 training sentences, 663 development and 670 testing sentences. We consider all the sentences in other languages as test set.
In order to be used as input to the parser, liu2018discourse first convert the DRS into tree-based representations, which are subsequently linearized into PTB-style bracketed sequences. This transformation is lossless in that re-entrancies are duplicated to fit in the tree structure. We use the same conversion in this work; for further detail we refer the reader to the original paper.
Finally, it is worth noting that lexical predicates in PMB are in English, even for non-English languages. Since this is not compatible with our copy mechanism, we revert predicates to their original language by substituting them with the lemmas of the tokens they are aligned to (since gold alignment information is included in the PMB).
Methods ::: Cross-lingual features
In order to make the model directly transferable to the German, Italian and Dutch test data, we use the following language-independent features.
Multilingual word embeddings. We use the MUSE BIBREF17 pre-trained multilingual word embeddings and keep them fixed during training.
UD relations and structure. We use UDPipe BIBREF18 to obtain parses for English, German, Italian and Dutch. UD relation embeddings are randomly initialized and updated.
Universal POS tags. We use the Universal POS tags BIBREF19 obtained with UDPipe parser. Universal POS tag embeddings are randomly initialized and updated during training.
Methods ::: Model comparison
We use the BiLSTM model as baseline (Bi) and compare it to the child-sum tree-LSTM (tree) with positional information added (Po/tree), as well as to a treeLSTM initialized with the hidden states of the BiLSTM(Bi/tree). We also conduct an ablation study on the features used, where WE, PE and DE are the word-embedding, PoS embedding and dependency relation embedding respectively. For completeness, along with the results for the cross-lingual task, we also report results for monolingual English semantic parsing, where word embedding features are randomly initialized.
Methods ::: Evaluation
We use Counter BIBREF20 to evaluate the performance of our models. Counter looks for the best alignment between the predicted and gold DRS and computes precision, recall and F1. For further details about Counter, the reader is referred to van2018evaluating. It is worth reminding that unlike other work on the PMB BIBREF21, BIBREF0 does not deal with presupposition. In the PMB, presupposed variables are extracted from a main box and included in a separate one. In our work, we revert this process so to ignore presupposed boxes. Similarly, we also do not deal with sense tags which we aim to include in future work.
Results and Analysis
Table TABREF12 shows the performance of our cross-lingual models in German, Italian and Dutch. We summarize the results as follows:
Dependency features are crucial for zero-shot cross-lingual semantic parsing. Adding dependency features dramatically improves the performance in all three languages, when compared to using multilingual word-embedding and universal PoS embeddings alone. We hypothesize that the quality of the multilingual word-embeddings is poor, given that models using embeddings for the dependency relations alone outperform those using the other two features.
TreeLSTMs slightly improve performance only for German. TreeLSTMs do not outperform a baseline BiLSTM for Italian and Dutch and they show little improvement in performance for German. This might be due to different factors that deserve more analysis including the performance of the parsers and syntactic similarity between these languages. When only dependency features are available, we found treeLSTM to boost performance only for Dutch.
BiLSTM are still state-of-the-art for monolingual semantic parsing for English. Table TABREF14 shows the result for the models trained and tested in English. Dependency features in conjunction with word and PoS embeddings lead to the best performance; however, in all settings explored treeLSTMs do not outperform a BiLSTM.
Results and Analysis ::: Error Analysis
We perform an error analysis to assess the quality of the prediction for operators (i.e. logic operators like “Not” as well as discourse relations “Contrast”), non-lexical predicates, such as binary predicates (e.g. Agent(e,x)) as well as unary predicates (e.g. time(t), entity(x), etc.), as well as for lexical predicates (e.g. open(e)). Results in Table TABREF15 show that predicting operators and binary predicates across language is hard, compared to the other two categories. Prediction of lexical predicates is relatively good even though most tokens in the test set where never seen during training; this can be attributable to the copy mechanism that is able to transfer tokens from the input directly during predication.
Related work
Previous work have explored two main methods for cross-lingual semantic parsing. One method requires parallel corpora to extract alignments between source and target languages using machine translation BIBREF11, BIBREF22, BIBREF23 The other method is to use parameter-shared models in the target language and the source language by leveraging language-independent features such as multilingual word embeddings, Universal POS tags and UD BIBREF24, BIBREF25, BIBREF26, BIBREF27. For semantic parsing, encoder-decoder models have achieved great success. Amongst these, tree or graph-structured decoders have recently shown to be state-of-the-art BIBREF5, BIBREF7, BIBREF0, BIBREF28, BIBREF8.
Conclusions
We go back to the questions in the introduction:
Can we train a semantic parser in a language where annotation is available?. In this paper we show that this is indeed possible and we propose a zero-shot cross-lingual semantic parsing method based on language-independent features, where a parser trained in English – where labelled data is available, is used to parse sentences in three languages, Italian, German and Dutch.
What would that require? We show that universal dependency features can dramatically improve the performance of a cross-lingual semantic parser but modelling the tree structure directly does not outperform sequential BiLSTM architectures, not even when the two are combined together.
We are planning to extend this initial survey to other DRS parsers that does not exclude presupposition and sense as well as to other semantic formalisms (e.g. AMR, MRS) where data sets annotated in languages other than English are available. Finally, we want to understand whether adding a bidirectionality to the treeLSTM will help improving the performance on modelling the dependency structure directly.
Acknowledgements
This work was done while Federico Fancellu was a post-doctoral researcher at the University of Edinburgh. The views expressed are his own and do not necessarily represent the views of Samsung Research.

Question: How many lexical features are considered?
Answer: No

3: In addition to word embedding, there is a POS tag embedding and a dependcy relation embedding. 

Unanswerable

Question: What is the performance for the three languages tested?
Answer: Best authors achieved (different models) in terms of F1 score is:
German - 0.6446
Italian - 0.6999
Dutch - 0.6057

Max-F Scores for German .6446, Italian .6999. Dutch .6057 compared to 0.8748 for English

Question: How many Universal Dependency features are considered?
Answer: Unanswerable

Unanswerable

Question: Do they evaluate any non-zero-shot parsers on the three languages?
Answer: Unanswerable

No

Question: How big is the Parallel Meaning Bank?
Answer: 4405 English sentences, 1173 German sentences, 633 Italian sentences and 583 Dutch sentences

6794 sentences

Question: What is the source of the crosslingual word embeddings?
Answer: MUSE BIBREF17

MUSE BIBREF17