Venue: NAACL 2021

# Improving Zero-Shot Cross-lingual Transfer for Multilingual Question Answering over Knowledge Graph

Yucheng Zhou\({}^{1}\), Xiubo Geng\({}^{2}\), Tao Shen\({}^{3}\), Wenqiang Zhang\({}^{1\dagger}\), Daxin Jiang\({}^{2\dagger}\)

\({}^{1}\)Fudan University, Shanghai, China

\({}^{2}\)Microsoft, Beijing, China

\({}^{3}\)Australian AI Institute, School of CS, FEIT, University of Technology Sydney

{yczhou18,wqzhang}@fudan.edu.cn, tao.shen@student.uts.edu.au {xieng,djiang}@microsoft.com

Work is done during internship at Microsoft.Corresponding authors.

###### Abstract

Multilingual question answering over knowledge graph (KGQA) aims to derive answers from a knowledge graph (KG) for questions in multiple languages. To be widely applicable, we focus on its zero-shot transfer setting. That is, we can only access training data in a high-resource language, while need to answer multilingual questions without any labeled data in target languages. A straightforward approach is resorting to pre-trained multilingual models (e.g., mBERT) for cross-lingual transfer, but there is a still significant gap of KGQA performance between source and target languages. In this paper, we exploit unsupervised bilingual lexicon induction (BLI) to map training questions in source language into those in target language as augmented training data, which circumvents language inconsistency between training and inference. Furthermore, we propose an adversarial learning strategy to alleviate syntax-disorder of the augmented data, making the model incline to both language-and syntax-independence. Consequently, our model narrows the gap in zero-shot cross-lingual transfer. Experiments on two multilingual KGQA datasets with 11 zero-resource languages verify its effectiveness.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding Authors.

## 1 Introduction

With the advance of large-scale human-curated knowledge graphs (KG), e.g., DBpedia Auer et al. (2007) and Freebase Bollacker et al. (2008), question answering over knowledge graph (KGQA) has become a crucial natural language processing (NLP) task to answer factoid questions. It has been integrated into real-world applications like search engines and personal assistants, so it attracts more attention from both academia and industry Liang et al. (2017); Hu et al. (2018); Shen et al. (2019).

Recently, a rising demand of KGQA systems is to answer the multilingual questions, motivating us to focus on multilingual KGQA. However, building a large-scale KG, as well as annotating QA data, is costly for each new language, not to mention many minority languages with a few native annotators. Therefore, we adopt a zero-shot cross-lingual transfer setting - a KGQA model is developed to perform inference on multilingual questions with the only access to training data and associated KG in a high-resource language (e.g., English).

Providing the success of pre-trained monolingual encoders Peters et al. (2018); Liu et al. (2019), some works (e.g., mBERT Devlin et al. (2019) and XLM-R Conneau et al. (2020)) pre-train a Transformer encoder Vaswani et al. (2017) on large-scale non-parallel multilingual corpora in a self-supervised manner. Then given an NLP task, a general paradigm for zero-shot cross-lingual transfer is to fine-tune a pre-trained multilingual encoder on the data in a data-rich (_source_) language. And the fine-tuned model is generalizable enough to perform inference in other low-resource (_target_) languages with surprising quality of prediction. This paradigm can be adapted to KGQA to build symbolic logical forms (e.g., query graph Yih et al. (2015)) for KG query. However, it is witnessed that there is a considerable KGQA performance gap between source and target languages, which is consistent with the empirical results on a wide range of other tasks by prior works Conneau et al. (2020).

To bridge the gap, translation approaches are proven effective on multilingual benchmarks Hu et al. (2020); Liang et al. (2020). As a way of data augmentation, they perform source-to-target translation to obtain multilingual training data. Further with advanced techniques Cui et al. (2019); Fang et al. (2020), they achieve state-of-the-art effectiveness. But these approaches rely heavily on a well-performing translator. The translator is not always available especially for a minority language since its training requires a large volume of parallel bilingual corpus. Therefore, to be applicable to

more languages, we assume that neither translators nor parallel corpora are available in this work.

In this paper, to adapt the translation approaches in our zero-resource scenario, we naturally propose to replace the full-supervised machine translator with unsupervised bilingual lexicon induction (BLI) for word-level translation. Specifically, as in prior works Lample et al. (2018); Artetxe et al. (2018), a BLI model is first trained on non-parallel bilingual corpora. Then, via bilingual word alignments in BLI, we map the training questions in source language into those in target languages to obtain augmented multilingual training data. Consequently, even simply learning a KGQA model on the augmented data can circumvent language inconsistency between training and inference and thus bridge the performance gap in zero-shot cross-lingual transfer. To explain why BLI is competent, it is observed that KGQA mainly involves phrase-level semantics Berant et al. (2013). Compared to other tasks depending on sentence-level contextualization, KGQA is insensitive to long-term dependency but benefits from the language consistency.

Moreover, we propose an adversarial strategy to mitigate the syntax-disorder caused by BLI. Specifically, we present a discriminator on top of the encoder, which is trained to distinguish whether the input is a grammatical question in source language or a BLI-translated one in target language. Meanwhile, jointly with KGQA goal, the encoder is fine-tuned to fool the discriminator so that the questions' representations are both language- and syntax-agnostic. So the trained KGQA model is robust to syntax-disorder and becomes insensitive to the question language, leading to superior performance on multilingual KGQA.

Experiments conducted on two multilingual KGQA datasets with 11 zero-resource languages verify the effectiveness of our approach.

## 2 KGQA Task Definition

We give a background of monolingual KGQA, followed by multilingual KGQA and its data format.

Monolingual Kgqa.A knowledge graph \({\cal G}\) is comprised of a set of directed triples \((h,p,t)\), where \(h\in{\cal E}\) denotes a head entity, \(t\in{\cal E}\bigcup{\cal L}\) denotes a tail entity or literal value, and \(p\in{\cal P}\) denote a predicate between \(h\) and \(t\). KGQA aims at generating answers for a natural language question \(q\) based on \({\cal G}\). Usually a model \({\cal M}\) first parses the question \(q\) into an intermediate logical form, which is then transformed into a SPARQL query, and the answer is derived by executing the SPARQL query on \({\cal G}\). An example is shown in Figure 1: the question in the bottom, intermediate logical form in the upper right and the corresponding SPARQL query in the top. Following Maheshwari et al. (2019), we take a restricted subset of \(\lambda\)-calculus - query graph, as the intermediate logical form. Typically, a query graph consists of four types of nodes: grounded entity(s) (in rounded rectangle), existential variable(s) "_?y_" (in circle), a lambda variable "_?x_" (in shaded circle), and an aggregation function (in diamond).

Considering entity-linking is a standalone system and there are many tools, we assume grounded entities in a question are given. This avoids uncertainty caused by entity-linking, and facilitates us to focus on the query graph construction process.

Multilingual Kgqa.We focus on a zero-shot cross-lingual transfer setting of KGQA. That is, we only have a labeled dataset \({\cal D}^{src}\) = \(\{(q_{l}^{src},s_{l}^{src})_{l=1}^{N}\}\), as well as the associated knowledge graph \({\cal G}\), in a high-resource language \(src\), where \(q_{l}^{src}\) and \(s_{l}^{src}\) denote a natural language question and a formal query, respectively. We will omit subscript \(l\) of example index in \({\cal D}^{src}\). Multilingual KGQA is to learn a model \({\cal M}\) which can answer questions \(q^{tgt}\) in multiple target languages \(tgt\). A recent baseline is to fine-tune pre-trained multilingual models (e.g. mBERT) in \(src\) and directly perform inference in \(tgt\).

## 3 Methodology

This section starts with a base framework for monolingual KGQA, followed by our proposed multilingual solutions. Lastly, details about training and inference are elaborated.

### Base Monolingual Framework

Following Maheshwari et al. (2019), we present a base pipeline framework as in Figure 1 to construct query graphs. It consists of three modules: 1) inferential chain ranking, 2) type constraint ranking, and 3) aggregator classification.

Inferential Chain Ranking.An inferential chain (IC) refers to a sequence of directed predicate from a grounded entity to lambda variable ?_x_. Given an entity \(e\) grounded from the question \(q\), we first search its chain candidates \({\cal C}^{e}\) = \((c_{1}^{e},\ldots,c_{n}^{e})\) by exploring legitimate predicate sequences starting from \(e\) in \({\cal G}\). Following previous works Yih 

et al., 2015; Maheshwari et al., 2019), we fetch the chains whose length \(\leq 2\). For example, as in the middle left of Figure 1, chain candidates are generated from the entity "_<dbr:Ven.-Ram.>_" within 2-hop on \({\cal G}\). Then, a model is presented to measure the semantic relatedness between the question \(q\) and each candidate of inferential chain \(c^{e}_{i}\), i.e.,

\[a^{e}_{i}\!=\!{\rm SemMatch}(q,c^{e}_{i};\theta^{(IC)}),\forall i\!=\!1,\ldots,n, \tag{1}\]

where \(a^{e}_{i}\) is a score for their relatedness, and \(\theta^{(IC)}\)-parameterized \({\rm SemMatch}(\cdot)\) can be any model for pairwise relatedness, such as Co-Attention network Chen et al. (2019) and BERT-based Matching Devlin et al. (2019). Finally, the resulting of this module is the top-1 ranked inferential chain, i.e.,

\[\bar{c}^{e}=\arg\max_{c^{e}_{i}}(a^{e}_{i},~{}\forall i=1,\dots,n). \tag{2}\]

Note, if there are multiple grounded entities in \(q\), we will predict an inferential chain for each entity.

Type Constraint Ranking.Type constraints (TC) refer to the entity types specified in the question for each variable on an inferential chain. They can be used to disambiguate the entities and thus boost KGQA performance. For example, answer entity(s) to the example question in Figure 1 are constrained by type _Scientist_. Hence, type constraint ranking is proposed to capture such information, which is also achieved by a semantic matching model. Specifically, given the resulting inferential chain \(\bar{c}^{e}\), we first enumerate type candidates \({\mathcal{T}}^{e}_{y}=\{t^{e}_{y1},\dots\}\) for the existential variable and \({\mathcal{T}}^{e}_{x}=\{t^{e}_{x1},\dots\}\) for the lambda variable. Then, because there is scarcely overlap of gold type constraints between the two variables, a single semantic matching model is adequate for both. Thus, we define the model to derive relatedness scores as

\[b^{e}_{*j}={\rm SemMatch}(q,t^{e}_{*j};\theta^{(TC)}),\] (3) where, \[\forall*\in\{y,x\},\text{and}~{}\forall j=1,\dots\]

Finally, we get the type constraints for existential and lambda variable with a threshold \(\gamma^{(thresh)}\), i.e.,

\[\overline{{\mathcal{T}}^{e}_{*}}=\{t^{e}_{*j}|b^{e}_{*j}>\gamma^{(thresh)},~{} \forall j=1,\dots\}. \tag{4}\]

Aggregator ClassificationGiven several answer formats in the dataset, aggregator classification (AC) is presented to distinguish the format among _Bool_, _Count_ and _Entity(s)_. The principle of each is detailed in the middle right of Figure 1. Formally, a simple text classifier can satisfy, i.e.,

\[p^{(AC)}={\rm Classifier}(q;\theta^{(AC)})\in\mathbb{R}^{3}, \tag{5}\]

where the \({\rm Classifier}(\cdot)\) is composed of a contextualized encoder, a pooler and an MLP with \({\rm softmax}\).

Once the above is completed, their results can compose a query graph, which is transformed into SPARQL and then executed on \({\cal G}\) for the answer.

### Proposed Multilingual KgQA Approach

Built upon the base framework detailed before, we extend it with a multilingual inference capability,

Figure 1: Base framework for monolingual KG, consisting of three modules to construct a query graph.

 

i.e., multilingual KGQA. We are in line with a recent popular zero-shot transfer paradigm (Conneau et al., 2020; Fang et al., 2020) that: a pre-trained multilingual encoder is only fine-tuned in \(src\), and a translation-based data augmentation technique is integrated to narrow the performance gap between \(src\) and \(tgt\). To emphasize the gap in KGQA, 65% F1 score in English (\(src\)) vs. 54% in Italian (\(tgt\)) is observed by mBERT zero-shot transfer in our pipeline without any multilingual augmenting.

Distinct from prior works in this paradigm requiring well-trained translators, we propose a fully unsupervised way for wide applicability with neither \(tgt\) KGQA data nor \(src\)-\(tgt\) parallel corpora. It is natural to resort to bilingual lexicon induction (BLI) with unsupervised training and acceptable word-level translating quality. In the following, we first present a BLI-based augmentation for multilingual training data, followed by our adaptation of the monolingual base framework (SS 3.1) to the augmented data. Finally, we propose an adversarial learning strategy coupled with BLI-based augmentation for robust cross-lingual transfer. An illustration of our proposed semantic matching model with symbolic candidates is in Figure 2.

#### 3.2.1 BLI-based Multilingual Augmentation

We leverage the BLI model by Lample et al. (2018b). First, it pre-trains monolingual word embeddings \({\mathbf{U}^{src}}\in\mathbb{R}^{d\times|\mathcal{V}^{src}|}\) and \({\mathbf{U}^{tgt}}\in\mathbb{R}^{d\times|\mathcal{V}^{tgt}|}\) in \(src\) and \(tgt\) respectively. Then, it learns a linear transformation to unsupervisedly align the word embeddings in two languages to one space, i.e.,

\[\bar{\mathbf{W}}\!\!=\!\underset{\mathbf{W}\in M_{d}(\mathbb{R})}{\arg\min}\!\sum \limits_{\mathbf{W}\in M_{d}(\mathbb{R})}\!\sum\limits_{\mathbf{W}\in M_{c}(\mathbb{ R})}\mathrm{Distance}(\mathbf{W}{\mathbf{U}^{src}_{:,k}},{\mathbf{U}^{tgt}_{:,l}}). \tag{6}\]

The unsupervised alignment between \(k\)-th \(src\) word and \(l\)-th \(tgt\) word is captured by adversarial learning, and \(\mathrm{Distance}(\cdot)\) is implemented by cross-domain similarity local scaling (CSLS). Please refer to (Lample et al., 2018b) for its details.

Based on the BLI model, we can build a word-by-word translator, \(\mathrm{BLI}^{(trans)}_{src+tgt}\), from \(src\) to arbitrary \(tgt\), as long as its monolingual corpus is available. Note, when performing word-level translation, we also employ CSLS to mitigate the hubness problem and find the most likely alignment. Then, we translate each question \(q^{src}\) in \(\mathcal{D}^{src}\) to other languages:

\[q^{tgt}=\mathrm{BLI}^{(trans)}_{src+tgt}(q^{src}), \tag{7}\]

where \(src\) denotes English (en) in our experiments while \(tgt\) can be one of 11 other languages, such Farsi (fa), Italian (it), etc. Consequently, \(q^{tgt}\) is the augmented multilingual data for model training.

_Remark:_ Although BLI provides multilingual data, open questions still remain. 1) _Why is BLI competent here:_ It is observed KGQA mainly involves word-/phrase-level semantics of symbolic candidates, rather than sentence-level one in most other NLP tasks. As the Module 1 and 2 in Figure 1, the matching only involves morphological similarity (e.g., _scientist_ vs. _<dbo:Scientists>_), synonym (e.g., _won an award_ vs. _<dbp:prizes>_), etc. Thus, KGQA is less sensitive to long-term context than other tasks. This has been leveraged by Berant et al. (2013) to propose a phrase matching model for monolingual KGQA. 2) _Will BLI lead to error propagation:_ Since BLI model achieves a high Precision@10 but a relatively low Precision@1, wrong translation and the corresponding ground truth are semantically similar. Intuitively, their word embeddings are spatially close to each other, so wrong word-level translation is equivalent to applying tiny noise to word embeddings, which hardly leads to error propagation when robust pretrained Transformer-based encoder is used.

#### 3.2.2 Multilingual Models

Symbolic Candidate ProcessingFor an inferential chain, we enrich each predicate on the chain by 1) transforming each camel-represented phrase into sequence-formatted words 2) prefixing +/- for directional information, and 3) concatenating top-frequent types in local closed-world assumptions (Krompass et al., 2015). For a type constraint, we simply transform each camel-represented phrase into sequence-formatted words. In the following,

Figure 2: Syntax-agnostic semantic matching between a BLI-augmented multilingual \(q\) and its symbolic candidates.

 

we denote the text of a processed symbolic candidate as \(z\) no matter it is a chain or type.

Multilingual Semantic Matching Model.As detailed in SS3.1, both inferential chain ranking and type constraint ranking modules are built upon a semantic matching model between the question \(q\) and a symbolic candidate \(z\). Note, \(z\) is always in \(src\) while \(q\) can be in either \(src\) or BLI-translated \(tgt\). Following the common practice, we first concatenate \(q\) and \(z\) with special tokens Devlin et al. (2019), which is passed into a pre-trained multilingual Transformer encoder, i.e.,

\[{\mathbf{v}}=\mathrm{Pool}(\mathrm{Transformer}(text)), \tag{8}\] \[\text{where, }text=(\texttt{[CLS]},q,\texttt{[SEP]},z,\texttt{[ SEP]}).\]

\(\mathrm{Pool}(\cdot)\) denotes using the contextualized embedding of [CLS] to represent the entire input. In this paper, the encoder is alternative between mBERT Devlin et al. (2019) and XLM-R Conneau et al. (2020). Lastly, a 1-way multi-layer perceptron (MLP) built upon \({\mathbf{v}}\) is presented to calculate the matching score in Eq.(1) or Eq.(3).

Multilingual Classification Model.As detailed in SS3.1, a text classification model is required to identify aggregator. To fit into our zero-resource multilingual scenario, the model, consisting of a pre-trained multilingual encoder and an MLP-based predicting layer, can be directly fine-tuned on the augmented questions, i.e., \(q^{src}\) and \(q^{tgt}\).

#### 3.2.3 Syntax-agnostic Adversarial Strategy

Although training the KGQA model on BLI-augmented multilingual data circumvents language inconsistency, it inevitably introduces syntax disorder and grammatical problem, which could hurt the performance. We thus present an adversarial strategy in pair with BLI-augmented data to push the Transformer encoder deriving language- and syntax-independent representations. Formally, a discriminator is built upon the single vector representation \({\mathbf{v}}\) produced by the Transformer encoder:

\[p^{(src)}=\mathrm{Simoid}(\mathrm{MLP}({\mathbf{v}};\theta^{(dis)})), \tag{9}\]

where \(p^{(src)}\) is the probability of the question in source. The discriminator is trained to minimize

\[\mathcal{L}_{\theta^{(dis)}}^{(adv)}\!\!=\!\mathbb{I}_{(src)}\!\log p^{( src)}\!\!-\!\mathbb{I}_{(tgt)}\!\log(1\!\!-\!p^{(src)}). \tag{10}\]

On the contrary, the Transformer encoder is learned to fool by minimizing an adversarial loss, i.e.,

\[\mathcal{L}_{\theta^{(enc)}}^{(adv)}=-\mathbb{I}_{(tgt)}\log p^{(src)}. \tag{11}\]

\(\mathbb{I}_{(tgt)}\) denotes if the question in BLI-translated \(tgt\), and \(\theta^{(enc)}\) is encoder's parameters in each module.

### Training

Before constructing the objectives, we conduct uniform negative sampling for the two ranking models with the maximum negative number limited to 100.

First, gold labels of a \(q\) for the three modules stem from the formal query \(s^{src}\). A margin-based hinge loss is defined for inferential chain ranking:

\[\hat{\mathcal{L}}^{(IC)}=\frac{1}{|\mathcal{D}|}\sum_{\mathcal{D}}\frac{1}{| \mathcal{N}|}\sum\nolimits_{i=1}^{|\mathcal{N}|}(\lambda\!-\!\tilde{a}^{c}\!+ \!\hat{a}^{c}_{i}), \tag{12}\]

where, \(\mathcal{D}\) is the augmented dataset, \(\mathcal{N}\) is a set of negative chains, \(\tilde{a}^{e}\) is derived from the gold chain and \(\hat{a}^{e}_{i}\) is derived from a negative chain. Similarly, the loss defined for type constraint ranking is

\[\hat{\mathcal{L}}^{(TC)}=\frac{1}{|\mathcal{D}|}\sum_{\mathcal{D}}\frac{1}{2| \mathcal{N}|}\sum_{*\in\{y,x\}}\sum_{j=1}^{|\mathcal{N}|}(\lambda-\tilde{b}^{ e}_{*}+\hat{b}^{e}_{*j}).\]

Lastly, the loss of aggregator classification is

\[\hat{\mathcal{L}}^{(AC)}=-\frac{1}{|\mathcal{D}|}\sum\nolimits_{\mathcal{D}} \log{\mathbf{p}}_{[i=\bar{g}]}^{(AC)}, \tag{13}\]

where \({\mathbf{p}}_{[i=\bar{g}]}^{(AC)}\) denotes probability corresponding to gold aggregator class.

During training, the adversarial loss is added to the loss function of each module to compose the final training objective, i.e.,

\[\mathcal{L}^{(*)}=\hat{\mathcal{L}}^{(*)}\!+\!\alpha\mathcal{L}_{\theta^{( enc)}}^{(adv)},*\!\in\!\{IC,TC,AC\}. \tag{14}\]

### Inference Algorithm

As in Algorithm 1, we provide a detailed procedure for model inference in target language.

We also provide an explanation of query graph in Figure 1. As the example query graph shown in the right of the figure: a topic entity is first grounded as \(e=\)"<_dbr:Ven-Ram>_" in rounded rectangle, an existential variable in circle denotes intermediate entity set ?\(y=\{h|(h,\textit{leaderName},e)\}\), a lambda variable in shaded circle denotes the answer entity set ?\(x=\{h|(h,\textit{prizes},e)\wedge\forall e\in?y\}\), and an aggregator Count is finally applied to ?\(x\) that is constrained by entity type "<_doo:Scientist>_". Note that, the existential variable can not exist if only 1-hop relation is expressed in a question, and if multiple topic entities are grounded, multiple "?\(x\)" will be merged by intersection 



## 4 Experiments

### Datasets and Evaluation Metrics

We evaluate the proposed approach on two datasets, LC-QuAD Trivedi et al. (2017) and QALDmultilingual Usbeck et al. (2018), both of which contain questions with corresponding SPARQL queries over DBpedia1. DBpedia is a large-scale knowledge graph extracted from Wikipedia pages with 6 million/60 thousands/13 billion entities/predicates/triples in the English edition.

Footnote 1: We use the 2016-10 version, which can be downloaded at [https://wiki.dbpedia.org/downloads-2016-10](https://wiki.dbpedia.org/downloads-2016-10).

LC-QuAD.LC-QuAD is a large-scale complex question answering dataset, which contains 5000 English question-SPARQL pairs2. We follow the official split with 1000 questions in the test set, and further split the original training set into training/valid with 3500/500 questions. To evaluate the effectiveness of multilingual KGQA, questions in the test set are translated into 10 languages (fa, de, ro, it, ru, fr, nl, es, hi, pt)3 using Google Translator4.

Footnote 2: [https://github.com/AskNowQA/LC-QuAD](https://github.com/AskNowQA/LC-QuAD).

Footnote 3: [https://github.com/yczhou001/Multilingual-KBQA-Dataset/tree/main/L-C-QuAD](https://github.com/yczhou001/Multilingual-KBQA-Dataset/tree/main/L-C-QuAD).

Footnote 4: [https://translate.google.com/](https://translate.google.com/).

**QALD-multilingual.**   QALD is a series of evaluation campaigns on question answering over linked data5. We collect all multilingual questions along with their SPARQL queries from QALD4 to QALD9 and filter out some out-of-scope ones6. There are overall 429 distinct question-SPARQL pairs and most are expressed in 12 languages (en, fa, de, ro, it, ru, fr, nl, es, hi_IN, pt, pt_BR). Considering the small size of this dataset, we take all QALD-multilingual questions as test set, and use the training data of LC-QuAD for model training.

Footnote 5: [https://github.com/ag-sc/QALD](https://github.com/ag-sc/QALD).

Evaluation Metrics.We adopt two widely-used metrics as following Maheshwari et al. (2019), i.e., inferential chain accuracy (ICA) and macro F1 score. The former is used to measure the accuracy (i.e., Precision@1) of inferential chain model, and defined as the percent of correctly-predicted inferential chains. The macro F1 score is used to measure the performance of final answers. Please refer to Maheshwari et al. (2019) for the details.

### Experimental Setting

We evaluate our approach with 2 multilingual encoding models, i.e. mBERTbase and XLM-Rbase. The embedding and hidden size in both models are set to 768. We use Adam optimizer Kingma and Ba (2015) to optimize the KGQA loss with the learning rate of \(5\times 10^{-5}\) and a linear warm-up Vaswani et al. (2017). The maximum training epoch, warm-up epoch, and batch size are set to 35, 3, and 32. The discriminator is trained along with each module's objective, with \(\alpha\) set to \(5\times 10^{-4}\) for learning to fool. The discriminator is optimized via the Adam optimizer with a learning rate of \(5\times 10^{-5}\). \(\gamma^{(thresh)}\) for the type constraint model is set to 0.7. We follow Maheshwari et al. (2019) and use the same values for other parameters in model training.

### Main Results

We compare our approach with a natural, widely-used baseline, which fine-tunes a pre-trained multilingual model (e.g., mBERT, XLM-R) on source language, and then directly apply it to target languages. The comparison on QALD-multilingual and LC-QuAD with mBERT are reported in Table 1 and 2 respectively. It is showed that our approach outperforms the baseline significantly on both datasets for all languages. ICA is improved by 1%-4%, and 2.9% on average on the QALD dataset. The improvement on LC-QuAD is even larger, i.e., averaged ICA and F1 score of all languages are increased by around 7% and 4% respectively. Notably, with the BLI-augmented data and syntax

agnostic adversarial learning, the performance of source-language (i.e., English) questions are also increased by a large margin, i.e., F1 score increases from 65% to 66.7% on QALD, and from 80% to 85% on LC-QuAD. We also evaluate the propose approach using XLM-R as the multilingual encoder. The comparison on QALD-multilingual is shown in Table 3. We can observe similar improvements as in mBERT, where both averaged ICA and F1 score are increased by around 1%, verifying the effectiveness of our proposed approach.

### Ablation Study

Our approach consists of two important components, BLI-based data augmentation and a syntax-agnostic learning strategy. We conduct an ablation study to investigate the effect of each component. Table 4 reports the averaged results of all target-languages on QALD-multilingual and LC-QuAD-multilingual. From the table we can see that, with BLI-based data augmentation, our approach increases the ICA score on QALD by 1.7%, and the syntax-agnostic adversarial learning further improves it by 1.2%. Similar improvements are observed on LC-QuAD, which verifies the effectiveness of both components in our approach.

### Analysis

Impact of BLI Accuracy.We assess the impact of BLI accuracy on five Romance languages (i.e. it, fr, es, pt, and ro) by injecting noise into BLI results. Specifically, when mapping source-language words into a target language via BLI, we randomly replace translated words with wrong ones with a probability of \(p\) (10%, 20%, 30%, 40%, and 50%). The averaged performance of our approach on the five languages is reported in Figure 3. It is observed, with more noise added, the performance of our approach drops, which is in accordance with intuition. But even when 50% of the translated words are noisy, our method still outperforms the baseline model. For example, it is superior than the baseline by 1% in terms of ICA with 50% noise, showing the robustness of our approach.

Deep Dive into Adversarial Learning.We take the inferential chain ranking model as an example, and take a deep dive into the impact of syntax-agnostic adversarial learning. The adversarial learning involves a discriminator to distinguish whether a question is grammatical or syntax-disorder, and an inferential chain ranking model to identify the gold chain. Their loss values, i.e., \({\cal L}^{(dis)}_{\theta^{(dis)}}\) and

Figure 3: Impact of BLI Accuracy in our approach. The x-axis represents the percentage of noise we inject into BLI results, while y-axis represents the performance in terms of ICA in Figure (left) and F1 score in Figure (right).

 

\(\hat{\mathcal{L}}^{(IC)}\), are plot in Figure 4. We can see that the classification loss of the discriminator quickly drops and then slowly goes up, indicating that the discriminator gets good performance and then it is fooled later by the language-/syntax-agnostic embeddings generated by mBERT. Meanwhile, the inferential ranking loss drops quickly and stays very small in following epochs, showing that when mBERT is generating syntax-agnostic embeddings, it also supports the inferential chain ranking very well.

### Case Study

We take several examples of inferential chain ranking to show how our approach works. We use t-SNE Maaten and Hinton (2008) to map the embedding of a question-chain pair into a two-dimensional data point. A question in a specific language is paired with its golden inferential chain and top-1 ranked negative candidate. Figure 5 compares the baseline with our approach for two questions. Positive and negative examples of the same question in different languages are plot in the same figure. We can see that the baseline model can not distinguish positive inferential chains from negative ones well, while our approach can learn a language-agnostic representation that focuses more on ranking inferential chain candidates.

## 5 Related Work

There are mainly two categories of approaches to handle monolingual question answering over knowledge graph (KGQA) task. (1) _Information retrieval-based_ approaches align a question with its answer candidates in the same semantic space,

Figure 4: Adversarial losses on validation set w.r.t different epochs in training phase.

Figure 5: Case study via t-SNE visualization. Different points in a graph represents different languages for the same question. (Upper left) and (upper right) show embeddings of baseline and our approach for the question “what is the population of Cairo?”. (Lower left) and (lower right) show embeddings of baseline and our approach for the question “which species does an elephant belong?”.

 

where the candidates usually stem from KG neighbors of the topic entity detected in the questions (Bordes et al., 2014b, a; Dong et al., 2015; Jain, 2016; Xu et al., 2016; Hao et al., 2017; Chen et al., 2019). (2) _Semantic parsing-based_ approaches first translate a question into the corresponding logical form, e.g., program (Guo et al., 2018; Shen et al., 2019) or query graph (Yih et al., 2015; Jia and Liang, 2016; Xiao et al., 2016; Dong and Lapata, 2016; Liang et al., 2017; Dong and Lapata, 2018; Maheshwari et al., 2019), and then execute the logical form over KG to derive the final answer. Note a logical form is usually composed of a series of grammars or operators pre-defined by experts. This paper is in line with the second category to generate query graph for KG execution. To the best of our knowledge, there are only few works targeting multilingual KGQA (Hakimov et al., 2017; Veyseh, 2016), which rely on extensive multilingual training data with hand-crafted features while are inapplicable to the zero-shot transfer scenario. So we adopt the pipeline by Maheshwari et al. (2019) for monolingual scenario as our base model but update the encoders with the Transformer (Vaswani et al., 2017) to strengthen their expressive power and facilitate recent pre-trained multilingual initializations.

Given task-specific data in a source language, cross-lingual models are trained to perform inference in target languages in a low- or zero-resource scenario. Typically, cross-lingual models are proposed in two paradigms. 1) _Universal encoding_-based paradigm represents multilingual natural language text into language-agnostic embeddings the same semantic space. Early works focus on aligning multilingual word embedding Mikolov et al. (2013); Faruqui and Dyer (2014); Xu et al. (2018), while recent efforts are mainly made on large-scale pre-trained multilingual encoder, such as mBERT Devlin et al. (2019), XLM Conneau and Lample (2019), Unicoder Huang et al. (2019a), XLM-R Conneau et al. (2020), InfoXLM Chi et al. (2020), and ALM Yang et al. (2020). They can perform zero-shot cross-lingual transfer by training in the source language while directly inference in target language. 2) _translation_-based paradigm employs well-trained machine translators to map the training or test examples in source language to those in target translation. Recent common practice tends to leverage the second paradigm to generate multilingual data to narrows the zero-shot cross-lingual performance gap in the first paradigm, which leads to state-of-the-art results on several cross-lingual benchmarks. In contrast, we consider a zero-resource scenario where translators are unavailable and we thus resort to unsupervised BLI in light of KGQA's characteristics.

As a branch of universal encoding at word level, bilingual lexicon induction (BLI) (a.k.a cross-lingual word embedding - CLWE) is learned to align bilingual word embeddings in the same space, where the embeddings are pre-trained on monolingual corpora and the alignment is trained in either a (semi-)supervised or unsupervised manner Smith et al. (2017); Lample et al. (2018); Artetxe et al. (2018, 2019); Huang et al. (2019b); Patra et al. (2019); Karan et al. (2020); Zhao et al. (2020); Ren et al. (2020). To alleviate "hubness" problem Dinu and Baroni (2015) in BLI, alternatives of the distance measurement are proposed to substitute nearest neighbor (NN) during the alignment, such as inverted-softmax Smith et al. (2017) and CSLS Lample et al. (2018). In addition to building bilingual dictionary via word-level translation, a well-trained BLI model can serve as a weak baseline of sentence-level translation Lample et al. (2018), a seed model for unsupervised translation Lample et al. (2018) or a bilingual variant of copy mechanism in summarization Zhu et al. (2020).

Moreover, adversarial training is usually integrated into cross-lingual models for language-agnostic representation learning, such as unsupervised BLI Lample et al. (2018); Zhang et al. (2017), unsupervised translation Lample et al. (2018), cross-Lingual sequence labeling Kim et al. (2017); Huang et al. (2019c) and cross-Lingual classification Dong et al. (2020). In contrast, our adversarial strategy not only considers language-agnostic representations but also aims at making the model insensitive to syntax-disorder and thus competent in zero-resource scenario.

## 6 Conclusion

We propose a novel approach for zero-shot cross-lingual transfer in multilingual KGQA, which augments training data by bilingual lexicon induction, and leverages a syntax-agnostic adversarial learning strategy to alleviate the syntax-disorder problem caused by BLI. Experimental results on two multilingual KGQA datasets in 11 zero-resource languages verify its effectiveness 



## References

* [1]M. Artetxe, G. Labaka, and E. Agirre (2018) A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, Volumes 789-798, pp. 789-798. External Links: Link Cited by: SS3.4.
* M. Artetxe, G. Labaka, and E. Agirre (2019) Bilingual lexicon induction through unsupervised machine translation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, Volumes 5002-5007, pp. 5002-5007. External Links: Link Cited by: SS3.4.
* S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives (2007) Dbpedia: a nucleus for a web of open data. In The semantic web, pp. 722-735. Cited by: SS1.
* J. Berant, A. Chou, R. Frostig, and P. Liang (2013) Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1533-1544. External Links: Link Cited by: SS1.
* K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor (2008) Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp. 1247-1250. Cited by: SS1.
* A. Bordes, S. Chopra, and J. Weston (2014a) Question answering with subgraph embeddings. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 615-620. External Links: Link Cited by: SS1.
* European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part I, Lecture Notes in Computer Science, Vol. , pp. 165-180. External Links: Document, ISSN 1558-1221, Link Cited by: SS1.
* Y. Chen, L. Wu, and M. J. Zaki (2019) Bidirectional attentive memory networks for question answering over knowledge bases. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 2913-2923. External Links: Link Cited by: SS1.
* Z. Chi, L. Dong, F. Wei, N. Yang, S. Singhal, W. Wang, X. Song, X. Mao, H. Huang, and M. Zhou (2020) InfoMix: an information-theoretic framework for cross-lingual language model pre-training. CoRRabs/2007.07834. External Links: Link Cited by: SS1.
* A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzman, E. Grave, M. Ott, L. Lettlemoyer, and V. Stoyanov (2020) Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 8440-8451. External Links: Link Cited by: SS1.
* A. Conneau and G. Lample (2019) Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 7057-7067. External Links: Link Cited by: SS1.
* Y. Cui, W. Che, T. Liu, B. Qin, S. Wang, and G. Hu (2019) Cross-lingual machine reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 1586-1595. External Links: Link Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Vol. , pp. 4171-4186. External Links: Link Cited by: SS1.
* G. Dinu and M. Baroni (2015) Improving zero-shot learning by mitigating the hubness problem. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, Cited by: SS1.
* L. Dong and M. Lapata (2016) Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers, Vol. , pp. 731-742. External Links: Link Cited by: SS1.
* L. Dong and M. Lapata (2018) Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, Vol. , pp. 731-742. External Links: Link Cited by: SS1.



* Dong et al. (2015) Li Dong, Furu Wei, Ming Zhou, and Ke Xu. 2015. Question answering over freebase with multi-column convolutional neural networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers_, pages 260-269. The Association for Computer Linguistics.
* Dong et al. (2020) Xin Dong, Yaxin Zhu, Yupeng Zhang, Zuohui Fu, Dongkuan Xu, Sen Yang, and Gerard de Melo. 2020. Leveraging adversarial training in self-learning for cross-lingual text classification. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGR 2020, Virtual Event, China, July 25-30, 2020_, pages 1541-1544. ACM.
* Fang et al. (2020) Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, and Jingjing Liu. 2020. FILTER: an enhanced fusion method for cross-lingual language understanding. _CoRR_, abs/2009.05166.
* Farqui and Dyer (2014) Manaal Farqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, April 26-30, 2014, Gothenburg, Sweden_, pages 462-471. The Association for Computer Linguistics.
* Guo et al. (2018) Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. 2018. Dialog-to-action: Conversational question answering over a large-scale knowledge base. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montreal, Canada_, pages 2946-2955.
* Hakimov et al. (2017) Sherzod Hakimov, Soufian Jebbara, and Philipp Cimiano. 2017. Amuse: multilingual semantic parsing for question answering over linked data. In _International Semantic Web Conference_, pages 329-346. Springer.
* August 4, Volume 1: Long Papers_, pages 221-231. Association for Computational Linguistics.
* Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. _CoRR_, abs/2003.11080.
* Hu et al. (2018) Sen Hu, Lei Zou, and Xinbo Zhang. 2018. A state-transition framework to answer complex questions over knowledge base. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2098-2108.
* Huang et al. (2019a) Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Davxin Jiang, and Ming Zhou. 2019a. Unicoder: A universal language encoder by pretraining with multiple cross-lingual tasks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 2485-2494. Association for Computational Linguistics.
* Huang et al. (2019b) Jiaji Huang, Qiang Qiu, and Kenneth Church. 2019b. Hubbness nearest neighbor search for bilingual lexicon induction. In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_, pages 4072-4080. Association for Computational Linguistics.
* Huang et al. (2019c) Lifu Huang, Heng Ji, and Jonathan May. 2019c. Cross-lingual multi-level adversarial transfer to enhance low-resource name tagging. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 3823-3833. Association for Computational Linguistics.
* Jain (2016) Sarthak Jain. 2016. Question answering over knowledge base using factual memory networks. In _Proceedings of the Student Research Workshop, SRW@HLT-NAACL 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016_, pages 109-115. The Association for Computational Linguistics.
* Jia and Liang (2016) Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics.
* Karan et al. (2020) Mladen Karan, Ivan Vulic, Anna Korhonen, and Goran Glavas. 2020. Classification-based self-learning for weakly supervised bilingual lexicon induction. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 6915-6922. Association for Computational Linguistics.
* Kim et al. (2017) Joo-Kyoung Kim, Young-Bum Kim, Ruhiari Sakkaya, and Eric Fosler-Lussier. 2017. Cross-lingual transfer learning for POS tagging without cross-lingual resources. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_, pages 2832-2838. Association for Computational Linguistics.

 

* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* ISWC 2015
- 14th International Semantic Web Conference, Bethlehem, PA, USA, October 11-15, 2015, Proceedings, Part I_, volume 9366 of _Lecture Notes in Computer Science_, pages 640-655. Springer.
* May 3, 2018, Conference Track Proceedings_. OpenReview.net.
* May 3, 2018, Conference Track Proceedings_. OpenReview.net.
* August 4, Volume 1: Long Papers_, pages 23-33. Association for Computational Linguistics.
* Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fengfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Davin Jiang, Guibang Cao, Xiaodong Fan, Bruce Zhang, Rahul Agrawal, Edward Cui, Siining Wei, Traon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Rangan Majumder, and Ming Zhou. 2020. XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation. _CoRR_, abs/2004.01401.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. _CoRR_, abs/1907.11692.
* van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of machine learning research_, 9(Nov):2579-2605.
* ISWC 2019
- 18th International Semantic Web Conference, Auckland, New Zealand, October 26-30, 2019, Proceedings, Part I_, volume 11778 of _Lecture Notes in Computer Science_, pages 487-504. Springer.
* Mikolov et al. (2013) Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. _CoRR_, abs/1309.4168.
* Patra et al. (2019) Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R. Gormley, and Graham Neubig. 2019. Bilingual lexicon induction with semi-supervision in non-isometric embedding spaces. In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_, pages 184-193. Association for Computational Linguistics.
* Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_, pages 2227-2237. Association for Computational Linguistics.
* Ren et al. (2020) Shuu Ren, Shujie Liu, Ming Zhou, and Shuai Ma. 2020. A graph-based coarse-to-fine method for unsupervised bilingual lexicon induction. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 3476-3485. Association for Computational Linguistics.
* Shen et al. (2019) Tao Shen, Xiubo Geng, Tao Qin, Daya Guo, Duyu Tang, Nan Duan, Guodong Long, and Dexin Jiang. 2019. Multi-task learning for conversational question answering over a large-scale knowledge base. _arXiv preprint arXiv:1910.05069_.
* Smith et al. (2017) Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
* ISWC 2017
- 16th International Semantic Web Conference, Vienna, Austria, October 21-25, 2017, Proceedings, Part II_, volume 10588 of _Lecture Notes in Computer Science_, pages 210-218. Springer.
* Usbeck et al. (2018) Ricardo Usbeck, Ria Hari Gusmita, Axel-Cyrille Ngonga Ngomo, and Muhammad Saleem. 2018. 9th challenge on question answering over linked data (QALD-9) (invited paper). In _Joint proceedings of the 4th Workshop on Semantic Deep 

Learning (SemDeep-4) and NLIWoD4: Natural Language Interfaces for the Web of Data (NLIWOO-4) and 9th Question Answering over Linked Data challenge (QALD-9) co-located with 17th International Semantic Web Conference (ISWC 2018), Monterey, California, United States of America, October 8th - 9th, 2018_, volume 2241 of _CEUR Workshop Proceedings_, pages 58-64. CEUR-WS.org.
* Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA_, pages 5998-6008.
* Ben Veyseh [2016] Amir Pouran Ben Veyseh. 2016. Cross-lingual question answering using common semantic space. In _Proceedings of TextGraphs-10: the workshop on graph-based methods for natural language processing_, pages 15-19.
* Xiao et al. [2016] Chunyang Xiao, Marc Dymetman, and Claire Gardent. 2016. Sequence-based structured prediction for semantic parsing. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics.
* Xu et al. [2016] Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016. Question answering on freebase via relation extraction and textual evidence. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics.
* November 4, 2018_, pages 2465-2474. Association for Computational Linguistics.
* Yang et al. [2020] Jian Yang, Shuming Ma, Dongdong Zhang, Shuanghi Wu, Zhoujun Li, and Ming Zhou. 2020. Alternating language modeling for cross-lingual pre-training. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_, pages 9386-9393. AAAAI Press.
* Yih et al. [2015] Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers_, pages 1321-1331. The Association for Computer Linguistics.
* August 4, Volume 1: Long Papers_, pages 1959-1970. Association for Computational Linguistics.
* Zhao et al. [2020] Xu Zhao, Zihao Wang, Yong Zhang, and Hao Wu. 2020. A relaxed matching procedure for unsupervised BLI. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 3036-3041. Association for Computational Linguistics.
* Zhu et al. [2020] Junnan Zhu, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2020. Attend, translate and summarize: An efficient method for neural cross-lingual summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 1309-1321. Association for Computational Linguistics.