SlowGuess/NAACL_2021_OCR huggingface dataset:
Yucheng Zhou $^{1*}$ , Xiubo Geng $^{2}$ , Tao Shen $^{3}$ , Wenqiang Zhang $^{1\dagger}$ , Daxin Jiang $^{2\dagger}$
$^{1}$ Fudan University, Shanghai, China
$^{3}$ Australian AI Institute, School of CS, FEIT, University of Technology Sydney
{yczhou18, wqzhang}@fudan.edu.cn, tao.shen@student.uts.edu.au
Multilingual question answering over knowledge graph (KGQA) aims to derive answers from a knowledge graph (KG) for questions in multiple languages. To be widely applicable, we focus on its zero-shot transfer setting. That is, we can only access training data in a high-resource language, while need to answer multilingual questions without any labeled data in target languages. A straightforward approach is resorting to pre-trained multilingual models (e.g., mBERT) for cross-lingual transfer, but there is a still significant gap of KGQA performance between source and target languages. In this paper, we exploit unsupervised bilingual lexicon induction (BLI) to map training questions in source language into those in target language as augmented training data, which circumvents language inconsistency between training and inference. Furthermore, we propose an adversarial learning strategy to alleviate syntax-disorder of the augmented data, making the model incline to both language-and syntax-independence. Consequently, our model narrows the gap in zero-shot cross-lingual transfer. Experiments on two multilingual KGQA datasets with 11 zero-resource languages verify its effectiveness.
With the advance of large-scale human-curated knowledge graphs (KG), e.g., DBpedia (Auer et al., 2007) and Freebase (Bollacker et al., 2008), question answering over knowledge graph (KGQA) has become a crucial natural language processing (NLP) task to answer factoid questions. It has been integrated into real-world applications like search engines and personal assistants, so it attracts more attention from both academia and industry (Liang et al., 2017; Hu et al., 2018; Shen et al., 2019).
Recently, a rising demand of KGQA systems is to answer the multilingual questions, motivating us
to focus on multilingual KGQA. However, building a large-scale KG, as well as annotating QA data, is costly for each new language, not to mention many minority languages with a few native annotators. Therefore, we adopt a zero-shot cross-lingual transfer setting – a KGQA model is developed to perform inference on multilingual questions with the only access to training data and associated KG in a high-resource language (e.g., English).
Providing the success of pre-trained monolingual encoders (Peters et al., 2018; Liu et al., 2019), some works (e.g., mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020)) pre-train a Transformer encoder (Vaswani et al., 2017) on large-scale non-parallel multilingual corpora in a self-supervised manner. Then given an NLP task, a general paradigm for zero-shot cross-lingual transfer is to fine-tune a pre-trained multilingual encoder on the data in a data-rich (source) language. And the fine-tuned model is generalizable enough to perform inference in other low-resource (target) languages with surprising quality of prediction. This paradigm can be adapted to KGQA to build symbolic logical forms (e.g., query graph (Yih et al., 2015)) for KG query. However, it is witnessed that there is a considerable KGQA performance gap between source and target languages, which is consistent with the empirical results on a wide range of other tasks by prior works (Conneau et al., 2020).
To bridge the gap, translation approaches are proven effective on multilingual benchmarks (Hu et al., 2020; Liang et al., 2020). As a way of data augmentation, they perform source-to-target translation to obtain multilingual training data. Further with advanced techniques (Cui et al., 2019; Fang et al., 2020), they achieve state-of-the-art effectiveness. But these approaches rely heavily on a well-performing translator. The translator is not always available especially for a minority language since its training requires a large volume of parallel bilingual corpus. Therefore, to be applicable to
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5822-5834 June 6-11, 2021. ©2021 Association for Computational Linguistics
more languages, we assume that neither translators nor parallel corpora are available in this work.
In this paper, to adapt the translation approaches in our zero-resource scenario, we naturally propose to replace the full-supervised machine translator with unsupervised bilingual lexicon induction (BLI) for word-level translation. Specifically, as in prior works (Lample et al., 2018b; Artetxe et al., 2018), a BLI model is first trained on non-parallel bilingual corpora. Then, via bilingual word alignments in BLI, we map the training questions in source language into those in target languages to obtain augmented multilingual training data. Consequently, even simply learning a KGQA model on the augmented data can circumvent language inconsistency between training and inference and thus bridge the performance gap in zero-shot crosslingual transfer. To explain why BLI is competent, it is observed that KGQA mainly involves phrase-level semantics (Berant et al., 2013). Compared to other tasks depending on sentence-level contextualization, KGQA is insensitive to long-term dependency but benefits from the language consistency.
Moreover, we propose an adversarial strategy to mitigate the syntax-disorder caused by BLI. Specifically, we present a discriminator on top of the encoder, which is trained to distinguish whether the input is a grammatical question in source language or a BLI-translated one in target language. Meanwhile, jointly with KGQA goal, the encoder is fine-tuned to fool the discriminator so that the questions' representations are both language- and syntax-agnostic. So the trained KGQA model is robust to syntax-disorder and becomes insensitive to the question language, leading to superior performance on multilingual KGQA.
Experiments conducted on two multilingual KGQA datasets with 11 zero-resource languages verify the effectiveness of our approach.
2 KGQA Task Definition
We give a background of monolingual KGQA, followed by multilingual KGQA and its data format.
Monolingual KGQA. A knowledge graph  $\mathcal{G}$  is comprised of a set of directed triples  $(h, p, t)$ , where  $h \in \mathcal{E}$  denotes a head entity,  $t \in \mathcal{E} \cup \mathcal{L}$  denotes a tail entity or literal value, and  $p \in \mathcal{P}$  denote a predicate between  $h$  and  $t$ . KGQA aims at generating answers for a natural language question  $q$  based on  $\mathcal{G}$ . Usually a model  $\mathcal{M}$  first parses the question  $q$  into an intermediate logical form, which
is then transformed into a SPARQL query, and the answer is derived by executing the SPARQL query on  $\mathcal{G}$ . An example is shown in Figure 1: the question in the bottom, intermediate logical form in the upper right and the corresponding SPARQL query in the top. Following Maheshwari et al. (2019), we take a restricted subset of  $\lambda$ -calculus - query graph, as the intermediate logical form. Typically, a query graph consists of four types of nodes: grounded entity(s) (in rounded rectangle), existential variable(s) “?y” (in circle), a lambda variable “?x” (in shaded circle), and an aggregation function (in diamond).
Considering entity-linking is a standalone system and there are many tools, we assume grounded entities in a question are given. This avoids uncertainty caused by entity-linking, and facilitates us to focus on the query graph construction process.
Multilingual KGQA. We focus on a zero-shot cross-lingual transfer setting of KGQA. That is, we only have a labeled dataset  $\mathcal{D}^{src} = \{(q_{l}^{src}, s_{l}^{src})_{l=1}^{N}\}$ , as well as the associated knowledge graph  $\mathcal{G}$ , in a high-resource language  $src$ , where  $q_{l}^{src}$  and  $s_{l}^{src}$  denote a natural language question and a formal query, respectively. We will omit subscript  $l$  of example index in  $\mathcal{D}^{src}$ . Multilingual KGQA is to learn a model  $\mathcal{M}$  which can answer questions  $q^{tgt}$  in multiple target languages  $tgt$ . A recent baseline is to fine-tune pre-trained multilingual models (e.g. mBERT) in  $src$  and directly perform inference in  $tgt$ .
This section starts with a base framework for monolingual KGQA, followed by our proposed multilingual solutions. Lastly, details about training and inference are elaborated.
3.1 Base Monolingual Framework
Following Maheshwari et al. (2019), we present a base pipeline framework as in Figure 1 to construct query graphs. It consists of three modules: 1) inferential chain ranking, 2) type constraint ranking, and 3) aggregator classification.
Inferential Chain Ranking. An inferential chain (IC) refers to a sequence of directed predicate from a grounded entity to lambda variable  $?x$ . Given an entity  $e$  grounded from the question  $q$ , we first search its chain candidates  $\mathcal{C}^e = (c_1^e, \ldots, c_n^e)$  by exploring legitimate predicate sequences starting from  $e$  in  $\mathcal{G}$ . Following previous works (Yih
Figure 1: Base framework for monolingual KG, consisting of three modules to construct a query graph.
et al., 2015; Maheshwari et al., 2019), we fetch the chains whose length  $\leq 2$ . For example, as in the middle left of Figure 1, chain candidates are generated from the entity “<dbr:Ven.-Ram.>” within 2-hop on  $\mathcal{G}$ . Then, a model is presented to measure the semantic relatedness between the question  $q$  and each candidate of inferential chain  $c_i^e$ , i.e.,
$$
a _ {i} ^ {e} = \operatorname {S e m M a t c h} \left(q, c _ {i} ^ {e}; \theta^ {(I C)}\right), \forall i = 1, \dots , n, \tag {1}
$$
where  $a_{i}^{e}$  is a score for their relatedness, and  $\theta^{(IC)}$ -parameterized SemMatch( $\cdot$ ) can be any model for pairwise relatedness, such as Co-Attention network (Chen et al., 2019) and BERT-based Matching (Devlin et al., 2019). Finally, the resulting of this module is the top-1 ranked inferential chain, i.e.,
$$
\bar {c} ^ {e} = \arg \max _ {c _ {i} ^ {e}} (a _ {i} ^ {e}, \forall i = 1, \dots , n). \qquad (2)
$$
Note, if there are multiple grounded entities in  $q$ , we will predict an inferential chain for each entity.
Type Constraint Ranking. Type constraints (TC) refer to the entity types specified in the question for each variable on an inferential chain. They can be used to disambiguate the entities and thus boost KGQA performance. For example, answer entity(s) to the example question in Figure 1 are constrained by type Scientist. Hence, type constraint ranking is proposed to capture such information, which is also achieved by a semantic matching model. Specifically, given the resulting inferential chain  $\bar{c}^e$ , we first enumerate type candidates
$\mathcal{T}_y^e = \{t_{y1}^e,\dots \}$  for the existential variable and  $\mathcal{T}_x^e = \{t_{x1}^e,\ldots \}$  for the lambda variable. Then, because there is scarcely overlap of gold type constraints between the two variables, a single semantic matching model is adequate for both. Thus, we define the model to derive relatedness scores as
$$
b _ {* j} ^ {e} = \operatorname {S e m M a t c h} \left(q, t _ {* j} ^ {e}; \theta^ {(T C)}\right), \tag {3}
$$
where,  $\forall * \in \{y, x\}$ , and  $\forall j = 1, \ldots$
Finally, we get the type constraints for existential and lambda variable with a threshold  $\gamma^{(thresh)}$ , i.e.,
$$
\bar {\mathcal {T}} _ {*} ^ {e} = \left\{t _ {* j} ^ {e} \mid b _ {* j} ^ {e} > \gamma^ {(t h r e s h)}, \forall j = 1, \dots \right\}. \tag {4}
$$
Aggregator Classification Given several answer formats in the dataset, aggregator classification (AC) is presented to distinguish the format among Bool, Count and Entity(s). The principle of each is detailed in the middle right of Figure 1. Formally, a simple text classifier can satisfy, i.e.,
$$
\boldsymbol {p} ^ {(A C)} = \operatorname {C l a s s i f i e r} (q; \theta^ {(A C)}) \in \mathbb {R} ^ {3}, \tag {5}
$$
where the Classifier  $(\cdot)$  is composed of a contextualized encoder, a pooler and an MLP with softmax.
Once the above is completed, their results can compose a query graph, which is transformed into SPARQL and then executed on  $\mathcal{G}$  for the answer.
3.2 Proposed Multilingual KGQA Approach
Built upon the base framework detailed before, we extend it with a multilingual inference capability,
Figure 2: Syntax-agnostic semantic matching between a BLI-augmented multilingual  $q$  and its symbolic candidates.
i.e., multilingual KGQA. We are in line with a recent popular zero-shot transfer paradigm (Conneau et al., 2020; Fang et al., 2020) that: a pre-trained multilingual encoder is only fine-tuned in src, and a translation-based data augmentation technique is integrated to narrow the performance gap between src and tgt. To emphasize the gap in KGQA,  $65\%$  F1 score in English (src) vs.  $54\%$  in Italian (tgt) is observed by mBERT zero-shot transfer in our pipeline without any multilingual augmenting.
Distinct from prior works in this paradigm requiring well-trained translators, we propose a fully unsupervised way for wide applicability with neither tgt KGQA data nor src-tgt parallel corpora. It is natural to resort to bilingual lexicon induction (BLI) with unsupervised training and acceptable word-level translating quality. In the following, we first present a BLI-based augmentation for multilingual training data, followed by our adaptation of the monolingual base framework (§ 3.1) to the augmented data. Finally, we propose an adversarial learning strategy coupled with BLI-based augmentation for robust cross-lingual transfer. An illustration of our proposed semantic matching model with symbolic candidates is in Figure 2.
3.2.1 BLI-based Multilingual Augmentation
We leverage the BLI model by Lample et al. (2018b). First, it pre-trains monolingual word embeddings  $\mathbf{U}^{src} \in \mathbb{R}^{d \times |\mathcal{V}^{src}|}$  and  $\mathbf{U}^{tgt} \in \mathbb{R}^{d \times |\mathcal{V}^{tgt}|}$  in src and tgt respectively. Then, it learns a linear transformation to unsupervisedly align the word embeddings in two languages to one space, i.e.,
$$
\bar {\boldsymbol {W}} = \underset {\boldsymbol {W} \in M _ {d} (\mathbb {R})} {\arg \min } \sum \operatorname {D i s t a n c e} \left(\boldsymbol {W} \boldsymbol {U} _ {:, k} ^ {s r c}, \boldsymbol {U} _ {:, l} ^ {t g t}\right). \tag {6}
$$
The unsupervised alignment between  $k$ -th src word and  $l$ -th tgt word is captured by adversarial learning, and Distance( $\cdot$ ) is implemented by cross-domain similarity local scaling (CSLS). Please refer to (Lample et al., 2018b) for its details.
Based on the BLI model, we can build a word-byword translator,  $\mathrm{BLI}_{src\rightarrow tgt}^{(trans)}$ , from src to arbitrary tgt, as long as its monolingual corpus is available. Note, when performing word-level translation, we also employ CSLS to mitigate the hubness problem and find the most likely alignment. Then, we translate each question  $q^{src}$  in  $\mathcal{D}^{src}$  to other languages:
$$
q ^ {t g t} = \operatorname {B L I} _ {s r c \rightarrow t g t} ^ {(t r a n s)} \left(q ^ {s r c}\right), \tag {7}
$$
where  $src$  denotes English (en) in our experiments while tgt can be one of 11 other languages, such Farsi (fa), Italian (it), etc. Consequently,  $q^{tgt}$  is the augmented multilingual data for model training.
Remark: Although BLI provides multilingual data, open questions still remain. 1) Why is BLI competent here: It is observed KGQA mainly involves word-/phrase-level semantics of symbolic candidates, rather than sentence-level one in most other NLP tasks. As the Module 1 and 2 in Figure 1, the matching only involves morphological similarity (e.g., scientist vs. <dbo:Scientist>), synonym (e.g., won an award vs. <dbp:prizes>), etc. Thus, KGQA is less sensitive to long-term context than other tasks. This has been leveraged by Berant et al. (2013) to propose a phrase matching model for monolingual KGQA. 2) Will BLI lead to error propagation: Since BLI model achieves a high Precision@10 but a relatively low Precision@1, wrong translation and the corresponding ground truth are semantically similar. Intuitively, their word embeddings are spatially close to each other, so wrong word-level translation is equivalent to applying tiny noise to word embeddings, which hardly leads to error propagation when robust pretrained Transformer-based encoder is used.
3.2.2 Multilingual Models
Symbolic Candidate Processing. For an inferential chain, we enrich each predicate on the chain by 1) transforming each camel-represented phrase into sequence-formatted words 2) prefixing  $+ / -$  for directional information, and 3) concatenating topfrequent types in local closed-world assumptions (KrompaB et al., 2015). For a type constraint, we simply transform each camel-represented phrase into sequence-formatted words. In the following,
we denote the text of a processed symbolic candidate as  $z$  no matter it is a chain or type.
Multilingual Semantic Matching Model. As detailed in §3.1, both inferential chain ranking and type constraint ranking modules are built upon a semantic matching model between the question  $q$  and a symbolic candidate  $z$ . Note,  $z$  is always in src while  $q$  can be in either src or BLI-translated tgt. Following the common practice, we first concatenate  $q$  and  $z$  with special tokens (Devlin et al., 2019), which is passed into a pre-trained multilingual Transformer encoder, i.e.,
$$
\boldsymbol {v} = \operatorname {P o o l} (\text {T r a n s f o r m e r} (t e x t)), \tag {8}
$$
where,  $text = ([CLS], q, [SEP], z, [SEP])$ .
$\mathrm{Pool}(\cdot)$  denotes using the contextualized embedding of [CLS] to represent the entire input. In this paper, the encoder is alternative between mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020). Lastly, a 1-way multi-layer perceptron (MLP) built upon  $\mathbf{v}$  is presented to calculate the matching score in Eq.(1) or Eq.(3).
Multilingual Classification Model. As detailed in §3.1, a text classification model is required to identify aggregator. To fit into our zero-resource multilingual scenario, the model, consisting of a pre-trained multilingual encoder and an MLP-based predicting layer, can be directly fine-tuned on the augmented questions, i.e.,  $q^{src}$  and  $q^{tgt}$ .
3.2.3 Syntax-agnostic Adversarial Strategy
Although training the KGQA model on BLI-augmented multilingual data circumvents language inconsistency, it inevitably introduces syntax disorder and grammatical problem, which could hurt the performance. We thus present an adversarial strategy in pair with BLI-augmented data to push the Transformer encoder deriving language- and syntax-independent representations. Formally, a discriminator is built upon the single vector representation  $\pmb{v}$  produced by the Transformer encoder:
$$
p ^ {(s r c)} = \mathrm {S i g m o i d} (\mathrm {M L P} (\boldsymbol {v}; \theta^ {(d i s)})), \qquad (9)
$$
where  $p^{(src)}$  is the probability of the question in source. The discriminator is trained to minimize
$$
\mathcal {L} _ {\theta (d i s)} ^ {(a d v)} = - \mathbb {I} _ {(s r c)} \log p ^ {(s r c)} - \mathbb {I} _ {(t g t)} \log \left(1 - p ^ {(s r c)}\right). \tag {10}
$$
On the contrary, the Transformer encoder is learned to fool by minimizing an adversarial loss, i.e.,
$$
\mathcal {L} _ {\theta (e n c)} ^ {(a d v)} = - \mathbb {I} _ {(t g t)} \log p ^ {(s r c)}. \tag {11}
$$
$\mathbb{I}_{(tgt)}$  denotes if the question in BLI-translated  $tgt$ , and  $\theta^{(enc)}$  is encoder's parameters in each module.
Before constructing the objectives, we conduct uniform negative sampling for the two ranking models with the maximum negative number limited to 100.
First, gold labels of a  $q$  for the three modules stem from the formal query  $s^{src}$ . A margin-based hinge loss is defined for inferential chain ranking:
$$
\hat {\mathcal {L}} ^ {(I C)} = \frac {1}{| \mathcal {D} |} \sum_ {\mathcal {D}} \frac {1}{| \mathcal {N} |} \sum_ {i = 1} ^ {| \mathcal {N} |} \left(\lambda - \tilde {a} ^ {e} + \hat {a} _ {i} ^ {e}\right), \tag {12}
$$
where,  $\mathcal{D}$  is the augmented dataset,  $\mathcal{N}$  is a set of negative chains,  $\tilde{a}^e$  is derived from the gold chain and  $\hat{a}_i^e$  is derived from a negative chain. Similarly, the loss defined for type constraint ranking is
$$
\hat {\mathcal {L}} ^ {(T C)} = \frac {1}{| \mathcal {D} |} \sum_ {\mathcal {D}} \frac {1}{2 | \mathcal {N} |} \sum_ {* \in \{y, x \}} \sum_ {j = 1} ^ {| \mathcal {N} |} (\lambda - \tilde {b} _ {*} ^ {e} + \hat {b} _ {* j} ^ {e}).
$$
Lastly, the loss of aggregator classification is
$$
\hat {\mathcal {L}} ^ {(A C)} = - \frac {1}{| \mathcal {D} |} \sum_ {\mathcal {D}} \log \boldsymbol {p} _ {[ i = \tilde {g} ]} ^ {(A C)}, \tag {13}
$$
where  $\pmb{p}_{[i = \tilde{g} ]}^{(AC)}$  denotes probability corresponding to gold aggregator class.
During training, the adversarial loss is added to the loss function of each module to compose the final training objective, i.e.,
$$
\mathcal {L} ^ {(*)} = \hat {\mathcal {L}} ^ {(*)} + \alpha \mathcal {L} _ {\theta (e n c)} ^ {(a d v)}, * \in \{I C, T C, A C \}. \tag {14}
$$
3.4 Inference Algorithm
As in Algorithm 1, we provide a detailed procedure for model inference in target language.
We also provide an explanation of query graph in Figure 1. As the example query graph shown in the right of the figure: a topic entity is first grounded as  $e = "<dbr:Ven.-Ram>"$  in rounded rectangle, an existential variable in circle denotes intermediate entity set?  $y = \{h|(h,leaderName,e)\}$ , a lambda variable in shaded circle denotes the answer entity set?  $x = \{h|(h,prizes,e)\land \forall e\in ?y\}$ , and an aggregator COUNT is finally applied to  $?x$  that is constrained by entity type "<dbo:Scientist)". Note that, the existential variable can not exist if only 1-hop relation is expressed in a question, and if multiple topic entities are grounded, multiple "?x" will be merged by intersection.
Algorithm 1 Inference in Target Language.
Require: A  $q$  in tgt and its grounded topic entities  $\mathcal{E}^q$ ; KG  $\mathcal{G}$ ; Models  $\theta^{(IC)}$ ,  $\theta^{(TC)}$ ,  $\theta^{(AC)}$
1: Search the chain candidates  $\mathcal{C}^e$  on  $\mathcal{G},\forall e\in \mathcal{E}^q$
2: Rank each  $\mathcal{C}^e$  by Eq.(1), and keep top-3 in  $\mathcal{C}^e$
3:  $\mathcal{C}^e\gets \{c^e |c^e\in \mathcal{C}^e\wedge \mathrm{Size}(?x\in c^e) > 0\}$
4:  $\overline{c}^e\gets Null$
5: if  $\mathrm{Size}(\mathcal{C}^e) > 0$  then
$\bar{c}^e \gets$  the top1 inferential chain in  $\mathcal{C}^e$
6: end if
7: Merge chains  $\{\bar{c}^e |\forall e\in \mathcal{E}^q\land \bar{c}^e$  is not Null}
8: Rank type constraint candidates by Eq.(3) and apply the top-1 constraint w/ score  $>\gamma^{(thresh)}$
9: Generate SPARQL and execute on  $\mathcal{G}$  for answer entity set  $\mathcal{A}$
10: Identify the aggregator for  $q$  by Eq.(5)
11:  $\mathcal{A}\gets$  Aggregate(A) by following Figure 1
12: return  $\mathcal{A}$ ;
We evaluate the proposed approach on two datasets, LC-QuAD (Trivedi et al., 2017) and QALD-multilingual (Usbeck et al., 2018), both of which contain questions with corresponding SPARQL queries over DBpedia<sup>1</sup>. DBpedia is a large-scale knowledge graph extracted from Wikipedia pages with 6 million/60 thousands/13 billion entities/predicates/triples in the English edition.
LC-QuAD. LC-QuAD is a large-scale complex question answering dataset, which contains 5000 English question-SPARQL pairs<sup>2</sup>. We follow the official split with 1000 questions in the test set, and further split the original training set into training-valid with 3500/500 questions. To evaluate the effectiveness of multilingual KGQA, questions in the test set are translated into 10 languages (fa, de, ro, it, ru, fr, nl, es, hi, pt)<sup>3</sup> using Google Translator<sup>4</sup>.
QALD-multilingual. QALD is a series of evaluation campaigns on question answering over linked data<sup>5</sup>. We collect all multilingual questions along with their SPARQL queries from QALD4