SlowGuess/NAACL_2021_OCR huggingface dataset:
to QALD9 and filter out some out-of-scope ones. There are overall 429 distinct question-SPARQL pairs and most are expressed in 12 languages (en, fa, de, ro, it, ru, fr, nl, es, hi_IN, pt, pt_BR). Considering the small size of this dataset, we take all QALD-multilingual questions as test set, and use the training data of LC-QuAD for model training.
Evaluation Metrics. We adopt two widely-used metrics as following (Maheshwari et al., 2019), i.e., inferential chain accuracy (ICA) and macro F1 score. The former is used to measure the accuracy (i.e., Precision@1) of inferential chain model, and defined as the percent of correctly-predicted inferential chains. The macro F1 score is used to measure the performance of final answers. Please refer to (Maheshwari et al., 2019) for the details.
We evaluate our approach with 2 multilingual encoding models, i.e.  $\mathrm{mBERT}_{\mathrm{base}}$  and XLM- $\mathbf{R}_{\mathrm{base}}$ . The embedding and hidden size in both models are set to 768. We use Adam optimizer (Kingma and Ba, 2015) to optimize the KGQA loss with the learning rate of  $5 \times 10^{-5}$  and a linear warm-up (Vaswani et al., 2017). The maximum training epoch, warm-up epoch, and batch size are set to 35, 3, and 32. The discriminator is trained along with each module's objective, with  $\alpha$  set to  $5 \times 10^{-4}$  for learning to fool. The discriminator is optimized via the Adam optimizer with a learning rate of  $5 \times 10^{-5}$ .  $\gamma^{(thresh)}$  for the type constraint model is set to 0.7. We follow (Maheshwari et al., 2019) and use the same values for other parameters in model training.
We compare our approach with a natural, widely-used baseline, which fine-tunes a pre-trained multilingual model (e.g., mBERT, XLM-R) on source language, and then directly apply it to target languages. The comparison on QALD-multilingual and LC-QuAD with mBERT are reported in Table 1 and 2 respectively. It is showed that our approach outperforms the baseline significantly on both datasets for all languages. ICA is improved by  $1\% -4\%$ , and  $2.9\%$  on average on the QALD dataset. The improvement on LC-QuAD is even larger, i.e., averaged ICA and F1 score of all languages are increased by around  $7\%$  and  $4\%$  respectively. Notably, with the BLI-augmented data and syntax
$^{6}$ https://github.com/yczhou001/Multilingual-KBQA-Dataset/tree/main/QALD.
<sup>1</sup>We use the 2016-10 version, which can be downloaded at https://wiki.dbpedia.org/downloads-2016-10.
$^{2}$ https://github.com/AskNowQA/LC-QuAD.
<sup>3</sup>https://github.com/yczhou001/Multilingual-KBQA-Dataset/tree/main/LC-QuAD.
4https://translate.google.com/.
$^{5}$ https://github.com/ag-sc/QALD.
ICA	en	fa	de	ro	it	ru	fr	nl	es	hi_IN	pt	pt_BR	Avg	Avg w/o en
Baseline	80.7	76.0	77.8	76.8	76.5	80.4	76.9	78.5	77.6	79.3	80.9	86.3	79.0	78.8
Ours	83.7	77.6	80.5	79.2	80.5	83.1	80.3	80.5	81.7	82.5	85.3	87.4	81.9	81.7
Lift	+3.1	+1.6	+2.8	+2.5	+4.0	+2.7	+3.4	+2.0	+4.0	+3.2	+4.4	+1.1	+2.9	+2.9
F1	en	fa	de	ro	it	ru	fr	nl	es	hi_IN	pt	pt_BR	Avg	Avg w/o en
Baseline	65.0	58.0	60.8	60.2	53.7	60.5	59.8	64.3	55.2	59.3	60.5	70.0	60.6	60.2
Ours	66.7	60.0	62.2	62.1	57.7	63.5	63.6	65.9	58.8	62.6	63.5	70.0	63.0	62.7
Lift	+1.7	+2.0	+1.4	+2.0	+4.0	+3.0	+3.8	+1.7	+3.7	+3.2	+3.1	+0.0	+2.5	+2.5
Table 1: Comparison on QALD-multilingual using mBERT.
ICA	en	fa	de	ro	it	ru	fr	nl	es	hi_IN	pt	Avg	Avg w/o en
Baseline	87.0	83.8	88.3	86.1	86.0	86.0	86.9	87.2	88.2	83.7	86.6	86.3	86.3
Ours	94.7	91.7	93.3	93.1	93.2	92.7	93.1	94.2	94.1	92.6	93.4	93.3	93.2
Lift	+7.7	+7.9	+5.0	+7.0	+7.3	+6.7	+6.2	+7.0	+5.9	+9.0	+6.8	+6.9	+6.9
F1	en	fa	de	ro	it	ru	fr	nl	es	hi_IN	pt	Avg	Avg w/o en
Baseline	80.1	66.6	78.3	68.9	69.1	71.1	69.5	75.8	72.9	66.5	69.3	71.6	70.8
Ours	85.5	71.7	82.4	72.6	72.3	74.5	73.2	80.9	76.1	71.9	74.0	75.9	74.9
Lift	+5.4	+5.1	+4.1	+3.6	+3.2	+3.4	+3.6	+5.1	+3.2	+5.5	+4.7	+4.3	+4.2
Table 2: Comparison on LC-QuAD-multilingual using mBERT.
agnostic adversarial learning, the performance of source-language (i.e., English) questions are also increased by a large margin, i.e., F1 score increases from  $65\%$  to  $66.7\%$  on QALD, and from  $80\%$  to  $85\%$  on LC-QuAD. We also evaluate the propose approach using XLM-R as the multilingual encoder. The comparison on QALD-multilingual is shown in Table 3. We can observe similar improvements as in mBERT, where both averaged ICA and F1 score are increased by around  $1\%$ , verifying the effectiveness of our proposed approach.
Our approach consists of two important components, BLI-based data augmentation and a syntax-agnostic learning strategy. We conduct an ablation study to investigate the effect of each component. Table 4 reports the averaged results of all target-languages on QALD-multilingual and LCQuAD-multilingual. From the table we can see that, with BLI-based data augmentation, our approach increases the ICA score on QALD by  $1.7\%$ , and the syntax-agnostic adversarial learning further improves it by  $1.2\%$ . Similar improvements are observed on LC-QuAD, which verifies the effectiveness of both components in our approach.
Impact of BLI Accuracy. We assess the impact of BLI accuracy on five Romance languages (i.e. it, fr, es, pt, and ro) by injecting noise into BLI results. Specifically, when mapping source-language
Figure 3: Impact of BLI Accuracy in our approach. The x-axis represents the percentage of noise we inject into BLI results, while y-axis represents the performance in terms of ICA in Figure (left) and F1 score in Figure (right).
words into a target language via BLI, we randomly replace translated words with wrong ones with a probability of  $p$  (10%, 20%, 30%, 40%, and 50%). The averaged performance of our approach on the five languages is reported in Figure 3. It is observed, with more noise added, the performance of our approach drops, which is in accordance with intuition. But even when 50% of the translated words are noisy, our method still outperforms the baseline model. For example, it is superior than the baseline by 1% in terms of ICA with 50% noise, showing the robustness of our approach.
Deep Dive into Adversarial Learning. We take the inferential chain ranking model as an example, and take a deep dive into the impact of syntax-agnostic adversarial learning. The adversarial learning involves a discriminator to distinguish whether a question is grammatical or syntax-disorder, and an inferential chain ranking model to identify the gold chain. Their loss values, i.e.,  $\mathcal{L}_{\theta (dis)}^{(dis)}$  and
ICA	en	fa	de	ro	it	ru	fr	nl	es	hi_IN	pt	pt_BR	Avg	Avg w/o en
Baseline (XLM-R base)	81.5	76.9	75.6	77.7	76.7	80.9	76.5	78.8	77.4	80.2	80.4	84.2	78.9	78.7
Ours (XLM-R base)	84.0	78.1	77.5	79.0	77.1	80.9	77.8	79.4	78.1	81.3	80.9	85.3	79.9	79.6
Lift	+2.5	+1.2	+1.9	+1.4	+0.4	+0.0	+1.3	+0.7	+0.7	+1.2	+0.5	+1.1	+1.1	+0.9
F1	en	fa	de	ro	it	ru	fr	nl	es	hi_IN	pt	pt_BR	Avg	Avg w/o en
Baseline (XLM-R base)	63.4	57.1	54.7	58.8	50.1	59.4	56.3	61.3	51.2	59.2	57.5	66.1	57.9	57.4
Ours (XLM-R base)	64.6	57.6	56.1	61.4	50.9	59.4	58.2	62.1	52.2	60.6	57.4	66.1	58.9	58.4
Lift	+1.2	+0.5	+1.4	+2.6	+0.8	+0.0	+1.9	+0.8	+1.0	+1.3	-0.1	+0.0	+1.0	+0.9
Table 3: Comparison on QALD-multilingual using XLM-R.
Avg w/o en	QALD	LC-QuAD
ICA	F1	ICA	F1
BLI-only	80.5	60.9	91.7	74.2
BLI-only vs. Baseline	+1.7	+0.7	+5.5	+3.4
BLI+Adv.	81.7	62.7	93.2	74.9
BLI+Adv. vs. BLI-only	+1.2	+1.8	+1.4	+0.7
Table 4: Ablation study. "BLI-only vs. Baseline" represents the effect of BLI. "BLI+Adv. vs. BLI-only" represents effect of syntax-agnostic adversarial learning.
Figure 4: Adversarial losses on validation set w.r.t different epochs in training phase.
$\hat{\mathcal{L}}^{(IC)}$ , are plot in Figure 4. We can see that the classification loss of the discriminator quickly drops and then slowly goes up, indicating that the discriminator gets good performance and then it is fooled later by the language-/syntax-agnostic embeddings generated by mBERT. Meanwhile, the inferential ranking loss drops quickly and stays very small in following epochs, showing that when mBERT is generating syntax-agnostic embeddings, it also supports the inferential chain ranking very well.
We take several examples of inferential chain ranking to show how our approach works. We use t-SNE (Maaten and Hinton, 2008) to map the embedding of a question-chain pair into a two-dimensional data point. A question in a specific
Figure 5: Case study via t-SNE visualization. Different points in a graph represents different languages for the same question. (Upper left) and (upper right) show embeddings of baseline and our approach for the question "what is the population of Cairo?". (Lower left) and (lower right) show embeddings of baseline and our approach for the question "which species does an elephant belong?"
language is paired with its golden inferential chain and top-1 ranked negative candidate. Figure 5 compares the baseline with our approach for two questions. Positive and negative examples of the same question in different languages are plot in the same figure. We can see that the baseline model can not distinguish positive inferential chains from negative ones well, while our approach can learn a language-agnostic representation that focuses more on ranking inferential chain candidates.
There are mainly two categories of approaches to handle monolingual question answering over knowledge graph (KGQA) task. (1) Information retrieval-based approaches align a question with its answer candidates in the same semantic space,
where the candidates usually stem from KG neighbors of the topic entity detected in the questions (Bordes et al., 2014b,a; Dong et al., 2015; Jain, 2016; Xu et al., 2016; Hao et al., 2017; Chen et al., 2019). (2) Semantic parsing-based approaches first translate a question into the corresponding logical form, e.g., program (Guo et al., 2018; Shen et al., 2019) or query graph (Yih et al., 2015; Jia and Liang, 2016; Xiao et al., 2016; Dong and Lapata, 2016; Liang et al., 2017; Dong and Lapata, 2018; Maheshwari et al., 2019), and then execute the logical form over KG to derive the final answer. Note a logical form is usually composed of a series of grammars or operators pre-defined by experts. This paper is in line with the second category to generate query graph for KG execution. To the best of our knowledge, there are only few works targeting multilingual KGQA (Hakimov et al., 2017; Veyseh, 2016), which rely on extensive multilingual training data with hand-crafted features while are inapplicable to the zero-shot transfer scenario. So we adopt the pipeline by Maheshwari et al. (2019) for monolingual scenario as our base model but update the encoders with the Transformer (Vaswani et al., 2017) to strengthen their expressive power and facilitate recent pre-trained multilingual initializations.
Given task-specific data in a source language, cross-lingual models are trained to perform inference in target languages in a low- or zero-resource scenario. Typically, cross-lingual models are proposed in two paradigms. 1) Universal encoding-based paradigm represents multilingual natural language text into language-agnostic embeddings the same semantic space. Early works focus on aligning multilingual word embedding (Mikolov et al., 2013; Faruqui and Dyer, 2014; Xu et al., 2018), while recent efforts are mainly made on large-scale pre-trained multilingual encoder, such as mBERT (Devlin et al., 2019), XLM (Conneau and Lample, 2019), Unicoder (Huang et al., 2019a), XLM-R (Conneau et al., 2020), InfoXLM (Chi et al., 2020), and ALM (Yang et al., 2020). They can perform zero-shot cross-lingual transfer by training in the source language while directly inference in target language. 2) translation-based paradigm employs well-trained machine translators to map the training or test examples in source language to those in target translation. Recent common practice tends to leverage the second paradigm to generate multilingual data to narrows the zero-shot
cross-lingual performance gap in the first paradigm, which leads to state-of-the-art results on several cross-lingual benchmarks. In contrast, we consider a zero-resource scenario where translators are unavailable and we thus resort to unsupervised BLI in light of KGQA's characteristics.
As a branch of universal encoding at word level, bilingual lexicon induction (BLI) (a.k.a crosslingual word embedding - CLWE) is learned to align bilingual word embeddings in the same space, where the embeddings are pre-trained on monolingual corpora and the alignment is trained in either a (semi-)supervised or unsupervised manner (Smith et al., 2017; Lample et al., 2018b; Artetxe et al., 2018, 2019; Huang et al., 2019b; Patra et al., 2019; Karan et al., 2020; Zhao et al., 2020; Ren et al., 2020). To alleviate "hubness" problem (Dinu and Baroni, 2015) in BLI, alternatives of the distance measurement are proposed to substitute nearest neighbor (NN) during the alignment, such as inverted-softmax (Smith et al., 2017) and CSLS (Lample et al., 2018b). In addition to building bilingual dictionary via word-level translation, a well-trained BLI model can serve as a weak baseline of sentence-level translation (Lample et al., 2018a), a seed model for unsupervised translation (Lample et al., 2018a) or a bilingual variant of copy mechanism in summarization (Zhu et al., 2020).
Moreover, adversarial training is usually integrated into cross-lingual models for language-agnostic representation learning, such as unsupervised BLI (Lample et al., 2018b; Zhang et al., 2017), unsupervised translation (Lample et al., 2018a), cross-Lingual sequence labeling (Kim et al., 2017; Huang et al., 2019c) and cross-Lingual classification (Dong et al., 2020). In contrast, our adversarial strategy not only considers language-agnostic representations but also aims at making the model insensitive to syntax-disorder and thus competent in zero-resource scenario.
We propose a novel approach for zero-shot crosslingual transfer in multilingual KGQA, which augments training data by bilingual lexicon induction, and leverages a syntax-agnostic adversarial learning strategy to alleviate the syntax-disorder problem caused by BLI. Experimental results on two multilingual KGQA datasets in 11 zero-resource languages verify its effectiveness.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 789-798. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2019. Bilingual lexicon induction through unsupervised machine translation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 5002-5007. Association for Computational Linguistics.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In The semantic web, pages 722-735. Springer.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1533-1544. ACL.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247-1250.
Antoine Bordes, Sumit Chopra, and Jason Weston. 2014a. Question answering with subgraph embeddings. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar; A meeting of SIGDAT, a Special Interest Group of the ACL, pages 615-620. ACL.
Antoine Bordes, Jason Weston, and Nicolas Usunier. 2014b. Open question answering with weakly supervised embedding models. In Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part I, volume 8724 of Lecture Notes in Computer Science, pages 165-180. Springer.
Yu Chen, Lingfei Wu, and Mohammed J. Zaki. 2019. Bidirectional attentive memory networks for question answering over knowledge bases. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2913-2923. Association for Computational Linguistics.
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, XianLing Mao, Heyan Huang, and Ming Zhou. 2020. Infoxm: An information-theoretic framework for cross-lingual language model pre-training. CoRR, abs/2007.07834.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8440-8451. Association for Computational Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 7057-7067.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2019. Cross-lingual machine reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 1586-1595. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Georgiana Dinu and Marco Baroni. 2015. Improving zero-shot learning by mitigating the hubness problem. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings.
Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 731-742. Association for Computational Linguistics.
Li Dong, Furu Wei, Ming Zhou, and Ke Xu. 2015. Question answering over freebase with multicolumn convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pages 260-269. The Association for Computer Linguistics.
Xin Dong, Yaxin Zhu, Yupeng Zhang, Zuohui Fu, Dongkuan Xu, Sen Yang, and Gerard de Melo. 2020. Leveraging adversarial training in self-learning for cross-lingual text classification. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 1541-1544. ACM.
Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, and Jingjing Liu. 2020. FILTER: an enhanced fusion method for cross-lingual language understanding. CoRR, abs/2009.05166.
Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, April 26-30, 2014, Gothenburg, Sweden, pages 462-471. The Association for Computer Linguistics.
Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. 2018. Dialog-to-action: Conversational question answering over a large-scale knowledge base. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pages 2946-2955.
Sherzod Hakimov, Soufian Jebbara, and Philipp Cimiano. 2017. Amuse: multilingual semantic parsing for question answering over linked data. In International Semantic Web Conference, pages 329-346. Springer.
Yanchao Hao, Yuanzhe Zhang, Kang Liu, Shizhu He, Zhanyi Liu, Hua Wu, and Jun Zhao. 2017. An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 221-231. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. CoRR, abs/2003.11080.
Sen Hu, Lei Zou, and Xinbo Zhang. 2018. A state-transition framework to answer complex questions