SlowGuess/NAACL_2022_OCR huggingface dataset:
As shown in Table 2, our method, Crake, outperforms all former methods by a large  $\sim 17\%$  margin on F1, becoming the new SoTA of LC-QuAD 1.0. Surpassing methods requiring aux tools (I) on all metrics, we present the effectiveness of independent pipelines (II) that avoid cascading errors. Also, we achieve consistent answer precision and recall to surpass other methods in II on F1, showing the superiority of our pipeline design, which is further discussed in the sections below.
Methods	Decoder Parameters	NE Accuracy	GSG Accuracy	End-to-end
P	R	F1	EM	Actual	P	R	F1
Seq2seq	76.67M (×1)	.895	.901	.897	.695	.768	.653	.674	.654
TF	0.66M (×1/100)	.895	.901	.897	.728	.795	.655	.674	.657
TF+SMTL	0.66M (×1/100)	.901	.904	.902	.735	.805	.665	.684	.667
TF+Causal	3.03M (×1/25)	.909	.914	.911	.755	.828	.677	.696	.680
Table 3: Experiments on table-filling and causal-modelling. Seq2seq and TF adopt a generative decoder and table-filler in GC respectively, while both deal with NE and GC by separate models. TF+SMTL (simple multitask learning) co-trains NE and GC by directly adding losses without modelling their intrinsic causal effects. TF+Causal denotes our full approach which models the causal effects between NE and GC by label transfer. We report the node-level P/R/F1 in NE, the exact-match (EM) and actual accuracy (that ignores variable mentions in judging accuracy) in GSG, and the overall answer-level P/R/F1 on LC-QuAD 1.0 dev set for comparison.
5.2 Effects of Tabel-Filling
As explained in Section 3.2, modelling GC as a sequence-generation causes a few issues that can be overcome by table-filling. Specifically, the sequence ambiguity confuses the learning process and requires large decoders to grasp the sequence generation policy, which may slow down the convergence. Besides, the exposure bias harms the decoding accuracy of the model at inference. This section, we try to verify such effects by experiments. To enable the comparison with sequence-generation, we construct a generative decoder as in Zhang et al., 2021 as the baseline, which sequentially generates the connected node pairs in the graph structure to represent the edges. We train the generative model under the same settings (e.g. learning rate, warmup, epochs, etc.), resulting in the performance of Seq2seq in Table 3.
Comparing with the table-filling model (i.e. TF in Table 3), Seq2seq comes short in the accuracy of graph structure, indicating the negative effects of the exposure bias on predicting accuracy. Meanwhile, TF requires only 1/100 of Seq2seq's parameters to achieve comparable or better results, we attribute this to the removal of sequence ambiguity which frees the model from acquiring the complex and ambiguous scheme of sequence-generation. This speculation is further verified in Figure 5, in which TF converges distinctly quicker than Seq2seq since the simultaneous decision of all edges is well-defined and easier to learn. Thus, compared with sequence-modelling, handling GC via table-filling reduces model size and boosts training, which is essential for real-world applications.
Figure 6: Case study on the effects of causal-modelling.
5.3 Effects of Causal-Modelling
We propose a joint model to learn the NE-GC causalities in Section 3.3, to discuss its effects, we compare it with two alternatives in Table 3: 1) using two separate models for NE and GC (TF in Table 3) like Ravishankar et al., 2021, 2) co-training NE and GC by sharing encoder and adding losses  $(\mathrm{TF} + \mathrm{SMTL}$  in Table 3) like Shen et al., 2019. As shown, co-training consistently surpasses separate models by grasping the shared knowledge between NE and GC, nevertheless, our causal-modelling approach  $(\mathrm{TF} + \mathrm{Causal})$  further outperforms cotraining. In detail, though TF+Causal has similar results with TF+SMTL in NE, it achieves better accuracy for overall GSG  $(\mathrm{NE} + \mathrm{GC})$  and excels in end-to-end metrics. Therefore, we infer that causalmodelling improves the integrity of the GSG stage
Methods	Accuracy	Efficiency
P	R	F1	1-hop	2-hop	3-hop
Baseline	.560	.566	.556	0.12s	42.4s	84.2s
BeamSearch	.677	.696	.680	0.12s	1.06s	2.72s
Table 4: Performance comparison between our beam-search RE algorithm and its baseline in Section 4. Accuracy refers to the answer-level P/R/F1, efficiency is measured by the average run time on 1/2/3-hop queries.
by expressing the internal causalities between its subtasks. To better understand this, we perform a case study in Figure 6, in which TF fails to realize that "skier" also corresponds to a type node; in contrast, TF+SMTL extract all nodes correctly by learning both NE and GC labels, but it still fails in generating a correct graph structure. Finally, TF+Causal utilizes the VT tag of "skier" in NE predictions and correctly connects the II-IV edge in GC. Thus, Figure 6 demonstrates the usage of causal effects to reach higher accuracy in GSG.
5.4 Analysis on Beam-Search RE
In this section, we compare our beam-search RE algorithm with its baseline. As stated in Section 4, by alternately performing retrieval and ranking on each edge (rather than retrieving the candidates of every edge before ranking), our approach lowers the scale of candidate predicates on multi-hop queries to get better efficiency, which is verified in Table 4. In detail, BeamSearch costs substantially less time than Baseline in 2 and 3-hop queries (note that for 1-hop queries, two methods reduce to a same process with similar time costs). Since BeamSearch only operates on the neighbors of certain KB nodes, it avoids the retrieval of 2-hop neighbors, which requires considerable time on DBpedia, to improve efficiency. In addition, by pruning off useless candidates in Baseline, BeamSearch also achieves higher overall KBQA accuracy in Table 4. Therefore, Algorithm 1 transcends previous methods to reveal an efficient and accurate solution for ranking-based RE scalable to KB size and query complexity.
KBQA via Semantic Parsing A mainstream to solve KBQA is semantic parsing (Yih et al., 2016) which converts a question to a KB query to get answers. Due to the graph-like structure of KB queries, prior works construct query graphs
to represent queries in semantic parsing. Among them, some works (Zafar et al., 2018; Chen et al., 2020b) only focus on predicting the graph structure given node inputs. To perform end-to-end QA, Hu et al., 2017 leverages the dependency parsing tree to match KB subgraphs for answers; Kapanipathi et al., 2021 builds the query graph by transforming and linking the AMR (Banarescu et al., 2012) of the question; Hu et al., 2021 uses the constituency tree to compose an entity description graph representing the query graph structure. Requiring aux tools or data structures, these works may be subjected to cascading errors. Yih et al., 2015 overcomes this by an independent stage-transition framework to generate the query graph, Hu et al., 2018 extends the transitions to express more complex graphs. Besides, Zhang et al., 2021 adopts a pointer generator to decode graph structure, Ravishankar et al., 2021 generates the query skeleton by a seq2seq decoder. Unlike these methods that model the query graph as a sequence (by state-transition or generative decoder), we decode all edges at once via a table-filler in graph structure generation.
Modelling causal effects Causality occurs in various deep-learning scenarios between multiple channels or subtasks, existing works models the causality for better performance. Niu et al., 2021 mitigates the false causal effects in VQA (Antol et al., 2015) to overcome language bias; Zeng et al., 2020 dispels the incorrect causalities from different input channels of NER by generating counterfacts. Chen et al., 2020a utilizes the inter-subtask causalities to improve multitask learning for JERE (Li and Ji, 2014), ABSA (Kirange et al., 2014), and LJP. Unlike them, we formulate and utilize the internal causal effects in KBQA.
In this work, we formalize the generation of query graphs in KBQA by two stages, namely graph structure generation (GSG) and relation extraction (RE). In GSG, we propose a table-filling model for graph composition to avoid the ambiguity and bias of sequence-modelling, meanwhile, we encode the inherent causal effects among GSG by a label-transfer block to improve the stage integrity. In RE, we introduce an effective beam-search algorithm to retrieve and rank predicates in order for each edge, which turns out to be scalable for large KBs and multi-hop queries. Consequently, our approach substantially surpasses previous state-of-the-arts in
KBQA, revealing the effectiveness of our pipeline design. Detailed experiments also validate the effects of all our contributions.
Admittedly, our approach endures certain limitations as discussed below.
Query Expressiveness Like most semantic parsing systems, we fail to cover all the operations of SPARQL, limiting our capability to compose queries with complex filter or property path. For the conciseness of our system, we only focus on constructing triples in the multi-hop query graph in this paper, while we plan to incorporate more functions into Crake in the future to improve the expressiveness of the system.
Annotation Cost Training models with node mentions require expensive manual annotations, which is impractical for us to conduct on every popular KBQA dataset. As explained in Section 5, without data-oriented optimization, we believe the significant gain presented adequate to verify our contributions. Further, we expect to extenuate such costs in two directions for the future: 1) some modules of our framework (e.g. NE) is generalizable to other English questions, gifting it the potential to be transferred to other datasets without re-training; 2) few-shot (Wang et al., 2020) and active (Aggarwal et al., 2014) learning techniques aids the model to reach competitive performance with a small portion of annotated data, which can be explored in our framework to reduce annotation cost.
This work was supported by National Key R&D Program of China (2020AAA0105200) and NSFC under grant U20A20174. The corresponding author of this work is Lei Zou (zoulei@pku.edu.cn). We would like to thank Zhen Niu and Sen Hu for their kind assistance on this work. We also appreciate anonymous reviewers for their valuable comments and advises.
Charu C Aggarwal, Xiangnan Kong, Quanquan Gu, Jiawei Han, and S Yu Philip. 2014. Active learning: A survey. In Data Classification, pages 599-634. Chapman and Hall/CRC.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick,
and Devi Parikh. 2015. VQA: visual question answering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425-2433. IEEE Computer Society.
Ram G Athreya, Srividya K Bansal, Axel-Cyrille Ngonga Ngomo, and Ricardo Usbeck. 2021. Template-based question answering using recursive neural networks. In 2021 IEEE 15th International Conference on Semantic Computing (ICSC), pages 195-198. IEEE.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In *The semantic web*, pages 722-735. Springer.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2012. Abstract meaning representation (amr) 1.0 specification. In Parsing on Freebase from Question-Answer Pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle: ACL, pages 1533-1544.
Wenqing Chen, Jidong Tian, Liqiang Xiao, Hao He, and Yaohui Jin. 2020a. Exploring logically dependent multi-task learning with causal inference. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2213-2225, Online. Association for Computational Linguistics.
Yongrui Chen, Huiying Li, Yuncheng Hua, and Guilin Qi. 2020b. Formal query building with query structure prediction for complex question answering over knowledge base. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3751-3758. ijcai.org.
Yongrui Chen, Huiying Li, Guilin Qi, Tianxing Wu, and Tenggou Wang. 2021. Outlining and filling: Hierarchical query graph generation for answering complex questions over knowledge graph. arXiv preprint arXiv:2111.00732.
Wanyun Cui, Yanghua Xiao, Haixun Wang, Yangqiu Song, Seung-won Hwang, and Wei Wang. 2017. Kbqa: Learning question answering over qa corpora and knowledge bases. Proceedings of the VLDB Endowment, 10(5).
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net.
Sen Hu, Lei Zou, Jeffrey Xu Yu, Haixun Wang, and Dongyan Zhao. 2017. Answering natural language questions by subgraph matching over knowledge graphs. IEEE Transactions on Knowledge and Data Engineering, 30(5):824-837.
Sen Hu, Lei Zou, and Xinbo Zhang. 2018. A state-transition framework to answer complex questions over knowledge base. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2098-2108, Brussels, Belgium. Association for Computational Linguistics.
Xixin Hu, Yiheng Shu, Xiang Huang, and Yuzhong Qu. 2021. Edg-based question decomposition for complex question answering over knowledge bases. In International Semantic Web Conference, pages 128-145. Springer.
Pavan Kapanipathi, Ibrahim Abdelaziz, Srinivas Ravishankar, Salim Roukos, Alexander Gray, Ramón Fernández Astudillo, Maria Chang, Cristina Cornelio, Saswati Dana, Achille Fokoue-Nkoutche, et al. 2021. Leveraging abstract meaning representation for knowledge base question answering. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3884-3894.
DK Kirange, Ratnadeep R Deshmukh, and MDK Kirange. 2014. Aspect based sentiment analysis semeval-2014 task 4. Asian Journal of Computer Science and Information Technology (AJCSIT) Vol, 4.
Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 402-412, Baltimore, Maryland. Association for Computational Linguistics.
Yinnian Lin, Minhao Zhang, Ruoyu Zhang, and Lei Zou. 2021. Deep-ganswer: A knowledge based question answering system. In Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint International Conference on Web and Big Data, pages 434-439. Springer.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Weili Nie, Nina Narodytska, and Ankit Patel. 2019. *Relay: Relational generative adversarial networks for text generation*. In *7th International Conference on Learning Representations*, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. 2021. Counterfactual vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12700-12710.
Eric Prud'hommeaux. 2008. Sparql query language for rdf, w3c recommendation. http://www.w3.org/TR/rdf-sparql-query/.
Srinivas Ravishankar, June Thai, Ibrahim Abdelaziz, Nandana Mihidukulasooriya, Tahira Naseem, Pavan Kapanipathi, Gaetano Rossilleo, and Achille Fokoue. 2021. A two-stage approach towards generalization in knowledge base question answering. arXiv preprint arXiv:2111.05825.
Tao Shen, Xiubo Geng, Tao Qin, Daya Guo, Duyu Tang, Nan Duan, Guodong Long, and Daxin Jiang. 2019. Multi-task learning for conversational question answering over a large-scale knowledge base. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2442-2451, Hong Kong, China. Association for Computational Linguistics.
Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. 2017. Lc-quad: A corpus for complex question answering over knowledge graphs. In International Semantic Web Conference, pages 210-218. Springer.
Christina Unger, Lorenz Bühmann, Jens Lehmann, Axel-Cyrille Ngonga Ngomo, Daniel Gerber, and Philipp Cimiano. 2012. Template-based question answering over RDF data. In Proceedings of the 21st World Wide Web Conference 2012, WWW 2012, Lyon, France, April 16-20, 2012, pages 639-648. ACM.
Svitlana Vakulenko, Javier David Fernandez Garcia, Axel Polleres, Maarten de Rijke, and Michael Cochez. 2019. Message passing for complex question answering over knowledge graphs. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019, pages 1431-1440. ACM.
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. 2020. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34.
Yijun Wang, Changzhi Sun, Yuanbin Wu, Hao Zhou, Lei Li, and Junchi Yan. 2021. Unire: A unified label space for entity relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 220-231.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Mohamed Yahya, Klaus Berberich, Shady Elbassuoni, Maya Ramanath, Volker Tresp, and Gerhard Weikum. 2012. Natural language questions for the web of data. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 379-390, Jeju Island, Korea. Association for Computational Linguistics.
Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1321-1331, Beijing, China. Association for Computational Linguistics.
Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201–206, Berlin, Germany. Association for Computational Linguistics.
Hamid Zafar, Giulio Napolitano, and Jens Lehmann. 2018. Formal query generation for question answering over knowledge bases. In European semantic web conference, pages 714-728. Springer.
Xiangji Zeng, Yunliang Li, Yuchen Zhai, and Yin Zhang. 2020. Counterfactual generator: A weakly-supervised method for named entity recognition. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7270-7280, Online. Association for Computational Linguistics.
Minhao Zhang, Ruoyu Zhang, Lei Zou, Yinnian Lin, and Sen Hu. 2021. NAMER: A node-based multitasking framework for multi-hop knowledge base question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, pages 18-25, Online. Association for Computational Linguistics.
Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. 2019. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4334-4343, Florence, Italy. Association for Computational Linguistics.
A Details in Entity and Type Linking
We link each non-variable node to a KB entry by its mention. For entity nodes, we directly link it to an entity with the same name as its mention if such entity exists in the KB (e.g. link mention "New
York" to dbr:New_York); otherwise, we recall entities by DBpedia Lookup $^2$  and further prioritize ones whose lower-cased name is the same as the lower-cased mention (e.g.dbr:new_york). Then, the prioritized entity with the highest lookup score is linked to the node; if no entity is prioritized, the entity with the highest score is selected.
For type nodes, we build a dictionary  $D$  based on the mention-type pairs (e.g. authors-dbo:Writer) in train data and directly use the link result from  $D$  if the mention exists in  $D$ . Otherwise, we singularize and capitalize the mention to construct an URI with prefix dbo (e.g. bands  $\rightarrow$  dbo:Band), if this URI presents in the KB, the type node is linked to this URI. If no entry is found for either an entity or a type afterall, we simply discard the node from our query graph.
Note that although we involve no extra disambiguation step, DBpedia Lookup itself has certain mention-level disambiguation abilities to refine mention-relevant candidates. Admittedly, sentence context also contributes to a precise linking decision, leaving such context-level disambiguation a future direction to improve our work.
B Training Details of the Candidate Ranking Model
We mainly follow NAMER (Zhang et al., 2021) in training the candidate ranking model mentioned in Section 4. Basically, the positive and negative training samples are obtained from the gold query. For instance, in Figure 1, we obtain the candidates between ?class ( $m_1$ ="class") and ?person ( $m_2$ ="person") by constructing "select ?r { ?person dbp: type dbo: person. ?class ?r ?person. dbr: Swinhoe's_Crake dbp: class ?class}" and "select ?r { ?person dbp: type dbo: person. ?person ?r ?class. dbr: Swinhoe's_Crake dbp: class ?class}" with query results  $P_p$  and  $P_r$  respectively. Let  $p^* = dbp$ : named_by be the correct predicate, we collect model inputs  $\{(q, m_1, m_2, p^*)\}$  as a positive sample (i.e. of label 1) and  $\{(q, m_1, m_2, p_i) | p_i \in P_{pos} \setminus \{p^*\} \cup \{(q, m_2, m_1, p_i) | p_i \in P_{rev}\}$  as negative samples (i.e. of gold label 0).
Further, we follow the augmentation process in NAMER to learn the effects of mention order on model predictions. Specifically, we add  $\{(q,m_1,m_2,p_i)|p_i\in P_{rev}\backslash \{p^*\} \cup \{(q,m_2,m_1,p_i)|p_i\in P_{pos}\}$  to negative samples when training. With the aforesaid process repeated
$^{2}$ http://wiki.dbpedia.org/projects/dbpedia-lookup
on each query graph edge, we get the full training samples to train a ranking model.
Besides, similar with NAMER, we observe a performance decay when forcibly co-training RE and GSG module, in this regard, we leave RE a separate module alongside GSG in the system. As discussed in NAMER, the different input channels between RE and GSG may result in unequal semantic spaces for the model. Thus, despite the causal association between RE and GSG, we conjecture that the model fails to acquire beneficial causalities between incompatible semantic spaces.
LC-QuAD 1.0 is an English open-domain KBQA dataset widely used to evaluate KBQA systems. With a GPL-3.0 licence, this dataset is intended for training and testing models to answer a question via querying the knowledge base, permitting modifications on the dataset for experiments, which is consistent with the way we use the dataset (annotating node mentions for each data entry, train several models for KBQA on the train data and test the system performance on the test data).