SPAGHETTI: Open-Domain Question Answering from Heterogeneous Data Sources with Retrieval and Semantic Parsing (2024)

Abstract: We introduce SPAGHETTI: Semantic Parsing Augmented Generation for Hybrid English information from Text Tables and Infoboxes, a hybrid question-answering (QA) pipeline that utilizes information from heterogeneous knowledge sources, including knowledge base, text, tables, and infoboxes. Our LLM-augmented approach achieves state-ofthe-art performance on the COMPMIX dataset, the most comprehensive heterogeneous opendomain QA dataset, with 56.5% exact match (EM) rate. More importantly, manual analysis on a sample of the dataset suggests that SPAGHETTI is more than 90% accurate, indicating that EM is no longer suitable for assessing the capabilities of QA systems today.1

heading: 1 Introduction
text: Open-domain question answering (QA) grounded in knowledge corpora has long been an active topic of research in natural language processing (Chen et al., 2017; Wang et al., 2018; Lee et al., 2019; Asai et al., 2020; Izacard and Grave, 2021; Khattab et al., 2021; Asai et al., 2022). With the rise of LLMs, new state of the art has been established with QA separately on free-text documents (Semnani et al., 2023; Jiang et al., 2023; Gao et al., 2023; Khattab et al., 2023), databases (Pourreza and Rafiei, 2023; Nan et al., 2023; Zhang et al., 2023), and graph databases (Xu et al., 2023; Luo et al., 2023; Li et al., 2023).
In practice, we need to fully leverage hybrid data sources. For instance, Wikipedia alone offers a wealth of knowledge through nearly 7M free-text articles; many of these articles contain structured information in tables and infoboxes; Wikidata is a knowledge graph containing over 17 billion triples. This paper investigates how to leverage LLMs to answer questions on all the different types of data.
1Code is available at https://github.com/stanfordoval/WikiChat.
The premise of this paper is that we need hybrid data and we need hybrid access methods. Our main contribution is twofold. First, we introduce a hybrid LLM-based system (Fig. 1), SPAGHETTI, that combines information retrieval with semantic parsing for question answering and achieves SOTA of 56.5% exact match rate on COMPMIX, the most comprehensive open-domain QA dataset on heterogeneous sources. Second, we show via careful evaluation and analysis that measuring the accuracy of LLM-based QA systems with the exact-match metric against hand-annotated answers is obsolete. By using evaluation methods closer to human judgment, we find that SPAGHETTI is more than 90% accurate on COMPMIX, surprisingly higher than traditional methods would suggest.
1663

heading: 2 Related Work
text: TextQA, TableQA. and KBQA have all been individually studied extensively (Zhao et al., 2023a; Lu et al., 2024; Pan et al., 2024, inter alia). However, the task of answering questions from two or more sources, known as heterogeneous QA, is under-studied. Some literature investigate two of the three sources, including those on closed domain (Miller et al., 2016; Chen et al., 2020; Pramanik et al., 2021; Liu et al., 2023; Lei et al., 2023) and open domain (Chen et al., 2021; Zhao et al., 2023b; Han and Gardent, 2023; Ma et al., 2022a, 2023), but very limited existing work experiments on all three.
CONVMIX (Christmann et al., 2022) collected the first conversational QA dataset that requires knowledge from all three heterogeneous sources. Crowdworkers were asked to pick an entity of their interest and find the answer from one of the Wiki sources - Wikidata, Wikipedia text, Wikipedia tables, or Wikipedia infoboxes. Christmann et al. (2023a) later collated the completed conversations to derive the COMPMIX dataset with 9410 selfcontained question-answer pairs.
Oguz et al. (2022), Ma et al. (2022b), and Christmann et al. (2022) proposed pipelines to answer questions from all three sources, by linearizing all structured information and applying text retrieval methods. Christmann et al. (2023a), on the other hand, unifies all the sources by representing all relevant information in a knowledge graph and uses GNN message passing to find the answer. The former gives up the advantage of using formal query languages on structured data, which can support operations such as ranking and averaging. The latter gives up the advantage of the expressiveness and versatility of free-text knowledge representation. Concurrent with our work, Lehmann et al. (2024) adopts another view that breaks down QA solution processes as tool calls and thoughts. They propose a human-like approach that teaches LLMs to gather heterogeneous information by imitating how humans use retrieval tools, which requires human-annotated demonstrations.

heading: 3 SPAGHETTI
text: SPAGHETTI is a hybrid QA pipeline that takes advantage of both structured and unstructured information. We obtain evidence from heterogeneous sources in parallel, including structured knowledge bases, plain text, linearized tables / infoboxes, and
LLM-generated claims that are verified, and gather those evidence to generate the final answer using a few-shot LLM (Fig. 1).

heading: 3.1 Knowledge Base
text: Xu et al. (2023) proposes a semantic parsing framework for Wikidata. By integrating a named entity linker and a fine-tuned LLaMA trained with modified SPARQL, they establish a strong baseline on the WikiWebQuestions dataset. We adopt their methodology as the interface to WikiData.
As noted by Xu et al. (2023), most of the semantic parsing errors are due to the failure of the entity linker model ReFinED (Ayoola et al., 2022). To improve on their approach, we propose a novel entity linking method where we first ask an LLM to detect entity mentions and generate a brief (maximum 10 words) description of each detected entity. We then feed the list of detected entities and descriptions to ReFinED to obtain the corresponding Wikidata entity IDs. Leveraging the world knowledge of an LLM in this fashion provides an additional mechanism to detect entity mentions and provide more context for ReFinED to disambiguate and link entities. Figure 2 illustrates how our entity linking module works. Here, ReFinED alone fails to identify the entity “Academy Award”, but it succeeds with LLM-provided context.

heading: 3.2 Text
text: Retrieval-augmented generation is a common approach for grounding LLMs in textual knowledge sources like Wikipedia. To avoid LLM hallucination, Semnani et al. (2023) proposes the WikiChat pipeline that combines retrieval with verification of
LLM-generated response, achieving significantly higher factual accuracy than GPT-4. We adopt a similar approach when handling text.
We first extract Wikipedia text using Wikiextractor 2. ColBERT (Santhanam et al., 2022) is used to retrieve Wikipedia passages that may answer a given query, and each of the top-k retrieved passages goes through a few-shot LLM summarizer.
As shown in the rightmost path of Figure 1, similar to WikiChat, SPAGHETTI also makes use of the internal factual knowledge of LLMs by first generating a response and then verifying the claims made in the response using retrieved information, retaining only grounded claims.

heading: 3.3 Tables and Infoboxes
text: Most NLP research using Wikipedia simply ignores the embedded tables and infoboxes, as extraction and preprocessing are challenging. With the help of tools such as WikiTextParser 3 and regex matching, we programmatically extract 9 million tables and infoboxes from Wikipedia pages and linearize them so that they can be encoded as a set of ColBERT (Santhanam et al., 2022) index for retrieval. Being linearized, the retrieved item can then be read by LLMs directly.
Below is an example linearized table from the Wikipedia article “Arundhati Roy”:
Fiction ; No.: 1, Title: “The God of Small Things”, Publisher: Flamingo, Year: 1997, ISBN: ISBNT|0-00-655068-1.<tr> No.: 2, Title: “The Ministry of Utmost Happiness”, Publisher: Hamish Hamilton, Year: 2017, ISBN: ISBNT|0241-30397-4 .<tr>
For each table, we include the section title, two preceding sentences, and two succeeding sentences of the table as additional context, if there are any in the current section. Table rows are formatted as “column_name: cell_content, ...” with “<tr>” as the row separator.
Since ColBERT is pretrained with textual passages and not tables, we finetune ColBERT for table retrieval. After retrieval, the retrieved table is then fed into a few-shot LLM to extract information directly relevant to the query.

heading: 3.4 Putting it Together
text: At the final stage, we gather and combine evidence from all sources. The answer from Wikidata is formatted as “Wikidata says the answer to <query> is:
2https://github.com/attardi/wikiextractor 3https://github.com/5j9/wikitextparser
<answer>.” The retrieved text and tables/infoboxes each goes through an LLM summarization prompt, as mentioned earlier, attempting to extract relevant information from each retrieved item. The verified claim(s) from the LLM-generated answer (if any) is also added to the evidence pool.
Finally, all evidence is fed to a few-shot LLM prompt to generate a single answer to the query. In some cases the answer may be contained in more than one information source, and such redundancy can help reduce errors introduced in earlier stages of the pipeline.

heading: 4 Experiments
text: We evaluate SPAGHETTI on the COMPMIX development and test sets, which contain 1680 and 2764 questions respectively.
For querying Wikidata, we use the LLaMA-7B semantic parser from Xu et al. (2023) trained on both WikiWebQuestions and QALD-7 (Usbeck et al., 2017). We use GPT-3.5 as the LLM in our entity linking module.
We experiment with LLaMA-7B, GPT-3.5-turboinstruct, and GPT-44, respectively, as the LLM backbone in all the stages for handling retrieved evidences and for answer generation. We use fewshot prompts for GPT-3.5 and GPT-4, and use the LLaMA model from Semnani et al. (2023), which is distilled from the teacher GPT-4.
To fine-tune the ColBERT table retriever, we obtain training data from the NQ-Tables dataset (Herzig et al., 2021), where each example matches one gold table to a query. For each positive example, we sample 10 negative tables to obtain a total of 95K training triplets. We confirmed on the NQ-Tables dataset that the fine-tuned version improves table retrieval Recall@3 by 10%.
Evaluation Metrics. Bulian et al. (2022) and Kamalloo et al. (2023) have established that exact match (EM) against gold answers, which is commonly used for evaluating QA systems, cannot evaluate generative models properly as they often generate lexically different, but semantically equivalent answers. To properly assess our approach, we introduce two additional evaluation metrics: (1) Superset: whether the gold answer is a substring of the generated answer, as the latter tends to spell out the answer in long form and may include a more
4We access GPT models via the Microsoft Azure OpenAI API. We use the GPT-4 snapshot from June 13th, 2023.
complete answer. (2) GPT-4 Matching: using GPT4 with a few-shot prompt to determine whether the generated answer matches the gold, similar to Kamalloo et al. (2023).
Moreover, datasets may have ambiguous queries or even wrong annotations. To assess the quality of COMPMIX, we sample 100 questions and carefully use online information sources to find the answers and decide if the generated answers are correct. We refer to this metric as platinum evaluation. GPT-4 for answer matching eliminates formatting issues, providing a result closer to human judgment than EM, while platinum evaluation further considers annotation issues and eliminates errors caused by ambiguous queries.

heading: 5 Results
text: SPAGHETTI (GPT-4) achieves 56.5% EM rate on the test set of COMPMIX, improving on the previously reported state-of-the-art (GPT-3) by 6.3%, and improves on the GPT-4 baseline by 3.7% (Table 1). SPAGHETTI (GPT-3.5) also improves upon all of the baselines. We note that the EM scores of the GPT-3.5-turbo-instruct baseline are low because this model tends to be more verbose.
SPAGHETTI (GPT-4) achieves 81.9% test set accuracy by GPT-4 matching, and 92% platinum accuracy on the 100 development set examples. Of the 8 errors cases, 3 have unanswerable questions (e.g. “FC Cincinnati soccer club?”), thus the true accuracy rate is 92/97 (94%).
Ablations. We evaluate the contribution of each knowledge source by ablating different parts of the system (Table 2). Using text alone already outperforms the previous SOTA, with each additional source further improving the result. Note that for many questions, information exists in multiple sources; the relatively little contribution from Wikidata and tables reflects mainly on the makeup of COMPMIX, not their value as knowledge sources. For detailed experimental results on our Wikidata entity linking approach, see Appendix A.
Human Evaluation We examine how our human “Platinum” evaluation (92%) differs from the EM metric (60%) on our sample of 100 cases. We report findings on the SPAGHETTI (GPT-4) responses, specifically in 32 annotated examples where EM fails but the response is indeed correct (Figure 3). Out of the 32 discrepancies, the unsophisticated “Superset” metric resolves 7, and GPT-4 matching resolves an additional 14. Platinum evaluation identifies that 4 questions have incorrect gold labels, and 7 questions are ambiguous and the generated answers are correct though different from the gold. We include examples for each of these resolved cases in Appendix B.
Of the 5 true errors, one is because SPAGHETTI cannot find the answer in any of the four information sources; in the other 4 cases, the answer generator cannot identify the correct answer re-
trieved due to conflicting or misleading evidence. See Appendix C for details on each error case.

heading: 6 Conclusion
text: We propose SPAGHETTI, a hybrid open-domain question-answering system that combines semantic parsing and information retrieval to handle structured and unstructured data.
SPAGHETTI achieves an exact match rate improvement of 6.3% over the prior state-of-the-art on the COMPMIX dataset. More importantly, we show that our approach is likely to reach an accuracy of over 90%, if we account for differences in the answer wording and incompleteness/errors in gold labels. This, however, does not mean opendomain QA is solved. Further research is needed to handle open-domain questions that require complex structured queries or composition of answers from multiple information sources, none of which are included in COMPMIX.

heading: Limitations
text: This work focuses specifically on open-domain QA with heterogeneous knowledge sources, and we only report results on the COMPMIX dataset due to the limited availability of high-quality datasets in this domain. A natural future work is to develop more diverse and advanced datasets that further push the need to utilize each knowledge source.
We evaluate on single-turn QA and do not work with conversations in this paper, and SPAGHETTI can be extended to handle fact-based conversational questions or even chitchat that involves facts.
We have a relatively small sample size for human evaluation, because the expert manually checks the correctness of each example with Internet searches,
which is labor-intensive. However, we acknowledge that a larger sample size would increase the statistical confidence of our evaluation.
Finally, we note that a number of Wikipedia tables are not well-formatted after preprocessing and linearization. Since Wikipedia tables are embedded as HTML elements that allow for idiosyncrasies like a table with one cell spanning multiple columns or color-highlighted cells, some are hard to parse correctly. Solving such edge cases engineering-wise would further improve TableQA.

heading: Ethical Considerations
text: To facilitate reproducibility and continued research, we will make the code available upon publication.
No new datasets were gathered specifically for this study, and we did not employ crowd-sourced labor. We use Wikipedia data under the terms of the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA) and the GNU Free Documentation License (GFDL). Wikidata is under Creative Commons CC0 License, which is equivalent to public domain. The COMPMIX benchmark is licensed under a Creative Commons Attribution 4.0 International License. We use the benchmark as it is intended.
The experimental phase involved approximately 80 hours of computation time on an NVIDIA A100 GPU to fine-tune the retrieval model and index Wikipedia content. We reused the LLaMA-7B model trained in prior work, thus avoiding extra GPU usage.
We do not anticipate adverse effects stemming from the proposed methods in this study.

heading: Transactions of the Association for Computational
text: Linguistics (TACL).
Fangyu Lei, Xiang Li, Yifan Wei, Shizhu He, Yiming Huang, Jun Zhao, and Kang Liu. 2023. S3HQA: A three-stage approach for multi-hop text-table hybrid question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1731– 1740, Toronto, Canada. Association for Computational Linguistics.
Tianle Li, Xueguang Ma, Alex Zhuang, Yu Gu, Yu Su, and Wenhu Chen. 2023. Few-shot in-context learning on knowledge base question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6966–6980, Toronto, Canada. Association for Computational Linguistics.
Shicheng Liu, Jialiang Xu, Wesley Tjangnaka, Sina J. Semnani, Chen Jie Yu, Gui Dávid, and Monica S. Lam. 2023. SUQL: Conversational search over structured and unstructured data with large language models.
Weizheng Lu, Jiaming Zhang, Jing Zhang, and Yueguo Chen. 2024. Large language model for table processing: A survey.
Haoran Luo, Haihong E, Zichen Tang, Shiyao Peng, Yikai Guo, Wentai Zhang, Chenghao Ma, Guanting Dong, Meina Song, and Wei Lin. 2023. Chatkbqa: A generate-then-retrieve framework for knowledge base question answering with fine-tuned large language models.
Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2022a. Open-domain question answering via chain of reasoning over heterogeneous knowledge. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5360– 5374, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2022b. Open domain question answering with a unified knowledge interface. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1605–1620, Dublin, Ireland. Association for Computational Linguistics.
Kaixin Ma, Hao Cheng, Yu Zhang, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2023. Chain-of-skills: A configurable model for open-domain question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1599–1618, Toronto, Canada. Association for Computational Linguistics.
Alexander Miller, Adam Fisch, Jesse Dodge, AmirHossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language
Processing, pages 1400–1409, Austin, Texas. Association for Computational Linguistics.
Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. 2023. Enhancing text-to-SQL capabilities of large language models: A study on prompt design strategies. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14935–14956, Singapore. Association for Computational Linguistics.
Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2022. UniK-QA: Unified representations of structured and unstructured knowledge for open-domain question answering. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1535–1546, Seattle, United States. Association for Computational Linguistics.
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2024. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering, page 1–20.
Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of textto-sql with self-correction.
Soumajit Pramanik, Jesujoba Alabi, Rishiraj Saha Roy, and Gerhard Weikum. 2021. UNIQORN: unified question answering over RDF knowledge graphs and natural language text. CoRR, abs/2108.08614.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3715–3734, Seattle, United States. Association for Computational Linguistics.
Sina Semnani, Violet Yao, Heidi Zhang, and Monica Lam. 2023. WikiChat: Stopping the hallucination of large language model chatbots by few-shot grounding on Wikipedia. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2387–2413, Singapore. Association for Computational Linguistics.
Ricardo Usbeck, Axel-Cyrille Ngonga Ngomo, Bastian Haarmann, Anastasia Krithara, Michael Röder, and Giulio Napolitano. 2017. 7th open challenge on question answering over linked data (qald-7). In Semantic web evaluation challenge, pages 59–69. Springer.
Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018. R^3: Reinforced ranker-reader for open-domain question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
Silei Xu, Shicheng Liu, Theo Culhane, Elizaveta Pertseva, Meng-Hsi Wu, Sina Semnani, and Monica Lam. 2023. Fine-tuned LLMs know more, hallucinate less with few-shot sequence-to-sequence semantic parsing over Wikidata. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5778–5791, Singapore. Association for Computational Linguistics.
Yunjia Zhang, Jordan Henkel, Avrilia Floratou, Joyce Cahoon, Shaleen Deep, and Jignesh M. Patel. 2023. Reactable: Enhancing react for table question answering.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023a. A survey of large language models.
Wenting Zhao, Ye Liu, Tong Niu, Yao Wan, Philip S. Yu, Shafiq Joty, Yingbo Zhou, and Semih Yavuz. 2023b. Divknowqa: Assessing the reasoning ability of llms via open-domain question answering over knowledge base and text.

heading: A Wikidata Experiments
text: Xu et al. (2023) fine-tuned two LLaMAs on Wikidata. The training data for the first model consists solely of WikiWebQuestions (Xu et al., 2023), while the other consists of the combination of WikiWebQuestions (Xu et al., 2023) and QALD-7 (Usbeck et al., 2017). We experiment with both models on the development set of COMPMIX, each with (1) entities predicted by ReFinED, (2) our entity linking approach with GPT-3.5 as the LLM (prompt in Figure 6), and (3) the dataset-provided oracle entities.
As shown in Table 3, the model using entities predicted by our approach outperforms the model using the baseline ReFinED entities. It achieves considerably closer performance with the model using oracle entities. We also observed that the the model trained on both WikiWebQuestions and QALD-7 outperforms the model trained on WikiWebQuestions only.

heading: B Details on Platinum Evaluation
text: Figure 3 shows the distribution of cases that we resolve using more advanced evaluation metrics. Numbers are reported on the first 100 dev examples with SPAGHETTI (GPT-4).
Examples of each evaluation error type can be found at Figure 12, Figure 13, Figure 14, Figure 15, and Figure 16.

heading: C Error Analysis
text: We include the five error cases after platinum evaluation in Figure 7, Figure 8, Figure 9, Figure 10, and Figure 11.

heading: C.1 Conflicting or Misleading Evidence
text: We analyze the 388 error cases from SPAGHETTI (GPT-3.5) as determined by GPT-4 Matching. We separate out evidence retrieval errors from answer generation errors by identifying how often the gold answer appears in the evidences using a substring matching heuristic (Table 4).
In 154 out of all 388 error cases, the system does not produce the gold answer despite the successful retrieval of evidence containing it. This observation indicates that a significant portion of the error cases are due to conflicting or misleading information in the evidence, where further improvements in selecting and merging evidences would be helpful. In the majority of the error cases (234 out of 388) where gold is not in the evidence, the system has
no high-quality candidates to select from. Note, however, that this is an overestimate, due to the use of substring matching for deciding whether an evidence is correct or not.
In the breakdown of gold answer sources, the source that contains the most gold answers is Text (87 out of 154 cases), and Wikidata contains the least gold answers (51 out of 154 cases).

heading: C.2 Combiner Hallucination
text: We investigate the ratio of generated answers that were hallucinated by our model. We manually checked the first 300 cases in our evaluation set and found 2 cases (0.67%) where the model ignored the evidence and hallucinated an incorrect answer. This low ratio of hallucination highlights the faithfulness of our system to the evidence retrieved. We include these cases in Figure 4 and Figure 5.

heading: Dev Dev (KB subset)

heading: Future Work
text: A natural future work is to develop more diverse and advanced datasets that further push the need to utilize each knowledge source. We evaluate on single-turn QA and do not work with conversations in this paper, and SPAGHETTI can be extended to handle fact-based conversational questions or even chitchat that involves facts. We have a relatively small sample size for human evaluation, because the expert manually checks the correctness of each example with Internet searches,
which is labor-intensive. However, we acknowledge that a larger sample size would increase the statistical confidence of our evaluation. Finally, we note that a number of Wikipedia tables are not well-formatted after preprocessing and linearization. Since Wikipedia tables are embedded as HTML elements that allow for idiosyncrasies like a table with one cell spanning multiple columns or color-highlighted cells, some are hard to parse correctly. Solving such edge cases engineering-wise would further improve TableQA