OpenTab: Advancing Large Language Models as Open-domain Table Reasoners
Authors: Kezhi Kong
Jiani Zhang
Zhengyuan Shen
Balasubramaniam Srinivasan
Chuan Lei
Christos Faloutsos
Huzefa Rangwala
George Karypis
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Table Reasoning
Primary area: generative models

Abstract: Large Language Models (LLMs) trained on large volumes of data excel at various natural language tasks, but they cannot handle tasks requiring knowledge that has not been trained on previously. One solution is to use a retriever that fetches relevant information to expand LLM's knowledge scope. However, existing textual-oriented retrieval-based LLMs are not ideal on structured table data due to diversified data modalities and large table sizes. In this work, we propose OpenTab, an open-domain table reasoning framework powered by LLMs. Overall, OpenTab leverages table retriever to fetch relevant tables and then generates SQL programs to parse the retrieved tables efficiently. Utilizing the intermediate data derived from the SQL executions, it conducts grounded inference to produce accurate response. Extensive experimental evaluation shows that OpenTab significantly outperforms baselines in both open- and closed-domain settings, achieving up to 21.5% higher accuracy. We further run ablation studies to validate the efficacy of our proposed designs of the system.

Published as a conference paper at ICLR 2024
OPENTAB: ADVANCING LARGE LANGUAGE MODELS AS OPEN-DOMAIN TABLE REASONERS
Kezhi Kong Jiani Zhang Zhengyuan Shen?
Balasubramaniam Srinivasan"
Chuan Lei² Christos Faloutsos Huzefa Rangwala* George Karypis²
'University of Maryland, College Park
Amaazon Web Services
kongêcs, und. edu {zhajiani, donshen, srbalasu] amazon. com {chuanlei, faloutso, rhuzefa, gkarypis (emppiss@aaaaaaaa @amazon. com
ABSTRACT
Large Language Models (LLMs) trained on large volumes of data excel ious natural language tasks, but they cannot handle tasks requiring knowledge that has not been trained previously. One solution is use a retriever that fetches relevant information to expand LLM's knowledge scope. However, ex- isting textual-oriented retrieval-based LLMs are not ideal structured table data due to diversified data modalities and large table sizes. In this work, we propose OPENTAB, an open-domain table reasoning framework powered by LLMs. Over- all. OPENTAB leverages table retriever to fetch relevant tables and then generates SQL programs to parse the retrieved tables efficiently. Utilizing the intermediate data derived from the SQL executions, it conducts grounded inference to produce accurate response. Extensive experimental evaluation shows that OPENTAB sig- nificantly outperforms baselines in both open- and closed-domain settings, achiev- ing up to 21.5% higher accuracy. We further run ablation studies to validate the efficacy of our proposed designs of the system.
INTRODUCTION
The field of natural language processing has rapidly advanced tasks of text generation (Brown et al. (2200) and knowledge reasoning (Huang & Chang) 2023). driven by general-purpose large- scale generative models like GPT4 (OpenAl]]022)
However, these generative models have inher- ent drawbacks, like hallucinations (Ye & Durrett, 2022) and limited specialized knowledge. They cannot be used to perform tasks involving data that has not been trained on, like answering questions about recent events about proprietary enterprise data. Recent advancements in retrieval-based methods (Asai et al. 2023) that aim to equip generative models with expanded information, allows for grounded responses based on real-time or proprictary data.
Many retrieval-basee -based LLMs primarily target textual data from the web or text corpora, overlooking the wealth of information stored structured tables. When applied the tables, retrieval-based LLMs encounter the following challenges: (I) Structured tables have diverse data types, notably numerical data presented in large or precise numbers, which results in high token usage and po- tentially leads to memory and computational constraints for the model. (2) LLMs, mainly those optimized for natural language understanding, fail to comprehend the complex relationships within tabular data for effective data transformation and answer extraction. (3) The restricted maximum context length of LLMs poses difficulties managing a multitude of retrieved tables of varying sizes, particularly when dealing with tables containing millions of rows.
This work aims to develop an LLM- -bsed framework for the open-domain table reasoning task (Herzig et al. [021]] Kweon al. 2023). Here, concentrate on two specific tasks: table- based question answering (QA) and fact verification. For both tasks, the framework is presented
"Work conducted during an internship at Amazon Corresponding author Huzefa Rangwala is on LOA as Professor of Computer Science at George Mason University. This paper describes work performed Amazon.

---

Published as a conference paper at ICLR 2024
Query
Response
Fabrice Sentore wit more the Australian open Wimbbeddd
Fabrica Sentaru wan
LLM-Anower
BINDER
more at Australaan open
LLM-SQL
Openlab
Table corpus
RETRIFVER
REASONER
SQL
Accuracy (%6
40
Cooth
READER
30
81
18
Exccution
5gg. &
271
10
ROWSELFCTOR
top-1
top-2
top-5 top-10
Number of retrieved tables
Figure 1: An overview of OPENTAB pipeline. OPENTAB uses a RETRIEVER to retrieve relevant sampled tables from a corpus of tables for a given natural language query, and then use REASONER to output natural language response.
Figure 2: The accuracy with an in- creasing numbers of retrieved ta- bles. OPENTAB has consistent in- crease in accuracy with more tables.
with natural language queries and corpus of tables. In the open-domain setting, the gold tables, which serve as the evidence for queries to generate responses, are not provided. Therefore, the chal- lenge lies in automatically identifying and retrieving pertinent knowledge from the table corpus and also formulating coherent natural language response. A task example of table QA is in Figure !
Existing LLM-based table reasoning methods, like BINDER (Cheng et al. 2023) and DATER YYe et al., (023), operate in the closed-domain setting, in which table retrieval is not required and pre-defined gold table is available for each query during test time. While this assumption simplifies the reasoning process, it does not hold in most real-world scenarios manually providing gold grounding information is costly and impractical. In the open-domain context, predictions must be grounded in selection of relevant tables, and the system should possess the reasoning capabilities to sift through incorrect tables (Liu et al.) 2023b) to derive the correct output. Mcanwhile, existing open-domain table reasoning systems (Chen et al. 2021b] Karpukhin et al. 20200] Chen al. 2022) have a fine-tuned encoder for retrieval and a fine-tuned table reasoner (Herzig et al. 2020) for QA as well, which necessitates specialized training for specific tasks and less flexible. Moreover, although they perform well on observed data after fine-tuning, the performance drops on new tables due to constrained transferability.
We propose OPENTAB, an open-domain & end-to-end table reasoning framework. As illustrated in [ OPENTAB leverages RETRIEVER to fetch relevant tables, generates programs from a CODER intermediary reasoning steps, and delegates the final solution to READER. Following thorough empirical assessment, we opt to implement the BM25 (Robertson et al.] 2009a) algo- rithm the table RETRIEVER because of its scalability and effectiveness. To mitigate challenges of LLMs in processing and understanding structured tables, we utilize an LLM as a CODER to generate high-quality SQL programs for efficient table parsing. Moreover, an LLM-based READER works to formulate the final response based on the SQL execution results. By breaking down the complex rea- soning job into programmatic steps, we achieve enhanced accuracy and reliability in open-domain table reasoning. Notably, the CODER and the READER modules are guided by few-shot prompting, without relying on training or fine-tuning. Our approach has several key advantages:
High Accuracy. Open-domain table reasoning presents a challenge due to the trade-off between retrieval recall and precision. Higher recall/coverage requires retrieving more tables, but this might introduce noise from less pertinent tables and therefore reduces precision and prediction accuracy.
Specifically, we propose a reranking strategy called Generative Reranking & Sequential Reason- ing to prioritize the tables with higher similarity between the natural language query and the corre- sponding generated SQL programs to address the trade-off, which enhances prediction accuracy as shown in Figure 2
Scalability. Our approach can efficiently handle large quantities and sizes of tables. The framework is capable of generating accurate responses with access only a limited number of rows from each table effectively extracted by ROWSELECTOR.
Robustness. We propose simple-to-compee prompting, a flexible and robust progressive program generation and execution strategy. By sequentially generating SQL programs with increasing levels of complexity, starting from basic column selection to advanced operations like aggregation and
2

---

Published as a conference paper at ICLR 2024
text operations, the model explores wider range of possible solutions. This approach offers more adaptability in finding the optimal SQL query for a given task, and it also serves as a structured reasoning process for generating effective SQL code. Additionally, it allows for easy fallback to simpler queries if the more complex ones fail, enhancing the overall robustness the system.
Effectiveness. We provide extensive experimental results showing that method registers com- petitive performances without any fine-tuning on the target dataset. We show that OPENTAB sig- nificantly outperforms baselines in both open- and close-domain settings. For example OPENTAB outperforms the best baseline with a remarkable 21.5% higher accuracy under the open-domain scenario. We further run detailed ablation studies to validate the efficacy of our proposed designs.
DEFINITION OF
TABLE REASONING
We formally define the open-domain table reasoning task follows: Given natural language input query and a database with tables T, the task is to generate a response a based on multiple retrieved tables, denoting as Tg C T. while the gold table set for query is denoted as Ty. The form of the (g, a) pair is determined by the corresponding downstream task. this work, we focus on following two tasks:
Question answering, where is a natural language question and a is a natural language answer. E.g. qF "Which team won the NBA championship in 2022?", a: "Golden State Warriors."
Fact verification, where is statement and is a judgement given T, like entailment or contra- diction. E.g., q: "In àlex corretja career there be no game in may 1998", a:: "Entailed".
We approach the task modeling P(alg, 7). Technically, T be large collection of tables, it is intractable to directly solve P(alg. T). Thus, we follow the two-step approach that is often adopted in natural language retrieval augmented models (Borgeaud et al.) 2022). Namely, given query q, we firstly retrieve set of evidence tables T, C T, and subsequently model P(alg, T₂) in place of P(alg, 7). In this work, we expect to leverage LLMs as the backbone for modeling given their capability in sew-shot inference and reasoning (Chen 2023).
Note that the key difference between open open-domain and closed-domain table reasoning task is that in the closed- -doeddddomain setting the tables Ty are explicitly provided for each query g. In contrast, in the open-domain setting, there is possibility that Ty nTa 0 depending on the ability of the retriever.
3 METHOD
Overview. We present an overview of OPENTAB in Figure To enable our pipeline to automatically handle large-scale tabular data in terms of both table size (number of rows and columns) and table quantity, we divide the overall system into two components: a RETRIEVER (in Section 3.1 and a REASONER (in Section 3.2). For RETRIEVER, we advocate the BM25 for table retrieval tasks as it provides scalable and competitive retrieval performance. For the REASONER, we leverage the CODER powered by LLM to generate SQL queries based on retrieved tables Tg. The final response is then extracted by an LLM-based READER module to ensure the accuracy, efficiency, and robustness of the REASONER against generation stochasticity. In Section 3.3, we introduce a Generative Reranking & Sequential Reasoning (GRSR) strategy for open-domain reasoning, which sequentially generates SQLs for a set of retrieved tables and then reranks them based on query similarity, effectively addressing the hallucination issues.
3.1 TABLE RETRIEVER
Efffective table retrieval in the open-domain setting Wang et al. 2022 Kweon et al. 2023) remains an open question (Ren et al. 2022). We select BM25 (Robertson al. 2009b) as the table retriever in our framework for the following considerations. First, BM25, probabilistic- based ranking func- tion used information retrieval systems to rank documents, is a standard sparse retrieval method that works efficiently on large corpora, considering term frequency and inverse document frequency
Implementation hased bttp:://gthha..cmm/ddaaaanwwowne hm255

---

Published as a conference paper at ICLR 2024
Text trunsformation ranking.
Jose
hahal
Eustrolion
open
0ushroo spen
22
oustrid on
0/34
12:34
wimbledon
D/14
11
wirbrrddr
0/14
11
11
US open
open
spcn
indin wells
0/13
16 13
Indlan well
0/13
16 13
indion wells
0/13
16-13
Basic
Intermedicte
Advanced
Figure 3: Examples illustrating the progressive Simple-to-complex SQL proficiency. The basic SQL queries mainly select specific columns. The intermediate incorporates both column and row selec- tion. The advanced SQL employs additional operations like aggregation and text operations that can manipulate and transform the tabular data. The cells blue are outputs the SQL programs.
with saturation. Second, is easy use without the necessary fine-tuning process Dense Passage Retrieval (DPR) (Karpukhin et al. 2020). Third, empirically, show that BM25 achieves compet- itive retrieval performance compared with dense methods such as DPR in Section 4.3 Given these, BM25 RETRIEVER together with REASONER powered LLMs make the OPENTAB system off-the-shelf usable without the necessity of fine-tuning on target databases, which a significant advantage for industrial applications.
3.2 TABLE REASONER
The REASONER processes natural language queries along with table schema and sampled rows, fa- cilitated by CODER and ROWSELECTOR. The ROWSELECTO ECTER ensures that the pertinent rows provided to the LLM for effective processing. The CODER generates SQL programs of increas- ing complexity. Finally, the READER leverages an LLM to parse and extract the final response.
3.2.1 CODER
On the table reasoning task, LLMs (e.g,, ChatGPT and Falcon) are capable of generating SQL queries that can be executed on databases extract the final answer (Chen al.) [02 Liu al.] 2023a Sun et al., 2023). This approach is referred to as the LLM-SQL baseline. However, the generated SQLs towards direct question answering can be less robust with both syntax and semantic errors during execution when facing complex reasoning tasks (Cheng et al.) 2023). which might
require multiple operations such as aggregation, comparison, and sorting (Kweon et al. 2023).
In CODER, propose new himple-to-comple prompting strategy for effective SQL generation. Specifically, for each input query. we prompt the LLM to sequentially generate three SQL programs with ascending complexities and functionalities, which are:
SQL-busic: focusing column selection, which sets the groundwork for understanding how to fetch specific data from a database.
SQL-intermediate: incorporating both column and row selection. This means extracting particular columns and filtering rows based on specified criteria, enhancing precision in data gathering.
SQL-advanced: empowering additional operations including but not limited aggregation func- tions and text operations. Aggregation empowers data summarization, while text operations facil- itate the manipulation and transformation of string data.
Figure 3 shows the high-level idea of this cimple-to-comppe prompting strategy. The reason for such a design is that generating solid SQLs can be challenging for complex input queries, so we incre- mentally generate SQLs in reasoning steps from simple to complex. Moreover, with the READER module (explained later), the generated SQL programs do not necessarily need to produce the fi- nal answer directly. This is infeasible when the input query is not solvable by pure SQL programs (Cheng et al. 2023). Therefore, SQL-basic and itn-intereeiiae can not only function as reason- ing steps towards the ultimate solid SQI-advanced but also lead final correct response.
4

---

Published as a conference paper at ICLR 2024
This provides more flexibility and robustness than generating single SQL. After the generations of three SQL programs, we first test SQL-advanced by execution as verification. We proceed to the READER module for final response extraction if it returns valid non-empty results. If the SQL fails the verification, we turn to verify simpler ones until all SQLs are exhausted. Figure 5 displays the prompt and generation structure of CODER. For concrete problem solving example, see Figure
3.2.2 READER
To expand the limited capability of SQL in solving natural input queries, we use READER mod- ule that leverages LLM to digest the intermediate SQL execution results and formulate the final response. Rather than providing only the execution results, we further supply the READER with broader context from CODER, including table schema, sampled rows, and generated SQL query. This enables the READER to better understand the contextual background and semanties needed for accurate predictions. We opt to provide READER with these additional contexts based on analysis showing poor performance when only giving the execution results without exposure to the richer contextual information from CODER (see ablation studies). As shown in Figure [5] our framework prompts the READER with this broader context encapsulating the entire SQL generation process.
3.2.3 ROWSELECTOR
Besides table schema, table contents are also critical for effective SQL generation. Idcally, the entire table should be input to the L.M for the complete view. However, LLMS can only handle tables with large sizes within their token capacity limits. In order to balance scalability and reasoning ability, we propose to use the RowSELECTOR be placed into the prompt of LLMs. Technically, we leverage BM25 to rerank and choose the top-k rows based on relevance to q.
harness few rows that are most relevant the input g to
3.3 OPEN-DOMAIN TABLE REASONING
In contrast a closed-domain setting where the table used for reasoning is provided, open-domain reasoning operates over a much larger and unconstrained table store, causing an inherent trade-off' between precision and recall of the retrieved tables. More tables need to be retrieved to achieve higher coverage (i.e., recall values), which inevitably brings irrelevant information and, thus, lower precision. Therefore, precisely identifying the correct table from the retrieved set is crucial for the pipeline to achieve accurate reasoning performance over the end-to-end open-domain task.
We propose novel Generative Reranking & Sequential Reasoning (GRSR) strategy to address this pain point. Specifically, given a query q and T of k tables fetched by RETRIEVER, SQLs sequentially generated for each table using CODER. We then rerank the tables based on the similarity between / and the generated SQL computed using pretrained Cross Encoder transformers (Reimers & Gurevych| 2019) P The tables with the highest resultant similarity scores computed by corresponding SQLs are then selected for downstream predictions. GRSR is effective because of its ability to combat the hallucination tendency of LLMs. Ideally, CODER should only be able to generate valid SQL programs if gold set Ty is retrieved, such that the execution result of the SQL can be used to locate the gold tables. However. in practice, we find that CODER likely generate valid SQL that can pass the execution verification even though the retrieved table is irrelevant from the query q. This hallucination problem greatly limits the expected effectiveness of generative feedback from L.MS toward retrieved tables. Given these, GRSR evades noisy retrieved tables by detecting hallucinated SQLs using a pretrained Cross Encoder. With more tables retrieved, we expect to higher recall value but also likely with precision degradation. GRSR effectively counters such negativity introduced by more tables retrieved while leveraging the advantage of higher recall by more precisely locating the gold table and ignoring the noisy ones.
To better reveal the efficacy of the GRSR strategy, we introduce two baseline algorithms for com- parison. Joint Reasoning approach, where all retrieved tables are aggregated into the prompt of CODER for SQL generation and prediction simultaneously. Sequential Reasoning only, which is basic sequential approach that verifies the validity of SQLs in the order defined by the retriever. If we reach a valid SQL without an execution error, we select the corresponding SQL. By default,
Huggingface checkpoint with model name cross cross-encodere www.eeooddddeemeearraooo Mi 1M-1.-12-v
5

---

Published as a conference paper at ICLR 2024
Table 1:
The accuracy on two open-domain table reasoning tasks using two different backbone LLMs. The results are presented at the top- top-1/2/5/10 retrieved tables.
gpt-3 3. 5-turbo
faicon- 180B
Method
Top-1
Top-2
Top-5
Top-10
Top-1
Top-2
Top-5
Top-10
Open- WikiTables
Open- WikiTables
LLM-Answer
0.273
0.102
0.047
0.008
0,203
0.047
0,016
0,008
LLM-SQI.
0.336
0.313
0.250
0.109
0.195
0.094
0.023
0.008
BINDER
0.234
0.305
0.375
0.375
0.164
0.188
0.227
0.227
OPENTAB
0.491
0.523
0.556
0.565
0.375
0.391
0.437
0.437
FEVEROUS
FEVEROUS
LLM-Answer
0.672
0.508
0.492
0.453
0.680
0.476
0.375
0.289
LLM-SQL.
0.609
0.460
0.515
0.422
0.328
0.242
0.195
0.094
BINDER 0.375
0,383
0.391
0.391
0,219
0.266
0.305
0,281
OPENTAB 0.695
0.508
0.563
0.508
0.742
0.515
0.555
0.484
we leverage GRSR in OPENTAB. the ablation section, we show the performance gain of GRSR over Sequential Reasoning and Joint Reasoning, justifying the efficacy of GRSR in mitigating the hallucination problem and locating the gold table.
EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets. To evaluate the proposed approach, we use Open-Wikelable (Kweon et al. 2023), Wik- iTableQuestions (Pasupat & Liang 2013). and FEVEROUS (Aly et al. (021) datasets. Open- WikiTable is an open-domain table question-answering dataset, where the specific table related to given question is not provided and must be retrieved. Experiments were limited by budget to 2,000 random samples from the validation set. The corpus contains 24,680 candidate tables. In contrast, WikiTableQuestions. also a table-based QA dataset, provides the relevant table containing the nec- essary information, making dataset that requires the identification of multiple relevant tables for reasoning and verifying factual questions. We adapted the dataset for open-domain table reasoning by filtering 323 examples from the validation set that rely solely table data, using a corpus of 26, 177 candidate tables from the FEVEROUS dataset. line with BINDER (Cheng et al., 2023), we use execution accuracy (EA) primary metric for evaluating performance on the Open-WikiTane WikiTable and WikiTableQuestions datasets. For FEVEROUS we adopt a metric that demands the pipeline not only to make the correct predictions but also correctly identify the gold evidence table(s) for making those predictions. We report both metrics as the accuracy term in the following descriptions. See details in Appendix AA.3)
FEVEROUS, on the other hand,
verification
Baselines. For table retrieval experiments, of BM25 and DPR, where DPR uses BERT model both with and without being fine-tuned on the target dataset. In the context of the table reasoning task, compare with BINDER (Cheng al., 2023), an end-to-end table QA model that generates specialized symbolic languages to be executed on the table database. As the original LLM model used in BINDER (OpenAI Codex) no longer available, we reproduce BINDER'S ' results using gpt 3 5 turbo with BINDER's 's official im- plementation. The official implementation of BINDER does not support the open-domain scenario; and the database interface cannot support identification of multiple relevant tables. We leverage the straightforward Sequential Reasoning (SR) strategy to reduce the multi-table scenario to single-table to make BINDER applicable in the open- open-domain setting. Furthermore, we implement two additional baselines: (1) LLM-An to prompt LLM to directly output the text-format answer based on the table and question; (2) LLM-SQL, to let LLM respond with SQL programs whose execution result will be the output. If not specified, the LLM backbone used in this work is gpt-3. 5-turbo (default 4oken version), and the in-context learning examples are 2-shot.
conduct experiments compare the performances
Huggingface checkpoint with model name bewt-base--eeee -base-uncased

---

Published as a conference paper at ICLR 2024
Table The accuracy on the closed-domain table QA
task (VikiTaableQuestions))
Table
The accuracy results on the
Method
Acc
closed-domain table QA task (Open- WikiTable).
T3-3B Xie et al.
2022
0.519
Tapex
Liu el al.
2022
0.604
Fine-tuned
TaCube (Zhou el al. 2022
0.611
Method
Acc
Omni'abb (nang et
2022
0.633
LLM-Answer
0.378
LLM-Ans
0.375
LLM-SQL.
0.579
LLM-SQL
0.414
BINDER (Cheng
2023)
0.421
w/o Fine-tuning
BINDER (Cheng 2023
0.428
OPENTAB
0.710
DATER eY al. 2023
0.453
OPENTAB
0.641
4.2 MAIN RESULTS
Open Domain. For this setting, we first retrieve top-k relevant tables Tg from the database us- ing the same BM25 algorithm. We run extensive experiments with different LLM backbones (gpt-3.5-tum & falcon-180b) on two datasets Sopen-WikiTables & FEVEROUS), Ta- ble summarizes the experimental results. From Table [ cantly outperforms baselines on the Open-WikiTables dataset. For example with spt-3. 3
that OPENTAB signifi-
OPENTAB outperforms the best baseline with remarkable 21.5% higher accuracy under the top- 10 scenario. Meanwhile on the FEVEROUS dataset, OPENTAB can mostly win over the baselines with non-trivial improvements. Note that the baseline methods require full tables be fed into LLM for reasoning, which may exceed the upper bound of the input token limit, rendering an invalid predic- tion. While OPENTAB does suffer from scalability issues CODER is capable of generating high-quality SQLs with three selected representative rows. Moreover, to reproduce BINDER' 's per- formance, we follow Cheng et al. (2023) to use 14-shot examples when doing in-context learning- For BINDER, we also deploy gpt 3. 5.55-t Stuurrb---- 11---666 6k to fit the long input sequences. This group of experiments shows that OPENTAB is the more capable system at dealing with the challenging open-domain table reasoning tasks.
Closed Domain. To complement the empirical investigation, we further conduct experiments under the closed-domain scenario. where the golden evidence table is directly given without the retrieval requirements. We show the performances in Table on the Open- WikiQuestion dataset. Similarly, to make BINDER applicable,
gpt-3. 5-turbo-111 6k with its official 14-shot examples. We can see that OPENTAB has a non-marginal 13% improvements over the baselines, further validating the efficacy of OPENTAB designs. We further evaluated the WikiTableeQestionns dataset (Pasupat & Liang 2015), which is intrinsically a closed-domain table QA dataset. Results are shown in Table It reveals the competitiveness of OPENTAB of even outperforming fine-tuned methods.
4.3 TABLE RETRIEVAL RESULTS
We report the table retrieval performance in Table Results on the Open- WikiTable dataset from both Kweon et al. 2023) and our implementations. BM25 has competitive performances compared the DPR, which has been fine-tuned the labeled training dataset. On the FEVEROUS dataset, BM25 even outperforms fine-tuned BERT, which supports our intuition that sparse retrievers are simple but powerful table retrievers.
ABLATION STUDIES & ANALYSIS
Ablation studies on the proposed modules of OPENTAB. Table we show the effects of de- signed components OPENTAB. The evaluations happen under the closed-domain scenario. We can see that, the performance drops drastically from 0.71 to 0.54 due to the removal the STC mod- ule, highlighting its significance. Moreover with the ablation of Row SELECTOR, the performance continues to drop. Lastly, by disabling the longer- functionality READER, the performance reaches the lowest, showing the significance of incorporating detailed context information including
7

---

Published as a conference paper at ICLR 2024
Table Recall & scores for table retrieval the open-domain Opkn-Wiks and FEVEROUS datasets. BM25 wins among the non fine-tuned methods.
Setting
Method
@
Recall 10
Recall @22
Recall @ 50
Open- WikiTables
BM25
0.832
0.893
0.940
0.972
w/o Fine-tuning
BM25*
0.422
0.489
0,561
DPR-BERT
0
0
0
DPR-BERT BERT*
0.895
0.950
0.973
Fine-tuned
0.870
0.933
0.975
0.990
FEVEROUS
BM25
0.919
0.942
0.956
0.973
w/o Fine-tuning
DPR -EET
0
0
0.001
0.003
Fine-tuned
OPR-BERS
0.698
0.767
0.831
0.902
Others are implemented in this work. 2 The non fine-tuned DPR scores are approximate to O on both datasets due to poor trans- ferability.
Results of mmehhods are from Kweon al. 2023).
Table 5: Ablation studies on different proposed modules in OPENTAB. On the closed-domain Open- WikiTables dataset. STC stands for the Simple-to- -ompllp prompting SQL generation strategy.
Method
Accurate
OPENTAB OPENTAB w/o STC
0.710
0.539
OPENTAB w/o STC-ROWSELECCOR
0.515
OPENTAB w/o STC-ROWSELECTOR +broad-context READER
0.421
that the SQL generation process. For clarification, by disabling the longer-context functionality, do not forward the table schemas and generated SQLs into READER.
Ablation studies on Simple-to-Complex SQLs. We test the individual efficacy of SQL-basic, SQL- intermediate, and SQL-advanced. Instead of moving onto the next in order if program fails to provide valid result, we focus on one SQL program at time for evaluation individually. Table 6 show the results. We see that our design of synergizing all three kinds of design has the best performance compared with merely using solo SQLs. This is because in our strategy, the solidness of the generated SQL programs will be verified on the fly, thus we can try to evade invalid SQLs that will either have syntax errors or empty execution results, which is common for solo SQL generation.
We verify the efficacy of the Joint Reasoning, Sequential Reasoning, and GRSR strategies. Results are summarized in Figure From the plot we can see that the best per- forming model is OPENTAB with GRSR strategy, which is intuitive because such strategy implicitly leverages the generative power of the LLM towards identifying the correct table to build reasoning on. From the figure we can see that the accuracy of OPENTAB-GRSR constantly increases with more tables retrieved (higher recall value) despite the negative impact of more incorrect tables pre- sented. The novel generative reranking design leads to more precisely grabbing the core information by mitigating the hallucination issue.
Open-domain strategy.
RELATED WORKS
LLMs for Table Reasoning, Recent work has explored leveraging LLMs for table reasoning tasks without task-specific fine-tuning. Chen (2023) showed that LLMs like GPT-3 can perform substan- tially on table QA and fact verification datasets like WikiTableQQestioons and TabFact with few-shot prompting. However, their capability degrades on large tables with 30+ rows. To address this limi- tation, Ye et al. proposed using LLMs to extract relevant sub-tables and decompose complex questions into simpler sub-questions. Their DATER method improves LIM performance large
8

---

Published as a conference paper at ICLR 2024
OpenTab-RR OpenTab-GRSR
56
Table 6: Ablation studies the STC SQL generations.
52.3 $1.4
50.0
Closed- domain
Open-domain
48
top2
OPENTAB
0.710
0.523
OPENTAB-adv.
0.585
0.367
OPENTAB-int.
0.609
0,406
Figure 4: Ablation study the open-domain strategy. "JR" stands for "Joint Reasoning" and "SR" stands for "Sequential Reasoning".
OPENTAB-bar
0.617
0,390
and complex table reasoning tasks. Alternatively, BINDER (Cheng al. 2023) maps the natural language questions into programing languages such as SQL. and Python with API calls to invoke LM functions, improving the reasoning capability beyond the basic programming language gram- mar. He et al. (2022) proposed a strategy that involves rethinking with retrieval to enhance LLM faithfulness by retrieving relevant knowledge using the chain-of-thought prompting, showing the effectiveness in a series of reasoning tasks, including tabular reasoning.
Table Retrieval. Wang al. (202) investigated the necessity of designing sable-specific dense retrievers with structure-related modules. The authors showed that DPR (Dense Passage Retriever) and DTR (Dense Table Retriever) have comparable performances, justifying the effectiveness of deploying DPR in table retrieval tasks. Furthermore, K kwenn et al. (2023) shared similar findings on the Open-WikiTaal
dataset. Despite the efforts of applying dense retrievers on tables, number of existing works rely sparse methods to realize the retriever. (Schlichtkrull et al. (2021) leveraged TF-IDF to retrieve tables in the open-domain setting. |Aly et al. (2021) introduce the FEVEROUS dataset and also use sparse method DrQA (Chen al.) 2017) to retrieve both text and tables. Moreover, DCUF (Hu et al., 2022) utilized DIQA to retrieve and BM25 to rerank the retrieved documents. Chen et al. (2022) proposed to fine-tuned DPR open-domain table retrieval task, and the answers are heavily dependent on the generated SQL queries. These dependencies can result in subpar retrieval performance for unseen tables and cannot handle the cases when the generated SQL queries fail to execute. In summary, effectively retrieving relevant tables from large data corpus still remains an open question.
Retrieval Augmented Generators (RAG). RAGs leverage retrievers to fetch information from ex- ternal knowledge database and augment the text input the language models (Lewis et al. 2020). REALM (Guu et al., (2020) and RETRO (Borgeaud et al.) (022) pretrains the retrieval augmentation framework the end-to-end manner so the language model learns to better utilize the retrieved information, while ATLAAS (Izacard et al. 2022) jointly trains the retriever well as the language model. We point out that these well-known RAG models are specialized in the text reasoning do- main and cannot be directly applied to table reasoning tasks without fine-tuning, thus inapplicable to the experimental setup of this work.
CONCLUSION
In this work, we study the task open-domain table reasoning, where complex questions are an- swered using knowledge and evidence stored in structured tables. The reasoning part is empowered by Large Language Models (LL.Ms), enabling the OPENTAB to work directly on the target dataset without being specifically fine-tuned. This is highly desired property for several applications, from both efficiency and privacy perspectives. The proposed method OPENTAB incorporates CODER, ROWSELECOOR, and READER that are effective in generating valid SQL programs, highlighting informative content and absorbing longer context allowing for accurate response. We perform com- prehensive empirical studies demonstrating the efficacy of OPENTAB. In both open-domain and closed-done -omsed- settings, OPENTAB outperforms baseline methods by a large margin, revealing its advanced capability.

---

Published as a conference paper at ICLR 2024
8 REPRODUCIBILITY
To ensure reproducibility of OPENTAB, we provide detailed discussion regarding the experimental setup in S including the LLM backbone usage, pretrained checkpoint model name, few-shot sample count, etc. We share the prompt used in the experiments in the Appendix. We discuss the details of datasets and the specific metries used in Section A.3
REFERENCES
Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. Feverous: Fact extraction and verifi- cation over unstructured and structured information. In Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, volume 1, 2021.
Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. Tutorial proposal: Retrieval- lan- guage models and applications. In The 6l Annual Meeting of the Association for Computational Linguistics (ACL): Tutorial Abstracts, pp. 41, 2023.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring. Tom Hennigan, Saffron Huang. Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162, pp. 2206-2240. PMLR, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 17:18791 2020.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computa- fional Linguistics (ACL), pp. 1870- 1870, 2017.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michacl Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021a
Siqin Chen, Yubo Liu, Jie Wu, and Mengshu Hou. Retrieval augmented via execution guidance in open-domain table qa. In Proceedings of the 2022 5th International Conference on Algorithmms, Computing and Artificial Intelligence (ACAI), pp. 1-6, 2022.
Wenhu Chen. Large language models are few(1 -hoot table reasoners. In Findings of the Association for Computational Linguistics (EACL), pp. 1120-1130, 2023.
Wenhu Chen, Ming- Wei Chang, Eva Schlinger, William Wang, and William W Cohen. Open ques- tion answering over tables and text. In The Ninth International Conference Learning Repre- sentations (ICLR), 2021b.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rabul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Binding lan- guage models in symbolic languages. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
10

---

Published as a conference paper at ICLR 2024
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-39323 PMLR. 2020.
Hangfeng He, Hongming Zhang, and Dan Roth. Rethinking with retrieval: Faithful large language model inference. arXiv preprint arXiv:2301.0030: 2022.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisen- schlos. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 4320-4333, 2020.
Jonathan Herzig, Thomas Müllr, Syrine Krichene, and Julian Eisenschlos. Open domain question answering over tables via dense retrieval. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 512-519, 2021.
Nan Hu, Zirui Wu, Yuxuan Lai, Xiao Liu, and Yansong Feng. Dual-channel evidence fusion for fact verification over texts and tables. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 5232-5242 2022.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics (ACL), pp. 1049-1065, 2023.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Y Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208. 03399, 2022.
Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neubig, and Weizhu Chen. OmniTab: Pretraining with natural and synthetic data for few-shot table-based question answering. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 932- 942, 2022.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Pro- ceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6788 2020.
Sunjun Kweon, Yeonsu Kwon, Seonhee Cho, Yohan Jo, and Edward Choi. Dataset for open domain question answering with complex reasoning over table. In Findings of the Association for Computational Linguistics (ACL), pp. 8285-8299 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel. et al. Retrieval-augmente genera- tion for knowledge-intenn intensive nlp tasks. Advances Neural Information Processing Systems, 33: 9474,9474, 2020.
Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S. Yu. A comprehensive evaluation of chatgpt's zero-shot text-to-sql capability, 2023a.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03112. 2023b.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. TAPEX: Table pre-training via learning neural SQL executor. In The Tenth International Con- ference on Learning Representations (ICLR), 2022.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
Panupong Pasupat and Percy Liang. Compositional semantic parsing semi-structured tables. In Proceedings of the 53d Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing, pp. 1470-1480, 2015.
11

---

Published as a conference paper at ICLR 2024
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using stamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3982-3992, 2019.
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qifei Wu, Yuchen Ding, Hua Wu, Haifeng Wang, and Ji-Rong Wen. thorough examination on zero-shot dense retrieval. arXiv preprint arXiv:2204.22755, 2022.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ Information Retrieval, 3(4):333-389 2009a.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Retrieval, 3(4):333-389, 2009b.
Michael Sejr Schlichtkrull, Vladimir Karpukhin, Barlas Oguz, Mike Lewis, Wen-tau Yih, and Se- bastian Riedel. Joint verification and reranking for open fact checking over tables. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Inter- national Joint Conference on Natural Language Processing. pp. 6787-6799, 2021.
Ruoxi Sun, Sercan O. Arik, Hootan Nakhost, Hanjun Dai, Rajarishi Sinha, Pengcheng Yin, and Tomas Pfister. Sql-palm: Improved large language model adaptation for text-to-sql, 2023.
Zhiruo Wang, Zhenghao Jiang, Eric Nyberg, and Graham Neubig. Table retrieval may not necessi- tate table-specific model design. In Proceedings of the Workshop Structured and Unstructured Knowledge Integration (SUKT), pp. 36-46, 2022.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien- Sheng Wu, Ming Zhong, Pengeheng Yin, Sida Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. UnifiedSKG: Unifying and multi- tasking structured knowledge grounding with text-to-text language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 602-631, 2022.
Xi Ye and Greg Durrett. The unreliability of explanations in few-shot prompting for textual reason- ing. Advances in Neural Information Processing Systems (NeurIPS), 35:30378-30392, 2022.
Yunhu Ye, Binyuan Hui, Min Yang. Binhua Li, Fei Huang, and Yongbin Li. Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 174-184, 2023.
Fan Zhou, Mengkang Hu, Haoyu Dong, Zhoujun Cheng, Shi Han, and Dongmei Zhang. Tacube: Pre-computing data cubes for answering numerical-reasoning questions over tabular data. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2278-2291, 2022.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 CODER IN-CONTEXT LEARNING PROMPT (2 SHOTS)
Given the table schema and threee example rows out of the table, write a squite program
+ extract the sub-table that contains the information needed answer the questions. The squite does not need to directly answer the question. you alwaya
information when executing the sqlite.
II you cannot generate the sptice with high conf_dence that it is correct, then generate some SqLite
complex
correct.
Try to use Tuzay-match for values if are not sure about the values.
Generate SQLite programs with respect to the question seperated by [SQLSIP), the + complexities of the sqLite programs
ascend (basic, intermediate, advanced)
TARLE Paorice, Faorice__antoon _
sow_id int,
name text.
_2001 text,
_202 text,
_200
2004 text,
_000 text,
_007 text,
_2008 text,
2003 text,
2010
career_nar text,
cazees_nwin
/-
example rows:
SELECC FROM
1,TMT 3;
row_id
zame _001
_2002
_203
_2004
2005
_200
_2007
_200
_2009
_010
@areer_nwin__eee
o australian open
2r
1r
3:
2t
qt
3r
2r
3t
12
1 french open
4z
2z
2r
3r
1x
1r
1r
2F
1:
2 wimbledon
3r
28
2r
2r
2r
2r
2r
2r
a
14
11
1.4
*/
Q: did he wia more at the australian open or indian wells?
SQLite:
SELECT name, carser_nwin_ FROX Fabrice_ SericceSanttroo [SQLSEP]
SELECT name, carper_nwin_ FROM Fabrice_ sa_ntoooo name "Laustral openk* name
4 LIKE "Sindian wellst"; [SQLSEF]
WETE wins AS
SE3ECT
nane, CA9TssiBBTK(caeeer..ii....
INT) wins, 11 AS INT) AS losses WHERE name LIKE "Saustralian open'" OR name LIKE *Siadian wellas"
LMSIA/career (AST,55BBTTRCarrrrrLnnn.L_ INSTR (cararrrrrrarcnnloee
FROM Fabrice_Santone
SELECT name,,
Lotal_wins, (1oases) Losse_ Wins name;
CREATE
TABLE htrp__ddoor__rttematiioooio____ Aitpprtt
row_id int, zank text, city text, passengers text, ranking text, sirline text)
/+
example zows:
LIMIT 3; zow zank city passengers ranking airline
SELECT FROM
united states, angeles 749 nan
eirlines
1 2 united states, houston 5,465 nan united express
3
canada, calgary
3,761 nan tranaat, westjet
*/
Q: how many more passengers flew to los angeles then to sssketoon from manzanillo sisport in + 2013?
sgLite:
SELECT city, passengers Playa_de cecee_rr___eeeeeeeeeeeeeeeeeeeee Iwt.rrnttiinalliieeee.... Airportt [SQLSEP] SELECT city, passengers FROM playa_de_Grr_______ wwt.inennaiinal_lrrree WHERE city LIKE *llos angeles city "Ksaskatoon""; [80TSRP]
13

---

Published as a conference paper at ICLR 2024
#essengertounts
SELECT
city,
CAST (pessenggee ',", (1] A8 IN2) AS passenger_coun:
FROM ins.roonnnttttionssa..a. "$los angeless" city "kaaskstoons"
WHERE
SUMICASR WHEK city LTKE *Mlos angeleak* THEN passenger count F1.55 RMD)
SUMICASE WHEK city LIKE "Beaskacoou"* THEN passençer__our ELSE END) AS passenger_ fleeeeeeeeeddeeeeeeee Frecence
FROM PassengesCounts
A.2 READER IN-CONTEXT LEARNING PROMPT (2 SHOTS)
ven execution
attained running ng solite, extract final of + queation Irom the table.
table, you give + senae the question based the given table. If the answer conteins multiple items, seperate them by [98P].
you slways find the answer
CREATE TABLE Fabrice__n coorice_aantoro/
int,
name text,
text,
_2002 text,
_203 text,
_204
2005 text,
_200
2007 text,
2008 text,
2008 text,
2010 text,
text,
cazeer_nwin_leee text)
3 example rows:
SELECT Fabrice_$antoro
3;
row_id [d
name 2001 _2032
2003
_2004
_2005
_2006
_200
_2008
_2009
2010
corser_nsr
carecr_awin, mwin nwwin_less__
career
1r
2r
2r
3r
4
0 / 18
22
18
french open
0
20
17
2.0
wimbledd
2r
2r
14
11
14
+/
australian open indian wella?
SQLite:
name,
(SURSTR
1, (N losa, (-1)
TNT) total_wing
FROM
Fabrice Sa_ataoo
WHERE
(name LIKE fSaustrelian opent') ) OR
(name 'iindian wellst')
Execution Result:
name total_wina
australisn open 22
indian wells 23
indian wells
CREATE TABLE derya__d__r_______e____ dtt..orro__tteettttioooaaaaaa sew_id int, zank text, city text, passengers text, ranking text, airline text)
example
SELECT FROM plays_de__res aattr__nnaaraatiool__ppeee LIMIT 3;
city passengers ranking airlins
0 1 united states, los angeles 14, nan slaska airlines
united states, houston 5,465 nan united express
14

---

whose verification Published as a conference paper at ICLR 2024
Table 7: Applicability of OPENTAB and baselines.
Method
Open-domain
Large table
w/o Fine-tuning
FEVEROUS base (Aly ct al.
2021
DCUF (Hu 2022)
OpenW base (K weon ct al
2023
BINDER CCheng el 2023
DATER et al.
2023)
OPENTAB (proposed)
2
3
canada, calgary 7,761 nan
alz Liansat, westjel
Q: how many more passengers flem to los angeles then to saakatoon from manzanille airport in. 2013?
SuLite: SELECT
city,
AEPLACE '//, (1) AB paszenger_count
FROM
Playa_da_Orr intprr__nntenncoool_.iii
WHERE
(city LIKE *llos angelest') OR
tclty 1.RR "Sssskatoon8")
Execution
city paszenger_ccunt
states, los angeles maxico, saskatoon 10000 4749
A.3 DATASETS
Open- WikiTables. We directly leverage the Ween-WiiiTablls dataset provided by (Kweon et al. 2023) in our experiments. Note that this dataset has limited amount of cases whose final ground truth answer is empty, which will not be used in our experiments. Our evaluations happen on the validation set of the dataset. Due to the budget limitation, the experiments
carried out 2,000 random samples out of the validation set with a fixed random seed 42 to make sure all the models and methods are evaluated the same subset fairly. Note that for this dataset each input question will only has one gold table as evidence. Table corpus size is 24,680. For the evaluation metric, adopt the semantic-match evaluator used in BINDER (Cheng et al. 2023), which is a more reasonable metric to evaluate the table QA task.
FEVEROUS. FEVEROUS (Aly et al. 2021) dataset is fact verification dataset consisting of evi- dences in the format of both natural language text and structured tables. To adapt the dataset into the open-domain table reasoning setting, we filter 323 claims out the validation
is only based on table information. Also because the tables are noisy web tables where a lot of sam- ples incorporate undesired properties such as empty sells, multi-header, cell count mismatch, etc. These tables cannot naturally be used in the SQL database interface claims that need to be verified on such tables. The table corpus the FEVEROUS dataset used here is of 26, 177 different tables. Note that for the processed subset, each claim will only have one gold table as evidence. The prediction space limited to "refutes" and "supports". For the evaluation metric, inspired by the FEVEROUS score metric in Aly al. (202I). which requires predictions to be made on the correct evidences, we further invent our metric as to make the right prediction while locating the correct evidence table in the same time. Such scenario will be regarded successful prediction. As long as the model does not make the right decision, or makes decision based on the wrong table, will be considered a failure case.
need to further remove
A.4 SANITY CHICK ON TEXT-TO-SQL DATASET
STC is proposed in the context of end-to-end table QA task, which SQL programs work as an ef- ficient and scalable interface to query the database and extract relevant information for READER to make inference on. SQL generation is an intermediate tool in the QA pipeline, instead of outcome. While text-to-SQL. benchmarks, ground truth is SQL program itself, which is different from
15

---

Published as a conference paper at ICLR 2024
Coder Generation
Prompt
Codor
Coder
SCL
Reader Instructions
Reader
Coder Instructions
SQL Intermediato
SOL Selected
SOL Exccution Results
Response
Query
LLM
SQJ Advanced
Context Query
Context Query
Table Schema
Example Rows
Query
Figure 5: Prompt and generation structures of both CODER and READER.
Qucar
Rester Pringe
SALECCT GELECTIOIA Canir JAin Juns
Divan Te Litk scmomna on on
Coder
Geven
nussing SDI ln entrrttt froa InttAr of
Raader matruiiva
ectret the adbiaaii Red dutt
SOL
FROM aawio Sartau
alveys End un avave from the lubly, do
cuudionn: The SCA does sor nead
CRtran USK "Nh riNccrrh
Berqueltic tauad yeer latin
kivaya have orough information when
SOL Gelecke
ocule with righ coriibenee Phtiiia
SELECT 1818 Ced 110T116TTocueee rei Jena B.
leoo ocmpieeeee fut coveest
FROM Fabris Sonken
Carar: meris Jooa
22-18
(Ml save l88 "iuine wdwr/1)
17-30
Content Questic
PROM aarko, Setono. Wins
Corred & Guantion
LIWI
Cerme Cunsioor
Quadion
LIM
CHizat TABL outeou Sumbee
not 2013 Ina.. 2014 lot
2003 2000 2010 careL For caree
Farric Conter: Autrmior
2008 hoi, 2000 adi 2010 lod
fench open dr 2r te
tr ara 07.0-7700 17:33
spin
Figure 6: Concrete prompt and generation structures of both CODER and READER.
motivation. While text-to-SQL task is not the focus of this work, we carried out complementary ex- periments on the "flight_4" database of the Spider dataset to verify the effectiveness of our CODER module with the STC prompting strategy, Below we list the results in the order of question ground truth, and SQL generated by CCDER. also show the stats of the large-scale database in the table.
How many routes does American Airlines operate? SELECT count (x) FROM airlines AS T1 JOIN routes AS T2 ON Ti.alid + 72.alid WHERE T1 ra.eame = 'American Airlines'
SELECT COUNT (+) FROM routes r INNER JOIN airlines a ON r.alid a.
+ alid WHERE a.name = 'American Airlines';
How many routes end in a Canadian airport? SELECT count (x) FROM airports AS T1 JOIN routes AS T2 ON T1.apid
4 12.dst_apid WHERE country = 'Canada'
SELECT COUNT (+) FROM routes WHERE dst_ dst_apid (SELECT apid FROM 4 airports WHERE country = 'canada');
What are the names of all airports in Cuba or Argentina?
SELECT name FROM airports WHERE country 'Cuba' OR country = + 'Argentina' SELECT name FROM airports WHERE country LIKE "$cubas" OR country + LIKE "Sargentinas";
16

---

Published as a conference paper at ICLR 2024
What are the countries of all airlines whose names start with + Orbit?
SELECT country FROM airlines WHERE name LIKE 'Oriit8' SELECT DISTINCT country FROM airlines WHERE name LIKE "Orbit3";
In how many cities are there airports in the country of Greenland?
SELECT count (DISTINCT city) FROM airports WHERE country + Greenland'
SELECT COUNT (DISTINCT city) as airport_cities FROM airports WHERE + country LIKE "kgreenland8";
17


Reviews:
[{"review_id": "Va6jgjKaLQ", "reviewer": "Reviewer_9EJC", "summary": "The paper proposes OpenTab, an open-domain table reasoning framework. The framework includes A) a \"retriever\" module to retrieve a subset of structured tables from a table corpus (using BM25), and B) a \"reasoner\" module that is composed of a \"coder\" module for generating SQL queries, a \"rowselector\" module to rerank rows, and a \"reader\" module to parse SQL execution and its accompanying context into a natural language response. The authors also propose a Generative Reranking & Sequential Reasoning (GRSR) strategy to rerank retrieved tables using SQL and query similarity, and a simple-to-complex (STC) prompting strategy for SQL generation. \n\nThe OpenTab framework contributes a method for open-domain table reasoning that has higher accuracy than existing baselines, and that is reliant only on few-shot prompting (no fine-tuning).", "questions": "**Questions**:\n\n1. The ICL prompt for SQL generation includes this line: \"Generate 3 SQLite programs with respect to the question separated by [SQLSEP], the complexities of the SQLite programs generated ascend (basic, intermediate, advanced).\" to \"sequentially generate three SQL programs with ascending complexities\", SQL-basic, SQL-intermediate and SQL-advanced.\n\n    - a) This prompt seems intuitively underspecified in guiding the model to generate SQL that focuses on column selection in the first SQL, and column+row selection in the second SQL. In other words, the wording of the prompt doesn't specify what kind of SQL should be generated in each stage. How did the authors verify the SQL was actually of that specific functionality?\n\n    - b) How did the authors verify that the SQL written was actually of ascending complexity?\n\n2. SQL generation performance: In claiming the new simple-to-complex prompting strategy for SQL generation, how did the authors verify the performance of this method on text-to-sql benchmarks like Spider or BIRD-SQL?\n\n3. Lost in the middle phenomenon: Was there any prompt engineering done for the ICL prompts to counter the \"lost in the middle\" phenomenon, particularly for the reader module which incorporates substantial context?\n\n4. From Section 4.2, given that the baseline methods \"require full tables to be fed into LLM for reasoning, which may exceed the upper bound of the input token limit, rendering an invalid prediction.\", do the results for baseline methods include or exclude these invalid predictions? \n\n5. From Table 4, what explains the large gap in difference for BM25 and BM25* for the \"w/o Fine-tuning\" row? This seems surprising given that BM25 is just supposed to be the authors' replication of BM25* presented in Kweon et al., 2023.\n\n6. On the effectiveness of GRSR: how does the effectiveness of GRSR vary with different choices of top-k (where k is number of tables retreived)?\n\n\n**Suggestions**:\n- In Section 3.1, \"DPR\" is first used. Suggest to include the full name of Dense Passage Retrieval when it is first used, and use the acronym subsequently.\n- In Section 5, explain why using all three SQL difficulties leads to the best performance, as was done in Section 3.2.1\n- It is likely relevant for the authors to include a section on retrieval augmented generation in LLMs under Section 6.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is largely written clearly, and easy to follow. The OpenTab framework is a notable contribution to the table reasoning and semantic parsing community for its approach to open-domain reasoning. The paper's combination of several approaches into the OpenTab framework can be considered as an original application to the problem of open-domain reasoning.", "weaknesses": "- The paper claims to provide a new simple-to-complex prompting strategy for text-to-SQL generation using LLMs. However, the details of this prompting strategy are relegated to Appendix A.1, and it is not clear how the LLM is guided to increase the complexity of successive SQL generations. Moreover, for this claim of a new strategy to be sound, one would expect to see it applied to standard text-to-SQL benchmarks like Spider or the new BIRD-SQL.\n\n- The paper's framework for retrieval and generation for open-domain table reasoning is analogous to retrieval-augmented LLMs, but there is no mention of RAG (for e.g. [Lewis et al., 2020](https://arxiv.org/abs/2005.11401)). The paper is thus strangely not situated within the RAG literature (but the paper contains fair contributions to RAG, such as GRSR).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VeeaMzFrlX", "reviewer": "Reviewer_zsjW", "summary": "This paper proposes a new method for open-domain table reasoning based on large language models. The method contains a table retriever, row selector, coder, and a reader. The table retriever is built on BM25. For the coder, the authors propose the simple-to-complex prompting strategy to generate SQL queries for three levels to resolve the complex input queries and infeasible queries by SQL itself. Then a reader module takes the context and the intermediate SQL execution results to produce the final answer to the query. \n\nTo resolve the hallucination problem of LLM when generating SQL queries, the authors also propose a generative reranking & sequential reasoning (GRSR) strategy by assessing the similarity of the generated SQL with the query so as to evade noisy tables. \n\nThe authors conduct comprehensive experiments to demonstrate the superiority of the proposed method as well as ablation studies to show the effectiveness of each proposed module.", "questions": "Are you going to release the code?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The proposed method is novel and effective, as demonstrated by experiments. Each proposed module is well-motivated and well-ablated.", "weaknesses": "Some technique details are not clear. \nWhen using BM2.5 for the retriever, are you using the full table? \nFor the proposed simple-to-complex strategy for Coder, how is it working with GRSR? \nFor the GRSR, how to deal with SQL queries grounded on multiple gold tables?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RlGbsaJElo", "reviewer": "Reviewer_VJ1H", "summary": "The paper introduces OpenTab, an open-domain table reasoning framework powered by LLMs. Unlike most LLMs which rely on knowledge stored in their parameters or retrieved from documents, OpenTab is specifically designed to utilize the rich information contained in structured data. The framework consists of several key components: a retriever to collect relevant tables, a row selector to prune only the pertinent rows of records, a Coder to generate SQL queries, and finally, a Reader to verify the answers. Experiments conducted on Open-WikiTable, WikiTableQuestion, and Feverous demonstrate that OpenTab achieves superior performance compared to previous reader and parser-based methods on tables across both open and close domains.", "questions": "- In Figure 3, reader seems to appear after the SQL is selected, but in the explanation, the authors mention that the reader is used to select the SQL. Which is correct?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is clearly written and easy to follow.\n- Figures 1, 3, and 4 effectively illustrate the overall pipeline and modules.\n- The proposed pipeline is novel and generally reasonable for addressing the specific task at hand.\n- OpenTab outperforms the previously proposed table-reasoning methods.", "weaknesses": "- The proposed pipeline might be applicable only to open-domain wikipedia-based tables and not to tables with a large number of rows as in Spider or BIRD. For instance, the row selector, although I agree with the motivation, can result in incorrect candidate tables as some questions may require a large number of rows. Determining the appropriate number of rows to retrieve using a heuristic seems to assume that the pipeline is not suited for dealing with complex tables. One of the main benefits of using tables is their ability to store a large number of records (mostly in close-domain scenarios). I would like to hear the authors’ thoughts on this aspect.\n- Minor typos in the manuscript (e.g., by by)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)