MMQA: Evaluating LLMs with Multi-Table Multi-Hop Complex Questions
Authors: Jian Wu
Linyi Yang
Dongyuan Li
Yuliang Ji
Manabu Okumura
Yue Zhang
Venue: ICLR 2025 Oral (2025)
Keywords: LLM evaluation
multi-table question answering; multi-hop question answering
Primary area: datasets and benchmarks
TL;DR: A novel multi-table evaluation benchmark that evaluate LLMs' multi-table understanding and reasoning ability.

Abstract: While large language models (LLMs) have made strides in understanding tabular data, current tabular evaluation benchmarks, such as WikiTableQuestions and WikiSQL, are focus on single-table scenarios, which cannot necessarily reflect the complexity of real-world applications. To bridge this gap, we present a \textbf{M}ulti-table and 
Multi-hop Question Answering (MMQA) dataset to assess LLMs' understanding and reasoning capabilities in handling multi-table tasks. The MMQA dataset demands that models perform multiple inferences by drawing evidence from various tables, which are designed to be connected with each other and require models to identify and utilize relationships such as foreign and primary keys. Then, we introduce a comprehensive evaluation framework that tailors to assess LLMs' capabilities in several aspects including Multi-Table Retrieval, Text-to-SQL Generation, Multi-Table QA, Primary Key Selection, and Foreign Key Selection. 
Finally, we propose a novel multi-table retrieval method that achieves state-of-the-art (SOTA) performance on the MMQA dataset compared to several strong baselines. 
Our experiment results reveal that, compared with human performance, both open-source and commercial LLMs leave significant performance room for improvements in multi-table understanding and reasoning tasks. We believe that the MMQA benchmark will enhance and facilitate LLMs' multi-table capabilities in real-world scenarios.

Published as a conference paper at ICLR 2025
MMQA: EVALUATING LLMs WITH MULTI-TABLE BULTI-HOP COMPLEX QUESTIONS
Jian Wu Linyi Yang"* Dongyuan Li Yuliang Ji
Manabu Okumura' Yue Zhang
Institute of Science Tokyo Uniiersityy College London Schooo of Engineering, Westlake Univeristy "The University of Tokyo "Najjing University of Science and Technology.
ABSTRACT
While large language models (LLMs) have made strides in understanding tab- ular data, current tabular evaluation benchmarks, such WikiTTableQuestionss and WikiSQL,, are focus on single-table scenarios, which cannot necessarily re- flect the complexity of real-world applications. To bridge this gap, we present a Multi-table and Multi-hop Question Answering (MMQA) dataset to assess LLMs' understanding and reasoning capabilities in handling multi-table tasks. The MMQA dataset demands that models perform multiple inferences by drawing evidence from various tables, which are designed to be connected and require models to identify and utilize relationships such as foreign and primary keys. Then, we introduce a comprehensive evaluation framework that tailors to assess LLMs' capabilities in several aspects including Multi-Table Retrieval, Text-to- SQL Generation, Multi-Tabb QA, Primary Key Selection, and Foreign Key Se- lection. Finally, propose novel multi-table retrieval method that achieves state-of-the-an (SOTA) performance on the MMQA dataset compared to several strong baselines. Our experiment results reveal that, compared with human per- formance, both open-source and commercial LLMs leave significant performance room for improvements in multi-table understanding and reasoning tasks. We believe that the MMQA benchmark will enhance and facilitate LLMs' multi-table capabilities in real-world scenarios. The Whole MMQA data are available httpss//anonymous 4open. science/r/MMQA- 34B1
INTRODUCTION
Table is one of the fundamental structured data types in real-world scenarios, its widespread applica-
tion includes relational databases and spreadsheet forms RRaffel et al.) 2019). Recent studies have
shown the strong capabilities of LLMs on table-related
tasks (Zhu et al. 2021 Zhao et al. 2023
Hegselmann et al. 2022 Li al. 2023 Zhang et al.
2024b
Lu et al.)
2024). However, L.MS
multi-table understanding and reasoning performance remain relatively
less explored. Previous
table-related studies such Table-QA
(Chen al.
2020b
Zhu al.
2021
Pasupat & Liang
2015 et al. 2017 Yu et al.
2018a| Cheng et al.
2021 Katsis et al. 2021 Nan et al.
2022
Jauhar et al.) 2016 Li et al. 2021 Chen et al. 2020a). Table Fact Verification (Chen al. 2019
Günther et al., 2021). 1). Table-to-text Generation (Moosavi et al. 2021:
Suadaa et al. 2021
Lebret
al. 2016). and Column type & Relation classification (lida al. [022 Deng et al. (0200) all focus
on single-table tasks. However, in real-world scenarios, operations such as join, union, intersection,
and foreign key identification are frequently used in multi-table reasoning. but there have been very
few benchmarks for their comprehensive evaluation (Pal et al., 2023 Zhang et al. 2024a).
To this gap, we build a Multi-table and Multi-hop Question Answering evaluation datasets (MMQA), aiming to evaluate LLMs' understanding and reasoning multi-table data. For com- prehensive understanding of LLMs' performance on multi-table table tasks, we propose an evalua- tion framework over different aspects (Multi-table Retrieval, Text-to-SQL Generation, Multi-table
Equal contribution. Jian Wu did this work during his internship at Westlake University Corresponning author.

---

Published as a conference paper at ICLR 2025
Head
Management
Department
Head ID
Name
Born State
Department ID
Head ID
Deparmment ID
Name
Crestion
Ranking
Num Employees
Tiger Woods
30266
Dudley Hart
California
Commerce
1903
36000
Question
Step Find Woods head id with born the state Alabama in the Head tabie
What are the distinct creation years of the departments managod by a secretary bomn Alabama?
Step Based on the head find the dopartment of the thn.aaaaammmetttaaa table
Find the creation year based the department in the Department table
Multi Toble Reasoning
Answer:
Figure I: An example from our MMQA benchmark. Typical challenges in MMQA are: 1) Determine the reasoning order; 2) Identify the primary keys and foreign keys; 3) Retrieve pieces of evidence from different tables for multi-hop reasoning. The words in deep blue in tables are evidence for reasoning the answer.
Question Answering, Foreign Key Selection, and Primary Key Selection), jointly considering coarse- grained and fine fine-graine ree--grined information of tables, at the table, column, and cell level. Figure depicts an example from our MMQA, a question involving three tables. The question is "What are the distinct creation years of the departments managed by a secrelary born in state Alabama?" and the three input tables are "Head", "Management" and "Depariment". The words in blue are the keywords for reasoning across tables. I.IMS expected to first understand the question well as the input tables to determine the reasoning order. Starting reasoning from a specific table and column, and then locating the key columns for jumping another table. For instance, the Head table, the columns are Head ID, Name, and Born State, where the column Head ID is both the foreign key and the primary key of the table, which helps LLMs to jump the table Management, In the Head table, we can determine the head ID is "1" based on the state "Alabama", " Then, from Head ID number "7" department. find the candidate answer "1903".
to Depariment ID in the Management table, LLMS could determine Finally, based on the department "7" in Department table, Consequently, in this example, LLMs need to reason intra-table and inter-table at both column level and cell level, which is much more complex than single-table reasoning. The differences between multi-table qa and single-table qa are reminiscent of those between Multi-hop QA (Yang et al. 2018) and songle-hop QA (Rajpurkar et al. 2016).
We conduct extensive experiments to assess the multi-table and multi-hop understanding and reason- ing abilities of the LLMs on our MMQA dataset. The results demonstrate the superiority of human performance over current SOTA LLMs, shedding light on the challenges encountered by existing models in performing multi-table tasks. Based on that, we propose a novel multi-table retrieval method, named MTR, which incorporates the Question Decomposition module to decompose multi-hop question into series of sub-questions and converts multi-table retrieval task into several rounds of single-table retrieval task. In each round, MTR jointly considers question-table relevance and table-table relevance for single-table retrieval. To the best of our knowledge, our research the first introduce a multi-table and multi-hop QA benchmark, evaluating LLMs' multi-table complex reasoning abilities. On the MMQA dataset, MTR shows an advantage achieving the best results over range of the previous strong baselines.
RELATED WORK
Single-Table QA. Table Question Answer (TQA) involves retrieving answers from one or several table cells from given table, such as Wik/TableeQesstinns (Pasupat & Liang 2015). WikiSQL (Zhong ct al. 2017). SPIDER (Yu et al., 2018a), TABFACT (Chen et al. 2019). However, these datasets mainly focus on reasoning tables and ignore important knowledge stored in the textual corpus. Consequently, QA covering both tabular and textual knowledge has gained increasing interest. Chen et al. (2020b) pioneered passage-table QA benchmark. HybridQA, with Wikipedia tables linked to relevant free-form text passages (c.g., Wikipedia entity-definition pages). The OTT-QA (Chen et al. 2020a) benchmark extended HybridQA the open domain setting, where system needs
2

---

Published as a conference paper at ICLR 2025
to first retrieve relevant set of tables and passages before trying answer questions. Moreover, the links from the table and the passage are not explicitly provided. FinQA (Chen et al., (202T) and AIT-QA (Katsis et al. (02T) are predominantly target financial and airline tables, suggesting complex reasoning challenges that require models not only to interpret but also to compute and extract
nuanced information precisely. TableBench (Wu et al. 2024b), comprehensive and complex tabular benchmark, including 18 fields within four main categories to evaluate the TQA capabilities of LLMs.
Despite the significant advances made by LLMs in TQA (Li et al. 2022 Singha al. 2023 Li et al.
2023), there is still critical need for benchmarks that reflect the multi-table reasoning complexity encountered in real-world scenarios. Our work differs from this line of work, incorporating real-world complexities into its evaluation scenarios.
LLM for Table Reasoning. Despite the remarkable performance LLMss textual reasoning, their reasoning capabilities on tabular tasks are still limited. (Zhu et al. (202T) proposes a TAT-LLMM for reasoning over hybrid of tabular and textual data, on FinQA (Chen et al. 202T). TAT-QA (Zhu et al. ) and TAT-DQA Zhu al. (2022) benchmarks. Cheng et al. (2022) and Chen (2022) focus
on utilizing LLMs to reason over single-table data with zero-shot setting. TableLLM (Zhang et al. 2024b) and TableLlama (Zhang al., 2023) two table-related LLMs, pre-tra pre-trained and evaluated on single-table datasets. TAT-LLM (Zhu et al., 2024) tackles the snuestion-answert
task (QA) by
proposing a step-wise pipeline including Extractor, Reasoner, and Executor assist LLMs in better performing discrete reasoning over a hybrid of tabular and textual data. Ye et al. (2023) harnesses the multi-step reasoning capabilities of LLMs to first decompose complex questions into sub-questions by generating intermediate SQL queries for tabular data. TAP4LLM (Sui et al. (023) is a versatile pre-processing toolbox to generate table prompts to enhance the complex reasoning ability of LLMs over tabular data. However, all LLM--ased table reasoning methods focus on single-table reasoning. The
understanding and reasoning problem remains under exploration.
Multi- Table Tasks. Pal et al. (2023) pioneer a multi-table pre- pre-training task of answering questions over multiple tables and targets to generate sub-tables from input multi-tables. [iu al. (2023)
furtherly proposes document-level summarization dataset that jointly considers textual information as well as multi-table content in documents. However, the two methods focus on the sub-table level, which is coarse-grained. In contrast, consider the multi-hop and multi-table tasks. Chen et al. (2024) propose a multi-table retrieval method that considers the relevance between column and sub-questions, without alignment of whole table headers. Our work is different from previous works in two main aspects: 1) a comprehensive multi-table understanding and reasoning evaluation; and 2) evaluation on different granularities, i.e., table, column, and cell levels.
3 MMQA
As shown in Figure 2 the evaluation framework has two main steps: 1) Multi-table Retrieval; and 2) Multi-table Evaluation. The evaluation tasks consist of four categories: Multi-table retrieval, Text-to-SQL Generation, Multi-table Question Answering, and Key Selection (primary key and foreign key).
3.1 MMQA CONSTRUCTION
This section details the dataset construction process, including data annotation and quality verification, and reasoning steps compared with the previous Table QA benchmarks.
Data Collection and Annotation We develop the MMQA benchmark over Spider database Yu et al. 2018a), which is cross-domain complex semantic parsing dataset for Text-to-SQL Generation. Spider consists of 5,693 SQL queries and more than 200 databases of multiple tables covering 138 different domains. We randomly select a total of 5,000 samples from Spider and each sample contains two or three tables.
Question Answer Annotation Then, we follow the paradigm of Pal et al. (2023) to synthesize multi-table SQL queries through the 45 manually crafted templates over the Spider database and hand-crafted rules for operation types. After getting the SQL queries, we prompt the tables and SQL queries into LLMs such as GPT-4-turbo to generate natural language questions. Finally, we asked two human experts with computer science backgrounds to annotate the foreign keys and primary keys of each table. The primary key of one table is a column or a constraint that uniquely identifies each record in the table. The foreign key is column or combination of columns that is used to establish
3

---

Published as a conference paper at ICLR 2025
NL Question Find the first nome students who living the Smith Hall
STEP-I: Multi- Retrieval
First name, Age, Sex, Major)
Retrieve Tables
Dorm (dormid, dorm_name,, Tuudent_ccppcity..... gender)
Lives_in (SruID, dormid, room_number)
LLM
Table Corpus
STEP-2: Multi- Table Evaluation
Text-to-SOL
Multi-Table OA
Key Sol ection
Evaluation
SELECT frome FROM student A5 TI TINO lives AS T2 ON T1.suudd
12 stad JOIN dorn AS T3 ON
13 dormi 12.dormid WHERE
T3.dom_rrame
Tables
LLM
The framework of our multi-table evaluation, LLMs are firstly required to retrieve tables from a given table corpus. Then, we evaluate LLMs' reasoning and understanding abilities on MMQA. The multi-table evaluation involves Text-to-SQL Generation, Multi-Table Question Answering, Primary Key Selection, and Foreign Key Selection.
Figure 2:
and enforce a link between the data in two tables to control the data that can be stored in the foreign key table. For answer annotation, two human experts are required to give answers based on the tables and generated questions.
Quality Verification After obtaining the anno- tation results, in cases discrepancies, third expert was invited to review the annotations of the two experts, serving as guidelines im- prove consistency, and the final results were determined by majority voting. To ensure the quality of the annotated data, discard the questions whose correct answers could not be extracted from the given table or have grammar issues. The inter-agreemmnt is the average score computed based on the three experts' check re- sults. We compute the results of experts as hu- man performance, compared with LLMs. We list the properties of our benchmark in Table [
Table 1: Summary of statisties of MMQA and inter-human agreement.
Propertics
table
table
Total Tables
2591
721
Avg rows per table
1833.31
1369.01
Avg columns per table Avg foreign keys per table
6,04
4.78
2.81
1.95
Avg primary keys per table
3.35
2.41
Inter-human agreement
86%
82%
Question Length
77.11
85.38
Question Category
Question Categories Drawing from real-world scenarios and user demands for multi-table data, we design four primary question categories: Nu- merical (numeric operation, sum, average, etc.), Seleet
Numerical
889
289
List
214
44
Count
200
1289
346
List (list operation that showcases all answers meet the conditions), Count (count the number of answers that meet the conditions), and Select (select specific answer that meets the conditions). We illustrate several examples of different categories of questions in Table
Reasoning Steps We compare the data complexities of different datasets by caleulating the number of reasoning steps required to solve the multi-hop questions. Figure demonstrates that the average reasoning step of MMQA is significantly higher than that of existing datasets.
Finally, after obtaining MMQA, a comprehensive and complex benchmark consisting of 3,312 tables in total with the corresponding natural language question, SQL query, gold answer, foreign key, and primary key annotation.
3.2 MULTI TABLE RETRIEVAL (MTR)
Different from the single-table retrieval task, a multi-table retrieval task over a couple of tables be defined follows. Specifically, given question Q, table corpus C (T, J. retrieve
4

---

Published as a conference paper at ICLR 2025
Table Differences between our MMQA with previous table QA benchmarks. We here abbreviate the natural language as 'NL'. Our benchmark can be applied to evaluate LLM's multi-table under- standing and reasoning abilities more comprehensively.
Benchmarks
Question formot
sizee
sounce
Multitable
WTO
Pasupat & Liang
2015
NL question
20,000 table questinn pairs
Wilkipedia
Single-table QA
X
WikksQL
(Zhong nl.
XI17
SQL. query
24241 tables
Wikeipedia
Single-table QA
By HyhridOA
Chen et al
2020b
NL question
70k table question puirs
Wikipodia
Table-tzxt QA
byeer
2017
question
6,066 unique questions
Wikipodia
Singlettable
X
ReTaQA
Nan ml
20722
NI question
10,350 tables
Wikipedila
Single-table
Spider
sQQ-query query
question
and SQU query pairs
questions
Crowdsouncing
Text-4o-SQL.
X
NL question &
12.751 questions
BIRD
1. al.
20144
SQL
and SQL query
Kaggle
Test40-SQU.
SPINACH
SQL query
question
and SQL query pairs 3681 tables
320
Crowdsouncing
Texs-to-QLL
x
WTQSQAAFTIIA
Tahlehench
Wu
2024bb
NL.
/FIRQA/ATTQAA
Single-table
MMOA(Ouns)
question SQL. query
3,312 tables
Wikippdia
Text-to-SQL. Multi-able QA
Multi-tablc
Primary Key & Foreign Key Selection
Table 3: Examples of MMQA question types and tables. We emphasize keywords for the related
table columns.
Table Type
Question Type
Multi-hop Question
What are the of all stations that have a latitude above 37.4 have never had less than 7 bikes available?
Numerical
2 table
List
List the customers' first and last name of 10 least expensive invoices. led by heads who are mentioned? What are the ids of the courses that are registered or attended by the student whose What is the salary and name of the employee who has the most number
Count
How many departments
Select
121?
Numerical
table
of certificates aircrafi with distance of more than 5000?
Find the cell mobile number the candidates whose assessment code
List
Fail?
For each course id, how many students are registered and what are the course names? ?
Count
What are the distinct creation years the departments managed by
Select
secretary born state 'Alabama'?
a table Ti from C that contains the answer of Q. Single-table retrieval task is to select the most question-related table from C. However, in the multi-table retrieval task, our problem is to retrieve a list of question-relatee tables, which can be joint reasoning with the connection of foreign keys. (f,,,t,.... where f; C and must be joinable
Here, denote the retrieved tables as R(t) with another I E R(I). For example, in Figure I the table Head is joinable with table Management because they have the same column Head ID. Consequently, to identify the most related tables corpus of tables, we need to consider two aspects: retrieving the question-related tables, and retrieving the table-related tables. Inspired by the previous multi-hop question decomposition work (Wu et al. 2024a), which generatively decomposes the multi-hop questions for enhancing question-related evidence retrieval performance.
We propose novel multi-table retrieval method (MTR) that iteratively retrieves question-related related and table-related tables. Given a multi-hop question Q. we first use the GPT-4-turbo the question decomposer and feed the multi-hop question into the decomposer directly by a set of prompts (Appendix
B with a one-shot setting and get series of sub-questions 91, 92, (2. .n. Then, we fine-tune TableLlama-7b (Zhang et al. (023), as well as SGPT-5.8B (Muennighoff 2022), as the single-table retrieval model on the off-the-shelf sing-table QA datasets Chen et al. (0019 2021):
Katsis al, (2021): Zhu et al. (2021).
We treat the multi-table retrieval task into several rounds of ting-table retrieval tasks. For n decom- posed sub-questions, from the table corpus. In the first round, we only consider the guestion-table relevance score, and each retrieved table is assigned table-relevance score. Then rank the scores to get the top K
iterate rounds to retrieve the top K (K=2,5,10) sub-question-related tables
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 Multi-Table Retrieval
Initialize:
Input: Multi-hop Question Q. LLM: GPT-4-turbo.
Ouput: Retrieved Tables
First Round: 7 + + (q tables)
> only compute question-relevanee scores in 1st round
for i to do
for j E to M do
for h E 0 to M do
Compute Relevance Scores
Y + alg tablet) 3(tablex 4-1 ,table))
end for end for end for for i AArgSort descending=True) do
table; +- m table;)
Select top K relevant tables
end for Return tables
tables. From the second round the end, the previously retrieved tables treated as seeds. Where we rank the top tables based on the question relevance score as well as the table relevance score. The question relevance score is computed with single-table retrieval model, and the table relevance score is computed based on the overlap of table columns. If the retrieved tables overlap columns with tables retrieved in the previous round, then assign score of I and O. When 0 is assigned, stop iterations. In each round, the score is the product of the question relevance score and table relevance score. After all the sub- uub-questions have been used retrieve the tables, we sum all the scores:
Σ iSNJEMASM
alqi, lable)) B(lablet ,lable))
where aq,, table)) is the relevance score, given by the single table retrieval model, between ith sub- question with jth retrieved table in round i.a(te table)) is the table relevance score between the k-th retrieved table in i round and j-th retrieved table in round i. The implementation details are illustrated in Appendix
3.3 SUBTASKS
The multi-table evaluation is referred to as find- ing answers complex questions that require reasoning multiple times from given tables. We employ multi-faceted metric set for Multi- Table Retrieval, Multi-Tall Table Question Answer- ing, Text-to-SQL, and Key Selection.
Averuge Reasoning Steps across
6.0
Multi-Table Retrieval. We evaluate the LLMs performance the Multi-Table retrieval task on MMQA whose questions require reasoning over multiple tables to answer. The goal to answer the following question: To what extent can LLMS retrieval multi-table that, jointly con- siders question-tabe relevance? We build a table pool that includes all tables of MMQA and input multi-hop questions into LLMs to retrieve all auestion-related tables from the pool. This task fundamental because in the real-world scenario, the subsequent steps of reasoning and answering performance based on the quality of the retrieved tables.
WTO u BQA FoTsOA II AITOA HybroOA TekreBench MMDA
Figure 3: Data complexity comparison with exist- ing datasets in reasoning steps.
relevance and table-table
Text-to-SQL. Following the text generation evaluation task, we utilize Rouge-1, Rouge-L (Lin 2004), and BLEU (Papineni et al. 2002) evaluate the I.L--generated SQL. query quality against

---

Published as a conference paper at ICLR 2025
Table The main experiment results on Multi-Table Retrieval, our MTR outperforms all previous strong baselines.
Top
Top-5
Top-
S-able
2-table
3-oble
2-able
Scatle
Besviiuss
DM25
6.2
4.3
5.1
4,4
5.2
4.8
8.7
9.2
6.3
7.5
10.9
9.2
99
8.3
mdf
4.6
4.
3.1
4.8
9.3
8.5
11.4
11.1
8.9
DTR
et et
29.9
30.4
35.
33.2
52.9
16.1
38.2
35.4
SGPT-1258
37.4
99.1
40
SCIP-5589
42.7
53.6
53.2
62.5
59.3
(SCIPE-S 8R)
51.4
59.5
60.2
61.7
61.6
63.5
MIR (Tibleluna
72.3
64.7
68.3
69.5
66.2
67.8
74.
71.8
73,0
72.9
70.6
71.7
74.5
73.3
74.9
73.6
71.9
72.7
w/o QU
05.3
62.3
633
64.7
63.5
64.1
70.2
68.3
74.6
03.5
67.5
68.0
70,3
08.5
09.6
69.5
67.7
08.6
ground truth. Different from single-table QA reasoning tasks, the multi-table SQL query is much more complex with more operations.
Multi-Table Question Answering. The primary goal of Multi-table weesstion-answeriiee evaluation is to measure the LLMs' proficiency in understanding complex queries, navigating through various tables, and generating correct and coherent answers. This evaluation crucial for determining the effectiveness of LLMs in real-world applications, where they often need to interact with and extract information from multiple data sources simultaneously.
Primary Key & Foreign Key Selection. Unlike rows, which represent records in databases, columns represent attributes where the column header provides semantic meaning to the values. Moreover, the primary key, as well as the foreign key, is the important column feature of multi-table data. Hence a correct key selection interprets LLM's column-level understanding ability across multi-tables. Primary key & Foreign key selection is the percentage of correctly selected columns among all target columns in the evaluation set.
4 EXPERIMENTS
We design a series of experiments based on the MMQA benchmark, aiming answer three questions: 1) How do LLMs perform on multi-table QA tasks compared to human performance? 2) What is the LLMs' performance
multi-table related tasks such Primary Key Selection and Foreign Key Selection? 3) The average number of rows of MMQA is more than 1,000. How do LLMss perform on long tables?
4.1 EXPERIMENTAL SETTINGS
Datasets. Specifically, we divide our MMQA benchmark into two parts: 2-table (2591 samples, average of 1833.31 rows and 6.04 columns) and 3-table (721 samples average of 1369.01 and 4.78 columns) subsets.
Models. We employ the proprictary and open-source LLMs our experiments and enhance reproducibility, we set the temperature to 0.7 for proprictary models, and all the experiment results the average scores of three experiment results. For proprietary models, adopt GPT-4 (Achiam et al. 2023), GPT-3.5 (Ouyang et al., 2022), Gemini-pro (Team et al. 2023), and OnldOO-ppee 01-preview. For open-source LLMs, we evaluate on TableLlama-76 (Zhang al. (023) and Mistral-7b (Jiang et al.)
2023 The prompts of different evaluation tasks are shown in the Appendix
Evaluation Metrics. For the table retrieval task, we utilize precision, recall, and F1 scores to measure the performance of multi-table retrieval. For multi-table reasoning. LLMS were assessed using a combination of metrics, including Exact Match (EM) and Partial Match (PM) for multi-table QA, Rouge-1, Rouge-L (Lin 2004). BLEU (Papineni et al. 2002) for Text-to-SQL task, and accuracy scores for Primary Key Selection (PKS) and Foreign Key Selection (FKS). Partial Match indicates the partial semantic match scores between LLMs' generated answers and gold answers. We utilize GPT-44turbo the answer evaluator give the scores and the prompt is available in Appendix
7

---

Published as a conference paper at ICLR 2025
Table 5: The main results of different baselines on the 2 table dataset. We divided our benchmark into a 3-table subset and a 2-table subset. We use to denote the zero-shot setting and tto denote the one-shot setting. We here abbreviate the Primary Key Selection "PKS", and the Foreign Key Selection as "FKS". PM indicates the partial semantic match of LLMs' generates evaluated with GPT-4-turbo with the prompt in Appendix
Dataset
2 table
Evaluation Methods
Table QA
Text-to-SQL
PKS
FKS
Metries
EM
PM
Rouge1
RougeL
BLEU
Acc
Acc
Open Source LLMs
Laaia 76"
7.58=03
8.06=0.11
9.12 =02
7.89 +022
1.82+0.1
17.86:10.2
13.75++0.2
Tablel 7bf
8.23-011
8.57
10.34 =0.3
9.53 2022
2.9240.2
20.25::0.2
15.8940.2
Mistral-7h
5.36-01
5.89=0.1
7.25=02
6.3620.1
1.79:40.1
14.15:0.1
13.98400.
Mistral-7b f
6,26-02
6.72=0.11
9.55=0.1
8.45
2.49+
17,65+00.2
16.17:40.2
LlaMA-2-13b
9.45
10.13.0.1
17.34_022
15.81
5.44.10.1
25.34.10.1
22.78.10.1
LJaMA-2-13h
11.28
13.04002
20.45-0.1
18.17.0.1
7.5910.2
28.89
25.27
Proprietary LLMS
GPT-3.5'
02
29.34.1.11
31.75
27.89
2.71110.3
28.06
19.25
GPT-3355
26.79=0.1
29.78=0.1
33.96=0.2
29.74
4.82+0.2
38.75+0..1
28.83++0.1
GPT-4"
25.17=02
31.35=0.2
32.51 =0.2
28. +0.
2.53+0.1
31.59 +0.2
21,27 #002
GPT-4f
28.88=0.1
34.57=0.1
39.64 09.64-0.2
35.07=0.2
5.77 +0.2
42.78
+:02
26.88++0.1
Gemini-pro"
27.16-02
32.72:102
33.13=002
29.28
2.69
32.77
22.06:10.2
Gemini-prof
28.58-02
33.89=0.1
35.26=0.1
30. 15:02
5,3440.2
44.194402
28.38
Ol O1-preview
46.25=02
49.72=0.2
38.41=022
37.75=0.3
6.79+0.3
42.81+0011
30.53+t0.1
01 O1-previewx
50.78
53.85
43.62=022
39.52.10.3
7.581002
49.53
34.17
Human
89.8
82.7
96.5
95.3
These metrics provide a comprehensive view of the models' performance, from their ability to generate accurate SQL queries (Text-to-SQL) to their capacity to understand and reason over tabular data.
4.2 MULTI-TABLE RETRIEVAL EVALUATION
Table 4 presents the results of a comprehensive experiment evaluating different baselines multi- table retrieval task on MMQA with different choices of (K=2,5,10). Across different values of K. our MTR achieves SOTA performance compared to previous strong retrieval approaches (BM25, TF-IDE DTR, and open-source LLMs). Notably, the BM25, TF-ID, and Table Dense retrieval which are based on the small models show poor performance
retrieval of multiple interconnected tables. While SGPT-5.8B and TableLlama-77 7b could handle long structure input, the guestion-relevane score and table-relevance score in MTR comprehensively consider the intra- and inter-connections. The multi-table retrieval experiment results indicate discernible variation in performance among the evaluated LLMs. Specifically,
MTR outperforms the open-source LLMs, SGPT, and TableLlama, and demonstrates superior precision, achieving a score of 72.3%, which suggests a high accuracy in identifying the most relevant tables for given question. This performance attributed to MTR's advanced question understanding and its ability to discern the intricacies of multi-table relations. In contrast, the MTR combined with TableLlama-77 while showing commendable recall with a score of 64.7%. lagged in precision, indicating tendency to retrieve a broader set of tables that occasionally included less relevant ones. The F1 score, which harmonizes precision and recall, was highest for MTR at 68.3%, reflecting balanced performance in both identifying relevant tables and minimizing false positives. These results underscore the importance understanding table relationships in multi-table retrieval tasks, area where MTR outperforms open-source counterparts. We also ablate the Question Decomposition (QD) module, and we found that QD plays a vital role in MTR. QD provides noticeable improvements over MTR multi-table retrieval tasks, for instance, in Top-2 retrieval, precision is improved from 65.3 to 72.3 and recall is improved from 62.3 to 64.7 7 in the 2-table subset.
8

---

Published as a conference paper at ICLR 2025
Table 6; The main results of LLMs on the 3-table subset.
Datasel
3 table
Evaluation Methods
Table QA
Text-to-SQL toL
PKS
FKS
Metrics
EM
PM
Rougel
RougeL
BLEU
Acc
Acc
Open Source I.MMs
Tableelamaa7b
7.42+0.1
8.12:0.11
8.96+0.
7.58+03
1.7740.1
16.17=02
11.57+00.2
TableLlama-7bi
7.82.0.1
8.38.0.1
9.36.10.1
7.92100
2.151001
18.05
13.57
Mistral-7b Mistral-7b i
5.27+0.1
5.91
3.72++0.1
2.46=0.2
1.68+0.1
16.86=011
12.58+00.
5,88+02
6.26=0.11
4.33++0.2
3.08+0.2
2.58+0.1
18.24=0.1
15,06++0.2
I.aMMA2--331
8.62+0.1
9.24
14.2240.2
12.75
4.79+0.1
21,27
20.09
LlaMA-2-13b
9.65+0.1
11.74
18.6649.2
15.73=002
6.29
24.37=02
22.58++0.1
Proprietary LLMs
G.T-3355'
20.66+02
24.29=001
31.48.0.3
27.39=0.3
2.67 #0.1
27.36 =0.2
18.3..
GPT-3.5+
24.64+02
28.55=#022
39.25+002
33.26=0.3
5.26 +0.1
40.16=0.1
25.01
GPTT-4"
27.16::02
31.48:10.2
33.13 13.132
29.28
2.6940.1
32,77=001
22.06+:0.2
GPT-44
28.58+002
33.21
35.26++0.1
30. 15.22
5.34
44.19=0.2
28.38++0.2
Gemini-pro"
24.25
28.59.0.11
29.78.10.
26.17 1.22
3.02.10.1
30.31 1.1.1
21.92.10.2
Gemini-prot
26.3810.1
30.88.10.1
33.44.10.1
29.25 1,02
4,88.002
38,85
31.52.10.2
Ol-preview"
42.37=00.1
45, 45.97=0.2
36.29 +0.2
35.73
5.28+0.1
41,89=0.2
32.32:+0.2
O1 O1-previewf
48.28+02
52.95=0.2
42.41::0.2
36.29
7.08 +0.1
46.84=0.1
40.78
Human
92.3
86.9
98.7
97.5
4.3 MULTI-TABLE REASONING EVALUATION
The main results of various LLMs presented in Tables and 6 Taking Table example, O1- preview outperforms numerous multi-table tasks on our MMQA benchmark, demonstrating superior performance across complex reasoning scenarios. Partieularly in the Table QA task (50.77 78 EM score), Primary Key Selection (49.53 accuracy), and Foreign Key Selection (34.17 accuracy), the O1-preview maintains a noticeable level of performance, significantly surpassing GPT-4, (28.88 EM score Table QA, 42.78 accuracy in Primary Key Selection, and 26.88 in Foreign Key Selection). Despite these advancements, proprietary and open-source LLMs still lag significantly behind human performance (89.8, 82.7, 96.5 and 95.3) on multi-table comprehension and reasoning tasks. Nevertheless, certain advanced LLMs, especially table-related LLMs, demonstrate potential in these scenarios.
Text-to-SQL. In the Text-to-SQL. generation task, L.MMs are assessed on their ability to translate natural language questions into accurate SQL queries. O1-preview with one-shot setting emerged as a top performer, with Rougel, RougeL, and BLEU scores of 43.62, 39.52, and 7.58 in Table respectively. This high score can be attributed O1-preiew's sophisticated language parsing and structured output generation capabilities, which are critical for understanding the semantic nuances of questions and mapping them onto the corresponding SQL syntax. GPT-4, while achieving respectable BLEU score, showed lower Rouge-L and Rouge-1 scores, suggesting that although it could capture the overall structure of the SQL query, struggled with the finer details of the query syntax. The Text-to-SQI. results reveal that while L.MSs are making strides in this area, there is still considerable room for improvement, particularly in generating queries that are not only syntactically correct but also semantically aligned with the original questions.
Multi-table QA. The multi-table qa task evaluates the models' capacity to retrieve evidence, navigate across multiple tables, and finally extract correct answers. O1-preview with one-shot setting exhibited an impressive Exact Match score of 50.78 and 48.28 Tables and showcasing its proficiency in comprehending complex, inter-table relationships to retrieve accurate answers. This performance is likely due to Model E's ability to effectively utilize foreign keys and primary keys to traverse between tables and identify the relevant data points. Conversely, GPT-4, despite solid showing in other tasks, only managed an Exact Match score of 28.88, and 28.58, respectively. in Tables 5land 6. suggesting that it encountered difficulties in integrating information from multiple tables to

---

Published as a conference paper at ICLR 2025
form coherent answer. The variance in performance across models highlights the complexity of multi-table reasoning and the need for LLMs to develop more sophisticated strategies for inter-table data integration.
Primary Key Selection and Foreign Key Selection. The accuracy of primary and foreign key selection is pivotal for establishing correct table relationships, which was tested in this task. O1- preview stood out with an accuracy score of 46.84% for Primary Key Selection and 40.78% for Foreign Key Selection in [ indicating a robust understanding of table schemas and the ability to accurately identify critical columns that facilitate data linkage across tables. These high scores reflect Ol-preview's s advanced feature engineering capabilities and its nuanced grasp of database structures. On the other hand, GPT-4 while competent in other tasks, achieved lower accuracy scores of 42.78% and 26.88% for primary and foreign key selection, respectively. This performance discrepancy could be due to GPT-4's less refined ability to discern the significance of certain columns within the context of table relationships. The results from this task emphasize the importance of precise key identification for effective multi-table data processing, an aspect that remains a challenge for many LLMs and warrants further research and refinement.
Impact of Table Length. We randomly select data with average table lengths of 500, 600, 700, 800, 900, and 1,000, sampling 50 samples for each type of data to evaluate the performance of LLM under different length tables.
an insightful evaluation of the Multi-Table QA and Text-to-SQL Generation tasks' performance across various table lengths. Figure reveals that there is an elbow point in the Multi-Table QA task. When the table length does not exceed 800 rows, the performance of LLMs decreases gently as the table length increases. But when the length of the table exceeds 800 rows, there is an elbow point where the performance of L.LM rapidly drops to a Generation task, matter how the length of the
significantly low level. While, in the Text-to-SOL table changes, the performance of LLMs has maintained a slow decline rate. This may be because, in Text-to-SQL tasks, LLMs only focus on the table header rather than the table content itself.
Tible 0A 3-table
Tabe QA 2-bbllble
55
Gentiiy
39-
O. gegrrii
30-
Oi O-geerizr
49
72
15
35
35
25
20
20
530
600
730
500
900
95n
table rows
Figure 4: The evaluation of different lengths of input tables on multi-table QA task. We divided MMQA into 6 subsets: tables' lengths of 500, 600, 700, 800, 900, and 1000. All LLMs are evaluated on a zero-shot setting.
5 CONCLUSION
We introduce a new MMQA benchmark for assessing of LLMs' capabilities in handling multi-table tasks. The extensive experiments conducted reveal both the promise and limitations of current LLMs in navigating complex, interconnected data. Although existing strong LLMs such as GPT- 4 and O1-preview showcase a strong performance on complex tasks, LLM still lacks the ability to comprehensively understand and reason over tables, especially in multi-table tasks, and lags significantly behind human performance in Multi-Tabh
Question Answering tasks and Foreign Key Selection tasks. As the field progresses, the MMQA dataset and its associated challenges will undoubtedly serve as a critical catalyst for innovation, driving the development of LLMs that can more effectively be reasoning across multiple tables, tackling weal-world data complexities.
10

---

Michael Günther, Maik Thiele, Julius Gonsior, and Wolfgang Lehner. Pre-trained web table em- beddings for table discovery. In Proceedings of the Fourth International Workshop on Ex- ploiting Artificial Intelligence Techniques for Data Management, aiDM '21, pp. 24-31, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978 7781450385300 doi: Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint 2:Xxv:2303308774. 2023.
Simran Arora, Avanika Narayan, Mayee F Chen, Laurel J Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. Ask me anything: A simple strategy for prompting language models. arXiv preprint arXiv:2210.02441 2022.
Peter Baile Chen, Yi Zhang, and Dan Roth. Is table retrieval solved problem? exploring join-aware multi-table retrieval. In Annual Meeting of the Association for Computational Linguistics, 2024. URL htttp:://ppl. htt.:pppa.sammmttsccscoaaa...rrr wtt.://s......r.r////////////1/0
Wenhu Chen. Large language models are few(I)-shot table reasoners. ArXiv, abs/2210.06710, 2022.
URL https://api Seenaanniccccoooarrrrr///orppsps
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang. SHIYANG LI, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. ArXiv, abbs/990.022144 2019. URL https: //api. wwmanticschollaat wwtp:/orgg//oiiiiiiioo 198917339
Wenhu Chen, Ming- Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. Open question answering over tables and text. ArXiv, abs/2010.10439, 2020a. URL https://api
Semanticscholar. wwtp//orrg///ao:.o.ooo 22:0000330022203066 224803601
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. Hybridqa: A dataset multi-hop question answering over tabular and textual data. Findings of EMNLP 2020, 2020b.
Zhiyu Chen, Wenhu Chen, Charese Smiley. Sameena Shah, Iana Borova, Dylan Langdon, Reema N Moussa, Matthew I. Beane, Ting-Hao 'Kenneth' Huang, Bryan R. Routledge, and William Yang Wang. Finqa: A dataset of numerical reasoning over financial data, ArXiv, abs/2109.00122 2021.
URL phttps://ap1 wwwanticscholar...... wtwpsshoo.ar..rr//rrrusSSDD:::: 235399966,
Z Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir R. Radev, Marilyn Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Bind- ing language models in symbolic languages. ArXiv, abs/2210.02875. 2022. URL https
//api. www.:esssaaiiccchol.o.o wrgg/ooppgssi....oo 252734772
Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo. Yan Gao, Shi Han, Jian-Guan Lou, and Dongmei Zhang. Hitab: A hicrarchical table dataset for question answering and natural language generation. arXiv preprint arXiv: 108.067/2 2021.
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. Turl. ACM SIGMOD Record, 51:33 40, 2020.
org/Corpus ottpap/eeeeaatatcccoooooooooorrr
220128303
Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. Active prompting with Chain-of-thoughe for large language models. arXiv preprint arXiv:2302.112244 2023.
00.1145334645093446488 3364892. URL
http///o.oorg/.... 100...44133449..... 3464892
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David A. Sontag. Tablim: Few-shot classification of tabular data with large language models. ArXiv, abs/2210.10723, 2022. URL httpp:://ppp...... wwwanticschollatt wwwpchhoaarrrgo/ooroo www.or////ooos..... 252999281
Jonathan Herzig, Thomas Müller, Syrine Krichene, and Julian Eisenschlos. Open domain question answering over tables via dense retrieval. In Kristina Toutanova, Anna Rumshisky, Luke Zettle- moyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the
11

---

Published as a conference paper at ICLR 2025
Association for Computational Linguisties: Human Language Technologies, pp. 512-519, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 maacc- maacl-main.44 m3.n.43.
URL hnttss://aclaathoolggy . org/ 2021. naanl-aaalnnnnnm
Hiroshi lida, Dung Ngoc Thai, Varun Manjunatha, and Mohit lyyer. Tabbie: Pretrained representations of tabular data. In North American Chapter of the Association for Computational Linguistics, 2021.
URL https://api semanti wttpaatttscccoooooooo/r//rr////s wrwpoooooss.: 233864627
Mohit Iyyer. Wen-tau Yih, and Ming- Chang. Search-based neural structured learning for sequential question answering. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (
(Volume 1: Long Papers), pp. 1122-11833 Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-11 167. UBLLhttts:///cclantiooooy/ wttp:///ggg..oo////111111
Sujay Kumar Jauhar, Peter D. Turney, and Eduard H. Hovy. Tabmeq: A dataset of gen- cral knowledge tables and multiple-choice questions. ArXiv, abs/1602.03960, 2016. URL
https://api wwwastmmmniiic semanticscoolaar http:///ssaaaaaaoaoooooooooooooa
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard I Lavaud, Marie-Ana Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, www.seemainnchhttoooeeeee
2023. URL UtLLnttpp///ppp..ennntcccooolll
www. wwtrg/CorpussD:: 263830494
Yannis Katsis, Saneem A. Chemmengath, Vishwajeet Kumar, Samarth Bharadwaj, Mustafa Canim, Michael R. Glass, A. Gliozzo. Feifei Pan, Jaydeep Sen, Karthik Sankaranarayanan and Soumen Chakrabarti, Ait-qa: Question answering dataset over complex tables in the airline industry. ArXiv, abs/2106. 12944, 2021, http::ttps::/ppptt semanticscholarr www.Corgroooooooooooooooooo PrrorpusD:::: 235623770
Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1203- 1203-1213, 1213, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v 1/D1 6- 128. URL /tRLittts///acaanthooo. www.gg..ooogoooooooo.......o.... org/D16-1 1128
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang. Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024.
Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. Table-gpt: Table-tuned gpt for diverse table tasks. ArXiv, abs/2310.00263.. 2023. URL httpt: www.snnticccoooororrrrro///////e 2ttp://Cooopss0:22127777711 264127877
Xiao Li, Yawei Sun, and Gong Cheng. Tsga: Tabular scenario based question answering. ArXiv, abs/2101.11429, 2021. URL http:: //api.
Yinghui Li, Qingyu Zhou, Yangning Li, Zhongli Li, Ruiyang Liu, Rongyi Sun, Zizhen Wang, Chao Li, Yunbo Cao, and Hai-Tao Zheng. The past mistake is the future wisdom: Error-driven contrastive probability optimization for Chinese spell checking. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 1202-3215 Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. URL https: httpss///cllaanhoooogyy or 2022. fen.ffiinnnnaaal.c 252
18653/v 1/2022. 1/222...nnnn.
Chin-Yew Lin, ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
URL https: http::////allathhllogoooo wwwphhooooooyooo///-0--00000
org/N00-1013
Shicheng Liu, Sina J. Semnani, Harold Triedman, Jialiang Xu, Isaac Dan Zhao, and Monica S. Lam. Spinach: Sparql-based information navigation for challenging real-world questions. ArXiv,
abs/2407.11417, 2024. URL /https://app. aeeemaniicchoo. wrtp://rrpss.::2211185553
Shuaiqi Liu, Jiannong Cao, Ruosong Yang, and Zhiyuan Wen. Long text and multi-table summariza- tion: Dataset and method. ArXiv, abs/2302.03815, 2023. URL hhtph:ttpss... //api. semanticscholar
org/CorpuSID: 256631 057
12

---

Published as a conference paper at ICLR 2025
Weizheng Lu, Jiaming Zhang. Jing Zhang. and Yueguo Chen. Large language model for table processing: A survey. ArXiv, abs/2402.05121, 2024. URL https:://app semanticscholar
org/CorpusID: 267548080
Nafise Sadat Moosavi, Andreas Ruckl'e, Dan Roth, and Iryna Gurevych. Learning to reason for text generation from scientific tables. ArXiv, abs/2104.08296, 2021. URL https://api
semanticscholar.c http://oaaarrrog/rroore/o orgg/oorpssI::2333330
Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904, 2022.
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojeiech Kryšciński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma. Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. FeTaQQA Free-form table question answering. Transactions of the Association for Computational Linguistics, 10:35-49, 2022. doi: 10.1162/tac a a_0466.
URL /ttpss//
accannhhlogyy.. htwhhooogy.oog///e/ org 2022. tac1-1.3
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02 2022. URL httphttp::/// //api. semant commaniicschlarr org. wttg//rrpssD::://0mmm 246426909
Vaishali Pal, Andrew Yates, Evangelos Kanoulas, and Maarten de Rijke. MultiTabQA: Generat- ing tabular answers for multi-table question answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6322-6334 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/11/223.acaaaao https://aclanthoology htt.aa/aaaaatoooooooo/////// acl long. .38
URL
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi:
1040
Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Annual Meeting of the Association for Computational Linguistics, 2015. URL https://api
semanticscholar wrg.:.orro/ooo /Ctgg///oooII::::: 9027681
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2019. URL https://api
semanticscholar wtt.:/scchooaaaoororrrrrorsssspa (0: 204838007
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/ D16- 1264. URL
1264
Ananya Singha, José Pablo Cambronero, Sumit Gulwani, Vu Le, and Chris Parnin. Tabular rep- resentation, noisy operators, and impacts on table structure understanding tasks in 1lms. ArXiv, abs/2310.103588 2023. URL https://appp www.stiiscoootttteoo. .rg. http//orppssDD... 264146582
Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, and Hiroya Takamura. Towards table-to-text generation with numerical reasoning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1451-1465. Online, August 2021. Association for Computational Linguistics. doi: 10. 186653/v1/222 long. 115. URL https
Taa.lannhology.orgg///o acl acl-llng........... 115
13

---

Published as a conference paper at ICLR 2025
Yuan Sui, Jiaru Zou, Mengyu Zhou, Xinyi He, Lun Du, Shi Han, and Dongmei Zhang. Tap4llm: Table provider on sampling, augmenting, and packing semi-structured data for large language model reasoning. ArXiv, abs/2312.090333 2023. WtL/https:://// httpp/pp....maacccccoooooooooo// org/CorpusID/ 266210509
Gemini Team, Rohan Anil, Sehastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Jian Wu, Linyi Yang. Yuliang Ji, Wenhao Huang, Börje F. Karlsson, and Manabu Okumura. Gendec: A robust generative weweettiiondccooosiitiioo method for multi-hop reasoning. ArXiv, abs/2402. 1166, 2024a. https: www.nticschol.eeeeeee org/CorppsID: ottp://.eaa..aaioooooooooooooora 267750855
Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang. Jiaheng Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, al. Tablebench: A comprehensive and complex benchmark for table question answering. arXiv preprint ar arXiv:2408.000 09/74, 2024b.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP). 2018.
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning. Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023. URL https: //api.:://ppp.ppp. www.seeeeaaenttoieeeee. or org/CorpusIDD 25641 6408
Tao Yu, Rui Zhang. Kai Yang, Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujü (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3911-3921, Brussels, Belgium, October-November 2018a. Association for Computational Linguistics. doi: 10.18653/v//D18- 1425. URL http:/tts///ccaattoooooo/ 001.1/o///oo//01--2233333
Tao Yu, Rui Zhang, Kai-Chou Yang. Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Z Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. Spider: A large-scale dataset for complex and cross- semantic parsing and text-to-sql task. ArXiv, abs/1809. 809008877 2018b. URL https://apii wet.aatttttcccoo...ooooorrooosoa
Tianshu Zhang, Xiang Yue, Yifei Li, and Huan Sun. Tablellama: Towards open large generalist models models for tables. ArXiv, abs/2311 1,0206, 2023.
26521 3406
Weijia Zhang, Vaishali Pal, Hu-Hongg Huang. E. Kanoulas, and Maarten de Rijke. Qfitt: Generating query-focused summaries multi-table inputs. ArXiv, abs/2405.05109, 2024a. |RLhttps:
//api. semppniismmtiiccchollooo .rg. wrgg/CoppsstDCD ID: ID: 269626608
Xiaokang Zhang, Jing Zhang. Zeyao Ma, Yang Li, Bohan Zhang, Guanlin Li, Zijun Yao, Kangli Xu, Jinchang Zhou, Daniel Zhang-Li, et al. Tablellm: Enabling tabular data manipulation by Ilms in real office usage scenarios. arXiv preprint arXiv:2403. 19318, 2024b.
Bowen Zhao, Changkai Ji, Yuejie Zhang, Wen He, Yingwen Wang. Qing Wang, Rui Feng, and Xiaobo Zhang. Large language models are complex table parsers. Conference on Empirical Methods in Natural Language Processing, 2023. URL http:///aap.ssmnniccco...o....oo org/ hcr//rrppss
266163842
Ruiqi Zhong, Tao Yu, and Dan Klein. Semantic evaluation for text-to-SQL with distilled test suites. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 396-411, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v www.://ee.l/p......eeee mainllppmmii..999 URL https://acllnnthooogy..rgg/ we:e0..mmnlppp.. emm.lpmmin......
14

---

Published as a conference paper at ICLR 2025
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103, 2017.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. TAT-QA: A question answering benchmark and textual content in finance. In Chengqing Zong, Fei Xia. Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 1 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Pupers), pp- 3277-3287, Online, August 2021. Association for Computational Linguistics. doi: (0.18653/v1/202) aal--lccl.255555 254. URL_hhtps://accaaanhooo.. wwwgy.org//ggg 2021 acl long. 254
hybrid of tabular
Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat seng Chua. Towards complex document understanding by discrete reasoning. Proceedings of the 30th ACM Interna- fional Conference Multimedia, 2022. URL. httpps://ppp www.nticschoollar......ocmm org/CorppsII)
251041071
Fengbin Zhu, Ziyang Liu, Fuli Feng, Chao Wang, Moxin Li, and Tat seng Chua. Tat-llm: A specialized language model for discrete reasoning over tabular and textual data. ArXiv, abs/2401.13223, 2024.
hhttps://ap smmmmniissstoooorrrrgg/rrrruuoop
15

---

Published as a conference paper at ICLR 2025
Test-recitt 2dable
Teshia-QQ 342hle
CBTA3
GF135
Gariiiip Clepoiin
Cleparicy
Rong
20
316
000
700
500
980
1000
300
000
135
2003
900
1000
tablc
table
Figure 5: The evaluation of different lengths of input tables Text-to-SQU Generation task. We divided MMQA into four subsets: tables' lengths around 500 (400-600), 700 (600-800), 900 (800- 1000), and 100 (1000-1200). All LLMs are evaluated on zero-shot setting.
A REPRODUCIBILITY STATEMENT
To make the results and models reproducible and verifiable, we provide our full data annotation guideline, data link, implementation details, and prompts: We detail the process of data annotation in section 3.1 and the implementations are Appendix C results illustrated in Appendix
All the prompts required to reproduce the
B PROMPTS
When evaluating large language models, prompting is brittle process wherein small modifications to the prompt can cause large variations the model predictions, and therefore significant effort should be dedicated designing a painstakingly crafted perfect prompt for the given task (Arora et al. 2022 Diao et al. 2023). In this study, We investigate the performance of zero-shot on our benchmark. To climinate the randomness, we manually select one demonstration for each task, ensuring that all tasks are covered.
We give our designed input examples for three different tasks to help readers understand imple- mentation, as shown in Table 7 respectively.
C IMPLEMENTATION DETAILS
For proprictary models, we employ official APIs to interact with exclusive LLMs and prompts are well-defined. For open-source models, all experiments are conducted on 8 A100 GPUs. For fine-tuning single-table-rettee rettivell models, we conduct supervised fine-tuning of TableLlama- and SGPT-5.8B on the single-table QA dataset. We set the initial learning rate 2e-5 and conducted training over three epochs. Optimization is performed using the Adam optimizer, with a batch size of 4 and a maximum input sequence length 4,096.
D CORRELATIONS OF PARTIAL MATCH BETWEEN GPT-4 AND HUMAN CHECK
We also check the person scores between GPT-4 Partial Match score and Human Check score. We randomly selected 100 data from MMQA (50 from the 2-table subset, 50 from the 3-table subset) and manually checked the partial match score. For answers generated by Ol-preview, GPT-4 gives the 53 partial match score which indicates that 53 answers can be aligned to ground truth. Human check gives 59 partial score which indicates the 59 answers can be aligned ground truth. We collect two lists with 100 elements, the element is "0" or "1". One list is the GPT-4 partial match score list and another is the Human Check partial match score list. We compute the Pearson Correlations between
16

---

Published as a conference paper at ICLR 2025
Table 7: The prompt templates of table-related tasks. We here take 2 table data as an example.
[WORDS] denotes the information we should provide.
Prompts of Question Decomposition
Prompt are an expert multi-hop question decomposition. you need to decompose the given multi-hop question [Question] based on the given example. Please only output the results without any other words in the JSON format of: "Subb-uestions":: List]."
[Question/ The given multi-hop question.
[Example] The given example of question and sub-questions.
Prompts of TeXI-to-SQL.
Prompt You are an expert at text-to-SQI., you need to generate SQL query based on the given multihop question [Question] and given tables [TABLEI], [TABLE2]. Please only output the results without any other words in JSON format of: ( ("SQL.": String)." [Question] The given multi-hop question.
[TABLEI] The given table 1.
[TABLE2] The given table 2.
Prompts of Multi-iable QA
Prompt "You are an expert at multi-table question answering, you need to extract answers based on the given multi-hop question [Question] and given two tables [TABLEI]. and [TABLE2]. Please only output the results without any other words in the format of: ("Answers": List). [Question] The given multi-hop question. [TABLEI] The given table [TABLE2] The given table 2.
Prompts of Foreign Key Selection
Prompt "You are an expert foreign key selection, you need select foreign keys based on the given two tables [TABLEE]],
[TABLFF2]. Please only output results without any other words the ISON format of: ("foreign keys": List).
[TABLEI] The given table 1.
[TABLE2] The given table 2.
Prompts of Primary Key Selection
Prompt You are an expert at primary key selection, you need to select primary keys based on the given two tables [TABLEI], and [TABLE2]. Please only output the results without any other words in the JSON format of: ("primary kcys": List].
[TABLEI] The given table
[TABLE2] The given table 2.
Prompts of Partial Match Evalutaion
Prompt You are Answer evaluator, you need to measure the semantic similarity between [Generated Answer] and [Gold Answer], and give the score, means equal, means not, Some answers may have abbreviations or alias, for example, Lionel Messi is equal to Messi, Donald Trump is equal to Trump. Please only output the score 1 or O without any other words.
[Generated Answer] The I.IM generated answer.
[Gold Answer] The ground truth.
the two lists. The results are illustrated in Table 8 and we find that the Human Check partial match score is highly correlated to the GPT-4 Partial Match score. For example, given the question "Find the number of male (sex is 'M') students who have some food type allergy." The answer is "10", while the generated answer is "ten". GPT-4 PM could treat "ten" as the correct answer. However, human partial match is better than GPT-4 PM, For example, the question is 'What are all the employee IDs and the names the countries in which they work?" One the answers is "CA", while the L.LM--geerrteed answer is "Canada". The GPT-4 PM assigned score is 0.
17

---

Published as a conference paper at ICLR 2025
Table Pearson Correlations between GPT-4 Partial Match Score and Human Check Partial Score.
Model
EM
GPT-4 PM
Human Check PM
Pearson Correlation
O1-preview
45.7
0.8852
GPT-4
31.6
41
45
0.8273
GPT-3.5
26.7
41
0.7784
E DIFFERENCE BETWEEN ORIGINAL QUESTIONS AND PARAPHRASED QUESTIONS
We randomly selected 100 questions (50 from the 2-table subset and 50 from the 3-table subset) that were paraphrased manually and sent the questions with corresponding tables into LLMs and evaluated Table QA tasks with EM score. The results illustrated in Table After paraphrasing. the table column-related information is reduced and the performance of LLMs also drops. For example, the original question is: "Show the name and number of employees for the departments managed by heads whose temporary acting value is "Yes"?" The paraphrased question is: "What are the names and number of employees of the department heads who are acting now?" The column-related information such as "temporary" and "managed" are eliminated.
Table 9: Performance between original SQL query generated questions and paraphrased questions.
Models
2-Table
3- T-Tble
Settings
Original
Paraphrased
Original
Paraphrased
O1-preview
43,5
40.7
39,8
34.4
GPT-4
29.6
25.8
26.2
23.6
GPT-3.5
26.3
23.1
24.5
21.9
F TEST SUITE ACCURACY EVALUATION FOR TEXT-TO-SQL
We utilize the ESM Zhong et al. 2020) for evaluating LLMs' Text-to-SQL performance on our 2-table and 3-table subsets. Table 10 illluustrtes that LLMs although get a relatively high pass rate of the ESM score, there are still large proportion of false positive SQL queries.
Table 10: The false positive/negative rate of the ESM metric.
Models
2-table
3-table
GPT-4
11.5/24.7
13.6/27.8
GPT-3.5
15.4/28.1
17.9/31.2
O1 O1-preview
8.7/19.4
11.3/22.6
G LIMITATIONS
In this paper, we focus on the evaluation LLMs' multi-table understanding reasoning ability on annotated counterfactual MMQA dataset. Although LLMs show obvious performance gap between humans, the evaluation methods remain improving. for example, the Exact Match is not sufficient for replying the real results. Secondly, although LLMs could generate SQL queries of relatively good quality, whether the generated SQL queries could be executed to get correct answers not is still unknown.
18


Reviews:
[{"review_id": "Veij7aYPiS", "reviewer": "Reviewer_P67K", "summary": "This paper introduces a new benchmark for multi-table multi-hop reasoning using question answering. This benchmark evaluates models on 5 aspects: multi-table retrieval, text-to-sql generation, multi-table QA, Foreign Key selection, and Primary Key Selection. They also provide 2 splits: one for questions where reasoning is needed across 2 tables, and another one across 3 tables (and hence more challenging). This benchmark is novel because prior works focus on single-table QA, while the proposed one focuses on multi-table, which is also a more realistic setup since it is common to perform unions and joins across tables in real-world scenarios. They conduct evaluations on multiple proprietary LLMs and open-weights LLMs and compare them with human and baselines and multiple retrieval methods. The paper aims to answer the following questions:\n\nHow well are LLMs at solving multi-table QA tasks compared to humans? -> there is still a large gap to close\nHow well can LLMs solve structured reasoning tasks such as Primary/Foreign Key Selection? -> only o1 achieves an accuracy > 40, which shows LLMs struggle on this subtask, which is key for proper multi-table reasoning.\nHow does the performance of LLMs decrease when the table sizes increases? -> when the table is too long, performance rapidly decreases indicating there is a (length, #rows) threshold where LLMs cannot understand tables well.", "questions": "* How good is the partial match evaluation? Does this metric correlate to a human partial match evaluation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Very complete benchmark for evaluating table reasoning in LLMs.\n* The benchmark includes different subsets for different difficulty levels, it also includes multiple subtasks, which can be useful to evaluate intermediate steps needed to solve multi-table QA.\n* Analysis of multiple open-weights and close-weights LLMs.\n* Includes short and longs tables, which can be useful for analysis of LLMs’ long-context understanding.\n* More generally, this benchmark can be very useful for future research on table reasoning.", "weaknesses": "* The paper uses a partial match evaluation that depends on GPT4 (which can be fine), but they don’t provide and evaluation of the quality of this metric.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hKHXgKwiFW", "reviewer": "Reviewer_PPNX", "summary": "This paper addresses the task of information retrieval and question answering from multiple tables, focusing on questions that require information from multiple tables and reasoning. For this task, the paper introduces the _Multi-table and Multi-hop Question Answering (MMQA) dataset_, along with several subtasks that multi-table QA systems might need to solve: Table retrieval, Table QA, Text-to-SQL, Primary key selection, and Foreign key selection.\n\nThis dataset is constructed by template-based synthesis of SQL queries from the tables in the Spider dataset, and converting them into natural language questions using GPT-4-Turbo, in addition to quality checks conducted by experts.\n\nThe paper also introduces _MTR_, a new method for the multi-table retrieval subtask. MTR uses question decomposition and both question-table relevance (using single-table retrievers from prior work) and table-table relevance, to tackle this subtask.\n\nFinally, the paper benchmarks several open LLMs (TableLLaMA-7B and Mistral-7B) and proprietary LLMs (GPT-3.5, GPT-4, Gemini-pro, and O1) on the MMQA dataset and conducts ablation studies to understand the effects of table length on the performance of various systems. These experiments show that there is a significant gap between the performance of LLMs and human experts on this dataset and task.", "questions": "1. In Table 1, what is the unit of \"question length\"? Is it measured in characters?\n\n1. Line 199: How do you measure the number of reasoning steps?\n\n1. What is the input for the Table QA, Text-to-SQL, \"Primary Keys Selection,\" and \"Foreign Key Selection\" tasks? Are the gold tables used as input?\n\n1. In Table 4, what is the difference between, for example, \"TableLlama-7b\" and \"MTR (TableLlama-7b)\"? Is it just the question decomposition part done by GPT-4-Turbo, or is their training data also different?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper identifies a task where large language models (LLMs) are significantly behind humans, and open models lag behind closed commercial models (like O1). This is quite valuable for the research community to continue developing better LLMs.\n\nMMQA has a comprehensive set of question categories, including \"list/count all answers,\" which are often overlooked in TableQA datasets.\n\nThe experiments are thorough, and include a good mixture of open and closed LLMs. I imagine future research will be able to easily compare their systems against these baselines.", "weaknesses": "## 1. The MMQA Dataset is Synthetic\nTo construct MMQA, this paper generates SQL using templates and then converts them to natural language using an LLM. There are two potential downsides to this approach. One is that synthetic evaluation and test sets pose the risk of overestimating the performance of models [4]. This may occur due to the lower diversity of natural language questions or the fact that, since they are generated from SQL, they may be too close to SQL or otherwise \"unnatural\" compared to questions that users of such systems may ask. \n\nFor example, the question in Figure 1, \"What are the distinct creation years of the departments managed by a secretary born in state Alabama?\" uses the terminology from the tables (\"creation,\" \"department,\" \"state\") and the SQL keyword \"distinct.\" In a more realistic setting, a user might instead ask, \"When were the departments managed by an Alabamian secretary established?\". Given that the user is most likely not aware of column names in the database. Tackling this setting involves additional challenges like mapping the terms users use into column names and dealing with ambiguities.\n\n\n## 2. Evaluation Metrics Need More Thought\nUsing ROUGE and BLEU for SQL is not really appropriate. To compare SQL queries directly, exact match is often used. Execution accuracy is another commonly used evaluation metric in the literature. It involves executing the resulting SQL against the database and comparing its answer against the answer from the gold SQL. Please note that the original Spider dataset paper uses these two metrics. \nEven more suitable metrics are discussed in [3] and [2]. For example, [2] proposes a generalization of execution accuracy, which could be useful for evaluating \"list\" and \"count\" questions in MMQA.\n\nUsing Exact Match and Partial Match for QA underestimates the performance of models. This is especially noticeable when evaluating LLMs (as this paper does). Also, I wonder how these metrics handle \"list\" answers, given that they are quite sensitive to order. Or if a system is off by one when answering a \"count\" question because it missed one of N rows, shouldn't it get a partial score? See [5] for an example of how using an LLM judge to compare a model's predictions against the gold answer can help alleviate this problem.\n\nIn addition, it might make sense to add a unified text-to-SQL and TableQA subtask. This means that a system proposed in future work can answer the question in any way it chooses, either by generating and executing an SQL query or by directly reading the tables, and the final answer is evaluated against the gold answer. This would have the additional benefit that text-to-SQL methods and TableQA methods can use this dataset and compare their work across the two lines of research.\n\n## Other minor issues\n- In Algorithm 1, it should also be specified that a single-table retriever is used, in the same way it mentions that GPT-4-Turbo is used. Additionally, “outputs” are not sub-questions but tables.\n\n## Suggestions\n- Please consider adding a setting for directly benchmarking text-to-SQL without first retrieving tables. It is conceivable that in the near future, as long-context LLMs improve, they can generate SQL directly given the entire schema as input.\n\n- To improve clarity, there could be a section dedicated to describing the various subtasks. For example, Section 3.3 could be renamed and dedicated to “subtasks.”\n\n- MTR uses both query-table relevance and table-table relevance. I’m curious about the impact of each on the final quality of the retrieval system. An ablation study here would be informative.\n\n- [1] converts BIRD and Spider text-to-SQL datasets to the open-domain format, where table retrieval is needed. What is the difference between that and MMQA? This could be added to Table 2 and Figure 3.\n\n- There are many other Question Decomposition papers e.g. [6]; using one of them in MTR would be interesting.\n\n\n**References:**\n1. Is Table Retrieval a Solved Problem? Exploring Join-Aware Multi-Table Retrieval\n\n2. SPINACH: SPARQL-Based Information Navigation for Challenging Real-World Questions\n\n3. Semantic Evaluation for Text-to-SQL with Distilled Test Suites\n\n4. AutoQA: From Databases To QA Semantic Parsers With Only Synthetic Training Data\n\n5. Evaluating Open-Domain Question Answering in the Era of Large Language Models\n\n6. Break It Down: A Question Understanding Benchmark", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JRL36bDYAH", "reviewer": "Reviewer_VktP", "summary": "This paper introduces MMQA, a novel multi-table multi-hop question-answering dataset. The authors randomly sampled from the Spider dataset and prompted LLMs to generate natural language questions. The synthesized dataset then went through human experts' manual verification. Compared to existing table question-answering datasets, MMQA features a more diverse task collection, longer reasoning chains, and evaluations on different granularities. Extensive experiments run on MMQA indicate that there is still a large performance gap between the current SOTA LLMs and humans. The authors also propose a strong baseline, MTR, for the multi-table retrieval task. Additional insights and analysis on the evaluation results are also provided.", "questions": "## **1. What kinds of annotation disagreement occurred?**\n\nIn 3.1 the authors described the annotation quality verification process. I was wondering what kinds of disagreement patterns were observed. Were the annotators disagreeing with each other because they made mistakes themselves all the time? Was there any case that was due to query ambiguity or GPT-4-turbo's incorrect NL translation? How did you manage those situations, if any?\n\n## **2. Did you notice any bias, information loss, or noises introduced by GPT-4-turbo as the query translator?**\n\nOne concern about GPT-4-turbo translating the template-based queries is that GPT-4-turbo might introduce bias (e.g., some wording patterns get overrepresented), information loss (e.g., the semantics in the SQL might not be fully faithfully translated into NL), or noises (e.g., hallucinated or ignored columns/entities/tables/etc). Did you observe any of such risks and how did you handle them? Would it make sense to also include the caveats and potential issues with the approach in e.g. appendix?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "## **1. The task is novel and important**\n\nThe task of multi-table question answering is novel and important because it is very common in real product environments to consider multiple tables together, yet previous table question answering datasets have mainly focused on single-table scenarios. MMQA is able to close this gap. From this perspective, I believe the paper is well motivated and the problem is realistic and important. \n\n## **2. The quality of the dataset is high**\n\nThe dataset contains 3,312 expert-verified tables with 7.61 average reasoning steps and a diversity of task types. The experiment results show a large gap where current SOTA LLMs such as OpenAI o1-preview only achieve around 50% success on MMQA evaluation metrics while humans achieved 80-90%. \n\n## **3. A strong baseline is proposed**\n\nThe authors also proposed a strong baseline, MTR, that significantly (around 10-15% better on retrieval metrics) outperformed existing LLMs on the multi-table retrieving task. This model can serve as a good reference for future models on MMQA.\n\n## **4. The paper is in general well written**\nThe paper is in general clear and easy to follow. The logic flow is straightforward and clearly presented. The authors also submitted supplementary dataset samples that give a sense of what the dataset will look like, which is helpful.", "weaknesses": "## **1. Diversity and naturalness of queries in the dataset**\n\nThe dataset is constructed by sampling from 45 SQL templates in the Spider dataset and converting them into natural language using GPT-4-turbo. This approach inherently limits the diversity of query types. Additionally, due to the synthetic nature of the dataset, its distribution does not fully align with that of real-world user queries.\n\n## **2. Baseline models tested are relatively weak**\n\nThe current baselines are mostly general-purpose LLMs except TableLlama, which is a smaller 7B model. I would recommend adding more recent LLM-based table question answering agentic systems as baselines, such as the ones on top of the Spider leaderboard: https://yale-lily.github.io/spider", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)