Large Language Models are Complex Table Parsers
Authors: Bowen Zhao
Changkai Ji
Yuejie Zhang
Wen He
Yingwen Wang
Qing Wang
Rui Feng
Xiaobo Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Complex Table QA
GPT-3.5
Large language models

Abstract: With the Generative Pre-trained Transformer 3.5 (GPT-3.5) exhibiting remarkable reasoning and comprehension abilities in Natural Language Processing (NLP), most Question Answering (QA) research has primarily centered around general QA tasks based on GPT, neglecting the specific challenges posed by Complex Table QA. In this paper, we propose to incorporate GPT-3.5 to address such challenges, in which complex tables are reconstructed into tuples and specific prompt designs are employed for dialogues. Specifically, we encode each cell's hierarchical structure, position information, and content as a tuple. By enhancing the prompt template with an explanatory description of the meaning of each tuple and the logical reasoning process of the task, we effectively improve the hierarchical structure awareness capability of GPT-3.5 to better parse the complex tables. Extensive experiments and results on Complex Table QA datasets, i.e., the open-domain dataset HiTAB and the aviation domain dataset AIT-QA show that our approach significantly outperforms previous work on both datasets, leading to state-of-the-art (SOTA) performance.

Large Language Models are Complex Table Parsers
Bowen Zhao', Changkai Ji?, Yuejie Zhang', Wen He', Yingwen Wang', Qing Wang', Rui Feng! Xiaobo Zhang
School of Computer Science, Shanghai Key Laboratory of Intelligent Information
Processing, Fudan University, Shanghai 200433," Academy for Engineering and Technology. Fudan University, Shanghai, Children's 's Hospital of Fudan University, National Children's 's Medical Center, Shanghai, China
221100222,. fudan. cn, (fengrui, yjzhang, hwwen y@f edu. cn, @ 163.com
Abstract
With the Generative Pre-trained Transformer 3.5 (GPT-3.5) exhibiting remarkable reasoning and comprehension abilities in Natural Lan- guage Processing (NLP), most Question An- swering (QA) research has primarily centered around general QA tasks based on GPT, ne- glecting the specific challenges posed by Com- plex Table QA. In this paper, we propose to in- corporate GPT-3.5 to address such challenges, in which complex tables are reconstructed into tuples and specific prompt designs are em- ployed for dialogues. Specifically,, we encode each cell's hierarchical structure, position in- formation, and content as a tuple. By enhanc- ing the prompt template with explanatory description of the meaning of each tuple and the logical reasoning process of the task, we effectively improve the hierarchical structure awareness capability of GPT-3.5 to hetter parse the complex tables. Extensive experiments and results on Complex Table QA datasets, i.e., the open-domain dataset HiTAB and the avi- ation domain dataset AIT-QA show that approach significantly outperforms previous work on both datasets, leading to state-of-the- art (SOTA) performance.
facouive
CSV
Teoept Tomplieee
Figure 1: The overall architecture of complex table pars- ing. achieves through table structure reconstruction and prompt template designing based on GPT-3.5. Texts in the pink background are prompts, where the bold text can be replaced; texts in the yellow background mean the format of the tables in the original dataset; and texts the orange background represent tables after refac- toring. GPT-3.5 indicates the model text-davinci-oe OOL indicates that the number of tokens we requested exceeds the length limit of GPT-3.5. Code represents piece of Python code insert to assist multi-turn QA.
1 Introduction
Complex tables, characterized by multi-level struc- ture headers and numerous merged cells, are a prevalent data format that includes diverse data types and intricate associations among cells (Lim and Ng, 1999; Chen and Cafarella. 2014: Jin et al., 2022). In this context, the Complex Table QA task emerges as an important and challenging problem within the field of NLP.
studies focused on improving the conversion of questions into logical forms (e.g.. SQLs and code) that could be directly executed on tables to retrieve answers (Jin et al., 2022; Dong et al., 2022). To this end, wide range of strategies has been in- troduced, such as reinforcement learning, memory enhancement, type awareness, relationship aware- ness, (Pasupat and Liang, 2015; Zhong et al., 2017: Liang al., 2018: Yu al., 2018; Yin al., 2020). recent years, there has been a notable progress of Large Language Models (LLMs), such as BERT (Devlin et al., 2019), GPT (Radford et al.,
In traditional Table QA tasks, the majority of re- search efforts focused on simple flat tables (i.c..
tables with single-level column headers and no merged cells). Earlier, a substantial number of
"Corresponding author

---

2018), GPT-2 (Radford et al., 2019), RoBERTa (Liu al., 2019), GPT-3 (Brown al., 2020), and T5 (Raffel et al., 2020) in the field of NLP. This enable the Table QA tasks to generate answers di- rectly without the need for intermediate logical forms, which leverage the rich language representa- tions and knowledge acquired through large-scale text pre-training (Herzig et al., 2020; Chen al., 2020; Eisenschlos et al., 2021; Yang et al., 2022; Chen, 2023).
code snippet facilitate the answering process. Through careful study and meticulous experimen- tal analysis, we conclude that GPT-3.5 can be a great parser for complex tables. To sum up, the contributions of this paper are:
We present novel approach that leverages GPT-3.5 a parser address Complex Table QA tasks, which enhances the ability of the model to perceive the hierarchical structure of complex tables by restructuring the data format and devising prompt templates.
While significant attainment has been made in the studies above-mentioned, their primary empha- sis has been on the development of simple flat ta- bles, overlooking the ubiquitous complex tables. Although (Katsis et al., 2021; Cheng et al., 2022) endeavored construct QA datasets, specifically tailored for complex tables, and evaluated the per- formance of the SOTA Table QA models, the out- comes have not met expectations.
We resolve the constraint on the input token limitation for GPT-3.5 by crafting single-turn and multi-turn dialogue prompts tailored to complex tables of different lengths, as well as incorporating code snippets for assistance in multi-turn dialogue.
Most recently, with the advent of ChatGPT', an advanced NLP model derived from GPT-3, has showcased remarkable capabilities in generation (Maddigan and Susnjak, 2023; Dong et al., 2023;
Extensive experiments are conducted on both pubic benchmark datasets (i.e., HiTAB and AIT-QA), and the results show that our method outperforms the SOTA methods.
Liu al., 2023a), contextual understanding (Bang
et al., 2023; Gao et al., 2023b; Amin et al., 2023),
2 Related Works
and reasoning (Liu et al., 2023b; Gao et al., 2023a). has profoundly impacted on the field of NLP. A multitude of research actively explores and lever- ages the comprehension and generation abilities of ChatGPT for QA tasks (Tan al., 2023; Zuccon and Koopman, 2023; Wang et al., 2023; Singhal al., 2023). However, we have not come across any relevant work that harnesses the competencies of GPT-3.5 for Complex Table QA task as yet.
2.1 Complex Table Question Answering
Complex table QA tasks refer to information re- trieval and answer generation complex tables containing multi-level headers and large num- ber of merged cells. In previous work, most re- search has focused on the simple flat table dataset, such as SQA (Pasupat and Liang, 2015), Spider (Yu et al., 2019), Hybridq (Chen et al., 2020), Fe- TaQA (Nan et al., 2021), etc. Recently, (Katsis et al., 2021) introduced the domain-specific Ta- ble QA dataset AIT-QA, which consists of 515 QA pairs authored by human annotators on 116 tables, and experimented with the most popular Table QA models, such as TaPAS (Herzig et al., 2020), TaBERT (Yin et al., 2020), and RCI (Glass et al., 2021), but the results were not satisfactory. (Cheng et al., 2022) also proposed a new dataset HiTab, which contains 10,672 QA pairs based on 3,597 complex tables from open -pmn-ddmain, and ex- amined with Table QA models, but the results fell short of expectations. Furthermore, they proposed serrarhy-awwar-baaee method, which improved the performance on complex tables with the models trained on simple flat tables. However, the perfor- mance of these models on simple flat tables still outperformed the hierarchy serrarchy-wware. aware- based method.
In this paper we first attempt incorporate GPT- 3.5 to achieve complex table parsing, in which com- plex tables are reconstructed into tuples and spe- cific prompt designs are employed for dialogues. As illustrated in Figure 1, we reconstruct the table stored in JSON format into tuples, incorporating the hierarchical information, position information, and the value of each cell into different elements of the tuple. Subsequently, we fill in and calculate the length using the designed single-turn dialogue prompt templates. If the token count does not ceed the limit of GPT-3.5, we adopt a single-turn dialogue scheme for QA task. Otherwise, we uti- lize multi-turn dialogue scheme, in which we break down the question into sub-questions based on the logic of the single-turn prompt and introduce a

---

steck
Rlerarchicnl
Crock options
Index
mortt
Col
now Ponision Index
CSV
5283
67O
Riscazchiool 2av
Row
Raw
Indax
Posttion Endox
Figure 2: The illustration of table reconstruction. First, we rewrite the rows and column headers of the input JSON-formateed table into a tree structure, where, except for the root node, all nodes have unique precursor node. Morcover, encode the "Hierarchhiall-ndee"" of and column headers according the hierarchy sequence number of the tree. For example, "Col 1" indicates that 2018, 2017, 2016, the three headers are all located at the first layer column header, the encoding
addition, we encode the row and column position information of the header, for example, the "Column Position Index" of 2018 is O, the "Column Position Index" of 2017 is 1, etc. Finally, we encode cach cell in the table based on "Hierarchical-Indee Position Index". In the refactored table, the "Non-header" represents the cell other than the "Column Header" and "Row Header", and the "Title" represents the table title.
"Row Position Index", and "Column
Different from them, we propose to reconstruct the table format and design suitable prompt templates to fully unleash the comprehension and reasoning power of GPT-3.5, which exhibits outstanding per- formance on both complex table datasets.
Method
In this section, we reconstruct the table from JSON- formatted to tuples (Section 3. ). To better leverage the reasoning and generation capabilities of GPT- 3.5, we introduce the single-turn prompts designed for tables that do not exceed the API input limit on tokens (Section 3.2) and the multi-tun prompts for tables that exceed the limit. Moreover, we add a piece of code to assist question answering in multi- turn QA (Section 3.3). Here, we use Python for coding and calling API.
2.2 Prompt Engineering
Prompt Engincering refers to making better use of the knowledge from the pre-training model by adding additional texts to the input. In the era of LLM, (Wei et al., 2023) proposed a new method called chain-of-thought (CoT), series of inter- mediate reasoning steps to unleash the power of LLMs to perform complex reasoning. Soon, (Ko- jima et al., 2023) proved LLMs be an effec- tive zero-shot reasoners by incorporating "Let's think step by step" before each answer. (Zhang et al., 2022) also proposed Auto-CoT (automatic chain-of of -f-thhgghh), which could generate prompts automatically. However, the power of LLMS as good reasoners has not been applied to implement complex table parsing. In this paper, we specifi- cally focus on designing the appropriate prompts to make full use of the remarkable understanding and reasoning capabilities of GPT-3.5 to realize the structure- umncture-awareness of complex tables.
3.1 Table Reconstruction
As shown in Figure 2, we reconstruct the table from fooomttttt to tuples. For the row and column headers, we encode each cell as a five-tuple. The first element in the tuple serves as the label to in- dicate that it represents the row or column header (T stands for column header and L stands for row header). The second element denotes the hierar- chical index of the cell, while the third element represents the ordinal number of the cell's start- ing row column. The fourth element signifies the cell's ending row or column number, and the fifth element encapsulates the cell's value. For non- header cells, we utilize a quadruple to depict each

---

Example Single-Pue Dialogue
Regulations
(Roia play!
Suppose expert
17able Sescripsien]
The table is described an follows:
[Exarpice]
For examples!
(7,1,0,9,9) represents
Example of Rolti-Tucm Disicqua
tat's think stap by step
Exaeple Malti-Puen Dialogue
Requlatione
1Output Control]
quastion
Ioola
Iapat Context
Pappese you aze on expect
Input
tande desorsted as foll follown:
"Colume beadac"; "L...]"
"Ace heades" "[...""
Irible mitlo]
ecumplest
[colunn Beader]
(7.1.0,3.gl represente
calima hasdec: 1115,0,0, 0,4, '2eas snied perember 3h, "11"
queetion:
column header:
Zepet Custent
0,, '....!'
[Acessant kivwords)
[nader]]
g: Find Bottiwett 's APM in 2018
nas-dasder: 1100 18,0, 1225, 1135, 345-) ...
(ar '13"
zow header: "1',,,0,0,0,, D.
Discrell
Octpac
KOTT anower queotion
[Ko--Badde]]
Tayaceds:
tiglet
Ducotion::
nce haadee:::: "['C,,0,,,200',....
Saethwert, 10V, 2012
2014
[Doestiinn)
Ispet: Costact 92
A:
q: "What age Chief Operating
officer of Alaska?
1d0lect Teateral
</>
Dode
Outgut
"Cuism beader") "I..''"
Dacember 24, (7.1,5,0 (20191)
"the Inader": "...""
beadect 2.2215.11 "Jtrvenue parmuspur
Output
(all: 16.25,0 221, 265))
MS5
Operation: Netrieve the volue
Colume header: (2.0,1,1.App)
Octps: #2
Ansver: 133.365
Row header: 1,, Benito Minioaccs)
Call: (c,1,1,51)
Colaze bandec: (2, 0,0,4 ended
Operation:
Answer: 51
silas (AP9A) (4n aillions)
(a) '?
(a) An example of Single-Turn Dialogue.
(b) An example of Multi-Ture Dialogue.
Figure 3: Examples of Single-Ture Turn and Multi-Turn Dialogue.
cell. The first element in the tuple designates the label, indicating that it pertains to a non-header cell. The second element represents the row number of the cell, while the third element signifies the col- umn number of the cell. Lastly, the fourth element encapsulates the value of the cell. For example, (I., 0, 0, 3, "Compensation cost:") means that the tuple is a row header with hierarchical-inder O, starting row position index O, ending row position index 3, and the value is "Compensation cost:".
3.2 Single-Turn Prompt
In this section, we adopt a single-turn dialogue scheme for complex tables that do not exceed the limit of the API input token. As shown in Figure 3(a), we reformulate the prompt of single-turn dialogue into Regulations and Input Context.
Regulations aim to dictate and guide the be- havior of GPT-3.5 make more capable of reasoning on complex tables. In Figure 3(a), we make the role assumption of GPT- 3.5, and the reconstructed table is described detail and illustrated with examples. Also, we provide a detailed description of CoT for the entire Complex Table QA task to drive GPT- 3.5 starting at the top level header, to find the relevant sub-headers layer by layer and locate the non-header cells by the location informa- tion in header tuples. is worth noting that in the part of "Output Control", we require the model to output the selected "Column header".
We successfully integrate the category informa- tion, position information, and value of each cell in the complex table, as well as the hierarchical information of the row and column headers into the tuple by table reconstruction. Thus, not only the information of the whole table clearly expressed, but also the problem of token numbers beyond the limit caused by the information redundancy of the original format can be solved.

---

"Row header", "Cell", "Operation" and "An- swer" in return. In this way, we can better evaluate the reasoning and understanding abil- ities of the model (See Appendix A.1 for more details of single-turn prompt).
sites render of upet living le acrarget ten rimifizensll sigher probibiiitt liring fiod inseccie sescelalll, main socs
A: Towe simniticantly gestubility living food insoesse bestibald.
vine
Input Context contains a large number of fill slots, which can be filled at: "title: [7", "column header: 1J", "row header: Ul", "Q: according to different QA pairs, where "title" represents the title of the table, "col- unn header" and "row header" refers to the column headers and row headers of the table respectively. and "Q" denotes the question. Give the input in Figure 3(a) as an example, "title: [tab-10 10ab-102""Ma is generated by filling "title:
Figure 4: Schematic comparison of the answers gen- crated by GPT-3.5 with the original answers. GPT-3.55 represents the answers generated by text-davinci-003. Ori represents the standard answer in the dataset. Texts in the green background represent the question. Texts in the blue background represent the answer.
Dotasct
number
Tabll
QA
Avcrage numober
of bokens
Domala
Dev
HIDAB ATTIQA
7,417
1871
Open
spectfic
[]" with "tab- 102"
Table I: Dataset statistics. Average number of tokens represents the average number of tokens after tokenizing the table in the original format when calling API.
3.3 Multi-Turn Prompt and Code Assistant
In multi-turn dialogue, as shown in Figure 3(b), similar to the single-turn dialogue, we partition the prompt into two components: Regulations and In- put Context. However, we divide the dialogue into three turns with four modules. More specifically. we not only split the Chain-of-thought in the Regu- lations part of the single-round dialogue and ussign it to the three turns of the multi-turn dialogue, but also move the Output Control part the last turn of the dialogue. piece code in the third module to assist the cell selection.
mitigating the relatively slow API calls, and significantly enhances the results by reducing the accumulation of errors resulting from the multi-turn dialogue.
In the fourth module (i.e., the third prompt turn), we prompt GPT-3.5 for all relevant row headers, column headers, and non-header tu- ples, ask questions, and require the model to answer them in our prescribed format.
worth noting that we add a
In the first module (i.e., the first prompt turn), we extract the keywords from the question.
4
Experiment
In the second module (i.c., the second prompt turn), we pick the relevant tuples in the row and column header tuples and record them based the keywords select in the first prompt turn.
4.1
Datasets
We evaluate our approach on Complex Table QA task with HiTab (Cheng et al., 2022) and AIT-QA (Katsis et al., 2021). As shown in Table 1, we provide the statistical results of the datasets and calculate the average length of the tokenized table. Moreover, it be seen in Table that the AIT-QA dataset is divided into four subsets. According to whether the question related to Key Performance Indicators (KPIs), the dataset is divided into "KPI- driven" and "Table-driven". Similarly, AIT-QA is also divided into "Row header hierarchy" and "No row header hierarchy". according whether the answer relies the row header hicrarchy.
In the third module (i.e., the code assistant module), we incorporate a code snippet to fa- cilitate the dialogue. Specifically, we pass the row and column header tuples selected in the second round of dialogue into the third module, extract the row and column position information in these tuples through the code, and retrieve the non-header tuples of the ta- ble based on this position information. All the tuples matching the location information are returned and passed to the last module. This optimization expedites the experiment by
It is worth noting that the data in HiTAB comes from statistical reports across 28 different fields and Wikipedia, offering rich table information with

---

complex structures. In contrast, the data in AIT- QA are all selected from airline companies, and the information and hierarchical structures contained in the tables are relatively simple compared to HiTAB.
Method
Dev
Test
Weak Supervision (Cheng al., 2022)
MAPO w. original legical form
31.9
29.2
TaPas wh. legical foem
39.7
38,9
MML ba logind fonm
38.9
36.7
REINFORCE v has logical form
42.7
38.4
MAPO w. hss Intisal form
43,5
40.7
Partial Supervision (Cheng al., 2022)
TaPus wio legical feem
41.2
40.1
4.2 Baselines
MML ha logioll form
45.4
45.1
Since we have not found any further work related to HiTAB and AIT-QA, we compare our work with the evaluation results in their original paper, including 1) supervision-basel method: MAPO (Liang et al., 2018), TaPas (Herzig al., 2020), MML (Dempster al., 1977) and REINFORCE (Williams, 1992); and 2) zero-shot-based methods: TaBERT (Yin et al., 2020), TaPas (Herzig al., 2020) and RCI (Glass et al., 2021).
REINFORCE,
logical form
44.0
39.7
MAPO w. ha logical form
44.8
44.3
Zero-shot
Ours GPT-3.
49.0
50.0
Table 2: Accuracy on dev/test of HiTAB. h.a. stands for hierarchy-aware.
Models
Thita subset
TaBERT
TaPaS
RCI
Single-tur
Multi-turn
Kiiv-driven
41.37
48.26
60.00
76.92
66.67
74,48
Tablo-drivan
31.08
76.67
80.00
Row header herrarcyy
21,92
47.26
45 89
61.72
61.11
61.64
4.3 Evaluation
hicrabby
38.75
54.20
82.13
75.95
Because of the presence of a substantial number non-numerical type answers in the datasets, di- rect alignment or similarity evaluation cannot effec- tively assess our method. As illustrated in Figure 4, it is evident that the answers generated by GPT-3.5 have essentially the same meaning the original answer. However, there exists a notable difference in their representation. The model model-generated an- swers tend to provide more intricate details, while the original answers exhibit a more concise nature. Therefore, we use Accuracy as our evaluation met- ric following (Cheng et al., 2022), which indicates the percentage of QA pairs with correct answers, to align whether the generated answers equiva- lent to the original answers in the case of a specific question and context (i.c., table content).
Overall
33.08
49.30
51 84
76.73
70.27
76.36
Table 3: Accuracy on AIT-QA. We compare with other baselines: TaBERT, TaPaS, RCT (Katsis et al., 2021).
HÎTAB. Among the partial supervision methods on HiTAB, our approach still outperforms previous works by a large margin. Table 3 reports the results AIT-QA. In the context of shoo learning, the accuracy of our method outperforms TaBERT, TaPas, and RCI by 42.28, 26.94, and 24.42 on the overall dataset, respectively. Combining the results on these two datasets, can be demonstrated that GPT-3.5 can achieve the parsing of complex tables given appropriate data format and prompt.
4.5.2 Results HiTAB
4.4 Model
As shown in Table 2, our method outperforms all previous methods by over 3.5 in accuracy across the board. At the same time, it can be seen from Table 4 that, the results of the single-turn dialogue are significantly higher than the overall results. These indicate that our approach of leveraging the Large Language Model as a parser for complex tables is effective. In multi-turn dialogue, our method achieves 43.5 and 47.0 accuracy on the Dev and Test sets, respectively. Notice that, our work is built upon the framework of zero-shot learning, differing from the supervised methods utilized in previous studies. Even if our results may not consistently surpass those of these previous works, we assert our method is still comparable to the SOTA reported in the original paper (Cheng et al., 2022) Although the training set is also applicable, we find performs slightly worse partially due to the
We conduct experiments mainly with the text- davinci-003 from OpenAI, which is improved on GPT-3 and can understand as well as generate nat- ural language or code. Text-davinci-003 supports 4,097 input tokens at most, which means that the combined count of the input tokens and the gener- ated tokens can not exceed 4,097.
4.5 Results
4.5.1
Main Results
The results on HiTAB are shown in Table 2. Specif- ically, the absolute accuracy improvement of 5.5 on the Dev set and 9.3 on the Test set. Specifically, the absolute accuracy improvement of 5.5 on the Dev set and 9.3 on the Test set can be observed if com- pared to the previous best weak supervision method MAPO with the hierarchy-awee aware logical form on

---

Train
Dev
Test
Overall
Accuracy
Idn
Single-tum
52.0
51.1
51.1
51.7
HiTAB
Multi-turn
40.8
43.5
47.0
42.1
Table w. spt & w/o ref
20.3
9.9
Overall
48.9
49.0
50.0
49.3
Table w. mpt & w:
49.3
0.6
AIT-QA
Table 4: Accuracy of different dialogue schemes on trin/deeveest/vverall of HiTAB.
Table w. spt & w/o ref
64.7
14.2
Table w. mpt & w. ref
76.3
0.2
dataset bias.
Table 5: Ablation sresults on HITAB and AIT-QA. spr represents simple prompt. mpn represents the prompt that combines the information of the reconstructed table with the CoT design of Complex Table QA. ref repre- sents SSON-formatted table. Idn stands for don't know". We ask the model to answer "I don't know" if it could not infer the answer based on the existing context in the prompt. Please refer to Appendix A.1, A.2 and A.3 for more details.
4.5.3 Results on AIT-QA
We divide the AIT-QA dataset into four subsets according to the method in the original paper (Kat- sis et al., 2021), and analyze them separately, as shown in Table 3. Overall, compared other mod- els trained on tabular data, our method soundly outperforms all previous works. The overall accu- racy of single-turn dialogue reaches 76.73, and of multi-turn dialogue, reaches 70.27, which is sig- nificant leap compared to the 51.84 accuracy of the the previous best method RCI (Glass et al., 2021). Furthermore, method achieves the best results two subsets "KPIIdriven" and "Table-driven", respectively, as well as on two subsets determined by whether or not the answer relies on the row header hierarchy.
This demonstrates that our method can effectively improve the perception ability of the model regard- ing complex table structures, and consequently en- hance its capacity to analyze complex tables.
Analysis
The results show that our method proposed for com- plex table parsing based on GPT-3.5 effectively out- performs the optimal methods on the existing two complex table datasets. However, compared to the significant performance with single-turn dialogue, the accuracy with multi-turn dialogue, especially on the HiTAB dataset, performs slightly worse. We further analyze the results in this section.
noteworthy that, on the AIT- QA dataset, GPT-3.5 still exhibits a trend where the accuracy of single-turn dialogue is generally higher than that of multi-turn dialogue. We con- sider that this is due to the inability of multi-turn dialogue to preserve historical information, result- ing in information loss and the accumulation of errors throughout multiple interactions.
5.1 Effects of the Input Token Limit
In addition, an absolute improvement over pre- vious work is achieved on all subsets of AIT-QA. The results on subsets "Row header hierarchy" and "No row header hierarchy" show that the accuracy of answers that do not rely the row header hier- archy is significantly higher than those that do rely on We altribute this to a bias in the attention of GPT-3.5 to row and column headers.
Due to the inherent length of complex tables, as seen in Table 1, filling them into prompt templates increases the likelihood of exceeding the input to- ken limit of GPT-3.5, which has the following im- plications for task.
5.1.1 Context Truncation
The excessive length of the input text leads con- text truncation, resulting the omission of critical information. As shown in Table 5, the accuracy on the HiTAB dataset is quite low without table reconstruction, which is attributed to the fact that after filling the prompt template slot with the table information, large number of prompts exceed the input token limit, and the direct truncation of the context leads the missing of valuable informa- tion. As solution, we employ combination of single-turn and multi-turn dialogues to accomplish complex table QA.
4.6 Ablation Study
As shown in Table 5, we conduct ablation exper- iments on the HiTAB and AIT-QA datasets. The results indicate that by restructuring the tables and integrating the restructured structural information with the CoT design prompts for the Complex Ta- ble QA task, the ability of GPT-3.5 to parse com- plex tables is significantly enhanced. Simultane- ously, there a notable reduction in the instances where the model responds with "I don't know."

---

@: vazu vecan redts Living poivase dratting cansia day Use Line of Uhe oolo inseviiwt
which refers to a "nursing home".
Dalt
5.3 Effect of Hierarchical Row Header
A: Koow living in a petinase dwelllng et cetara day 2011 vace oche intarcine (57.5) 22.48)-
In conjunction with Table 3, it can be observed that the performance of not only our method but also others on the subset "Row header hierarchy" begins to decline when the answer relies on the row header hierarchy compared with the subset "No row header hierarchy".
nerntog 2084
Figure 5: An example of hallucination of GPT-3.5.
During the analysis of experimental results on the HiTAB dataset, we also encountered the same issue: the model exhibits different levels of atten- tion to row headers and column headers in tables. Under the same data format and prompt conditions, GPT-3.5 typically tends to focus more on column headers. We consider that this behavior might be attributed to the inherent ability of GPT-3.5 to in- terpret tables in markdown format, which typically only has column headers places emphasis them. Consequently, when attempting to under- stand tables in a new format, GPT-3.5 may transfer its prior knowledge, thereby affecting its parsing of the new tables.
5.1.2 Trade-off between Depth and Breadth When opting for single-turn dialogue for Complex Table QA, we input all information at once. The model is exposed to wide and comprehensive range of information sources, but due to the ab- sence of interactive prompts, its ability to focus on key information in the questions and tables is limited. In contrast, when employing multi-turn dialogue, through continuous interactive prompt- ing, we consistently guide the model to identify key information from each dialogue turn. We then utilize this information to further prompt the model in subsequent turns. Multi-turn dialogue enables the model be more focused on key information, but there is a possibility that it may overlook the global context.
5.4 Error Analysis
We randomly sample 50 instances, each from the erroneous results of single-turn dialogue and multi- turn dialogue, and analyze them in terms of the accuracy in the row and column header localiza- tion and cell selection. The results indicate that the errors are primarily attributed the model ei- ther locating incorrect row and column headers or locating too few row and column headers.
Additionally, given that GPT-3.5 cannot auto- matically retain historical conversations, errors ac- cumulate dialogue turns increase, rendering the answers increasingly prone inaccuracy.
5.1.3 Prompt Design Limitation
(Pan et al., 2023) pointed out that providing both ex- amples and descriptions in prompt can effectively improve the performance of ChatGPT. However, due to the length of the complex table, we cannot provide an example with complete dialogue and description in the prompt concurrently.
In the analysis of single-tun turn dialogue and multi- turn dialogue, it can be found that errors are mainly caused by incorrect localization of row and col- umn headers. When the model locates non-header tuples based on incorrect or incomplete row and column header tuples, it tends to make erroneous selections. Furthermore, when the model selects a broad range of row and column headers, the num- ber of non-header tuples that are selected based on them increases. The excess information can also lead to incorrect selections (another main reason for the erroneous answers of the model). This sug- gests that the comprehension capability of GPT-3.5 might decline when the input content is lengthy and complex. In addition, the model sometimes chooses the wrong cells even under the guidance of correct row and column headers, such as generat- ing fictitious answers on their own, indicating that GPT-3.5 is sometimes prone to hallucinations.
5.2 Effect of Hallucination
Hallucination refers to the generation of content that may seem plausible but is not grounded in the source information or is outright incorrect. As seen in the Figure 5, when we pose the question, "were women still living in a private dwelling on census day 201 I more likely to be unmarried or were those living in nh at the time of the echs interview?". GPT-3.5 responded with "Women" and further pro- vided detailed explanation stating "(57.5% vs 72.4%)". Evidently, if we make our selection based on the amalysis provided in the answer, the appro- priate response would be "women living in an n",

---

6 Conclusion
(PROJECT NO: 2100241 and Auxiliary Di- agnosis and Rare Discase Screening System for Children's Pneumonia (No.yg2022-7).
Inspired by the powerful reasoning and genera- tion capabilities of LLMs, propose to lever- age GPT-3.5 parser for complex tables with table reformat and prompt design. Extensive ex- periments show that our method achieves the best performance on HÎTAB and AIT-QA. However, limited by the input length and inability to store historical information of GPT-3.5, how to utilize multi-turn dialogue to obtain more valuable context for Complex Table QA remains to be explored.
References
Mostafa Amin, Frik Cambria, and Björ W. Schuller. 2023. Will affective computing emerge from foun- dation models and general ai? a first evaluation chatgpt.
Yejin Bang. Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung. Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilin- gual, multimodal evaluation of chatgpt reasoning, ballucination, and interactivity.
Limitations
When utilizing GPT-3.5, generative model, for the Complex Table QA task, comparing its gener- ated outputs to the original answers is an important issue.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33: 877-19911
GPT-3.5 typically generates answers in its own words. When paraphrasing the informa- tion from the context or that has learned during training, may provide details that do not align granularity with the original answer.
Wenhu Chen. 2023. Large language models are few(1)- shot table reasoners.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong. Hong Wang, and William Wang. 2020. Hybridqa: A dataset of multi-hop hop question answering over tabular and textual data. arviv preprint xxiv:2004.07344
GPT-3.5 is unstable. When posing the same question within the same context, does not always provide consistent responses.
Zhe Chen and Michael Cafarella. 2014. Integrating spreadsheet data via accurate and low-effort extrac- tion. In Proceedings of the 20th ACM SIGKDD in- ternational conference Knowledge discovery and data mining, pages 1126-1135.
GPT-3.5 is prone to hallucinations. It is ca- pable of generating answers that bear rel- evance to the original text or the original an- swer.
Zhoujun Cheng. Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo. Yan Gao, Shi Han. lian-Guang Lou, and Dongmei Zhang. 2022. Hitab: hierarchical table dataset for question answering and natural language generation.
Given these issues, it is important to define an evaluation metric that takes into account the com- plexity and variability of the responses generated by GPT-3.5.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. 1977. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39(1):1-22.
Acknowledgements
We deeply indebted to the anonymous reviewers from EMNLP for their constructive feedback. We are also express our gratitude to our tutors and peers for their invaluable insights and unflagging support throughout the course of this research. This work is also supported by National Natural Science Foundation of China (No.62172101), and by the Science and Technology Commission of Shanghai Municipality (No.2251 106000); and Regional so- cial experimentation with the "Clinical Decision Support System for Paediatric Outpatient Clinics"
Jacob Devlin, Ming-Was Hiig-Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing.
Haoyu Dong, Zhoujun Cheng. Xinyi He, Mengyu Zhou, Anda Zhou, Fan Zhou, Ao Liu, Shi Han, and Dong- mei Zhang. 2022. Table pretraining: survey on model architectures, pretraining objectives, and downstream tasks. arXiv preprint 22Xi::2001.09755
Yihong Dong, Xue Jiang. Zhi Jin, and Ge 2023. Self-collaboration code generation via chatgpt.

---

Julian Martin Eisenschlos, Maharshi Gor, Thomas Müller, and William W Cohen. 2021. Mate: multi- view attention for table transformer efficiency. arXiv preprint 2xx::2100.044333 04312.
Paula Maddigan and Teo Susnjak. 2023. Chat2vis: Gen- erating data visualisations via natural language us- ing chatgpt, codex and gpt-3 large language models. IEEE Access.
Jinglong Gao. Xiao Ding, Bing Qin, and Ting Liu. 2023a. Is chatgpt a good causal reasoner? a compre- hensive evaluation.
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryšciński. Nick Schoelkopf, Riley Kong, Xiangru Tang. Murori Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, and Dragomir Radev. 2021. Fetaqa: Free-form table ques- tion answering.
Yuan Gao, Ruili Wang, and Feng Hou. 2023b. How to design translation prompts for chatgpt: An empirical study.
Michael Glass, Mustafa Canim, Alfio Gliozzo, Sa- neem Chemmengath, Vishwajeet Kumar, Rishav Chakravarti, Avi Sil, Feifei Pan, Samarth Bharad- waj. and Nicolas Rodolfo Fauceglia. 2021. Cap- turing and column semantics in transformer based question answering over tables. arXiv preprint arXiv:2104.08300
Wenbo Pan, Qiguang Chen, Xiao Xu, Wanxiang Che. and Libo Qin. 2023. A preliminary evaluation of chatgpt for zero-shot dialogue understanding, arXiv preprint arXiv:2304.04256.
Panupong Pasupat and Percy Liang. 2015. Composi- tional semantic parsing on semi-structured tables.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisen- schlos. 2020. Tapas: Weakly supervised table parsing via pre-training. arXiv preprint arXiv:2004.023::
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language under- standing by generative pre-training.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Nengzheng Jin, Joanna Siebert, Dongfang Li, and Qing- cai Chen. 2022. A survey on table question answer- ing: Recent advances.
Yannis Katsis, Saneem Chemmengath, Vishwajeet Ku- mar, Samarth Bharadwaj. Mustafa Canim, Michael Glass, Alñio Gliozzo. Feifei Pan. Jaydeep Sen, Karthik Sankaranarayanan, et al. 2021. Ait-qa: ques- tion answering dataset over complex tables in the airline industry. arXiv preprint arXiv:2106. 12944.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research. 21(1):5485-555
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2023. Large lan- guage models are zero-shot reasoners.
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulezyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schacker- mann, Amy Wang. Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Sem- turs, Sara Mahdavi, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Shekoofch Azizi, Alan Karthikesalinaam, and Vivek Natarajan. 2023, Towards expert-level medical question answering with large language models.
Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le. and Ni Lao. 2018. Memory augmented policy optimization for program synthesis and se- mantic parsing. Advances in Neural Information Processing Systems, 31.
Seung-Jin Lim and Yiu-Kai Ng. 1999. An automated approach for retrieving hierarchical data from html tables. In Proceedings of the eighth international conference on Information and knowledge manage- ment, pages 466 474.
Yiming Tan, Dehai Min. Yu Li, Wenbo Li, Nan Hu. Yongrui Chen, and Guilin Qi. 2023. Evaluation of chatgpt as a question answering system for answering complex questions.
Chao Liu, Xuanlin Bao, Hongyu Zhang, Neng Zhang, Haíbo Hu, Xiaohong Zhang, and Meng Yan. 2023a. Improving chatgpt prompt for code generation.
Zezhong Wang, Fangkai Yang. Pu Zhao, Lu Wang. Jue Zhang, Mohit Garg, Qingwei Lin, and Dong- mei Zhang, 2023. Empower large language model to perform better on industrial answering.
Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023b. Evaluating the logical reasoning ability of chatgpt and gpt-4.
question
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1162..
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elic- its reasoning in large language models.

---

Ronald Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Reinforcement learning, pages 5-32.
Jingfeng Yang, Aditya Gupta, Shyam Upadhyay. Luheng He. Rahul Goel, and Shachi Paul. 2022. Tableformer: Robust transformer modeling for table- text encoding. arXiv preprint arXiv:2203.0022
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Se-
bastian Riedel. 2020. Tabert: Pretraining for joint understanding of textual and tabular data. arXiv preprint arXiv:2005.083/4
Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. 2018. Typesql: Knowledge-basee type- aware neural text-to-sql generation. arXiv preprint arXiv:1804.09769.
Tao Yu, Rui Zhang, Kai Yang. Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2019. Spider: A large-scale noman-labeled dataset for complex and cross-domain semantic pars- ing and text-to-sql task.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting large language models.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arxiv: 1e0::1709.000 709,00103.
Guido Zuccon and Bevan Koopman. 2023. Dr chatgpt, tell me what want to hear: How prompt knowledge impacts health answer correctness.
A
Appendix
A.1 Single-Turn Prompt
The Regulation and Input Context parts of the single-turn dialogue prompt are detailed in Tables 6 and 7, respectively.
A.2 Multi-Turn Prompt
As shown 8, 9 and 10, we describe the prompt the first, second and third prompt turn in detail.
A.3 Simple Prompt
Table 11 shows the full text of specific prompt designed for the original table format.

---

Regulations
# Role play
Suppose you are an expert in statistical analysis.
You will be given table described in special format. Your task is to answer the questions based on the content of the table. # Table Description
The table is described as follows:
1. The title mcans the title the table.
2. A tuple (T, TI, T2, T3, T4) represents a column header, where T indicates it's a column header,
Tl denotes its level, T2 and T3 indicate the start and end column of the header, and T4 specifies the content.
3. A tuple (L, L1, L2, L3, L4) represents row header, where L indicates it's row header, LI denotes its level, L2 and L3 indicate the start and end row of the header, and L4 specifies the content.
We represent non-header tuples as (C, C1, C2, C3), where denotes non-header tuple, C1 denotes the row, C2 denotes the column, and C3 denotes the content. C1 corresponds to the row header tuple's 1.2 and L33 that are related to it, while C2 corresponds to the column header tuple's ss T2 and T3 that are related to it.
5. The tuple of column header contains T1, representing the level of the header, with 0 being the highest level and larger T1 indicating lower levels. If the T2 and T3 of tuple A are between T2 and T3 of tuple B (can be equal), then there is parent-child relationship between A and B, A is sub-header of B, B is parent-header of A, and A's T1 must be smaller than B's T1. The lowest level header's tuple has T2=T3. Similarly for row headers. The specific tuples are Table Content. # Examples For examples:
The tuple (T, 1, 0, 0, g) denotes a column header with level 1, spanning from column 0 to column 0, with the content "g".
The tuple (L. 0, 6, 6, karlsruher sc) denotes row header with level O, spanning from row 6 to row 6, with the content "karlsruher se".
The tuple (C, 7. 0, 416) represents non-header cell row column O, with value of 416. Make sure you read and understand these instructions carefully.
# Chain-of-thougl!
Let's think step by step as follows and give full play to your expertise as a statistical analyst: 1. Clearly understand the question and the information needed to answer the question to determine the necessary information to extract.
2. Have comprehensive understanding of the data in the table, including the meaning, data types, and formats of each column and row tuples (Note: There are usually summative tuples in the table, such as all, combine, total, sum, average, mean, etc. These tuples help you skip a lot of operations). Based on the question, select the row and column header tuples that are most relevant to the question and then locate the non-header tuples based on the row and column header tuples you selected before.
4. Perform statistical, calculation, sorting, grouping, or other operations on the tuples you selected before to extract useful information based on the question's requirements.
# Output Control
You MUST answer each question in the format below line by line (Note: Keep your answer concise):
Column header:
The column header tuples most relevant to the answer.
Row header:
The row header tuples most relevant to the answer.
3. Cell: The non-header tuples most relevant to the answer.
4. Operation: the operation you performed on the tuples you selected.
5. Answer: your answer (A number, noun, phrase, or set of data).
And if the answer is not contained within the context, say "I don't 't know".
Table 6: Full text of the "Regulations" part of the single-tum dialogue prompt. The text in "[]" can be replaced
according to the specific information in the QA process.

---

Input Context
Table Title
Title: "TALLE_TTLEE ]ITLE_HERE]
# Column Header
Column header:
COLUMN_HEADER HEADER HERE]
# Row Header
Row header: WABLE_ROW, HEW___EEDERRHHE HERE]
# Non-Header
Non-header: SAALLENNO HEADER HEREE
Question
Q: DUUSIION_ERRI
A:
Table 7: Full text of the "Input Context" part of the single-turn dialogue prompt. The text "[]" can be replaced
according to the specific information in the QA process.

---

Regulations
# Role play
Suppose you are an expert in statistical analysis.
You will be given table described in special format. Your task is to answer the questions based on the content of the table. # Table Description
The table is described follows: The title means the title the table.
2. A tuple (T, T1, T2, T3, T4) represents a column header, where T indicates it's a column header,
Tl denotes its level, T2 and T3 indicate the start and end column of the header, andT4 specifies the content.
3. A tuple (L, L1, L., L3, L4) represents a row header, where L. indicates it's row header, L1 denotes its level, L2 and L3 indicate the start and end row of the header, and L4 specifies the content.
We represent non-header tuples (C, C1, C2, C3), where C denotes non-header tuple, C1 denotes the row, C2 denotes the column, and C3 denotes the content. C1 corresponds the row header tuple's L2 and L3 that are related to it, while C2 corresponds to the column header tuplc's T2 and T3 that are related to it.
5. The tuple of column header contains T1, representing the level of the header, with 0 being the highest level and larger TI indicating lower levels. If the T2 and T3 of tuple A are between T2 and T3 of tuple B (can be equal), then there is parent-child relationship between A and B, A is sub-header of B, B is parent-header of A, and A's TI must be smaller than B's TI. The lowest level header's tuple has T2=T3. Similarly for row headers. The specific tuples in Table Content.
Examples
For examples:
The tuple (T, 1, 0, g) denotes a column header with level 1, spanning from column 0 to column 0,
with the content "g".
The tuple (L, 0, 6, 6. karlsruher sc) denotes row header with level 0, spanning from row 6 to row 6, with the content "karlsruher sc".
The tuple (C, 7, 0, 416) represents non-header cell row 7, column O, with value of 416.
Make sure you read and understand these instructions carefully.
Input Context #1 # Abstract Keywords
Extract the key words in the question.
Q: (QUESTION_HERE)
A:
Table 8: Full text of the first prompt turn of the multi-tum dialogue. The text in "0" can be replaced according the specific information in the QA process.

---

Regulations and Historical Dialogue
# Role play
Suppose you
expert in statistical analysis.
You will be given table described in a special format. Your task is to answer the questions based on the content of the table. # Table Description
The table is described as follows:
1. The title means the title of the table.
2. A tuple (T, T1, T2, T3, T4) represents column header, where T indicates it's column header,
Tl denotes its level, T2 and T3 indicate the start and end column of the header, andT4 specifies the content.
tuple (L, LI, L2, L3, L4) represents row header, where L indicates it's row header, LI denotes its level, L. and L3 indicate the start and end row of the header, and L4 specifies the content.
4. We represent non-header tuples as (C, CI, C2, C3), where C denotes a non-header tuple, CI denotes the row, C2 denotes the column, and C3 denotes the content. C1 corresponds to the row header tuple's s L.2 and L3 that are related to it, while C2 corresponds to the column header tuple's s T2 and T3 that are related to it.
5. The tuple of column header contains T1, representing the level of the header, with 0 being the highest level and larger T1 indicating lower levels. If the T2 and T3 of tuple are between T2 and T3 of tuple (can be equal), then there is parent-child relationship between and B, A is a sub-header B, is parent-header of A, and A's T1 must be smaller than B's T1. The lowest level header's tuple has T2=T3. Similarly for row headers. The specific tuples are Table Content.
# Examples For examples:
The tuple (T, 1, 0, 0, denotes column header with level 1, spanning from column 0 to column 0, with the content "g".
The tuple (L. 0, 6, 6, karlsruher sc) denotes a row header with level O, spanning from row row 6, with the content "karlsruher se".
The tuple (C, 7. O, 416) represents a non-header cell at row 7, column O, with a value of 416.
Make sure you read and understand these instructions carefully.
# Output of Turn
Extract the key words in the question.
Q: (QUESTION_HERS
A: [ANSWER OF TURN_I
Input Context #2
# Select Headers
Here are table title and the tuples of the table's rows and headers, please try to locate the lowest level of headers that match the question and keywords you extracted:
"Title": [TABLE TITLE HERE]
"Column header": [TABLE COLLECNOONLLLLL _EEDERREERRR..EE _EEREER_HERE]
"Row header": [TABLE ROW HEADER HEREJ
# Output Control
You MUST output your selection in the following format:
1. Column header:
2. Row header:
Table 9: Full text of the second prompt turn of the multi-tur dialogue. The text in "[]" can he replaced according to the specific information the QA process.

---

Regulations and Historical Dialogue
# Role play
Suppose you are an expert in statistical analysis.
You will be given table described in special format. Your task is to answer the questions based on the content of the table. # Table Description
The table is described as follows:
1. The title mcans the title the table.
2. A tuple (T, TI, T2, T3, T4) represents a column header, where T indicates it's a column header,
Tl denotes its level, T2 and T3 indicate the start and end column of the header, andT4 specifies the content.
3. A tuple (L, L1, L2, L3, L4) represents row header, where L indicates it's row header, LI denotes its level, L2 and L3 indicate the start and end row of the header, and L4 specifies the content.
We represent non-header tuples as (C, C1, C2, C3), where denotes non-header tuple, C1 denotes the row, C2 denotes the column, and C3 denotes the content. C1 corresponds to the row header tuple's 1.2 and L3 that are related to it, while C2 corresponds to the column header tuple's ss T2 and T3 that are related to it.
5. The tuple of column header contains T1, representing the level of the header, with 0 being the highest level and larger T1 indicating lower levels. If the T2 and T3 of tuple A are between T2 and T3 of tuple B (can be equal), then there is parent-child relationship between A and B, A is sub-header of B, B is parent-header of A, and A's T1 must be smaller than B's T1. The lowest level header's tuple has T2=T3. Similarly for row headers. The specific tuples are Table Content. # Examples For examples:
The tuple (T, 1, 0, 0, g) denotes a column header with level 1, spanning from column 0 to column 0, with the content "g".
The tuple (L. 0, 6, 6, karlsruher sc) denotes row header with level O, spanning from row 6 to row 6, with the content "karlsruher se".
The tuple (C, 7. 0, 416) represents non-header cell row column O, with value of 416.
Make sure you read and understand these instructions carefully.
# Output of Turn and Code
Here are all the tuples relevant to the question:
JANSWER OF [F_TURN_22
Non-header: [OUTPUT_OW) SF OF_CODE]
Input Context #2
# All Headers
"Column header": [TABLE_COUUM] HEADER HERE]
"Row hcader": [TABLE ROW HEADER HERE]
# Outpul Control
You MUST answer each question in the format below line by line (Note: Keep your answer concise):
1. Column header:
The column header tuples most relevant to the answer.
2. Row header:
The row header tuples most relevant to the answer.
Cell: The non-header tuples most relevant to the answer.
Operation: the operation you performed on the tuples you selected.
Answer: your answer (A number, noun, phrase, or set of data).
And if the answer is not contained within the context, say "I don't 't know".
Notes:
1. In the row and column header tuples, the third and fourth elements represent the row and column position information.
2. You must output non-header tuples that are valid in the above tuples.
Table 10: Full text of the third prompt turn of the multi- turn dialogue. The text in "["" can be replaced according to the specific information in the QA process.

---

Regulations
The text provided describes a table in json format, answer the question as truthfully as possible using the provided text and don't 't omit the decimal point, and if the answer is not contained within the text below, say "I don't know".
# Content EAGGINALTTABBER
Q: [QUESTION _ERRE] A:
Table Full text of the simple prompt. The text "[]" can be replaced according the specific information the QA process.


Reviews:
[{"review_id": "wlbHWIHIvk", "reviewer": "Reviewer_bgck", "summary": "", "questions": "1. For the multi-turn dialogue, why do not directly use the question to extract the row headers and column headers? How much improvement does keyword extraction bring?\n2. What’s the performance variance of the predictions of GPT3.5?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The title is somehow overclaimed to my understanding. The argument that LLMs are complex table parsers is somehow supported by the experiments on GPT-3.5 which is instruction-tuned LLM and improved with RLHF and other techniques. Without showing pure LLMs can achieve this complex table parsing ability, I am not convinced by the claim of the title.  \nR1: Thank you very much for providing valuable suggestions.  We recognize and appreciate your concerns regarding the scope of our research and the claim made in the title.  \nBefore diving into experiments with the model text-davinci-003, we conducted preliminary tests using the model gpt-3.5-turbo on the HiTAB dev dataset. While the results were not as prominent as those with the text-davinci-003, they were still on par with the performance level of the weak supervision method, as illustrated in Table 2 (MAPO) (Mehran et al. 2010).  From the experimental observation and analysis, it can be exhibited that instruction-tuned LLMs are capable of parsing complex tables.   \nGiven your feedback, if we have a chance to optimize the illustration, we will consider adjusting the expression to \"Instruction-Tuned LLMs are Complex Table Parsers\" to more accurately represent the content and contributions of our study.\n\n\nQ2: What if the simple prompt with original json format is augmented with instructions by showing the meaning of each field, will the performance be comparable with the designed tuple format? I feel that somehow the baseline is too weak.  \nA2: Thank you very much for the suggestion. We have redesigned the prompt for the original json format, and clarified the meaning of each field, then conducted tests on the dev dataset of HiTAB. The specific results are as follows.  \nPrompt for json format in Table 5: Acc 20.3%, Idn 9.9%;  \nPrompt for tuple format in Table 5: Acc 49.3%, Idn 0.6%;  \nPrompt for json format (redesign): Acc 29.2%, Idn 14.2%.  \nThese results indicate that your suggestion \"simple prompt with original json format with instructions by showing the meaning of each field\" indeed enhanced the model's understanding of the original json format, thereby significantly improving the baseline performance (Acc increased from 20.3% to 29.2%). However, it still falls short when compared to the tuple format we proposed (Acc 49.3%).  \nUpon initial analysis, it appears that this trend can be attributed to the model's tendency to generate \"I don't know\" responses in two scenarios: first, when it does not understand the table, and second when it understands the table but cannot find a suitable answer. The rise in the Idn percentage might be due to the second scenario, indicating that the model's comprehension ability has indeed improved, and a point is also corroborated by the increase in Acc.  \nThank you once again for your question. We will continue to refine our experiments. However, due to time constraints, we could only test on the dev dataset of HiTAB for now. We also plan to further explore the model's comprehension ability in the following research work.\n\n\nQ3: For the multi-turn dialogue, why do not directly use the question to extract the row headers and column headers? How much improvement does keyword extraction bring?  \nA3: Thank you for your question. We appreciate your astute observation, and there were certain aspects of the initial experimental design that were not sufficiently comprehensive. In response, we conducted fresh trials on the dev dataset of HiTAB, merging the formerly separate first and second round dialogue tasks. The resultant experimental findings exhibit a notable surge in multi-turn dialogue accuracy, ascending to 48.0% from the previous 43.5%. Our analysis attributes this progress to the diminished error accumulation due to the consolidation of tasks.  \nGiven the time constraints, we focused our efforts on the dev dataset of HiTAB. However,  we are dedicated to extending our efforts to encompass the remaining follow-up experiments and subsequent analysis. We sincerely extend our gratitude for your insightful queries.\n\n\nQ4: What’s the performance variance of the predictions of GPT3.5?  \nA4: The variance of the predictions of GPT3.5 on HiTAB and AIT-QA datasets is 0.250 and 0.181, respectively. The calculation process is as follows:  \n1)Variance=∑(Xi-μ)^2/N (N is the total number of test samples, and Xi is the execution result (1 for positive and 0 for negative) of the i-th sample. μ is the average of all sample execution results).  \n2)There are a total of 10,672 instances in HiTAB, out of which 5,237 are positive examples and 5,435 are negative examples. μ1=0.493, 1-μ1=0.507. There are a total of 514 instances in AIT-QA, 392 positive examples and 122 negative examples. μ2=0.763, 1-μ2=0.237.  \n3)For HiTAB, variance = ((1-μ1)^2)*5237+((μ1^2)*5435)/10672=0.250. For AIT-QA, variance = ((1-μ2)^2)*392+((μ2^2)*122)/514=0.181.  \n\n\nReferences:  \n[1] Mehran, N., Kimovski, D., & Prodan, R. (2019, October). MAPO: a multi-objective model for IoT application placement in a fog environment. In Proceedings of the 9th International Conference on the Internet of Things (pp. 1-8).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Explored LLM’s effectiveness on the complex table question answering. Proposed a new approach to improve the model ability of modeling hierarchical structure of complex table. The contribution is demonstrated by the ablation study of replacing the designed prompting method with simple table prompting.\n2. By combining the single-turn and multi-turn dialogues, the authors alleviate the input token limitation of GPT3.5.\n", "reasons_to_reject": "1. The title is somehow overclaimed to my understanding. The argument that LLMs are complex table parsers is somehow supported by the experiments on GPT-3.5 which is instruction-tuned LLM and improved with RLHF and other techniques. Without showing pure LLMs can achieve this complex table parsing ability, I am not convinced by the claim of the title. \n2. What if the simple prompt with original json format is augmented with instructions by showing the meaning of each field, will the performance be comparable with the designed tuple format? I feel that somehow the baseline is too weak.\n", "questions_to_authors": "1. For the multi-turn dialogue, why do not directly use the question to extract the row headers and column headers? How much improvement does keyword extraction bring?\n2. What’s the performance variance of the predictions of GPT3.5?\n", "paper_topic_and_main_contributions": "This work focuses on the task of question answering over complex tables. The model involves table linearization, regulations prompt design, input context design, and long context adaptation.\nMore specifically, in the regulation prompt, the authors add an explanatory description of the meaning of each tuple and the logical reasoning process of the task, to improve the hierarchical structure awareness capability of LLMs. To alleviate the long context issue, the LLM is used to extract important information (keywords) from the question and relevant table information for the usage in following-up question answering component. With extensive experiments, the authors show that the proposed method achieves state-of-the-art performance on HiTab and AIT-QA.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 357, “.” is missing.\n2. Typo in Table 7 where for Non-Header, the prompt should be non-header instead of row header.\n3. Missing “,” in Figure 3 (a) “0Position”\n"}, "extra_text": {}}, {"review_id": "fxA9mW13Wr", "reviewer": "Reviewer_n81y", "summary": "", "questions": "1. Is the chain of thought composed of generic instructions (as shown in Page 12)  for all examples ?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The evaluation of the model is performed with accuracy as mentioned in Section 4.3. However, accuracy provides little insight into the generated answer, and precision and recall might capture the differences in representation and verbosity of the generated answers. For example, the sample shown in Figure 4 would have high recall and low precision indicating that the generated text is more verbose.  \nA1: Thank you for your insightful comment. We acknowledge that we may not have clarified our evaluation metric sufficiently in Section 4.3, leading to a misunderstanding. We chose to follow the evaluation metric Execution Accuracy (EA) as employed by (Cheng et al., 2022), which is originally from (Pasupat and Liang, 2015). Specifically, EA is a performance metric used in the context of question answering over structured data, such as tables or databases. It measures the accuracy of the execution of a query generated by a model, rather than the accuracy of the query itself. In our research, our focus on the logical form and detailed analysis of the model's reasoning capabilities makes EA a more appropriate metric. For example, as shown in Figure 5, the question uses the abbreviation \"nh,\" which GPT also uses in its response, while the standard answer is \"nursing home\". In such cases, using precision and recall would not accurately reflect the model's performance, as both answers are correct but differ in presentation.  \nAfter conducting research on relevant QA work, we have found that precision and recall are widely employed in general QA tasks (Ferrucci et al. 2010; Yang et al. 2015; Rajpurkar et al. 2016). However, when facing complex table QA tasks involving intricate relationships among headers, especially concerning the model's capability to parse complex tables, the evaluation metric EA is better suited for such tasks.\nWe greatly appreciate your suggestions. If there is an opportunity for further revisions of the paper, we will consider adding a detailed discussion on the evaluation metrics of precision and recall. Moving forward, we will also continue to focus on identifying other metrics that are better suited for Complex Table QA tasks.\n\n\nQ2: Is the chain of thought composed of generic instructions (as shown in Page 12) for all examples? && It is not clear how the chain of thought was annotated for the questions. && Converting a concise structured table into a long sequence of text (prompt) increases the sequence length and may not be an ideal approach for any realistic table size. While table truncation problem persists in existing tableQA models, it is exacerbated when transforming the table into a text sequence as described in Section 3.3. It is unclear how converting a single-turn question into multi-turn will help address this issue.  \nA2: Thank you for your valuable feedback. While we aimed to clarify the multi-turn dialogue mechanism in Section 3.3, page limitations might have led to a little confusion.  \nInitially, as you rightly pointed out, converting a concise tabular structure into a long sequence of text (prompt) increases sequence length. Thus, we adopted a succinct tuple-based approach for complex table transformation. This preserves all original information and applies universally to non-database tables, including complex and flat tables. It is important to highlight that, distinct from flat tables with single-level column headers and no merged cells, complex tables possess multi-level structured headers and numerous merged cells, creating intricate positional relationships among cells (Lim and Ng, 1999; Chen and Cafarella, 2014; Jin et al., 2022). Preserving these header relationships comprehensively while maintaining conciseness during complex table parsing is crucial.  \nSecondly, concerning the \"chain-of-thought\" design and how multi-turn dialogues mitigate long prompt issues, we offer the explanation as follows.  \nIn single-turn dialogues, the entire table is inputted at once. The \"Chain-of-thought\" section guides the model through the steps as \"understanding the question - understanding the table - selecting relevant row and column headers - locating non-header cells based on relevant headers - operations\" (as illustrated in Section \"Chain-of-thought\" of Page 12). If token count surpasses limits, we shift from single-turn to multi-turn dialogues, guiding the reasoning incrementally. Unlike inputting the whole table in one turn, multi-turn dialogues prompt the model to construct a subtable for answering. Such a manner accommodates token constraints.  \nSpecifically, in the first dialogue round, we refrain from inputting the table. Instead, we establish the role assumption and table definition while inputting the question. We ask the model to extract keywords it deems important from the question, corresponding to the \"understanding the question - understanding the table\" steps in the single-turn dialogue's chain of thought. In the second dialogue round, building on the content from the first round, we provide the model with all row and column headers of the table, asking it to select headers that align with the keywords from the first round, mirroring the \"selecting relevant row and column headers\". Subsequently, utilizing the Code Assistant module, we use a programming approach to directly pinpoint specific non-header tuples based on the header tuples chosen in the second round, equivalent to \"locating non-header cells based on relevant headers\". By this point, we have constructed a smaller subtable based on the selected headers, further alleviating the token count limit. In the third round, we input the subtable (including row and column headers, as well as non-header tuples) to the model and request it to answer the question, corresponding to the \"operations\".\n\n\nReferences:  \n[1]Pasupat, P., & Liang, P. (2015). Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508.00305.  \n[2]Cheng, Z., Dong, H., Wang, Z., Jia, R., Guo, J., Gao, Y., ... & Zhang, D. (2021). Hitab: A hierarchical table dataset for question answering and natural language generation. arXiv preprint arXiv:2108.06712.  \n[3]Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A. A., ... & Welty, C. (2010). Building Watson: An overview of the DeepQA project. AI Magazine, 31(3), 59-79.  \n[4]Yang, Y., Yih, W. T., & Meek, C. (2015, September). Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (pp. 2013-2018).  \n[5]Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.  \n[6]Lim, S. J., & Ng, Y. K. (1999, November). An automated approach for retrieving hierarchical data from HTML tables. In Proceedings of the 8th CIKM International Conference on Information and Knowledge Management (pp. 466-474).  \n[7]Chen, Z., & Cafarella, M. (2014, August). Integrating spreadsheet data via accurate and low-effort extraction. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1126-1135).  \n[8]Jin, N., Siebert, J., Li, D., & Chen, Q. (2022, August). A survey on table question answering: recent advances. In China Conference on Knowledge Graph and Semantic Computing (pp. 174-186). Singapore: Springer Nature Singapore.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper designs prompts for single-turn and multi-turn QA over tables for GPT-3.5\n2. The ablation study on the prompt design in Section 4.6 is useful to understand which aspect of the prompt helps the model reason better.\n3. The analysis in Section 5 provides interesting limitations of prompting GPT-3.5 which can promote further research", "reasons_to_reject": "1. The evaluation of the model is performed with accuracy as mentioned in Section 4.3. However, accuracy provides little insights into the generated answer and precision and recall might capture the differences in representation and verbosity of the generated answers.  For example, the sample shown in Figure 4 would have high recall and low precision indicating that the generated text is more verbose.\n2. It is not clear how the chain of thought was annotated for the questions.\n3. Converting a concise structured table into a long sequence of text (prompt) increases the sequence length and may not be an ideal approach for any realistic table size. While table truncation problem persists in existing tableQA models, it is exacerbated  when transforming the table into a text sequence as described in Section 3.3. It is unclear how converting a single-turn question into multi-turn will help address this issue.", "questions_to_authors": "1. Is the chain of thought composed of generic instructions (as shown in Page 12)  for all examples ?", "paper_topic_and_main_contributions": "The paper focuses on designing prompts to GPT-3.5 to perform tableQA. The convert single-turn questions to multi-turn when the input sequence exceeds the maximum sequence length of the model. The examples of the prompts increase the understanding of the main text.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Table 1 and 5 and Figure 2 are too small and hard to read."}, "extra_text": {}}, {"review_id": "iG76DnWuox", "reviewer": "Reviewer_Didr", "summary": "", "questions": "1. What are the new challenges in applying LLM for complex table QA, compared with flat table QA? Is the multi-turn scheme also applicable in flat table QA? \n\n2. How many testing cases exceed the max input tokens? \n\n3. What is the concrete functionality of Code Assistant? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: What are the new challenges in applying LLM for complex table QA, compared with flat table QA? Is the multi-turn scheme also applicable in flat table QA? && The proposed approach is not well-motivated and the intuitions of the design choices are not elaborated clearly. Specifically, what kind of new challenges are encountered when using LLM for complex table QA, compared with using LLM for flat table QA? By diving into these new challenges, we can better understand the necessity and importance of a new approach. At the same time, we elaborate on each design choice with regard to a specific challenge, e.g., representing a cell independently with a tuple could represent a complex table concisely so as to reduce the risk of exceeding the max input token.  \nA1: Firstly, distinct from flat tables with single-level column headers and no merged cells, complex tables possess multi-level structured headers and numerous merged cells, creating intricate positional relationships among cells (Lim and Ng, 1999; Chen and Cafarella, 2014; Jin et al., 2022). Preserving these header relationships comprehensively while maintaining conciseness during complex table parsing is crucial. Hence, compared to flat tables, the new challenges in employing LLMs for complex table QA include:\n1) Linearizing complex tables: Unlike flat tables, which can be flattened directly (e.g., TaPas (Herzig et al., 2020)), complex tables with multi-level row/column headers cannot be linearized in the same manner. We propose a tuple-based linearization approach, using designated flags to succinctly retain all table information.\n2) Input token limits for complex tables: We address the token limit issue of GPT-3.5 through a combination of single-turn and multi-turn dialogues.\n3) Enhancing model understanding of transformed table format: We provide explanations and examples for the meaning of each tuple in the prompt, coupled with a description of the QA task logic, to enhance the model's structural awareness of complex tables.  \nRegarding the question of \"Is the multi-turn scheme also applicable in flat table QA?\", we conducted experiments on a subset of 200 cases randomly drawn from the WIKITABLEQUESTIONS (Pasupat and Liang, 2015), a classic flat table QA dataset on which Tapex (Liu Q et al., 2021) conducted experiments. We transformed the corresponding tables into tuple format and made slight adjustments to the prompts. The experimental results demonstrated an accuracy of 53.0% in single-turn dialogues. While this slightly falls short of the 57.5% accuracy achieved by Tapex, it is still sufficient to showcase the transferability of our method. We eliminated the test for multi-turn dialogues in our experiments with flat tables, primarily for two reasons. First, throughout our experiments, we did not encounter situations where input length exceeded the limit, and as per our original design, such tables do not necessitate multi-turn dialogues. Secondly, multi-turn dialogues introduce the potential for cumulative errors.  \nAdditionally, even if the length of a flat table exceeds the input token limit of a single turn, we can address this through table segmentation, conducting multiple single-turn Q&A sessions (e.g., dividing tables into subtables based on columns or rows, then combining subtables with headers). For the record, unlike a flat table, where straightforward table segmentation can be employed to address model input token constraints, complex tables exhibit an intricate interplay between the column and row headers that defies such solutions. The interdependencies and contextual significance embedded within the header arrangement necessitate a more nuanced approach to effectively handle the limitations posed by the model's input token. Due to time constraints, we can only test on a subset of data. If we have the opportunity to revise the paper, we will include additional experimental results from the complete dataset.\n\nQ2: Some simple yet important baselines are missing in the experiments. To demonstrate the effectiveness of the table reconstruction method, it is necessary to compare it with the well-established table linearization method, e.g., the one used in Tapex, even though they are only designed for flat tables.  \nA2: We sincerely appreciate the valuable suggestion, which emphasizes the importance of comparing our method with well-established table linearization approaches, e.g., the one used in Tapex. We understand that such a comparison is crucial for validating the effectiveness of our reconstruction method.\nHowever, we would also like to underscore that our research primarily focuses on handling the positional relationships within complex tables, rather than contrasting methods for flat tables. At the same time, our additional experiments in A1 have demonstrated the portability and effectiveness of our approach on flat table QA tasks.  \nWe once again express gratitude for your suggestion. We will consider including more comparisons in the subsequent research. However, at this juncture, through comparisons with our established baselines and the exhibited experimental results, significant efficacy has been demonstrated.\n\n\nQ3: How many testing cases exceed the max input tokens? && The effectiveness of the multi-turn dialogue schemes is not demonstrated.  \nA3: In the HiTAB and AIT-QA datasets, there are 7,746 and 477 cases respectively that can be directly solved using a single-turn scheme. There are 2,926 and 37 cases respectively in HiTAB and AIT-QA where the content exceeds the maximum input token limit. It is noticeable that, in the HiTAB and AIT-QA datasets, 27.42% and 7.20% of cases exceed the maximum input length for the single-turn scheme, respectively. As demonstrated in Tables 3 and 4 of the paper, these cases achieve considerable results within the multi-turn scheme (42.1% and 70.27% accuracy, respectively). This underscores the necessity and effectiveness of employing the multi-turn scheme to address the input token limitation.\nFurthermore, with ongoing improvements to LLMs (including further alleviation of input length limitations and automatic retention of chat history), the effectiveness of the multi-turn scheme is expected to exhibit even more substantial enhancement.\n\n\nQ4: What is the concrete functionality of Code Assistant?  \nA4: The Code Assistant serves two primary purposes:\n1) Constructing a subtable based on selected headers: In single-turn dialogues, the entire table is inputted at once. The \"Chain-of-thought\" section guides the model through steps as \"understanding the question - understanding the table - selecting the relevant row and column headers - locating non-header cells based on relevant headers - operations\" (as shown in Section \"Chain-of-thought\" of Page 12). For the Code Assistant module in multi-turn dialogue, we use a programming approach to directly pinpoint specific non-header tuples based on the header tuples chosen in the second round, equivalent to the \"locating non-header cells based on relevant headers\" step in the single-turn dialogue's chain of thought. By this point, we constructed a smaller subtable based on the selected headers, further alleviating the token count limit. \n2) Reducing model invocations, enhancing experimental efficiency, and lowering costs: In comparison to the slower response and generation speed of the text-davinci-003, code processing is significantly faster. Substituting a single interface call with code processing enhances experimental efficiency and reduces costs associated with invoking the interface.\n\n\nReferences:  \n[1] Lim, S. J., & Ng, Y. K. (1999, November). An automated approach for retrieving hierarchical data from HTML tables. In Proceedings of the 8th CIKM International Conference on Information and Knowledge Management (pp. 466-474).  \n[2] Chen, Z., & Cafarella, M. (2014, August). Integrating spreadsheet data via accurate and low-effort extraction. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1126-1135).  \n[3] Jin, N., Siebert, J., Li, D., & Chen, Q. (2022, August). A survey on table question answering: recent advances. In China Conference on Knowledge Graph and Semantic Computing (pp. 174-186). Singapore: Springer Nature Singapore.  \n[4] Herzig, J., Nowak, P. K., Müller, T., Piccinno, F., & Eisenschlos, J. M. (2020). TaPas: Weakly supervised table parsing via pre-training. arXiv preprint arXiv:2004.02349.  \n[5] Liu, Q., Chen, B., Guo, J., Ziyadi, M., Lin, Z., Chen, W., & Lou, J. G. (2021). TAPEX: Table pre-training via learning a neural SQL executor. arXiv preprint arXiv:2107.07653.  \n[6] Pasupat, P., & Liang, P. (2015). Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508.00305.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem studied is highly valuable. \n\n2. Overall, the paper is clearly written and easy to follow. \n\n3. The proposed approach achieved state-of-the-art performance on HiTab and AIT-QA.", "reasons_to_reject": "1. The proposed approach is not well-motivated and the intuitions of the design choices are not elaborated clearly. Specifically, what kind of new challenges are encountered when using LLM for complex table QA, compared with using LLM for flat table QA? By diving into these new challenges, we can better understand the necessity and importance of a new approach. At the same time, we elaborate on each design choice with regard to a specific challenge, e.g., representing a cell independently with a tuple could represent a complex table concisely so as to reduce the risk of exceeding the max input token. \n\n2. Some simple yet important baselines are missing in the experiments. To demonstrate the effectiveness of the table reconstruction method, it is necessary to compare it with the well-established table linearization method, e.g., the one used in Tapex, even though they are only designed for flat tables.  \n\n3. The effectiveness of the multi-turn dialogue schemes is not demonstrated. ", "questions_to_authors": "1. What are the new challenges in applying LLM for complex table QA, compared with flat table QA? Is the multi-turn scheme also applicable in flat table QA? \n\n2. How many testing cases exceed the max input tokens? \n\n3. What is the concrete functionality of Code Assistant? ", "paper_topic_and_main_contributions": "This paper targets the task of question answering on complex tables. For complex tables, the authors mainly refer to those containing multi-level headers and a large number of merged cells. The authors propose incorporating advanced generative language models, such as GPT-3.5 to tackle the task. Specifically, a novel complex table representation method is proposed to feed a complex table’s information into a prompt. Also, a carefully designed prompting mechanism is innovated to boost the prompting performance and resolve the max input token limitation of GPT-3.5. Experimental results on HiTab and AIT-QA demonstrated the effectiveness of the proposed approach. The fine-grained empirical analysis also sheds light on future work. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It is necessary to study prior table representation methods and discuss the differences between the proposed method and existing ones. "}, "extra_text": {}}]

Author rebuttal:

Meta-review: Problem: This paper targets the task of question answering on complex tables. For example, those containing multi-level headers and many merged cells. 

Proposed solution: They convert single-turn questions to multi-turn prompts that convey information about the schema of the table and the gist of its content. This boosts the prompting performance and resolves the max input token limitation of GPT-3.5. Experimental results on HiTab and AIT-QA demonstrated the effectiveness of the proposed approach. 


- From the title, it is unclear what aspect of LLMs led to these gains (scale of models, scale of pre-training data, RLHF?) The authors have proposed to revise their title. 
- More insight is needed for Section 4.3. The authors promise to add more analysis to the document. 
- Details of other prompting techniques need to be included. The authors promise to add the missing details in the text.


Overall, the reviewers have no concerns regarding the "soundness" of the paper. So, I think this paper got accepted to either Finding or the main track. However, given the limited "excitement," I am **not** willing to defend its acceptance to the main track.  

----------------------------------------------

Aside: Leaving these formatting suggestions for the authors for their revision: (1) the fonts across most figures/tables are too small. (2) "(Pan et al., 2023) pointed out" should be "Pan et al., (2023) pointed out" since the citation is used in the subject of your sentence.
Decision: Accept-Main