HiddenTables and PyQTax: A Cooperative Game and Dataset For TableQA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies
Authors: William Watson
Nicole Cho
Tucker Balch
Manuela Veloso
Venue: EMNLP 2023 Main (2023)
Keywords: table question answering
dataset
large language models
data privacy
agents
cooperative game
TL;DR: We propose HiddenTables and PyQTax - a cooperative game that evaluates an LLM’s ability to align natural language schemas to programmatic commands for TableQA tasks across a myriad of question complexities and taxonomies to probe current limitations.

Abstract: A myriad of different Large Language Models (LLMs) face a common challenge in contextually analyzing table question-answering tasks. These challenges are engendered from (1) finite context windows for large tables, (2) multi-faceted discrepancies amongst tokenization patterns against cell boundaries, and (3) various limitations stemming from data confidentiality in the process of using external models such as gpt-35-turbo. We propose a cooperative game dubbed "HiddenTables" as a potential resolution to this challenge. In essence, "HiddenTables" is played between the code-generating LLM "Solver" and the "Oracle" which evaluates the ability of the LLM agents to solve TableQA tasks.  This game is based on natural language schemas and importantly, ensures the security of the underlying data. We provide evidential experiments on a diverse set of tables that demonstrate an LLM's collective inability to generalize and perform on complex queries, handle compositional dependencies, and align natural language to programmatic commands when concrete table schemas are provided. Unlike encoder-based models, we have pushed the boundaries of "HiddenTables" to not be limited by the number of rows - therefore we exhibit improved efficiency in prompt and completion tokens. Our infrastructure has spawned a new dataset "PyQTax" that spans across 116,671 question-table-answer triplets and provides additional fine-grained breakdowns and labels for varying question taxonomies. Therefore, in tandem with our academic contributions regarding LLMs' deficiency in TableQA tasks, "HiddenTables" is a tactile manifestation of how LLMs can interact with massive datasets while ensuring data security and minimizing generation costs.

HiddenTables & PyQTax: A Cooperative Game and Dataset For TableQA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies
William Watson and Nicole Cho and Tucker Balch and Manuela Veloso
J.P. Morgan Al Research New York, NY, USA
(william. watson, tucker balch, manuela. nicole. jhorjamargan. com
Abstract
models typically prioritize and highlight the meth- ods' achievement in accuracy (Herzig et al., 2020; Liu et al., 2022). However, in many cases, prereq- uisite for these upproaches to achieve such accuracy is the exposition of tabular content in its entirety and the indulgent ingestion of tokens (Herzig et al., 2020; Yin et al., 2020; Yu et al., 2021; Liu et al., 2022). Such liberal dispositions towards privacy and efficiency can be deemed impractical in the tangible deployment process of language models within institutions. Moreover, such necessities to expose the underlying data begs the question whether the model actually understands the ques- tion to provide an accurate answer. In essence, our endeavor is also an intellectual pursuit to an- swer the "chinese room argument" with regards to language models (Cole, 2023). Therefore, we propose an alternative approach for table question- answering tasks cooperative game dubbed "Hid- denTables". HiddenTables is comprised of two agents: an "Oracle" and a "Solver", in which the latter generates code to answer user queries relying solely on the Oracle's instructions and relaying of schema. In other words, the game is played with- out the Solver knowing the tabular content. The Solver's code is then evaluated by the secure Oracle that relays the answer to the user or asks follow- up questions to the Solver. Figure I summarizes the environmental set-up that our method enables between a user and gpt-3. 3pt-3.5-turbbe 5-turbo.
myriad of different Large Language Mod- els (LLMs) face a common challenge in con- textually analyzing table tasks. These challenges engendered from (I) finite context windows for large tables, (2) factte discrepancies amongst tokeniza- tion patterns against cell boundaries, and (3) various limitations stemming from data con- fidentiality in the process of using external models such as a cooperative game dubbed "HiddenTables" potential resolution this challenge. In essence, "HiddenTTblles" is played between the code-generating LLM "Solver" and the "Ora- cle" which evaluates the ability of the LLM agents to solve Table QA tasks. This game based on natural language schemas and im- portantly, ensures the security of the underly- ing data. We provide evidential experiments on a diverse set of tables that demonstrate an LLM's collective inability to generalize and perform on complex queries, handle composi- tional dependencies, and align natural language to programmatic commands when concrete ta- ble schemas are provided. Unlike encoder- based models, we have pushed the boundaries of "HiddenTTables to not be limited by the num- ber of rows therefore we exhibit improved ef- ficiency in prompt and completion tokens. Our infrastructure has spawned new dataset "PyQ- Tax" that spans across I 116,671 question-table- answer triplets and provides additional fine- grained breakdowns labels for varying ques- tion taxomomies. Therefore, in tandem with academic contributions regarding LLMs' defi- ciency in TableQA tasks, "HiddenTables" is tactile manifestation of how LLMs can inter- act with massive datasets while ensuring data security and minimizing generation costs.
We propose
Therefore this paper sets forth general system architecture that can be employed across a myriad of taxonomies and tabular formats. We find that the accuracy of 3-5t-bbb has decreased with our cooperative game albeit with lesser tokens and tightened privacy. In summary, HiddenTables and its pertinent experiments have brought forth the fol- lowing contributions to the academic community:
1 Introduction
We have devised construct that can com- plement an encoder-baue approach in table question-anse answering tasks for language mod-
siccoder-bseed approaches in contextually analyz- ing table question- ceastioo-answering tasks for language

---

Oracle: On P Premise
Solver: Off Premise
User Question
many people before been subpoenoed?
cutoff date "1990-01-07
Schema
subscenaed df cutoff [df[
Oracie encopsulates the schema of dota toble
True)]
Calore Calrre
num, subpoenaed len(subgoenaed)
(rint(f""cuu__sss" (ntr"" people")
Data Table
Ecalion
cutoff '1990-01-01"
["[["DO"] po. datetiree((( [oor)
-1..7
131223333
Secure
Code
suppoenzed [arl ['D0e']
Interpreter
cutoff date] a [dF[ SStppenn""
10:39
17/0991
"Yes"3]
nan, subpoenaed (en(subpaenaed)
KBB
230.09
621.09
080213222
3334-5555
Initial Code
mor
print (nun_subboenaes) people")
46d5
440558385
cutoff date '1990-01-01"
Answer Error
del 'D0 dff qutoff_datt (dF[ sutpoens "Y')l
AB5
326.9
080143222
64:40
Final Code
Code
nun subsoenasd [su(supponaadd) printy subpocnaod) people")
Answer
3 people
Conversation Channel
Figure I: Overview of our system apparatus to encourage HiddenTables. The setup requires two agents, an Oracle and the Solver, which may or may not be on the same device. For our purposes, the Solver is a 1.pt-3.5-urras LLM agent that handles generation off-site, and therefore potentially offers risk of adversarial attacks. We outline the conversation between our agents, which is pesssagepassing channel that transfers solution code along with follow-up questions, without exposing any information from datalake. Finally, the Oracle provide the answer to the user.
cls. as a less costly and more secure alterna- tive with significantly decreased risk in data exploitation.
Related Work
Since the advent of Transformer-based attention models, pre-trained language models have shown remarkable success in learning and encoding the semantics of tabular content (Vaswani et al., 2017). Methods employing encoder-based architectures rely on Masked Language Modeling (MLM) to learn semantics and dense representations of tab- ular content. Yet they are pre-trained on natural language text tokenized by byte-pair-encoding WordPiece (Devlin et al.. 2019: Sennrich al., 2016) which can misalign with tabular structure. TaPaS (Herzig et al., 2020) employed an en- coder that pre-trained with whole word mask- ing, TaBERT (Yin et al., 2020) leveraged Masked Column Prediction and Cell Value Recovery to learn structure, and GraPPa (Yu et al., 2021) aug- mented pre-training with synthetic SQL to inject structural properties into the model. In contrast these encoder-basse approaches, TaPEx (Liu al., 2022) relies on BART mncoder-decodee backbone (Lewis et al., 2019) to encode tables and generate answers in an autoregressive fashion.
Leveraging oode-generation capabilities of language models allows for a full chain of thought exposition via programmatic com- mands, enabling further interpretability into the answer retrieval process than what prior encoder or sequence-to-sequenee models pro- vided.
Our cooperative game is a robust demonstra- tion that the accuracy of gp 5-turbo de- creases rapidly when language models are not given the entirety of the data yet improves with consecutive rounds of feedback.
Therefore, our study contributes to not only the institutional adoption process language models but also the critical question of gen- cral intelligence capabilities language mod- els with regards to table question-answering tasks.
Additionally, HiddenTables has generated new dataset "PyQTax" that encom- passes 116,671 -aaittionntalle-nnweerr quadruplets of varying degrees and tax- onomies for promising future academic ex- periments.
However, HiddenTables relies solely on the gen- erative power of autoregressive decoders (Brown et al., 2020) and instruction-aligned models trained with reinforcement learning from human feedback

---

You are an Al Assietant that can answer questions from tables by writing python pandes code.
R
+ You will receive question and will have to write to best answer
O(1)
You will only be provided the columns to the table and their type.
Fallow these instructions"
must write python code operate a pandas dataframe nomed
O(1)
Instructions
Use (ind__) after any groupby operation involving aggregation, and sort'
....
+ 8) you think you cannol enswer the question, look for columns such ss note, comments, etc. that and the correct item"
contain the
Schema
has following columns: company(string), (g(Stniing) -.yy (alles(Murber). Asset (Number)" O(C))
Query
What the average assets in billions of the company Bank of United S States of America, which has less than $49.01 biliions sales?
0(1)
Figure 2: Outline of our Role, Instructions, Schema, and Question (RISQ) prompt template that the Oracle generates for the Solver. Each instruction was curated align the Solver's code work with our tables. For instance, all string comparisons are case insensitive and Unicode normalized. For each prompt component we outline the token complexity, which is bounded by the number of columns ((c) schema.
(RLHF) (Ouyang et al., 2022), to generate solu- tions based prompts (Liu et al., 2023) rather than fine-tuning.
guage models truly comprehend natural language and align language the correct solution when only given the table schema? In HiddenTables, two agents exist: the Oracle and the Solver. The clear delineation between these two agents' respective roles not only allows the user to test the model's holistic ability to comprehend tabular content but also enables the preservation of privacy with re- gards to the underlying data on-premise. In this context, our proposed apparatus allows the two agents engage in conversation. in which the Oracle may ask questions and the Solver will gen- erate code that could solve the Oracle's question. Next, the Oracle will evaluate and follow-up which enables the Solver to correct any mistakes or misun- derstandings. This game is played for a maximum of seven rounds to prevent infinite cycles between the agents. Throughout this process, no data entries exposed to the Solver the Solver must produce executable code relying solely the schema and the of instructions.
Furthermore, prior work shows that language models can more effectively solve problems when decomposing them into steps or a chain of thought (Wei et al., 2023; Nye et al., 2021). Hidden Tables is inspired by using chain of thought through code, demonstrated by (Liang et al., 2022) for robotic programs, action plan generation in robotics (Ahn al., 2022), web browsing (Nakano et al., 2022), tool APIs (Schick al., 2023), or the generation of valid programs for arithmetic computation (Gao et al., 2022). ReAct (Yao et al., 2023) explores how LLMs can improve their chain of thought reasoning via intermediate actions and interactions with exter- nal sources. Furthermore, BINDER (Cheng et al., 2023) demonstrated neural-symbot approach to mapping questions program, building upon the work in (Rajkumar al., 2022) for semantic parsing and code generation.
Also, previous literature has explored how LLMs can interact with themselves through intermedi- ate followups (Press et al., 2023), chained LLM prompts (Wu et al., 2022), or cascades (Dohan et al., 2022). (Reynolds and McDonell, 2021) has proposed how LLMs can be encouraged to gener- ate their own prompts for solving tasks. Finally, MemPrompt (Madaan et al., 2022) demonstrated that memories of errors and user feedback can be incorporated as part the conversation to help prevent repetitive mistakes.
3.1 The Oracle
The Oracle takes the user query and crafts ap- propriate prompt for the Solver, which is structured as a role, instruction, relevant schema, and the question (RISQ).. It will not exposc any individ- ual data entries in the table. This allows the Oracle to protect highly confidential information in fire- walled system from any adversaries. This prompt is then sent to the Solver, which is fully outlined in Figure 2. Furthermore, the prompt burden ($3.8) juxtaposed against holistic encoder methods (Table 1).
include discussion
3 Methodology
Our proposed framework is inspired from the "chi- nese room argument" to what extent could lan-
Further details about our RISQ prompt can be found in Appendix $C

---

Dataset
Split
Samples
Query
Table
Answer
Total
Train
56,355
913,649
31,357,980
229,264
32,500,893
Val
8,421
137,067
4.399,557
33,147
4,569,771
WikisQL
Test
15,878
258,362
9,562,411
65,560
9,886,333
Total
80,654
1,309,078
45,319,948
327,971
46,956,997
Train
11,321
146,080
9,180,294
57,677
9,384,051
WikiTable Questions
Val
2,831
36,525
2,153,166
14,448
2,204,139
Test
4,344
55,385
3,343,142
21,538
3,420,065
Total
18,496
237,990
14,676,602
93,663
15,008,255
Train
12,276
126,536
4,907,917
386,288
5,420,741
Val
2,265
23,113
947,744
70,886
1,041,743
SQA
Test 3,012
29,180
1,262,051
106,919
1,398,150
Total 17,553
178,829
7,117,712
564,093
7 7,860,634
Grand Totals
116,703
1,725,897
67,114,262
985,727
69,825,866
Table I: Number of Tokens required to be analyzed by gpt-3. b-turbb if holistic table encoding approach was adopted, as (Herzig et al., 2020; Liu et al., 2022). Query, Table and Answer totals are provided per dataset and in aggregate. Note that the largest table dimensions encountered were 1,956 rows, 44 columns, and 1 11,600 entries. Our system seeks to minimize the token usage through schemas only therefore bounding the number of tokens used number of columns, instead to number entries.
The Oracle also maintains the datalake in the Se- cure Interpreter, that executes the code produced by the Solver ($3.2). Moreover, the Secure Interpreter ensures that any request expose the dataset via code injections rejected and that only returns the answer to the user's query. We provide more details into the Oracle's followups Section $3.3.
the instructions to the Solver. The instructions are an itemized list that dictates the format of the Solver's ' response. The instructions and rationale outlined in Figure 2.
Next, the Solver responds with what deems be the best sequence of commands to answer the query. This is sent to the Oracle as free-text along with embedded code, including artifacts pertaining to explanations and chain of thought. Consequently, the Oracle sets up fire-walled with its dataset. Aforementioned, this environment ensures that any arbitrary execution of code is non-destructive and any exposure of the underlying tabular data is disabled.
3.2 The Solver
The Solver is gode-generating LLM agent that ac- cepts the Oracle's instructions, question, and tablu- lar schema. Then, strives translate and align the prompt into a sequence of executable operators that can be applied to the hidden table. prior literature, the main choice query language was SQL (Zhong et al., 2017); however, within our con- struct, the Solver does not need to be restricted to any specific programming language. HaddenTables opted to use Python the Solver's language of choice, it is dynamically typed, easily readable, and procedure-orientee to view the ththuuht through iterative com- mands. Finally, byproducts our generative exper- iments have yielded an amalgamation of verified python programs grounded to each question-table table- answer triplet that are linked to varying taxonomies we introduce this new dataset as PyQTax ($3.9).
environment, locally
As result of this conversation, there are two states that will be defined in detail state of "successful retrieval" or one of "failure". A state of successful retrieval is defined an answer has been generated from executing the Solver's code in the Oracle's secure environment. This answer could be text entry from the table, an aggregated value such as sum, or a list of table entries. In contrast, a state of failure is defined as an error message, such as Value or Index errors, NULL answers that provide no identifiable answer (empty dataframes) nor any executable code, or the Solver's comment that cannot answer. For each type of failure, the Oracle handles the state differ- ently. Firstly, errors can be sanitized remove any data references and fed back to the Solver, as prior literature regarding celf-correcting code has
in which
Therefore, convenient
3.3 The Conversation
We now outline the communication channel be- tween the two agents. Foremost, the Oracle sends

---

discussed (Madaan al., 2022). Secondly, empty dataframes can be conveniently identified with the Solver being informed that the generated code pro- duced no valid results. Thirdly, if the Solver is conservative answering the question and pro- vides no executable code, the Oracle reassures the Solver that the question can be answered from the table provided. Within this context, new failures be re-prompted to the Solver for correction by the Oracle.
3.6 Question, Table, and Answer Token Counts
Table I outlines each set's total token count for gpt-3. 5-turbb if sent to the model. The number of tokens were determined by OpenAl's fast BPE encoder tiktoken". The dominating term for token counts in the table entries themselves: 96.1% of the outstanding burden is located here. How- ever, previous encoder-based methods were lim- ited by the model's ' sequence length and memory constraint in computing multi-header attention be- tween every cell (Liu et al., 2022). In contrast, our construct is comparably linear in its token usage. If and the number
With this apparatus correct initial failures while retaining the original context throughout the conversation session, we allow the Oracle and Solver to interact for a maximum of seven times before the conversation is halted and the final ver- dict for this query is designated as a failure. We have discovered that failures are common when the answer is within extractive span in single table cell (free text) if the answer resides in generic column such as 'comments' or 'notes' that complicates contextual inference.
define the number of rows of columns as c, then the total token count for a table is polynomial O(re), which quadratic in time complexity as either term increases. However, in HiddenTables, since the only dependent variable required for solving table query is bounded by the number of columns O(c), token growth is linear. Each table could add e many rows yet our task will still include the same number of columns
3.4 Minor Roles
3.7 Privacy
The User The user's query initiates game of Tabden
Another by-product of this setup is privacy. Since row entries are omitted and safe guarded by the Or- acle, the Solver must form a general solution from the schema only. More importantly, the Oracle be configured with additional safety prompts and code policies to ensure that any adversarial attacks by the Solver are properly handled. However, this system may potentially need additional safeguards against side-channel attacks to obfuscate successful retrievals from failures (Kocher, 1996).
Datalake The Oracle has read-access access to data- lake, which stores the tables and entries in a secured environment on-premise.
Firewall This is boundary denote in Figure 1 the on-premise and off-premise environments the agents operate in. This setup can enable guided entry into the on-premise environment.
3.8 Prompt Burden
Given the replacement table entries with our RISQ system prompt, we analyzed the distribution of usage tokens for all three of our datasets. Of the 116, 661 samples accepted by the Solver and responded to (without error), the average prompt burden was 279 tokens with standard deviation of 19 tokens. The minimum, median, and maximum prompt usage was 243, 275, and 630, respectively. Overall, the total amount of tokens used in the Solver's ' system prompt was 32, 546, 634. This is only 18.5% of the burden incurred by using the entire table. As mentioned in $3.6, our construct is efficient for large tables with many rows, the token burden remains constant for each new row of
3.5 Benefits of Demarcating the Roles
Demarcating the boundaries between the Oracle and Solver is ensure that the underlying dataset is protected. This can be beneficial because firstly, for many institutions that handle sensitive or con- fidential data such as personally identifiable infor- mation, the Oracle can prevent any off- aff-premise entities from accessing the data but still help gen- erate answers. Secondly, this demarcation ensures that code is executed in regulated and structured manner, regardless of the user's location or de- vice. Thirdly, an additional layer of control has been generated, while still allowing third-party API providers to operate on the data.

---

data. Our Solver generated an average 115 tokens per answer, with standard deviation of 61 tokens.
Difficulty
Count
Train
Val
Test
Easy
618
78.7
75.0
82.4
Medium
55,659
72.3
72.5
72.2
3.9 PyQTax
Hard
19,303
59.2
60.1
60.0
HiddenTables has produced PyQTax that aligns 116,671 caasstiontabll--nnwee triplets to Python code. addition, PyQTax categorizes every ques- tion into varying taxonomies, such as difficulty, table size, question type, operator, and sequence length (for SQA), With these two additions, fur- ther research can be conducted into bolstering low- performing taxonomies and improving LLM code generalization in MiddenTabbes with Python.
Extra Hard
5,044
55.0
55.3
55.5
Operator
Count
Train
Val
Test
SELECT
57,923
72.1
72.9
72.1
COUNT
7,347
61.5
58.5
60.4
МΑX
4,639
56.7
55.4
54.5
MIN
4,631
57.8
60.3
59.2
AVG
3,135
43.2
44.4
44.2
SUM
2,949
68.0
67.9
71.7
Table Size
Count
Train
Val
Test
4 Datasets
Small
20,332
68.7
70.2
68.1
Average
39,799
68.9
68.6
68.5
WikiSQL (Zhong et al., 2017) The original pur- pose of WikiSQL was to translate natural language into SQL, and have repurposed this task to write Python code. WikiSQL is comprised of sim- ple questions selecting and filtering table entries (71.8% that align well with the table schema. Ag- gregation operations only comprise 28.2% of the questions. It consists of 80, 654 total examples over 24, 241 tables. However, 2% of the set's answers are incorrect according to (Herzig et al., 2020; Liu et al., 2022).
Large
20,493
66.2
66.7
67.3
# Rows
Count
Train
Val
Test
Small
22,985
68.5
69.8
68.9
Average
38,263
68.3
68.1
67.3
Large
19,376
67.5
67.8
68.9
# Columns
Count
Train
Val
Test
Small
31,192
70.6
72.1
70.3
Average
35,072
68.8
68.6
68.3
Large
14,360
61.5
60.3
62.7
Conv Rnds
Count
Train
Val
Test
Round 1
62,478
79.5
79.5
78.9
WikiTQ (Pasupat and Liang, 2015) WikiTable-
Round 2
10,247
37.0
36.0
36.9
Questions more complex guestion-answering set sampled from tables Wikipedia. It consists of 18, 496 examples over 2, 108 tables. Annota- tors were tasked with composing a series of com- plex questions involving operations such com- parisons, superlatives, aggregation, and arithmetic create a challenging QA dataset.
Round 3
3,108
36.7
33.3
39.8
Round 4
785
28.3
27.1
23.2
Round 5
536
30.9
29.8
30.9
Round 6
243
20.7
11.5
27.9
Round 7
3,227
1.5
3.2
1.3
All
80,624
68.2
68.5
68.
Table We provide breakdowns of each WikiSQL, split by complexity of the required operations produce the answer and by each aggregator. The best performing taxonomies are Easy and Medium difficulty questions, SELECT, and tables with small amount of columns. Medium style questions comprise 69% of the overall set, with hard at 24%. SELECT the dominant operator at 71.8% of questions. TaPEx achieved denotation accuracy of 89.5% WikiSQL-Weal
SQA (lyyer et al., 2017) Building upon WikiTQ, SQA decomposes compositional questions into se- quential orderings, in which each resulting question can be answered by one or more table cells. The main distinguishing factor for SQA is that the ques- tions are conversational, built up from prior queries. The set consists of 17, 553 examples, 982 tables, and 6, 066 sequences with an average sequence length of 2.06 questions. The median sequence length is 2 and the maximum questions.
the middle 50%, and large tables the upper quar- tile (/= 75%).. This enables outlier categorization into the pertinent buckets that guide the amount of content any model processes to produce an answer. For WikiSQL, the first and third quartiles are 247 and 607 tokens. For WikiTQ. the quartiles are 288 and 805. For SQA, the quartiles are 248 and 492.
5 Analysis & Discussion
5.1 Table Size
We breakdown our analysis based upon the in- terquartile range on table tokens small tables rep- resent the lower quartile ( 25%), average tables

---

Operator
Count
Train
Val
Test
View
Count
Train
Val
Test
Aggregate
4,854
45.6
44.1
44.1
Q1
6,065
51.8
51.5
58.0
Filter
4,418
39.9
42.9
40.6
Q2
6,064
33.8
31.8
32.2
Superlative
3,048
37.0
34.5
41.9
Q3
4,035
34.0
35.3
38.7
Comparative
2,497
42.6
35.1
42.8
Q4
1,106
37.3
36.0
47.6
Select
1,943
36.6
45.8
37.1
Q5
222
35.1
60.0
42.1
Arithmetic
1,562
32.0
31.2
33.2
Q6
38
32.1
0.0
44,4
Other
107
46.0
54.2
45.0
Q7
15
18.2
100.0
Group
67
35.3
41.7
57.1
Q8
6
16.7
Table Size
Count
Train
Val
Test
Operator
Count
Train
Val
Test
Small
4,505
43.7
46.7
45.5
Aggregate
596
31.8
34.5
36.9
Average
9,111
40.1
39.7
38.9
Filter
3,860
40.3
43.4
46.2
Large
4,870
36.9
38.0
41.0
Superlative
3,887
42.3
37.4
45.5
# Rows
Count
Train
Val
Test
Comparative
4,269
38.4
42.8
41.6
Small 5,555
39.8
42.0
43.4
Select
1,685
36.6
34.3
45.2
Average
8,355
41,4
39.2
40.5
Arithmetic
273
37.9
24.0
41.5
Large
4,576
38.6
40.3
39.8
Other
87
52.2
50.0
29.6
Group
20
45.5
55.6
# Columns
Count
Train
Val
Test
N/A
2,874
42.7
Small
7,346
40.6
38.3
40.1
Average
7,439
40.7
42.4
42.0
Table Size
Count
Train
Val
Test
Large
3,701
38.5
40.8
41.4
Small
4,545
42.1
42.6
48.0
Average
8,315
40.5
40.3
41.0
Conv Rnds
Count
Train
Val
Test
Large
4,691
38.1
37.4
44.2
Round 1
11,031
53.7
53.9
55.4
Round 2
3,258
32.4
31.9
31.3
# Rows
Count
Train
Val
Test
Round 3
1,129
25.4
27.2
22.7
Small
5,144
41.1
42.5
45.2
Round
441
20.4
17.5
14.4
Average
8,202
39.5
38.2
41.3
Round
192
20.5
29.6
23.7
Large
4,205
41.0
40.3
46.7
Round 121
22.5
20.0
15.4
# Columns
Count
Train
Val
Test
Round 2,314
1.3
0.5
1.8
Small
7,430
41.1
38.1
45.2
All
18,496
40.2
40.3
41.1
Average
7,259
39.2
42.8
42.8
Large
2,862
40.9
39.0
43.5
Table provide breakdowns of each WikiTQ. split by the type of operation, table size by entries, rows, and columns, and the number of conversation rounds required by the Solver. WikiTQ provides insight into how language models can handle complex QA chal- lenges. We employ few-shot categorization to label each question ($5.4). The best performing taxonomies are Aggregate and Comparatin for operators, small tables with limited entries, and tables solved in Round 1. Note that the Solver consistent in performance regard- ing row size. TaPEx acheived denotation accuracy of 57.00 57.0% and 5% on the Dev and Test set respectively.
Conv Rnds
Count
Train
Val
Test
Round 1
10,152
50.6
49.6
53.6
Round 2
3,574
43.9
46.7
45.7
Round
1,019
25.6
30.2
27.7
Round
358
17.4
21.3
21.3
Round
161
17.9
18.2
9.1
Round
111
21.1
21.1
43.8
Round
2,176
0.7
0.3
0.9
All
17,551
40.3
40.0
43.9
Table 4: Experimental results SQA for sequence lengths, operators, table sizes, and conversation length. Accuracy is reported only when applicable. Catego- rizations are reused from WikiTQ Solver performance is strong on QT. Filter, Superla- tive, Other operators, and conversation rounds & 2. The Arithmetic and Select operators are the most de- ficient compositional errors propagate downstream. TaPEx achieved SQA test accuracy of 74.5%
N/A otherwise.
We follow the same procedure for the number of table rows and columns, relying the interquartile range delineate small, average, and large tables. For WikiSQL, our quartiles for rows are Q1
Q3 18. For WikiTQ, our quartiles for rows are
Q 10, Q3 = 25. For SQA, our quartiles for rows are Q1 = 9. Q3 = 17. The first and third

---

Scenario
Conversation Rounds
Dataset
Split
Count
R2
R5
R7
Train
56,335
61.4
66.1
67.6
67.8
68.0
68.2
8,417
62.2
66.7
67.9
68.2
68.4
68.4
68.5
WikiSQL
Test
15,872
61.4
66.0
67.5
67.8
68.0
68.1
68.1
Total
80,624
61.5
66.2
67.6
67.9
68.1
68.1
68.2
Train
11,313
31.7
37.5
39.1
39.7
39.9
40.1
40.2
Val
2,831
32.6
38.1
39.5
39.9
40.2
40.3
40.3
WikiTQ
Test
4,342
33.6
38.9
40.3
40.6
40.8
40.9
41.1
Total
18,496
32.3
37.9
39.5
39.9
40.2
40.3
40.3
Train
12,274
28.0
38.0
39.6
39,9
40.1
40.2
40.3
Val
2.265
27.7
37.4
39.2
39.6
39.8
40.0
40.0
SQA
Test
3,012
33.5
41.7
43.1
43.4
43.5
43.8
43,9
Total
17,551
29.5
38.6
40.1
40.5
40.6
40.8
40.9
Table 5: Ablation results for the cumulative accuracy gains per additional conversation round. Each round includes the cumulative total of correct solutions, even the conversation ended prematurely. Incremental gains accuracy level off after the third conversation round, consequence of a dwindling pool of remaining unsolved problems. Furthermore, issues from parsing persist in the later conversation rounds as the Solver struggles to find the right formats forgets the original task.
column quartiles were Q1 = 5, Qs = for all three datasets. Our experiments show that demar- cations for columns show the largest differentials in performance favoring small tables, while our Solver is consistent across any number of rows. It is difficult generalize the performance regarding table entries since the size is obfuscated by either the number rows or columns.
to the number of components our Solver must in- teract with to arrive at an answer, which is pro- portional to the number of operations any Python program must also use. We designate queries by the number of AST nodes such that Easy is S 8, Medium is 15, Hard is 20, and Extra Hard 20. The experimental results for WikiSQL are provided Table 2.
5.2 Conversation Length Cumulative Accuracy
Operator Difficulty We also evaluate in Table 2 the accuracy of our approach by SQL aggrega- tor, which includes SELECT, MAX, MIN, COUNT, SUM, and AVG operations. WikiSQL is relatively simple as reflected by 71.8% of SELECT questions, with COUNT as the next prominent operator at 9.1%. The top operators are SELECT and SUM. In contrast, Hid- den'Tables exposes 3.pt-3.5turaee fetching extrema within a column with MIN/MAX or simple counting. AVG underperforms, as signifi- cant number of tables include grand total entry.
For all datasets, show the necessary number of attempts to write fully executable code. Our experiments show that while the probability of successful retrieval decreases with more rounds, considerable number of samples are being solved correctly each round. As reported in Table 5, HiddenTables sees significant cumulative increases in the Solver's s accuracy when paired with Oracle agent for the first three conversation rounds. Af- terwards, additional rounds yield very diminished accretive benefits.
deficiency in
5,4 WikiTTableQuestions
Operator Difficulty We tag each question in WikiTQ as a Select, Filter, Aggregate, Su- perlative, Arithmetic, Comparative, Group or Other operator, as inspired by (Liu et al., 2022), to further understand the limitations regarding gpt-3. Ept-3.5-turke 5-turbo. Table 3 enumerates the opera- tor types and the performance breakdown by split. In order to quickly tag each question, we used 7-shot approach using one example per type of question, then leveraged 3pt-33.5turfe to gener- the best category for the question. This provides insight into how the model handles each question during inference time, as the same assumptions
5.3 WikiSQL
SQL Query Difficulty Following a similar anal- ysis by (Yu et al., 2018; Liu et al., 2022), we break- down our WikiSQL results by difficulty, yielding insights into how well the Solver can assemble the required steps based on how many SQL elements appear in the original query. For our analysis, used SQLGlot² create an abstract syntax tree that shows the query's complexity. The number of nodes in an abstract syntax tree (AST) corresponds
Phtnps//github..co//abamymoo

---

categorizing the question influence the generated code.
lem. This results in fewer tokens required during the inference step of an LLM, resulting in lower latency and faster response times. Our approach used 48.5% of the total tokens, table contexts are considered. This proportion will decrease as table sizes increase in either rows or columns.
5.5 SQA
Dependency Difficulty As conversa- tional dataset, SQA allows the profiling of 5tt-3.5-turbo's performance on follow-up questions. In Table 4, we denote the accuracy across several facets. We profile the overall accuracy for each sample and denote the accuracy for the sequence. For intermediate questions Qi we showcase the accuracy of the i-th question in the conversation. As expected, highly composi- tional questions tend to struggle more than initial sequence questions.
Hidden Tables comes with drawback in terms of accuracy. When relying solely the schema, the problem shifts from multi-span extraction task to semantic parsing and code generation task. This added complexity requires LLMs to interpret and comprehend the question alongside the table structure. As result, we see that HiddenTables s final accuracy is below TaPEx (Liu et al., 2022). By forcing L.MSs to align the interpretation of queries to structure, errors in understanding the format of data dominates most failure cases. While addi- tional conversation rounds mitigate this risk, other errors such as relying on extraction within full text column still prove difficult.
Operator Difficulty Since SQA builds off of compositional questions from WikiTQ, there is sig- nificant overlap between the two. Therefore, reuse our generated 7-shot question taxonomies for all SQA samples found in the WikiTQ set. If not found, the category defaults to N/A (2,874 sam- ples).
Conclusion
In this work, we introduced a novel approach to evaluating the generalizability of LLMs across 3 table question-answering datasets. By creating a co- operative game that withholds the underlying data from the the model, HiddenTables challenges the Solver to make educated guesses via programmatic commands and operators be in a state of suc- cessful retrieval. We have shown that this construct enables a computationally efficient large-scale test- ing of L.Mss on massive datasets in tandem with ensuring the security of the tabular data. Also, our study provides insights that this task is consider- ably more difficult than traditional holistic models yet lends itself to potentially large-scale indus- trial applications. We have also quantified this efficiency by showcasing the number of generated tokens in contrast with those of conventional mod- els. We also contribute PyQTax, a dataset aligning generated python code to table questions and vari- ous taxonomies for 1 116,703 samples. Overall, our work provides promising direction for future re- search in the field of table question-answe answerrin and has devised novel construct in the deployment process of language models.
5.6 Privacy & Efficiency vs. Accuracy: Tradeoff
HiddenTables has demonstrated that in order to have full privacy and efficiency in the context of table
the lack of illustrative examples or the holistic table degrades accuracy. Privacy is a crucial concern when working with sen- sitive data, especially in industries that are highly regulated. By generating code derived only from the question and schema whole table, data exposure can be limited. There- fore, the Oracle, via the Secure Interpreter, only ac- cesses the relevant portions of the data on- on-rremise, mitigating the risk of any data leaks. HiddenTables compensates the substantial increase in difficulty from blindly solving TableQA by implementing the pair-programming iterative approach between the Solver and Oracle LLMs, as outlined in The Con- versation ($3.3). This iterative approach to prob- lem solving yields +6.7% increase for WikiSQL, a +8.2% increase for WikiTQ, and a +11.4% in- crease in SQA.
table, ruther than the
Efficiency is another consideration regarding large knowledge bases computationally inten- sive tasks. First, generating code allows systems to focus computational resources on subsets of the data internally, rather than processing the entire set as a multi-span extraction or aggregation prob-
Limitations
While our work presents a novel approach evaluating the generalizability of LLMs on table- question answering datasets, it imperative to dis-

---

cuss several limitations to our system. Foremost, our approach requires a Solver to generate code and answer the user query, which may be infea- sible. Additionally, our system's 's reliance on pro- grammatic commands and operators may result in a lack of flexibility when it comes to answering certain types of queries.
References
Michael Ahn, Authony Brohan, Noah Brown, Yov- gen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz. Brian lchter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jes- month, Nikhil Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Kuang-Huel Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Pe- ter Pastor, Jomeel Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes. Pierre Sermanet, Nico- las Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. 2022. Do as can and not as i say: Grounding language in robotic af- fordances. In arXiv preprint arXiv:2204.00 01691..
Next, while Tabdes protects the informa- tion in the tables by withholding the underlying data from the LLM, it may not be able to address the issue of data privacy in cases that the table schema may contain sensitive information. More- over, our system's 's reliance on an Oracle to evaluate the Solver's 's code may not be scalable in cases when there is high volume in user queries.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Tlya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Lastly, while our results demonstrate the effec- tiveness on English language datasets, its scala- bility to other languages with more complex mor- phologies and diacrities is area that requires further investigation. Additionally, questions tailored to each dataset, where WikiSQL questions reiterate column names to align language to table retrieval. The discrepancy between experimental questions and real-life user queries can be substan- tial and warrants further investigation, In summary, while our system presents a promising direction for future research in table question-answering, these limitations must be acknowledged to enable its wider adoption.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. Binding language models symbolic languages.
Acknowledgements
David Cole. 2023. The Chinese Room Argument. In Edward N. Zalta and Uri Nodelman, editors. The Stanford Encyclopedia of Philosophy, Summer 2023 edition. Metaphysics Research Lab, Stanford Univer- sity.
This paper was prepared for informational purposes by the Artificial Intelligence Research group of JP- Morgan Chase & Co and its affiliates ('J.P. Mor- gan"" and is not a product of the Research De- partment of J.P. Morgan. J.P. Morgan makes no representation and warranty whatsoever and dis- claims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment re- search or investment advice, or recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistiex: Human Language Tech- nologies. Volume (Long and Short Papers), pages 4171-4186. Minneapolis, Minnesota. Association for Computational Linguistics.
David Dohan, Winnie Xu, Aitor Lewkowycz, Ja- cob Austin, David Bieher, Raphael Gontijo Lopes, Yuhuai Wu, Hemryk Michalewski, Rif A. Saurous. Jascha Sohl-dickstein, Kevin Murphy, and Charles Sutton. 2022. Language model cascades.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2022. Pal: Program-aided language models. ArXiv, abs/2211. 10435.

---

Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4320 4333, Online. Association for Computa- tional Linguisties.
David Dohan, Aitor Lewkowyez. Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratchpads for intermediate computation with language models.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Mohit lyyer, Wen-tau Yih, and Ming-Woi Chang. 2017. Search-based neural structured learning for sequen- tial question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1821 1831, Vancouver, Canada. Association for Computa- tional Linguistics.
Panupong Pasupat and Percy Liang. 2015. Composi- tional semantic parsing on semi- structured tables. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages 1470- 1480, Beijing, China. Association for Computational Linguistics.
Paul C. Kocher. 1996. Timing attacks on implementa- tions of diffie-hellman, rsa, dss, and other systems. In Advances in Cryptology - CRYPTO '96, pages 104-113, Berlin, Heidelberg. Springer Berlin Heidel- berg.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zeulemoyer. 2019. Bart: Denoising suueence-t--eqqeeee pre-training for nat- ural language generation, translation, and compre- hension. arXiv preprint urXiv:I9100 13461.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bah- danau. 2022. Evaluating the text-to-sql capabilities of large language models.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian lchter, Pete Florence, and Andy Zeng. 2022. Code as policies: Language model programs for embodied control. In arXiv preprint arXiv:2209.07553.
Laria Reynolds and Kyle McDonell. 2021. Prompt pro- gramming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Com- puting Systems, CHI EA '21, New York, NY, USA. Association for Computing Machinery.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput, Surv. 55(9).
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer. Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022. TAPEX: Table pre-training via learning neural SQL executor. In International Conference on Learning Representations.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers). pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.
Aman Madaan, Niket Tandon, Peter Clark, and Yim- ing Yang. 2022. Mesory-assisted prompt editing to improve GPT-3 after deployment. In Proceed- ings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 2833-2861, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser- assisted censstionnanswring with human feedback.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elic- its reasoning in large language models.
Tongshuang Wa, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie Cai. 2022. Promptchainer: Chaining large language
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber,

---

model prompts through visual programmiing. Ex- tended Abstracts of the 2022 CHI Conference on
B Implementation: Secure Interpreter
Human Factors in Computing Systems, CHI EA '22,
To execute code generated by the Solver, we pro- vided the Oracle a secure interpreter that can di- reetly interact with the data on-premise. This means that our setup, in order to preserve privacy, is executed locally. The Solver's generated out- put is checked for any malicious code, in case of a potential attack through code injection or exter- nal requests. First, the interpreter is fire-walled have no external connections, as the data already on on-premise. Second, the interpreter does not allow for any additional packages to be imported. The generated code is inspected for import *, import * *, from import * and replaced with empty string. The namespace of the interpreter is pre-installed with verified packages. Finally, avoid malicious code intended to erase or corrupt data, all operations are performed on a copy of the table. a copy is not feasible, the database only allows for read operations. Any write or in-place operations on the source data are strictly denied. In- termediate artifacts are allowed to be manipulated during execution.
New York, NY, USA. Association for Computing Machinery.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023, React: Synergizing reasoning and acting in language models.
Pengcheng Yin. Graham Neubig, Wen-tau Yih, and Se- bastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426, On- line. Association for Computational Linguistics.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang. Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2021. Grappa: Grammar-augmennte augmented pre- pre-training for table semantic parsing. In International Conference Learning Representations.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao. Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic pars- ing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods Natural Lan- guage Processing, pages 391 1-3921, Brussels, Bel- gium. Association for Computational Linguistics.
C Instructions for RISQ
We outline the instructions and the (rationale) in parenthesises.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning.
You must write python code and operate on a pandas dataframe named df. (Aligns Solver to the start variable to operate on)
A Few-Shot Categorization of Questions
2.
Use reset_index () after any groupby opera- tion involving aggregation, and sort (Common eттor is to access an aggregated variable, yet Pandas stores these in the table's 's index)
To provide better clarity into the generalizability of gpt-3. totrbo, we breakdown WikiTQ into seven categories of questions. By using the same LLM and the Solver, we gain insight into how gpt-3.5-turbe recognizes and understands what kind of operations should be performed for given question, based on semantics. To label each ques- tion, we select representative example for each question category, and provide this as a 7-shot prompt to the model. We include the candidate question and a directive to label it, then parse and reconcile the generated category with the pre- scribed eight (Other is fallback category). For SQA, there is an overlap between WikiTQ, and therefore we reuse the same labels when applica- ble. See Table 17 for an example of each question type, plus the semantic span that correlates with the category.
3. All contains MUST BE case insensi- tive (case False). (Helps improve the hit rate within column when filtering)
4. Do not use inplace operators save each inter- mediate variable (Improves chain of thought with code by saving each step as a new vari- able)
5. The final answer must be saved as iinal_answer (Easier to find the gen- erated answer, although optional)
Do not ask for clarification, you have every- thing you need to answer the question through the column headings (Gives confidence to

---

3pt-3.5-ture directly answer the ques- tion and take risks)
20
21
12 Q2: Who was the top winner 2002 of the division undergraduate?
7. You cannot look at the data just write code instead (Reiterates the main generation objec- tive)
23
24
25 Group by Year and Division 26 Undergraduate, and find the max
27 the Division I Overall
78 g grouped of -f- groupby Year', "Division Undergraduate }) D'Division I Overall J. max ()
8. If you think you cannot answer question, look for column such note, comments, that may contain the answer and return the row item. (Guides gpt-3. 5-turb to attempt re- trieval code on free-texi columns, which may contain the answer)
29
30
31
32
Reset the index
33 grouped grouped. reset_index
34
35 Filter the result for the year 2002
36 and sort by Division I Overall fi filtered = grouped[
Sample Solver Code
37
38
grouped[ == '2002'
]. sort_vllees( by='Division I Overall', ascending=false)
Q1: what was the highest vote number for Bush? max vus_votes_bush "[''Bus*"]] "Iah#"J.aa maxax
#
The top winner is the first row 44 of the filtered result final_answer filtered. il]c[0][ DDivssoo I Undergraduate
# Q2: What is the Position of the
7 ff person with birthplace of phoenix, new yorsk?
#
]
final Linal_answer = df[
Listing 2: Generated Python Code from Natural
df[ Birthplaceeeess s..str. contains
'phoenix, new york', case=False)
Language Questions (WikiTQ)
][ Position ilo
Q1: S1| Who are all the players? all_players = df[ Pllayer' ']. unique
Q3: Which Home team has Tie no #
of replay, Date of
Q2: S1 What event did an American get medal in?
november 1985, and
#
Away team of tranmere rovers? Final_answer df[
df[ df[ Gold']. containss
(if no no'l. contains)
"USA United States', case=false) Ji containss Unntte States', case=false)
'replay". case=False))
(dte'Date']
(df['Awa) november 1985')
'tranmere rovers')
d contacnna
][ Home team 'J. iloc[0]
UnSA|United States', case=false)]
Listing I: Generated Python Code from Natural
final_ (medal_winners ['Event J. i.)
Language Questions (WikiSQL)
What is difference, years, between the Founding of the heidelberg football club eltham football club? df[
2
21
Q3:
S1 Which baseball players have than 500 hame
scored runs?
the
23
5 #
24
S2 those, which have scored less than 600?
siidelberr_foundder df[' cl strtrr contains
25
26
S3 among those players reached the mark se0 home runs in month of september of any year?
'heidelberg' case=False) Fouundd [0]
27
28
10
29
11 eltham_ founded df[
eltham_founded
30
$4 these who has the least home runs?
12 Club str.r. contains(
31
13
"hllmha". case=False)
32
more. more_ chan_500 df[ df[ HR ]. as type (int) 580]
14
Fouuudd iloc [0]
33
15
34
16
difference = abs(
35
less_ than_haaaa_ 600 more than_ 500
17
36
more_ tire__tha__5d00 HR astype (int)
18
than_
37
< 600]
19
38

---

september _50 df[ Date reached 500 .srr. lower ()
Solver produced code that accesses a column not available the transformed tables, either through hallucination or as byproduct of aggregation.
42
str.. contains ( ''eptemeer', case=false)]
Examining the Effect of Table Size on Performance
least_home runs
less _ess_than tha_thhn_eee 680. loc[ less_ than_ 608 HR ast (int) .idxmin(, P]ayer']
46
Listing 3: Generated Python Code from Natural
Partition
Small
Average
Large
Easy
86.2
78.3
73.6
Language Questions (SQA)
Medium
73.8
72.7
69.7
Hard
58.8
59.6
59.8
Extra Hard
53.5
56.0
55.0
E
Common Error Codes
SELECT
74.0
72.6
69.2
COUNT
64.4
60.9
57.8
MAX
51.8
57.0
59.5
Error Code
Count
Percent
MIN
55.1
60.0
58.7
No Code Provided
4,573
51.17
AVG
41.3
44.9
43.7
IndexError
1,254
14.03
SUM
69.0
68.4
68.8
AttributeError
865
9.68
ValueError
715
8.00
Table 7: Cross- cooss-taxonomy accuracy for all WikiSQL sets by difficulty and operator against table size (tokens).
KeyError
567
6.34
IndentationError
346
3.87
For difficulty, we see performance degrade as the overall table increases.
NameError
300
3.36
SyntaxError
160
1.79
TypeError
102
1.14
DateParseError
14
0.16
OutofBoundsDatetiime
12
0.13
Partition
Small
Average
Large
IndexingÉrror
12
0.13
Aggregate
50.2
45.0
42.3
RedefinitionErroo
Filter
48.1
41.2
35.5
IntCastingNaNE
Superlative
43.3
36.0
36.7
Undef IndefinedVarian rablerrr
Comparative
46.2
39.8
40.8
FileNotFoundError
Select
40.2
38.9
36,2
InduleNotFoundErree
Arithmetic
36.2
32.4
29.4
Other
25.0
53.8
46.2
Table 6: Common error codes encountered during HiddenTables, with proportional percentages. Total number encountered: 8,937.
Group
55.6
41.5
41.2
Table Cross-taxonomy accuracy for all WikiTQ sets by operator against table (tokens). Generally, performance decreases tables grow larger in tokens.
We outline common errors in Table 6. The largest failure case was gpt-3.5-turbo not providing any executable code. This usually occurs when a ques- tion does not aligning with any column names. Furthermore, IndexError exclusively occurs at attempting to directly access table value that is strictly out of bounds for the index, which is expected if the Solver does not know how many records are contained in the table after a Filter, Comparative, or Superlative operator. The next most common issue was an AttributeError, of- ten triggered by gpt-3.5 3pt-3.5-turbe. turbo being unable to infer the correct type of variable the code oper- ates on. For instance, the most common objec- tion of the interpreter was "Can only use .str accessor with string values!" indicating failure correctly apply string methods onto pan- das dataframe. ValueError arose when boolean indexing that had NA NaN values of which a fix is to include str. www.sttccttatt contains na=False). Finally, KeyError is fairly straightforward the
Partition
Small
Average
Large
Q2
52.9
53.6
51.3
Q2
35.9
32,5
32.3
Qs
40.4
33.8
31.8
Q4
41.5
39.0
37.8
Qs
43.9
33.9
45.8
Qs
14.3
50.0
18.2
Q7
50.0
33.3
40.0
Qs
0.0
0.0
50.0
Aggregate
38.9
34.3
23.6
Filter
42.7
42.0
41.9
Superlative
46.2
39.8
44.2
Comparative
41.2
34.3
Select
42.3
37.2
39.7
Arithmetic
22.7
35.2
40.2
Other
55.6
46.7
58.8
Group
45.5
55.6
39.4
N/A
43.2
42.5
42.5
Table Cross-taxonomy accuracy for all SQA by question sequence and operator against table size (tokens). Generally, performance decreases as table size increases, with few exceptions.

---

G Examining the Effect of the Number of H Examining the Effect of the Number of Rows on Performance
Columns on Performance
Partition
Small
Average
Large
Partition
Small
Average
Large
Easy
80.1
79.1
75.3
Easy
78.8
77.7
80.6
Medium
71.9
72.7
72.5
Medium
75.9
72.7
64.0
Hard
58.7
58.6
61.5
Hard
61.2
59.0
54.8
Extra Hard
55.4
53.8
56.2
Extra Hard
58.3
54.8
49.9
SELECT
72.8
72.1
69.2
SELECT
74.5
72.1
69.2
COUNT
61.4
61.2
60.0
COUNT
60.4
62.8
58.1
MAX
55.0
55.0
59.4
MAX
57.3
55.9
54.7
MIN
58.9
57.7
58.5
MIN
57.2
57.9
61.0
AVG
42.7
43.4
45.2
AVG
51.6
40.6
29.7
SUM
69.7
67.0
69.6
SUM
68.3
69.8
66.3
Table Cross-taxonomy accuracy sets by difficulty and operator against the number of table rows. There are no discernible trends, highlighting that HiddenTaales not dependent on the number of rows for performance. Therefore, the trends in Table 7 are exclusively driven by the number of columns.
WikiSQL
Table 13: Cross-taxonomy accuracy for all WikiSQL sets by difficulty and operator against the number of table columns. As difficulty increases, the number of table columns has more influence on performance, yet for simple questions shows differentiation. No dis- cernible trend can inferred for SQL operator.
Partition
Small
Average
Large
Partition
Small
Average
Large
Aggregate
44.3
45.9
Aggregate
45.8
45.4
42.6
Filter
43.1
40.3
Filter
40.1
40.0
Superlative
40.0
35.5
Superlative
37.6
32.2
Comparative
40.7
43.2
39.5
Comparative
38.8
41.6
46.2
Select
36.1
39.3
Select
38.0
36.6
Arithmetic
33.5
32.7
29.5
Arithmetic
29.5
33.1
35.0
Other
36.4
60.0
44.4
Other
49.1
53.1
33.3
Group
37.5
47.6
33.3
Group
38.7
41.7
41.2
Table 11: Cross-taxonomy accuracy for all WikiTQ sets by operator against the number of table rows. Gen- erally, performance for most partitions greatest on average sized tables.
Table 14: Cross-taxonomy accuracy for all WikiTQ
sets by operator against the number of columns. Per- formance increases with more columns, suggesting that question complexity plays greater role than operator.
Partition
Small
Average
Large
Partition
Small
Average
Large
Q1
53.4
51.5
54,5
Q2
53.5
51.8
53.4
Q2
34.1
32.3
34.4
Q2
33.1
33.6
33.2
Qa
36.5
34.1
34.7
Qs
35.4
34.4
35,4
Q4
42.1
35.7
41.8
Q4
42.1
36.9
38.7
Q5
37.5
34.0
48.1
Qs
40.5
38.1
44.0
Qa
28.6
40.0
27.3
Qa
30.8
31.6
50.0
Q7
40.0
40.0
40.0
Q7
25.0
37.5
66.7
Qs
0.0
0.0
50.0
Qs
0.0
25.0
0.0
Aggregate
34.2
33.5
31.7
Aggregate
37.5
33.6
27.0
Filter
42.0
40.7
45.1
Filter
41.6
41.3
45.8
Superlative
42.7
40.1
46.2
Superlative
40.6
43.4
42,5
Comparative
42.5
39.1
37.7
Comparative
41.8
38.5
37.8
Select
43.4
36.5
35.6
Select
37.4
39.1
37.5
Arithmetic
28.9
32.3
48.5
Arithmetic
38.3
31.0
39,2
Other
38.5
60.7
36.4
Other
38.3
47.8
58.8
Group
45.5
55.6
Group
45.5
55.6
N/A
41.6
42.0
45.7
N/A
44.9
39.6
44.2
Table 12: Cross-taxones cross-taxonomy accuracy for all SQA sets question sequence and operator against the number of rows. Filter increases in performance, perhaps being agnostic the number of items, while Aggregate shows increased sensitivity to the inclusion of outliers.
Table 15: Cross-taxonomy accuracy for all SQA sets by question sequence and operator against the number of columns. There no discernible influence of columms on the performance of HiddenTables.

---

OP
Count
Query
Answer
COUNT
Name the number F'bruariee
12
Easy
MAX
264
What was the highest home total?
91143
MIN
345
List the lowest number of assists
10
SELECT
8
What names of the episodes that airs 2:00pm?
[-..]]
AVG
1,208
Name the average allendance with Dallas visitor
16839
Medium
COUNT
5,132
For how many years was urban population 57%?
4
MAX
2,386
What was the largest crowd where the home team was Fitzray?
18000
MIN
2,291
What year was the tournament first held Italy?
1930
SELECT
43,512
What are all the results for New York 7th district?
re-elected
SUM
1,130
How many specimens does the SANT instituion have?
71000
AVG
1,407
Name the average pop for Chūgoku and pretecture of Okayama
700646
COUNT
1,658
What is the total when silver is < 1 and rank >> 32
1.0
Hard
MAX
1,514
What is Honda's is highest grid with a time of 41:3
7
MIN
1,489
What was the lowest total for Italy when the latest win is 1950?
7
SELECT
11,893
What was the tonnage of the Great Britain ship Batna?
4399
SUM
1,342
What is the number of points ofr East Germany, and Places of 88?
170.54
AVG
520
Tell me the average rank for losses < 6 and wins < 11 for michigan state.
10
COUNT
556
How many years did the NY Giants win with result of 15-7 at Lincoln Financial?
1
Extra Hard
MAX
475
What is the latest year Rafacl Nadal was in the French Open,
2007
Roger Federer Wimbledon, and Roger Federer in the Australian Open?
MIN
506
What is the lowest total with rank <8, silver > 6, and 20 the bronze? Which Production in 2009 had a Result of Nominated
57
SELECT
2510
wicked
the Helpmann awards Award Ceremony?
What is the sum of Population when Area 5131, Pop Density 180.
SUM
477
72660
Subdivisions Parishes, and Capital Roseau?
Table 16:
Cross-taxonomy samples for all WikiSQL. sets by difficulty and operator. The diversity in answer types warrants a flexible approach to table QA through code. We hope enumerating samples without the corresponding tables proves that Hidden'Tables is difficult game, and the Solver may have to make several attempts before successful retrieval is made by the Oracle.
OP
Query
Answer
Betore 1999, how many series occurred?
6
Aggregate
How many species of hirds there in Guatemala?
684
What the total amount of students who took the test in 2007?
97136
Who was only Candidate with hometown of Tulsky?
Alissa Joanndova
Filter
Name the nations that won at least five silver medals.
Puerto Rico
Which song charted the but not the UK?
Set the Night Music
What opponent is at the top of the Chart?
Japan
Superlative
Which country took the least amount of time?
United States
Which team was the runner up the most times?
Arsenal
Which event occurred first: St. Paul Open or the Charlotte Open?
Charlotte Open
Comparative
Which nation won the same number of gold medals as Hungary?
Bulgaria
What country had the least amount of drivers, Germany or the UK?
Germany
What is the name of the first venue this list?
Riverside Montien Hotel
Select
In what country is Bologna?
Italy
Which 1965 film starred actors Elizabeth Taylor & Richard Burton?
The Sandpiper
How many more AM channels there than FM channels?
9
What the difference of weight between the Maria Bell & the Carolus Bell?
3145
Arithmetic
What was the difference, time, between the first place competitor
+0.400
and the third place competitor?
For each winning game, what was their score?
6-1
Group
What was the ranking each November game?
Name winners of the Carihbean Cup
Trinidad & Tohago Solano 3
What is next after chuchillo -2?
Other
What was the first outcome listed on this chart?
Winner Alberto García
What is the first name ranked?
Table 17: Taxonomy samples for WikiTQ generated through 7-shot classification procedure, outlined in the Appendix SA. We also highlight key semantics within each sample that aligns for the category.


Reviews:
[{"review_id": "aCVtxEhqjL", "reviewer": "Reviewer_fyQe", "summary": "", "questions": "Question A: How the system performs in the case of the table contents available to the \"Solver\"? Is it still underperform TAPEX?\n\nQuestion B: How does the Secure Interpreter check that the code received from the “Solver” is not malicious?\n\nQuestion C: Why are you using the complexity of the original SQL query as a measure of the difficulty in PyQTax which is in Python? Do you have any evidence that the complexity of the solution in SQL matches that of Python?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for engaging with our work and highlighting the key contributions our work presents to the community.  The reviewer’s feedback has provided great insights and we have great appreciation for this opportunity to further clarify any points.  In response to the raised points, we provide the following comments:\n1. Lack of baselines: We focused on evaluating the proposed system based on the “Solver”/“Oracle\" ChatGPT against BART-based TAPEX models to demonstrate its efficacy. However, we agree that including additional baselines would provide valuable insights for an end-to-end (full information) QA with ChatGPT. We will also expand the discussion surrounding the conversation rounds against a single round (schema-only) to demonstrate the benefits of the interactive system (by expanding section 5.2 Conversation Length & Accuracy). We will incorporate this baselines in our revised work to provide a comprehensive evaluation of our approach. For the single ChatGPT attempt (schema only), the accuracy for WikiSQL is (61.4 Train, 62.2 Val, 61.35 Test) which is 6 below the interactive version. Additionally, WikiTQ is 31.7 train / 32.6 val / 33.6 test (8 points below the interactive) and SQA is 28.9 train / 27.7 val / 33.5 test (10 points below the interactive game). Hence compared to a single round, our interactive game demonstrates the efficacy of improved performance.\n2. Lack of details about the Secure Interpreter: We agree that additional provision of details regarding the Secure Interpreter used in the \"Oracle\" component would be beneficial for our research. Our details for the secure interpreter will be expanded from lines 201-207, and for additional clarification, “the secure interpreter is a firewall python interpreter that does not have any external internet connections.”\n3. Question A) Performance of the system when the table contents are available to the “Solver\": We will include an analysis of this scenario in our revised work, to highlight the challenges posed by limited table information.\n4. Question B) Malicious Code & the Secure Interpreter: In the revised version, we will expand lines 201-207 with “The Secure Interpreter used in our system is a subprocess python shell, isolated within a virtual environment, and strictly prohibits write or external request access, ensuring its integrity and preventing any potential damage.” We will also expand implementation details in Python in the appendix.\n5. Question C) Complexity of the original SQL as difficulty for Python: Considering the similarities of operations between Python/Pandas and SQL, we believe the complexity measure based on the abstract syntax tree of the ground truth SQL query serves as a reasonable proxy for assessing the difficulty of Python solutions in PyQTax.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors propose an approach for TableQA which leverages the external LLMs while protecting the data in the tables, which would be applicable in real applications.\n- The proposed method has no restrictions on the size of the table since only the schema of the table is passed to the model to generate the solution.\n- Another contribution is a new dataset for TableQA with Python code instead of SQL that could be useful for future research.\n- The paper is well-written and easy to follow, and the proposed approach is simple and easy to replicate.", "reasons_to_reject": "- The main weakness of this paper is the lack of baselines. The authors experiment only with the proposed system based on the ChatGPT and BART-based TAPEX models, which does not show how certain components of the approach affect the quality. In particular, there is no baseline in which “Solver”, ChatGPT, has the full information about the table. Such a comparison would show how much more difficult the task is if the table content is not available for the model. Another missed baseline is ChatGPT as is (without any interactions) which would provide the basic performance on the TableQA task and probably show the benefits of the interactive systems like the proposed “Solver”-“Oracle”.\n\n    Update: I acknowledge that the authors have provided a non-interactive ChatGPT baseline in their rebuttal. I encourage them to include these results in the next revision and add the \"full information\" baseline to provide a deeper understanding of the proposed system.  \n- The authors do not provide any details about the Secure Interpreter used in “Oracle” which can hinder reproducibility.", "questions_to_authors": "Question A: How the system performs in the case of the table contents available to the \"Solver\"? Is it still underperform TAPEX?\n\nQuestion B: How does the Secure Interpreter check that the code received from the “Solver” is not malicious?\n\nQuestion C: Why are you using the complexity of the original SQL query as a measure of the difficulty in PyQTax which is in Python? Do you have any evidence that the complexity of the solution in SQL matches that of Python?", "paper_topic_and_main_contributions": "This paper presents an adversarial approach “HiddenTables” for Table Question Answering which consists of two components, “Solver” and “Oracle”. The first one, “Solver”, predicts executable operations only relying on the table schema. This enables the usage of external LLM while protecting the table content. The “Oracle” executes the code from the “Solver” in a safe environment and passes the feedback. The proposed approach can potentially work with tables of arbitrary sizes. In addition, the authors propose a new large-scale dataset for Table Question Answering, PyQTax, which is built on top of the existing datasets but utilizes Python instead of SQL.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g7HaIf57si", "reviewer": "Reviewer_xnAr", "summary": "", "questions": "1) \"Our adversarial game is a robust demonstration that the accuracy of gpt-35-turbo decreases rapidly throughout when language\". How exactly has this claim been shown?\n\n2) Do the variations within the \"game\" prompt rapidly change the results of the experimentation? Have the authors tried testing the method for sensitivity, as it would highly impact the interpretable component?\n\n3) Can the method be tested against baselines, i.e. CoT in zero/few-shot settings? Such an analysis can establish the advantages of the approach more clearly.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and their engagement with our work. Moreover, the reviewer’s expertise has deepened our research and we greatly appreciate it.  We have further clarified the specified comments below:\n1. Choice of terms and notations: We will carefully review the terminology used throughout the paper, including terms such as \"oracle\" and \"adversarial game,\" and make necessary revisions to ensure clarity and coherence as an interactive game.\n2. Elaboration on the accuracy degradation: We are happy to clarify this contribution, as it relates to the decrease in accuracy compared to BART/BERT based models (TAPEX, TAPAS). As addressed in other reviews, we will also include an additional baseline to highlight this claim. The interactive game in our approach involves iteratively generating code to solve Table QA tasks, and the claim about the accuracy decreasing refers to the observation that without full information or example data, language models still struggle to align questions to commands. However, we do compare our approach against encoder-decoder methods with full information (TAPAS/TAPEX models).\n3. Question 1) As stated above, our intention was to showcase that without “full information” such as TAPEX/TAPAS models, ChatGPT still struggles to solve TableQA tasks when provided only the schema of the data rather than the whole data. We will rephrase this contribution to highlight which baselines we are comparing our methodology with.\n4. Question 2) Variations within the \"game\" prompt and their impact on the experimentation results: We have taken into consideration the sensitivity of our method. We have tested varying prompts on a sample sample set to evaluate the effects of prompt variations on the interpretable component. We will include a detailed discussion of these findings in the revised version of the paper, expanding on the Appendix Section “Instructions for RISQ.” Each prompt rule already has the motivation in parenthesis that explains our intuition from experimentation when constructing the final prompt.\n5. Question 3) Alternative Baselines: We will include a discussion of further approaches to test against in the revised paper, alongside reporting our baselines for “full information” ChatGPT and single chat-turn conversations over the full interactive game. For the single ChatGPT attempt (schema only), the accuracy for WikiSQL is (61.4 Train, 62.2 Val, 61.35 Test) which is 6 below the interactive version. Additionally, WikiTQ is 31.7 train / 32.6 val / 33.6 test (8 points below the interactive) and SQA is 28.9 train / 27.7 val / 33.5 test (10 points below the interactive game).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper contributes valuable insights and novel solutions to the challenges in TableQA tasks. The content is solid, and with some refinement in language, it would be a strong addition to the conference. The method allows for the simultaneous solution of computational, data leakage and tokenization issues prevalent throughout the TableQA tasks. The offered mechanism allows for a more interpretable family of solutions while maintaining the cost-effective nature of the method. The provided ablations and qualitative analysis is helpful for understanding the complexity of the task. A dataset of code-aligned queries/table/code/answer touples is also provided.\n\n", "reasons_to_reject": "1) [Minor] The choice of some terms and notations may need reconsideration as it was hard to follow. To more accurately reflect the approach and align better with the reader's understanding of the subject matter, wording such as \"oracle\" and \"adversarial game\" can be reworded. In particular, as the iterative generation approach does not involve an actual adversary or adversarial generation, it is confusing to the reader to understand the scope/terminology within the paper.\n\n2) lines [95-97] \"Our adversarial game is a robust demonstration that the accuracy of gpt-35-turbo decreases rapidly throughout when language\". This point has not been widely explored throughout the paper and requires elaboration.", "questions_to_authors": "1) \"Our adversarial game is a robust demonstration that the accuracy of gpt-35-turbo decreases rapidly throughout when language\". How exactly has this claim been shown?\n\n2) Do the variations within the \"game\" prompt rapidly change the results of the experimentation? Have the authors tried testing the method for sensitivity, as it would highly impact the interpretable component?\n\n3) Can the method be tested against baselines, i.e. CoT in zero/few-shot settings? Such an analysis can establish the advantages of the approach more clearly.", "paper_topic_and_main_contributions": "The paper tackles the complex challenges faced by Large Language Models (LLMs) in contextually analyzing table question-answering tasks, proposing \"HiddenTables\" - an alternative method, which involves generating code to solve Table QA tasks using only the schema of the table. This approach is computationally faster and more cost-effective while simultaneously mitigating the common problems associated with tokenization (table parsing etc.). Furthermore, it eliminates the risk of private data leakage and includes inherent mechanisms for interpretability. \n\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "k2Mfw2Qmcv", "reviewer": "Reviewer_6csL", "summary": "", "questions": "1. Can you provide a more detailed breakdown of accuracy by table size and question complexity? Where does the Solver struggle most?\n2. How do the accuracy results compare if the full table is exposed to the LLM? Is there a way to quantify the privacy/efficiency vs accuracy tradeoff?\n3. How scalable is the iterative Oracle/Solver conversation protocol? What are the limits on number of queries before performance degrades?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review of our paper and appreciate your valuable feedback. In addition, thank you for bringing the missing reference to our attention and will ensure that the reference to \"Binding Language Models in Symbolic Languages” is included in the revised manuscript. Below, we have considered your comments:\n1. Accuracy Degradation: In our revised version, we will include a more detailed breakdown of accuracy by table size versus question complexity. In addition, we are happy to expand the discussion section regarding on the accuracy degradation for complex questions that relied on multi-hop reasoning (WikiSQL, WikiTables) or prior solutions (SQA) - which can suffer for error propagation if prior steps are wrong. For example, in WikiSQL Easy, the accuracy for Small is 96%, Average is 81%, and Large is 74%. We can provide this in any number of ways to highlight degradation or stagnant performance.\n2. Privacy and Efficiency Tradeoff: We are happy to add a comprehensive discussion of this tradeoff between accuracy and privacy, and relating it to real world examples. We will also aim to provide a clearer understanding of the practical implications of utilizing the HiddenTables framework in regards to PPI, HIPA, criminal information, trade secrets, or FERPA. \n3. Scalability of “Oracle”/“Solver” Conversation Protocol: We are happy to expand on this by analyzing the token usage, latency as conversation length grows, and generation of new tokens.\n4. Question 1) Accuracy Breakdown by Table & Complexity: We can add a cross section between any dimension and permutation in the appendix to highlight the struggles by the solver. Each solution is tied to multiple taxonomies - allowing for any comparison. We will include this in the appendix in the revised version.\n5. Question 2) Full Information Accuracy: We will conduct experiments in our revised version where the full table is exposed to the LLM.\n6. Question 3) Scalability:  We limit the conversation to 7 chat turns, and for each dataset we enumerate the total number of solutions that are correct per turn. We will add a section to address scalability, in addition to breaking down table size by rows/columns and the number of conversation cycles. We have already analyzed token consumption and can expand on this section with latency and timing information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The idea of restricting table access for LLMs to test true language understanding is insightful. The proposed framework is well-motivated.\n- Thorough experiments across diverse datasets (WikiSQL, WikiTableQuestions, SQA) validate the utility of HiddenTables.\n- The framework enables large-scale testing of LLMs on massive datasets not possible with standard approaches, with built-in privacy", "reasons_to_reject": "\n-  The accuracy degradation under HiddenTables, while expected, seems quite significant on complex questions. More analysis on where the Solver struggles would strengthen the findings.\n- It's not clear if the gains in privacy and efficiency outweigh the loss in accuracy for practical applications. Some discussion of this tradeoff would be useful.\n- The scalability of the proposed conversation protocol between Oracle and Solver needs more analysis, especially for high query volumes.\n\n", "questions_to_authors": "1. Can you provide a more detailed breakdown of accuracy by table size and question complexity? Where does the Solver struggle most?\n2. How do the accuracy results compare if the full table is exposed to the LLM? Is there a way to quantify the privacy/efficiency vs accuracy tradeoff?\n3. How scalable is the iterative Oracle/Solver conversation protocol? What are the limits on number of queries before performance degrades?", "paper_topic_and_main_contributions": "This paper proposes a novel adversarial framework called \"HiddenTables\" for evaluating the ability of large language models (LLMs) to solve table question answering (QA) tasks without exposing the full table content to the model. The key ideas are:\n\n   - An \"Oracle\" agent that holds the table data and interacts with the LLM \"Solver\" agent via a restricted interface by providing schema and natural language instructions.\n   - The Solver must generate executable Python code to retrieve answers, based solely on the schema and instructions from the Oracle. This tests the LLM's ability to map language to solutions without seeing real table data.\n   - A secure \"conversation\" protocol between Oracle and Solver to allow iterative refinement of generated code.\n\nThe main contributions are:\n- The HiddenTables framework itself, which provides privacy and efficiency benefits compared to exposing full tables to LLMs.\n- Analysis of limitations of LLMs on table QA using HiddenTables, showing degraded accuracy but more interpretability.\n- A new dataset PyQTax with 116k samples linking questions to Python code and taxonomies.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Binding Language Models in Symbolic Languages: https://arxiv.org/abs/2210.02875"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The proposed methodology is novel, interesting and also timely (because it involves application of instruction-tuned LLMs for table-QA). The reviewers have pointed out to the absence of several meaningful baselines/ablations, which the authors have acknowledged and promised to include in the revised version of the paper. Overall, the suggestions of the reviewers, if incorporated, will make the paper a good contribution in EMNLP. I would strongly encourage the authors to add the suggested baselines and the analysis (e.g., the analysis on detailed breakdown of accuracy by table size and question complexity).
Decision: Accept-Main