CABINET: Content Relevance-based Noise Reduction for Table Question Answering
Authors: Sohan Patnaik
Heril Changwal
Milan Aggarwal
Sumit Bhatia
Yaman Kumar
Balaji Krishnamurthy
Venue: ICLR 2024 spotlight (2024)
Keywords: Table Question Answering
Large Language Models
Noise Reduction
Unsupervised Relevance Scoring
Table Parsing
Relevant Cell Highlighting
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: A content relevance based noise reduction framework for table QA that weighs the table content based on its relevance to question without removing table content explicitly.

Abstract: Table understanding capability of Large Language Models (LLMs) has been extensively studied through the task of question-answering (QA) over tables. Typically, only a small part of the whole table is relevant to derive the answer for a given question. The irrelevant parts act as noise and are distracting information, resulting in sub-optimal performance due to the vulnerability of LLMs to noise. To mitigate this, we propose CABINET (Content RelevAnce-Based NoIse ReductioN for TablE QuesTion-Answering) – a framework to enable LLMs to focus on relevant tabular data by suppressing extraneous information. CABINET comprises an Unsupervised Relevance Scorer (URS), trained differentially with the QA LLM, that weighs the table content based on its relevance to the input question before feeding it to the question answering LLM (QA LLM). To further aid the relevance scorer, CABINET employs a weakly supervised module that generates a parsing statement describing the criteria of rows and columns relevant to the question and highlights the content of corresponding table cells. CABINET significantly outperforms various tabular LLM baselines, as well as GPT3-based in-context learning methods, is more robust to noise, maintains outperformance on tables of varying sizes, and establishes new SoTA performance on WikiTQ, FeTaQA, and WikiSQL datasets. We release our code and datasets here.

Published as a conference paper at ICLR 2024
CABINET: CONTENT RELEVANCE BASED NOISE REDUCTION FOR TABLE QUESTION ANSWERING
*oohnn Patnaik Herill Changwal', "Milan Aggarwal', Sumit Bhatia', Yaman Kumar Singla', & Balaji Krishnamurthy' MDSR Lab, Adobe; 2IT Kharagpur; 'IT Roorkee @oghanpatnaik10ee (milaggar, sumbhati, ykumar, kba kbalaji j
com
@adobe. com
ABSTRACT
Table understanding capability of Large Language Models (LLMs) has been ex- tensively studied through the task of question-answering (QA) over tables. Typ- ically, only small part of the whole table is relevant to derive the answer for a given question. The irrelevant parts act as noise and are distracting information, resulting in sub-optimal performance due the vulnerability of LLMS to noise. To mitigate this, we propose CABINET (Content RelevAnce Based NoIse Re- ductioN for TablE puesTion-Answeiigg - a framework enable LLMs to focus on relevant tabular data by suppressing extrameous information. CABINET com- prises an Unsupervised Relevance Scorer (URS), trained differentially with the QA LLM, that weighs the table content based on its relevance to the input ques- tion before feeding it to the question-answeringg LLM (QA LLM). To further aid the relevance scorer, CABINET employs weakly supervised module that gen- erates a parsing statement describing the criteria of rows and columns relevant to the question and highlights the content of corresponding table cells. CABINET significantly outperforms various tabular LLM baselines, as well as GPT3-based in-context learning methods, is more robust to noise, maintains outperformance on tables of varying sizes, and establishes new SoTA performance on WikiTQ, FeTaQA, and WikiSQL datasets. We release our code and datasets here.
INTRODUCTION
Understanding tabular data through MI. models has been extensively explored through various tasks such as question-answente
(QA) (Chen et al., 2022: Cheng et al., 2022: Nan al., 2022), fact- verification (Chen et al., 2020b; Wang et al., 2021; Aly et al., 2021), table-to-description genera- tion (Chen et al., 2020a; Parikh et al., 2020; Chen al., 2020c; Suadaa al., 2021; Nan al., 2021) and table grounded dialogue (Nakamura et al., 2022). Table QA has been studied with a specific focus as it allows to conveniently query the table in natural language to extract desired information. Large Language Models (LLMs), which have shown remarkable generalization on various Natural Language Processing (NLP) tasks, have also been used to reason over tables achieving impressive performance (Yu et al., 2021; Neeraja et al., 2021; Gu et al., 2022; Chen, 2023).
LLMs Tables contain information organized in rows and columns, and typical transformer- such BERT (Devlin et al., 2019), T5 (Raffel et al., 2020), and GPT (Brown et al., 2020) trained over unstructured natural language text using standard language modeling objectives do not account for the table structure and underlying compositionality of data (Yu et al., 2021). Many works on table understanding therefore, adapt LLMs for tables through joint learning over tabular and text content (Yin al., 2020), pre-training table semantic parsing (Liu et al., 2022; Jiang al., 2022) and synthesizing template- based questions to improve reasoning skills over tables (Gu et al., 2022). Typically, only a small number of cells contain the information required to derive the answer for a question. The irrelevant tabular data acts as distracting information optimal performance since LLMS are susceptible to noise in the input (Kumar et al., 2023; Chen et al., 2023a). Performance degradation is further amplified in large tables due to presence of even more data as illustrated Figure Section 4.3.
noise, resulting in sub-
"quall contribution; + work done during intersship at Adobe Media and Data Science Research Lah

---

Published as a conference paper at ICLR 2024
Sub-Table
Hecand 1:4
Faretion ligeriggeee
What highest amount points game an opponent?
Quastion:
(Decompooer)
DATER
DATER (Reasoner)
Answer 38
Question
egpanett
were
resut
Answer 45
GA LLM
CABINET
202
staniintggeaaa
64
146
week date
opponent
score
result
record
toronto argonauts
13-6
sept
hamiiton tiger cats
38-12
loss
1-5
Table
harritton tgger cats
45-0
lass
1-6
Relevance Score
Weighted Table
13
montreal alouettes
14-12
2-12
Figure 1: Comparison between CABINETard DATER GPT-3 based in in-context learning method). For the given example, DATER extracts a wrong sub-table through hard decomposition (resulting in loss of useful information) that causes QA reasoner to answer incorrectly. CABINET weighs rele- vant table parts higher without removing content explicitly allowing QA LLM to answer correctly.
Significant efforts have been made to mitigate the issue of noise by pruning tabular data, albeit at cost of accuracy (Krichene et al., 2021), and by retrieving content from table for QA (Wang al., 2022; Lei et al., 2023; Kumar et al., 2023). DATER (Ye et al., 2023), one of the sfate-of-the-ar methods for table QA, proposed decomposing a table into simpler sub-tables containing information needed to answer the question by providing in-context examples to GPT-3 based Codex (Chen et al., 2021).
Such a question-condiei condiioone hard decomposition of table sub-optimal as the subsequent QA model cannot correct the eтror made during decomposition if relevant information is not selected (as shown in Figure 1). To mitigate this, we propose CABINET (Content Relev GelevAnce-Bass Based Nolse ReductioN Answering)
for TablE on their relevance to the question without explicitly removing any content. CABINET comprises to
framework for table QA that weighs different table parts based
relevance scorer ($ 3.1), which takes question and table input provide relevance table content. The score is used to weigh corresponding content passed to the QA LLM, allowing it focus more the relevant content. The relevance scorer is unsupervised and trained with QA LLM differentiably due to lack of annotations denoting relevant table information. Although answer generation loss enables learning of relevance scorer, it acts as an indirect training signal.
Hence, to aid relevance scorer, inspired by how humans process tables, CABINET employs parsing statement generator (S 3.2) that describes which rows and columns are relevant to the question. For instance, consider the example in Figure 1, CABINET generates "consider rows with result as 'loss', and note the higher value in the 'score' column".. The parsing statement then used to identify corresponding cells, and their content is given more weight during relevance scoring. CABINET establishes new SoTA on three challenging table QA datasets (WikiTQ, FeTaQA and WikiSQL.) significantly outperforming various strong baselines (S 4.1.. We show that CABINET is more robust to noise in tables and structural biases i.c. row and column ordering ($ 4.2). Further, the performance gains achieved by CABINET are more pronounced for larger tables ($ 4.3), indicating that it successfully mitigates noisy table information irrelevant to a given question.
2 RELATED WORK
Table Specific Architecture: Tables contain information in a structured format, organized in rows and columns. Hence, many works have focused on developing table-specific models utilize the se- mantics of table structure through its description. TabBERT (Yin et al., 2020) pre-trains BERT (De- vlin et al., 2019) on paired table-text samples through masked language modeling (MLM). Deng et al. (2020) modified the bidirectional attention in BERT to incorporate table structure while per- forming MLM. TAPAS (Herzig et al., 2020) utilizes positional embeddings for rows and columns to explicitly capture cell location. Yang et al. (2022) noted that methods using positional embed- dings are vulnerable to column and row permutations. To address this, they introduce TableFormer, table-text encoding architecture that incorporates tabular structure through learnable attention bi- ases. We show that LLMs become less susceptible to such permutations by learning to focus on relevant table parts through CABINET' (S 4.2).
2

---

Published as a conference paper at ICLR 2024
Table QA Specifie Pre-training: Eisenschlos et al. (2020) argued that the MLM objective just fill in the blanks of table cells and descriptions is insufficient to capture relations between cells and associated text needed to perform table QA. They introduced additional pre-training tasks that require explicit question-table reasoning and complex table operations (such as aggregation). Other improvements include handling of numeric tokens (Han al., 2022), temporal relations (Zhao al., 2022), and selectively masking tokens that require table based reasoning (Gu et al., 2022). Methods like TAPEX (Liu al., 2022), OmniTab (Jiang et al., 2022) etc. typically involve joint training over natural Sagguage-SQ.. pairs so that the underlying model learns to map the information implied in the question to the required table operations. However, as discussed in experiments ($ 4.2 and 4.3), these methods suffer significant performance drop when dealing with large and noisy tables owing to their limited capability at identifying information relevant to question.
F Few/Zero-Shot Learning with Large Language Models:
Given the remarkable performance of LLMs on various tasks without any task-specific training. their use for table understanding has also been explored extensively. Chen (2023) have shown that LLMs perform strongly on various table QA tasks using Chain of Thought (CoT) (Wei et al., 2022; Wang et al., 2023) prompting. Since typ- ical LLMs are trained over unstructured text data, models specifically designed to handle structured data, such as StructGPT (Jiang et al., 2023) have also been used for table QA. LEVER (Ni et al., 2023) and BINDER (Cheng al., 2023) utilized code-optimized GPT-Codex (Chen et al., 2021) to generate SQL. statements that can be executed to answer questions over tabular data. DATER (Ye al., 2023) uses Codex to break table into sub-tables conditioned on given question through in- context learning. Such methods have no way to recover relevant table part to generate the correct answer in case it is omitted while generating sub-tables (as discussed in Figure 1).
METHODOLOGY
We summarize the architecture of CABINET in Figure 2. comprises two components: 1) Unsu- pervised Relevance Scorer, an unsupervised module comprising a transformer encoder that takes question and table as input and tokenizes them (steps 1 and relevance score to each table token (step 3 in Fig. 2). The relevance score is then multiplied with the corresponding token embedding at the time of giving it as input to QA LLM encoder (step in Fig. 2). This ensures that noisy content with lower relevance score get suppressed and the QA LLM can focus on relevant tokens. The unsupervised relevance scorer is connected to QA LLM in differentiable manner enabling it to be trained through answer generation loss (step 8 in Fig. 2).
Fig. 2) followed by assigning a
Even though answer generation loss enables learning of unsupervised relevance scorer, it acts as an indirect training signal. To aid relevance scoring, we propose weakly supervised module: 2) Relevant Cell Predictor through Table Parsing that parses table conditioned on question to highlight cells containing relevant information (steps 4 and 5 in Fig. 2). It comprises two sub- steps where we first train a Parsing Statement Generator that describes criteria in natural language about which rows and columns should be used to derive the answer (step 4 in Fig. 2). Table cells corresponding to the criteria described in the parsing statement (step 5 in Fig. 2) are highlighted such that score for content tokens in highlighted cells is boosted by combining it with unsupervised relevance score through a linear combination (step 6 in Fig. 2). We conduct extensive ablations to establish efficacy for different modules ($ 4.4). We now discuss the details of each component.
3.1 UNSUPERVISED RELEVANCE SCORER
The unsupervised relevance scorer is used to assign score table content tokens. Since annotating cells of a table relevant to given question is tedious, the relevance scorer is unsupervised and gets trained along with QA LLM through answer generation loss. Formally, consider pair of represents the question tokens, T = {cipl 1 < i < Nrow, 1 < j S Neoth. where Nrow and Noot indicate number of rows and columns in T respectively, and Cij represents string in cell in the 7th row and j column. To make T suitable to be fed as input transformer-basee LLM, we follow the commonly used lincarising scheme (Liu et al., 2022) where table is flattened as (step in Fig. 2):
table T and a question Q about T. Qtakens = {41, ... } (11)
Tyluttened = [HEDD]: en1 | C12 1
[ROW]]: : 021 |..- C2N... [OWW]2
(1)
[HEDD] and [ROW]k indicate start of column header row and tth data row respectively. We separate special tokens and cell content using pipe symbol 'p" The string in Equation is tokenized
3

---

Published as a conference paper at ICLR 2024
Question Flatiened Table
the oppowet?? (1C40]) week (dee recordl (low! 11 h-1 phov01 new 121-11212
Shared Embedding
What
Question: paints opponent?
Embedding
[ea, 0gs..]
Unsupervised Relevance Scorer
(o [.c]
Embodding
Embeoded Tokens
2
3
Table
upprrse
Hore
GL
Relevant Cell Predictor through Table Parsing
Jumdonn grose
3BG
Parsing Statement
"consider with resutt
e6 TLees, and note the highaz value
he " ....
Answer:
Generator
Highlighter
988-1211 1500 I..."
300
Overview of architecture of CABINET. The table is linearized (step 1) and embedded along with question through embedding layer of the underlying QA LLM (step 2). 'he embedded sequence is passed to the unsupervised relevance scorer that assigns a relevance score to each table token (step3). In parallel, the parsing statement generator describes the criteria for rows and columns relevant to deriving the answer (step 4) that is used to identify corresponding cells and assign a cell- based relevance score (step 5). The unsupervised and cell-based relevance is combined (step 6) and used to weigh the table content (step 7) to the QA LLM which generates the answer (step 8).
Figure 2:
using the tokenizer of underlying QA LLM to obtain table tokens Trokens = {4,, 12, ..., Tookenal Trokens is concatenated to Qtokens to obtain Ttokens = (Qtokens: Teokens) which is given as input (steps 2 and 3 in Fig. 2) Unsupervised Relevance Scorer (URS) comprising a transformer encoder ns. The contextualized representation hp E R of the nh token is estimated as:
6 62
URS URS
= Embeddingu ns (Itokens) ITokeenal
URS
(2)
h1, App
h = (e (e
URS .62
URS
(3)
We aim to predict relevance score for each table token, however, since annotations for relevant table parts are unavailable, token relevance is not explicitly observable and consider it as latent variable. Further, we hypothesize that the representation space of table tokens can be structured better for modeling relevance by clustering their encodings into two categories relevant and non- relevant. Variational Inference (VI) has been commonly used to estimate latent variable probability and group data points on the basis of latent topics (Srivastava & Sutton, 2017). Hence, we estimate relevance np of table token lp (1Qtakensl + < I Stakensl + [Trokens)) 1) as (step 3 in Fig. 2):
Mp = @u(hp): Op do (hp)
(4)
mp
uns (2gm)id(2p); 2p = Up + 5
(5)
8 is sampled from standard normal distribution, 62. and & are FC layers with weights Wr, E Rdx1
and Wa E Rdx1, sigmoid is applied to normalize the relevance score in the range 0 to 1. To enable the relevance scorer assign appropriate scores, we structure the latent space of T TUERS by clustering table tokens into relevant and non-relevant. We use the method of van der Maaten & Hinton (2008) (details in appendix A.1) which performs clustering in a trainable manner using clustering loss Letu We apply Letu. over latent representation ho of tokens which enables us to tune RE for clustering. During experiments, we observed that unit vectors for cluster centroids n parveleeant are not well separated. To mitigate this, we enforce a separation loss Seep that increases the distance between unit vectors representing cluster centroids:
pe Prelevant and
clu Ceep 2 - Praevannt Perrelevant)
olu
(6)
Further, it is desirable that relevance scores for tokens in one cluster (corresponding to irrelevant tokens) are low. To achieve this, we apply a sparsification loss Lsperse where the score logit zp is exponentiated with a negative coefficient to push logit values for relevant and irrelevant clusters to oo and -oo respectively that enables final score (after applying sigmoid) to be close to I and O:
Esparse
1Qrokensl + 1 S < Qtokensl + [Tokeess
Trokenel
At the time of providing question and table as input to transformer encoder T EqqA of the QA LLM, embedding (e,) corresponding to question tokens is used as is while embedding of each table token
4

---

Published as a conference paper at ICLR 2024
is multiplied by its corresponding relevance score (steps 7 and 8 in Fig. 2):
C1, 62, ... ITookenal Embeddingq. (Itokens)
(8)
ep- 1pp epi 11okens (oaienss++2200 S p S Looknns + Ttokenel
(9)
hi IIJrroonnal = (EE
, .gIzononal)
(10)
a1 02," aN = TD (1))
(h), (i
(11)
'O' indicates scalar multiplication with vector operation, TDQA represents the transformer decoder of the QA LLM that generates the answer tokens an sequentially. Tunss, TE and T are trained in an end-to-end manner through cross-entropy loss LoE between the generated and ground-truth answer tokens. Thus, the total loss L becomes:
(12) The answer generation loss acts as indirect training signal for relevance scorer. To aid unsuper- vised scorer, we propose a weakly-superrised module (trained separately from URS and QA LLM) that highlights relevant cells (discussed in next subsection). Table tokens for highlighted cells are assigned cell-based score nieil that is combined with unsupervised relevance score "/ through linear combination (step 6 in Fig. 2). Thus, final relevance score Tp used in Eq. 9 is:
C= L-E + detu * Letu + Xeep Ceep + Aspparse * Leparse
Up Auns Up
uns Aoell * Up
cell
(13)
3.2 RELEVANT CELL PREDICTOR THROUGH TABLE PARSING
As discussed above, we train separate module to highlight table cells relevant to a given question in reakly-superviser manner. Since there is no Table QA dataset that contains annotations for table cells useful to answer a given question, we adopt a two-stage approach where we first train a Parsing Statement Generator (step 4 in Fig. 2) to generate natural language text describing criteria for rows and columns relevant to the given question. Subsequently, we train another model that takes the parsing statement and table input identify the cells matching the criteria (step 5 in Fig. 2).
Parsing Statement Generator (PSG) comprises a pre-trained LLM Flan T5-xl (Chung et al., 2022a) that is fine-tuned to take the question and table as input (Izokens) generate a parsing statement perlparse (step 4 in Fig. 2). The statement describes criteria stating which rows and columns useful to derive the answer. To bootstrap training of PSG, we manually annotate very few (~~300) question-table pairs with parsing statement. For instance, for table and question shown in Figure 2. we annotate parsing statement as 'To derive answer, note the values of higher score in rows with result as loss'. To circumvent annotating samples for each table QA dataset, we choose WikiTabbleQuestions (WikiTQ) dataset (Pasupat & Liang. 2015) to select the samples for annotation since it is the most complex QA dataset containing variety of samples. We sample diverse set of questions for annotation (please refer appendix A.2 for details and examples). The sampled question along with its table are manually annotated with parsing statement which is used to fine-tune PSG. The trained PSG model is then used to generate parsing statement for any question-table pair from datasets studied for experiments. We release the dataset of manually annotated parsing statements.
Cell Highlighting based on Parsing Statement: To identify table cells for the criteria described in the parsing statement parse: we need way map the statement to corresponding cells. To this end, we use ToTTo dataset (Parikh et al., 2020) that contains samples of (table, list of highlighted cell coordinates) pairs. Each pair is accompanied by text description summarising the content of the corresponding list of cells. We fine-tune a cell highlighting model Cell_ Hii.Higgghhhrrrr comprising of Flan T5-xl on ToTTo dataset where the table and summarising text are given as input to generate the content of corresponding highlighted cells. Once Cell Highlighter LLM is trained, we provide the table and texlparse as input to identify and generate content of corresponding cells. For instance, consider example in Figure 2, given the parsing statement shown in figure as input, the cell predictor generates '38- 12 || loss || c hightighted
45 || loss I|...' (step 5). More formally,
|| c e hightighted Cell H Highlighter..dd (T, .texfporse)
(14)
of Nightighted represents the string of th highlighted cell predicted based on parsing statement. M is variable number, 'II' is delimiter to separate cell content. For 1 < < M, if highlighted exactly matches with the content of some cell T. then the tokens lp of matching cell is assigned a cell relevance score (ngell) of 1. nfeil is set to 0 for table tokens belonging to cells in T whose content does not match with any dhighlighted .n splell is then combined with unsupervised relevance score np as in Eq. 13. We now discuss experiments performed to validate the efficacy of our approach.
5

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS AND EVALUATION
Implementation Details: For the encoder (T (QA) and decoder (TDQA) of the QA LLM, we employ the OmniTab (Jiang et al.. 2022) backbone (pre-trained for table understanding) comprising of BART-Large (Lewis al., 2020). The embeddings of unsupervised relevance scorer (URS), Embeddingu RS. and QA model, Embeddingqa. are shared. URS encoder (T ('EuRs) RS) is initialized with the architecture and weights of QA LLM encoder (I EQA). though they do not share weights during QA training. Consequently. the hidden dimension d of T Fu is 1024. We train CABINET and baselines (wherever needed) for 30 epochs on an effective batch size (BS) of 128 using 8 80GB A100 GPUs (BS of 8/GPU with gradient accumulation 2) using a learning rate of 1e with cosine annealing (Loshchilov & Hutter, 2017) through AdamW optimizer (Loshchilov & Hutter, 2019).
Datasets and Evaluation Metries: We evaluate CABINET on three commonly used datasets (i) WiliTaabeeuuestinn (WikiTQ) (Pasupat & Liang, 2015) which is one of the most commonly used and highly complex datasets consisting of about 2100 HTML tables from Wikipedia and about 22, 033 questions that require complex operations such as comparison, aggregation, and arithmetic operations to arrive at the answer; (ii) FeTaQA (Nan et al., 2022), challenging dataset consist- ing of about 10, 000 questions that have a long-form natural language answer (18 words on aver- age) such that requires fetching multiple entities from the table, aggregating and reasoning over these entities, and structuring the inferred information to produce a coherent answer; and (iii) Wik- iSQL (Zhong et al., 2017) that comprises roughly 80, 654 questions over 24, 241 Wikipedia tables. It also provides the equivalent SQL query for each question to obtain the correct answer. While we do not generate SQL (or other implicit logical forms) and only use the natural language questions and answers from this dataset, it serves as useful benchmark to compare CABINET with table understanding methods that generate explicit logical forms to extract relevant answers from table. The ground- trutt answers in both WikiTQ and WikiSQL datasets are short (1-2 words). Hence, we use exact-match accuracy (Acc.) to compare various methods. For FetaQA dataset, ground truth an- swers being long-form (=s 18 words on average), we employ commonly used overlap- based metric Sacre-BLEU (S-BLEU) (Post, 2018). We report performance on test split for all datasets.
4.1 PERFORMANCE OF CABINET ON TABLE QA
We present a detailed comparative analysis of results achieved by CABINET with a variety of baselines. We consider three different categories of methods (i) LLMs specifically pre-trained for table under- standing and fine-tuned for QA, such as TAPEX (Liu et al., 2022), ReasTAP (Zhao et al., 2022) and Om- niTab (Jiang et al., 2022); (ii) fine-tuning LLMs (pre- trained on text only) such as T5-3b (Raffel et al., 2020) and Flan T5-xI (Chung et al., 2022b); and (iti) few or zero shot prompting of LLMs like Struct- GPT (Jiang et al., 2023) and approaches that employ such LLMS for in-context learning like LEVER (Ni et al., 2023), BINDER (Cheng et al., 2023) and DATER (Ye et al., 2023).
Table 1: Comparison of CABINET with
different baselines on WikiTQ. CABINET achieves significantly better accuracy.
Method
Acc.
# params
Fine-tuning Tuble-specific LLMs
TAPAS (Herzig et al., 2020)
48.8
345 M
TaBERT (Yin el al,, 2020)
52.3
345
MATE (Eisenschlos et al., 2021)
51.5
340 M
GraPPa (Yu et al., 2021)
52.7
355 M
DoT (Krichene et ul., 2021)
54.0
299 M
TableForner (Yang et al., 2022)
52.6
345
TAPEX (I.iu et al., 2022)
55.5
405 M
TAP (Zhao et al., 2022)
58.6
406 M
TaCube (Zhou et al., 2022)
60.8
406 M
TaniTabb (Jiang et al., 2022)
62.7
406
Table 1 presents the performance of various methods on the WikiTQ dataset, and we can observe CAB- INET with an accuracy of 69.1% outperforms the best-performing baselines in each of the three cate- Specif-
Fine-tuning text-based LLMs 3b-3b (Xie et al., 2022)) F5an'55xxl (Chung el al., 2022a)
49.3
2
64.4
2.9 B
Few/zero shot Prompting of LLMs
gories and establishes new state-of-the-an ically, CABINET outperforms OmniTaab, DATER, and fine-tuned Flan T5-xl by 6.4%, 3.2% and 4.7%, in absolute terms, respectively. Also, note that sim- ple prompting of ChatGPT does not work well for Table QA. We want to highlight that CABINET performs much better than GPT-3 and Codex-based SoTA in-context learning methods despite contain- ing orders of magnitude fewer parameters.
Codex (Ye et al., 2023)
47.6
175 B
Codex-COT (Chen, 2023)
48.8
B
Binder (Cheng et al., 2023)
64.6
175
LEVER (Ni et al., 2023)
65.8
175 B
DATER (Ye et al., 2023)
65.9
175 B
CharGPT (Jiang et al., 2023)
43.3
175
StrucGGP (Jiang et al., 2023)
48.4
CABINET (Ours)
69.1
560 M

---

Published as a conference paper at ICLR 2024
Table Comparison with different categories of baselines on FeTaQQ. CABINET achieves significantly better Sacre-BIe -LeUBBBEU (S-BLEU).
Table 3: Comparison with different categories of baselines on WikiSQL. CABINET achieves bet- ter Accuracy (Ace.).
Method
S-BLEU
# params
Method
Acc.
# params
Fine-tuning Table-specific LLMs
Fine-tuning Table-specific LLMs
PeaQA (Pal ct al., 2022)
33.5
406 M
TAPAS (Herzig et al., 2020)
86.4
345 M
TAPEX (Liu et al., 2022)
34.7
406
GraPPa (Yu al., 2021)
84.7
355
OmniTab (Jiang et al., 2022)
34,9
406
DoT (Krichene al., 2021)
85.5
299
TAPEX (liu al., 2022)
86.4
406 M
Fine-tuning text-based LLMs
Omni (Jiang al., 2022)
87,9
406 M
T5-small (Nan et al., 2022)
21.6
60 M
UTP (Chen et al., 2023b)
88, 1
345 M
T5-base (Nan et al., 2022)
28.1
222
ReasTAP (Zhao et al., 2022)
88.8
406
TS-large (Nan al., 2022)
30.5
738
T5-3b (Xic et al., 2022)
33.4
2.9 B
Fine-tuning text-based LLMSs
FlanT55xx
36.2
2.9 B
T5-3b (Xie et al., 2022)
85.9
2.9 B
FlanTS-xl
87.8
2.9.B
Fewizero shot Prompting of LLMss Codex-COT (Chen, 2023)
27.0
175
Fewizero shot Prompting LLMs CharGPT (Jiang al., 2023) StructGPT (Jiang et al., 2023)
Codex (Ye et al., 2023)
27.9
175 B
51.6
175 B
DATER (Yc ct al., 2023)
30.9
175 B
54.4
175 B
CABINET (Ours)
40.5
560 M
CABINET (Ours)
89,5
560 M
Similar observations hold for FeTTaQA (Table 2) and WikiSQL (Table 3) datasets where CABINET achieves new SoTA performance. For generating long descriptive answers for questions FeTaQA, CABINET achieves SoTA S-BLEU of 40.5 outperforming OmniTab, fine-tuned Flan T5-xl and DATER by a margin of 5.6, 4.3 and 9.6 absolute percentage points, respectively. We report per- formance only for baselines that have explored the dataset in their work (except for T5 and Flan T5). We use their code/API for evaluation available or else specify performance as reported in their paper. Similarly, for WikiSQL dataset, CABINET pushes the SoTA by 0.7% already high performance of ReasTAP (current SoTA). Since best performance on WikiSQL is already high, the absolute performance gains of 0.7% should be interpreted as a proportion of scope of further im- provement possible, i.e., 0.7 0.7/(100~ 88.8), which is R 6%.
4.2 How ROBUST IS CABINET TO NOISE AND IRRELEVANT INFORMATION?
Despite the remarkable success of transformer- based models on table understanding, they are sen- sitive to noise and perturbations to the tabular data (Pi et al., 2022; Yang et al., 2022; Zhao et al., 2023). We examine the robustness and sensitivity of CABINET towards noise while performing Ta- ble QA. We introduce noise by perturbing tables in test split and report the relative percentage drop in performance. We perform four types of perturbations: 1) Row Addition (RA): insert noise into table by adding rows from another table that contains same number of columns; 2) Row Permu- tation (RP): randomly permute ordering of rows (Pi et al., 2022); 3) Column Permutation (CP): randomly permute column ordering: and (4) Cell Replacement (CR): replace content of certain cells with content from some other table. We perform each perturbation separately to obtain four perturbed test splits for each dataset. Please see appendix A.5 for further details about the procedure.
Figure 3 summarizes the relative drop in performance of CABINET and the dataset-specific best baseline for the three datasets. Note that for all the perturbation categories, CABINET leads sig- nificantly less drop in performance when compared with the corresponding baseline, highlighting the robustness and ability of CABINET identify the relevant portions of the underlying table. Specifically, CABINET is significantly less sensitive to row and column permutations (RP and CP), indicating that relevance scoring of tokens helps the QA L.MM to focus more on relevant informa- tion and reduces the potential ordering biases commonly observed in models pre-trained on tabular data (Yang et al., 2022). For the cell replacement (CR) and row addition (RA) perturbations, where extraneous information is explicitly added to the table, the drop in performance suffered by CAB- INET is significantly less compared to the baselines owing to the superior ability of CABINET to identify relevant information. For instance, in the case of WikiTQ, the relative drop in performance for RA is ss 19% for OmniTab, almost 40% higher than CABINET (+= 11.5%). This consistent trend holds for FeTaQQA and WikisQL. datasets as well.

---

Published as a conference paper at ICLR 2024
239
Performanc Dang P83
+5-
-150
-457
-2.32
Performanno
(60)
-10
10
-225
-55
-0.10
17.40
.1.92
4.0
20-
18.9
CrriTAB CARRET
-0--
31.0
OrmiTAi CARIETT
457
(a) WikiTQ
(b) FeTaQQ
(c) WikiSQL
Figure 3: Relative performance drop (%) with perturbations (RA Row Addition, RP Row Per- mutation, CP Column Permutation, CR Cell Replacement). We compare CABINET (green) with
OmniTab (red) WikiTQ and FeTaQA ; and against ReasTAP (red) WikiSQL. CABINET is more robust to addition of noise to table and shuffling of row and column ordering.
50
Own Iah CABNET
CABIET
47
Accuracy
80
A00
20-10
200
200
0400
5007
(a) WikiTQ
(b) Fa
(c) WikiSQL
Figure 4: Variation in performance with table size (# cells). We compare CABINET (green) with Tab (red) on WikiTQ (left) and FeTaQA (middle), and against ReasTAP (red) for WikiSQL (right). It can be seen that CABINET performs much better than the baselines on larger tables.
4.3 IMPACT OF TABLE SIZE ON PERFORMANCE
We now study how CABINET performs with tables of different sizes. Tables typically comprise large amount of data, so the entire information is usually not required to answer a given question and acts as distracting information (Neeraja et al., 2021). This noise irrelevant data poses severe challenge for table understanding models and leads to poor generalization for larger tables (Kumar al., 2023; Chen, 2023). We consider the number of cells in the table a proxy for its size and bin all the questions in the three datasets into six categories based on the number of cells (Figure 4) and compare the performance of CABINET with dataset-specific best-performing baseline. We note that for all the datasets, while model performance drops with increasing table size, CABINET consis- tently and significantly outperforms the baseline methods across all table size categories. Moreover, the differences become starker for larger tables. For instance, for the largest tables in FeTaQQA CAB- INET achieves double the S-BLEU scores compared OmniTab (36 vs. 18). Similarly, for the other two datasets, CABINET achieves significantly high performance for the largest tables (> 500 cells) compared to the baselines accuracy of 38 vs. OmniTaabss 28 for WikiTQ and 79 vs. ReasTAP's 70 for WikiSQL. These empirical observations provide further evidence for CABINET's ability to identify relevant content, making the QA LLM relatively robust table size.
4.4 DISCUSSION ON THE IMPACT OF DIFFERENT DESIGN CHOICES FOR CABINET
We study the impact of clustering the table tokens using their latent representations (discussed in Section 3.1). To do so, we toggle the clustering loss (Cot). cluster centroids separation loss (Csep), and score sparsification loss (Esparse) by setting their weight (Aetus Asep. Asparse) to 0 1. For this study, we only use unsupervised relevance scorer by turning off weakly supervised cell predictor to eliminate other influencing factors. Results are summarized in Table 4 where we can observe that applying all three losses yields the best performance (row 6). Specifically, for WikiSQL, clustering improves performance when score sparsification loss is applied (row 4 vs. row 3) which is due to the fact that sparsification enables categorizing scores into low and high. For WikiTQ and FeTaQA, adding the cluster centroids separation loss further increases the efficacy of clustering and sparsification yielding the best results.
Effect of Clustering Table Tokens:
Combining Unsupervised Relevance Scorer with Cell Predictor: We vary the relative importance given to relevance score predicted by unsupervised relevance scorer and weakly-superviiee cell pre- dictor by varying Auns and Aeell in Eq. 13. Table 5 shows that combining the two modules yields much better accuracy for WikiTQ and TaQA compared to just using unsupervised relevance scorer (row row 2). This highlights that the seakky--uperviiee cell predictor complements unsuper-
8

---

Published as a conference paper at ICLR 2024
Table 4: Effect of applying clustering (Cata,, centroid separation (Ceep) and relevance score sparsification loss (Ceparse) Clustering ta- ble tokens by enforeing sparsity in relevance scores and distance between cluster centroids improves performance.
Table 5: Impact of combining unsupervised relevance score (weight \uns) and weakly- supervised cell-based relevance score (weight Acst). Fusing the relevance from both compo- nents gives optimal performance.
Aceil
WikiTQ
FeTaQA
WikiSQL
Lets
Leep
Laparse
WikIQQ
FaTaQA
WikiSQI.
1
0
65.6
35.8
89.2
60.8
35.1
86.2
0.7
0.3
69.1
40.5
89.2
60.9
35.1
86.3
0.5
0.5
68.6
40.5
88.9
X
X
62.7
35.0
88.9
61.0
35.0
89,5
0.3
0.7
67.0
38.9
88.8
X
61.0
35.1
89.1
0
37.6
24.2
34.1
65.6
35.8
89.3
Queetion:
What the total number of awards jnominatoons Minsi beo and men??
Referincc
200
10
pile plu
pil
Prodicter
Honveleaar tolles Entvent
outh tasle make thay 3D spaco through
greate than setveree with others
wan polden Icon award best sctor- comedry series
shown
Upen veriiaation Biss foued center slove Also soen greaped
regiore tndiceting bonett
spane.
Figure 5: Visualisation depicting that Unsupervised Relevance Scorer (URS) assigns higher score to table parts relevant the question (rows where "two and half men" either won or got nominated for an award). Further, the weakly-supervised parsing statement based relevant cell predictor identifies the cells for row missed by URS (year 2006, golden icon award best actor comedy series)
vised scorer by identifying further relevant table content (Figure 5 depicts qualitative visualisation for the same). For WikiSQL, same performance is observed with and without the cell predictor. Fur- ther it observed that using only the cell predictor (last row) achieves significantly low performance due to the fact that the number of cells highlighted by the cell predictor is much lesser resulting in assigning a score of zero most table content in cases where misses identify important cells.
We show CABINET can be used with TAPEX backbone (instead of OmniTab) to improve it's perfor- mance showing generality of our framework (Appendix A.7). We show that giving parsing statement as input to QA I.IM,, replacing URS with BERT based similarity metric for relevance scoring, and using question directly instead of parsing statement to generate highlighted cells gives sub-optimal performance compared to CABINET, justifying our design choice (Appendix A.8), Appendix A. shows case study depicting how clustering losses interact to yield improvements. Appendix A.11 shows that CABINET can be used to improve other NLP tasks like reading comprehension.
CONCLUSIONS
We studied the problem of question-answering over tables and focused on identifying the relevant portions of the table to derive the answer. Generally, only a small subset of the tabular data is required to answer the question, and owing to the vulnerability of LLMs to noise, the extrancous information leads to sub-optimal performance. This problem is further exacerbated in the case of large tables. Our proposed framework, CABINET addresses this issue by weighing the table content based on its relevance to the question, identifying the relevant rows and columns, and high- lighting the content of the relevant cells. CABINET establishes new SoTA three commonly used challenging benchmarks, outperforming context learning with much larger GPT-3 scale models. We show empirically that CABINET' is more robust to noise and generalizes well for larger tables, indicating its efficacy in mitigating noise and overcoming table structural biases typically learned during training.
selei-s models, as well as methods that employ in-

---

Published as a conference paper at ICLR 2024
6 ETHICS AND REPRODUCIBILITY STATEMENT
We use publicly available datasets and LLMs (which are commonly used) to conduct the study in work. The only data that annotate is ~ 300 samples of table-question pairs with parsing statement describing rows and columns relevant to question. The parsing statement were written keeping in mind the safety and ethics guidelines without any potential concerns. To encourage reproducibility, we release our code and datasets (including manually written parsing statements) at this link. We describe the details of the datasets in & 4 (under 'Datasets and Evaluation Metries') and the LLMs used in $ 4 (under 'Implementation Details') and $ 3.2. Further, we provide the implementation details of our method in 4 (under 'Implementation Details') and discuss baselines used for comparison in $ 4.1. Finally, we elaborate further details of our method in Appendix Trainable clustering over latent representation of table tokens (A.1), Details of parsing statement annotation procedure (A.2) and Further details on table perturbation procedure (A.5).
REFERENCES
Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Chris- Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. Feverous: Fact ex- traction and verification over unstructured and structured information. In J. Van- schoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Sys- lems Truck on Dalasets and Benchmarks, volume 1. Curran, 2021. URL https: nneurrs neurips wttp:/eeeppppppppeeeeeeeeee /tape/////////////////
/ 2022//LLe/BBB1005557786833322222
roundl. pdd..
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., sateer_ffisspppppeeeeeeeeee
2020. URL
neurips cwpps.cc//aaee 2at--22266aaaa.r.eeeeea---
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmennted generation, 2023a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder. Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Mathew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- ter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URL https:ttt/////cctmm http//aaivvv./. orgg abs 2217 03374.
Nuo Chen, Linjun Shou, Ming Gong, Jian Pei, Chenyu You, Jianhui Chang, Daxin Jiang, and Jia Li. Bridge the gap between language models and tabular understanding, 2023b.
Wenhu Chen. Large language models are few(1) table reasoners. In Findings the Association for Computational Linguistics: EACL 2023, pp. 1120-1130, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. URL https: https///ccaatttt awllanthcloooyyy org/2023. mendings-eacl.. 88.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. Logical natural language generation from open-domain tables. In Proceedings of the 58th Annual Meeting
10

---

Published as a conference paper at ICLR 2024
of the Association for Computational Linguisties, pp. 7929-7944 Online, July 2020a. Asso- ciation for Computational Linguistics. doi: 10.18653/v 10.18653/w1/2020.ac aww.a......mii...77 URL https: / /aclanthology. org/2020 2200 acl-main, 708.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations, 2020b. URL https: / /openevieeee htt.m/orummmiom d=rriideerreeeknnn
Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, and William Yang Wang. Logic2Text: High-fidelity natural language generation from logical forms. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2096-2111, Online, November 2020c. Association for Computational Linguistics. doi: 10. 10.18653/1/2020. findings-emnlp.190 URL https: 190.
wwwptthooooo.oooo///ooo wwg//ogg/// f findings emnlp.
Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. ConvFinQA: Exploring the chain numerical reasoning in conversational finance question Conference on Empirical Methods in Natural Lan- guage Processing, pp. 6279-6292, Abu Dhabi, United Arab Emirates, December 2022. Asso- ciation for Computational Linguisties. doi: 10.18653/v 10//33/v12222.com www.e-eelllmm...... // //aclanthology. hwwpollooyorgg///// 2022 wwwnlmmmaaeeeeeeeeeeeeeeeeeeemam meain.421.
answering. In Proceedings of the 2022
URL https:
Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. HiTab: hierarchical table dataset for question answering and nat- ural language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), pp. 1094-1110, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10.18653/1120022.cl--boo attts:://aclatteee
URL
org 202p:/222.... att/20222...11onnn acl-1ong 78.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Binding lan- guage models in symbolic languages. In The Eleventh International Conference on Learning Representations, 2023. URL https: : /tt///pprreeeeeeeeeee net/ forummrmmioo (tw.mmmmi=1111422220
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction -inetuuned language mod- els. arXiv preprint arXiv:2210. 1416, 2022a.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction instructionfinetuueed language mod- els. arXiv preprini arXiv:22/0 20.0.1111000022211 1416, 2022b.
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. Turl: table understanding through representation learning. Proceedings of the VLDB Endowment, 14(3):307-311 2020.
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-110 1423. URL https: / //aclanttiii //aclanthology wtg//ils-ee 1423.
Julian Eisenschlos, Syrine Krichene, and Thomas Müller. Understanding tables with interme- diate pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 281 281-296. Online. November 2020. Association for Computational Linguistics. http//20000..nneeeeeeee
doi: 10. findi ngs- eanggemnllppppmmmmmm 1..
org/202c. URL https://aclanthhllee
Julian Eisenschlos, Maharshi Gor, Thomas Müller, and William Cohen. MATE: Multi-view attention for table transformer efficiency. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7606-7619 Online and Punta Cana, Dominican Republic,
11

---

Published as a conference paper at ICLR 2024
November Associatin for Computational Linguistics. doi: 10.18663//1/0222.cmmm main. 600. URL https: acl http:///laaaaaotoooa.oooooo...o. org. 2222 emnlp-main. 600.
Siddhant Garg, Thuy Vu, and Alessandro Moschitti. Tanda: Transfer and adapt pre-trained trans- former models for answer sentence selection. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7780-7788, 2020.
Zihui Gu, Ju Fan, Nan Tang, Preslav Nakov, Xiaoman Zhao, and Xiaoyong Du PASTA: Table- operations aware fact verification via sentence-table cloze pre-training. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4971-4988 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10. 11188///1122222mmmmmmm emn p-ma i .. 331.
URL https: / / antho wrg/202/22222
Hongwei Han, Jialiang Xu, Mengyu Zhou, Yijia Shao, Shi Han, and Dongmei Zhang. Luna: Lan- guage understanding with number augmentations on transformers via number plugins and pre- training, arXiv preprint arXhv:2212.02691/ 2022.
Pengeheng He, Jianfeng Gao, and Weizhu Chen. DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations, 2023. URL httpp:://oppt /opprr/ieppeeeiiee. net peet/foz /t//m//rmmmii/////
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisen- schlos. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computationall Linguistics, pp. 4320-4333, Online, July 2020. Association for Computational Linguistics. doi: 10. (0.18653/v1/2222.accc maiim...39 URL https::
/ //a/cllaatttttt /taaa/aaaaaattooooeeoeaaaoeoooae wtgg///00000 wal.000acl-mmmm in.inn... 398.
Roshni Iyer. Thuy Vu, Alessandro Moschitti, and Yizhou Sun. Question-ansee answer sentence graph for joint modeling answer selection. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 77th Conference of the European Chapter the the Association for Computational Linguistics, pp. 968-979, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10. 18653/v1/2023 www.-aain.ieee6 URL https: /ttp//acaaaa//aaa www.ccannnoooggyoooooooooo org/ eeclllllllee mmain. 68.
Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. Structgpt: A general framework for large language model to reason over structured data, 2023.
Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neubig, and Weizhu Chen. OmniTab: Pretrain- ing with natural and synthetic data for few-shot table-based question answering. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pp. 932-942 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 1118663/112022.naaaa main.mm.......... URL https: http://claantttttt http//aaaannttoooogoooo org 2022. nww.nnaclaaaam mac.://.oor.222...mmmm.a......a
68.
Syrine Krichene, Thomas Müller, and Julian Eisenschlos. DoT:
An efficient double transformer for NLP tasks with tables. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pp. 3273-3283, Online, August 2021. Association for Computational Linguis- http://ww........ngggaa........ findings-acl.288) URL https: http://aaaaaattcceeeeee
tics. doi: 10.18653/v/2021. findings- acl. 289.
Vishwajeet Kumar, Yash Gupta, Saneem Chemmengath, Jaydeep Sen, Soumen Chakrabarti, Samarth Bharadwaj, and Feifei Pan. Multi-row, multi-span distant supervision for Table+Text question answering. In Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguisties (Volume 1: Long Papers), pp. Association for Computational Linguistics. doi: 10. / //accantholl ogy. 22032 a wew.alcllll.... aacl-long 449.
Toronto, Canada, July 2023.
URL https:
Fangyu Lei, Xiang LI, Yifan Wei, Shizhu He, Yiming Huang, Jun Zhao, and Kang Liu. S3HQA: A stree-stage approach for multi-hop text-table hybrid question answering. In Proceedings of the 61 st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers),
12

---

Published as a conference paper at ICLR 2024
pp. 1731-1740, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10. 22t/2222222ssooos URL http:p::///.aaatooooo..roor///// sho-s 147.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequennc-tO-sequeeece pre- training for natural language generation. translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, On- line, July 2020. Association for Computational Linguisties. doi: 10. 18653//1/2020.acll1l www.2000..cl-ain.... URL https:::p..........t. / /tcl///ccccaaaa wwwcclnnhoollgoo ogy. org/ 2020 acl 703.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. TAPEX: Table pre-training via learning a neural SQL executor. In International Confer- Learning Representations, 2022. URL https: /ttp:///ppeeeeiieeeeee / net net/forum?id 050443AASC
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In In-
ternational Conference on Learning Representations, 2017. URL https://openreviee net Pust/Eoza Eorum? i d= -idiidgasse-eee Skq89Scxx:
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https: https://openreviett /ot///ppprrvviieeeeeeeeeeee net/ forum? id=
CqY7.
Kai Nakamura, Sharon Levy, Yi-Lin Tuan, Wenhu Chen, and William Yang Wang. HybriDialogue: An information-seeking dialogue dataset grounded on tabular and textual data. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 481-492, Dublin, Ireland, May URL
2022. Association for Computational Linguistics. doi: 10. (6655////222.ffnniinne.. https: attps://aclanthoot /alaaaaaannttoooooyoooooooo org/2022 findings-acl... 41.
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Ra- jani. DART: Open- Open-domain structured data record to text generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 432-447, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 1/2021 wwe.:02..aaalllmee .naacI-main.37 URL https: http://aaaaattaaaaoeaeeae org/
2021. www.:aaccllaa. manacl-male mwwn.an.... 37.
Linyong Nan, Chiachun Hsich, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojeiech Kryšcifíski, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Is- abel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. FeTaQA: Free-form table question answering. Transactions of the Asso- ciation for Compusational Linguistics, 10:35-49, 2022. doi: 10.1162/tacl a 00446. URL https: /t////alaattttt aaclannhhoooyy wwwp//gg........ooo tacl-
J. Neeraja, Vivek Gupta, and Vivek Srikumar. Incorporating external knowledge to enhance tabular reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, pp. 2799-2809, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 http:////221...ammaaaaaa .naacl-main.224 URL https: / / acc 1 anthol ogy. .rgg orgg/0211111 2021 www.aaclll-a.... -oaellmminn 2.244
Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-cor generation with execution. Proceedings of the 40th Internutional Conference on Machine Learning (ICML.23), 2023.
Vaishali Pal, Evangelos Kanoulas, and Maarten Rijke. Parameter-efficient abstractive question an- swering over tables or text. In Proceedings of the Second DialDoc Workshop on Document- grounded Dialogue and Conversational Question Answering, pp. 41-53, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 1/.18653/ 1.5. URL https: / //ccl anthhlogohee hengyoorg/22222 di a.doc-1.5.
13

---

Published as a conference paper at ICLR 2024
Ankur Parikh, Xuezhi Wang. Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra. Diyi Yang, and Dipanjan Das. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 1173- 1186, Online, November 2020. Association for Computational Linguistics. doi: 10. 18 2020.emnlp-main.95 URL pett:p::/accaanthttt oow.ooyoooooooo wrgg/220 emn p--malpppa- ma 89.
Panupong Pasupat and Percy Liang- Compositional semantic parsing on semi-structured tables. CoRR, abs/1 558.00005. 2015. URL htt.tttpp//aiiii. www./oggg////...... 158.1110800 00305.
Xinyu Pi, Bing Wang. Yan Gao, Jiaqi Guo, Zhoujun Li, and Jian- Jian-Guang Lou. Towards robustness of text-to-SLL models against natural and realistic adversarial table perturbation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I; Long Papers), pp- 2007-2022, Dublin, Ireland, May 2022. Association for Computational Lin- guistics. doi: 1/228666311120222el 142. URL https: https///aaaaaattahaaccee org/2022. acl-long. 142.
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Con- ference on Machine Translation: Research Papers, pp. 186-191, Brussels, Belgium, Octo- URL
ber 2018. Association for Computational Linguistics. doi: 10.18653/11/W18-60 https: http://ccaantttoogoo/oo wrg/orl//...............oo 6319.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
//jmlr: wttpa/.ror/ppaprs//2111 000122000000-20 074. html.
Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. In International Conference Learning Representations, 2017. URL htt.ttts:///ppeeeeeeeee met/foounm d==BbbtK99g,
Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, and Hiroya Takamura. Towards table-to-text generation with numerical reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1451-1466 Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2000 1202:/11211222222 1a222..looo URL https::///// /aaaaaaaanntooggo org/ 2021 aal-loon..11.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 9(86):2579 2008. URL http: jmlr. wrg/papers/vv/
vandermaaten rmaaten08a htm.l.
Nancy X. R. Wang, Diwakar Mahajan, Marina Danilevsky, and Sara Rosenthal. SemEval-2022 task 9: Fact verification and evidence finding for tabular data in scientific documents (SEM-TAB- FACTS). In Proceedings of the 15th International Workshop Semantic Evaluation (SemEval- 2021), pp. 317-326, Online, August 2021. Association for Computational Linguistics. doi: 10. 18653/v1/2021 semp:// 1,399. URL https: / //cl//acceeeeee anthology. .rgpy....rr//// org/2021 semeval-1. 39.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning language models. In The Eleventh International Conference on Learning Representations, 2023. URL /etps///eeeeeeieewve www.eviwwwweeeeeeeeeeeme.ee..eee www.:nneee./tneeeeeeeeeeee Doruumid==1PLNNNNT
Yingyao Wang, Junwei Bao, Chaoqun Duan, Youzheng Wu, Xiaodong He, and Tiejun Zhao. MuGER2: Multi-granularity evidence retrieval and reasoning for hybrid question answering. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6687-6697, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/22 1/t02//1 wtt.:3/w/122...nnnnnssseee findings-emnlp.988. URI. https: accl antlalnhhooloo ooy org/ Findings-ema 1p..mmmmpp....mmmmmmmmm.... 498.
Jason Wei. Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language
14

---

Published as a conference paper at ICLR 2024
models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022. URthttps: htttt:::///penrview...to. net// Forum?id=_vjoil VVOMMMSBBB..B.. J.
Tianbuo Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong. Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong. Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. UnifiedSKG: Unifying and multi-tasking structured knowledge grounding with text-lo-text language models. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 602- 631, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10. 18653866/v 1/2022. wwt.:22.mmlmpmmii.n.. emnlp-main. 3...pmaaan...9..................
URL https: /tc//aaaaanttt wwlalaannnntooooooooooooo org/ 2022.
Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. Table- Former: Robust transformer modeling for table-text encoding. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguisties (Volume I: Long Pupers), pp. 528-537, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 166.836111//22.. acl-long.40 l0. URL htt ps: / /tt///aaaaaatoooooooooooe wwo/or////// acl-long. 40.
Yunhu Ye, Binyuan Hui, Min Yang. Binhua Li, Fei Huang, and Yongbin Li. Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning. In Research and Development in Information Retrieval, SIGIR '23, pp. 174-184, New York, NY, USA, 2023, Association for Computing Machinery. ISBN 978 9781450394086 doi: 10. 00.11457353661883591703 URL https: / //doi. org/i.org/000 1ttp://0.1115333336111
Proceedings of the 46th International ACM SIGIR Conference
3591708.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 8413-8426, Online, July 2020. Asso- URL https:
ciation for Computational Linguistics. doi: // /acllanthlloyy..ooom crg.2020 acl-main. 745.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, bailin wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, richard socher, and Caiming Xiong. Gra(pp)a: Grammar-augmentedd pre-training for table semantic parsing. In International Conference on Learning Representations, 2021. URL
htttp:://openreviee
net www.mmmddkyyaaeeaaaa..
Zeyu Zhang, Thuy Vu, and Alessandro Moschitti. Joint models for answer verification in question answering systems. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp- 3252-3262, Online, August 2021. Association for Computational Linguistics. doi: 18.15v33/v1 2021 lonn...-ooo URL https: / /aaccaann woellanhhooogg. howgyy.ooo...ooooo.oo.ooo.oooo.. org/ /01//22211..0 www.0...1111.... moll-loon.2.. 252.
Yilun Zhao, Linyong Nan, Zhenting Qi, Rui Zhang, and Dragomir Radev. ReasTAP: Injecting table reasoning skills during pre-training via synthetic reasoning examples. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 9006-9018. Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10. 10.18853/v1/2022..oo www2222.eemlpmmaii. main.615. URL https: / awwaatthaltooyyoo org/2022. emn 1- n.n 615.
Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir Radev. Robut: A systematic study of table qa robustness against human-annotated adversarial perturbations. arXiv preprint arXiv:2306. 14321, 2023.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00003, 2017.
Fan Zhou, Mengkang Hu, Haoyu Dong, Zhoujun Cheng, Fan Cheng, Shi Han, and Dongmei Zhang. TaCube: Pre-computing data cubes for answering numerical-reasoning questions
15

---

Published as a conference paper at ICLR 2024
tabular data. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 2278-228222 12912299, Abu Dhabi, United Arab Emirates, December 2022, Associ- ation for Computational Linguistics. doi: 10. http:22eeel2.p2mmeeeee / llcclanhhlooyy org/ 2022. emn m mamii...................... i..iinn... 145.
URL https:
A APPENDIX
A.1 CLUSTERING LATENT VECTORS
As discussed in Section 3.1, the table tokens are clustered in a trainable manner (van der Maaten & Hinton, 2008) using their latent representations encoded through Unsupervised Relevance Scorer (URS). We discuss the details of the trainable clustering algorithm.
Formally, the probability of latent vector hp corresponding to the pd token belonging to jth cluster is given by Equation 15
(1 + 11 1/ tt Σ, (1+ - 1 11/ =
(15)
Here, hp the contextualised latent vector of the pa token obtained using Tu a is the degrees of freedom of the Student's distribution, 11g/ is the centroid of the the jth cluster and j E {0, 13. Here, p = Prefevant and pu Mirrelevant Moreover, the cluster centroids are learnable.
chu
The clustering process iteratively refined by enforcing KL divergence minimization between the probability distribution for each token and pseudo distribution generated using Opj. Mathemati- cally, the clustering loss is
1 =EKL Letu = B
1 -EΣ Zoglog B
Zpp
(16)
qpj
where 6 E 11, B}. B is the batch size, and Z is the target distribution. A naive approach to model would be setting each Ep to a delta distribution (to the nearest centroid) for representations above confidence threshold and ignoring the rest. However, because 9p soft assignments, it is more natural and flexible to use softer probabilistic targets. So, we model Epj with Equation 17
Lpg/Sos
Zpj Ey / where fps Σ Aps is the soft cluster frequency
(17)
A.2 DETAILS OF ANNOTATING TABLE PARSING STATEMENT
As discussed in Section 3.2, train parsing statement generator that generates criteria de- scribing which rows and columns contain information relevant to the table. The generated parsing statement is used to identify corresponding cells which then combined with the relevant ta- ble part detected by the unsupervised relevance scorer through relevance scoring. To bootstrap the training of the parsing statement generator, we manually annotate ~300 question-table pairs from the WikiTQ dataset with parsing statement. To select samples to be annotated, we sample questions from WikiTQ as it consists of diverse and complex set of reasoning questions over tables, thereby sampling questions from this dataset allows us to select a set that is representative of questions in other datasets for Table QA.
To select a diverse set of questions from WikiTQ for annotation, we group questions into 5 clus- ters by clustering their representations such that we randomly sample an equal number of questions from each cluster. Specifically, tions. Since the questions pertain to tables, we embed questions LLM possessing table understanding DeBERTaV2 (He et al., 2023) initialized with weights tuned through table understanding task proposed by PASTA (Gu et al., 2022).
an encoder-only model DeBERTa-VV V2 to encode the ques-
train split of WikiTQ through an
Once the questions clustered, sample small subset (2.5%) from each cluster to manually annotate with the parsing statement. Some examples of questions, the parsing statement and the corresponding answer from each cluster can be seen in Table 6.
16

---

Published as a conference paper at ICLR 2024
Table Examples of parsing statement annotated manually for questions sampled from each cluster.
Cluster Question
Answer
Parsing Statement
how many episades had nightly rank of 11?
to find number of episodes with nightly rank 11, we need to look column named "nightly rank" and count number of times the value 11 occurs.
how many games during this season were aired on cbs?
to find number games aired on ebs, we need to look the column tv and retrieve the rows having value chs. from table, there are oc- curenees of cbs in column.
2 which scason more 1996/97 successful, 1995/96 1996/977
to compare between success of the seasons 1995/96 1996/97, we need look at the final place both seasons. from table, place cor- responding to season 1995/96 19th, which is bad compared to the place 1st corresponding to season 1996/1997.
did john howard serve as more prime minister for more or less time than julia gillard?
for answering this question, we need to look at total time in office for both john howard and julia gillard. for prime minister john howard total time in office 4,284 days which more than 1,099 days i.e., total time office for julia gillard.
3 which party won the top place in the election?
Australian Labor Party
to find party with top place in the election, we need to compare the seats of each party. from table, australian labor party has the maximum number of seats.
which role is the most com- mon from all the titles?
Salesman
most common title refers to title which occurs the most number of times. from table, in the column role, the value salesman occurs the most number times.
what publication is listed before play magazine?
Nintendo Power
from table, in the column publication, the row before play magazine has the value nintendo power
what college has top en- rollment?
Cornell Uni- versity
to find top enrollment, we to find college with maximum number of enrollment. from table, the maximum value of enrollment column is 20,400 corresponding to the institu- tion cornell university.
what the difference be- tween the caps of henry carlsson and borge leander?
to find difference of caps between henry carl- son and borge leander, we need to first obtain the caps values. from table, caps of name henry carlson i.c., henry "garvis" carlson is and caps of borje leander is so difference 1.
what was the difference in time between the 8th place finisher and the first place finisher?
+17.32
to find difference between 8th place finisher and first place finisher, we need to look at the dif- ference column corresponding rank in the table. from table, value +17.32.
17

---

Published as a conference paper at ICLR 2024
A.3 ADDITIONAL RELATED WORK
Graphs have been used to capture table structure and similarity between table-question samples across the dataset (lyer et al., 2023). TANDA (Gar et al., 2020) performs the task of answer sentence selection (AS2) for a given question by fine-tuning the transformer- baaseeddr- model to select the right candidate from answer candidates. They first fine-tune an LLM on the AS2 task followed by adapting it to specific domain. Zhang et al. (2021) built over TANDA by considering remaining answer candidates as evidence while deciding the appropriateness of particular answer candidate. This was further extended by Iyer et al. (2023) who modeled the relation between other similar question samples and corresponding answers in the dataset with the given question and answer candidates using a graph. In our current work, we mainly focussed on identifying relevant parts of the table useful to derive the answer to the question, however, similar method can also be explored as future work that utilizes a graph to capture the table structure (positioning of different cells in the table) for a given sample and also model the similarity between multiple question-table pairs across the dataset. Another direction focuses on applying semantic parsing over the input text (question) and table to generate logical form (such SQL) which when executed fetches relevant information (Yu et al., 2021).
A.4 PERFORMANCE ON NUMERIC Vs NON-NUMERIC RETRIEVAL AGGREGATION-BAL BASED ANSWERS
We analyse the performance of CABINET generating answers of distinct types by categorizing them into four categories: numeric, non-numeric, retrieval, and non-retrieval (aggregation). This meticulous categorization allows gain nuanced understanding of how CABINET performs against baselines for table-question pair that require different types of answers to be generated.
80
OmniTab CABINET
ReasTAP CABINET
75.1
95
Accuracy
70
74.7
70.2
Accuracy
90.4
713
90
91.8
89.7
60-
59.7
59.8
85.7
84.4
84.3
51.1
50
48.2
81.1
80
Retrieval Aggregation
Numeric Non-numeric
Retneval Aggregation
Numeric Non-numeni
(a) WikiTQ
(b) WikiSQL
Figure 6: Performance comparison between CABINET with OmniTab on WikiTQ and with ReasTAP on WikiSQL when test samples are categorised based on the answer type retrieval, non- retrieval (aggregation), numeric and nom-nume We see CABINET provides significantly better performance on all answer type categories. It's noteworthy to mention that this analysis excludes FeTaQA, as comprises of free-form long answers, hence categorization not possible.
Figure 6 summarises the results. For both WikiTQ and WikiSQL, our method consistently demon- strates better performance for all four categories. We see an improvement of around 10% for WikiTQ and around 3 5% for WikiSQL for numeric and non-retrieval type answers, thereby high- lighting the improved aggregation capabilities using CABINET.
A.5 DETAILS OF TABLE PERTURBATION
In Section 4.2, we discussed the impact of perturbing the tables on performance. We discuss the detailed steps that we followed to inject the 4 types of perturbation separately in the tables. For Row Addition, we create sets of table with same number of columns. Further, based on the number of cells in table, we insert n rows from another table with the same number of columns at a random position. The exact scheme followed is, 1) n =
m < 150, 2) n = 2 if m > 5 if m 300 & < 450 and 4) = if m 450. For permutation and column permnutation, we randomly permute the rows and columns respectively without any specific conditioning over the number of and columns present in the table. For
150 & < 300, 3)
18

---

Published as a conference paper at ICLR 2024
cell replacement, we divide the dataset again into the 4 buckets same that of row addition and randomly replace 0.02%, 0.05%,,..11 and 0.12% of the total number of cells in the table.
A.6 ADDITIONAL IMPLEMENTATION DETAILS
As stated in section 4 (under implementation details) of the main paper, we employ the TmniTab backbone built over BART-Large architecture for the encoder (T (QA) and decoder (T (TD of the QA LLM. Further, URS encoder ES) is initialized with the architecture and weights of QA LLM encoder (T EQA). though, they do not share weights during training. However, the embedding layers Embeddingonng ns and Embeddingga share weights. The hidden dimension d of T FURS 1024, similar to that of both T' A and TDQA. We train CABINET and other baselines for 30 epochs on an effective batch size (BS) of 128 using 8 80 GB A100 GPUs (BS of 8/GPU with gradient accumulation 2) using learning rate of le - with Cosine Annealing through AdamW optimizer. We carry out hyper-parametts tuning based on the validation set to come up with the optimal values of learning rate (1e 5), scheduler (Cosine Annealing), batch size (8). gradient accumulation steps (2) and the optimizer AAdamW. We leverage text pre-trained Flan T5-xl as the backbone for Parsing Statement Generator (PSG) and the Cell Highlighter I.IMM The Parsing Statement Generator is trained for 50 epochs on the 300 manually annotated question-table-pann www.ttabbbppppaiggggg.e statement triplets with setting same as that of CABINET, however the only difference being the effective batch size of 16 (BS of 2/GPU with 8 GPUs without gradient accumulation). On similar note, the Cell Highlighter LLM is trained for 3 epochs on the ToTTo dataset to generate highlighted cells given the table and natural language statement describing subpart of the table as the input. During the inference phase of generating answer, parsing statements and highlighted cells, we use beam search decoding with a beam size of 3.
A.7 CAN CABINET BE USED WITH OTHER QA LLM BACKBONES TO IMPROVE PERFORMANCE?
To verify the generality of CABINET as a framework that can be used with any encoder-decoder style Table QA method, we use TAPEX (Liu al., 2022) as the backbone initialise the unsuper- vised contextual relevance scorer (instead of OmniTab). TAPEX is also used as the underlying QA LLM. Table summarises the results where it can be seen that the performance improves on all the three datasets. This highlights the generality of CABINET in improving performance.
Table 7: Performance improvement achieved through CABINET over TAPEX by using TAPEX as the underlying QA LLM (instead of OmniTab). Further, the encoder of TAPEX is used to initialise the unsupervised relevance scorer (instead of OmniTTa)). This highlights the generality of CABINET as a framework for improving performance.
Method
WikiTQ
FeTaQA
WikiSQL
TAPEX
55.5
34.6
86.4
CABINET w TAPEX Backbone
62.7
37.8
87.3
A.8 ABLATIONS ON ALTERNATE DESIGN CHOICES FOR CABINET
We discuss several explorations alternate ways of utlising the different components of CABINET framework. The results pertaining to those can be seen in table 8.
1. Instead of using the generated parsing statement to highlight cells for our cell-based scoring mechanism, we feed the parsing statement in the input to the QA LLM along with the ques- tion and table in order to verify if parsing statement alone can improve the QA performance when directly used as instruction to the model. We use the unsupervised relevance scorer along with the QA LLM. Table summarises the results. It can be seen that using the pars- ing statement input to the QA model gives significantly sub-optimal performance (row vs row 7 in Table 8) compared to highlighting cells to obtain the corresponding scores that can be combined with the relevance scores predicted by the unsupervised relevance scorer.
19

---

Published as a conference paper at ICLR 2024
2. As an alternate to URS, we explore similarity between semantic embeddings of question and table row for estimating relevance of tokens. Precisely, to perform this ablation, we encode the question and each in the table with BERT. Subsequently, compute the similarity score between the encoding of each table row and question encoding through cosine similarity. Each token in given table row is then assigned relevance score equal to the cosine similarity between the row and question encodings. This relevance score is then used (instead of URS) with CABINET. Clearly from 8, replacing URS with BERT- based similarity results in sub-optimal performance when compared against CABINET with URS and without cell highlighter (row 3 vs row 6), hence validating the importance URS.
3. Further, we experiment with another configuration where the weakly supervised cell high- lighter is used with off the shelf BERT based encoding similarity for relevance scoring and compare the performance with CABINET (with URS and cell highlighter). Here, a similar trend is observed where it is seen that CABINET with URS and cell highlighter performs much better than CABINET with BERT based relevance scoring and cell highlighter (row 4 vs row 7).
4. Utilising the question directly to generate the highlighted cells, can be observed that there is significant decline in performance for the task of TableQA compared to using the parsing statement as input to the cell highlighter to generate the relevant cells which is then used determine relevance score for table tokens for performing QA (row 5 vs row
7). This indicates that the parsing statement describing the criteria of rows and columns relevant to the question is essential to achieve the performance gains.
Table 8: Performance analysis for different design choices explored over CABINET. OmniTab as baseline (row 1), ablation using parsing statement as input to QA LLM instead of higlighting corresponding cells (row 2), leveraging BERT based relevance scoring (row and 4), directly using question input to cell higlighter LLM (row 5), produce sub-optimal performance when compared with CABINET.
Method
WikiTQ
FeTaQA
WikiSQL
TmniTab
63.1
35.9
85.8
CABINET w parsing statement as input to QA model instead of highlighting corresponding cells
66.2
34.9
85.9
CABINET with BERT based relevance scoring (as discussed above) without cell highlighter
61.8
34.9
83.7
CABINET with BERT based relevance scoring (as discussed above) with cell highlighter
64.5
36.7
85.1
CABINET with question as input to cell highlighter
63.7
34.4
85.7
CABINET with URS only and without cell highlighter
65.6
35.8
89.3
CABINET
69.1
40.5
89.5
A.9 DATASET STATISTICS
In this section, tabulate the number of samples in the training, validation and test set of all the three datasets in table 9.
A. CASE STUDY ON HOW CLUSTERING LOSSES INTERACT TO YIELD IMPROVEMENTS
We now discuss case study depicting how the loss functions interact yield improvements:
20

---

Published as a conference paper at ICLR 2024
Table 9: Dataset Statistics
Dataset
# #Triin samples
# Validation samples
#Test samples
WikiTQ
11321
2831
4344
WikisQL 56355
8421
15878
FeTaQA 7326
1001
2003
Consider the example in figure 2 in the paper pdf we plot the histogram of relevance score assigned to table tokens during inference for this example for differently trained variants of CABINET a) URS trained w/o any clustering losses, b) URS trained with clustering loss, URS trained with clustering and cluster mean separation loss, and d) URS trained with clustering, cluster mean separation and relevance score sparsification loss. The histogram plots depicting count of table tokens against relevance score assigned during inference for the 4 variants are shown in figure 7, we summarise observations below:
1. From the sub-plot figure 7(a), it can seen that when URS trained without any of the three losses, the relevance score for most of the tokens is in the range 0.7 0.9 which is undesirable since many tokens which are irrelevant are also assigned a decently high relevance
which is roughly equivalent to passing the table to the QA LLM as it is.
2. When URS is trained with clustering loss (sub-ple plot-plotten figure 7(b)), the frequency distribution relevance scores becomes bi-modal with the majority of tokens corresponding to the first mode assigned relevance score in the range 0.6-0.7 and for the second mode around 0.8-0.9. This shows that clustering loss trains the URS in structuring the latent space rep- resentation into two categories, however, still there are many tokens with a relevance score around 0.7 between the two modes which means that many irrelevant tokens are still as- signed decently high relevance.
3. When URS is trained with clustering and cluster mean separation loss (figure 7(c)), the first mode corresponding to relatively lower relevance score is observed around 0.55-0.65 while the second mode corresponding to a higher relevance score is observed around 0.8- 0.9. This shows that URS trained with cluster mean separation loss amplifies the effect clustering loss by pushing more tokens into lower relevance category resulting in more tokens assigned lower relevance around the first mode. Also note that the number of tokens in the range 0.7-0.8 also reduces owing to the increased gap between token representations belonging to categories.
4. Finally when URS is additionally trained with relevance score sparsification loss (fig- ure 7(d)), we observe that magnitude of relevance score assigned to tokens in low relevance score cluster decreases further which is useful since irrelevant tokens gets suppressed fur- ther and the downstream QA LLM can focus better on relevant parts. Further, more irrele- vant tokens are assigned a low relevance score. This can be seen figure 7(d) where more tokens are assigned relevance score in the range 0.45-0.6.
Further, since the URS is trained end-to-end with QA LLM differentiably, tokens in higher score category are likely to be relevant in order to enable the QA LLM to be able to generate the correct answer. Additionally, in figure 8, we visualise the t-SNE plots (corresponding to the same example in figure 2 as discussed above) of the latent representation of table tokens encoded during inference by URS corresponding the 4 variants trained differently a) URS trained w/o any clustering losses (figure 8(a)), b) URS trained with clustering loss (figure 8(b)). c) URS trained with clustering and cluster mean separation loss (figure 8(c)), and d) URS trained with clustering, cluster mean separa- tion and relevance score sparsification losses (figure 8(d)). The table tokens having relevance score greater than the average relevance score (assigned to table tokens in this example) are depicted as relevant tokens (in red) while those tokens having score less than the average relevance are depicted as non-relevant (in blue). It can be seen that URS model trained with clustering loss, cluster means separation loss and relevance score sparsification loss is better able to segregate the table tokens into two categories and assign lower relevance to tokens in one category while assigning higher rele- vance to tokens in the second category. Further, since the URS is trained end-to-end with QA LLM differentiably. tokens in higher score category are likely to be relevant in order to enable the QA I.LM to be able to generate the correct answer.
21

---

Published as a conference paper at ICLR 2024
60
60
Count Table Tokens
Count Tabll Tokens
40
40
20
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.4
0.5
0.6
0.7
0.8
0.9
10
Relevance
Relevance
(a) Histogram plot count table tokens against relevance score assigned during inference by URS trained without any clustering losses.
(b) Histogram plot of count of table tokens against relevance score assigned during inference by URS trained with clustering loss.
Count of Table Tokens
50
50
of Table
30
30
10
Relevance score
Relevance score
(c) Histogram plot of count of table tokens against relevance assigned during inference by URS trained with clustering loss and clusters mean sepa- ration loss.
(d) Histogram plot of count of table tokens against relevance assigned during inference by URS trained with clustering loss, clusters mean separation loss and relevance score sparsification loss.
Figure 7: Histogram plots of relevance example figure in the main paper. The plots correspond to variants of CABINET trained with different combination of loss functions for URS a) URS trained w/o any clustering losses; b) URS trained with clustering loss; e) URS trained with clustering and cluster mean separation loss; and d) URS trained with clustering, cluster mean separation and relevance acts complementary manner to enable URS to segregate table tokens better into two categories and assign low relevance to tokens in one category and high relevance to tokens in the second category. Further, since the URS is trained end-to-end with QA LLM differentiably, tokens in higher score category are likely to be relevant in order to enable the QA LLM be able generate the correct answer.
assigned during inference table tokens corresponding the
sparsification loss. It can be
that the three losses
22

---

Published as a conference paper at ICLR 2024
tokens
Nonrelevant tokens
(a) t-SNE plot of latent representation of table to- kens encoded during inference by URS trained with- out any clustering losses.
(b) t-SNE plot of latent representation of table tokens encoded during inference by URS trained with clus- tering loss only.
Non-relevant telkens Melevvant takens
Non-rolevant takenns
(d) (-SNE plot of latent representation of table tokens encoded during inference by URS trained with clus- tering loss, cluster means separation and relevance score sparsification loss.
(c) I-SNE plot latent representation of table tokens encoded during inference by URS trained with clus- tering loss and cluster means separation loss.
Figure 8: t-SNE plots (corresponding the same example as in figure of the latent representation of table tokens encoded during inference by URS corresponding to the variants trained differently a) trained w/o any clustering losses (sub-plot figure 8(a)), b) URS trained with clustering loss only (sub-plot figure 8(b)), URS trained with clustering and cluster mean separation loss (sub-plot figure 8(c)). and d) URS trained with clustering, cluster mean scparation and relevance score sparsification loss (sub-plot figure 8(d)). The table tokens having relevance score greater than the average relevance score (assigned to table tokens in this example) are depicted as relevant tokens (in red) while those tokens having score less than the average relevance are depicted as non-relevant (in blue). It can be seen that URS model trained with clustering loss, cluster means separation loss and relevance score sparsification loss is better able to segregate the table tokens into two categories and assign lower relevance tokens in one category while assigning higher relevance tokens in the second category. Further, since the URS trained end-to-end with QA LLM differentiably, tokens in higher score category are likely to be relevant in order to enable the QA LLM to be able to generate the correct answer.
23

---

Published as a conference paper at ICLR 2024
A.11 CAN CABINET USED FOR IMPROVING OTHER NLP TASKS LIKE READING COMPREHENSION?
We employ CABINET on the reading comprehension task, where given a paragraph and a corre- sponding question, LLMs need to answer the question based on the paragraph. Drawing an analogy from the Table QA task, certain tokens in the paragraph are more relevant for answering the given question. To achieve this, we employ the URS component of CABINET on top of traitrained BART-Large. We experiment with a commonly used benchmark SQuAD-v2 and report accuracy on the test set (Table 10), hence validating the task-level generality of CABINET. It can be seen that using CABINET helps improving the performance of vanilla BART-large LLM for the reading comprehension task.
Table 10: Performance improvement achieved upon employing URS of CABINET over BART for the Reading Comprehension task on SQuAD-v2
Method
SQuAD-v2 Test Accuracy
BaRT-large
42,9
URS of CABINET BART-large
48.0
24


Reviews:
[{"review_id": "kdro574QZ3", "reviewer": "Reviewer_NUWR", "summary": "To minimize the negative effects from the noisy and distracting information from irrelevant parts of the table, in this work, the authors propose CABINET framework that weighs different table parts based on their relevance to the question without explicitly removing any content. The main part of this framework is on the relevance score assignment, either with unsupervised learning or leveraging external weakly-supervised cell highlighter model, which provides the relevance measurement between table content and question. \nSpecifically, variational inference is used to obtain the relevance score with unsupervised fashion. Furthermore, CABINET leverages a parsing statement generator that describes which rows and columns are relevant to the question to provide more matching information. Experimental results show that: (1) The relevance score with three designed loss functions can improve the model performance. (2) The relevance scores obtained from unsupervised setting and weakly-supervised setting should be balanced to obtain good performance.", "questions": "What’s the performance of cell highlighter on ToTTo validation set? Do we have evaluations on some sampled examples from FetaQA, WikiTQ, and WikiSQL showing the quality of parsing statement generator? If the quality of parsing statement is high but putting more weight on that makes the model perform worse, then it would be an interesting point to discuss.\nFrom Table 4, there is a big improvement of using all three loss functions against the models without using all three loss functions. Do we have some case studies showing where the improvements come from?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The model achieves the state-of-the-art performance on the WikiTQ, FetaQA and WikiSQL using orders of magnitude fewer parameters when compared of LLMs such as Codex. \nFurthermore, the model is more robust on the perturbation than other counterparts. Also for large tables, the CABINET also shows better performance than OmniTab.  \nThe authors smartly use the ToTTo dataset to train a cell highlighter model for obtaining the relevance score from the parsing statement. \nThe ~300 manually annotated example for parsing statement generation will be very useful for the community.", "weaknesses": "Even though there is a good ablation experiments on the three loss functions, my overall feeling is that there is no good rational how these loss function interact with each other. For example, we can see that combining all three loss functions performs best. However, the benefits of a single loss function is hard to be observed. I think this strategy can be further applied to reading comprehension task. If it works, this can make the paper more comprehensive, having broader impact. No other weaknesses in my mind but there are some technical questions in Question section.\n\n-----\nWith responses and updated number for reading comprehension task, I am happy to change the rating accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ivsa0VSFru", "reviewer": "Reviewer_nyxN", "summary": "This paper introduces a novel architecture for table QA. It is with two components, which are the main contributions of this paper. Unsupervised Relevance Scorer provides a \"soft\" assignment score for each row of the table given the question. Relevant Cell Predictor through Table Parsing converts the question into a statement with highlighted column and row information that can be used to solve the QA tasks. This method achieves SOTA performances on WikiTable, FeTaQA, and WikiSQL. Overall, this is a great paper.", "questions": "* What is the average model inference latency compared with the baseline model (OmniTAB) for the three tasks\n* More ablation studies are needed to support the needs of the components introduced in this paper. \n  - The need for an Unsupervised Relevance Scorer: If we replace URS with BM25 or some out-of-the-box similarity metrics, will the model have a huge performance decline?\n  - The need for a Parsing Statement Generator: If the original question is directly used for cell highlighter other than the parsed text, will the model have a huge performance decline?\n  - The need for a Cell Highlighter: If the $\\eta^{cell}$ is the indicator for a cell value in the parsed text, will the model have a huge performance decline?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* This paper proposed a novel framework for table QA tasks\n* The proposed method achieves SOTA performances on three table QA tasks\n* The paper shows the robustness of the provided model", "weaknesses": "* Unlike a \"hard\" table token retriever, the computation of the model proposed by this paper may be extensive when the number of table tokens is many. \n* More ablation studies are needed to support the needs of the components introduced in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "myFVwtYn08", "reviewer": "Reviewer_e8r4", "summary": "This paper tackles the task of table Question-Answering (QA). The authors state that the approaches considering all the tables have too much noise when generating the answer, while approaches selecting a part of the table before answering might remove the relevant parts. The authors thus propose to estimate the relevance of each token based on a clustering approach with latent variable (Srivastava and Sutton, 2017), coupled with two custom losses (sparsity of the relevance cluster and distance between the centroids). They combine this token relevance score with a predictor of a table \"cell relevance\" based on a sequence-to-sequence model that outputs the relevant cells from a full table. \n\nResults reported on the main table QA datasets (WikiTableQuestion, FeTaQa, and WikiSQL) show a noticeable improvement (e.g. gaining 3 points in accuracy compared to a LLM/Codex prompting approach that select subtables, DATER, on WikiTQ). Experiments consisting in perturbing the tables (adding rows, column/row permutation, cell replacement) show that the method is more robust than OmniTAB and ReasTAP. Finally,", "questions": "- section 3.1, the whole paragraph discussing variational inference sounds strange: variational inference is a method to estimate laternt variable probabilities - stating that \"we model ... as a latent variable ... **through VI**\" is not correct\n\n- the experimental details (how the model was trained, how hyperparameters were set) are not given; could the authors provide the information (at least in the appendix)?\n\n- How is the model used during inference (i.e. how is the VI-related loss used to predict the relevance of each token)?\n\n- is the clustering model that important? Would predicting directly a probability of relevance for each token work?\n\n- p.9, please correct \"its performance\" to \"TAPEX performance\" for clarity", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The approach has good results on the main table QA dataset\n- The ablation study shows that each subcomponent of the model is important\n- The model outperforms LLM-based approaches with a much lower number of parameters (175B vs 3.5B) and other state-of-the-art approaches (e.g. OmniTAB)", "weaknesses": "- The overall model is quite complex, relying on various subcomponents of different natures: this paper provides a new baseline, but it is hard to build from that\n- The cell relevance loss is not learned, making the model not end-to-end\n- Using a latent variable for relevance, which is optimized as within a probabilistic model for relevance, but then used as a scalar, makes the model a bit inconsistent - a fully probabilistic model would have been much sounder\n- there are missing experimental details", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S3L83EEcGd", "reviewer": "Reviewer_mwZa", "summary": "The authors propose CABINET (Content RelevAnce-Based NoIse ReductioN for TablE QuesTion-Answering) – a framework to enable LLMs to focus on relevant tabular data by suppressing extraneous information. CABINET comprises an Unsupervised Relevance Scorer, trained differentially with the QA LLM, that weighs the table content based on its relevance to the input question before feeding it to the question-answering LLM (QA LLM). Further, it uses a weakly supervised module that generates a parsing statement describing the criteria of rows and columns relevant to the question and highlights the content of corresponding table cells. Authors release code and dataset to enable for reproducibility.", "questions": "Please see weaknesses section above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is technically sound and in general well written. Figure 2 is also informative.", "weaknesses": "- The authors need to better motivate what the practical utility of Table QA is, as most of the questions are based in the format of natural language text sentences. Moreover, tabular dataset can be indexed and converted to their freeform text representations which general open-domain QA systems are already able to solve. \n\n- In the Related Work Section, the authors should also discuss recent developments with LLM guided graph neural network (GNN) model for QA to capture tabular/graph structure:\n\n[1] EACL 2023: Question-Answer Sentence Graph for Joint Modeling Answer Selection. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 968–979, Dubrovnik, Croatia. Association for Computational Linguistics.\n[2] AAAI 2020: TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection. AAAI 2020: 7780-7788\n\n- It would be helpful if the authors could better summarize the statistics of their datasets in tabular as opposed to text format. Further, based on the text description, I have a concern that the datasets being used are relatively small-scale, e.g., in the order of thousands of nodes. The experiment results would be more conclusive if evaluated on large-scale datasets. To this end, the authors also should provide analysis on the runtime and memory complexities of their work, since M/B parameter LLMs may not be scalable for training time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)