An Inner Table Retriever for Robust Table Question Answering
Year: 2023

Abstract:
Recent years have witnessed the thriving of pretrained Transformer-based language models for understanding semi-structured tables, with several applications, such as Table Question Answering (TableQA). These models are typically trained on joint tables and surrounding natural language text, by linearizing table content into sequences comprising special tokens and cell information. This yields very long sequences which increase system inefficiency, and moreover, simply truncating long sequences results in information loss for downstream tasks. We propose Inner Table Retriever (ITR),1 a generalpurpose approach for handling long tables in TableQA that extracts sub-tables to preserve the most relevant information for a question. We show that ITR can be easily integrated into existing systems to improve their accuracy with up to 1.3-4.8% and achieve state-of-the-art results in two benchmarks, i.e., 63.4% in WikiTableQuestions and 92.1% in WikiSQL. Additionally, we show that ITR makes TableQA systems more robust to reduced model capacity and to different ordering of columns and rows.

heading: 1 Introduction
text: Tables offer a systematic way of storing information in the Web. Extracting information from Web tables poses different challenges than extracting information from relational databases with logical queries, especially when queried via Natural Language (NL) user questions. Table Question Answering (TableQA) is the task of answering such questions with factoid answers extracted from table content. This requires developing models with the ability to reason over and understand tables. Following the success of the pre-training paradigm for understanding NL text (Devlin et al., 2019), some recent research has focused on pretraining Transformer models (Vaswani et al., 2017)
∗Work done as an intern at Amazon Alexa AI. 1We make our code available at: https://github.com/
amazon-science/robust-tableqa
on large corpora of linearized tables in a selfsupervised fashion using encoder-only architectures (Herzig et al., 2020; Yin et al., 2020; Yang et al., 2022), or encoder-decoder architectures (Liu et al., 2022; Jiang et al., 2022). These so-called Tabular Language Models (Dong et al., 2022, TaLMs) were fine-tuned on the TableQA downstream task— among others—to achieve state-of-the-art performance (Herzig et al., 2020; Liu et al., 2022). However, the self-attention mechanism in TaLMs has a quadratic complexity on the dimensionality of input tables, which might consist of tens or even hundreds of rows and columns, thus yielding longer sequences than TaLMs can easily handle. State-ofthe-art TableQA models handle this limitation by truncating the linearized table to fit an input length budget, e.g., of 512 and 1024 tokens for Herzig et al. (2020, TaPas) and Liu et al. (2022, TaPEx), respectively. In other applications, simple sequence truncation might be reasonable, e.g., encoding only the initial paragraphs of a Wikipedia document pre-
9909
suming it comprises a summary, or dropping earlier turns in Conversational QA to focus on the recent ones. However, in TableQA it is not realistic to assume that relevance depends on the position within the linearized sequence, especially because different questions require various table regions to be properly answered. For example, even for a standard dataset such as WikiTableQuestions, naive truncation allows information loss affecting 18.1%- 44.9% of tables, which limits QA accuracy (see §4.1 for more details). This is also an important limitation in latency-constrained realistic use cases that use big tables, while limiting Transformer models even down to 64 input tokens. To this end, a content-driven strategy is needed to avoid information loss. We refer to tables exceeding the input length budget as overflow tables, as opposed to compact tables, which fit within the budget. An example of an overflow table is shown in Figure 1(a), where naive truncation leads to the wrong answer.
In this work, we propose ITR to improve on this problem by creating smaller sub-tables, i.e., within a length budget, based on dense retrieval of table rows and columns according to the relevance to the question. An example is shown in Figure 1(b). Our method is flexible and can be integrated off-theshelf into virtually any existing TableQA system. To the best of our knowledge, our work is the first to propose a neural-based sub-table selection in the context of TableQA that improves denotation accuracy (Pasupat and Liang, 2015), especially for the overflow tables, setting a new state of the art. Other input selection strategies, mainly heuristics-based, have also been proposed in the literature (Krichene et al., 2021; Yin et al., 2020; Eisenschlos et al., 2021), which we discuss further in Section 2.
To summarize, the contributions of this work are the following:
1. We propose ITR, an efficient approach to handling overflow tables for TableQA models, which produces sub-tables containing the most relevant information for answering a question while fitting within the budget.
2. We combine ITR with existing TableQA systems such as TaPas, TaPEx, and OmniTab (Jiang et al., 2022), and achieve a new state of the art for two standard benchmarks, WikiSQL and WikiTableQuestions.
3. We evaluate the robustness of ITR against current TableQA models on overflow tables,
when reducing the length budget, and when repositioning relevant table information.

heading: 2 Related Work
text: The works most related to ours employed different pruning strategies to handle large tables. Yin et al. (2020, TaBERT) introduce the concept of content snapshot as encoder input. This snapshot is composed of a small number of rows which are chosen based on the n-gram similarity between the question and column headers and cell values. In a similar fashion, Eisenschlos et al. (2020) explore Jaccard similarity to obtain the most similar columns. In addition, they leverage model tokenizer to reduce cell tokens to their first token, when necessary, and dropping entire rows that do not fit the length budget. However, lexical similarity and naive truncation are not flexible and lead to information loss, which has a drastic effect on TableQA performance, as we show in our experiments. Another line of work focuses on balancing model efficiency and accuracy when handling long tables. Krichene et al. (2021, DOT) first uses a smaller pruning transformer to selects top-K tokens from the input table, and then a larger second task-specific transformer takes into consideration only the selected K tokens and their pruning scores; Eisenschlos et al. (2021, MATE) can accept more tokens while not significantly increasing latency. Authors apply sparse self-attention and use different attention heads for tokens in the same column and row. However, their proposed mechanisms are deeply integrated with the task-specific model and must be jointly trained. In contrast, our ITR method, as a flexible plug-in process, can work independently of any underlying TableQA model. Moreover, our approach is complementary to theirs since ITR can drop irrelevant information in tables efficiently and pass the trimmed compact table to the underlying model. This can further exploit the potentials of virtually any TableQA models and improves their performance. Although not specific to TableQA, works such as Wang et al. (2021) and Chen et al. (2021) employ chunking strategies, i.e., encoding table chunks separately and then aggregate them together. However chunking is not widely employed in the literature (Dong et al., 2022), due to encoding overhead requiring multiple inference calls for each chunk.
Algorithm 1 Creating N sub-tables from T for q. 1: def ITR(q, T , EQ, ET , N , b): 2: Z ← [], Tsub ← [], Taux ← [], L← [] 3: eq ← EQ(q) 4: for each i ∈ Items(T ) do 5: et ← ET (i) 6: Z ← Z ∪ {sim(eq, et), i} 7: for each i ∈ Sorted(Z) do ▷ ↓ sim 8: Taux ← Taux ∪ i 9: if CheckValid(Taux) then
10: Tsub ← Tsub ∪ Taux 11: for each t ∈ Tsub do 12: L← L ∪ {Length(t), t} 13: Tsub ← [] 14: for each (l, t) ∈ Sorted(L) do ▷ ↓ length 15: if l > b then continue 16: Tsub ← Tsub ∪ t 17: return Tsub[ : N ]

heading: 3 Methodology

heading: 3.1 Task
text: Given a question q and a table T , TableQA systems return an answer denotation a, either by performing table cell selection or as the result of operations (such as counting) carried out over an aggregation of table cells. This work aims to find one or more sub-tables Tsub containing the most relevant information from T needed to answer q; Tsub can replace T as the input to virtually any existing TableQA system.
To this end, we map a table T into a set of items, where an item is either a complete row or a complete column. For an n × m table, this gives a set of items {r1, . . . , rn, c1, . . . , cm}. Then, we construct sub-tables by specifying subsets of these rows and columns: a sub-table consists of the cells at the intersection of the selected rows and columns. We refer to each such set of rows and columns as a mix, and note that a mix must contain at least one row and one column to specify a valid sub-table. The table in Figure 1(a) is defined as {r1, r2, r3, r4, c1, c2, c3, c4}, and the sub-table in Figure 1(b) is specified as the mix {r3, r4, c2, c4}. {c2, c4} is not a valid sub-table, since no cells will be intersected with only column-wise items.

heading: 3.2 Inner Table Retriever
text: ITR is a process of retrieving table rows and columns, and combining them to form sub-tables
Tsub, with q as a query. We describe the steps for creating sub-tables in Algorithm 1. Lines 2-6 compute item similarities to q, and the function Items(T ) in Line 4 maps T into its n+m items. Following Karpukhin et al. (2020, DPR), we have two fine-tuned encoders, one for questions (EQ) and another for table items (ET ).
We fine-tune DPR encoders using a question as a query and the row/columns that contain the gold answer cells as positive items. Then we sample the negative items from the remaining row/columns in the table. We leverage the standard DPR contrastive loss to fine-tune the two encoders so that the similarities between question and positive item embeddings are maximized.2 At inference time, we compute the contextual embeddings for the question (Line 3) and all the items in a table (Line 5) and compute their similarity, sim(·), as the dot product in Line 6. In practice, pre-computed embeddings of table items can be cached offline.
Creating sub-tables. In Lines 7-10 we loop through the table items ranked by highest similarity and aggregate in Taux. CheckValid( Taux) verifies that Taux is a valid sub-table, i.e., there exists at least one column-row intersection (see §3.1).
Choosing the most appropriate sub-tables. In Lines 11-16 we sort the sub-tables by their sequence length in descending order, and filter out any sub-table which exceeds the length budget b. Finally, in Line 17 we return top-N remaining largest sub-tables that fit the budget length to be used as input to the TableQA model. Each returned sub-table is guaranteed to contain the most relevant items, due to the sorting operation in Line 7.

heading: 3.3 TableQA with ITR
text: Through the ITR process, we obtain N sub-tables Tsub as replacement for the original table T . Each sub-table, together with the associated NL question q, is linearized into a sequence of tokens prior to encoding, using the corresponding TableQA tokenizer. We exemplify this in Figure 1, where both table (a) and sub-table (b) are similarly processed. Each linearized sequence is used as input to the TableQA model, thus obtaining N predictions, out of which we choose the most confident answer. We empirically find that the model performance is marginally better when N > 1, instead of only considering the largest sub-table (see §6; Appendix C).
2More details in Appendix A.1.

heading: 4 Experimental Setup

heading: 4.1 Datasets and Evaluation
text: We use two popular datasets for TableQA which are constructed from Wikipedia: WikiSQL3 (Zhong et al., 2017) and WikiTableQuestions4 (Pasupat and Liang, 2015, WikiTQ), with each posing different challenges. WikiSQL is a simpler TableQA dataset than WikiTQ as it requires mainly filtering and simple operations of table cells to answer a question. WikiTQ demands more complicated reasoning capabilities such as aggregations, comparisons, superlatives, or arithmetic operations, e.g., SUM, MAX. We measure Denotation Accuracy (DA) for validation and test sets, to assess whether the predicted answer(s) is equal to the ground-truth answer(s). Additionally, we introduce the distinction between compact tables and overflow tables, which is determined by the length of linearized questiontable pair. Statistics in Table 1 show that even when using a relatively high number of tokens, i.e., 1024 or 512—the allowed maximum supported by most of Transformer-based encoders— the range of overflow tables is very high, 7-33% in WikiSQL, and 18-45% in WikiTQ. ITR is applied only for overflow tables, as the compact ones already fit in the token budget. Finally, we evaluate ITR retrieval ability on WikiSQL using Recall@K, which measures whether all the gold rows/columns for answering a question are among the top-K retrieved items.

heading: 4.2 Training Setup
text: For both, the ITR retrieval component and the underlying TableQA systems that we train in-house,
3https://huggingface.co/datasets/wikisql 4https://huggingface.co/datasets/
wikitablequestions
we choose the best checkpoint based on the performance on the validation set. Otherwise, for TableQA systems in the literature, we use the released checkpoints from huggingface.
Since WikiTQ does not provide SQL annotations, which are needed to obtain gold answer cell coordinates for supervising the ITR retriever (see §3.2), we use the model trained on WikiSQL in a zero-shot fashion to retrieve the relevant table items for the WikiTQ TableQA task. We set the number of sub-tables N=10 when using TaPEx and OmniTab systems, and N=1 with TaPas (see Appendix C). We provide details and hyperparameters for ITR and TableQA models in Appendix A.

heading: 4.3 Comparison Systems
text: We evaluate the effectiveness of ITR by comparing our ITR-enhanced baselines with recent models from literature that report performance on WikiSQL or WikiTQ (Min et al., 2019; Herzig et al., 2020; Yu et al., 2021; Liu et al., 2022; Jiang et al., 2022). We provide detailed comparisons using TaPEx, TaPas, and OmniTab, with ITR included in inference alone, as well with ITR integrated into TaPEx training. TaPEx leverages BART (Lewis et al., 2020), an encoder-decoder model, and finetunes it to act as a SQL executor. For the TableQA downstream task, TaPEx takes as input a <question, table> pair and autoregressively generates an answer, implicitly performing any kind of aggregations. OmniTab extends TaPEx and leverages multi-tasking and data augmentation to establish a new state of the art in WikiTQ. TaPas is an encoder model built on top of BERT (Devlin et al., 2019) which takes as input a <question, table> pair and learns jointly to: i) select table cells by thresholding the cell confidence, and ii) predict an explicit operator to aggregate results from the cell selection. We use a recent version of TaPas proposed by Eisenschlos et al. (2020) which leverages intermediate pretraining and table pruning, e.g., cell truncation to the first token and dropping rows that exceed the limit, improving significantly the initially released model (TaPasv0). When applying ITR, we disable TaPas table pruning and use the full sub-table(s) (see Appendix D for a case study). Regarding their capacity, TaPEx and OmniTab support up to 1024 tokens, while TaPas can take up to 512 tokens.
It is worth noting that we have not been able to fully reproduce the reported results in the literature of TaPas, TaPEx and OmniTab with their
huggingface implementations, due to crossframework dependencies or any model-specific preprocessing or evaluation scripts. To assess the realistic contribution of ITR, we also report reproduced results using the same unified framework across all models. We discuss reproducibility in Appendix A.3.

heading: 5 Results

heading: 5.1 Main Results
text: We report the performance of our best ITRenhanced TableQA models for WikiSQL and WikiTQ in Tables 2 and 3, respectively, and compare with the state-of-the-art results as reported in the literature. In WikiSQL, ITR-enhanced models consistently outperform all previous baselines. ITR improves TaPEx and TaPas performance with 2.6 and 2.3 DA points in the Dev set, respectively. ITR → TaPas sets a new state of the art for WikiSQL, reaching a DA of 92.1% in the Test set. In WikiTQ, results are mixed: ITR shows slight degradation over TaPas in the Dev set. Combined with TaPEx, ITR improves DA by 4 points. Further, combined with OmniTab, ITR improves DA by 0.6%, reach-
ing a new state-of-the-art result on this task. In Table 4 we compare ITR in a unified experimentation setting using the released model checkpoints in huggingface for inference-only evaluation, and the in-house fine-tuned TaPEx model that enables both training and evaluation with ITR. For OmniTab, only the WikiTQ model is made available, thus we do not evaluate in WikiSQL. We also break down the performance into compact and overflow tables to better assess the contribution of ITR. When evaluated using the same settings, ITR consistently improves on top of baselines(#5-7 versus #1-3) on WikiSQL and WikiTQ, respectively: TaPas (+2.6% and +0.2%), TaPEx (+2.9% and +1.4%) and OmniTab (+1.3% on WikiTQ only). This is also true for the in-house trained TaPEx (#4 versus #8). ITR reduces the compact/overflow performance gap significantly: ITR increases overflow DA with up to 30% (with TaPEx) and 9.4% (with OmniTab) on WikiSQL and WikiTQ, respectively, making the underlying model more robust to larger tables. In fact, in WikiSQL we close the gap between compact and overflow tables (#5). These results show that ITR can be applied at inference time to always improve performance, by presenting more complete and relevant information in stateof-the-art model decision process, even without interfering with the model training process. Additionally, fine-tuning TaPEx with ITR sub-tables (#9) is better than plugging ITR only at inference time (#8). Training with ITR improves the overflow DA by 1.7% on WikiSQL and 3.7% on WikiTQ. The Test performance increases by 0.8% and 2.7%, respectively. Not only does ITR improve the decision process of underlying models at inference time, but it further increases performance if included in the model training process as well. This demonstrates that ITR can be flexibly applied to any TableQA model.

heading: 5.2 Repositioning Denotations
text: In Table 4 we notice that models that naively truncate table sequences (#1-4) may observe the correct denotations within the length budget, achieving 58.2-63.2% DA in WikiSQL even for overflow samples. This is because the original table happens to present the answer early, e.g., in the first column/row. To fully investigate the potential and usefulness of ITR, we create an extreme case by moving gold rows and columns altogether to the bottom right of the table, thus reducing the chances of arbi-
trary success due to dataset design. For this analysis, we use WikiSQL since it provides the gold cell annotations. We evaluate TaPEx in combination with ITR on both the original and extreme-case set and report results in Table 5. Unsurprisingly, the overall performance of TaPEx drops sharply from 87.7% to 82.8%, with the overflow accuracy dropping from 58.2% to only 23.6%. This suggests that in extreme cases failure to address the long table issue will harshly degrade model performance and that the reported DA values can be optimistic due to the convenient position of the relevant information on top of the table. ITR-enhanced TaPEx is less affected: the overall DA degrades by 1.4% and overflow accuracy by only 4.1% as opposed to 34.6% drop of TaPEx. The 4.1% drop is mainly due to the positioning bias that the TableQA system might have learned during training. In fact, if we introduce row/column repositioning also at training time (+train shuffled), i.e., making gold answer denotations appear equally in any possible position of the input sub-table, we observe a reduced impact of the extreme repositioning at inference time: the performance drop is only 0.7% overall and 1.9% in the overflow split. We discuss
row/column positioning effects in Appendix B.

heading: 5.3 Reducing the Input Length Budget
text: In production pipelines, latency is a crucial issue and 1024 tokens can be unrealistic in achieving a smooth user experience. We explore the input length budget within 64 to 1024 tokens and compare TaPEx and TaPas when combined with ITR in WikiSQL and WikiTQ. We recall from Table 1 that overflow rate increases drastically with shorter length budgets. TaPEx and TaPas use different linearization and tokenization strategies, thus they yield different overflow rates for the same budget. However, for both models and datasets, 64 tokens lead to an 100% overflow rate. We show the DA values in WikiSQL and WikiTQ in Figure 2. In WikiSQL, ITR helps the TableQA model remain in the region of the best accuracy even when reducing the budget to 256 and 128 tokens for TaPEx and TaPas, respectively. Without the aid of ITR, both models sharply degrade in performance. Interestingly, TaPEx-based models are less robust when reducing the number of tokens than TaPas-based model. This is because TaPEx linearization uses up tokens faster, therefore encoding less table informa-
25.6
42.0
65.9
82.1 88.7 74.2
84.8 89.7 91.6 91.6
64 128 256 512 1024 0
20
40
60
80
100
48.3
71.4
84.5 89.5
86.4 90.7 91.9 92.1
64 128 256 512
16.9
24.9
38.4
51.2 55.5
24.4 31.4
42.3
53.9 56.9
64 128 256 512 1024
25.6
39.3
47.3 50.6
33.5
41.8 48.1 50.8
64 128 256 512 0
20
40
60
80
100
baseline +ITR
D en
ot at
io n
A cc
ur ac
y (%
)
O ve
rf lo
w R
at e
(% )
WikiSQL WikiTQ
Maximum Length (tokens)
Figure 2: Impact of input length budget on Denotation Accuracy (line plots) and Overflow Rate (bar plots) for WikiSQL (left) and WikiTQ (right) Test sets.
tion overall (see Appendix D for a case study). In WikiTQ, while the trend remains unchanged, there is a more noticeable drop for all the models. This is due to the challenging nature of WikiTQ questions, which generally require visibility of larger portion of tables. However, ITR still improves over baselines, further widening the gap as we decrease the length budget to 128 and 64 tokens. Similarly to results in WikiSQL, TaPas is more robust than TaPEx in extreme scenarios in WikiTQ. Finally, while ITR benefits models in standard benchmarks, it is even more beneficial in extreme realistic scenarios.

heading: 6 Ablation Study

heading: 6.1 ITR Variants
text: In addition to our ITR, we investigate several variants: with varying number of sub-tables N , representing tables using columns or rows only, creating sub-tables with different strategies, or scoring items via a different measure of relevance.
Row/Column-only Items. Our ITR considers a mix of both row and column items. We redefine Algorithm 1 to consider only row or column items, but not both, creating the following ITR variants:
1. ITRcol: Items(T ) maps a table T into a set of columns, e.g., {c1, c2, c3, c4} in Figure 1(a). A sub-table is created by combining the retrieved columns, e.g., Figure 1(b) would be represented as {c1, c3} but containing all 4 rows for the 2 columns. 2. ITRrow: similar to ITRcol, but only considering row-wise items.
Reduction versus Addition. ITR returns the largest possible sub-tables by iteratively dropping
irrelevant items to fully leverage the length budget (here referred to as Reduction, and models are suffixed by ‘−’). As such, we do not consider the smallest sub-tables that only contain the top-N relevant items. To verify whether dropping the irrelevant items is a better approach, we contrast it with an Addition strategy (models suffixed by ‘+’), where we return the top-N sub-tables created by successively appending the top-N items by their similarity with q, i.e., after Line 10 in Algorithm 1.
Semantic versus lexical. Finally, inspired by table input selection strategies in literature (Eisenschlos et al., 2020; Yin et al., 2020), we use ngram similarity instead of dense retrieval in Algorithm 1 (Lines 5-6) and obtain ITRngram. We use ITRngram to assess the importance of dense retrieval for ITR and its benefits in TableQA.

heading: 6.2 Results
text: In Table 6 we compare the accuracy of ITR variants on WikiSQL and WikiTQ. In WikiSQL, ITR{+,−}row variations (#4 & #6) perform better than the baseline (#1) and the column-wise counterparts. In WikiTQ we see the opposite: ITR{+,−}col variations (#3 & #5) perform better than baseline (#1) and the row-wise counterparts. However, ITR (#2) demonstrates superior performance across the board by jointly ranking the most relevant rows and columns, which strikes the right balance between the preference for each dataset. Even with N=1, ITR significantly improves the TaPEx baseline. Indeed, N=10 delivers only a slight improvement over N=1, by 0.3-0.4% in Test set depending on the dataset. As such, N=1 is a more efficient solution for applications with limited computational resources. We
further discuss varying N values in Appendix C.
Semantic versus lexical input selection. In Figure 3, we compare ITR and ITRngram when performing both row-wise, and column-wise retrieval. Neural-based ITR outperforms ITRngram for all values of K and both item types. The retrieval performance of ITR converges after K > 5 for ITRcol and K > 10 for ITRrow. For ITRngram, higher values of K are needed to achieve full Recall@K.
The lower performance of ITRngram is explained by poor lexical matching of the question with cell values and column names, as compared to the embedding similarities used by neural counterparts. For example, two viable questions that can be answered with the ‘Rank" column in Figure 1 are “which mountain peak is the highest?” and the less natural “which is the mountain peak that has the lowest value in the rank?”. While the latter matches with the column name, the former is more natural
for human to ask and it does not match the column names via n-gram similarity. It is worth mentioning that WikiSQL is more canonicalized and ITRngram results in Figure 3 might be positively affected by the nature of WikiSQL. We expect the gap between ITR and ITRngram to be even larger in challenging datasets where the reference question is more human-like, e.g., WikiTQ. This is indeed evident when these variations are integrated in TableQA. Unsurprisingly, results in Table 6 (#7) show that, in contrast to ITR, ITRngram variation is able to improve the baseline performance (#1) in WikiSQL, but degrades the performance in WikiTQ.

heading: 7 Conclusions
text: In this paper we presented ITR for TableQA, an approach to create the most relevant sub-table(s) to efficiently answer a given question via TableQA. ITR is based on a dense retrieval component, which selects relevant rows and columns and combines them into a compact sub-table that satisfies length budget constraints. We combined ITR with different TableQA models from the literature, at training and/or inference time, and showed that ITR indeed captures the most relevant information, which enabled underlying models to perform better overall and become more robust, thus attaining state-ofthe-art results on WikiSQL and WikiTQ benchmarks. ITR is flexible, does not depend on the underlying model and can be easily integrated in future model developments to increase their robustness. As future work we can combine ITR with computational operations over different table elements (Zhou et al., 2022) to collapse its information in a more compact format, to benefit also questions that rely on table completeness.

heading: 8 Limitations
text: First, in this work we limit the experimentation to vertical relational web-tables only, following the format of benchmarks used in TableQA, i.e., WikiSQL and WikiTQ. While we believe that ITR can easily be extended to horizontal entity web-tables, e.g., tables from Wikipedia, we do not expect our algorithm to transparently work on other types of tables that we do not consider, e.g., matrix tables from scientific papers and/or spreadsheets (Wang et al., 2021), where table items can be represented differently. However, this is not a limitation of the algorithm itself and adjusting our assumptions to certain scenarios and type of data can be feasible in the future. Second, ITR selects the relevant table elements by using a question as query. Therefore, it can only be applied for tasks with table-text joint input such as TableQA we showcase in the paper, or also table entailment tasks, e.g., table fact verification. Unfortunately, ITR cannot be used for tasks where table is the only input, e.g., table-to-text task. Finally, while ITR is beneficial for questions that do not rely on table completeness, its effectiveness is limited when, for example, all table cells are required to be predicted. Consider a question that requires cell counting, and the gold cells satisfying the query can be more than what we can feed a model with, e.g., “how many championship did Player A get?” and Player A has won 500 champions. However, this limitation does not arise from our approach and is rather inherited by existing TableQA models in the literature. Indeed, it can be a potential future direction of our work, which requires model innovation and table transformation that focuses on representing the information in a compact form.

heading: B Column and Row Order Effect
text: Despite the order of items returned by ITR, after creating and choosing the sub-tables, we rearrange their columns and rows in the same order as that in the original table. We rely on the order of the original training data, which can have its own biases in data creation. In addition, we observe that:
1. Exposing the most relevant items first at training time, in which case we also have access to gold items, leads to quick model over-fitting. The model can be strongly biased to choose
cells that appear early in the linearized sequence, which is not desired for training a generalizable and robust TableQA model.
2. Baseline models have been trained without a strong bias on the column/row order, i.e., not enforcing that most relevant items are shown first. We show several experiments in which we apply ITR at inference time only. As such, introducing an ordering bias at inference time only decreases the performance.
Furthermore, to investigate whether the positioning of gold answers in the dataset can bias the trained model, we shuffle sub-table rows and columns to make the gold answers appear equally possible in any position of the input table. Results in Table 5) showed that shuffling at training time slightly increases the robustness of the model by 0.2-0.5 denotation accuracy points in WikiSQL Test and Dev sets respectively. Interestingly, shuffling has a bigger impact in the extreme scenario (see § 5.2) increasing the Overflowext. by 2 denotation accuracy points. In the literature different strategies have been employed in the model design to avoid positional biases. For example, TableFormer (Yang et al., 2022) disposed of positional embedding to make all token positions homogeneous. However, such modifications of the baselines are out of the scope of this paper, in which we show the contribution of ITR on the current settings of each baseline.
chosen N for the best model. N=0 indicates the baseline, i.e., using the full table.

heading: C Multiple Sub-tables Effect
text: For our main experiments we use N > 1 sub-tables at inference time for generation baseline systems, i.e., TaPEx and OmniTab. In particular, we use N = 10 for our main ITR variant and ITRngram, while for the column and row only ITR variants, we set N = 5 and N = 10. In §5 (and Figure 3), we showed that ITR retrieval performance converges after K > 5 for columns and K > 10 for rows, which justifies the values selected in TableQA for N for each variant.
In §6 we showed the marginal impact of N=1 versus N=10. For completeness, in Table 11 we report the effect of varying N sub-tables for ITR → TaPEx: on the WikiSQL Test set, we get an improvement of 0.4 accuracy points for querying TaPEx on N=10 sub-tables versus doing so only on N=1 sub-table. Increasing N up to 20 yields no further improvements. We realize that using a large enough number of sub-tables, one might consider even simpler methods that consider different regions and combinations of the table each time, delegating the selection of the most relevant subtable to the TableQA system, as per prediction confidence. For this reason, we also compare a naive baseline that uses up to N=10 randomly chopped sub-tables from a given table, without a specific notion of item relevance, combined with TaPEx. For this baseline, we simply sample columns and rows and combine them similar to ITR mix until a sub-table exceeds the token budget. Results show that N=10 random sub-tables might allow TaPEx to improve its performance by +2.3% in WikiSQL (versus +3.7% improvement from ITR). In WikiTQ, randomly choosing the sub-tables degrades the performance by -3.4% (versus +3.7%
improvement from ITR). This is because in WikiSQL questions require less interaction between rows/columns and it might be enough for the system to have visibility of the items containing the gold answer. In WikiTQ, questions require aggregations between different columns and rows, and therefore a random combination of them leads to performance degradation.
We recall that for TaPas, we use N=1 as, due to joint tasks of cell selection and aggregation classification, it is not straightforward to determine the probability of the output making it unfeasible to compare N > 1 predictions.

heading: D Case Study
text: In this Section we discuss two case studies. We illustrate the relevance assigned by ITR on the original table rows and columns as a heat-map where the color scale reflects the relevance scores per each column and row with respect to the question (green → yellow→ orange→ red). To obtain cell scores in their intersection, we sum up their corresponding column-wise and row-wise scores. As a result, more relevant cells are more red.
In Table 12 we show a side-by-side comparison of TaPas and TaPEx under the 64 token reduction scenario, and the benefit of applying ITR. TaPEx uses special tokens for encoding the table structure, which make the linearized sequence longer. TaPas instead, encodes the table structure via additional embedding layers. In addition, TaPas applies cell truncation to the first token for each cell, which are reconstructed as a post-processing step, and drops rows that exceed the token budget. This allows TaPas to be fed a larger portion of the table in the input, even if the cell information might be lost, e.g., in Figure 4 ‘OF-8’ is squeezed into only ‘OF’ removing the distinction between ‘Equivalent NATO Rank’ across rows. As such, under extreme scenarios TaPas performs better than TaPEx, due to the visibility of a larger portion of the table. ITR proves beneficial and enables TaPEx to view only the relevant information within the token budget (Figure 6, left) to correctly answer the question.
In Table 13 we show a comparison of table pruning strategies included in TaPas, such as cell and row truncation, which might cause information loss. We disable those when we apply ITR, and provide TaPas with the full information contained on the question-relevant cells, as determined by ITR. In Figure 6 ‘New York’ and ‘New England Patriots’
Question: What could a Spanish Coronel be addressed as in the commonwealth military? Gold Answer: Group Captain.
Model Input [question, table (serialized and tokenized)] Prediction Notes
TaPEx 
 what could a spanish coronel be addressed as in the commonwealth military? col : equivalent nato rank code | rank in spanish | rank in english | commonwealth equivalent | us air force equivalent row 1 : of-8 | general del aire | lieutenant general | air marshal | lieutenant general
 (truncated at 64 tokens) Air Marshal TaPEx uses separation tokens interleaving cell values.
TaPas [CLS] what could a spanish coronel be addressed as in the commonwealth military? [SEP] equivalent rank rank commonwealth us of general lieutenant air lieutenant of brigadier major air major of coronel colonel group colonel of teniente lieutenant wing lieutenant of mayor major squadron major of capitan captain flight captain of teniente first flying first (62 tokens, with only 1 token in each cell and truncated at 7 rows) Group Captain TaPas uses additional embedding layers to encode table structure. The TaPas tokenizer by default squeezes the number of tokens in each cell to fit the table (e.g., ‘OF-8’ is squeezed into only OF), during which process information may be lost.
are both truncated as ‘New’ by TaPas. This information is crucial for correctly answering the question. Indeed, TaPas fails to locate the right cell after truncation, and predicts a wrong answer. The sub-table created by ITR in Figure 7 presents the full information of relevant cells to the model, thus enabling TaPas to make the correct prediction.
Question: Which winning team beat the New York Yankees? Gold Answer: Arizona Diamondbacks.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 and Appendix A
3 B1. Did you cite the creators of artifacts you used? Section 4 and Appendix A
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. We do not release any artifact with this submission. We release code upon acceptance.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. We only use publicly available resources.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. We only use publicly available resources.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4 and Appendix A
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: However, this is not a limitation of the algorithm itself and adjusting our assumptions to certain scenarios and type of data can be feasible in the future. Second, ITR selects the relevant table elements by using a question as query. Therefore, it can only be applied for tasks with table-text joint input such as TableQA we showcase in the paper, or also table entailment tasks, e. g. , table fact verification. Unfortunately, ITR cannot be used for tasks where table is the only input, e. g. , table-to-text task. Finally, while ITR is beneficial for questions that do not rely on table completeness, its effectiveness is limited when, for example, all table cells are required to be predicted. Consider a question that requires cell counting, and the gold cells satisfying the query can be more than what we can feed a model with, e. g. , “how many championship did Player A get?” and Player A has won 500 champions. However, this limitation does not arise from our approach and is rather inherited by existing TableQA models in the literature. Indeed, it can be a potential future direction of our work, which requires model innovation and table transformation that focuses on representing the information in a compact form