LI-RAGE: Late Interaction Retrieval Augmented Generation with Explicit Signals for Open-Domain Table Question Answering
Year: 2023

Abstract:
Recent open-domain TableQA models are typically implemented as retriever-reader pipelines. The retriever component is usually a variant of the Dense Passage Retriever, which computes the similarities between questions and tables based on a single representation of each. These fixed vectors can be insufficient to capture fine-grained features of potentially very big tables with heterogeneous row/column information. We address this limitation by 1) applying late interaction models which enforce a finer-grained interaction between question and table embeddings at retrieval time. In addition, we 2) incorporate a joint training scheme of the retriever and reader with explicit table-level signals, and 3) embed a binary relevance token as a prefix to the answer generated by the reader, so we can determine at inference time whether the table used to answer the question is reliable and filter accordingly. The combined strategies set a new state-to-the-art performance on two public open-domain TableQA datasets.

heading: 1 Introduction
text: Tabular data is ubiquitous on the Web. Opendomain Table Question Answering (TableQA), the task of answering questions grounded in tables, is increasingly attracting attention of both public and commercial research, for its value in real-world applications. Research TableQA pipelines are typically implemented with two components: a retriever and a reader. The retriever chooses a small set from the entire pool of table candidates, while the reader generates answers processing each table candidate. State-of-the-art implementations use transformer-based models for both components. In particular, the retriever is built with variants of Dense Passage Retriever (Karpukhin et al., 2020, DPR), which computes question-table similarity by using single vector representations of the question and the table. Retriever and reader can be trained
∗Work done as an intern at Amazon Alexa AI.
separately (Herzig et al., 2021) or jointly (Pan et al., 2022) via Retrieval Augmented Generation loss (Lewis et al., 2020b, RAG). We observe three limitations which we address in this paper.
First, a table can be very large and might contain heterogeneous information across rows/columns; encoding into a fixed size vector risks information loss, which can have an impact in QA quality. One way to alleviate this issue is to replace DPR with a Latent Interaction (LI) model, which encodes text into token-level representations. In particular, we find ColBERT (Khattab and Zaharia, 2020) to be very effective, even if not pretrained for tables.
Second, RAG uses only an implicit signal to guide the retriever. Recently, Lin and Byrne (2022) proposed RAGE loss (Retrieval Augmented Generation with Explicit Signals) for visual QA, which in our setting rewards the retriever with table-level signals from the reader model in joint training.
Third, we observe empirically that the reader does not always rank answers coming from the gold table at the top. As our reader is a sequence-tosequence model, we propose a simple modification to the training data: we prepend binary relevance tokens (‘yes/no’) to the answer itself. The reader learns to generate a first token indicating whether the table is relevant to the question or not.
Using these techniques, we build an end-to-end framework, LI-RAGE, and achieve state-of-theart results on two benchmarks for open-domain TableQA, NQ-TABLES (Herzig et al., 2021) and E2E-WTQ (Pan et al., 2021). 1

heading: 2 Related Work
text: While open-domain TableQA is yet a relatively unexplored problem, with only a few applications in the past couple of years, there has been extensive work on table retrieval and TableQA separately. In table retrieval, recent advances in ma-
1We make our code available at: https://github.com/ amazon-science/robust-tableqa
1557
chine learning have enabled extracting deep features for tables with Transformers (Vaswani et al., 2017), by designing models to parse complex tabular structure (Herzig et al., 2021; Wang et al., 2021), or by simply linearizing tables with interleaving tokens to preserve its structure (Pan et al., 2022; Wang et al., 2022). In TableQA, until recently researchers assumed gold tables were given and focused on developing models that understood and answered questions over tables, i.e. the readers. Earlier models generated commands in logical forms (e.g. SQL queries) that were executable over tables (Yu et al., 2018; Lin et al., 2019; Xu et al., 2018), while recent state-of-the-art models directly predict the answers from the input question and table by either classification (Herzig et al., 2020; Yang et al., 2022, TaPas) or autoregressive generation (Liu et al., 2022, TaPEx). Following these advances, in open-domain TableQA the best performing systems are based on a retriever-reader pipeline (Herzig et al., 2021; Pan et al., 2022). Herzig et al. (2021, DTR) leverages TaPas (Herzig et al., 2020) to both initialize a DPR-like retriever and the reader. T-RAG (Pan et al., 2022) uses DPR as retriever of rows/columns by decomposing the table and generates the answer via a sequence-tosequence reader (Lewis et al., 2020a), applying the RAG loss to refine the retriever with implicit signals during end-to-end TableQA fine-tuning. Unlike DTR and T-RAG, CLTR (Pan et al., 2021) employs only retrieval of rows and columns and obtains the answer cell by intersecting the top-scored ones. In this work we focus mainly on the retriever, and unlike previous work that relies on single vector embeddings, we leverage late interaction retrievers (Khattab and Zaharia, 2020) to achieve a finer-grained interaction between questions and tables. In contrast to T-RAG and CLTR, we do not need to decompose the table into rows and columns, but retrieve a whole table from the corpus, ensuring that the reader is given all the relevant information. In addition, we explore different techniques for explicitly refining the retriever during end-to-end TableQA achieving superior performance.

heading: 3 Methodology
text: Given a question q, the tasks are to find the gold table t∗ from a table corpus T, i.e. table retrieval (§ 3.1), and to derive the answer denotations S (1 or more cells from the table), i.e. question answering over the retrieved tables (§ 3.2). We assume
that labeled datasets consisting of triples {(q, S, t∗)} are available to us. We flatten the tables into sequences with interleaving special tokens that encode its structure (see Appendix A).

heading: 3.1 Table Retrieval
text: In order to exploit question-table similarity at a finer-grained level than when using DPR models, we leverage LI models to encode and retrieve tables for a question. We use ColBERT, which consists of a question encoder Fq and a table encoder Ft, to encode questions and tables at the token level:
Q = Fq(q) ∈ Rlq×d;T = Ft(t) ∈ Rlt×d, (1)
where lq and lt are input token lengths of q and t. The relevance score accounts for the interactions between all question and table token embeddings:
r(q, t) =
lq∑
i=1
lt max j=1 QiT ⊤ j (2)
LI models extract multi-dimensional question/table embeddings and token-level similarity, as opposed to finding the similarity of single embeddings for the whole question/table in DPR, thus capturing a finer-grained interaction between them.
To train the model we exploit the gold (positive) table t∗ for each question q, i.e. explicitly considering the table-level ground truth. We use in-batch negative sampling for training, per Karpukhin et al. (2020). All documents in a training batch other than t∗ are considered negative for q, and denoted as N(q). We train with the contrastive loss LCL:
− ∑
(q,t∗)
log exp (r(q, t∗))
exp (r(q, t∗)) + ∑
z∈N(q) exp (r(q, z))
(3)
To this end, for each q, the retriever outputs K top-scoring tables {tk}Kk=1. Finally, following RAG, we obtain their (approximate2) conditional probability pθ(·|q) with the retriever parameters θ:
pθ(tk|q) = exp(r(q, tk))∑K j=1 exp(r(q, tj))
(4)

heading: 3.2 Retrieval-based TableQA
text: For the TableQA task we make use of a sequenceto-sequence Transformer-based model that directly
2because we sum over the top-K tables instead of all tables, assuming their probabilities are small and irrelevant.
produces an answer for a given question and table. The TableQA model pϕ takes as input a sequence composed of the question q and each of the retrieved tables tk as described in §3.1, and generates an answer yk for each input table tk:
yk = argmax y
pϕ(y|q, tk) (5)
Finally, the model returns the answer associated with the highest probability/confidence:
ŷ, t̂ = argmax y,tk
pϕ(y|q, tk) (6)

heading: 3.3 Joint Training of Retrieval and TableQA
text: We train both modules jointly using a compositional loss (Lin and Byrne, 2022, RAGE), which considers signals from table relevance and answer prediction, as follows:
− ∑
(q,S)
( K∑
k=1
log pϕ(s ∗ k|q, tk) +
∑
k∈P+(q,S) log pθ(tk|q)
)
(7) where s∗k is a concatenation of all comma-separated answers in S and P+(q,S) = {k : yk = s∗k ∧ tk = t
∗} is a subset of the retrieved K tables, which contains those tables that satisfy (1) being a gold table relevant to answering the question; (2) the answer generator successfully produces the correct answer from that table. The core idea is to leverage the signal from model prediction to decide which tables are beneficial to producing the correct answer. Their scores are dynamically adjusted during training, which tailors the retriever to better serve the answer generation.

heading: 3.4 Learned Table Relevance
text: The answer generator is trained to produce s∗k for each input (q, tk) pair. Ideally, we would assume that the answer generated from the gold table t∗ is also associated with the highest probability from the answer generator. However, it might happen that an answer derived from a non-gold retrieved table may achieve higher confidence than the answer derived from a gold retrieved table. We propose a simple yet effective approach to improve this process: we add a binary relevance token preceding s∗k as ‘yes’ if tk = t
∗, ‘no’ otherwise. This design aims at guiding the model to prioritize reliable answer sources at training time. At generation time, if the leading generation of a (q, tk) pair is ‘yes’, we consider tk to be a more reliable answer source and
prioritize it over other input tables—that generate ‘no’ instead—when selecting the final prediction. We rely on the confidence scores if the leading token of all the candidates is ‘no’.

heading: 4 Experimental Setup
text: Datasets and metrics. We evaluate our system on two benchmarks, i.e. NQ-TABLES (Herzig et al., 2021) and E2E-WTQ (Pan et al., 2021).3 NQ-TABLES contains generally hard questions extracted from the NaturalQuestions (Kwiatkowski et al., 2019) dataset, comprising the questions that can be answered from tables rather than plain text. For this benchmark, we evaluate the models using: Token F1, i.e. token-wise F1 score; and exact match (EM) or accuracy, i.e. whether predictions match the annotations. E2E-WTQ contains look-up questions that require cell selection operation and is a subset of WikiTableQuestions (Pasupat and Liang, 2015). In E2E-WTQ train/valid/test splits are the same as in WikiTableQuestions, with questions limited to those that do not aggregations across multiple table cells. We evaluate models via accuracy4.
In addition, we report Recall@K for the retrieval performance in both, which measures whether the gold table is among the top-K retrieved tables.5
System configurations. For the table retrieval component, we conduct contrastive experiments using both DPR and LI. We first fine-tune the official pretrained DPR or ColBERTv2 model on each dataset before using them in the joint retrieverreader training. We do not train the TableQA model from scratch, instead we warm-start the training with TaPEx, a state-of-the-art pre-trained model for tabular data understanding based on BART (Lewis et al., 2020a). Since the E2E-WTQ is very small and not enough for learning a robust TableQA model, we additionally fine-tune TaPEx on its superset, i.e. WikiTableQuestions. Note that no test samples are leaked due to this as the dataset splits of E2E-WTQ are the same as WikiTableQuestions. We select the best checkpoints based on the validation set. We set K=5 since it shows the best balance between performance and latency by both RAG and RAGE. Training details, computational cost
3Dataset statistics are shown in Appendix B. 4Also named as Hit@1 in Pan et al. (2021, 2022) 5We do not report metrics such as P@K, N@K, MAP used by T-RAG and CLTR, which decompose tables, being incompatible with our setting (see Appendix C).
and software solution are provided in Appendix D.
Comparison systems. We compare with models from the literature, i.e. DTR, CLTR, T-RAG (see §2), and BM25—sparse retrieval baseline. Moreover, we build the following model variants: LI-RAGE: our main system that leverages ColBERT as retriever, TaPEx as answer generator, RAGE loss for joint training and the binary relevance token in output. We also ablate the system showing the effectiveness of each feature. When disabling joint training, i.e., for ablating the model, the retriever is not updated. DPR-RAGE: similar to LI-RAGE, except for the retriever being a DPR model. RAG: we train the RAG (Lewis et al., 2020b) in TableQA data, initializing the retriever and answer generator with our fine-tuned DPR and TaPEx, respectively. Different from DPR-RAGE, RAG does not produce the binary relevance token and updates the retriever only with the RAG loss, which is an implicit signal from the reader.

heading: 5 Results and Discussions

heading: 5.1 Main Results
text: As shown in Table 1, LI-RAGE achieves the best performance across the board on both datasets, with more than 3 points improvements in Token F1 and EM in NQ-TABLES, and 11.45 points in
E2E-WTQ with respect to previously best reported results in the literature. We attribute these results to the high performance of the LI retriever. On NQTABLES it obtains the best recall rate (87.90%) when only 5 tables are retrieved, as opposed to the previous models that achieve a lower recall rate with K = 10 tables, and also performs better when compared with RAG and DPR-RAGE, by a large margin. Effects of Joint Training. Similar to the observation of Lin and Byrne (2022), joint training with RAGE improves over the frozen system on both retrieval and TableQA performance. As shown in Table 1, joint training improves the end-to-end TableQA performance on both datasets by ∼0.6- 2.83%, and shows a superior retrieval ability especially on NQ-TABLES (85.21 to 87.90). Effects of Binary Relevance Tokens. As shown in Table 1, removing the binary relevance tokens greatly reduces system performance, by around 3.6% Token F1 and EM in NQ-TABLES and 8.4% in E2E-WTQ accuracy. Effects of LI. We report the retrieval performance in Table 2. LI-RAGE achieves the highest recall, outperforming BM25 in both datasets, and DPR by ∼3% on NQ-TABLES and by over 20-30% Recall@5/1 on E2E-WTQ. The large margin on E2E-WTQ is because it contains generally long tables with diverse information, and LI models prove
beneficial in learning richer table representations.

heading: 5.2 Remarks of Design Rationale
text: We tailor our solution for TableQA, with the specific design of two main components, i.e., adding a relevance token and modifying the RAGE loss. Relevance token. In open-domain QA, openended questions may have multiple correct answers and can be answered by different passages. As a result, increasing the number of retrieved passages (K) often improves the retrieval performance by enlarging the coverage of search. However, this is not the case for tables; in open-domain TableQA, the question often has only one gold table and most of the questions focus on a particular cell in the gold table. In our experiments, increasing K decreased the performance when K > 5 since presenting more tables to the answer generator only increases confusion and chance of mistakes (overconfident on some wrongly retrieved tables). When using relevance tokens as per our design, increasing K does not adversely impact the performance since irrelevant tables are dropped. In addition, we also explored alternative strategies that leverage retrieval scores to determine document reliability. The first strategy predicts the final answer from the table with the highest retrieval score. This setting achieves 41.04 EM on NQ-TABLES, which is even lower than our ablated LI-RAGE w/o joint training & relevance tokens attaining 42.19 EM (see Table 1). A second strategy weights predictions from different tables with the corresponding retrieval score, i.e., by multiplying the retrieval score (from the retriever) with the answer confidence (from the answer generator) when using K=5. This again performs poorer than our ablated LI-RAGE w/o joint training & relevance tokens that uses only answer generator confidence, achieving 40.91 EM on NQ-TABLES and 42.19 EM, respectively. In summary, relevance tokens work better than document retrieval scores or combination of retriever and reader scores. RAGE loss. We modify the original RAGE loss (Lin and Byrne, 2022) to adapt it to the domain of tables. In particular, we dropped the third term in the equation, which penalizes documents when they do not contain gold answers and also do not contribute to successful question-answering. Enabling this term in the loss, penalizes K − 1 documents in most cases, which leads to collapsed performance of the retriever in joint training for
TableQA. This is motivated by the same fact that gold tables are relatively sparse in TableQA and penalizing wrong documents leads to instability of training and quick retriever overfitting. Disabling this term instead, softens the RAGE loss by only awarding “good” tables and distinguishing good tables from bad ones, which improved the performance by around 1% EM on NQ-TABLES.

heading: 6 Conclusion
text: We introduce a novel open-domain TableQA framework, LI-RAGE, that leverages late interaction retrievers to enable finer-grained interaction between questions and tables. Additionally, LI-RAGE incorporates the RAGE loss and binary relevance tokens which enable significant improvements over the state-of-the-art in two challenging TableQA tasks.

heading: 7 Limitations
text: Our proposed system was tested on two opendomain TableQA datasets, with one of them (E2EWTQ) being relatively small compared to the other. Also, the current open-domain TableQA datasets are limited to look-up questions. They do not cover more complicated questions that involve multiple cells and complex table operations, such as SUM/MAX/MIN/SUBTRACT in some questions of WikiSQL and WikiTableQuestion. Therefore, the effectiveness of our system should be further evaluated on more complicated datasets of larger scale in the future. Another limitation lies in the token length limit of modern Transformer models. The best-achieving models typically accept up to 1024 tokens (e.g. BART, the base model of TaPEx). This limitation becomes more obvious when tables grow longer and the information being sought go beyond the limit. We believe that, with better approaches addressing this limitation, our system can achieve better performance. The solution can be either applying sampling strategies to pick the rows and columns that are most relevant to answering the question, or increasing the capacity of future Transformer models.

heading: A Table Linearization
text: In the retriever component, the input table is linearized into a sequence with separation tokens interleaving the table elements to make the input structure-aware, e.g. “<SOT> [table title] <EOT> <BOC> mountain peak <SOC> elevation <EOC> <BOR> red slate mountain <SOR> 13,162 ft <EOR> <BOR> ...”.
In the reader component, the TaPEx tokenizer linearizes the table with structure-aware separation, for example, “[HEAD] mountain peak | elevation [ROW] 1 : red slate mountain | 13 , 162 ft [ROW] 2 ...”.

heading: B Dataset Statistics

heading: C CLTR and T-RAG Evaluation
text: In these open-domain TableQA datasets, each question is associated with only one gold table. As a result, Precision@K in retrieval has a certain upper bound at 1K . Therefore, evaluating the retriever with Recall@K is more reasonable in this case.
We confirmed with the authors of CLTR and TRAG that they decomposed tables into single rows and columns to form the table database. In evaluating their systems on the E2E-WTQ dataset, the authors reported some retrieval metrics including Precision@K (P@K) which goes beyond the 1K limit (e.g. T-RAG achieved 0.7806 P@5). This is because they reported a hit for a retrieved row/column as long as it belongs to the gold table. With different setups for table corpus, the retrieval metrics of their systems are not directly comparable. Therefore, we compare Recall@K with BM25 and DPR only, and compare the end-to-end TableQA accuracy with CLTR and T-RAG (which is called Hit@1 in their papers).

heading: D Technical Details
text: D.1 Hyperparameters
The training hyperparameters are shown in Table 4, 5, and 7. The tuning of hyperparameters was performed on validation performance. DPR: The dimension of the extracted table/question embeddings is d = 768. LI: The dimension of the extracted table embeddings is lt × d = lt × 128, where lt depends on the length of input tables. Following Santhanam et al. (2022b), the dimension of the extracted question embeddings is fixed to lq × d = 32× 128. We pad the questions with less tokens than lq.
D.2 Indexing and Dynamic Retrieval
DPR. Following Lewis et al. (2020b), onedimensional table embeddings are pre-extracted with the DPR model that has been finetuned on the retrieval task. The FAISS system (Johnson et al., 2019) is used to index all table embeddings which enables fast nearest neighbour search with sub-linear time complexity. In training LI-RAGE, question embeddings are dynamically extracted from the retriever, and tables with highest scores
are retrieved using the precomputed index. LI. Khattab and Zaharia (2020) proposed the first version of ColBERT, and Santhanam et al. (2022b) introduced ColBERTv2, which is an enhanced version of ColBERT. Santhanam et al. (2022a) developed an efficient search engine, PLAID, for ColBERTv2, which significantly improved the retrieval latency. We redirect readers to the aforementioned papers for more details. We started from the official ColBERTv2 implementation 6and refactored the code base. We integrated ColBERTv2 into our training framework, so that fast and dynamic retrieval can be done during end-to-end joint training.
D.3 Computational Cost In Table 6 we report computational cost of the proposed models. It is clear that time spent on the training of LI is not significantly increased compared to DPR training. This is because both models use contrastive learning in training. But we note that the index building time of LI is around 5 mins while that of DPR only takes 40 seconds.
In terms of joint training, the end-to-end training time of LI-RAGE is longer. This is due to (1) slightly slower dynamic retrieval during end-toend training; (2) refining the retriever via larger multi-dimensional embeddings in comparison to one-dimensional embeddings used in DPR-RAGE. However, the inference speed is not affected much (from 1.22 sec/iteration to 1.40). This suggests that when deployed as real applications, LI-RAGE does not bring significant increase in computation.
6https://github.com/stanford-futuredata/ ColBERT
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 and 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3 and 4, and Appendix B and D
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. We do not distribute artifacts with this submission. Upon acceptance we will release code.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. We use public datasets from the literature
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix B
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix D
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: com/stanford-futuredata/ ColBERT
ACL 2023 Responsible NLP Checklist