SEER : A Knapsack approach to Exemplar Selection for In-Context HybridQA
Authors: Jonathan Tonglet
Manon Reusens
Philipp Borchert
Bart Baesens
Venue: EMNLP 2023 Main (2023)
Keywords: Hybrid Question Answering
In-Context Learning
Integer Linear Programming

Abstract: Question answering over hybrid contexts is a complex task, which requires the combination of information extracted from unstructured texts and structured tables in various ways. Recently, In-Context Learning demonstrated significant performance advances for reasoning tasks. In this paradigm, a large language model performs predictions based on a small set of supporting exemplars. The performance of In-Context Learning depends heavily on the selection procedure of the supporting exemplars, particularly in the case of HybridQA, where considering the diversity of reasoning chains and the large size of the hybrid contexts becomes crucial. In this work, we present Selection of ExEmplars for hybrid Reasoning (SEER), a novel method for selecting a set of exemplars that is both representative and diverse. The key novelty of SEER is that it formulates exemplar selection as a Knapsack Integer Linear Program. The Knapsack framework provides the flexibility to incorporate diversity constraints that prioritize exemplars with desirable attributes, and capacity constraints that ensure that the prompt size respects the provided capacity budgets. The effectiveness of SEER is demonstrated on FinQA and TAT-QA, two real-world benchmarks for HybridQA, where it outperforms previous exemplar selection methods.

SEER: A Knapsack approach to Exemplar Selection for In-Context HybridQA
Jonathan Tonglet Manon Reusens", Philipp Borchert Bart Baesens
'Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AD), TU Darmstadt
Research Centre for Information Systems Engineering (LIRIS),
Faculty of Economics and Business, KU Leuven IESEG School of Management, 3 Rue de la Digue, 59000 Lille, France *Department of Decision Analytics and Risk, University of Southampton www.ukp.tu-dan tuddaansaddatsdeede
Abstract
the potential to automate business processes and enhance accessibility to the information they con- tain. Several benchmarks anchored in the financial domain have been introduced for hybrid question answering (HybridQA) (Chen et al., 2021c; Zhu et al., 2021; Zhao al., 2022). Figure 1 presents an example from the FinQA dataset. Despite ongo- ing progress, current HybridQA models have yet to achieve human expert performance (Zhu et al., 2021; Chen et al., 2021c).
Question answering over hybrid contexts is complex task, which requires the combina- tion of information extracted from unstructured texts and structured tables in various ways. Re- cently, In-Context Learning demonstrated sig- nificant performance advances for reasoning tasks. In this paradigm, large language model performs predictions based small set of supporting exemplars. The performance of In-Context Leamingg depends heavily on the selection procedure of the supporting exem- plars, particularly in the case of HybridQA, where considering the diversity of reasoning chains and the large size of the hybrid con- texts becomes crucial. this work, we present Selection of ExEmplars for hybrid Reasoning (SEER), novel method for selecting a set of exemplars that both representative and di- verse. The key novelty of SEER is that it for- mulates exemplar selection as a Knapsack Inte- ger Linear Program. The Knapsack framework provides the flexibility to incorporate diversity constraints that prioritize exemplars with desir- able attributes, and capacity constraints that sure that the prompt size respects the provided capacity budgets. The effectiveness SEER is demonstrated on FinQA and TAT-QA, two real-world benchmarks for HybridQA, where outperforms previous exemplar selection meth- ods!.
Recently, In-Context Learning (ICL) with Large Language Models (LLMs) has shown great perfor- mance on reasoning tasks. In this setting, set of exemplars, i.c. training instances with their an- swers, are provided as part of the input prompt assist LLMs in generating the correct answer. ICL an inference time technique that keeps the LLMs parameters frozen (Brown et al., 2020). The per- formance of ICL depends heavily on the quality of the provided exemplar set (Liu et al., 2022a). Various strategies, ranging from random selection to similarity- based retrieval, have been proposed to tackle this problem (Liu al., 2022b; Rubin et al., 2022; Li and Qiu, 2023; Lu et al., 2023). When selecting exemplars for HybridQA, special consid- eration must be given to the unique challenges of the task, including diverse reasoning chains, large context sizes (Chen et al., 2021c: Zhao et al., 2022), and the limited correlation between the question and its reasoning chain.
1 Introduction
In this work, we propose Knapsack Programs framework model exemplar selection for ICL. Knapsacks a family of Integer Linear Programs that search an optimal subset of items under linear constraints (Wolsey, 2020). For a given test stance, a Knapsack Program is solved to obtain the optimal exemplar set. This expressive framework
Hybrid documents, which combine tables and text paragraphs, are prevalent in various industries such finance, healthcare, and manufacturing. The development of question-answering systems capa- ble effectively handling these documents holds
Code availahle at github.com/jongglz/EEER

---

2013
2014
2015
2016
2017
Question what anualized return
s&p 500 from 2012 2017?
Cme
$
$
$
$
group
164.01
194.06
208.95
279.85
370.32
Answer:
inc.
Subtract(208.14. 100); Divide(HO, 100); Divide(1,5H Exp(#1,42); Subtrac((H3.1)
S&P 500
132.39
150.51
152.59
170.84
208.14
Python cade:
Peer
176.61
187.48
219.99
249.31
323.23
table conle_gurry__e 0 208.14
group
text v_riabrrabeeo0 = 100
[...] investment of 100 (with reinvestment of all divideuds assumed to have been made in class common stock, , [...] s&p 500 peer group 100 invested on 12/31/12 stock or index, [...]
step_ table _querr_rr____ text variable_ step stap table cable_query_0
step 1/5
step step step__
ans step_3-1
Figure I: An instance of the FinQA dataset (Chen al., 2021c). Text snippets of the paragraphs shown in the dashed box.
allows balancing the diversity and similarity of the selected exemplars while controlling the prompt size with user-defined linear constraints. We intro- duce SEER, novel method to select exemplars for HybridQA using Knapsack Programs. SEER reduces the candidate set with a nearest neighbor fil- tering, and leverages constraint modules to predict the attributes of the test instance. The attributes of HybridQA instance properties that influence the underlying reasoning chain, e.g.. the modal- ity (table, text, hybrid) and the answer type (span extraction, arithmetic reasoning, counting). By leveraging constraint modules, SEER shapes the Knapsack structure to prioritize the selection of exemplars that share similar attributes with the test instance.
(Zhao al., 2021; Lu al., 2022). Hence, several approaches have been explored for exemplar selec- tion, including nearest neighbor search (Liu et al., 2022a), reinforcement learning (Zhang et al., 2022; Lu et al., 2023), clustering (Zhang et al., 2023), search algorithms (Li and Qiu, 2023), and super- vised learning (Rubin et al., 2022: Ye et al., 2023), Rubin et al. (2022) consider token capacity indi- rectly by constructing the largest possible prompt with the selected exemplars. In contrast to previ- ous methods, we consider exemplar selection as an ILP Knapsack Program, which allows us to specify diversity-enhancing constraints, directly optimize the token capacity without simple heuristics, and leverage powerful solvers to find a performant ex- emplar selection.
The contributions of this work are follows: (1) we introduce Knapsack Programs framework for ICL exemplar selection. (2) We propose SEER, a novel exemplar selection method for In-Context HybridQA. (3) We address three challenges of HybridQA exemplar selection at the same time with fine-grained token budget constraints and attribute-guider selection. Extensive evaluation of two real-world HybridQA benchmarks shows that SEER outperforms state-of-the-art exemplar selection methods, especially under restricted token capacity budgets.
Hybrid Question Answering Chen et al. (2020, 2021b) introduced the task of HybridQA on open- domain Wikipedia pages. Later on, datasets based on real-world financial documents were introduced (Zhu et al., 2021; Chen et al., 2021c; Zhao et al., 2022; Chen et al., 2022b). Previous work has fo- cused on improving the retriever- reniever-generrtor frame- work (Lei et al., 2022: Sun et al., 2022; Zhang and Moshfeghi, 2022). Chen et al. (2022a) use few-shot chain-of-thoughe to solve the task of HybridQA. However, they focus on improving the prompt for- mat, while our work focuses on selecting good exemplars.
2 Related work
Integer Linear Programming for NLP Integer Linear Programming has been used in many NLP tasks (Martins, 2014), including coreference reso- lution (Denis and Baldridge, 2007; De Belder and Moens, 2012), sentence compression (De Belder and Moens, 2010), dependency parsing (Riedel and Clarke, 2006), semantic role labeling (Roth and
In-Context Learning LLMs have shown the abil- ity to perform wide range of tasks with only few exemplars provided as prompt while keep- ing all the model parameters frozen (Brown al., 2020). However, the performance is highly de- pendent on the quality of the provided exemplars

---

Yih, 2005), and translation (Germann al., 2004). In this work, we introduce novel application of ILP NLP: exemplar selection for ICL. Further- more, it is, to the best of our knowledge, the first time that the Knapsack family of ILP programs is used in NLP.
Question
Program
Answer
Modality
Answer type
Whot the percentage lun dlrare 2019 20187
[--10%'1
uabe
What was the percentage change cost of vaftmare under develogment between 2015 2019?
(16,254
6,509)
190.8
table
arttimetic
6,500
3 Methodology
On as avenaee loos rann geople are ruglored LLD feeii in 20/W?
['0000]]
lest
*pen
3.1 Integer Linear Programming
Whe is the soorage rumber Admunattaivv Statl employed?
(790 1884))33) 3
tahe
aeiltmutic
Linear Programming (LP) involves maximizing or minimizing an objective function while adher- ing to set of constraints. The objective func- tion consists of weighted linear combination of variables, while the constraints (in)equalities that involve linear combinations of these variables. These constraints serve to restrict the value ranges the variables and capture their interaction effects (De Belder and Moens, 2010). Integer Linear Pro- gramming (ILP) is a subset of LP, wherein variables are constrained to take only integer values. ILP is divided into program families, one of which is the Knapsack. Given set of items, the Knapsack's objective is to select the subset that maximizes the total value while remaining within the maximum capacity (Wolsey, 2020). More formally, the problem can be expressed as:
Figure 2: Four problem instances from TAT-QA. Similar questions do not always share similar problem attributes.
Q. a hybrid context consisting of text paragraphs P and a table T. and set of n exemplars E = [eI, Gn) where is tuple (QT..,,PPTT))
A argmar, (a P,P,T,
Prior studies (Liu al., 2022a; Lu al., 2023) have demonstrated that the thoughtful selection of the exemplar set can improve and stabilize the performance of ICL over a random selection baseline. However, selecting the optimal exemplar set poses three challenges for HybridQA problems. First, there is a high diversity in the type of ques- tions and in the approaches to solve them. The financial dataset FinQA for example contains more than 300 different numerical formulas. For ques- tions asking to compute a "percentage change", the training set counts unique formulas. Given this diversity, it is not possible to cover all possible rea- soning chains with a single set of exemplars. This challenge is partially addressed by similarity-based exemplar selection methods (Liu et al., 2022a). Secondly, Figure 2 shows the additional challenge of the low correlation between the problem's 's ques- tion and attributes. This might result in prediction errors, these problems seem semantically simi- lar, however, they require different modalities and answer types.
maximize
Σ NE
subject to
Σ iCS
4 34 S
iES
2; E {0,1}
iES
where 2; is variable that takes value 1 when item i in set S is selected, 0 otherwise. Wi and C parameters representing the value and cost of item i, respectively. C is the maximum capacity of the Knapsack. Depending on the setting, ad- ditional constraints and variables can added to the basic Knapsack program template shown above. The Knapsack is NP-hard, but several algorithms have been developed to find an optimal solution and Cut- ting Planes (De Belder and Moens. 2010). Several solvers provide efficient implementations of these algorithms (Martins, 2014).
efficiently, including Branch-and-
Thirdly, HybridQA problems, especially when deal- ing with real-world data like financial documents, involve large contexts. LLMs have a limit to the number of input and output text tokens they process, whether it due to organizational resource constraints the inherent limitation of the LLM. Consequently, it becomes crucial ensure that to- kenized exemplars fit within the LLM's capacity while reserving enough tokens for generating the
3.2 Challenges of Exemplar Selection for HybridQA
When solving HybridQA problems with ICL, the task is to predict an answer A given a question

---

desired output text. In the following, we propose a new exemplar selection method that addresses those three challenges by modelling them as ob- jectives and constraints of a Knapsack program. Notably, this work addresses explicitly the latter two challenges for the first time, contributing to the advancement of exemplar selection in this domain.
the original training set, composed of the k nearest neighbor of the test instance's. s. The Knapsack has a double capacity constraint. Firstly, M is the max- imum allowed number of exemplars. Secondly, L is the maximum combined length of the exemplars 4 in number of tokens. The value of L depends on the backbone LLM.
3.3 SEER
maximize
Σ ic$
Wi Zi
SEER generates Knapsack programs for exemplar selection in HybridQA. Given the training set and test instance, SEER constructs a unique Knapsack program using nearest neighbor filtering and con- straint modules. The selected exemplar set is the optimal solution to the Knapsack program. These exemplars, along with the test instance, are pro- vided as prompts to an LLM for predicting the final answer. Figure provides an overview of SEER's methodology.
subject to
1j M
4 24 S L
table; T; 2 M
other; x, 22 B M
24 C (0, other; = tezt, + hybrid;
i S
Similarity computation involves calculating co- sine similarity between pairs of HybridQA prob- lems' questions in the embedding space. The re- sulting similarity values serve as coefficients in the objective function. To ensure accurate compar- isons, preprocessing is applied to remove noise. (1) All questions are lowercased. (2) All punctuation is removed. (3) Dates, numerical values, and companies are replaced by their NER tags. Nearest Neighbor filtering involves applying an initial filter (Liu et al., 2022a) to identify the k can- didates from the training set that exhibit substantial surface similarity with the test instance, thus nar- rowing down the search space.
where
The program contains two or more diversity con- straints. The structure of those constraints depends on the attributes predicted by the constraint mod- ules. All possible diversity constraint configura- tions are presented in Appendix B. Parameter ranges from 0 to and controls the promotion of exemplars possessing the predicted attribute, i.e. "table" in our case. Parameter /3, also in the range [0,1], controls the promotion of the other attributes. holds that a+ I 1. tab, text,, and hybrid, are binary parameters with value when the modal- ity of candidate is the table, the text, or both. respectively, and 0 otherwise. The objective is to maximize the sum of the exemplar's similarity with the test instance w;, while respecting all constraints. Framing exemplar selection as a Knapsack program has several benefits. solvable with efficient de- terministic algorithms. gives full control to the end user over the prompt capacity in tokens and explicitly allows the user to balance the similarity and diversity of the selected exemplars in terms of the test instance's attributes. SEER's pseudo-code is shown in Appendix A.
locations,
Constraint modules predict an attribute of the test instance. We define an attribute as characteris- tic of the reasoning chain of HybridQA problem. Attributes include modality, answer type, and num- ber of reasoning steps. Inferring these attributes involves a standard classification task, where the question and hybrid context are provided as input, and the output corresponds to one of the attribute values. The task can be addressed through fine- tuning or ICL.
A SEER Knapsack is uniquely defined for test in- stance by the combination of the similarity weights and the predicted attributes. As an illustration, the base template for Knapsack programs with pre- dicted attribute "modality:table" is formulated as follows, where variable z; takes value if instance from the candidate set S is selected as an exem- plar, otherwise. The candidate set S is subset of
3,4 LLM Code Generation
The ICL exemplars selected by SEER and the test instance are concatenated and provided a prompt to an LLM. To remove irrelevant text paragraphs from the hybrid context relation to the ques- tion, we employ a pre-trained text retriever. Recent

---

SEER
Prediction
Candidate Exemplars
Python Engiie
KNN Selection
Program
Python
Train set Emneecdings
X Embeddings
Large Langunge Model
Constraiot modules
Retriever
Embodding model
Few Shot lixemplars
Train set
iest
Xyest
Xgest
Figure 3: Overview of SEER's architecture to select the optimal exemplar set for HybridQA problem.
studies have demonstrated the benefits formulat- ing answer derivation as Python code (Chen al., 2022a; Gao et al., 2023; Mishra et al., 2022). In- spired by these approaches, we adopt a code gener- ation formulation instead of text generation. The resulting code is executed using an external engine to derive the answer. Figure depicts the conver- sion from the original text answer to Python code answer.
Equations are written as one-line variable assign- ment. A prompt example is provided in Appendix C. Unlike FinQA which only contains arithmetic problems, TAT-QA includes other answer types such as (multí-)span extraction and counting prob- lems. Consequently, TAT-QA has two constraint modules, one for modality and one for the answer type.
The evaluation of the datasets is based on their spective metries: exccution accuracy (EA) and pro- gram accuracy (PA) (Chen et al., 2021c) for FinQA, and Exact Match (EM) and numeracy-focused FI (Dua et al., 2019) for TAT-QA. EA and EM assess the correctness of the final answer, while PA en- sures that the generated code is mathematically equivalent to the ground truth derivation. The numeracy-focused F1 (Dua et al., 2019) computes the Fl score over the bag-of-wordd representation of the generated answer. For TAT-QA, we exclude the sub-task scale prediction and reserve it for future work multi-task ICL HybridQA. The LLM has little exposure to syntactic conventions unrelated to correct reasoning, such as determining the ap- propriate scale to represent percentage ([0,1) or [0,100]). To take this into account, allow some flexibility in the answer evaluation script. For fur- ther details on the evaluation procedure, refer Appendix
4 Experiments
4.1 Datasets
We evaluate SEER on two HybridQA datasets with real-world contexts: FinQA (Chen et al., 2021c) and TAT-QA (Zhu et al., 2021), both anchored in the financial domain.
FinQA comprises 8,281 problems, each contain- ing a context with table and multiple text para- graphs. The answers in FinQA are expressed in a dopain-specific language as a sequence of opera- tors with two operands each. These expressions are then translated to Python code using an automated script, with each operation a separate line. Ta- bles linearized with " "I" " as column delimiter and 'nn as row delimiters. A prompt example is provided in Figure 1. This dataset includes constraint module for predicting the modality.
TAT-QA consists of 16,552 problems. Since the ground truth answers for the test set of TAT-QA are not publicly available, we employ the original dev set as the test set. To compensate, we create a new dev set of equal size by splitting the training set. The context consists of text paragraphs and table. Some tables flattened, while others exhibit complex structures with multiple header levels. The original answers are written as text or equations. We convert text to Python comments.
4.2 Baselines
We compare SEER with other exemplar selection strategies for ICL. The Random baseline randomly selects a of exemplars from the training for each test instance. We define CSP (Constraint Sat- isfaction Problem) as SEER without the objective function. candidate set is randomly selected among those that meet all the Knapsack's 's con-

---

straints. The Fixed set baseline employs the same set of exemplars for every test instance. Specifi- cally, we randomly sample 20 groups of 4 and exemplars from the training set for FinQA and TAT- QA, respectively. The group exhibiting the highest performance on the dev sets is selected. KATE (Liu et al., 2022a) selects the k nearest neighbors to the test instance. It corresponds to SEER without token capacity and diversity constraints. Diverse KATE, an extension of KATE, divides the training set into two subsets, one for text problems and one for table problems. An equal number of nearest neighbors is selected from each subset. This en- sures that both modalities are present in the exem- plar set. PromptPG (Lu et al., 2023) is reinforce- ment learning method that trains a policy network to select well pell-oerforinn exemplars among fixed candidate pool. At inference time, the policy de- cides which exemplars from the candidate set are selected for a given test instance, For training, we use the same parameters as Lu et al. (2023), except the size of the candidate exemplar set which is set to 20 and 40 for FinQA and TAT-QA respectively. To disentangle the exemplar selection performance from errors related to constraint modules, we also include results obtained using ground truth problem attributes, denoted SEER gold, which serves as an upper bound estimate for SEER's performance. Although our primary focus is on ICL, numerous studies have explored line-tuning approaches for HybridQA. We report the results of the SOTA fine- tuning approach. For a comparison of fine-tuning approaches with SEER, refer to Appendix F.
The ICL prompt starts by the text paragraphs, the linearized table and the problem's 's question, in that order. Then, a task-specific instruction provided at the end of the prompt. Instruction for modal- ity prediction: "Do you need data from the table, the text paragraphs, or both (hybrid) to answer this question? Answer by one of the following: table, text, hybrid.". Instruction for answer type predic- tion: "Does this question require extract spans from the document, to count, or perform an arith- metic reasoning? Answer by one of the following: span, multi-span, count, arithmetic.", We set the maximum token length to the maximum capacity of the LLM minus the token lengths of the prob- lem's s context and the longest answer in the training set. This conservative estimate of L ensures that the exemplars, the problem's context, and its an- swer can all fit within the imposed token limit. We use the GUROBI solver² find the optimal Knap- sack solution. The detailed parameter values are listed in Appendix D. Our experiments are run on a single NVIDIA GeForce RTX 3050 GPU, except for CODEX inferences, which are performed on the OpenAl servers.
4.4 Main results
Table I shows the main results, showcasing the superior performance of SEER compared to all exemplar selection baselines. Notably, SEER achieves better results than SEER by a margin of 0.4 to 1.5%, indicating that accurate attribute pre- diction by the constraint modules significantly con- tributes to the overall performance. The difference between KATE and SEER on TAT-QA is not signif- icant. However, SEERRoold is significantly different from KATE, showing that SEER ultimately outper- forms KATE with better constraint modules. For the detailed significance test of the main results, please refer to Appendix G. The lower performance of random selection reaffirms the necessity of care- ful exemplar selection. Despite its relatively low performance, CSP demonstrates the notable im- provements achievable by introducing constraints into the random selection process, surpassing the outcomes of a purely random approach. Interest- ingly, Diverse KATE marginally stronger than KATE on FinQA but 7% lower in EM on TAT-QA. While the SOTA fine-tuning models outperform ICL methods, SEER contributes to closing the gap. The average run time distribution is as follows:
4.3 Implementation details
We use CODEX, a 175 Billion parameters LLM available through the OpenAl API2 (Chen et al., 2021a), as a backbone for SEER and all base- lines. CODEX has demonstrated exceptional per- formance in code generation tasks (Chen et al., 2021a). For the computation of question and text paragraph embeddings, we employ Sentence- BERT models (Reimers and Gurevych, 2019). Re- garding the constraint modules, we explore two strategies. Firstly, fine-tuning BERT model on the training set. Secondly, employing ICL predictions with one exemplar per attribute value, leveraging CODEX as the underlying model. The ICL strat- egy offers the advantage of requiring only small number of training instances with attribute labels.

---

FinQA
TAT-QA
Method
EA
PA
EM
F1
PromptPG (Lu al., 2023)
53.56 + 3e-3
24.09 + le-3
51.64 + 0.27
58.86 0.27
Random
55.65 3.35
29.5
49.7
57.31 4.21
CSP
60.77 0.14
43.62
57.47
6..21
Fixed set
64.05 + 0.22
38.15
66.55
73.8
KATE (Liu et al., 2022a)
67.07 + 0.04
58.65 + 0.04
68.9
75.77 + 0.08
Diverse KATE
67.31 + 0.24
59.54 + 0.19
61.53 + 0.23
68.89 + 0.21
SEER
68.85 + 0.04
59.78 + 0.15
69.68 + 0.07
76.71 = 0.07
69.25 + 0.11
60.16 0.14
71.32 + 0.07
78.12 + 0.08
SOTA fine-tuned model
71.07
68.94
73.6
81.3
Table Table main results (%). EA indicates Execution Accuracy, the Program Accuracy, EM Exact Match, and FI the numeracy-focuser F1 score. Reported results averages over three iterations the test set. CODEX is the backbone LLM for ICL methods.
solving the Knapsack with Gurobi takes 0.2 sec- onds, predicting an attribute requires 0.05 seconds with BERT and 2.95 seconds with CODEX, and predicting the answer with CODEX takes 2.89 sec- onds, on average.
SEER
Excecutinn Accuracy
KATE
Fixed set
-50
Figure 4 presents a comparison between SEER and two strong baselines across four subsets of FinQA test instances. The set of instances with attributes correctly predicted by the constraint module, called Correct Attribute Predicted (CAP), and its comple- ment, Incorrect Attribute Predicted (IAP). The of instances with least one selected exemplar hav- ing the correct attribute, called Correct Attribute Selected (CAS), and its complement, Incorrect At- tribute Selected (LAS), It holds that CAP C CAS, IAS C TAP, and CAS n TAP + Ø. We conclude that SEER's hedge over baselines is the direct con- sequence of the prediction and selection of correct attributes. Interestingly, the baselines struggle with instances where incorrect attributes are predicted and selected, indicating a correlation between the difficulty of HybridQA instance and the chal- lenge predicting its attributes. Where no correct attribute is selected (IAS), the Fixed set baseline outperforms SEER and KATE by a slight margin.
40
CAP
CAS
IAP
TAS
Figure 4: EA (%) four subsets of the FinQA test instances. Results are averages over three iterations. CAP stands for Correct Attribute Predicted, CAS for Correct Attribute Selected, IAP for Incorrect Attribute Predicted, IAS for Incorrect Attribute Selected.
constraint modules, shown by the low precision of the "hybrid" modality in the confusion matrices in Figure 5. As result, we decided treat the "hy- brid" modality as an "uncertain" modality attribute. Hence, the diversity constraints the "hybrid" set- ting promote the usage of all three modalities.
Analysis
How sensitive is SEER to variations in con- straint parameters a and !? We analyze SEER's and SEER sensitivity to different (a,,5) value pairs. By increasing a, we encourage the selection of exemplars that share the predicted attributes of the test instance. Conversely, increasing & promotes the inclusion of exemplars with different attributes, thus ensuring diversity and mitigating errors introduced by the constraint modules. We evaluate the EA and EM on the dev set for the pairs (50,25), (75,0), (75,25), (100,0). The results, depicted Figure 6, indicate that the difference in EA and EM between the lowest and
4.5 Constraint modules results
Table 2 shows the performance of fine-tuned and ICL constraint modules on the dev and test sets. On two out of the three attribute prediction tasks, the fine-tuned BERT module performs best. How- ever, the ICL constraint module's performance is always close, making it viable alternative when attribute labels are not available in the training set. There is still room to improve the performance of

---

FinQA Modality
TAT-QA
Modality
Answer type
Dev
Test
Dev
Test
Dev
Test
Fine-TunedrEar 63.98
58.93
56.89
55.28
95.69
91.79
ICLCODEX
59.9
58.15
63.19
62.17
87.23
87.23
Table Accuracy (%) constraint modules on the dev and test sets.
Modality HieQA
Modality TAF-2A
Answer type
ubll
386
30
2699
uble
497
19
255
400
mo-gen spm
6.58
22
600
Thue label
3000
2.0
194
400
19
211
200
loxt
2.53
124
200
57
649
Infond
byloid
200
100
2
30
-0
sable
bybid
takle
tcM
bybrid label
zrab
count
Figure 5: Confusion matrices of the best constraint modules on the test sets.
SEER
restrictive to even fit most TAT-QA test instances. Hence, only consider FinQA for that setting. Our experiment involves varying values of M. representing the maximum number of exemplars to included in the selection. We compare SEER with KATE, the strongest baseline, which disregards token capacity budgets during the search for the most similar exemplars. Figures 7 and 8 report the coverage, i.e. the number of prompts that respect the token budget, and the performance of SEER and KATE. SEER consistently outperforms KATE across all capacity budgets, showcasing a margin of improvement of up to 26% for the more restrictive budgets. While imposing budget restrictions inevitably results in reduced performance, SEER demonstrates a superior ability to mitigate the negative impact compared to KATE Those results could further be improved with a better estimate of the maximum token capacity L. The current estimate is very conservative, as saves enough space generate the longest solution of the training sets, 309 and 800 tokens for FinQA and TAT-QA, respectively. Most real-world solutions will require fewer tokens than this upper bound.
Exsostion Assuney
Doost Moxt
09.2
2535
1525 (alphabbcaa) TAT-QA
(slptabeca)) FaiQA
Figure 6: EM and EA (%) for different pairs of (a/3) values. Results averaged over 5 iterations the dev set.
highest performing configurations is less than 1% Consequently, we conclude that SEER does not require extensive tuning of these two parameters to achieve satisfactory performance. SEER performs best when a is set to higher values and 3 is set to 0. As SEER gold leverages the ground truth attribute, there is need to mitigate attribute errors with B. This finding aligns with intuition that guiding exemplar selection based on problem attributes improves performance.
How does SEER perform under different token capacity budgets?
To evaluate the benefits of the double capacity constraint, we evaluate SEER under different token capacity budgets. While the default token capacity of CODEX is 4096, multiple reasons might lead reduction in that value, including using another LLM or financial and time constraints. the following, we consider capacities of 2048 and 1024 tokens. The 1024 token budget is too
Qualitative Error analysis
We sample 100 incorrect instances from the test sets at random and categorize them manually in different error categories, summarized in Table 3. On TAT-QA, incorrect values occur even when the exemplars use the required formula. Hence, it seems the error results from the limitations of

---

Coverage SEER
100
Score SEER
FinQA I
Cooerrug
Percentage
73
KATE
23
FinQA
TAT-QA
TAT-QA
Figure 7: Performance of SEER under a 2048 token capacity budget, as evaluated on the dev set. The coverage is the percentage of exemplar set respecting the budget. The score is the EA (%) for FinQA and EM (%) for TAT-QA.
Conclusion
Coverage
This paper investigates the problem of exemplar selection for ICL in HybridQA tasks. We propose ILP as a framework for exemplar selection and in- troduce SEER, novel method based on Knapsack programs. While existing methods only focus on the high diversity of questions and how to solve them, SEER explicitly tackles two other key chal- lenges of HybridQA exemplar selection in the form of integer linear constraints. Specifically, SEER focuses on the following challenges, the low cor- relation between the problem's 's questions and at- tributes on one hand and the large context required to solve these problems through diversity and ca- pacity constraints. Diversity constraints enhance performance by selecting exemplars that share the same attributes as the test instance. Capacity straints provide fine-grained control to the end-user over the token budget allocated the prompt, en- suring sufficient tokens to generate the desired out- put. This level of control beneficial overcom- ing the limitations of the backbone LLM and deal- ing with financial constraints. Evaluation on two financial HybridQA datasets shows that SEER out- performs previous ICL exemplar selection methods, especially under limited token capacity.
Percentage
Covergge KATE
20
FinQA M-2
FinQA M-4
Figure 8: Performance of SEER under 1024 token capacity budget, as evaluated on the dev set of FinQA.
Error category
FinQA
TAT-QA
Values error
76%
63 %
Operators error
45%
30 %
Ground truth error
10 %
0 %
Answer formatting
0 %
10 %
Execution error
4 %
0 %
Table 3: Percentage of errors per category.
CODEX. This is particularly true for Counting problems, where 4 or relevant exemplars provided, but CODEX fails make the correct count for the test instance. 25% of the FinQA value errors result from a poor computation of similarity with the candidate exemplars, based on semantically superficial elements. Furthermore, we observed that 17% of the FinQA operator errors are due to one missing reasoning step, e.g. the addition of constant term. The analysis highlights directions for future work. (1) SEER could benefit from fine-tuning methods compute the similarity weights (Rubin al., 2022). (2) SEER can be augmented with mechanisms that predict and control the required number reasoning steps.
For future research, we plan explore additional attributes and seek ways to enhance the overall per- formance of the constraint modules. Leveraging the flexibility of the Knapsack framework, we intend to study the potential of incorporating constraints defined by end users, such as domain experts, to fur- ther refine exemplar selection. Additionally, SEER can be extended beyond the HybridQA setting encompass other modalities like images and knowl- edge graphs, well other tasks, such as hybrid fact-checking (Aly et al., 2021), data-to-text gen- cration (Parikh et al., 2020), and multimodal fraud detection (Wang et al,, 2023).

---

Limitations
Acknowledgements
We identify two main limitations to the method introduced in this paper.
This research has been funded by the Statistics Flanders research cooperation agreement on Data Science for Official Statisties and supported by the LOEWE initiative (Hesse, Germany) within the emergenCrTY center. We express our gratitude to the Policy Research unit at OpenAI for provid- ing access to the language model CODEX via the OpenAl Research Access Program.
Firstly, we assumed that we could select ICL exem- plars from thousands of training instances with an- notated textual reasoning. However, in real-world scenarios, obtaining these textual annotations is time-consuming process that involves manual la- beling. In a recent study by Su et al. (2023), a method called vote-k was proposed to address this challenge. It involves selecting candidates to anno- tate from large unlabeled pool before performing similarity-based exemplar selection. By incorporat- ing vote-k into SEER, a fully annotated training set.
References
Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thome, Andreas Vlachos, Christos Christodoulopou- los, Oana Cocarascu, and Arpit Mittal. 2021. Fever- unstruc-
reduce the reliance
Secondly, while efficient solvers exist for finding the optimal solution to SEER's Knapsack program, it remains an NP-hard problem that can take expo- nential time to complete in the worst case. To mit- igate this issue, we impose a computation budget by setting a timeout of 5 seconds for the solver's execution. We empirically observed that SEER convergence the global optimum is slower in cases where many candidates have high similar- ity weight with the test instance.
Fact extraction and verification tured and structured information. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei, 2020. Language models few-shot learners. In Ad- vances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Ethics Statement
Intended Use HybridQA demonstrates extensive applicability in real-world scenarios, particularly in the fields of finance and education. In finance, it can reduce the time needed for financial analysts and journalists to process large volumes of financial reports. Additionally, it enhances accessibility to rinance-reltte information for a wider audience. In the education sector, it assists students in solving textbook exercises that involve hybrid context. Misuse potential While we have not identified any direct potential for misuse of SEER in particular, it is crucial acknowledge some general concerns with using automatic question answering systems and LLMs. Automatic question answering systems are not infallible. Blindly relying on the outputs generated by the model without double-checking the results can have adverse consequences for end- users, particularly in financial investment analysis. We recommend careful and critical usage of the content generated by LLMs, both with and without SEER. To mitigate this risk, we see potential in research on verifiers for program generation (Ni et al., 2023) and on new explainability methods for LLMs.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021a. Evaluating large lan- guage models trained on code. arXiv preprint arXiv:2107.08: 107.03374.
Wenhu Chen, Ming- wei Chang, Eva Schlinger, William Wang, and William Cohen. 2021b. Open question answering over tables and text. International Confer- ence Learning Representations (ICLR).
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022a. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks. arXiv preprint 2rXiv:2211.12588
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020. Hy- bridQA: A dataset multi-hop question answering over tabular and textual data. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pages 1026- 1036, Online, Association for Computa- tional Linguistics.

---

Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, lana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang. Bryan Routledge, and William Yang Wang. 2021c. FinQA: A dataset of nu- merical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Nut- ural Language Processing, pages 3697-371 1, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
encoder and tree-based decoder. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1379-1390, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Chenying Li, Wenbo Ye, and Yilun Zhao. 2022. Fin- Math: Injecting a tree-structured solver for question answering over financial reports. In Proceedings of the Thirleenth Language Rexources and Evaluation Conference, pages 6147-6152, Marseille, France. Fu- ropean Language Resources Association.
Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma. Sameena Shah, and William Yang Wang. 2022b. ConvFinQA: Exploring the chain of numerical rea- soning in conversational finance question answering. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 6279-6292, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Xiaonan Li and Xipeng Qiu. 2023, Finding support- ing examples for in-context learning. arXiv preprint 1:Xii:2302.13532
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022a. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Iniegration for Deep Learning Architectures. pages 100-1 14, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Jan De Belder and Franc-Fe Moens. 2010. In- teger linear programming for dutch sentence com- pression. In Proceedings of the 11th International Conference on Computational Linguistics and Intel- ligent Text Processing. CICLing' 10, page 711-723, Berlin, Heidelberg. Springer- Verlag.
Jan De Belder and Marie-Francine Moens. 2012. Coref- erence clustering using column generation. In Pro- ceedings of COLING 2012: Posters, pages 245 -454 Mumbai, India. The COLING 2012 Organizing Com- mittee.
Junpeng Liu, Kaiyu Huang, Jiuyi Li, Huan Liu, Jinsong Su, and Degen Huang. 2022b. Adaptive token-level pross-lingual feature mixing for multilingual neural machine translation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 10097-101 13, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Pascal Denis and Jason Baldridge. 2007. Joint deter- mination of anaphoricity and coreference resolution using integer programming. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computa- tional I. Linguistics; Proceedings of the Main Confer- ence, pages 236-243, Rochester, New York. Associa- tion for Computational Linguistics.
Pan Lu, Liang Qiu, Kai-Wer Chang, Ying Nian Wu, Song- Chhu-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan, 2023. Dynamic prompt learning via policy gradient for semi- eemi-structured mathematical reasoning, In International Conference on Learning Representations (ICLR).
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Min- neapolis, Minnesota. Association for Computational Linguisties.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 8086 8098, Dublin, Ireland. Association for Compu- tational Linguistics.
André F. T. Martins. 2014. Linear programming de- coders in natural language processing: From integer programming to message passing and dual decom- position. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Process- ing: Tutorial Abstracts, Doha, Qatar. Association for Computational Linguistics.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764-10799. PMIR.
Ulrich Germann, Michael Jahr, Kevin Knight, Daniel Marcu, and Kenji Yamada. 2004. Fast and optimal decoding for machine translation. Artificial Intelli- gence,
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpuro- hit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2022. LII.A: A unified bench- mark for mathematical reasoning, In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 5807-5832, Abu
Fangyu Lei, Shizhu He, Xiang Li, Jun Zhao, and Kang Liu. 2022. Answering numerical reasoning ques- tions in table-text hybrid contents with graph- graph-based

---

Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Computational Linguisties: Human Language Tech- nologies, pages 2655-2677 Seattle, United States. Association for Computational Linguisties.
Rungsiman Nararatwong, Natthawut Kertkeidkachom, and Ryutaro Ichise. 2022. Enhancing financial ta- ble and text question answering with tabular graph and numerical reasoning. In Proceedings of the 2nd Conference of the Pia--Pacific Chapter of the Asso- ciation for Compulational Linguisties and the 12th International Joini Conference on Natural Language Processing (Volume 1: Long Papers), pages 991- 1000, Online only. Association for Computational Linguistics.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf. Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. Selective annotation makes language models better few-shot learners. In International Conference on Learning Representations (ICLR).
Jiashuo Sun, Hang Zhang, Chen Lin, Yeyun Gong, Jian Guo, and Nan Duan. 2022. Apollo: An optimized training approach for long-form numerical reasoning. arXiv preprint 2rXiv:2112.07244
Ansong Ni. Srini Iyer. Dragomir Radev, Veselin Stoy- anov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In International Con- ference on Machine Learning, pages 26106-26122 PMLR.
Gang Wang, Jingling Ma, and Gang Chen. 2023. Atten- tive statement fraud detection: Distinguishing multi- modal financial data with fineggriined attention. De- cision Support Systems, 167:113913.
Ankur Parikh, Xuezhi Wang. Sebastian Gehrmann, Man- aal Faruqui. Bhuwan Dhingra, Diyi Yang, and Dipan- jan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173-1 186, Online. As- sociation for Computational Linguistics.
Laurence A. Wolsey. 2020. Formulations, chapter I. John Wiley & Sons, Ltd.
Yuancheng Xia, Feng Li, Qing Liu, Li Jin, Zequn Zhang. Xian Sun, and Lixu Shao. 2023. Reasonfuse: Rea- son path driven and global-local fusion network for numerical table-text question answering. Neurocom- puling, 16:1669-811
Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Qiang Fu, Yan Gao, Jian-Guang Lou, and Weizhu Chen. 2022. Reasoning like program executors. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 761- 779. Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In Proceedings of the 40th Inter- national Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19818-39833. PMLR.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Jiaxin Zhang and Yashar Moshfeghi. 2022. Elastic: Numerical reasoning with adaptive symbolic com- piler. In Advances in Neural Information Processing Sysiems, volume 35, pages 12647-12661. Curran As- sociates, Inc.
Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. Ac- tive example selection for in-context learning. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 9134- 9148, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Sebastian Riedel and James Clarke. 2006. Incremental integer linear programming for non-projective de- pendency parsing. In Proceedings of the 2006 Con- ference Empirical Methods in Natural Language Processing, pages 129-137, Sydney, Australia. Asso- ciation for Computational Linguistics.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023. Automatic chain of thought prompting in large language models. In International Confer- ence on Learning Representations (ICLR).
Dan Roth and Wen-tau Yih. 2005. Integer linear pro- gramming inference for conditional random fields. In Proceedings of the 22nd International Conference on Machine Learning, ICMI. '05, page 736-743, New York, NY, USA. Association for Computing Machin- ery.
Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. 2022. MultiHiertr: Numerical reasoning over multi hierarchical tabular and textual data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6588 36000 Dublin, Ireland. Association for Computational Linguistics.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. Proceedings of the 2022 Conference of the North American Chapter of the Association for

---

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shol performance of language models. In In- ternationnal Conference Machine Learning, pages 12697-12706. PMLR.
2) Predicted modality: text
subject to
text; ai 2 a M
RC
other _ 1j 2 B M
Fan Zhou, Mengkang Hu, Haoyu Dong, Zhoujun Cheng, Fan Cheng. Shi Han, and Dongmei Zhang. 2022a. TaCube: Pre-computing data cubes for answering numerical-reasoning questions over tabular data. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pages 2278-2291, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
iES other_i- = table; + hybrid;
where
3) Predicted modality: hybrid
subject to
tab; 2; 2 B M
Yongwei Zhou, Junwei Bao, Chaoqun Duan, Youzheng Wu, Xiaodong He, and Tiejun Zhao. 2022b. UniRPG: Unified discrete reasoning over table and text as pro- gram generation. In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing, pages 7494-7507, Abu Dhabi, United Arab Emirates. Association for Computational Lin-
text; 24 2 M
iES
hybrida
x4 2 B M
iES
4) Predict answer type: span
guistics.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat- Seng Chua. 2021. TAT-QA: question answering hybrid of tabular and textual con- tent in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 3277-3287, Online, Association for Computational Linguistics.
subject to
spant Ij 2 a M
other; Fi 8 M
benchmark
where
other; mssaan + arith, + count,
5) Predict answer type: multi-span
subject to
mspaa; 24 2 a M
Fengbin Zhu, Moxin Li, Junbin Xiao, Fuli Feng, Chao Wang, and Tat Seng Chua. 2023. Soargraph: Nu- merical reasoning over financial table-text data via semantic-oriented hierarchical graphs. In Compan- ion Proceedings of the ACM Web Conference 2023, WWW '23 Companion, page 1236-1244, New York, NY, USA. Association for Computing Machinery.
SE
other;
F4 2 M
EE spon; + arith; count;
where
other;
6) Predict answer type: arithmetic
subject to
SES x, 2 a M
arith;
The SEER algorithm
SEER is explained in pseudo-code in Algorithm 1.
other;
x, 2 3 M
ES
where
other;
span; + mspant + count;
B Diversity constraint templates
7) Predict answer type: count
We present the potential configurations of diversity constraints, which are determined by the predic- tions of the constraint modules. There exists one configuration per possible attribute value. 1) Predicted modality: table
subject to
count;
3; 2 a M
icS
other; 24 2 9 M
iS
where
other; = span, + mspan; + arith;
subject to
table; x; 2,20 a M
C TAT-QA Prompt examples
SES other 4; 2 3 M
Figure illustrate an instance from the TAT-QA dataset. The answer type is multi-span and the modality is text.
EES
where
other i = text, + hybrid,

---

Algorithm/ 1 Selection of ExEmplars for hybrid Reasoning (SEER)
Input Test instance Xtest, training set Strain: candidate pool size Output Exemplar selection Eselection
Ecandidates (N Strain k)
KNN cosine similarity
attributes
0 for module E constrainl_ do
attributes//oddule.uu <- coodule.predictt (eest)
end for
knapsak + get sr_jip(XXunn attributes) Eeclection solve (knapsack, Econdidates)
Generate the Knapsack
Solve the Knapsack program
2019
2018
2017
Question: : what are the contract
Parameter
Value
Floed
1432.4
1946.2
1030.9
types?
Preprocessing
Price
44.1
Answer: (Mxeed price
plustppus type.
Spacy model
'en mo__ore_webb
lime-and-mmlerte lype]
NER model
'bert-basee Bort-base-NER-eo Satt--bb----
Total sales
5
1202.9
1107.7
Derivation:
Scale:
Similarity computation
fixed
agrce
Python code:
Backbone
'all-mpnet- -bseettbbbeev
The contrnetuall stalementt.... cost-pi puu--pp contract, I.I] On time-and-en material type
MMxed price type',
"cos-gpuus type'.
Fine-tuned constraint modules
hours b..d
contract we are paie the bante of cinect labor
"ime and material type)
Model
bast-base-ccsed"
Epochs
3
Figure 9: TAT-QA instance example
Batch size
32
Learning rate
5e -5
ICL constraint modules
D Implementation details
Model
'code-davinci-002"
Table 4 presents an overview of the parameters em- ployed in the components of the SEER framework.
Temperature
Top P
1
Max tokens
5
E Complement on evaluation metrics
SEER FinQA
k M
200
The ground truth answers of FinQA and TAT-QA have specific syntactic rules that require access to many training examples to be learned. Those rules not related to the semantic correctness of the answer. ICL approaches are limited to few exemplars. Hence, they have a limited ability to learn those syntactic rules. As a result, we provide some flexibility in the evaluation scripts. The fol- lowing rules are applied: equivalence of answers written as percentage or decimal, removal of char- acters (S, "million"," "million","illlio""" "billionn, ...), equivalence up to a rounding of 2 decimals, removal of trailing Os after the comma. Examples are shown on Table 5
4 [' ['Modality"]
Constraint module
a
75%
B
0 %
solver
GGRROB CMD'
SEER TAT-QA
A:
200
M
8
Constraint module
['Modality' 'nnwer type']
a
50%
B
25 %
solver
GGRROBI CMD'
Text retrieval
Backbone
all-MiniLM-L6-v22
F Comparison with Fine- Tune- models
k
10
Code Generation
Tables 6 and 7 list the best-reported performance of models that followed fine-tuning strategy. SEER outperforms several models but lags in perfor- mance compared to the current SOTA. There are several trade-offss consider between fine-tuning and ICL approaches. Fine-tuning requires updating the model weights, which costs time and resources. On the other hand, ICL models can be adjusted to
Backbone
-ovde-davinci-022
Temperature
0
Top P
1
Max tokens
256
Table 4: Parameter values

---

Ground Truth
((4.1-3.9)/.9) 100
G Significance tests
Prediction
(4.113.9)/33
We evaluate the statistical significance of the results of Table using the Wilcoxon Signed-Rank non- parametric test. The significance is evaluated by comparing the average EA and EM over 3 iterations for FinQA and TAT-QA respectively, Results are reported in Tables 8 and 9.
Ground Truth
I'S 3.4 milliion']
Prediction
['3.4 millionn]
Ground Truth
["29.0";27.0']
Prediction
['29",'7'']
Ground Truth Prediction
45.4
45.3981
PrompriG
Table 5: Examples of differences between ground truth and predictions that are counted as correct by the evalu- ation scripts.
Fixed set
27
KATT:
7x-20
fed
0.02
Divee KATE
48
le.5
0.83
0.68
PomppPG
4c-13
8c-16
4c-4
3629
1629
SEER
263
0.24
Model
EA
Table 8: Significance tests for FinQA
APOLLO ensemble (Sun et al., 2022)
71.07
ELASTIC (Zhang and Moshfeghi, 2022)
68.96
SEER + CODEX
68.85
DSP
Prompilk)
APOL.LO(Sun al., 2022)
67.99
Se-41
1e:30
ReasonFuse (Xia et al., 2023)
66.17
Divese KATE
KATE
2:-49
1o-20
1c:30
Icc
0.012
Ro-14
SoarGraph (Zhu et al., 2023)
64.5
ProupPG
3:.9
20.5
9c:30
6o:36
8c-12
SEER
2o-51
4c-3
2-3
0.24
16-15
40-40
FinQANet (Chen et al., 2021c)
61.24
SUER
46-56
26-28
366
fe-3
16:19
0015
Table 9: Significance tests for TAT-QA
Table 6: FinQA Tune-Tund models vs SEER's 's perfor- mance the test (%).
Model
EM
RegHNT (Lei et al., 2022)
73.6
SoarGraph (Zhu al., 2023)
70.3
UniRPG (Zhou al., 2022b)
70.2
SEER + CODEX
69.68
GANO (Nararatwong et al., 2022)
68.4
FinMath (Li et al., 2022)
60.5
Poet-SQL (Pi et al., 2022)
59.1
TaCube (Zhou al., 2022a)
53.9
TagOp (Baseline) (Zhu et al., 2021)
55.2
Table 7: TAT-QA Fine-Tuned models SEER's perfor- mance on the dev set (%).
new use cases without retraining. It is interesting note that both approaches are still far from the human expert performance, which is 91.16% of EA for the test of FinQA and 84.1 1% of EM for the private test set of TAT-QA. Hence, despite a relatively superior performance of the SOTA fine- tuned models over ICL models, both approaches are equally far from the human performance. Given the recent progress made with LLMs, we believe that the importance of ICL will only increase in the future. Hence, developing strategies for exemplar selection like SEER is an important direction for future work.


Reviews:
[{"review_id": "3JmtNqwf5J", "reviewer": "Reviewer_dH6X", "summary": "", "questions": "1. Why are the results for PromptPG so low?\n2. A baseline could be to divide the M number of examples into two groups, one of each modality, followed by kNN search for each group. I think this baseline could be stronger than KATE, avoids retrieving no examples of the correct modality as pointed in Figure 20, 21. What do you think?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, thank you for your constructive and insightful feedback. We are glad that you recognize the contribution of SEER to solving HybridQA problems and the improvements it brings over existing baselines.\nWe address your remarks and questions below:\n\n> **The paper has a huge appendix which is not entirely optional.**\n\nWe acknowledge that the current appendix is too long and contains information that should be moved to the main text. However, as mentioned by Reviewer Fa3v, this can be solved with minor changes. The revision will contain the **following modifications** : \n- The sections dealing with preprocessing (App. G), the description of PromptPG (App. J), the confusion matrices and description of the constraint modules (App. K and L), and the error analysis (App. O) are moved to the main body.\n- Certain content, such as dataset statistics and the instance examples from the FinQA and TAT-QA datasets (App. B, C, F), is already well-documented in the original dataset papers. Hence, we remove them from the Appendices.\n- We will add the best fine-tuning approach (App. M) and the significance scores of SEER and KATE (App. N) into the main body. \n\nBy making those modifications, the Appendices are reduced to 3 pages, and will contain only the optional content in the camera-ready version of the paper.\n\n> **Why are the results of PromptPG so low?**\n\nPromptPG (Lu et al., 2023) trains a neural network with the REINFORCE (Williams, 1992) policy gradient algorithm to select exemplars from a fixed candidate pool. Like all reinforcement learning algorithms, it needs to balance **exploration** and **exploitation** to move towards a global optimum. Our empirical observations have led us to conclude that PromptPG **under-explores** the candidate pool. It tends to prematurely settle into a fixed suboptimal selection of exemplars after a limited number of learning episodes. Given\nthat the initial candidate pool is randomly selected, the performance of a fixed subset is close to the random baseline. From those observations, we conclude that PromptPG needs adjustment beyond the original configuration proposed in Lu et al. (2023).\n\n\n>**A baseline could be to divide the M number of examples into two groups, one of each modality, followed by kNN search for each group. I think this baseline could be stronger than KATE, avoids retrieving no examples of the correct modality as pointed in Figure 20, 21. What do you think?**\n\nYour insightful suggestion raises an interesting new baseline approach that has yet to be explored in our study! Intriguingly, we find that only 42% of the exemplar set selected by KATE on FinQA incorporates all modalities. Consequently, there is a **clear potential** for this new baseline, which we call **Diverse KATE**, to outperform KATE. \nIt is worth mentioning that this baseline closely resembles SEER when the hyperparameters $\\alpha$ and $\\beta$ are both set to 0.5. In that scenario, SEER requires to select an equal number of exemplars possessing the predicted attribute and those with a different attribute.\nWe have conducted an initial evaluation of a single iteration of the Diverse KATE approach on the FinQA dataset. We obtained an **Execution Accuracy** (EA) score of 67.12,  a small improvement over the EA of KATE (67.07). We will conduct additional rounds of evaluation for both the FinQA and TAT-QA datasets. We will **include and discuss the obtained results** in the camera-ready version of our paper.\n\n> **The \"correct answer\" for Figure 19 should be 200+500+750=1450 instead of 1000?**\n\nThank you for identifying this typo. The correct answer, also predicted by SEER, is indeed 1450. This corresponds to the instance with the identifier “ETR/2008/page_298.pdf-2” from the FinQA test set.\n\n> **It is not clear from the description in the paper about how the tables are encoded? Is there any specific format for linearizing the table?**\n\nThank you for pointing this out. The tables are linearized with **“|” as column delimiters** and **“\\n” as row delimiters**. This is a standard scheme for table encoding (Lu et al., 2023), and it offers the direct advantage of introducing minimal additional tokens. \nTo provide greater clarity, we will explain the table encoding in Section 4.1 Datasets.\n\n> **In line 418, should it be CAS \\subset CAP?**\n\nThank you for your keen observation. However, **our notation is accurate as presented**: CAP $\\subseteq$ CAS. When the constraint module correctly predicts the attribute of a test instance, it guarantees that said instance will be incorporated into the selection, as stipulated by the $\\alpha$ diversity constraint. Consequently, every element within the Correct Attribute Predicted (CAP) set belongs to the Correct Attribute Selected (CAS) set. \nThrough the $\\beta$ diversity constraint, it is possible that one of the selected exemplars possesses the correct attribute, even if an incorrect attribute was predicted by the constraint modules. As a result, the CAS set also includes elements of the Incorrect Attribute Predicted (IAP) set. Consequently, CAP is a subset of CAS.\n\n\nReferences\n\n- Pan Lu,Liang Qiu,Kai-Wei Chang,Ying Nian Wu, Song-Chun Zhu,Tanmay Rajpurohit,Peter Clark, and Ashwin Kalyan. 2023. Dynamic Prompt Learning via policy gradient for semi-structured mathematical reasoning. International Conference Learning Representations (ICLR).\n\n- Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning,8(3):229–256.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Stronger results compared to baselines", "reasons_to_reject": "1. The paper has a huge appendix which is not entirely optional. ", "questions_to_authors": "1. Why are the results for PromptPG so low?\n2. A baseline could be to divide the M number of examples into two groups, one of each modality, followed by kNN search for each group. I think this baseline could be stronger than KATE, avoids retrieving no examples of the correct modality as pointed in Figure 20, 21. What do you think?\n", "paper_topic_and_main_contributions": "The paper proposes an approach for selecting examples for few-shot prompting. They propose defining an optimization problem using Knapsack based ILP. The objective is to maximize the similarity of the example with the test example subject to variable number of constraints. The constraints include budget of maximum number of tokens, maximum number of allowed examples. They also include constraints on diversity of selected examples. One of the constraints is on the modality of the example, the modality being one of table, text, or both. Another constraint is on the type of answer including span, multi-span, and so on.\nThey evaluate on the task of financial QA including FinQA and TAT-QA\n\nThe contributions include:\n1. They show that proposing such a knapsack based ILP optimization outperforms other baselines including random selection of examples, k-nearest neighbor search.\n2. They show that the approach is effective for dataset which combines different forms of input including text, table, or both.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The \"correct answer\" for Figure 19 should be 200+500+750=1450 instead of 1000?\n2. It is not clear from the description in the paper about how the tables are encoded? Is there any specific format for linearizing the table?\n3. In line 418, should it be CAS \\subset CAP?\n"}}, {"review_id": "KX6Dtl4peh", "reviewer": "Reviewer_aS63", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for your helpful comments. We appreciate that you see a **clear motivation** in our proposed approach. SEER addresses a **key issue** in HybridQA (Reviewer Fa3v). We are the first to show that the Knapsack is well-suited for HybridQA problems that combine text, table, and hybrid data (Reviewer Fa3v, Reviewer dh6X). \n\nWe address your remarks and questions below:\n\n> **The paper, especially the figures, is not formed in a good layout.**\n\nWe acknowledge that certain figures might not have achieved the desired level of clarity in their current presentation. This issue **can be resolved** for the camera-ready version of the paper. We intend to reduce the amount of text shown on the illustrative examples of FinQA and TAT-QA (Figure 1, 2, 10) to make them more visually-engaging. We **will apply those changes** to the camera-ready version of the paper.\n\n> **Analysis experiments in the main body is not adequate.**\n \nThe analysis experiments in the main body consist of two parts. Allow us to elaborate on the **significance and rationale** behind these analyses, addressing their relevance and contributions.\n- The first analysis is concerned about controlling the length of the prompt while adhering to **token capacity constraints**. This aspect is vital due to the **inherent limitations of LLMs** in processing a finite number of tokens as input. Moreover, commercial LLMs, such as GPT-4, often use the token count as a pricing unit, rendering **token budgeting** a practical consideration for organizations. The **importance of this property**, and its applicability to other NLP tasks, is supported by  Reviewer Fa3v. In our detailed analysis, we compare SEER with its closest baseline, KATE, under different token budgets. We show how SEER's strategic incorporation of token capacity constraints yields **superior performance** compared to KATE, particularly in scenarios with limited token capacity budgets. \n- The second analysis evaluates SEER’s **sensitivity** to the values of the **hyperparameters** $\\alpha$ and $\\beta$. Those parameters govern the selection of exemplars based on their shared attributes with the test instance. Our analysis shows that the sensitivity to changes in hyperparameter values is low, meaning that SEER can produce **good results without extensive hyperparameter tuning**. \n\nIn essence, both analyses **contribute substantially** to SEER's assessment, addressing robustness and applicability.\n\nFollowing the recommendation of Reviewer Fa3v’s, we will also include a **qualitative error analysis** in the revised version of the paper. This analysis will highlight research directions to augment the approach in future work. \n\n> **The writing and story is not easy-reading**\n\nThank you for your feedback on the readability of the paper. We fully recognize the importance of ensuring a smooth and comprehensible reading experience for our audience. Could we ask you to provide pointers to the parts of the paper that are currently not easy-reading? We are **committed to addressing these concerns** for the camera-ready version.\n\nWe will take the following actions to improve the presentation : \n- **Reordering of Sections**: We will relocate the \"Related Work\" section after \"Introduction\" and before \"Methodology,\" in line with the feedback provided by Reviewer Fa3v. This adjustment will provide readers with background information on In-Context Learning and Integer Linear Programming, improving their understanding of the subsequent content.\n- **Visual clarity**: We will reduce the amount of text in the figures that illustrate FinQA and TAT-QA instances (Figures 1, 2, 10) . This modification aims to render the visual representation more engaging for readers.\n- **Results Presentation**: In response to the suggestion by Reviewer Fa3v, we will arrange the baselines in the results table from the worst to the best performing. This arrangement will provide a more intuitive representation of the relative performance of the different methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The code is provided.\n2. clear motivation.", "reasons_to_reject": "1. The paper, especially the figures, is not formed in a good layout.\n2. Analysis experiments in the main body is not adequate.\\\n3. The writing and story is not easy-reading.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes Knapsack Programs as a framework to model exemplar selection for ICL. For a given test instance, a Knapsack Program is solved to obtain the optimal exemplars set. This expressive framework allows balancing the diversity and similarity of the selected exemplars while controlling the prompt size with user-defined linear constraints. The paper introduces SEER, a novel method to select exemplars for HybridQA using Knapsack Programs. SEER reduces the candidates set with a nearest neighbor filtering, and leverages constraint modules to predict the attributes of the test instance. The experimental results show that the proposed method outperforms previous exemplar selection methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hpQpnEvJiG", "reviewer": "Reviewer_Fa3v", "summary": "", "questions": "end of section 3.2: Include the best fine-tuning approach in the comparison here; that is mention it in the text (there will be room once you do not refer to the appendix) and add a row to the main results table\n\nlines 399-402: at least state whether the non-gold SEER results are stat sig better than the next best system. \n\nFigure 4:  some of the results look very similar: are they stat sig different? Also, indicate the variance in the 3 runs (e.g. have a min/max bar overlaying the current ones).\n\nI think the knapsack approach could be used for other tasks that use ICL with LLM. Perhaps mention this in the conclusion as an area for future work/future impact of the proposed method.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for your constructive comments and detailed improvements suggestions. We are encouraged that you see in SEER an **excellent application of the Knapsack algorithm** to exemplar selection. We thank you for pointing out the **solid results** and **comparison with strong competitors**, also supported by Reviewer dh6X.\n\nWe address your remarks and questions below: \n\n> **Need to include a few items from the appendix into the main paper (minor and I think these can fit). Huge appendix, which is not realistic to ask reviewers to look at.**\n\nWe acknowledge that the current appendix is too long and some content needs to move to the main text. We **will take the following actions to correct this** : \n- The sections dealing with preprocessing (App. G), the description of PromptPG (App. J), the confusion matrices and description of the constraint modules (App. K and L), and the error analysis (App. O) are moved to the main body.\n- Certain content, such as dataset statistics and the instance examples from the FinQA and TAT-QA datasets (App. B,C,F), is already well-documented in the original dataset papers. Hence, we remove them from the Appendices.\n- We will add the best fine-tuning approach (App. M) and the significance scores of SEER and KATE (App. N) directly into the main body. \n\nBy making those modifications, the Appendices are reduced to 3 pages, and will contain only the optional content in the camera-ready version of the paper.\n\n> **No qualitative error analysis which could help in determining how to augment the approach in the future.**\n\nThank you for suggesting the **addition of a qualitative error analysis**. We had previously sampled 100 error cases per dataset and classified the errors in different categories (Table 11). Based on your recommendation, we explored again the sampled error cases together with their exemplar set and obtained **three major insights**: \n- 25% of the FinQA errors resulted from a poor ranking of the candidate exemplars based on their similarity with the test instance. SEER could benefit from more advanced methods to compute the similarity weights in the Knapsack objective function. Those are currently computed with a pre-trained Sentence-BERT model, but **more advanced fine-tuning** methods are available (Rubin et al., 2022).\n- 17% of the FinQA errors had one missing reasoning step, e.g. the addition of a constant term. In the future, the approach can be augmented with mechanisms that predict and control the **required number of reasoning steps**.\n- When a TAT-QA error contains incorrect values in the reasoning (31%), the exemplars often possess the correct attributes and have a high similarity with the test instance. Hence, it seems the error results from the limitations of the LLM itself, i.e., CODEX in our case. This is particularly true for Counting problems, where 4 or more relevant Counting exemplars are provided, but CODEX still fails to make the correct count for the test instance.\n\nWe will **include the qualitative error analysis** and the insights **in the main body** of the revised paper.\n\n> **The two above abilities are likely to be applicable to other NLP problems. I think the knapsack approach could be used for other tasks that use ICL with LLM. Perhaps mention this in the conclusion as an area for future work/future impact of the proposed method.**\n\nWe are heartened by your recognition of the **broader applicability of SEER's knapsack approach** in NLP applications and beyond. The current conclusion already highlights Hybrid Fact-Checking (Aly et al., 2021) as an avenue for future work. We will augment the conclusion to encompass other promising applications, such as Data-to-Text Generation (Parikh et al., 2020) and Multimodal Fraud Detection (Wang et al., 2023).\n\n\n>  **Include the best fine-tuning approach in the comparison here; that is mention it in the text (there will be room once you do not refer to the appendix) and add a row to the main results table. lines 399-402: at least state whether the non-gold SEER results are stat sig better than the next best system**\n\nThank you for this recommendation. As mentioned above, we will add the best fine-tuning approach (App. M) and the significance scores of SEER and KATE (App. N) directly into the main body, in the main results section.\n\n>**Figure 4: some of the results look very similar: are they stat sig different? Also, indicate the variance in the 3 runs (e.g. have a min/max bar overlaying the current ones).**\n\nThank you for suggesting those additions that reinforce the insights observed from the subset analysis. We evaluate the statistical significance with the Wilcoxon Signed-rank non-parametric test over the 3 runs.  SEER and KATE are **stat sig different (p-value=0.05)** for Correct Attribute Predicted (CAP) and Correct Attribute Selected (CAS). SEER and the Fixed set baseline are stat sig different for CAP, CAS, and Incorrect Attributed Predicted (IAP). KATE and Fixed set are stat sig different for CAP, CAS, and IAP. The variance over the 3 runs is low, except for the KATE baseline on CAS and IAP, and for the Fixed set baseline on IAS. The mean and standard deviation values are reported in the following table.\n\n|    | CAP |  CAS |  IAP |  IAS  |\n| ----- | ----  | ----  | ----   | ----   |\n|**SEER** | 75.48+-0.82 | 71.83+-0.94 | 59.42+-0.47 | 48.53+-0.47| \n| **KATE** | 72.8 +-0.47 |  69.77+-0.94 | 58.86+-0.0 | 48.75+-1.25 |\n| **Fixed set** | 70.03+-0.94 | 66.1+-2.16 | 55.56+-1.89 | 50.11+-0.47 |\n\n\nWe will edit Figure 4 to **include min/max bars for the standard deviations**.\n\n\n> **The paper is very well laid out and easy to follow. The English is generally excellent but there are a few systematic errors.**\n\nThank you for taking the time and effort to identify errors and **providing pointers** to their position in the text, which will **greatly assist us** in rectifying them. We will edit all typos and syntactic mistakes for the camera-ready version.\n\n>**Related work could be earlier. Some of the explanations are helpful in understanding ICL and ILP if the reader is not familiar with them.**\n\nThank you for this suggestion. We will move the related work section between Section 1 Introduction and Section 2 Methodology.\n\n>**I am glad the authors are thinking about possible misuse. What is cited is pretty much a standard concern. It is good to cite it, but perhaps mention more strongly that this is a general concern with using LLM and also for using automatic QA without double-checking the results.**\n\nWe propose the following modification to the original text : “While we have not identified any direct potential for misuse for SEER in particular, it is crucial to acknowledge **some general concerns** with using automatic question answering systems and LLMs. Automatic question answering systems are not infallible. Blindly relying on the outputs generated by the model without double-checking the results can have adverse consequences for end-users, particularly in financial investment analysis. [...]”\n\n\nReferences\n\n- Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. Feverous: Fact extraction and verification over unstructured and structured information. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran.\n- Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A Controlled Table-To-Text Generation Dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173–1186, Online. Association for Computational Linguistics.\n- Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning To Retrieve Prompts for In-Context Learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2671, Seattle, United States. Association for Computational Linguistics.\n- Gang Wang, Jingling Ma, and Gang Chen. 2023. Attentive statement fraud detection: Distinguishing multimodal financial data with fine-grained attention. Decision Support Systems, 167:113913", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Excellent use of knapsack algorithm to the problem.\n- Ability to adjust to input length constraints on LLMs.\n- The two above abilities are likely to be applicable to other NLP problems.\n- Solid results and comparison to strong SOTA competitors.\n- Well written and easy to understand.", "reasons_to_reject": "- Need to include a few items from the appendix into the main paper (minor and I think these can fit).\n- No qualitative error analysis which could help in determining how to augment the approach in the future.\n- Huge appendix, which is not realistic to ask reviewers to look at.", "questions_to_authors": "end of section 3.2: Include the best fine-tuning approach in the comparison here; that is mention it in the text (there will be room once you do not refer to the appendix) and add a row to the main results table\n\nlines 399-402: at least state whether the non-gold SEER results are stat sig better than the next best system. \n\nFigure 4:  some of the results look very similar: are they stat sig different? Also, indicate the variance in the 3 runs (e.g. have a min/max bar overlaying the current ones).\n\nI think the knapsack approach could be used for other tasks that use ICL with LLM. Perhaps mention this in the conclusion as an area for future work/future impact of the proposed method.\n", "paper_topic_and_main_contributions": "This paper addresses a key issue in hybridQA (text+table): exemplar selection. It treats this as a knapsack problem, which allows the system to more effectively select exemplars under various resource constraints (primarily length of input to an LLM).\nThe paper provides an excellent introduction to the task and to ICL and Knapsack approaches. The results are good and are compared to multiple other systems on two standard datasets.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is very well laid out and easy to follow. \nThe English is generally excellent but there are a few systematic errors:\n- do not have a space before colons (:) in English\n- modifiers in noun-noun modification should be in the singular even if they refer to a plural (e.g. \"candidate set\" not \"candidates set\")\n- \"allow to + infinitive\" requires an object (line 252: allows the user to balance)\n- \"train set\" and \"train instance\" => \"training set\" and \"training instance\"\n\nline 059: problem( => problem (\nline 090: insert space after commas\nline 225: what does \"neighbor of the test instance's\" mean? I think this is just a typo; please rephrase\nline 320: syntactical => syntactic\nline 375: estimates of => estimate of\nline 379: solver3to => solver3 to\n\nTable 1: It will be easier to read if the rows are ordered roughly worst to best system.\n\nSection 5: This could be earlier. Some of the explanations are helpful in understanding ICL and ILP if the reader is not familiar with them.\n\nline 620: potentialWhile => potential While\n\nEthical concerns: I am glad the authors are thinking about possible misuse. What is cited is pretty much a standard concern. It is good to cite it, but perhaps mention more strongly that this is a general concern with using LLM and also for using automatic QA without double-checking the results.\n"}}]

Author rebuttal:

Meta-review: This paper presents an ILP formulation for selecting exemplars -- for prompting LLMs in a task of hybrid (text+tables) QA. The idea is interesting and novel. Moreover, it allows for a better control of selected exemplars in terms of their diversity, length/number or other constraints. The code will be publicly available. All the reviewers agree that this contribution is sound and exciting.

However, there are some presentation issues. The main issue is the appendix (reviewers #1 and #3): it is very large and contains essential information. It should be reorganized to make the paper self-sufficient and easy to follow. Minor presentation issues (easily fixable) have been noted by the reviewers.
Decision: Accept-Main