Using counterfactual contrast to improve compositional generalization for multi-step quantitative reasoning
Year: 2023

Abstract:
In quantitative question answering, compositional generalization is one of the main challenges of state of the art models, especially when longer sequences of reasoning steps are required. In this paper we propose CounterComp, a method that uses counterfactual scenarios to generate samples with compositional contrast. Instead of a data augmentation approach, CounterComp is based on metric learning, which allows for direct sampling from the training set and circumvents the need for additional human labels. Our proposed auxiliary metric learning loss improves the performance of three state of the art models on four recently released datasets. We also show how the approach can improve OOD performance on unseen domains, as well as unseen compositions. Lastly, we demonstrate how the method can lead to better compositional attention patterns during training.

heading: 1 Introduction
text: Enterprise documents such as reports, forms, and analytical articles often include quantitative data in tabular form. The data in these tables can be self-contained, but more commonly the surrounding text provides more context that is necessary to understand the content. Answering questions over these hybrid tabular/text contexts requires reasoning that combines verbal and quantitative semantics.
Question answering over quantitative tabular/text data has gained recent traction with the release of datasets such as FinQA (Chen et al., 2021b), TAT-QA (Zhu et al., 2021), and HiTab (Cheng et al., 2022). Table 2 shows an example of a question that requires quantitative reasoning to derive the answer. Given the question and the tabular context, the output is a single-step program that leads to the final answer of -20.
A major challenge that state of the art models face is compositional generalization (Montague,
1973), especially when the number of reasoning steps grows (Chen et al., 2021b). In the context of quantitative QA, compositional generalization refers to the model’s ability to generalize to new compositions of previously seen elements. As an example, if the model has encountered training examples that demonstrate calculations for “growth rate” and “percent change”, we would like it to be able to come up with a reasonable hypothesis as to how to calculate “percent growth” or “rate of change”. Table 1 demonstrates how this challenge becomes more difficult as the number of reasoning steps grows. For questions that require longer chains of reasoning, the model learns spurious patterns and unsuccessfully tries to leverage these memorized patterns to solve new problems.
The Table also shows that as the number of steps grows, generating the wrong operator becomes a more dominant mistake than selecting the wrong operand. Not only is this error more dominant, but it can also have a more destructive impact on the chain of reasoning, as it can derail the model’s hidden representations from that point onward. As an example, our analysis of the FinQANet model (Chen et al., 2021b) output showed that if the model generates an incorrect operator, it is about 30% more likely to commit other errors in the following
14930
steps compared to when the model generates an incorrect operand.
In this paper, we propose CounterComp, an approach that can enhance compositional learning in multi-step quantitative QA. We take inspiration from the symbolic composition of arithmetic operations, and their correspondence to natural language phrases. Building on the work on attention alignments from previous studies, we propose an auxiliary metric learning loss that is focused on specific components of the input and output. Our sampling strategy is based on counterfactual scenarios. This means that the model learns proper representations for each component based on what-if scenarios. To the best of our knowledge, this is the first study that successfully applies component-wise counterfactual sampling as a metric learning strategy. We show how, when state of the art models are augmented with our auxiliary metric learning loss, they exhibit better performance in cases where multistep reasoning is required. CounterComp outperforms current baselines on four recently released datasets, and show stronger performance on OOD samples.

heading: 2 Related work
text: The typical architecture of a quantitative QA model is composed of a retriever and a generator (Jurafsy and Martin, 2021). The retriever identifies the particular context where the answer might be found. Since the context can be a mix of table cells and
sentences, often a tabular encoder (Herzig et al., 2020) or verbalizer (Chen et al., 2021b) is used to convert the cells into a natural language sequence. The retrieved context is referred to as retrieved facts. Next, the generator uses the question along with the facts to generate the output in a step by step fashion. In multi-step QA, the generator often combines a recurrent module with an attention mechanism (Chen et al., 2021b), as illustrated in top half of Figure 1.
The output can be assessed in terms of program accuracy as well as execution accuracy. Our study is focused on improving program accuracy by encouraging compositional generalization in the generator.
There are two common approaches to improving
compositional generalization. Attention alignment models encourage explicit alignments between natural language utterances (e.g. “rate of change”) and corresponding symbolic math operations (e.g. subtraction followed by division). Methods informed by counterfactuals use what-if scenarios to generalize to a wider variety of compositions and reduce the effect of memorization.

heading: 2.1 Attention alignments
text: Yin et al. (2021) showed that additional supervision can be used to promote explicit alignments between components in the input and in the output. They added a regularization loss that encourages the cross-attention module to adjust its attention weights according to gold alignments. Using as few as 16 examples, their model was able to improve generalization in a semantic parsing task. CompAQT (Nourbakhsh et al., 2022) extended this idea to multi-step quantitative QA. Instead of using additional supervision, it used natural language heuristics to create noisy alignment labels between input tokens and output symbols. The additional alignment loss improved the performance of three baseline models on multi-step reasoning tasks for four datasets.

heading: 2.2 Methods informed by counterfactuals
text: The success of alignment-based methods is limited by the fact that by heavily discouraging memorization, they underperform in settings where memorization can be helpful (Oren et al., 2020). To strike a balance between memorization and generalization, one approach is to generate new training examples that cover important semantic gaps in the training data. This is reminiscent of how adversarial training can help better define the semantic contours of compositional representations (Zhang et al., 2022). Contrastive or metric learning methods pursue a similar goal, but instead of generating new samples, they leverage existing samples within the training set (Jain et al., 2021).
Counterfactual data augmentation (CAD) methods strive to achieve this by generating new samples using what-if scenarios (Zmigrod et al., 2019; Liu et al., 2021; Chen et al., 2021a). This can be done by altering a minimally sufficient set of tokens in the input such that the output class changes (Kaushik et al., 2020). There are two main challenges to creating these samples. First, it is difficult to identify the minimal set of tokens necessary to alter the output. Second, there is no guarantee that
a counterfactual sample exists in the training set. To address these challenges, some studies employ human labelers (Kaushik et al., 2020) or a third party model (Huang et al., 2020). In domains like semantic parsing and quantitative QA where the output is symbolic, an alternative approach leverages the structure of the output to avoid the need for human labelers. Li et al. (2022) achieve this by intervening on the operands. Suppose that a question states “What was the net change in revenue from 2019 to 2020?” and the retriever produces two (verbalized) table cells: “2019 revenue was $80M” and “2020 revenue was $60M”. The output program for this question would be: subtract(80, 60). Given the numeric nature of the operands, it’s possible to generate new scenarios such as “What if 2019 revenue was $90?” with the updated output subtract(90, 60). Employing this method, Li et al. (2022) augment the TAT-QA dataset (Zhu et al., 2021) into a new dataset named TAT-HQA. They also enhance the verbal reasoning capacity of their model by offering the counterfactual scenario as a natural language prompt. Their model, named Learning to Imagine (L2I), outperforms state of the art models.
As mentioned in the previous section, models that struggle with compositional generalization suffer from errors in operator selection, whereas L2I is focused on the selection of operands. In this paper, we propose CounterComp, a method that focuses on counterfactual sampling for components that indicate operators1. Using natural language constraints from previous studies, we first find components that correspond to operators versus those that correspond to operands. Next, we use an auxiliary metric learning loss with positive and negative samples chosen based on those components. This helps us avoid the complexities associated with a data augmentation approach, such as the need for creation of additional human labels. The next section lays out our problem definition in more detail.

heading: 3 Problem formulation
text: Let us consider the example provided by Table 3. Suppose Q is the question, represented as a sequence of tokens q1, · · · , qN (i.e. “what”, “was”, “the”, · · · , “2020”, “?”).
F is the evidence obtained by the retriever, made up of a sequence of tokens f1, · · · , fM (i.e. “2019”,
1Please refer to Appendix C for a study on the use of CounterComp for operators versus operands.
“revenue”, “was”, · · · , “$60M”). The concatenation of these two sequences, i.e. Q||F , forms the input to the generator. The generator encodes Q||F using a neural language model such as RoBERTa (Liu et al., 2019), resulting in an embedding matrix U ∈ Rdenc×(N+M).
Consistent with Chen et al. (2021b), we represent the output S as a sequence of steps s1, · · · , sL. Each step sl can be an operator (such as add or divide), or an operand. Similar to Chen et al. (2021b), our programs are modeled as rightexpanding binary trees with each operator having exactly two operands. If necessary, one or more operands are set to NONE, where NONE is a special constant. L is pre-defined as the maximum number of steps allowed. In the example from Table 3, S is: subtract, 80, 60, NONE, NONE, NONE.
To generate the lth output step sl, the generator applies a cross-attention module to U , resulting in the attention weight matrix Al ∈ R1×(N+M) and the attention output X l ∈ RK . A recurrent module then generates the hidden vector hl, which is used to produce the output step sl.
hl = RNN(hl−1,X l) sl = NN(hl) (1)
where NN can be any neural module that projects hl onto the simplex sl ∈ RK , from which sl can be sampled: sl = argmaxk sl,k. Our goal is to encourage hl to be sensitive to the composition of the input Q||F with regards to the current output step sl. This means that hl needs to capture proper alignments between important terms in the input and the relevant operator/operand in the output.
To achieve this, we pursue a metric learning approach where positive and negative samples are generated according to counterfactual scenarios.

heading: 3.1 Counterfactual samples
text: Given a training example ([Q||F ](i), S(i)), we define an intervention target Q(i) as a subsequence of the question tokens, i.e. Q(i) = {q(i)n ;n ∈ N (i)} where N (i) ⊆ {1, 2, · · · , N}.
Suppose that changing the intervention target affects a single step in the output program S(i) = s(i)l , which we name the intervention outcome. Note that due to our focus on the generation of operators, we limit the intervention outcome to an operator. Since the output is composed of one operator followed by two operands followed by another operator and so on, l is selected from a limited index set:
l ∈ {1, 4, 7, · · · , L−3}. In the example from Table 3, the possible indices will be 1 and 4, representing the operators subtract and NONE.
Given this definition, it’s possible to mine positive and negative examples for the ith training instance. A positive example ([Q||F ](i)pos, S(i)pos) is an instance for which, despite a possible intervention in the target, the outcome remains the same, i.e. Q(i)pos ̸= Q(i) and S(i)pos = S(i). A negative example ([Q||F ](i)neg, S(i)neg) is an instance for which an intervention in the target leads to a change in the outcome, i.e. Q(i)neg ̸= Q(i) and S(i)neg ̸= S(i).
This allows us to define a triplet loss that encourages h(i)l to remain close to h (i) l,pos and far from h (i) l,neg with a margin of α (i):
L(i)triplet = max{||h(i)l − h (i) l,pos||22 − ||h (i) l − h (i) l,neg||22 + α(i), 0}
(2) Figure 1 illustrates the sampling process for one training example. Note that this metric learning approach will only be valid if causal assumptions with regards to the intervention target are valid, i.e. the change in S(i)neg is in fact the result of the intervention in Q(i)neg and not a change in any other part of the input. In a data augmentation setting, this can be achieved by keeping the input fixed and perturbing a small segment that functions as the intervention target similar to (Kaushik et al., 2020). However, as discussed in Section 2.2. this requires additional manual labor to annotate the perturbed examples.
In the next section, we describe how we impose certain constraints on the intervention target to achieve this in a self-supervised setting2.

heading: 4 Methodology
text: Our goal is to identify potential positive and negative samples for the anchor ([Q||F ](i), S(i)). Suppose the anchor is the one shown in the top three rows of Table 3. Bold italicized tokens are redundant between the question and the fact, e.g. “revenue”, “2019”, and “2020”. Those terms are often used by the retriever to find the correct facts. They are also used by the generator to find the correct order of operands.
2Note that the term “self-supervised” is used in this context to refer to the sampling strategy, i.e. no additional labeling is needed to generate the positive and negative samples.
There are also terms that are unique to the question, i.e. “What”, “the net change to”, “from”, and “to” (highlighted in blue). In CompAQT, the authors showed that these can be used as indicators for the operators. Lastly, there are terms that are unique to the facts, i.e. “was $80M” and “was $60M” (red italicized tokens). These can be used as indicators for the operands. We use these heuristics to guide our sampling strategy.
We flag all spans in the question that do not overlap with the facts, i.e. underlined blue segments. Those spans serve as candidate intervention spans. In the example from Table 3, this results in four candidates: “What”, “the net change in”, “from”, and “to”.
Next, we seek a positive and a negative example within the training set. A positive example is a sample in which, despite possible changes in the question, the operators in the output remain consistent with the operators in the anchor. Table 3 shows one such example. Several terms have been altered in the question. However, we would only focus on the changes in the candidate spans. Here, “was” has changed to “is”, “net change” has changed to “difference”, “from” to “between” and “to” to “and”. This results in a token-level Levenshtein distance of 5 (four edits and one insertion) (Yujian and Bo, 2007). We ignore the change from “revenue” to “operating expenses” and from “2019” to “2018”, because those changes have occurred outside of our candidate spans and only correspond to operands.
A negative example is a sample in which exactly one output operator is altered, deleted, or added. Table 3 shows one such example. Here, the output includes a new operator divide. The question has also been altered with a token-level Levenshtein distance of 4.
The given positive and negative example can now be plugged into Equation 2. Instead of a fixed margin, we use the edit distances mentioned before to dynamically adjust the margin. Let NLD(i)pos and NLD(i)neg be the normalized, token-level Levenshtein edit distance between the anchor and the positive example, and the negative example, respectively. We set the margin to: α(i) = 1−|NLD(i)neg − NLD(i)pos|
This encourages a larger margin for cases where the anchor is equally similar to the positive and the negative examples, and the model might have a harder time picking up on the nuances of each component.
indicates components that are unique to the question ( candidates for intervention). These terms often indicate an operator. Red italicized text indicates terms that are unique to the facts. These terms often indicate operands. Bold italicized text indicates terms that are shared between the question and facts. These terms often indicate metrics.

heading: 4.1 Runtime optimization
text: There are two runtime challenges to this proposed approach: 1) Sampling can be costly if the entire training set has to be scanned for each batch. This means an online sampling strategy cannot be used. On the other hand, an offline strategy introduces a large overhead. A hybrid approach is needed. 2) Calculating the edit distance metric is a costly operation with O(n2) steps.
To solve the first problem, we build two indices prior to training. One index groups the samples by their sequence of output operators. This index can be used to sample positive examples.
The other index includes all training examples, and for each example, it includes the full list of one-step perturbations applied to its output operators. By generating all possible perturbations, we are able to find other samples whose outputs match the perturbed sequence (i.e. negative samples). For
a sequence with n operators, all possible perturbations can be generated in O(n ×K) time, where K is the number of possible operators3.
Given the pre-generated positive and negative pools, we can also calculate and cache edit distances ahead of time. However, in practice, we realized that we could do so during training with little additional cost. This is because the edit distance is applied at the token-level4, and is limited to candidate spans, rendering it relatively fast. The decision as to whether distances should be cached or calculated on the fly depends on the average size of each pool versus the number of training steps.
The algorithm outlined in Appendix B summarizes our approach.

heading: 5 Experiments

heading: 5.1 Datasets
text: We use the hybrid CompAQT dataset, which is composed of four previously released datasets, namely FinQA (Chen et al., 2021b), TAT-QA (Zhu et al., 2021), HiTab (Cheng et al., 2022), and MULTIHIERTT (Zhao et al., 2022). The authors filtered these four datasets down to QA pairs that require single or multi-step quantitative reasoning. They also processed the tables and outputs in all four datasets to match the FinQA format.

heading: 5.2 Baselines
text: We apply our proposed auxiliary loss to three baselines: 1) FinQANet, originally developed for the FinQA dataset (Chen et al., 2021b). 2) TAGOP, originally developed for the TAT-QA dataset (Zhu et al., 2021). 3) Pointer-Verbalizer Network (PVN), originally proposed by Nourbakhsh et al. (2022). We also apply the CompAQT loss to each model as a secondary baseline in order to determine how CounterComp compares to an attentionalignment strategy.

heading: 5.3 Sampling success rate
text: Another possible concern is that our sampling strategy might be limited, in that positive and negative samples might not always be available in the training set, or that limited availability of samples might bias the training process. To remediate the problem of unavailable samples, when a positive sample is
3Since we follow Chen et al. (2021b), in all of our experiments K = 10.
4Since we’re using a language model that uses word-piece tokenization, in effect the runtime is at subword level.
missing, we use the anchor as the positive sample, and when a negative sample cannot be found, we use a uniformly sampled instance from the batch.
Table 4 shows some statistics about the success rate of the sampling algorithm. “% Failure” identifies the share of training examples for which either a positive or a negative example was missing. Unsurprisingly this never happens for single-step programs, is very rare for two-step programs, and with the exception of HiTab, happens in less than 10% of the cases for longer programs. The Table also shows the average number of positive and negative examples found for each anchor. Again, HiTab has the lowest number of available samples, making it the most challenging dataset. In Section 6.4, we demonstrate how, even in cases with few possible samples, the model is able to generalize to unseen examples.

heading: 5.4 Settings
text: Since we are focused on the generator, in the experiments discussed in this section we will use gold facts and encode the input using RoBERTalarge (Liu et al., 2019)5. We run the baselines with and without the additional Ltriplet for 50 epochs with a learning rate of 5e−5, the Adam optimizer (Kingma and Ba, 2015) with β1 = 0.9 and β2 = 0.999. At each step, we sample (with replacement) 5 positive and negative pairs per anchor, and add the average auxiliary triplet loss to the main model loss with a weight of λ. After a grid search with a step-size of 0.1, we set λ to 0.4 for all experiments. All experiments were conducted on 8 NVIDIA T4 GPUS with 16 GBs of memory.

heading: 6 Results and analysis
text: Table 5 shows the program accuracy of baselines (top row of each cell) compared the addition of CounterComp loss (bottom row of eachcell). Among the baseline models, TAGOP is not designed to generate multi-step programs. Therefore we only apply it to the TAT-QA dataset, which has a set of pre-determined operations (e.g. change ratio). We also apply the PVN model to the combined dataset, but since FinQANet outperforms it on all benchmarks, we will continue to use FinQANet as the reference baseline model for the remaining experiments in this section.
As Table 5 shows, CounterComp consistently
5Please refer to Appendix A for results using retrieved facts.
outperforms the baselines and the margin is often higher for longer programs. One notable exception is the TAT-QA dataset. As mentioned before, the dataset is not designed for open-ended multistep reasoning and includes a limited set of possible operations. Therefore methods that encourage memorization might achieve higher performance on TAT-QA. HiTab is another challenging dataset, but despite low performance on longer sequences, CounterComp offers an improvement over the baseline.

heading: 6.1 Auxiliary triplet loss versus auxiliary attention alignment loss
text: The middle row of each cell in Table 5 shows the program accuracy when CompAQT loss is added instead of CounterComp loss. As previously described, CompAQT imposes an auxiliary attention alignment loss such that tokens related to opera-
tors receive more attention during the generation of operators. Even though this leads to improvements over the baselines, CounterComp outperforms CompAQT in all experiments. This might be due to the fact that the regularizing effect of CompAQT loss is not as strong as the representation learning impact of CounterComp.
Despite the fact that CounterComp was not designed as an attention alignment model, it does have an impact on how attention patterns evolve during training. Table 6 shows the top-attended input tokens during the generation of a divide operator in various contexts. For a singular division operation, FinQANet attends to tokens such as “year” whereas CounterComp encourages the model to attend to more relevant tokens such as “net” and “change”. A subtraction followed by a division often indicates a percentage calculation, as captured by both models. An addition followed by a division often indicates an average calculation. Again, CounterComp is able to capture relevant tokens but the FinQANet baseline seems to attend to some memorized tokens such as “annual”.

heading: 6.2 Fixed versus adaptive margin
text: The last row of Table 5 shows the performance of the FinQANet model on the combined dataset using the CounterComp loss with a fixed margin of 1. The performance suffers, especially as the number of steps grows. This further demonstrates the importance of the adaptive margin α(i) that takes the edit distance into account.

heading: 6.3 Qualitative examples
text: Table 7 shows four qualitative examples from the FinQA dataset. The first two rows show how CounterComp enables the FinQANet model to represent concepts such as “decline” and “percentage” more accurately. The third example shows how CounterComp is able to determine the difference between a calculation question and a yes/no question. The last row shows a failure example that was also reported in Chen et al. (2021b). Here, CounterComp does not improve the performance of FinQANet. This particular example requires domain expertise to address, which goes beyond a direct mapping between components in the question and the evidence. This highlights the need for methods that allow domain expertise to be represented more effectively (Chen et al., 2021b).

heading: 6.4 Compositional v.s. OOD generalization
text: In a recent study Joshi and He (2022) showed that current approaches to counterfactual data augmentation do not necessarily lead to better generalization to out-of-distribution (OOD) samples. To test whether this holds for CounterComp, we conduct two studies. First, we train FinQNet with and without CounterComp loss on the FinQA dataset, then test it on the other three datasets. Note that the four datasets are based on different domains. FinQA and TAT-QA are both based on financial reports, but while FinQA was derived from US filings, TAT-QA is based on international filings and
therefore covers a wider variety of metrics. HiTab and MULTIHIERTT are both based on other types of corporate reports with highly complex tabular structures.
The first three columns of Table 8 show a slight improvement when CounterComp is used in this setting. In contrast, using CompAQT loss slightly hurts the performance, demonstrating CounterComp’s higher OOD generalization potential.
Next, we select a subset of samples from the FinQA dev set that have unseen compositions compared to the training set. This means that the particular combination of operations were never seen during training. As the last column of the table shows, CounterComp outperforms the baseline by more than 7 points. This further demonstrates how improving representation learning at the component level can enhance generalization to unseen contexts.

heading: 7 Conclusion
text: In this paper, we presented CounterComp, a method that leverages counterfactual contrast to enable metric learning for quantitative QA. We show how using the auxiliary CounterComp loss can improve compositional generalization in multistep reasoning tasks, especially as the number of steps grows.
Due to runtime challenges, we proposed a hybrid offline/online sampling strategy that uses predefined indices for easier lookup operations. This allows us to capture samples that have a contrast of one operator with the anchor. In future studies, we hope to capture contrastive samples with longer perturbation chains. We also hope to examine the effectiveness of counterfactual compositional contrast in other domains such as semantic parsing and question answering over multimodal input.
Lastly, we hope to extend the use of CounterComp to enhance the performance of the retriever, using the heuristics introduced in Section 4 (i.e. by
focusing on components in the question that overlap with the facts). This can result in a quantitative QA pipeline that is powered by compositional contrast in an end-to-end fashion.

heading: 8 Limitations
text: As previously mentioned, our study is focused on the generator component of a QA pipeline and ignores the retrieval task. In the experiments presented in the paper, we have used gold facts to report the results. For certain datasets such as HiTab and MULTIHIERTT which were designed for complex tabular structures, this might simplify the end-to-end challenge. In future studies, we hope to explore whether CounterComp can enhance the performance of retrievers.
The datasets used in our experiments were curated using enterprise documents such as financial reports or other corporate disclosures. Quantitative QA over these reports often involves multi-step reasoning that is limited to linear arithmetic operations such as addition, division, averaging, etc. A completely open-domain QA engine might need to cover more complex operators.
Lastly, we designed CounterComp to leverage existing data by sampling from the training set. Nevertheless, combining CounterComp with augmentation-focused methods such as CAD might lead to more robust models.

heading: A Results on retrieved facts
text: Table 9 shows the performance of FinQANet versus FinQANet+CounterComp on retrieved facts from the FinQA dataset. Similar to gold facts, CounterComp improves program accuracy, especially on multi-step output.

heading: B Training algorithm
text: Algorithm 1 details our pre-indexing, sampling, and training processes. Note that the algorithm is a simplified version of our implementation, e.g. it follows a basic SGD instead of a batch SGD process, and shows the process for only one epoch.

heading: C CounterComp for operators versus operands
text: CounterComp intervenes on operators, whereas operands provide another possible intervention target. As mentioned in Section 2, Learning to Image (L2I) (Li et al., 2022), which focuses on counterfactual scenarios for operands, was able to outperform TAGOP by a large margin. L2I was evaluated on TAT-QA, a dataset with a limited set of possible multi-step operations, resulting in the challenge of compositional generalization being mainly focused on operands. Since we were not able to recreate the results reported in the original L2I paper6, instead we evaluate an operand-focused approach via a metric learning method similar to CounterComp.
Given an anchor, we generate new samples using the operations laid out in the L2I paper (i.e. SWAP, ADD, MINUS, etc.), where one or more operands are perturbed in random. We apply the same perturbation in the facts. This effectively eliminates the "imagination" component but provides a baseline that is more comparable to CounterComp. These samples are used as positive examples, whereas negative examples are randomly sampled from the batch.
6This could be because we failed to generate a TAT-HQA dataset that was comparable to the one used in the original paper.
Algorithm 1 Training algorithm 1: Training data: {([Q||F ](i), S(i))}Ii=1 2: Parameters: λ 3: Model: model
// Create the indices for pos and neg samples 4: pos_index← {} 5: neg_index← {} 6: for i ∈ {1, . . . , I} do 7: O(i) ← s(i)1 , s(i)4 , · · · , s(i)L−3 8: add_to_index(pos_index, O(i), i)
//p is the perturbed output and l is the location of the perturbation
9: for p, l ∈ possible_perturbations(O(i)) do 10: j ← find_matching_sample(p) 11: add_to_index(neg_index, O(i), (j, l)) 12: end for 13: end for
// Train (single epoch, non-batch version) 14: for i ∈ {1, . . . , I} do 15: for j ∈ {1, 2, · · · , 5} do
// Basic model loss 16: L(i) ← loss(model.forward([Q||F ](i)), S(i))
// Pos/neg sampling 17: O(i) ← s(i)1 , s(i)4 , · · · , s(i)L−3 18: pos_sample← sample(pos_index[O(i)] \ i) 19: neg_sample, l← sample(neg_index[O(i)])
// Find candidate intervention spans 20: Q(i) ← find_intrvntn_span(i) 21: Q(i)pos ← find_intrvntn_span(pos_sample) 22: Q(i)neg ← find_intrvntn_span(neg_sample)
// Calculate edit distances and loss 23: NLD(i)pos ← norm_edit_dist(Q(i),Q(i)pos) 24: NLD(i)neg ← norm_edit_dist(Q(i),Q(i)neg) 25: α(i) = 1− |NLD(i)neg − NLD(i)pos| 26: pos_dist(i)j = ||h (i) l − h (i) l,pos||22 27: neg_dist(i)j = ||h (i) l − h (i) l,neg||22 28: L(i)tripletj = max{pos_dist (i) j −neg_dist (i) j +α
(i), 0} 29: end for 30: L(i) = (1− λ)L(i) + λ
5 ∑5 j=1 L (i) tripletj
31: end for 32: L = 1
I ∑I i=1 L(i)
33: model.backward(L)
Table 10 shows the program accuracy of CounterComp versus the new method when applied to each dataset. As expected, TAT-QA is the only dataset responsive to the perturbation of operands. All other datasets suffer from an exclusive focus on operands. For HiTab and MULTIHIERTT, the operand strategy also underperforms compared to the baseline FinQANet performance (see Table 5).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8
3 A2. Did you discuss any potential risks of your work? 6.3, 6.4, 8
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 5.1, 5.2
3 B1. Did you cite the creators of artifacts you used? 5.1, 5.2
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 5.1, 5.2
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 5.1, 5.2
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The data is based on publicly available corporate reports.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 5.1, 5.2
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 1
C 3 Did you run computational experiments? 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 5
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 6
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No packages used.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future studies, we hope to explore whether CounterComp can enhance the performance of retrievers. The datasets used in our experiments were curated using enterprise documents such as financial reports or other corporate disclosures. Quantitative QA over these reports often involves multi-step reasoning that is limited to linear arithmetic operations such as addition, division, averaging, etc. A completely open-domain QA engine might need to cover more complex operators. Lastly, we designed CounterComp to leverage existing data by sampling from the training set. Nevertheless, combining CounterComp with augmentation-focused methods such as CAD might lead to more robust models