Venue: ACL 2023

Using counterfactual contrast to improve compositional generalization for multi-step quantitative reasoning

Armineh Nourbakhsh

Language Technologies Institute, Carnegie Mellon University

J.P. Morgan AI Research anourbak@cs.cmu.edu

Sameena Shah

J.P. Morgan AI Research sameena.shah@jpmorgan.com &Carolyn Rose

Language Technologies Institute, Carnegie Mellon University cprose@cs.cmu.edu

###### Abstract

In quantitative question answering, compositional generalization is one of the main challenges of state of the art models, especially when longer sequences of reasoning steps are required. In this paper we propose CounterComp, a method that uses counterfactual scenarios to generate samples with compositional contrast. Instead of a data augmentation approach, CounterComp is based on metric learning, which allows for direct sampling from the training set and circumvents the need for additional human labels. Our proposed auxiliary metric learning loss improves the performance of three state of the art models on four recently released datasets. We also show how the approach can improve OOD performance on unseen domains, as well as unseen compositions. Lastly, we demonstrate how the method can lead to better compositional attention patterns during training.

## 1 Introduction

Enterprise documents such as reports, forms, and analytical articles often include quantitative data in tabular form. The data in these tables can be self-contained, but more commonly the surrounding text provides more context that is necessary to understand the content. Answering questions over these hybrid tabular/text contexts requires reasoning that combines verbal and quantitative semantics.

Question answering over quantitative tabular/text data has gained recent traction with the release of datasets such as FinQA Chen et al. (2021), TAT-QA Zhu et al. (2021), and HiTab Cheng et al. (2022). Table 2 shows an example of a question that requires quantitative reasoning to derive the answer. Given the question and the tabular context, the output is a single-step program that leads to the final answer of ~20.

A major challenge that state of the art models face is compositional generalization Montague (1973), especially when the number of reasoning steps grows Chen et al. (2021). In the context of quantitative QA, compositional generalization refers to the model's ability to generalize to new compositions of previously seen elements. As an example, if the model has encountered training examples that demonstrate calculations for "growth rate" and "percent change", we would like it to be able to come up with a reasonable hypothesis as to how to calculate "percent growth" or "rate of change". Table 1 demonstrates how this challenge becomes more difficult as the number of reasoning steps grows. For questions that require longer chains of reasoning, the model learns spurious patterns and unsuccessfully tries to leverage these memorized patterns to solve new problems.

The Table also shows that as the number of steps grows, generating the wrong operator becomes a more dominant mistake than selecting the wrong operand. Not only is this error more dominant, but it can also have a more destructive impact on the chain of reasoning, as it can derail the model's hidden representations from that point onward. As an example, our analysis of the FinQANet model Chen et al. (2021) output showed that if the model generates an incorrect operator, it is about 30% more likely to commit other errors in the following

\begin{table}
\begin{tabular}{l|c|c|c} \hline
**\# steps in** & **\% wrong** & **\% wrong** & **\% wrong order** \\
**output** & **operator(s)** & **operators** & **of operands** \\ \hline
1 step & 39.07 & 53.64 & 7.28 \\ \hline
2 steps & 46.75 & 46.75 & 6.50 \\ \hline
3 steps & 56.47 & 29.41 & 14.12 \\ \hline ¿= 4 steps & 52.00 & 40.00 & 8.00 \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \end{tabular}
\end{table}
Table 1: Share of FinQANet errors due to the selection of wrong operators or operands when applied to the FinQA dataset Chen et al. (2021), broken down by the number of steps in the output. Note that the numbers are based on program accuracy, which accounts for each type of error separately, resulting in the each row summing up to 1.

 

steps compared to when the model generates an incorrect operand.

In this paper, we propose CounterComp, an approach that can enhance compositional learning in multi-step quantitative QA. We take inspiration from the symbolic composition of arithmetic operations, and their correspondence to natural language phrases. Building on the work on attention alignments from previous studies, we propose an auxiliary metric learning loss that is focused on specific components of the input and output. Our sampling strategy is based on counterfactual scenarios. This means that the model learns proper representations for each component based on what-if scenarios. To the best of our knowledge, this is the first study that successfully applies component-wise counterfactual sampling as a metric learning strategy. We show how, when state of the art models are augmented with our auxiliary metric learning loss, they exhibit better performance in cases where multi-step reasoning is required. CounterComp outperforms current baselines on four recently released datasets, and show stronger performance on OOD samples.

## 2 Related work

The typical architecture of a quantitative QA model is composed of a retriever and a generator Jurafsy and Martin (2021). The retriever identifies the particular context where the answer might be found. Since the context can be a mix of table cells and sentences, often a tabular encoder Herzig et al. (2020) or verbalizer Chen et al. (2021) is used to convert the cells into a natural language sequence. The retrieved context is referred to as retrieved _facts_. Next, the generator uses the question along with the facts to generate the output in a step by step fashion. In multi-step QA, the generator often combines a recurrent module with an attention mechanism Chen et al. (2021), as illustrated in top half of Figure 1.

The output can be assessed in terms of program accuracy as well as execution accuracy. Our study is focused on improving program accuracy by encouraging compositional generalization in the generator.

There are two common approaches to improving

\begin{table}
\begin{tabular}{l|c|c|c|c|} \hline \multirow{3}{*}{**Question**} & \multicolumn{4}{c}{What was the net change in} \\  & revenue from 2019 to 2020? & \multicolumn{1}{c}{} \\ \cline{1-5}  & **Metric (SM)** & **2018** & **2019** & **2020** \\ \cline{2-6}
**Tabular context** & Operating expenses & 35 & 29 & 30 \\ \cline{2-6}  & Revenue & 70 & 80 & 60 \\ \hline
**Verbalized facts** & 2019 revenue was $80M. & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{1-5}
**Output program** & subtract(80, 60) & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \hline
**Answer** & -20 & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 2: Example of a quantitative QA problem over tabular data.

Figure 1: A high-level illustration of our proposed method. The input example (anchor) is processed by cross-attention and recurrent modules to produce the output step by step. In addition to a regular Cross-Entropy loss, CounterComp adds an auxiliary triplet loss based on positive and negative examples. Note that the anchor and pos/neg examples are all processed through the RNN before calculating the triplet loss, a process which we have not illustrated due to space limitations. Also note that multiple pos/neg examples are sampled at each step.

 

compositional generalization. Attention alignment models encourage explicit alignments between natural language utterances (e.g. "rate of change") and corresponding symbolic math operations (e.g. subtraction followed by division). Methods informed by counterfactuals use what-if scenarios to generalize to a wider variety of compositions and reduce the effect of memorization.

### Attention alignments

Yin et al. (2021) showed that additional supervision can be used to promote explicit alignments between components in the input and in the output. They added a regularization loss that encourages the cross-attention module to adjust its attention weights according to gold alignments. Using as few as 16 examples, their model was able to improve generalization in a semantic parsing task. CompAQT Nourbakhsh et al. (2022) extended this idea to multi-step quantitative QA. Instead of using additional supervision, it used natural language heuristics to create noisy alignment labels between input tokens and output symbols. The additional alignment loss improved the performance of three baseline models on multi-step reasoning tasks for four datasets.

### Methods informed by counterfactuals

The success of alignment-based methods is limited by the fact that by heavily discouraging memorization, they underperform in settings where memorization can be helpful Oren et al. (2020). To strike a balance between memorization and generalization, one approach is to generate new training examples that cover important semantic gaps in the training data. This is reminiscent of how adversarial training can help better define the semantic contours of compositional representations Zhang et al. (2022). Contrastive or metric learning methods pursue a similar goal, but instead of generating new samples, they leverage existing samples within the training set Jain et al. (2021).

Counterfactual data augmentation (CAD) methods strive to achieve this by generating new samples using what-if scenarios Zmigrod et al. (2019); Liu et al. (2021); Chen et al. (2021). This can be done by altering a minimally sufficient set of tokens in the input such that the output class changes Kaushik et al. (2020). There are two main challenges to creating these samples. First, it is difficult to identify the minimal set of tokens necessary to alter the output. Second, there is no guarantee that a counterfactual sample exists in the training set. To address these challenges, some studies employ human labelers Kaushik et al. (2020) or a third party model Huang et al. (2020). In domains like semantic parsing and quantitative QA where the output is symbolic, an alternative approach leverages the structure of the output to avoid the need for human labelers Li et al. (2022) achieve this by intervening on the operands. Suppose that a question states "What was the net change in revenue from 2019 to 2020?" and the retriever produces two (verbalized) table cells: "2019 revenue was $80M" and "2020 revenue was $60M". The output program for this question would be: subtract(80, 60). Given the numeric nature of the operands, it's possible to generate new scenarios such as "What if 2019 revenue was $90?" with the updated output subtract(90, 60). Employing this method, Li et al. (2022) augment the TAT-QA dataset Zhu et al. (2021) into a new dataset named TAT-HQa. They also enhance the verbal reasoning capacity of their model by offering the counterfactual scenario as a natural language prompt. Their model, named Learning to Imagine (L2I), outperforms state of the art models.

As mentioned in the previous section, models that struggle with compositional generalization suffer from errors in operator selection, whereas L2I is focused on the selection of operands. In this paper, we propose CounterComp, a method that focuses on counterfactual sampling for components that indicate operators1. Using natural language constraints from previous studies, we first find components that correspond to operators versus those that correspond to operands. Next, we use an auxiliary metric learning loss with positive and negative samples chosen based on those components. This helps us avoid the complexities associated with a data augmentation approach, such as the need for creation of additional human labels. The next section lays out our problem definition in more detail.

Footnote 1: Please refer to Appendix C for a study on the use of CounterComp for operators versus operands.

## 3 Problem formulation

Let us consider the example provided by Table 3. Suppose \(Q\) is the question, represented as a sequence of tokens \(q_{1},\cdots,q_{N}\) (i.e. "what", "was", "the", ..., "2020", "?").

\(F\) is the evidence obtained by the retriever, made up of a sequence of tokens \(f_{1},\cdots,f_{M}\) (i.e. "2019", 

"revenue", "was", ..., "$60M").

The concatenation of these two sequences, i.e. \(Q||F\), forms the input to the generator. The generator encodes \(Q||F\) using a neural language model such as RoBERTa Liu et al. (2019), resulting in an embedding matrix \({\mathbf{U}}\in\mathbb{R}^{d_{\text{enc}}\times(N+M)}\).

Consistent with Chen et al. (2021b), we represent the output \(S\) as a sequence of steps \(s_{1},\cdots,s_{L}\). Each step \(s_{l}\) can be an operator (such as add or divide), or an operand. Similar to Chen et al. (2021b), our programs are modeled as right-expanding binary trees with each operator having exactly two operands. If necessary, one or more operands are set to NONE, where NONE is a special constant. \(L\) is pre-defined as the maximum number of steps allowed. In the example from Table 3, \(S\) is: subtract, 80, 60, NONE, NONE, NONE.

To generate the \(l\)th output step \(s_{l}\), the generator applies a cross-attention module to \({\mathbf{U}}\), resulting in the attention weight matrix \({\mathbf{A}}_{l}\in\mathbb{R}^{1\times(N+M)}\) and the attention output \({\mathbf{X}}_{l}\in\mathbb{R}^{K}\). A recurrent module then generates the hidden vector \({\bf h}_{l}\), which is used to produce the output step \(s_{l}\).

\[\begin{split}&{\bf h}_{l}=\text{RNN}({\bf h}_{l-1},{\mathbf{X}}_{ l})\\ &{\bf s}_{l}=\text{NN}({\bf h}_{l})\end{split} \tag{1}\]

where NN can be any neural module that projects \({\bf h}_{l}\) onto the simplex \({\bf s}_{l}\in\mathbb{R}^{K}\), from which \(s_{l}\) can be sampled: \(s_{l}=\arg\max_{k}{\bf s}_{l,k}\). Our goal is to encourage \({\bf h}_{l}\) to be sensitive to the composition of the input \(Q||F\) with regards to the current output step \({\bf s}_{l}\). This means that \({\bf h}_{l}\) needs to capture proper alignments between important terms in the input and the relevant operator/operand in the output.

To achieve this, we pursue a metric learning approach where positive and negative samples are generated according to counterfactual scenarios.

### Counterfactual samples

Given a training example \(([Q||F]^{(i)},S^{(i)})\), we define an _intervention target_\(\mathcal{Q}^{(i)}\) as a subsequence of the question tokens, i.e. \(\mathcal{Q}^{(i)}=\{q_{n}^{(i)};n\in\mathcal{N}^{(i)}\}\) where \(\mathcal{N}^{(i)}\subseteq\{1,2,\cdots,N\}\).

Suppose that changing the intervention target affects a single step in the output program \(\mathcal{S}^{(i)}=s_{l}^{(i)}\), which we name the _intervention outcome_. Note that due to our focus on the generation of operators, we limit the intervention outcome to an operator. Since the output is composed of one operator followed by two operands followed by another operator and so on, \(l\) is selected from a limited index set: \(l\in\{1,4,7,\cdots,L-3\}\). In the example from Table 3, the possible indices will be 1 and 4, representing the operators subtract and None.

Given this definition, it's possible to mine positive and negative examples for the \(i\)th training instance. A positive example \(([Q||F]^{(i)}_{\text{pos}},S^{(i)}_{\text{pos}})\) is an instance for which, despite a possible intervention in the target, the outcome remains the same, i.e. \(\mathcal{Q}^{(i)}_{\text{pos}}\neq\mathcal{Q}^{(i)}\) and \(\mathcal{S}^{(i)}_{\text{pos}}=\mathcal{S}^{(i)}\). A negative example \(([Q||F]^{(i)}_{\text{neg}},S^{(i)}_{\text{neg}})\) is an instance for which an intervention in the target leads to a change in the outcome, i.e. \(\mathcal{Q}^{(i)}_{\text{neg}}\neq\mathcal{Q}^{(i)}\) and \(\mathcal{S}^{(i)}_{\text{neg}}\neq\mathcal{S}^{(i)}\).

This allows us to define a triplet loss that encourages \({\bf h}_{l}^{(i)}\) to remain close to \({\bf h}_{l,\text{pos}}^{(i)}\) and far from \({\bf h}_{l,\text{neg}}^{(i)}\) with a margin of \(\alpha^{(i)}\):

\[\begin{split}&\mathcal{L}^{(i)}_{\text{triplet}}=\\ &\max\{||{\bf h}_{l}^{(i)}-{\bf h}_{l,\text{pos}}^{(i)}||_{2}^{2 }-||{\bf h}_{l}^{(i)}-{\bf h}_{l,\text{neg}}^{(i)}||_{2}^{2}+\alpha^{(i)},0\} \end{split} \tag{2}\]

Figure 1 illustrates the sampling process for one training example. Note that this metric learning approach will only be valid if causal assumptions with regards to the intervention target are valid, i.e. the change in \(\mathcal{S}^{(i)}_{\text{neg}}\) is in fact the result of the intervention in \(\mathcal{Q}^{(i)}_{\text{neg}}\) and not a change in any other part of the input. In a data augmentation setting, this can be achieved by keeping the input fixed and perturbing a small segment that functions as the intervention target similar to Kaushik et al. (2020). However, as discussed in Section 2.2. this requires additional manual labor to annotate the perturbed examples.

In the next section, we describe how we impose certain constraints on the intervention target to achieve this in a self-supervised setting2.

Footnote 2: Note that the term “self-supervised” is used in this context to refer to the sampling strategy, i.e. no additional labeling is needed to generate the positive and negative samples.

## 4 Methodology

Our goal is to identify potential positive and negative samples for the anchor \(([Q||F]^{(i)},S^{(i)})\). Suppose the anchor is the one shown in the top three rows of Table 3. _Bold italiczed_tokens are redundant between the question and the fact, e.g. "revenue", "2019", and "2020". Those terms are often used by the retriever to find the correct facts. They are also used by the generator to find the correct order of operands.

 

There are also terms that are unique to the question, i.e. "What", "the net change to", "from", and "to" (highlighted in blue). In CompAQT, the authors showed that these can be used as indicators for the operators. Lastly, there are terms that are unique to the facts, i.e. "was $80M" and "was $60M" (red italicized tokens). These can be used as indicators for the operands. We use these heuristics to guide our sampling strategy.

We flag all spans in the question that do not overlap with the facts, i.e. underlined blue segments. Those spans serve as candidate intervention spans. In the example from Table 3, this results in four candidates: "What", "the net change in", "from", and "to".

Next, we seek a positive and a negative example within the training set. A positive example is a sample in which, despite possible changes in the question, the operators in the output remain consistent with the operators in the anchor. Table 3 shows one such example. Several terms have been altered in the question. However, we would only focus on the changes in the candidate spans. Here, "was" has changed to "is", "net change" has changed to "difference", "from" to "between" and "to" to "and". This results in a token-level Levenshtein distance of 5 (four edits and one insertion) [21]. We ignore the change from "revenue" to "operating expenses" and from "2019" to "2018", because those changes have occurred outside of our candidate spans and only correspond to operands.

A negative example is a sample in which exactly one output operator is altered, deleted, or added. Table 3 shows one such example. Here, the output includes a new operator divide. The question has also been altered with a token-level Levenshtein distance of 4.

The given positive and negative example can now be plugged into Equation 2. Instead of a fixed margin, we use the edit distances mentioned before to dynamically adjust the margin. Let NLD\({}_{\text{pos}}^{(i)}\) and NLD\({}_{\text{neg}}^{(i)}\) be the _normalized, token-level Levenshtein edit distance_ between the anchor and the positive example, and the negative example, respectively. We set the margin to: \(\alpha^{(i)}=1-|\text{NLD}_{\text{neg}}^{(i)}-\text{NLD}_{\text{pos}}^{(i)}|\)

This encourages a larger margin for cases where the anchor is equally similar to the positive and the negative examples, and the model might have a harder time picking up on the nuances of each component.

### Runtime optimization

There are two runtime challenges to this proposed approach: 1) Sampling can be costly if the entire training set has to be scanned for each batch. This means an online sampling strategy cannot be used. On the other hand, an offline strategy introduces a large overhead. A hybrid approach is needed. 2) Calculating the edit distance metric is a costly operation with \(O(n^{2})\) steps.

To solve the first problem, we build two indices prior to training. One index groups the samples by their sequence of output operators. This index can be used to sample positive examples.

The other index includes all training examples, and for each example, it includes the full list of one-step perturbations applied to its output operators. By generating all possible perturbations, we are able to find other samples whose outputs match the perturbed sequence (i.e. negative samples). For

\begin{table}
\begin{tabular}{l|p{71.1pt}|c} \hline \multirow{2}{*}{**Anchor**} & **Question** & What are the change in _retense_ \\  & & _from 2019_ / _2020?_ \\ \cline{2-3}  & **Facts** & _2019_ _recure was $80Mi_ \\  & & _2020_ _prereme were $56Mi_ \\ \cline{2-3}  & **Program** & subtract(50, 60) \\  & & NME(ONE, , None) \\ \hline \multirow{3}{*}{**Negative sample**} & **Condidate interaction spans** & What are the edit change in from \\  & & _from \\  & & _to_ \\ \hline \multirow{3}{*}{**Positive sample**} & **Question** & What is the difference in _opening expenses_ between **2018_ / _2020?_ \\  & **Facts** & _2018 operating expenses_ were $53Mi_ \\  & & _2020_ _operating expenses_ were $50Mi_ \\  & & Subtract(35, 30) \\  & & NME(None, None) \\ \hline \multirow{3}{*}{**Negative sample**} & **Question** & What are the ratio of change of _opearing income from 2018_ to **2019?_ \\  & **Facts** & _2018 incoming room operating activities_ **see $63Mi_ \\  & & _2019__income from operating activities_ **van $60Mi_ \\  & & Fitback(35, 60) \\  & & Update(60, 50) \\ \hline \multirow{3}{*}{**Variables**} & \(\mathcal{Q}^{(i)}\) & \(\{q_{1}^{(i)},q_{1}^{(i)},q_{1}^{(i)},q_{2}^{(i)},q_{3}^{(i)},q_{4}^{(i)}, q_{4}^{(i)},q_{4}^{(i)}\}\); \\  & & what the net change in from to \\  & \(S^{(i)}_{s}\); & Noise \\ \cline{2-3}  & \(\overline{Q}^{(i)}_{\text{max}}\) & \(\{q_{1}^{(i)},q_{1}^{(i)},q_{4}^{(i)},q_{4}^{(i)},q_{4}^{(i)},q_{4}^{(i)},q_{4}^{(i)}\}\); \\  & & what is the difference in between and \\  & \(S^{(i)}_{\text{min}}\) & \(s^{(i)}_{t}\); None \\ \cline{2-3}  & \(\underline{\text{effit dist}}\) & \(\mathbf{5}\) \\ \hline \multirow{3}{*}{\(\overline{Q}^{(i)}_{\text{max}}\)} & \(\{\overline{Q}^{(i)}_{\text{max}}\)} & \(\{\overline{Q}^{(i)}_{\text{max}},q_{4}^{(i)},q_{4}^{(i)},q_{4}^{(i)},q_{4}^{(i)}, q_{4}^{(i)},q_{4}^{(i)}\}\); \\  & & what the rate of change of no \\ \cline{1-1}  & \(S^{(i)}_{\text{log}}\) & \(s^{(i)}_{t}\); div side \\ \cline{1-1}  & \(\text{edit dist}\) & 4 \\ \hline \end{tabular}
\end{table}
Table 3: Example of positive and negative sampling using counterfactual components. Blue underlined text indicates components that are unique to the question ( candidates for intervention). These terms often indicate an operator. _Red italicized_ text indicates terms that are unique to the facts. These terms often indicate operands. _Bold italicized text_ indicates terms that are shared between the question and facts. These terms often indicate metrics.

 

a sequence with \(n\) operators, all possible perturbations can be generated in \(O(n\times K)\) time, where \(K\) is the number of possible operators3.

Footnote 2: Since we follow Chen et al. (2018), in all of our experiments \(K=10\).

Given the pre-generated positive and negative pools, we can also calculate and cache edit distances ahead of time. However, in practice, we realized that we could do so during training with little additional cost. This is because the edit distance is applied at the token-level4, and is limited to candidate spans, rendering it relatively fast. The decision as to whether distances should be cached or calculated on the fly depends on the average size of each pool versus the number of training steps.

Footnote 4: Since we’re using a language model that uses word-piece tokenization, in effect the runtime is at subword level.

The algorithm outlined in Appendix B summarizes our approach.

## 5 Experiments

### Datasets

We use the hybrid CompAQT dataset, which is composed of four previously released datasets, namely **FinQA**Chen et al. (2021), **TAT-QA**Zhu et al. (2021), **HiTab**Cheng et al. (2022), and **MultiHiertt**Zhao et al. (2022). The authors filtered these four datasets down to QA pairs that require single or multi-step quantitative reasoning. They also processed the tables and outputs in all four datasets to match the FinQA format.

### Baselines

We apply our proposed auxiliary loss to three baselines: 1) **FinQANet**, originally developed for the FinQA dataset (Chen et al., 2021). 2) **TagOp**, originally developed for the TAT-QA dataset Zhu et al. (2021). 3) **Pointer-Verbalizer Network**PVN), originally proposed by Nourbakhsh et al. (2022). We also apply the **CompAQT** loss to each model as a secondary baseline in order to determine how CounterComp compares to an attention-alignment strategy.

### Sampling success rate

Another possible concern is that our sampling strategy might be limited, in that positive and negative samples might not always be available in the training set, or that limited availability of samples might bias the training process. To remediate the problem of unavailable samples, when a positive sample is missing, we use the anchor as the positive sample, and when a negative sample cannot be found, we use a uniformly sampled instance from the batch.

Table 4 shows some statistics about the success rate of the sampling algorithm. "% Failure" identifies the share of training examples for which either a positive or a negative example was missing. Unsurprisingly this never happens for single-step programs, is very rare for two-step programs, and with the exception of HiTab, happens in less than 10% of the cases for longer programs. The Table also shows the average number of positive and negative examples found for each anchor. Again, HiTab has the lowest number of available samples, making it the most challenging dataset. In Section 6.4, we demonstrate how, even in cases with few possible samples, the model is able to generalize to unseen examples.

### Settings

Since we are focused on the generator, in the experiments discussed in this section we will use gold facts and encode the input using RoBERTalarge Liu et al. (2019)5. We run the baselines with and without the additional \({\cal L}_{\text{triplet}}\) for 50 epochs with a learning rate of \(5\mathrm{e}{-5}\), the Adam optimizer Kingma and Ba (2015) with \(\beta_{1}=0.9\) and \(\beta_{2}=0.999\). At each step, we sample (with replacement) 5 positive and negative pairs per anchor, and add the average auxiliary triplet loss to the main model loss with a weight of \(\lambda\). After a grid search with a step-size of 0.1, we set \(\lambda\) to 0.4 for all experiments. All experiments were conducted on 8 NVIDIA T4 GPUs with 16 GBs of memory.

Footnote 5: Please refer to Appendix A for results using retrieved facts.

## 6 Results and analysis

Table 5 shows the program accuracy of baselines (top row of each cell) compared the addition of CounterComp loss (bottom row of each-cell). Among the baseline models, TagOp is not designed to generate multi-step programs. Therefore we only apply it to the TAT-QA dataset, which has a set of pre-determined operations (e.g. change ratio). We also apply the PVN model to the combined dataset, but since FinQANet outperforms it on all benchmarks, we will continue to use FineQANet as the reference baseline model for the remaining experiments in this section.

As Table 5 shows, CounterComp consistently 

outperforms the baselines and the margin is often higher for longer programs. One notable exception is the TAT-QA dataset. As mentioned before, the dataset is not designed for open-ended multi-step reasoning and includes a limited set of possible operations. Therefore methods that encourage memorization might achieve higher performance on TAT-QA. HiTab is another challenging dataset, but despite low performance on longer sequences, CounterComp offers an improvement over the baseline.

### Auxiliary triplet loss versus auxiliary attention alignment loss

The middle row of each cell in Table 5 shows the program accuracy when CompAQT loss is added instead of CounterComp loss. As previously described, CompAQT imposes an auxiliary attention alignment loss such that tokens related to operators receive more attention during the generation of operators. Even though this leads to improvements over the baselines, CounterComp outperforms CompAQT in all experiments. This might be due to the fact that the regularizing effect of CompAQT loss is not as strong as the representation learning impact of CounterComp.

Despite the fact that CounterComp was not designed as an attention alignment model, it does have an impact on how attention patterns evolve during training. Table 6 shows the top-attended input tokens during the generation of a divide operator in various contexts. For a singular division operation, FinQANet attends to tokens such as "year" whereas CounterComp encourages the model to attend to more relevant tokens such as "net" and "change". A subtraction followed by a division often indicates a percentage calculation, as captured by both models. An addition followed by a division often indicates an average calculation. Again, CounterComp is able to capture relevant tokens but the FinQANet baseline seems to attend to some memorized tokens such as "annual".

### Fixed versus adaptive margin

The last row of Table 5 shows the performance of the FinQANet model on the combined dataset using the CounterComp loss with a fixed margin of 1. The performance suffers, especially as the number of steps grows. This further demonstrates the importance of the adaptive margin \(\alpha^{(i)}\) that takes the edit distance into account.

\begin{table}
\begin{tabular}{l|c|c|c} \hline  & \multicolumn{3}{c}{**Top attended tokens during**} \\  & \multicolumn{2}{c}{**the generation of divide**} \\ \cline{2-4}
**Model** & **divide** & **subtract** & **add** \\  & **divide** & **divide** & **divide** \\ \hline FinQANet & share, year & ratio, percent & annual, per \\ \hline +CounterComp & net, change & share, percent & average, per \\ \hline \end{tabular}
\end{table}
Table 6: Top attended tokens during the generation of the division operator in various sequences. The dataset used for this experiment is FinQA.

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c|c|c} \hline \multirow{2}{*}{**Dataset**} & \multicolumn{7}{c|}{**1 step**} & \multicolumn{8}{c|}{**2 steps**} & \multicolumn{3}{c}{**3+ steps**} \\ \cline{2-9}  & **\% Failure** & **Avg. \# pos** & **Avg. \# neg** & **vg. \# res** & **Avg. \# pes** & **Avg. \# pos** & **Avg. \# neg** & **samples** \\ \hline
**FinQA** & 0 & 1457 & 3254 & 0.2 & 913 & 630 & 8.2 & 41 & 190 \\ \hline
**TAT-QA** & 0 & 1808 & 638 & 0 & 295 & 958 & 1.3 & 1055 & 66 \\ \hline
**HiTab** & 0 & 221 & 554 & 2.5 & 30 & 29 & 29.8 & 4 & 24 \\ \hline
**MultiHertt** & 0 & 189 & 533 & 0.2 & 326 & 335 & 7.8 & 92 & 190 \\ \hline \end{tabular}
\end{table}
Table 4: The failure rate of sampling from each dataset (when no positive or no negative sample can be found for a given anchor), as well as the average number of positive and negative samples found for each anchor.

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c|c} \hline \multirow{2}{*}{**Dataset**} & \multicolumn{5}{c|}{**1 step**} & \multicolumn{5}{c|}{**2 steps**} & \multicolumn{3}{c}{**3+ steps**} \\ \cline{2-9}  & **\% Failure** & **Avg. \# pos** & **Avg. \# neg** & **vg. \# res** & **Avg. \# pos** & **Avg. \# neg** & **Avg. \# neg** \\ \hline
**FanQ** & **sample** & **samples** & **values** & **samples** & **samples** & **samples** & **samples** \\ \hline
**FinQA** & 0 & 1457 & 3254 & 0.2 & 913 & 630 & 8.2 & 41 & 190 \\ \hline
**TAI-QA** & 0 & 1808 & 638 & 0 & 295 & 958 & 1.3 & 1055 & 66 \\ \hline
**HiTab** & 0 & 221 & 554 & 2.5 & 30 & 29 & 29.8 & 4 & 24 \\ \hline
**MultiHertt** & 0 & 189 & 533 & 0.2 & 326 & 335 & 7.8 & 92 & 190 \\ \hline \end{tabular}
\end{table}
Table 4: The failure rate of sampling from each dataset (when no positive or no negative sample can be found for a given anchor), as well as the average number of positive and negative samples found for each anchor.

 



### Qualitative examples

Table 7 shows four qualitative examples from the FinQA dataset. The first two rows show how CounterComp enables the FinQANet model to represent concepts such as "decline" and "percentage" more accurately. The third example shows how CounterComp is able to determine the difference between a calculation question and a yes/no question. The last row shows a failure example that was also reported in Chen et al. (2021). Here, CounterComp does not improve the performance of FinQANet. This particular example requires domain expertise to address, which goes beyond a direct mapping between components in the question and the evidence. This highlights the need for methods that allow domain expertise to be represented more effectively [3].

### Compositional v.s. OOD generalization

In a recent study Joshi and He (2022) showed that current approaches to counterfactual data augmentation do not necessarily lead to better generalization to out-of-distribution (OOD) samples. To test whether this holds for CounterComp, we conduct two studies. First, we train FinQNet with and without CounterComp loss on the FinQA dataset, then test it on the other three datasets. Note that the four datasets are based on different domains. FinQA and TAT-QA are both based on financial reports, but while FinQA was derived from US filings, TAT-QA is based on international filings and therefore covers a wider variety of metrics. HiTab and MultiHiertt are both based on other types of corporate reports with highly complex tabular structures.

The first three columns of Table 8 show a slight improvement when CounterComp is used in this setting. In contrast, using CompAQT loss slightly hurts the performance, demonstrating CounterComp's higher OOD generalization potential.

Next, we select a subset of samples from the FinQA dev set that have unseen compositions compared to the training set. This means that the particular combination of operations were never seen during training. As the last column of the table shows, CounterComp outperforms the baseline by more than 7 points. This further demonstrates how improving representation learning at the component level can enhance generalization to unseen contexts.

## 7 Conclusion

In this paper, we presented CounterComp, a method that leverages counterfactual contrast to enable metric learning for quantitative QA. We show how using the auxiliary CounterComp loss can improve compositional generalization in multistep reasoning tasks, especially as the number of steps grows.

Due to runtime challenges, we proposed a hybrid offline/online sampling strategy that uses predefined indices for easier lookup operations. This allows us to capture samples that have a contrast of one operator with the anchor. In future studies, we hope to capture contrastive samples with longer perturbation chains. We also hope to examine the effectiveness of counterfactual compositional contrast in other domains such as semantic parsing and question answering over multimodal input.

Lastly, we hope to extend the use of CounterComp to enhance the performance of the retriever, using the heuristics introduced in Section 4 (i.e. by

\begin{table}
\begin{tabular}{l|p{56.9pt} p{56.9pt}|p{56.9pt}} \hline
**Question** & **Kelience** & **Goldgram** & **FingQuANet + CounterComp** \\ What was the gross margin decline in fact & 1) the gross ranging per 2004 is 27.3% & subtract(27.5, 27.3) divided(40, 27.5) \\ What percentage of amounts exposed in 2009 comes from heterogeneity company correlation? & 2) average includes a discretionary contributor

focusing on components in the question that overlap with the facts). This can result in a quantitative QA pipeline that is powered by compositional contrast in an end-to-end fashion.

## 8 Limitations

As previously mentioned, our study is focused on the generator component of a QA pipeline and ignores the retrieval task. In the experiments presented in the paper, we have used gold facts to report the results. For certain datasets such as HiTab and MultiHiertt which were designed for complex tabular structures, this might simplify the end-to-end challenge. In future studies, we hope to explore whether CounterComp can enhance the performance of retrievers.

The datasets used in our experiments were curated using enterprise documents such as financial reports or other corporate disclosures. Quantitative QA over these reports often involves multi-step reasoning that is limited to linear arithmetic operations such as addition, division, averaging, etc. A completely open-domain QA engine might need to cover more complex operators.

Lastly, we designed CounterComp to leverage existing data by sampling from the training set. Nevertheless, combining CounterComp with augmentation-focused methods such as CAD might lead to more robust models.

## References

* T. Berg-Kirkpatrick, D. Burkett, and D. Klein (2012)An empirical investigation of statistical significance in NLP. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Jeju Island, Korea, pp. 995-1005. External Links: Link Cited by: SS3.4.
* H. Chen, R. Xia, and J. Yu (2021)Reinforced counterfactual data augmentation for dual sentiment classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 269-278. External Links: Link Cited by: SS3.4.
* Z. Chen, W. Chen, C. Smiley, S. Shah, I. Borova, D. Langdon, R. Moussa, M. Beane, T. Huang, B. Routledge, and W. Y. Wang (2021)Finaq: a dataset of numerical reasoning over financial data. Proceedings of EMNLP 2021. Cited by: SS3.4.
* Z. Cheng, H. Dong, Z. Wang, R. Jia, J. Guo, Y. Gao, S. Han, J. Lou, and D. Zhang (2022)HiTab: a hierarchical table dataset for question answering and natural language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 1094-1110. External Links: Link, Document Cited by: SS3.4.
* J. Herzig, P. Krzysztof Nowak, T. Muller, F. Piccinno, and J. Eisenschlos (2020)TaPaS: weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 4320-4333. External Links: Link, Document Cited by: SS3.4.
* Po-Sen Huang, H. Zhang, R. Jiang, R. Stanforth, J. Welbl, J. Rae, V. Maini, D. Yogatama, and P. Kohli (2020)Reducing sentiment bias in language models via counterfactual evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online, pp. 65-83. External Links: Link, Document Cited by: SS3.4.
* P. Jain, A. Jain, T. Zhang, P. Abbeel, J. Gonzalez, and I. Stoica (2021)Contrastive code representation learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 5954-5971. External Links: Link, Document Cited by: SS3.4.
* N. Joshi and He He (2022)An investigation of the (in)effectiveness of counterfactually augmented data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 3668-3681. External Links: Link, Document Cited by: SS3.4.
* D. Jurafsy and J. H. Martin (2021)Speech and language processing. 3 edition, chapter 23. Cited by: SS3.4.
* D. Kaushik, E. Hovy, and Z. Lipton (2020)Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations, Cited by: SS3.4.
* D. P. Kingma and J. Ba (2015)Adam: a method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Cited by: SS3.4.
* M. Li, F. Feng, H. Zhang, X. He, F. Zhu, and T. Chua (2022)Learning to imagine: integrating counterfactual thinking in neural discrete reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 57-69. External Links: Link, Document Cited by: SS3.4.
* Q. Liu, M. Kusner, and P. Blunsom (2021)Counterfactual data augmentation for neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for 

Computational Linguistics: Human Language Technologies_, pages 187-197, Online. Association for Computational Linguistics.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _ArXiv_, abs/1907.11692.
* Montague (1973) Richard Montague. 1973. _The Proper Treatment of Quantification in Ordinary English_, pages 221-242. Springer Netherlands, Dordrecht.
* Nourbakhsh et al. (2022) Armineh Nourbakhsh, Cathy Jiao, Sameena Shah, and Carolyn Rose. 2022. Improving compositional generalization for multi-step quantitative reasoning in question answering. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 1916-1932. Association for Computational Linguistics.
* Oren et al. (2020) Inbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner, and Jonathan Berant. 2020. Improving compositional generalization in semantic parsing. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 2482-2495, Online. Association for Computational Linguistics.
* Yin et al. (2021) Pengcheng Yin, Hao Fang, Graham Neubig, Adam Pauls, Emmanouil Antonios Platanios, Yu Su, Sam Thomson, and Jacob Andreas. 2021. Compositional generalization for neural semantic parsing via span-level supervised attention. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2810-2823, Online. Association for Computational Linguistics.
* Yujian and Bo (2007) Li Yujian and Liu Bo. 2007. A normalized levenshtein distance metric. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 29(6):1091-1095.
* Zhang et al. (2022) Le Zhang, Zichao Yang, and Diyi Yang. 2022. TreeMix: Compositional constituency-based data augmentation for natural language understanding. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5243-5258, Seattle, United States. Association for Computational Linguistics.
* Zhao et al. (2022) Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. 2022. MultiHiertt: Numerical reasoning over multi hierarchical tabular and textual data. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 6588-6600, Dublin, Ireland. Association for Computational Linguistics.
* Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and TatSeng Chua. 2021. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3277-3287, Online. Association for Computational Linguistics.
* Zamigrod et al. (2019) Ran Zamigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. 2019. Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1651-1661, Florence, Italy. Association for Computational Linguistics.

 

Results on retrieved facts

Table 9 shows the performance of FinQANet versus FinQANet+CounterComp on retrieved facts from the FinQA dataset. Similar to gold facts, CounterComp improves program accuracy, especially on multi-step output.

## Appendix B Training algorithm

Algorithm 1 details our pre-indexing, sampling, and training processes. Note that the algorithm is a simplified version of our implementation, e.g. it follows a basic SGD instead of a batch SGD process, and shows the process for only one epoch.

```
1:Training data:\(\{([Q||F]^{(i)},S^{(i)})\}_{i=1}^{I}\)
2:Parameters:\(\lambda\)
3:Model:model//Createtheindicesforsposandnegsamples
4:pos_index \(\leftarrow\{\}\)
5:neg_index \(\leftarrow\{\}\)
6:for\(i\in\{1,\dots,I\}\)do
7:\(O^{(i)}\gets s_{1}^{(i)},s_{4}^{(i)},\cdots,s_{L-3}^{(i)}\)
8:add_to_index(pos_index, \(O^{(i)},i\))  /p is the perturbedoutputand\(l\)is thelocationofthe perturbation
9:for\(p,l\in\text{possible\_perturbations}(O^{(i)})\)do
10:\(j\leftarrow\text{find\_matching\_sample}(p)\)
11: add_to_index(neg_index, \(O^{(l)},(j,l)\))
12:endfor
13:endfor
14://Train(singleepoch,non-batchversion)
15:for\(i\in\{1,\dots,I\}\)do
16:for\(j\in\{1,2,\cdots,5\}\)do
17:/Basicmodelloss
18:\(L^{(i)}\leftarrow\text{loss}(\textit{model},\textit{forward}([Q||F]^{(i)}), S^{(i)})\)  / Postrangesampling
19:\(O^{(i)}\gets s_{1}^{(i)},s_{4}^{(i)},\cdots,s_{L-3}^{(i)}\)
20: pos_sample \(\leftarrow\text{sample}(\textit{poss\_index}|O^{(i)}|\setminus i)\)
21: neg_sample, \(l\leftarrow\text{sample}(\textit{neg\_index}|O^{(i)}|)\)   / Find candidateinterventionspans
22:\(\mathcal{Q}^{(i)}\leftarrow\text{find\_intrvrntn\_span}(i)\)
23:\(\mathcal{Q}^{(i)}_{\Theta}\leftarrow\text{find\_intrvrntn\_span}(\textit{pos\_ sample})\)
24:\(\mathcal{Q}^{(i)}_{\Theta}\leftarrow\text{find\_intrvrntn\_span}(\textit{ range\_sample})\)
25://Calculate(edit distancesand loss)
26:\(\mathrm{ND}^{(i)}_{\Theta}\leftarrow\text{norm\_edit\_dist}(\mathcal{Q}^{(i)}, \mathcal{Q}^{(i)}_{\Theta})\)
27:\(\mathrm{NLD}^{(i)}_{\Theta}\leftarrow\text{norm\_edit\_dist}(\mathcal{Q}^{(i)}, \mathcal{Q}^{(i)}_{\Theta})\)
28:\(\alpha^{(i)}=1-\lvert\mathrm{NLD}^{(i)}_{\Theta}-\mathrm{NLD}^{(i)}_{\Theta}\rvert\)
29: pos_dist\({}_{ij}^{(i)}=\lvert\lvert\mathbf{h}^{(i)}_{i}-\mathbf{h}^{(i)}_{i,\log}\rvert _{2}^{2}\)
30: neg_dist\({}_{ij}^{(i)}=\lVert\mathbf{h}^{(i)}_{i}-\mathbf{h}^{(i)}_{i,\log}\rVert _{2}^{2}\)
31:\(\mathcal{L}^{(i)}_{\operatorname*{unique}_{j}}=\max\{\text{pos\_dist}_{ij}^{(i)}- \text{neg\_dist}_{ij}^{(i)}+\alpha^{(i)},0\}\)
32:endfor
33:\(\mathcal{L}^{(i)}=(1-\lambda)\mathcal{L}^{(i)}+\frac{\lambda}{5}\sum_{j=1}^{5} \mathcal{L}^{(i)}_{\operatorname*{triplet}_{j}}\)
34:endfor
35:\(\mathcal{L}=\frac{1}{1}\sum_{i=1}^{I}\mathcal{L}^{(i)}\)
36:\(\textit{model\_backward}(\mathcal{L})\)
```

**Algorithm 1** Training algorithm

The main advantage of FiniQANet is that we can use the same method as in the original test case. The first step consists of two parts: 1) the input and 2) the output. In this way, we can use the same method as in the previous section. We will show how to perform FiniQANet with respect to the random variable.

To improve FiniQANet, we need to compute FiniQANet using the following methods:

[MISSING_PAGE_POST]

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline \multirow{2}{*}{**Model**} & \multicolumn{4}{c}{**Program accuracy**} \\ \cline{2-4}  & **1 step** & **2 steps** & **3+ steps** & **Overall** \\ \hline 



\begin{table}
\begin{tabular}{l|c|c|c|c} \hline
**Model** & **FinQA** & **TAT-QA** & **HiTab** & **MultiHiertt** \\ \hline CounterComp (operators) & **74.49** & **70.01** & **32.61** & **40.85** \\ \hline CounterComp (operands) & 68.98 & 70.80 & 28.88 & 37.67 \\ \hline \end{tabular}
\end{table}
Table 10: Program accuracy of CounterComp versus a sampling strategy focused on operands. FinQANet was used for all experiments.





**ACL 2023 Responsible NLP Checklist+**

Footnote †: margin: The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

A For every submission: A1. Did you describe the limitations of your work? \(8\) A2. Did you discuss any potential risks of your work? _6.3, 6.4, 8_ A3. Do the abstract and introduction summarize the paper's main claims? A4. Have you used AI writing assistants when working on this paper? _Left blank._ B Did you use or create scientific artifacts? _5.1, 5.2_ B1. Did you cite the creators of artifacts you used? _5.1, 5.2_ B2. Did you discuss the license or terms for use and / or distribution of any artifacts? _5.1, 5.2_ B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? _5.1, 5.2_ B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? _The data is based on publicly available corporate reports._ B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? _5.1, 5.2_ B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. \(1\) C Did you run computational experiments? C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used?

C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 5 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 6 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? _No packages used._
* D **Did you use human annotators (e.g., crowdworkers) or research with human participants?** _Left blank._
* D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? _No response._
* D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? _No response._
* D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? _No response._
* D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? _No response._
* D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? _No response._