EXPLAIN, EDIT, GENERATE: Rationale-Sensitive Counterfactual Data Augmentation for Multi-hop Fact Verification
Authors: Yingjie Zhu
Jiasheng Si
Yibo Zhao
Haiyang Zhu
Deyu Zhou
Yulan He
Venue: EMNLP 2023 Main (2023)
Keywords: multi-hop fact verification
counterfactual data augmentation
out-of-domain generalization
TL;DR: In this paper, we present a rationale-sensitive pipeline counterfactual data augmentation method to generate logical, diverse, and label flipping counterfactuals for multi-hop fact verification.

Abstract: Automatic multi-hop fact verification task has gained significant attention in recent years. Despite impressive results, these well-designed models perform poorly on out-of-domain data. One possible solution is to augment the training data with counterfactuals, which are generated by minimally altering the causal features of the original data. However, current counterfactual data augmentation techniques fail to handle multi-hop fact verification due to their incapability to preserve the complex logical relationships within multiple correlated texts. In this paper, we overcome this limitation by developing a rationale-sensitive method to generate linguistically diverse and label-flipping counterfactuals while preserving logical relationships. In specific, the diverse and fluent counterfactuals are generated via an Explain-Edit-Generate architecture. Moreover, the checking and filtering modules are proposed to regularize the counterfactual data with logical relations and flipped labels.  Experimental results show that the proposed approach outperforms the SOTA baselines and can generate linguistically diverse counterfactual data without disrupting their logical relationships.

EXPLAIN, EDIT, GENERATE: Rationale-Sensitiv Counterfactual Data Augmentation for Multi-hop Fact Verification
Yingjie Zhu"" Jiasheng Si Yiho Zhao', Haiyang Zhu', Deyu Zhou't, Yulan He 'School of Computer Science and Engineering, Key Laboratory of Computer Network and Information Integration, Ministry of Education, Southeast University, China
2 Department of Informatics, King's s College London, UK
3 'The Alan Turing Institute, UK
{yi_zhu, jasenchn, yibozhao, haiyangzhu, d. d.zhou]@seu, edu. cn yulan. he@kcl. ac.uk
Abstract
often perform poorly under the multitude of distri- bution shifts due to an over-reliance on spurious correlations between input text and labels (Guru- rangan et al., 2018; 8; Schuster et al., 2019; Geirhos et al., 2020). It can potentially be addressed by Counterfactual Data Augmentation (CDA), using counterfactual instances generated by perturbing causal features within the input (Khashabi et al., 2020). Several works have revealed that training with counterfactual data enhances the capability of the model to identify causal features and dimin- ish its reliance on spurious correlations between the input text and the label, thus resulting in the improvement in cut-OffDoomal (OOD) general- ization (Vig et al., 2020; Eisenstein, 2022).
Automatic multi-hop fact verification task has gained significant attention in recent years. De- spite impressive results, these well-designed models perform poorly on out-of-domain data. One possible solution is to augment the training data with counterfactuals, which are generated by minimally altering the causal features of the original data. However, current counterfac- tual data augmentation techniques fail to handle multi-hu hop fact verification due their incapa- bility to preserve the complex logical relation- ships within multiple correlated texts. this paper, overcome this limitation by devel- oping a pationale-sensitiv method to generate linguistically diverse and label-flipping counter- factuals while preserving logical relationships. specific, the diverse and fluent counterfactu- als are generated via an Explain- Cesplin-Ediit--ee Eeit-Geeereate architecture. Moreover, the checking and fil- tering modules are proposed to regularize the counterfactual data with logical relations and flipped labels. Experimental results show that the proposed approach outperforms the SOTA baselines and can generate linguistically di- verse counterfactual data without disrupting their logical relationships!.
In this paper, we seek to generate counterfactuals for multi-hop fact verification, instead of explor- ing the causal bias for specific model. However, due to the complex logical relationships within the multi-hop input texts, developing such an approach poses some significant challenges. As shown in the first row of Table 1, most CDA methods are designed for NLP tasks without requiring intricate reasoning over the input, such as the sentiment anal- ysis task (Yang et al., 2021; Howard et al., 2022). Their local modification of the causal feature in a single sentence (e.g.. "amazing" in Table + "ter- rible") is difficult to constrain the logical relation- ships between different causal features in multiple correlated texts, resulting in unverifiable counter- factuals. Furthermore, the prior attempt, CrossAug (Lee et al., 2021), is primarily designed to generate counterfactuals for single-hop fact verification via consistently editing the causal features in the claim and in the one piece of evidence (e.g., "over 30 days in the second row of Table + "less than 10 days"). Nevertheless, its claim-only based gen- eration strategy struggles preserve the complex logical relationships when faced with multiple hops of evidence, and fails to ensure label flpping and linguistic diversity in the counterfactuals, which
1 Introduction
Multi-hop fact verification task, which discerns the truth from falsehood based on multiple hops of reliable evidence, becomes crucial countering misinformation and counterfeit news spread on cur- rent social media platforms (Vosoughi et al., 2018; Botnevik et al., 2020), especially in some specific domains such politics (Alhindi et al., 2018; Os- trowski et al., 2021), science (Wadden al., 2020, 2022) and public health (Kotonya and Toni, 2020; Sarrouti al., 2021). However, many recent works
"qqual Contribution. Corresponding Author 'The code and datascts are available at pttps://giite thub. con/ Com/AAAndy-Zuin SaayyyZhhRRCCEE

---

Task
Inference
Input (label
Sentiment Analysis
This is amazing book, I'm already immersed in the storyline. (POSITIVE)
Single-hop
C: Little Miss Sunshine was filmed over 30 days. (SUPPORTS)
Fact Verification
E: Little Miss Sunshine filming began on June and took place over 30 days in Arizona
Reassning Graph
C: The Ford Fusion intraduced for model year 2006. The Rookie of The Year in the 1997
CART season drives it in the NASCAR Sprint Cup Series. (SUPPORTS)
Multi-hop
Ford Fusion manufactured and marketed by Ford. Introduced for the 2006 model year... Patrick Carpentier competed in the NASCAR Sprint Cup Series, driving
Fact Verification
Fusion. E3: The 1997 CART PPG World Series season, Rookie of the Year was Patrick Carpentier.
Table 1: Comparison between different tasks.
are crucial for CDA (Joshi and He, 2022).
motivation here is that these rationales provide the intrinsic semantic and relational information for in- ferring its label, and present the factual consistency with its claim (Raha et al., 2023).
For multi-hop fact verification, as shown in the third row of Table 1, the set of possible causal fea- tures is more complex, and exploring them may necessitate intricate reasoning about the logical relationships between multiple hops of evidence and between the claim and the evidence. For ex- ample, the "Patrick Carpentier" in E2, which is invisible to the claim, bridges the connection be- tween the causal features "Introduced for the 2006 model year" in El and "Rookie of the Year" in E3, thus leading to the alignment of the multi-hop evi- dence with the claim C' (as shown in the Reasoning Graph). Without considering such complex logi- cal relationships within the correlated input, the generated counterfactual claims potentially tend to be unreasonable or unverified. Furthermore, en- suring the label flipping and linguistic diversity of generated counterfactuals become increasingly difficult with the premise of logical relationships, which are critical factors to assure the quality of the counterfactuals.
It should be pointed out that RACE requires no external knowledge as used in Paranjape et al. (2022) besides the original training data, and is able to generate linguistically diverse and label-flipping counterfactuals while preserving logical relation- ships. Compared alternative approaches (e.g.- ChatGPT (OpenAI, 2022)) ($ 4), training on the counterfactuals generated by RACE reveals the im- provement in performance under different settings ($ 5.1), including in-domain, out-of-domain (Paran- jape et al., 2022), and challenge settings (Gardner et al., 2020). In addition, the intrinsic evaluation shows that the counterfactual claims generated by RACE are more logical and linguistically diverse than those produced by the baselines ($ 5.3, $ 5.4). Finally, we compare the results based on different generation models with baselines, illustrating that our method is generation model-agnostic. ($ 5.5).
To address these challenges, we develop a novel pipeline method, RACE (RAtionale-sensitive Counterfactual gEneration), by focusing on the causal features within the rationales extracted from the multi-hop evidence using explainability method. In specific, for each original instance, the Explainer and Editor modules are employed to produce the counterfactual evidence that logically corresponds to - but factually distinct from - the original claim. Then, according to the counterfac- tual evidence, an entity- Generator gener- ates the counterfactual claims by synthesizing the semantic information across multi-hop evidence. During the above process, the Checking and Filter- ing modules are used regularize the reasonable- ness of the output of each module from different aspects, resulting in fully labeled examples that can be used directly to augment the training data. The
Related Works
Debiasing Fact Verification A variety of ad- vanced multi-hop fact verification methods have recently emerged in various domains due the de- velopment of pre-trained models (Das et al., 2023). Nevertheless, most models exhibit poor OOD gen- eralization, primarily due to their over-reliance on spurious correlations between inputs and labels (Gururangan et al., 2018; Schuster et al., 2019; Geirhos et al., 2020). Thus, several works focus on the debiasing of fact verification models. Schuster et al. (2019) have identified strong cues for pre- dicting labels solely based on the claim. Zhu et al. (2022) proposed an entity debiasing framework that mitigates entity bias from a cause-effect perspec- tive. Lee et al. (2021) addressed the debiasing of fact verification models by augmenting the data

---

SUPPORTSS Instane
REFUTES Iestance
Coutinlacuul
Frideno
Rationales
Counerfactual Evidence
0.0
Coumberfachie chon Clain Countretccual Dual Claim]
Claim
EXPLA
PETOR
FRATOR
FIMTER
Poidenoe
REFULTS Irstincc
St SUPPORTS Instance
Clsim
Counerfatual
Rationales
0.0
Counlafaaiee actua Claing Counletor Uud Claini
Claim
GENERATOR
FINTER
Evidenoe
The overall pipeline of RACE. The SUPPORTS and REFUTES instances are processed differently, as indicated by the black and red arrows, respectively.
Figure I:
with contrastive instances. Atanasova et al. (2022) explored what information is sufficient to verify claim, and proposed a CDA schema for learning of (in)sufficient information.
can also be challenging. To this end, we propose a general pipeline, RACE, to tackle these challenges. As shown in Figure 1, our RACE consists of four stages: (I) Explainer: rationale extraction ($3.1), (II) Editor: evidence editing ($3.2), (IIII Generator: claim generation ($3.3), (IV) Filtering ($3.4). Note that our method handles SUP and REF instances differently, as the large difference in generation space between these two types of instances.
Counterfactual Data Augmentation There is growing academic interest in CDA to improve model robustmess. Initial studies focus on human- crafted counterfactuals (Kaushik al., 2020; Gard- ner al., 2020). Recently, numerous automatic CDA methods have been proposed for sentiment analysis (Wang and Culotta, 2021; Yang et al., 2021: Howard et al., 2022), question answering (Paranjape et al., 2022; Dixit et al., 2022), and natural language inference (Glockner et al., 201 8). However, these methods are primarily targeted to NLP tasks without requiring complex reasoning about the input. Thus, their direct application to the multi-hop fact verification task presents consid- erable challenges.
3.1 Explainer: Rationale Extraction
Our RACE focuses on identifying the causal fea- tures within rationales that can be perturbed. To this end, we use CURE (Si et al., 2023a), multi- granular rationale extraction method, to simultane- ously extract sentence rationales R. and token ra- tionales R, from the multi-hop evidence E for both SUP P and REF instances. In essence, the token ra- tionales R2 reflect the logical correlation within the evidence (blue words in Table and the factual relationship between the claim and the evidence (red words in Table 1). Considering the causal re- lationship of the rationales to the prediction label (Wu et al., 2022), we regard the extracted rationales as the causal features that are to be further pro- cessed. The detailed algorithm can be found in Si et al. (2023a).
3 Methodology
Given a claim with its associated evidence E (C1, 62, .2, Cx), the aim of multi-hop fact verifica- tion is to infer whether the claim is supported refuted by the evidence. We denote an in- stance in the dataset D as triplet E,y). where y E {SUP, REF) the verification label. The goal RACE is to generate counterfactual data (d,, F, v/) or (c, F', v/) that differ in some meaning- ful way from the original instance (c. E, y). where o + c' and E' denote the counterfactual claim and counterfactual evidence, respectively. This setting poses some unique challenges, such as re- quiring to identify the causal features to be edited, ensuring sound logical relations in evidence edit- ing and claim generation, and avoiding unverifiable claims. Meanwhile, ensuring the semantic diversity and the minimal perturbation of the counterfactuals
3.2 Editor: Evidence Editing
In general, entities contained within the multi-hop evidence possess rich trove of factual knowledge and crucial information (e.g., date, location, orga- nization, person, and the correlation between them), facilitating more precise multi-hop fact verification (de Jong al., 2021: Rani al., 2023). Therefore, meticulously design set of simple entity-based evidence editing rules to control the semantie per- turbation while preserving the multi-ho hopii-hop correla- tion within the evidence, and an Ad- Ad-Checking mod-

---

ule to filter out the under-edited or over-edited evi- denee. Additionally, Tan et al. (2023) highlight that controlling the generation for REF more chal- lenging due to its significantly broader generation scope compared to P. As such, we focus on editing the evidence E for instances (c,, E, SUSUU) P) rather than for instances (c, E, REF).
the original claim e as input, which is fine-tuned on HOVER (Jiang et al., 2020) dataset with instances labeled as SUP, REF, and NEI. The edited evi- dence that yields a REF prediction is retained as counterfactual evidence E' (i.e., (c. F²) -EF F). If not, we discard this case for generating counter- factuals. See Appendix B for details.
After Editing and Ad-Checking, to proceed with the claim generation for the SUP and REF instances. We retain the original REF instance as E,,,,T, REF), and have perturbed the SUP instance (c, F,, T, SU to E', E', T", Rs, REF), where T and T'' denote the set of original and edited causal entities ex- tracted from the token rationales Rs, respectively. Up to this step, generate the counterfactuals (c, E', REF) by altering the causal entities within the multi-hop evidence for (c, S)P.
ready
Editing We first utilize an off-the- shelf NER tool, Stanza (Qi et al., 2020), to identify various types of causal entity T from token rationales R2 Follow- ing Rani et al. (2023), we only retain entities with specific types, including ORG, PERSON, DATE, GPE, and NUM. Then, we automatically edit the evidence according to the following rules.
in-Dataset: Randomly replace entities of type GPE, DATE and NUM with other entities of the same type present in the entire dataset, e.g., 2006 model year + 2008 model year in Table 1. in-Inteacc: If all the token rationales in ev- idence E contain two or more PERSON/ORG enti- ties, their positions are randomly swapped between different pieces of evidence, e.g., Ford (PERSON)
3.3 Generator: Claim Generation
As Tan et al. (2023) notes, the direct generation of refuted claims is challenging and may require addi- tional ontology-like mechanisms to ensure that the generation is plausible but reversed. Thus, we opt to generate counterfactual claims c' that are sup- ported by the evidence E/E' E' from the instances. Notably, we do not intervene too much in its gener- ation process, apart from regulating the generated claim d' sensitive to the causal entities T T/T'. This allows us to ensure the linguistically diverse gener- ation while preserving the factual consistency with evidence E/E'.
Patrick Carpentier (PERSON) in Table 1.
Consistent Edit: The same entity token is processed consistently throughout all pieces of evi- dence, to preserve the multi-hop correlation within the evidence. For example, if an entity is identified in one piece of evidence, will be consistently replaced or swapped across all pieces of evidence within the instance.
We use the editing rules to produce one edited evidence for each instance based on a random seed. Notably, the PERSON and ORG entities are unique to each instance, rather than across the en- tire dataset. Thus, we prefer random in-instance swapping over in-dataset replacing to avoid intro- ducing irrelevant information from the dataset into the edited evidence. See examples in Appendix A.
Generation We first use pre-trained generation model (e.g., T5 (Raffel et al., 2020)) fine-tuned on the SUP instances in FEVEROUS dataset (Aly et al., 2021), using the concatenation of all the gold-standard evidence E as input and the corre- sponding claim as the target text (i.e., E -> c). Unlike prior work on editing the original claim c, this encourages the linguistically diverse genera- tion by synthesizing the semantic and correlation information between the multi-hop evidence.
Ad-Checking The random operation in our edit- ing rules may raise the under-editing evidence (i.e., SSU P) or the over-editing evidence (i.e, FT) for P instances, resulting in the generated claim c' based on this evidence being an incorrect seman- tic perturbation compared to its original claim c. To this end, we use existing fact verification model verify the original claim based on the edited evidence, thus ensuring that this evidence is still valid for further providing to the claim Generator. We adopt the RoBERTa (Liu et al., 2019) model, with the concatenation of the edited evidence and
Then, to ensure that the generated claim d' presents factual consistency with the evidence E E', we apply constrained beam search decoding (Anderson et al., 2017; Post and Vilar, 2018; Hu et al., 2019) with entity constraints to guide the claim generation, by taking the concatenation of all sentence rationales Rs in E/ E' as input. Specifically, regarding the list of causal entity tokens de, = [ts,1, t..,..., tx,s within each piece

---

of evidence as disjunctive constraints, where 4,, e T/T" denotes the causal entity in the i-th evidence, and j is the number of the entities, we acquire the conjunction constraint of the beam search by combining of all disjunctive constraints,
et al., 2000), measures the semantic similarity be- tween two sentences. We thus use this metric calculate semantic remanticudelitty score between each coun- terfactual claim in C" and its corresponding original claim e, evaluating the semantic change between these two claims.
CONS = de1 A doz A A don,
(1)
Entity Filtering We introduce the entity fidelity score by calculating the overlap rate of entities be- tween strings of claim (c', c) pair. This allows us to ensure topic consistency between e' and filtering out the irrelevant claims from topic perspective (Si al., 2021).
where n is the number evidence. The conjunctive constraint during decoding encourages the gener- ated claim e to contain at least one causal entity from each piece of evidence, thus ensuring factual consistency with the multi-hop evidence. After repeated generation, we generate k (k 10 in our experiments) candidate counterfactual claims C² C²= (c ...,,,c.c d for each instance.
One generated claim d' E C² with the highest sum score over semantic fidelity score and entity fidelity score is retained for each instance. Finally, our RACE produces the counterfactual data for each instance (c, E, y) in the dataset, including (d', F., y') and (c, F',g)).
Tost-Checking The claim generation model can be noisy, potentially leading to the non-reversed predictions of a claim ć' given E. To ascertain the label flipping between claim e' and c, i.e, E) y' # y: by taking the concatena- tion of each candidate counterfactual claim & with its corresponding original evidence as input, we use the same three-way fact verification model as in Ad-Checking to filter the candidate counterfactual claims. We retain those candidate claims in C' that yield predicted label y' #
4 Experiments
We generate counterfactual data for HOVER? training set (Jiang et al., 2020), a multi- hop dataset with facts sourced from Wikipedia. We evaluate the model generalization on three types of development sets, (I) In-domain setting (sourced from Wikipedia), including FEVER (Thorn et al., 2018) and FEVEROUS (Aly al., 2021). (II) Out-of-domain setting (sourced from specific do- mains), including PolitiHop (political news) (Os- trowski et al., 2021), SCIFACT (scientific articles) (Wadden et al., 2020), Health (Sarrouti et al., 2021) and PubHealth (public health) (Kotonya and Toni, 2020). (III) Challenge setting (contrastive data), including FM2 (Eisenschlos et al., 2021) and VITAMINC (Schuster et al., 2021). Details and statistics of datasets are presented in Appendix C.
Datasets
Discussion Claim generation also be done by very large language models (LLMs) (e.g., ChatGPT (OpenAI, 2022)) with in-context learning (Brown et al., 2020; Wei et al., 2022). However, since our editing may introduce inconsistencies with com- mon sense, we empirically find that the edited evi- dence E" is more likely to conflict with the internal knowledge of LLMs, thus leading to the irrelevant content even failure in generating the claim c'. Thus, we choose the fine-tuned generation models.
3.4 Filtering
Baselines We use three types of baselines aug- ment the HOVER training set, (I) Data augmenta- tion method: EDA (Wei and Zou, 2019). (II Coun- terfactual data augmentation methods: CrossAug (Lee et al., 2021) and POLYJUICE (Wu et al., 2021). (III) LLMs: GPT-3 (text-davinci-003) (Brown et al., 2020) and ChatGPT (gpt-3. 1.ptt-.55-00 3.5-3.tubbr. 5-turbo-0301) (OpenAI, 2022). More details are presented in Ap- pendix D.
Unlike prior work that relies on a curated set of minimal edits (e.g., Yang al. (2021)), the strategy in our Generator maybe over-generate claim c' with over diverse semantic shift compared to c. Thus, following Paranjape et al. (2022), we use post-hoc filtering with two modules on generated claims C" to ensure the minimal semantic (Keane and Smyth, 2020) and topic perturbation compared the original claim
Semantic Filtering The MoverScore (Zhao et al., 2019), which combines the contextualized repre- sentations with the Earth Mover distance (Rubner
"Since the VER dataset contains explicit multi-hop cor- relation among evidence based on different reasoning type, we choose to generate counterfactuals and report results in this paper.

---

In-domain
Out-of-domain
Challenge
Source of data
[Drom
HOVER
FEVER FEVEROUS
PolitiHpp
SCIFACT
HealthVVrr
PubHealth
FM2
VItaMINC
None
18,171
82.55
76.70
69.43
48.74
62.77
54.98
53.01
61.51
67.05
EDA
36.342
82.35
73.60
68.22
54.62
62.77
53.68
45.99
60.56
59..6
CrossAug
29.174
82.28
65.92
70.05
54.62
57.98
49.24
39.15
56.12
61.66
POLYJUICE
25.190
81.10
67.94
57.98
54.65
46.28
57.14
62.29
GPT-3
24,171
80.75
72.30
67.56
51.26
64.36
49.46
42.91
61.25
62.21
ChatGPT
24,171
80.13
77.77
69.04
44.54
60.64
51.84
45.00
50.74
68.14
OWL RACE (BART)
24,398
82,78
76.07
70.63
47.06
61.17
46.97
42.81
59.17
$9.54
Our RACE (GPT-2)
23.645
82.53
77.15
66.07
45.38
65.43
54.87
53.52
62.36
67.88
Our RACE (15-large)
26,638
83.18
78.11
71.55
47.06
62.77
55.84
56.59
61.16
67.71
our RACE (TS-buse)
26,917
83.15
75.05
70.50
52.94
68.43
55.41
53.52
62.19
66.50
-CONS
28.468
"82.33
73:93
70.09
48.74
59.04
52.7
49.06
62.28
67.31
-EDIT
28.359
80.75
71.50
68,13
54.62
60.64
52.92
47.47
61.33
62.72
-EDIT&CONS
27,682
83.00
76.84
70.69
43.70
60.11
52.60
53,42
60.05
64.74
[c. (REF)
RACE (BART)
27.909
83.53
76.65
69.16
41.18
59.57
54,00
44,20
61.42
66.16
RACE (GPT-2)
27,156
82.78
75.31
70.52
51.26
62,77
51.62
51.83
59.97
62.18
RACE (TS-large)
30,149
82,90
78 69
69.29
47,90
64,89
55.41
52.03
61.08
66.31
our RACE (I5-base)
30,428
76.73
70.90
57.14
60.11
55.63
61.33
67.66
Table Fact verification accuracy of various data augmentation methods on different development sets three settings. DDraio shows the total number of training instances, including 18,171 171 original HOVER training instances. w (c, E", REF) denotes the incorporation of counterfactual instances (c, REF) into the training set. -CONS denotes the use of beam search instead of constrained beam search in claim generation. -EDIT denotes that the evidence editing stage is skipped, and counterfactual claims are generated directly from the original evidence for each original instance. The best of the main results are marked bold. The results with further improvement in model performance after the incorporation of (c, E", REFF are boxed.
Implementation Details In the experiments, we fine-tune basic multi-hop fact verification model, an additional RoBERTa- base (Liu et al., 2019), on the original training data (c, E, y) and the counter- factual data generated by each method. The model is evaluated the development set of different datasets.
data (c', E,y') generated by the different methods. Meanwhile, we also report the results after incorpo- rating (c, E", -REF) into the training set (bottom of Table 2).
Out-of-domaii Setting Table shows the effects of the data generated by RACE and baselines on the OOD generalization. We can observe that, (I) RACE significantly improves model performance on PolitiHop, SCIFACT and PubHealth compared to the results without data augmentation, and out- performs baselines on almost all OOD datasets, demonstrating the effectiveness of our augmenta- tion strategy for multi-hop fact verification task. (II) RACE significantly outperforms POLYJUICE, showing that the general- CDA method, designed for tasks without requiring complex rea- soning on the input, fails to achieve acceptable results on multi-hop fact verification task, and even impairs the OOD generalization. (III) The coun- terfactual data generated by LLMs provides little improvement in OOD generalization, demonstrat- ing that CDA for multi-hop fact verification task remains challenging for LLMs by using the in- context learning alone. (IV) The incorporation of (c, E', REF) further improves the model general- ization to a certain extent on PolitiHop, indicating
For the basic multi-hop fact verification model, we concatenate the claim and all evidence as input sequence, and limit its maximum length to 130. We set the batch size to 4 and optimize the model through cross entropy loss using the AdamW optimizer (Loshchilov and Hutter, 2019) with the learning rate of le-5. For claim generation, conduct experiments with four generation models: BART-base (Lewis et al., 2020), Tasbase, TS-large (Raffel et al., 2020) and GPT-2 (Radford et al., 2019). The beam size is 30 and the max length of generated text is 96.
5 Results and Discussion
5.1 Main Results
Neglecting the logical relationships within the cor- related input results in a failure to generate coun- terfactual evidence E' for baselines. Thus, we mainly compare the effects of the counterfactual

---

Original Instance
Claim
The 1994 British iomattic comedy that Charlome Nincn Coleman played Searlett in festured the song "Love"
[Reg Presley] Ile wrote the song "Love is Around". which lestured in the films Touur Weddings and Funeral" and "Love" Acmallyy..
Evidence
[Charlons Coleman] Charlott Nincn Coleman (3 April 1968 November 2001) was an Englissh actress best known for playing Scarleti the film TTour Weddingss and Tuneraa. Jess in the television drama "Oranges Are Not the Only Fruil", and her childhood roles of Soe in "Worzel Gummidge" and the character Marmalade Atkins.
Weddings a Funeral] Weddings and a Tumeral
British romanii comedy film directed by Mike Newell.
Label
SU PPORTS
Counterfactual Claims
CrossAng
1994 Brirish romantic comedy that Charlotte Ninon Coleman played Scarlett featured the song SLove The 1994 British romamiic comedy that did not win Charloti Ninon Coleman played Scurtet, leatured the song "Love".
POLYJUICE
ChalGPT
The 1994 American romuntic comeddy that Charlotie Ninon Coleman played Scazieti in featured the song "Love". Marmalade Arkins directed the 1948 British romantic comedy thar Reg Presley played Charlate Caleman in. featured the song "Love".
RACE (TS-base)
Table 3: Examples of counterfactual claims HOVER training derived by different methods. The difference
between the counterfactual claim and the original claim is highlighted in blue. See Table in Appendix A for the corresponding edited evidence and more examples.
Method
Flip. 1
Flu. J
Sim.
Div.
M.b.
the elimination of spurious correlations between features in claim and labels.
CrossAng
0,3138
209.34
0.6100
2.24
0.6090
POLYJUICE
0.6066
195.18
0.5969
1,20
0.5960
GPT-3
0,3970
96.84
0,5873
1.47
0.5865
In-domain Setting As shown in Table 2, RACE
ChatGPT
0,4160
107.94
0.5906
1.73
0.5898
improves the model performance on in-domain data, while most baselines tend to degrade it. No- tably, our method has the most significant improve- ment on the FEVEROUS development set, which requires four pieces of true evidence to verify each claim on average. This further demonstrates the effectiveness of our method for multi-hop fact veri- fication task.
RACE (TS-large)
0,9402
$5.03
0,5770
11.22
0.5763
RACE (I5-base)
0.9457
55.81
0.5770
11.19
0.5763
-FITEE
0.8388
58.83
0.5773
0.5766
Table 4: Automatic intrinsic evaluation results. For Flip rate (Flip.). we use RoBERTa-based classifier fine-tuned on the HOVER training set to calculate the verification accuracy of the instance (d', E,y.). For Flueney (Flu.), following previous work (Atanasova et al., 2020; He et al., 2023), we use the perplexity scores calculated by GPT-2 to evaluate the fluency of d. For Similarity (Sim.), we calculate the MoverScore between c' and c. For Diversity (Div.), following Rani the inverse of the BLEU score
5.2 Ablation Study
We conduct ablation studies on evidence editing and claim generation stage to verify the reasonable- ness of causal entities in token rationales. All the experiments are conducted based on RACE (T5- base).
et al. (2023), (Papineni et al., 2002) to measure dissimilarity between d' and c. For Multi hop (M.h.). we employ MoverScore to calculate the average semantic similarity between e' and e, where e; E E, to evaluate the coherence like He al. (2023). -FILTER denotes the evaluation all the generated claims before post-checking and filtering stage. The best results are marked in bold.
Firstly, we use ordinary beam search instead of constrained beam search during the claim gener- ation stage (i.e., -CONS in Table 2). The results in Table 2 reveal that significant performance de- crease occurs on both in-domain and OOD data. It might be explained by constraints based enti- ties in token rationales, which allow the generated claim to be multi-hop and topic consistent with the original claim, resulting in more efficient counter- factual. In contrast, we note a slight improvement on the challenge datasets, which might be attributed to the shorter length of claims in both datasets (cach claim contains about 13 words on average).
that the edited evidence still remains multi-hop cor- related and reasonable.
Challenge Setting Comparing the results on challenging contrastive datasets, as Table 2 shows, training with RACE data improves the fact veri- fication accuracy, while almost all the baselines degrade the performance of the model. This phc- nomenon confirms that our method improves model robustness spurious correlations. Additionally, the incorporation of the (c, F', REF) yields no improvement in verification aecuracy, probably be- cause these datasets are constructed in response to
Then, we skip the evidence editing stage and di- rectly generate the counterfactual claims for all the instances (i.e, -EDIT in Table 2) by a T5-base lan- guage model. The model is fine-tuned on FEVER- OUS to generate claims that are supported or re-

---

futed by the input evidence via setting the prefix. As shown in Table 2, the accuracy decreases sub- stantially on almost all datasets, except for Poli- tiHop. can be explained by the fact that polit- ical news typically focuses on event information rather than entity information, hence entity-based evidence editing fails to improve model generaliza- tion on PolitiHop.
Mullia
Diancty
Malloy
scoeddeg
a aaalybl
a D.ussi
our ANCE (1--ane
6d The resallss of SUP semples
(b) The results of REF sumplles
Figure The results of human evaluation, where in- dicates complete breach of the criteria and 5 indicates full compliance. The inter-rate agreement measured by Krippendorf''s s a (Krippendorff, 2011) 0.54.
Finally, we further remove both the constrained beam search and evidence editing stage (i.e., CONS&EDIT in Table 2). A significant decrease in accuracy is observed on both OOD and chal- lenge data, which demonstrates that the proposed evidence editing based on rationales and claim gen- eration based on entities are crucial for improving the generalization and robustness of the multi-hop fact verification models.
pared to the one before filtering, which illustrates the necessity of the filtering stage for generating high-quality counterfactual data. For automatic evaluation of Multi hop, we follow He al. (2023) to use MoverScore to evaluate the multi hop of counterfactuals. And all methods achieve compara- ble results. However, we argue that this is compro- mised since its solely semantic comparison cannot reflect whether all the evidence can be aggregated as whole to verify the counterfactual claim.
5.3 Intrinsic Evaluation
For further analysis of the quality of the gener- ated counterfactual claims, following Chemmen- gath et al. (2022) and Dixit et al. (2022), we au- tomatically and manually evaluate the generated counterfactual claims according to the following five criteria: (I) Flip rate (Flip.), measuring if the label of the generated claim is flipped based on the original evidence; (II) Fluency (Flu.), measuring whether the generated claim is grammatically cor- rect and semantically meaningful; (III) Diversity (Div.), reflecting the linguistic diversity of the gen- erated claim compared to the original claim; (IV) Similarity (Sim.), measuring the degree of semantic similarity between the generated claim and the orig- inal claim, where we use MoverScore (Zhao et al., 2019) instead of Levenshtein edit distance (Leven- shtein et al., 1966) in the automatic evaluation to balance with diversity; (V) Multi hop (M.h.), mea- suring whether the generated claim is multi-hop and relevant to the evidence.
Manual Evaluation To address the limitations of the automatic evaluation, we adopt the human evaluation to qualify the counterfactuals from dif- ferent aspects. Specifically, we randomly select 30 SU P instances and 30 REF instances and ask three postgraduate students with an NLP back- ground score counterfactual claims in likert scale of I to according to the above criteria. Since CrossAug can only generate counterfactuals for SUP instances, we compare the results on SUP and REF instances separately. The evaluation re- sults are shown in Figure 2. It can be observed that RACE well outperforms baselines, particularly in terms of diversity, which illustrates the ability of RACE to generate human-readable, diverse, and label-flipping counterfactual claims. Meanwhile, entity constraint-based generation enables RACE to generate multi-hop claims.
Automatic Evaluation For a fair comparison, the claims generated before and after the post- checking and filtering are compared with the base- lines separately. As shown in Table 4, RACE out- performs baselines significantly in terms of flip rate, diversity, and fluency. It demonstrates the abil- ity of RACE to generate fluent and linguistically diverse counterfactual claims based on the edited evidence, while keeping label flipping and logical relationships with the original evidence. Moreover, the counterfactual claim after the filtering stage achieves a higher flip rate and fluency score com-
Overall, both the automatic and manual eval- uation results show the effectiveness of RACE from different aspects for multi-hop fact verifica- tion task.
5,4 Qualitative Evaluation
Table 3 presents example of the original instance and the counterfactual claims generated by differ- ent methods. The words that differ from the origi- nal claim are highlighted. It can be observed that RACE generates a linguistically diverse and flu-

---

ent counterfactual claim, and the original label is successfully flipped. Obviously, the counterfactual claim generated by RACE can be combined with the original evidence to form a valid multi-hop fact verification instance, which is logical and can be verified according to the given evidence. Moreover, the claim generated by RACE is semantically and lexically similar to the original claim, benefiting ca- sual entities in multi-hop rationales. Nevertheless, the baselines tend to simply modify the original claim, despite the use of language models. As shown in Table 3, most of the baselines (including LLMs), prefer to add "not" to the original claim or make antonym substitutions. Such modifications make the counterfactual claims lexically similar to the original claim, but are not valid for multi-hop fact verification and cannot generate diverse and logical counterfactual claim (as evidenced by lower flip rate and diversity in Table 4 and Figure 2).
counterfactuals for multi-hop fact verification task. An Explain-Edit- Geet-ee Generate architecture is con- structed to generate diverse and logical counterfac- tual claims based on the rationales. Then, a filter process with two modules is employed to further regularize semantic and topic consistency. Experi- mental results reveal the improvement in OOD gen- eralization and robustness of the proposed method. Intrinsic evaluation and qualitative evaluation of counterfactual claims show that RACE can gener- ate linguistically diverse and label-flipping counter- factual data while preserving logical relationships.
Limitations
As multi-hop fact verification is relatively com- plex reasoning task, designing an effective method to generate counterfactuals for this task requires consideration of the logical relationships between the claim and the evidence and between multiple pieces of evidence, making our proposed method more complex and cumbersome. Meanwhile, the use of heuristic rules in the editing process results in the inability to generalize to other tasks and the need to recreate the rules. In addition, the prompts given to LLMs for generating counter- factual claims can be further elaborated, e.g., us- ing chain-of-thouuht,, exploit more potential of LLMs on CDA for multi-hop fact verification task. In the future, due to the flexible generation of LLMs, we will explore the construction of effective prompts to generate counterfactuals for multi-hop fact verification using the Chain-of-Thoughh
5.5 Effect of Generation Models
We adopt different generation models to test the ef- fect of the generation ability on our method, which aims to illustrate the independence of our proposed method from a particular generation model (i.e., Generation Model-Agnostic)) As shown in Table 2. compared to the baselines, our RACE yields a comparable or improved performance based on different generation models, especially the results based on T5-base and T5-large. Besides, We em- pirically find that different generation models have more prominent performance on specific datasets, e.g., GPT-2 on SCIFACT and FM2 datasets, and T5 on 6 datasets.
A Acknowledgement
To explore the effect of the number of parame- ters, we further compare the results based on T5- base and T5-large. As Table 4 and 2 shows, com- pared to 75-base, counterfactuals generated by fine- tuned Ta-lerge are more fluent and linguistically di- verse, and further improve the model performance on most datasets. This illustrates that it is possible to further improve the effectiveness of our method by using a more powerful generation model. Thus, for the choice of the generation model, we recom- mend choosing the powerful possible generation model in the absence of the priors to the data.
The authors would like to thank the anonymous reviewers for their insightful comments. This work is funded by the National Natural Science Foun- dation of China (62176053) and supported by the Big Data Computing Center of Southeast Univer- sity. YH is supported by a Turing Al Fellowship funded by the UK Research and Innovation (grant no. EP/V020559/1, EP/V020579/2).
References
Tariq Alhindi, Savvas Petridis, and Smaranda Mure- san. 2018. Where is your evidence: Improving fact- checking by justification modeling. In Proceedings of the First Workshop on Fact Extraction and VE cation (FEVER), pages 85-90.
6 Conclusion
We present a novel rationale-sensitive pipeline counterfactual data augmentation method (RACE) to generate logical, diverse, and label-ftipping
Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thoree, Andreas Vlachos, Christos Christodoulopou-

---

los, Oana Cocarascu, and Arpit Mittal. 2021. Fever- ous: Fact extraction and verification over unstruc- tured and structured information. In Proceedings of the NeurIPS Track on Datasets and Benchmarks.
Julian Eisenschlos, Bhuwan Dhingra, Jannis Bulian, Benjamin Börschinger, and Jordan Boyd-Graben 2021. Fool me twice: Entailment from Wikipedia gamification. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistiex: Human Lunguage Tech- nologies, pages 352-365.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided open vocabulary im-
age captioning with constrained beam search. In Pro-
ceedings of the Conference on Empirical Methods in
Jacob Eisenstein. 2022. Informativeness and invariance: Two perspectives on spurious correlations in natu- ral language. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 43311
Natural Language Processing, pages 936-949 945.
Pepa Atanasova, Jakob Grue Simonsen, Christina Li- oma, and Isabelle Augenstein. 2022. Fact checking with insufficient evidence. Transactions of the Asso- ciation for Computational Linguistics, 10:746-763.
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hamnanch Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nel- son F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models' local decision boundaries via contrast sets, In Proceedings of Findings of the Association for Computational Linguistics: EMNLP. pages 1307-1323.
Pepa Atanasova, Dustin Wright, and Isabelle Augen- stein. 2020. Generating label cohesive and well- formed adversarial claims. In Proceedings of the Conference on Empirical Methods Natural Lan- guage Processing, pages 3168-3177.
Bjarte Botnevik. Eirik Sakariassen, and Vinay Setty. 2020. Brenda: Browser extension for fake news detection. In Proceedings of ACM SIGIR Confer- ence Research and Development in Information Retrieval, page 2117-2120.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Robert Geirhos,
Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shorteut learning in deep neural networks. Nature Machine Intelligence, 2(1 1766666673
Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the Anmual Meeting of the Association for Computa- tional Linguistics, pages 650-655.
Saneem Chemmengath, Amar Prakash Azad, Ronny Luss, and Amit Dhurandhar. 2022. Let the CAT out of the bag: Contrastive attributed explanations for text. In Proceedings of the Conference on Empiri- cal Methods in Natural Language Processing, pages 7 7190-7206.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language in- ference data. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 107-112.
Wenhu Chen, Hongmin Wang. Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. Tabfact: A large-scale dataset for based fact verification. In Interna- tional Conference on Learning Representations.
Anubrata Das, Houjiang Liu, Venelin Kovatchev, and Matthew Lease. 2023. The state of human-centeree nlp technology for fact-checking, Information Pro- cessing & Management, 6032):103219
Bing He, Mustaque Ahamad, and Srijan Kumar. 2023. Reinforcement learning-based counter- study
misinformation response generation: of covid-19 vaccine misinformation. In Proceedings of the ACM Web Conference, pages 2698-2709.
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGer- ald, Fei Sha, and William W. Cohen. 2021. Mention memory: incorporating textual knowledge into trans- formers through entity mention attention. CoRR, abs/2110.06176.
Phillip Howard, Gadi Singer, Vasudev Lal, Yejin Choi, and Swabha Swayamdipta. 2022. NeuroCounterfac- tuals: Beyond minimal-edin counterfactuals for richer data augmentation. In Proceedings of Findings of the Association for Computational Linguistics: FMNLP, pages 5056-5072.
Tanay Dixit, Bhargavi Paranjape. Hannanch Hajishirzi, and Luke Zettlemoyer. 2022. CORE: A retrieve- then- edit framework for counterfactual data generation. In Proceedings of Findings of the Association for Computational Linguistics: EMNLP, pages 2964-
J. Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. 2019. Improved lexically constrained decoding for translation and monolingual rewriting.
2984.

---

In Proceedings of the Conference of the North Amer- iean Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-850.
the Association for Computational Linguistics, pages 7871-7880.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized BERT pretraining approach. CoRR, tttp/1907.1166
Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. HoVer: A dataset for many-hop fact extraction and claim verification. In Proceedings of Findings of the Association for Computational Linguistics: EMNLP, pages 3441-344600
Nyy Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Nitish Joshi and He He. 2022. An investigation of the (in fineeffcctvessss of counterfactually augmented data, In Proceedings of the Annual Meeting of the Asso- ciation for Computational Linguistics, pages 3668- 3681.
OpenAL. 2022. Introducing chatgpt.
Wojeiech Ostrowski. Arnav Arora, Pepa Atanasova, and Isabelle Augenstein. 2021. Multi-hop fact check- ing of political claims. In Proceedings of the Inter- national Joint Conference on Artificial Intelligence, pages 3892-3898.
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning the difference that makes a differ- ence with wtwperrffactallly-aggeeeeee data. In Inter- national Conference on Learning Representations.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 311-318.
Mark Keane and Barry Smyth. 2020. Good counter- factuals and where to find them: A case-based tech- nique for generating counterfactuals for explainable (xai). In Proceedings of International Conference on Case-Based Reasoning Research and Develop- mens, pages 163-178.
Bhargavi Paranjape, Matthew Lamm, and Ian Tenney. 2022. Retrieval-guided counterfactual generation for QA. In Proceedings of the Annual Meeting of the A Association for Computational Linguistics, pages 1670-1686
Daniel Khashabi, Tushar Khot, and Ashish Sabharwal. 2020. More bang for your buck: Natural perturbation for tobust question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 170.
Matt Post and David Vilar. 2018. Fast lexically con- strained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1314-1324.
Omar Khattab, Christopher Potts, and Matei Zaharia. 2021. Baleen: Robust multi-hop reasoning at scale via condensed retrieval. In Advances in Neural Infor- mation Processing Systems, pages 27670 27682.
Neema Kotonya and Francesca Toni. 2020. Explainable automated fact-checking for public health claims. In Proceedings of the Conference Empirical Methods Natural Language Processing, pages 7740-7754.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the Annual Meeting of the Association for Computational Linguistics: Sys- tem Demonstrations, pages 101-108.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
Minwoo Lee, Seungpil Won, Juae Kim, Hwanhee Lee, Cheoneum Park, and Kyomin Jung. 2021. Crossaug: contrastive data augmentation method for debi- asing fact verification models. In Proceedings of the ACM International Conference on Information & Knowledge Management, pages 3181-3185.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1:9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research. 21:5485-5551.
Vladimir Levenshtein et 1966. Binary codes capa- ble of correcting deletions, insertions, and reversals. In Sovier physics doklady, pages 707-710.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyamov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequeete pre-training for natural language generation, translation, and com- prehension. In Proceedings of the Annual Meeting of
Tathagata Raha, Mukund Choudhary, Abhinav Menon, Harshit Gupta, KV Srivatsa, Manish Gupta, and Va- sudeva Varma. 2023. Neural models for factual in- consistency classification with explanations. arXiv preprint arXiv:2306.08872

---

Anku Rani, SM Tonmoy, Dwip Dalal, Shreya Gautam, Megha Chakraborty, Aman Chadha, Amit Sheth, and Amitava Das. 2023. Factify-5wqa: 5w aspect-based fact verification through question answering. arXiv preprint arXiv:2305.0433 04339..
Jesse Vig, Sebastian Gehrmann. Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. 2020. Causal mediation analysis for interpreting neural nip: The case of gender bias. arXiv preprint UrXiv:2004.12255.
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. 2000. The earth mover's s distance as a metric for image retrieval. International journal of computer vision, 40(2)999.
Soroush Vosoughi, Deb Roy. and Sinan Aral. 2018. The spread of true and false news online. Science, 359(6380): 146-1151.
Mourad Sarrouti, Asma Ben Abacha, Yassine Mrabet, 2021. Eyidence-basse fact-checking of health-related claims. In Proceed- ings of Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3499-3512.
David Wadden, Shanchuan Lin, Kyle Lo. Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannanch Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the Conference on Empirical Methods in Natural Language Process- ing, pages 7534-7550.
and Dina Demner-Fushine
Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with con- trastive evidence. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Techmologies, pages 224-643.
David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, and Hannaneh Hajishirzi. 2022. Mul- tiVerS: Improving scientific claim verification with weak supervision and full-document context. In Pro- ceedings of Findings of the Association for Computa- rional Linguistics: NAACL, pages 61-76.
Tal Schuster. Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the Conference on Empir- ical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing, pages 3419-3425.
Zhao Wang and Aron Culotta. 2021. Robustness to spurious correlations in text classification via auto- matically generated counterfactuals. In Proceedings of the AAAI Conference on Artificial Intelligence. pages 14024-14031.
Jiasheng Si, Deyu Zhou, Tongzhe Li, Xingyu Shi, and Yulan He. 2021. Topic-aware evidence reasoning and stance-awar aggregation for fact verification. In Proceedings of the Annual Meeting of the Associa- tion for Computational Linguistics and International Joint Conference on Natural Language Processing, pages 1612-1622.
Jason Wei, Xuezhi Wang. Dale Schuurmans. Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Jason Wei and Kai Zou. 2019. EDA: Easy data aug- mentation techniques for boosting performance on text classification tasks. In Proceedings of the Con- ference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing, pages 6382-66 -6388.
Jiasheng Si. Yingjie Zhu, and Deyu Zhou. 2023a. Con- sistent
rationale extraction for ex- plainable multi-hop fact verification. arXiv preprint arXiv:2305.09-00
Jiasheng Si, Yingjie Zhu, and Deyu Zhou. 2023b. Ex- ploring faithful rationale for multi-hop fact verifi- cation via salience-aware graph learning. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence.
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2021. Polyjuice: Generating counter- factuals for explaining, evaluating, and improving models. In Proceedings of the Anmual Meeting of the Association for Computational Linguistics and International Joint Conference on Natural Language Processing, pages 6707-6723
Neset Tan, Trung Nguyen. Josh Bensemann, Alex Peng, Qiming Bao, Yang Chen, Mark Gahegan, and Michael Witbrock. 2023. Multi2Claim: Generat- ing scientific claims from multi-choice questions for scientific fact-checking. In Proceedings of the Con- ference of the European Chapter of the Association for Computational Linguistics, pages
Yingxin Wu, Xiang Wang. An Zhang, Xiangnan He, and Tat-Seng Chua. 2022. Discovering invariant ra- tionales for graph neural networks. In International Conference on Learning Representations.
James Thorne, Andreas Vlachos. Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 809-819.
Linyi Yang, Jiazheng Li, Padraig Cunningham, Yue Zhang, Barry Smyth, and Ruihai Dong, 2021. Ex- ploring the efficacy of automatically generated coun- terfactuals for sentiment analysis. In Proceedings of the Annual Meeting of the Association for Computa- tional Linguistics and International Joint Conference on Natural Language Processing, pages 306-316.

---

Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized em- beddings and earth mover distance. In Proceedings of the Conference on Empirical Methods in Natural Lan- guage P Processing and International Join Conference on Natural Language Processing, pages 563-578.
C
Datasets
HOVER (Jiang et al., 2020), a dataset for multi-hop fact verification, which challenges models to extract relevant evidence from sev- eral Wikipedia articles and verify whether the claim is SUPPORTED or REFUTED by the evidence. We construct the dataset follow- ing Khattab al. (2021), where each claim is associated with five pieces of evidence.
Yongchun Zhu, Qiang Sheng, Juan Cao, Shuokai Li, Danding Wang. and Fuzhen Zhuang. 2022. General- izing to the future: Mitigating entity bias in fake news detection. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2120-2125.
FEVER (Thorne et al., 2018), a large-scale fact verification dataset with the claims gen- erated by altering sentences extracted from Wikipedia. The claims in FEVER are clas- sified SUPPORTS, REFUTES NOT ENOUGH INFO (NEI) by annotators and more than 87% of them only require infor- mation from a single Wikipedia article. We remove the instances with NEI label and only retain the other two classes of instances in our experiments.
A Evidence Editing
Table 5 shows examples of the evidence edited by RACE. We can observe that rationale- and entity- based editing enables the edited evidence to still retain multi-hop correlation with each other and present a completely different fact from the original evidence. Hence, the claim generator can generate logical, fluent, and linguistically diverse counter- factual claims based on the edited evidence.
FEVEROUS (Aly et al.. 2021), a large-scale multi-hop fact verification dataset consisting of claims verified against Wikipedia pages and labeled as SUPPORTS, REFUTES or NOT ENOUGH INFO (NEI). Each claim has ev- idence in the form of sentences and/or cells from tables on Wikipedia. Following Chen et al. (2020) and Si et al. (2023b), we employ the simple table linearization template gen- erate contextualized sequence representations for table evidence. We remove the instances with NEI label and only retain the other two classes of instances in our experiments.
B Checking Module
For the ad- and post-checking module, we fine-tune a RoBERTa-base classifier to filter invalid edited evidence and counterfactual claims, respectively. To improve the quality the retained data, we fine- tune it on the SUP P, REF, and EI instances rather than just the SUP and REF instances.
Considering that we perform CDA on HOVER training set during the experiment while no N NEI instances are available in HOVER, we first con- duct data augmentation on HOVER dataset to in- corporate NEI instances by perturbing existing instances. Specifically, for random instance in HOVER, we randomly remove one piece of true evi- dence randomly pair the claim with the evidence of another instance. To avoid imbalance classes, randomly select half of the SU instances and half of the REF instances for perturbation and each per- turbation strategy is employed with equal probabil- ity. Finally, the fine-tuned RoBERTa- base classifier has 81. 31.23% on label accuracy of claim verification on NEI augmented HOVER development set. The statistics of NEI augmented HOVER are shown in Table 6.
PolitiHop (Ostrowski et al., 2021), multi- hop fact verification dataset of real-world claims with manual annotations of evidence from PolitiFact articles. The labels include FALSE, HALF-TRUE and TRUE. In our experiments, we remove the instances with HALF-TRUE label and only retain the other two classes of instances.
SCIFACT (Wadden et al., 2020), scientific fact verification dataset of 1.4K expert-written scientific claims paired with evidence. As with the above dataset, we only retain the in- stances with SUPPORTS and REFUTES bels to evaluate the model.
Other implementation details are the same as the fact verification model in the OOD generalization experiment described in Section 4.

---

Original Instance
Claim
The 1994 British romantic comedy that Clirroote Ninon Coleman played Scarlett in featured the song "Love".
1. [Reg Presley] He wrote the song "Love Is All Around", which was featured in the films "Four Weddings and a Funeral" and "Love Actually".
Evidence
[Charlotie Coleman] Charlotte Ninon Coleman (3 April 1968 November 2001) was an English actress best known for playing Scarlett in the film "Four Weddings and Funcra",
television drama "Oranges Are Not the Only Fruit", and her childhood roles of Sue "Worzel Gummidge" and the character Marmalade Alkins.
FFour Weekdings and Funeral] Four Woddings and a Funeral is 1994 Brilish romantic comedy film directed Mike Newell.
Label
SUPPORTS
Edited Evidence
1. [Mike Newell] He wrote the song "Love Is All Around", which was featured in the films "Four Weddings and a Funeral" and "Love Actually".
IReg Prestey] Reg Presley (3 August 1987 - 26 June 2000) was an Eaglish actress best known for playing Charlotte Coleman in the film "Four Waddingg and Funena", Scarletr television drama "Oringes Are Not Only Fruit", her childhood of Sue in "Worzel Gummidge" and the character Jess. [Four Weddings and Funeral] Four Weddings and Funeral 1948 British romantic comedy film directed Marmalade Arkins.
Edired Evidence
Counterfictual Claims
CrossAug POLYJUICE ChatGPT
The 1994 British romantic comedy that Charlotte Ninon Coleman played Scarlett in featured the song "Love" The 1994 British romantic comedy that did not win Charlotte Ninon Coleman played Scarlett featured the song "Love".
The 1994 American romantic comedy that Charlotte Ninon Coleman played Scarlett featured the song "Love". Marmalade Atkins directed the 1948 British romantic comedy that Reg Presley played Charlotte Coleman in. It featured the song "Love".
RACE
Original Instance
Cluim
Bruce Geller who died in 1978 developed American television detecrive show Mannix. [Mannix] Created Richard Levinson and William Link and developed executive producer Bruce Geller, the litle character. Joe Mannix, is a private investigator.
Evidence
2. [Brace Geller] Bruce Bemard Geller (Octoher 13, 1930 May 21, 1978) was an American lyricist, screenwriter, director, and lelevision producer.
Label
SUPPORTS
Edited Evidence
[Mannix] Created by Joe and Richard Levinson developed executive producer William Link, title Bruce Geller, is private investigator.
Edired Evidence
[William Link] William Link (Decemher 14, 1898 April 30, 1977) American lyricist, screenwriter, director, and television producer.
Counterfactual Claims
CrosAng
Geller passed away developed American television detective show Mannix.
POLYJUICE
Bruce Geller died 1978, did not developed American television detective show Mannix.
ChatGPT
Bruce Geller, who passed away in 1985, developed the American television detective show Mannix.
The executive producer of American television detective show Mannix died in 1877. The show was created by Joe Mannix and Richard Levinson.
our RACE
Table 5: Examples of edited evidence and counterfactual claims on HOVER training set. The differences from the original instance are highlighted in blue.
Augmented HOVER
Num.SUP
Num.REF
Nom.NEK
Total
REFUTE, and NEUTRAL. We remove the instances with the NEUTRAL label. As the evidence provided by Health Ver contains sev- eral sentences, we split it into multiple pieces of evidence to simulate multi-hop scenario.
Train
7.148
9,086
27,572
Dev
2,0000
2,000
2,000
6,000)
Table 6: The statistics of augmented HOVER with NEI instances. SUP, Num.REF number of SUP P instances, REF instances, and NEI instances, respectively.
REF and Num.NEI are the
PubHealth (Kotonya and Toni, 2020), a 4- way classification dataset for explainable fact verification with gold standard explanations by journalists in the public health setting. We only retain the instances with TRUE and FALSE labels, and the explanation provided is split into separate sentences as multiple pieces of evidence.
Health Ver (Sarrouti et al., 2021), an evidence- based fact verification dataset for health- related claims, where the relations between each piece of evidence and the associated claim are manually annotated as SUPPORT,

---

Given original claim with corresonndngg evidence und label (SUPPORTS or REFUTES), generute counterfactual claim based the evidence, taking
that the generated counterfactual claim is similar possible the original cluim, while being aware of linguistic diversity and the change labels.
Example:
Claim:
Bettany Hughes, an English historian scholar, born May 15th, 1967, presented "The Spartans".
Evidence:
The Spartans (documentary): " 'The Spartans was 3-part historical documentary scries first broadcast on UK terrestrial Channel 4 in 2003, presented by Bettuny Hughes.
Bettany Hughes: Bettany Hughes born May 15, 1967 is an English historian, author, and broadcaster.
Label: SUPPORTS
Generate counterfactual claim:
"The Spartans" is documentary presented by Bettany Hughes, American historian scholar born March 24, 1980.
Claim:
The writer Norman Alfred William Lindsay enjoyed boxing, but the author of The Hundred Secret Senses did not.
Evidence:
Amy Tan: Amy Tan born February 19, 1952) ) is an American writer whose works explore mother-daughter relationships and the Chinese American experience.
The Hundred Secret Senses: The Hundred Secret Senses bestselling 1995 novel by Chinese-Americun writer Amy Tun.
Norman Lindsay: Norman Alfied William Lindsay February 21 November 1969 was an Australian artist, etcher, sculptor, writer, editorial cartoonist, scale modeller, and an accomplished amateur boxer.
Label: SUPPORTS
Generate counterfactual claim:
Table 7: An example of prompt given to GPT-3 and ChatGPT for generating counterfactual claims.
PORTS and REFUTES labels in our experi- ments.
Dataset
Num.SUP
Num.REF REF
Total
HOVER Dev
2,000
2,000
4,000
FEVER Dev
6,666
6,666
13,332
We only test the performance of the basic multi- hop fact verification model the development of the above datasets in our experiments. The statisties are shown in Table 8.
FEVEROUS Dev
3,908
3,481
7,389
PolitiHop Dev
98
119
SCIFACT Dev
124
64
188
Health Ver Dev
533
391
924
PubHealth Dev
628
544
1,172
D Baselines
FM2 Dev
596
573
1,169
In our experiments, we compare our method with the following baselines.
VITAMINC Dev
31,484
22,528
54,012
Table 8: The statistics of the datasets we used in our experiments.
EDA (Wei and Zou, 2019), a data augmen- tation method that applies four simple opera- tions, including synonym replacement, dom insertion, random swap, and random deletion, original sentences to generate new instances.
FM2 (Eisenschlos et al., 2021), large-scale dataset of challenging claim-evidence pairs collected through a fun multi-player game which encourages adversarial instances and drastically lowers the number of the instances with "shorteuts". All the claims need to be verified (SUPPORTS, REFUTES).
CrossAug (Lee et al., 2021), a counterfac- tual data augmentation method that employs a two-stage augmentation pipeline to generate contrastive claims and evidence from existing SUP instances.
VITAMINC (Schuster et al., 2021), large- scale contrastive fact verification dataset, where each contrastive claim is manually writ- ten by annotators based on Wikipedia revi- sions. We only retain the instances with SUP.
POLYJUICE (Wu et al., 2021)), general- purpose counterfactual generator based on fine-tuned GPT-2 that allows for control over perturbation types and locations.

---

GPT-3 (text-davinci-003) (Brown et al.. 2020), large autoregressive language model with superb few-shot and in-context learning capabilities.
ChatGPT hupt--.--turb--0301 (OpenAI, 2022), a powerful GPT-3 based model which is trained to follow an instruction in prompt and provide a detailed response.
For EDA 3 and CrossAug", all the experimental setups of them are followed from the original pa- pers and all hyperparameters are set to the same values in the official code. For POLYJUICE', set the control code "negation", the beam size to 10, and generate one counterfactual claim for each original claim. All the inputs the above baselines are only the original claim.
For GPT-3 and ChatGPT, we make use of the APIs provided by OpenAI for generating coun- terfactual claims and design prompt with task introduction and demonstration as input, as shown in the Table 7.
/nttps::/gittubb.cmm http://hhhhhbccco//ssssse/iieeee cctpc///sssommeee////caaa.
compprduut.c "nttpp/u/openai.cmmmprruuutt
"https://openal


Reviews:
[{"review_id": "XtjV9fPd0V", "reviewer": "Reviewer_gQhw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for the insightful comments, and they are helpful for us to improve our work. We will re-proofread the paper and revise the typo grammar style and presentation based on your suggestions. Furthermore, our point-to-point responses to your comments are given below.\n\n>**Q1**: \nThe authors state that previous methods of generating counterfactuals, based on making local edits to the claim, struggle to preserve “logical relationships” between multiple hops of evidence. Furthermore, they state that the edits made by the baselines tested are “not valid for multi-hop fact verification” (lines 603-604). No evidence is given for either of these claims, and in fact, Table 4 shows that the baselines achieve a higher “Multi hop” score than their method.\n\n**R1**: \n\nMulti-hop fact verification is a task that necessitates **the inference of logical relationships between claim and evidence, as well as within the multiple pieces of evidence**, i.e., by aggregating the multi-hop evidence to verify the truthfulness of the claim. As shown in the third row of Tab.1, there are complex relationships between claim and multi-hop evidence, and within the multiple pieces of evidence. \n\n**(I) Qualitative Description**: Existing CAD methods generate counterfactual claims solely by modifying the original claim (e.g., *CrossAug* deletes the words in the claim), rather than using the information of multi-hop evidence. We empirically find that this way will **compromise the semantic integrity of the generated counterfactuals**. As the *CrossAug* shown in Tab.3, the counterfactual claim is generated by removing the entity \"Love\" (the name of the song), while this entity is the key word to connect the claim and evidence E1, as well as the evidence {E1,E2,E3}. Without this information, we are unsure that whether the same song that the claim and evidence discussed, which results in the multiple pieces of evidence cannot be aggregated as a  whole to verify the claim, thus producing the invalid counterfactuals.\n\n**(II) Quantitative Evaluation**: To our knowledge, there currently exists no automatic metrics to evaluate the logical relationship within the instance. In our paper, we adopt the metric *MoverScore* to evaluate the *Multi hop* of the counterfactuals, while this is compromised. The more reasonable approach to evaluate the logical relationship is adopting the **Human Evaluation**. To this end, we ask three postgraduate students to evaluate the generated counterfactuals of different methods, as shown in Fig.2, our RACE outperforms all other baselines on *Multi hop*, which validate the effectiveness of our method. \n\nFurthermore, *label flipping* is a basic factor in the validity of the counterfactual instances. The lower *label flipping rates* of baselines (as shown in Tab.4 and Fig.2) also present the invalidity of counterfactuals generation by previous methods.\n\n>**Q2**: \nThey also claim that their approach “outperforms the SOTA baselines” (line 24). However, the results in Table 2 seem more mixed to me. A RACE model achieves the highest accuracy on six of the nine datasets, but the results vary widely based on the generation model used (BART vs. GPT-2 vs. T5). Also, the authors do not mention how many runs were performed; given the small effect sizes (in most cases, the difference between the best RACE model and the best baseline is within one or two points), I am unsure how robust the results are.\n\n**R2**:\n\n**(I)** We adopt different generation models to test the effect of the generation ability on our method. Overall, compared to the baselines, our method yields a comparable or improved performance based on different generation model, especially the results based on T5-base. We empirically find that different generation models have more prominent performance on specific datasets, e.g., *BART* on *FEVEROUS* dataset, *T5* on 5 datasets. In fact, by experiments conducted on the *T5-large*, we find that the performance can be further improved on most datasets, as shown in the table below. We will report more experimental results and the detailed analysis in the revised paper.\n\n**(II)** The results reported in Tab.2 are mean of the accuracy over three different seeds, we will report the standard deviation of all the methods in the revised paper to illustrate the robust of our method.\n\n| Source of CFs   | HoVer | FEVER | FEVEROUS | PolitiHop |SCIFACT | HealthVer | PubHealth | FM2 | VitaminC |\n|:----------------|:-----:|:---------:|:---------:|:--------:|:---------:|:--------:|:---------:|:---------:|:--------:|\n| RACE (T5-base)  | 83.15 | 75.05     | 70.50     | 52.94    | 65.43    | 55.41   | 53.52    | 62.19     | 66.50    |\n| RACE (T5-large) | **83.18** | **78.11**   | **71.55**     | 47.06   | 62.77    | **55.84** |**56.59**    | 61.16    | **67.71**    |\n\n\n\n\n>**Q3**:\nWhen using GPT-3 and ChatGPT to generate counterfactuals, it seems that those models are given an example of a counterfactual claim in the prompt (as in Table 7). My concern is that it may be possible to engineer GPT-3/ChatGPT to generate counterfactuals with certain properties, based on the example counterfactual. For instance, if in the prompt, the example counterfactual is lexically similar to the original claim, then the counterfactuals generated by GPT-3/ChatGPT may analogously be lexically similar to the given claim, i.e., score lower on diversity. More detail should be given about how the prompt was designed.\n\n**R3**: \n\nThrough extensive experiments on GPT-3/ChatGPT, we have found that such a simple prompt performs better than the other prompts. \n\n**(I)** We attempt the examples with linguistically diverse and lexically distinct from the original claim. However, it has found that LLMs tend to improve the linguistic diversity of counterfactuals by replacing the keywords with irrelevant entities during generation (e.g., *Charlotte Ninon Coleman* in Tab.3 --> *Charlotte Bronte*), resulting in counterfactual claims can not be verified.\n\n**(II)** We attempt the examples with various reasoning types. Unfortunately, we experimentally found that LLMs have difficulty in understanding the multi-hop reasoning type, resulting in the poor quality of generated counterfactual claims. Furthermore, the generalization of the models trained on these counterfactuals declined on the vast majority of out-of-domain datasets. \n\nThe following is a comparison of the prompt and the corresponding generated counterfactual claim. We can observe that the Prompt with various reasoning types generates worse results compared to our simple prompt, i.e., LLMs cannot generate counterfactual claims well around the anchor entity **boxing**.\n\n***Prompt in our paper:***\n```\nGiven a original claim with corresponding evidence and label (SUPPORTS or REFUTES), generate a counterfactual claim based on the evidence, taking care to ensure that the generated counterfactual claim is as similar as possible to the original claim, while being aware of linguistic diversity and the change of labels.\nExample:\nClaim: Bettany Hughes, an English historian scholar, born May 15th, 1967, presented \"The Spartans\".\nEvidence:\nThe Spartans (documentary): “ The Spartans ” was a 3-part historical documentary series first broadcast on UK terrestrial Channel 4in 2003 , presented by Bettany Hughes .\nBettany Hughes: Bettany Hughes ( born May 15 , 1967 ) is an English historian , author , and broadcaster .\nLabel: SUPPORTS\nGenerate a counterfactual claim:\n“The Spartans” is a documentary presented by Bettany Hughes, an American historian scholar born on March 24, 1980.\n\nClaim: The writer Norman Alfred William Lindsay enjoyed boxing, but the author of The Hundred Secret Senses did not.\nEvidence:\nAmy Tan: Amy Tan ( born February 19 , 1952 ) is an American writer whose works explore mother-daughter relationships and the Chinese American experience .\nThe Hundred Secret Senses: The Hundred Secret Senses is a bestselling 1995 novel by Chinese-American writer Amy Tan .\nNorman Lindsay: Norman Alfred William Lindsay ( 22 February 1879 – 21 November 1969 ) was an Australian artist , etcher , sculptor , writer , editorial cartoonist , scale modeller , and an accomplished amateur boxer .\nLabel: SUPPORTS\nGenerate a counterfactual claim:\n```\n***Generated counterfactual claim:***\n```\nThe writer Norman Alfred William Lindsay not only enjoyed boxing, but he also authored The Hundred Secret Senses.\n```\n\n***Prompt with examples of various reasoning types: (E denotes evidence and C denotes claim.)***\n```\nGiven a original claim with corresponding evidence and label (SUPPORTS or REFUTES), generate a counterfactual claim based on the evidence, taking care to ensure that the generated counterfactual claim is as similar as possible to the original claim, while being aware of linguistic diversity and the change of labels.\nExample:\n\n## Reasoning type: {E1<->E2}->C\n\nClaim: Bettany Hughes, an English historian scholar, born May 15th, 1967, presented \"The Spartans\".\nEvidence:\n(E1)The Spartans (documentary): “ The Spartans ” was a 3-part historical documentary series first broadcast on UK terrestrial Channel 4in 2003 , presented by Bettany Hughes .\n(E2)Bettany Hughes: Bettany Hughes ( born May 15 , 1967 ) is an English historian , author , and broadcaster .\nLabel: SUPPORTS\nGenerate a counterfactual claim:\n“The Spartans” is a documentary presented by Bettany Hughes, an American historian scholar born on March 24, 1980.\n\n## Reasoning type: E1 <-> E3 <-> E2, E1 -> C <- E2\n\nClaim: Gina Bramhill was born in an Italian village. The 2011 population of the unitary authority area that includes this village was 167,446.\nEvidence:\n(E1)North Lincolnshire: The population of the Unitary Authority at the 2011 census was 167,446 .\n(E2)Gina Bramhill: Gina Bramhill was born in Eastoft , where she grew up on a farm .\n(E3)Eastoft: Eastoft is a village and civil parish in North Lincolnshire , England .\nLabel: REFUTES\nGenerate a counterfactual claim:\nIn 2011, the population of the North Lincolnshire, including an English village where the birthplace of Gina Bramshill is located, was 167,446.\n\n## Reasoning type: E1 -> C <- E2\n\nClaim: University of Wisconsin Madison is a research university and University of Notre Dame is a research university.\nEvidence:\n(E1)University of Wisconsin–Madison: The University of Wisconsin–Madison ( also known as University of Wisconsin , Wisconsin , UW , or regionally as UW–Madison , or simply Madison ) is a public research university in Madison , Wisconsin , United States .\n(E2)University of Notre Dame: The University of Notre Dame du Lac ( or simply Notre Dame ) is a Catholic research university located adjacent to South Bend , Indiana , in the United States .\nLabel: SUPPORTS\nGenerate a counterfactual claim:\nThe American University of Wisconsin-Madison is a public research university while Notre Dame is not.\n\n\nClaim: The writer Norman Alfred William Lindsay enjoyed boxing, but the author of The Hundred Secret Senses did not.\nEvidence:\nAmy Tan: Amy Tan ( born February 19 , 1952 ) is an American writer whose works explore mother-daughter relationships and the Chinese American experience .\nThe Hundred Secret Senses: The Hundred Secret Senses is a bestselling 1995 novel by Chinese-American writer Amy Tan .\nNorman Lindsay: Norman Alfred William Lindsay ( 22 February 1879 – 21 November 1969 ) was an Australian artist , etcher , sculptor , writer , editorial cartoonist , scale modeller , and an accomplished amateur boxer .\nLabel: SUPPORTS\nGenerate a counterfactual claim:\n```\n***Generated counterfactual claim:***\n```\nThe writer Norman Alfred William Lindsay not only enjoyed boxing, but also shared a passion for it with the author of The Hundred Secret Senses, Amy Tan.\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper describes some very interesting ideas regarding how to generate counterfactuals for multi-hop fact verification, that are worthy of further exploration. The Explain-Edit-Generate pipeline intuitively makes a lot of sense given the goals. The authors successfully show that their generated counterfactuals are linguistically diverse and label flipping.", "reasons_to_reject": "The authors state that previous methods of generating counterfactuals, based on making local edits to the claim, struggle to preserve “logical relationships” between multiple hops of evidence. Furthermore, they state that the edits made by the baselines tested are “not valid for multi-hop fact verification” (lines 603-604). No evidence is given for either of these claims, and in fact, Table 4 shows that the baselines achieve a higher “Multi hop” score than their method.\n\nThey also claim that their approach “outperforms the SOTA baselines” (line 24). However, the results in Table 2 seem more mixed to me. A RACE model achieves the highest accuracy on six of the nine datasets, but the results vary widely based on the generation model used (BART vs. GPT-2 vs. T5). Also, the authors do not mention how many runs were performed; given the small effect sizes (in most cases, the difference between the best RACE model and the best baseline is within one or two points), I am unsure how robust the results are.\n\nWhen using GPT-3 and ChatGPT to generate counterfactuals, it seems that those models are given an example of a counterfactual claim in the prompt (as in Table 7). My concern is that it may be possible to engineer GPT-3/ChatGPT to generate counterfactuals with certain properties, based on the example counterfactual. For instance, if in the prompt, the example counterfactual is lexically similar to the original claim, then the counterfactuals generated by GPT-3/ChatGPT may analogously be lexically similar to the given claim, i.e., score lower on diversity. More detail should be given about how the prompt was designed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors describe a method to generate counterfactual data for a multi-hop fact verification task, specifically by passing original instances through RACE, an Explain-Edit-Generate pipeline. First, the Explainer extracts sentence and token rationales from the given multi-hop evidence. Then, the Editor identifies certain named entities among the token rationales, and either replaces or permutes them, making sure that the edited evidence now refutes the original claim. The Generator then creates a counterfactual claim supported by the edited evidence (and refuted by the original evidence). After filtering the counterfactual claims for minimal semantic shift from the original claim, the authors train a basic multi-hop fact verification model on the augmented dataset, comparing it to models trained on the original data as well as on datasets augmented with (counterfactual) instances from other sources. They find that their counterfactual data augmentation scheme results in comparable or better results on the multi-hop fact verification task, while the generated counterfactuals outscore those from other sources in flip rate, fluency, and diversity.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper would benefit from another round of proofreading. For example:\n\nline 3: delete “with”\n\nline 4: “brutal” → I would just say “poorly” or something like that\n\nline 7: “is” → “are”\n\nfigure 2: “inner-rate” → “inter-rater”\n\netc.\n\nAlso, while the paper usually uses the term “logical” to refer to the logical relationships between multiple hops of evidence, in line 605 (and possibly line 131?) it is used to mean “label flipping”; this should be made consistent."}}, {"review_id": "g8Ch2tKDkV", "reviewer": "Reviewer_HUk1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging words and constructive comments. In the following, your comments are first stated and then followed by our point-by-point responses.\n\n>**Q1**:\nThis paper adopts three generation models as the backbone model and compare their performances with SOTA methods such as GPT-3 and ChatGPT. In my opinion, GPT-3/ChatGPT has far more parameters than BART/GPT-2/T5, whether such comparison is fair? Can the proposed method adapts well to those huger LLM models?\n\n**R1**:\n\nAs your mentioned, our method generates counterfactual claims by fine-tuning generation models with relative less parameters, instead of using in-context learning to bootstrap LLMs with frozen parameters. Although GPT-3/ChatGPT has far more parameters than BART/GPT-2/T5, the performance of our method still outperforms these LLMs, which validates the effectiveness of our proposed method.\n\nTo explore the effect of the number of parameters, we conducted additional experiments on the *T5-large*. We empirically find that compared to T5-base, counterfactuals generated by fine-tuned T5-large are more fluent and linguistically diverse, and further improve the model performance on most datasets. This illustrates that it is possible to further improve the effectiveness of our method by using a more powerful generation model. The results are shown in the tables below.\n\nDuring our experiments, we empirically find that it is difficult for LLMs to generate high quality of counterfactuals according to our framework, and no improved performance can be observed. We will release the source code to promote the further research in this filed.\n\n| Source of CFs   | HoVer | FEVER | FEVEROUS | PolitiHop |SCIFACT | HealthVer | PubHealth | FM2 | VitaminC |\n|:----------------|:-----:|:---------:|:---------:|:--------:|:---------:|:--------:|:---------:|:---------:|:--------:|\n| RACE (T5-base)  | 83.15 | 75.05     | 70.50     | 52.94    | 65.43    | 55.41   | 53.52    | 62.19     | 66.50    |\n| RACE (T5-large) | **83.18** | **78.11**   | **71.55**     | 47.06   | 62.77    | **55.84** |**56.59**    | 61.16    | **67.71**    |\n\nTable 1: Fact verification accuracies of data augmentation methods on different development set.\n\n| Method | Flip.$\\uparrow$ | Flu.$\\downarrow$ | Sim.$\\uparrow$ | Div.$\\uparrow$ | M.h.$\\uparrow$|\n|:------|:-----:|:---------:|:---------:|:--------:|:------:|\n| RACE (T5-base)  | 0.9457 | 55.81 | 0.5770 | 11.19 | 0.5763 |\n| RACE (T5-large) | 0.9402 | **55.03** | 0.5770 | **11.22** | 0.5763 |\n\nTable 2: Automatic intrinsic evaluation results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The method is explained clearly and easy to follow.\n- The proposed method has high interpretability. \n- Introducing the idea of counterfactual data augmentation seems novel.\n- Experiments are solid and comprehensive.", "reasons_to_reject": "- This paper adopts three generation models as the backbone model and compare their performances with SOTA methods such as GPT-3 and ChatGPT. In my opinion, GPT-3/ChatGPT has far more parameters than BART/GPT-2/T5, whether such comparison is fair? Can the proposed method adapts well to those huger LLM models?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a counterfactual data augmentation method to handle the multi-hop fact verification task. An Explain-Edit-Generate architecture is leveraged to generate diverse and fluent counterfactuals, and the checking and filtering modules are proposed to regularize the counterfactual data. Experiments demonstrate the effectiveness of the proposed method.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "A5a7cHz2tr", "reviewer": "Reviewer_gZJT", "summary": "", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and suggestions, and they are exceedingly helpful to improve our paper. We will re-proofread the paper and revise the typo grammar style and presentation based on your suggestions. Furthermore, our point-to-point responses to your comments are given below.\n\n>**Q1**:\nThe proposed method has multiple pipeline stages, which are quite complex and might be difficult to adapt to other tasks or datasets. Dependence on heuristic rules in the editing process also reduces generalization capabilities.\n\n**R1**:\n\nDue to the complexity of the multi-hop fact verification task, to generate valid and effective counterfactuals for this task, we design a relative complex pipeline method by regularizing the different properties (i.e., logical relationship, label flipping, and diversity) at different stage during our generation process. In fact, we empirically find that it is very challenging to generate valid counterfactuals using an end-to-end method, even with the LLMs (e.g., *ChatGPT* in Tab.2).\n\nTo make our method reproducible, we have simplified the heuristic rules as much as possible by only considering the **entity type** in the editing process. Even though, to ensure the effectiveness of the proposed method on multi-hop fact verification task, we have unfortunate to sacrifice the generalization capabilities of the method on other tasks.\n\n>**Q2**:\nAlthough the results show improvement, it is still limited. For instance, the advantage is not significant in some out-of-domain results. More analysis is needed on its robustness and generalization.\n\n**R2**:\n\nIn fact, substantial generalization improvement is difficult due to the complexity of the multi-hop fact verification task. Compared to the original model (i.e., **None** in Tab.2), we empirically find that the previous methods tend to decrease the generalization ability of the model in most datasets, e.g., in Tab.2, the *CrossAug* shows a decrease in accuracy of 5.74% and 13.86% on HealthVer and PubHealth, respectively. However, our proposed method results in a comparable or better performance on the generalization of the model in most cases, illustrating the effectiveness of our proposed method. We will report the standard deviation of each method to validate the robustness of our method.\n\n>**Q3**:\nThere could be a more thorough exploration of using large language models for generation. The prompts given to them can be improved to fully leverage their potential.\n\n**R3**:\n\nWe have experimentally found that it is difficult for LLMs to generate counterfactuals with one step for multi-hop fact verification tasks via in-context learning. So far, we have attempted a variety of complex prompts for LLMs, but still failed to achieve satisfactory results. Please refer to the Response **R3** to **Reviewer 1** for details. A potential way is to introduce complex prompts such as Chain-of-Thought, Graph-of-Thought, Tree-of-Thought, etc., so that LLMs can accomplish the counterfactuals generation step by step. That's what we'll explore in the future.\n\n>**Q4**:\nApart from manually analyzing some samples, there is no manual evaluation of the counterfactual statements. The existing sample analysis is not sufficient and more rigorous manual evaluations should be undertaken.\n\n**R4**:\n\nIndeed, we conduct a thorough manual evaluation of the generated counterfactual claims as Section 5.3 described. The results are shown in Fig.2. Note that we only manually evaluate the counterfactual claims since the previous method cannot generate the counterfactual evidence as our proposed method. The description in line 564 of the paper seems to be a bit off, and we will clarify this point in the revised paper.\n\n\n>**Q5**:\nThe section on experimental results could further enhance result analysis, such as discussing more about the reasons behind the results, extracting main findings, etc., to enrich the content of this section.\n\n**R5**:\n\nThank you for your constructive suggestions, we will report more results and the corresponding analysis to enhance the content of our experiment in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\nBased on my understanding of this paper, it may bring the following benefits:\n\n1. It provides a new approach for enhancing the generalization and robustness of models for multi-hop fact verification tasks. While counterfactual data augmentation is gaining traction in NLP, research focused on complex reasoning tasks is relatively scarce. This paper provides valuable exploration into how to generate high-quality counterfactual data for such tasks.\n\n2. It designs a targeted counterfactual data generation pipeline, considering the characteristics of multi-hop verification tasks, and generating high-quality training data in terms of logic and semantics. This could serve as a reference or foundation for future research.\n\n3. It introduces the concept of rationales extracted using interpretability methods, focusing on the causality characteristics of the text. This attempt to integrate machine learning interpretability can provide certain inspirations and references for related research.\n\n4. It enriches the application of counterfactual learning in the NLP field, demonstrating the potential of this technique in enhancing model generalization capabilities. This can promote more research focusing on data augmentation and robustness.\n\n5. It empirically verifies that counterfactual data augmentation has a good effect in improving the generalization ability of multi-hop fact verification. This provides new insights for research in this task.", "reasons_to_reject": "The potential shortcomings and assumed risks in the paper could be:\n\n1. The proposed method has multiple pipeline stages, which are quite complex and might be difficult to adapt to other tasks or datasets. Dependence on heuristic rules in the editing process also reduces generalization capabilities.\n\n2. Although the results show improvement, it is still limited. For instance, the advantage is not significant in some out-of-domain results. More analysis is needed on its robustness and generalization.\n\n3. There could be a more thorough exploration of using large language models for generation. The prompts given to them can be improved to fully leverage their potential.\n\n4. Apart from manually analyzing some samples, there is no manual evaluation of the counterfactual statements. The existing sample analysis is not sufficient and more rigorous manual evaluations should be undertaken.", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "The main focus of this paper is the Counterfactual Data Augmentation (CDA) method for multi-hop fact verification tasks. The paper argues that current CDA methods mainly cater to single-sentence tasks and struggle to handle the complex logical relations in multi-hop fact verification tasks. To address this issue, a new pipeline method called RACE is proposed, which can generate logically coherent, linguistically diverse, and label-flipping counterfactual data to enhance the generalization and robustness of multi-hop fact verification models.\n\nThe main contributions of this paper include:\n\n1. The proposal of an Explain-Edit-Generate framework, which generates counterfactual evidence that is logically consistent with the original evidence but differs factually, based on the extracted rationales for evidence editing and statement generation.\n\n2. During the statement generation, a constrained beam search and entity constraint are utilized to ensure the generated counterfactual statements are semantically diverse and have multi-hop relations with the evidence.\n\n3. The introduction of a Checking and Filtering module, which regularizes the generated counterfactual data from aspects of semantic reasonability and topic consistency.\n\n4. Experimental results demonstrate that the counterfactual data generated using the method in this paper can improve the model performance on in-domain, out-of-domain, and adversarial datasets. At the same time, the counterfactual statements generated in this study exhibit superior logical coherence and linguistic diversity.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "While reading the paper in detail, I noticed some issues with language expression that need to be improved:\n\n1. On line 07, I suggest changing \"minimally altering\" to \"minimal perturbations\" for a more precise expression.\n2. On line 459, change \"provides\" to \"provide\" to ensure subject-verb agreement.\n3. On line 126, change \"reveals\" to \"reveal\" for subject-verb agreement.\n\nFurthermore, regarding the structure of the paper, I suggest:\n\nThe section on experimental results could further enhance result analysis, such as discussing more about the reasons behind the results, extracting main findings, etc., to enrich the content of this section."}}]

Author rebuttal:

Meta-review: The paper presents a data augmentation technique for Multi-hop Fact Verification which generates diverse counterfactuals via a proposed  Explain-Edit-Generate architecture. The experimental results highlights the potential of the approach in generating linguistically diverse counterfactual data while preserving the logical relationships. The writing requires further polishing (starting from the abstract, but this is minor could be addressed in the current version). Several points were clarified during the discussion phase which I strongly encourage the authors to incorporate into their papers.
Decision: Accept-Main