ClaimDiff: Comparing and Contrasting Claims on Contentious Issues
Year: 2023

Abstract:
With the growing importance of detecting misinformation, many studies have focused on verifying factual claims by retrieving evidence. However, canonical fact verification tasks do not apply to catching subtle differences in factually consistent claims, which might still bias the readers, especially on contentious political or economic issues. Our underlying assumption is that among the trusted sources, one’s argument is not necessarily more true than the other, requiring comparison rather than verification. In this study, we propose ClaimDiff, a novel dataset that primarily focuses on comparing the nuance between claim pairs. In ClaimDiff, we provide 2,941 annotated claim pairs from 268 news articles. We observe that while humans are capable of detecting the nuances between claims, strong baselines struggle to detect them, showing over a 19% absolute gap with the humans. We hope this initial study could help readers to gain an unbiased grasp of contentious issues through machine-aided comparison.

heading: 1 Introduction
text: With an ever-increasing amount of textual information on the web, many researchers have focused on detecting misinformation from diverse sources, such as fake news (Potthast et al., 2018; Nguyen et al., 2020) and rumor tweets (Zubiaga et al., 2016; Kochkina et al., 2018). In particular, fact verification has become a popular task due to its utility and the availability of reliable datasets such as FEVER (Thorne et al., 2018; Aly et al., 2021).
For contentious issues, however, fact verification alone is not sufficient; comparing and contrasting claims from the opposing sides are necessary to
⋆This work was done during internship in NAVER AI Lab.
†Is now at Samsung Research. ‡Is now at Google.
gain an unbiased understanding of the issue. Figure 1 presents two articles reporting on the treatment of long COVID – long-term physical and mental symptoms that can occur after COVID-19 infection. Although both articles are published in the same week, their perspectives on long COVID are quite different; article A downplays the risk of long COVID stating that it has become less common in the recent COVID variants, while article B expresses concerns about the increase in the number of people suffering from long COVID. In this way, even articles from trusted sources may provide biased views about an issue.
In this paper, we present ClaimDiff, a novel dataset consisting of 2,941 claim pairs extracted from 268 news articles on 134 contentious issues.1 ClaimDiff comes in two variations—ClaimDiff-S and ClaimDiff-W—each consisting of labels targeting a different relation: determining whether a claim Strengthens and Weakens another claim, respectively. For instance, the two claims in Figure 1 weaken each other, providing inconsistent perspectives surrounding the undisputed factual information — the human body has a natural ability to heal from long COVID. More specifically, claim A argues that medical treatments should be geared toward supporting the natural ability to heal, while claim B calls for more active interventions for fast recovery. ClaimDiff focuses on recognizing such relations between two claims on an issue; this is distinguished from existing tasks such as fact verification — verifying the veracity of a claim using evidence text (Vlachos and Riedel, 2014; Thorne et al., 2018) — and stance detection — identifying the stance of a claim toward a topic of interest (Mohammad et al., 2016; Derczynski et al.).
We also demonstrate the efficacy of ClaimDiff on two tasks it supports — relation classification and rationale extraction — and an extended ap-
1The articles are collected from allsides.com, licensed under a CC BY-NC 4.0 license.
4711
plication — document-level ClaimDiff. Relation classification experiments tests the ability to identify strengthen and weaken relations. Rationale extraction tests the ability to find the rationale for a given relation in extractive and generative scenarios. Lastly, although ClaimDiff only provides sentence-level annotations, models trained on it allow document-level analyses. In this documentlevel ClaimDiff, we test the potential of analyzing points of agreement and disagreement in documents.
The main contributions of our work are:
• We present ClaimDiff, a novel dataset of claim pairs on contentious issues. Each pair comes with strengthen and weaken relation labels, as well as a rationale for choosing the label.
• We present competitive baselines for the dataset, leveraging finetuning, parameterefficient finetuning, and zero-shot approaches, along with human performance.
• We showcase how models trained on ClaimDiff can be used to compare documents on an issue, indicating specific points of agreement and dispute, in document-level ClaimDiff.

heading: 2 Related Works
text: Dealing with Misinformation Detecting and avoiding misinformation received intense interest with a massive amount of information on the web. Existing works introduced benchmarks with a broad spectrum of sources, including rumors in social media (Potthast et al., 2018; Kochkina et al.,
2018) and fake news (Zubiaga et al., 2016). Other researchers focus on dealing with an exploding amount of misinformation on global events, such as the COVID-19 pandemic (Saakyan et al., 2021; Jiang et al., 2021; Alam et al., 2021; Weinzierl and Harabagiu, 2022). While many existing benchmarks aim to detect less reliable information on unverified sources, our work targets subtle differences in trusted sources.
Claim Verification Claim verification verifies the factuality of the target sentence with respect to a reliable truth from verified sources. Automatic verification shows remarkable progress with the introduction of rich claim verification datasets (Vlachos and Riedel, 2014; Thorne et al., 2018; Hanselowski et al., 2019; Aly et al., 2021; Khan et al., 2022). Existing claim verification datasets introduce many variants, including a shift in domains and languages of claims; claims from political sources (Wang, 2017; Garimella et al., 2018), scientific claims (Wadden et al., 2020), climate change-related claims (Leippold and Diggelmann, 2020), Arabic claims (Baly et al., 2018; Alhindi et al., 2021), and Danish claims (Nørregaard and Derczynski, 2021). Our assumption is different from claim verification in that one claim is not necessarily more true than the other. As a result, ClaimDiff focues on comparison between claims rather than verification.
Stance Detection Stance detection aims to predict the stance of a claim toward a specific topic between agreeing or opposing perspectives. Mohammad et al. (2016) propose SemEval challenges to
predict the stance of tweets toward target keywords. Derczynski et al. further presents a sub-challenge to detect the stance of corresponding threads of rumor tweets. Other benchmarks are introduced with diverse challenges, such as claim-based stance detection (Ferreira and Vlachos, 2016; Bar-Haim et al., 2017), stance detection with evidence (Chen et al., 2019), and stance detection over political domains (Li et al., 2021).

heading: 3 ClaimDiff
text: In this section, we formally define the tasks supported by ClaimDiff, describes how the dataset is constructed, and provide the statistics and analysis of the resulting dataset.

heading: 3.1 Task Description
text: ClaimDiff comes in two variations—ClaimDiffS targeting strengthen relations, and ClaimDiffW targeting weaken relations. Both variations of the dataset were designed to support the following tasks.
Relation Classification Relation classification aims to determine if claims from two different documents are in a relation: strengthen for ClaimDiffS, and weaken for ClaimDiff-W. For instance, as shown in Figure 1, claim A and claim B are both about the treatment of long COVID. Claim A and B exhibit opposing positions for the body’s natural recovery, respectively. We want to classify this case into weakens as claim A weakens claim B. Although, in this case, claim B weakens A as well, note that the relationship is not guaranteed to be symmetric in general.
More formally, for ClaimDiff-S, given a claim pair (c1, c2), the objective is to return true if c1 strengthens c2, and false, otherwise. For ClaimDiffW, given a claim pair (c1, c2), the objective is to return true if c1 strengthens c2, and false, otherwise. Note that for any given claim pair, the answer cannot be true for both variations of ClaimDiff.
Rationale Extraction Rationale extraction aims to extract phrases from a claim in a relation: strengthen for ClaimDiff-S, and weaken for ClaimDiff-W. For instance, in Figure 1, a rationale for ClaimDiff-W is ‘body’s natural ability to heal itself could be an important’, which weakens claim B about the view that more active interventions for fast recovery.
More formally, given a claim pair (c1, c2) in a relation, the objective is to extract phrases in c1 that provide a rationale for relation. Note that a single pair of claims can have multiple rationales.

heading: 3.2 Constructing ClaimDiff Dataset
text: Raw Data Collection We first collect a group of news articles from AllSides2 headlines. Allsides provides a balanced search of news with all sides of the political spectrum. More specifically, in AllSide headlines, there are groups of news articles about the same issues from different media press. The article groups have a broad political spectrum; each belongs to one of (left, center, and right) political stances. We crawl the headline pages uploaded from 2012-06-01 to 2021-11-21, covering more than 180 media sources. We choose two articles with left and right labels if possible.
To construct the claim pairs, we filter the nonoverlapping sentence pairs from the article pair. As most claim pairs do not provide overlapping contents, a large proportion of pairs are filtered out. We apply an additional filtering process using Amazon Mechanical Turk (MTurk) 3 to collect the overlapping claim pairs. Each worker is asked to answer the question, ’Does the target sentence overlap with a given sentence?’, where each sentence is extracted from two different articles. Given a single example, three workers made a response. We collected the pairs if at least two workers answered that the given pairs are "overlapping". The details of the filtering process are shown in Appendix A.
Annotation Process After the filtering process, we conducted an annotation process with 15 inhouse expert annotators to obtain the final data. Given a pair of claims, the annotators were requested to determine the stance among strengthen, weaken and no effect. If they choose strengthen or weaken, the annotators had to select the phrases from the claim that strengthen or weaken the other claim. The overall interface for the data collection process is shown in Figure 4.
For each single claim pair, three to five participants submitted their responses. We collect the responses and convert the relation options to scaled values. We first consider strengthen as 1, weaken as -1, and no effect as 0 and average the choices after conversion. We filter out the pairs with absolute average values between (0, 0.5), which means the
2https://www.allsides.com/unbiased-balanced-news/ 3https://www.mturk.com/
relations are ambiguous.4 The pairs with positive values are mapped into strengthening claims (1 for ClaimDiff-S), and the pairs with negative values are mapped into weakening claims (1 for ClaimDiffW). If the resulting values are exactly 0, the claims become 0 for both ClaimDiff-S and ClaimDiff-W. We measure the average inner-annotation agreement by Krippendorff’s alpha (Hayes and Krippendorff, 2007). The scores are 0.46 and 0.47 for ClaimDiff-S and ClaimDiff-W, respectively.
Constructing Test-doc Dataset We aim to build an application for understanding contentious issues with diverse views on a fine-grained level. Rather than classifying an article in a single label (i.e., containing left or right political bias), ClaimDiff enables a comparison between two articles in a sentence-wise manner. However, this requires a further extension of ClaimDiff from sentence pairs to document pairs, having significantly different distributions. To be coherent with the realworld distribution over whole articles, we provide an additional test dataset, test-doc, following the distribution over article pairs. Test-doc can be considered as an unfiltered test dataset, resulting in a high ratio of non-overlapping pairs. We collect the non-overlapping claim pairs of articles in the test dataset that are obtained from the previous filtering step. Over 70% of the claim pairs from article pairs are not overlapped, resulting in a highly skewed distribution. We combine filtered-out claim pairs with a standard test dataset to construct the final test-doc dataset.

heading: 3.3 Dataset Statistics
text: We extract the pairs from a group of articles sharing the same topic, published by multiple presses. Our final dataset contains articles from 47 presses. The top-3 presses with the highest appearance are Fox News, CNN, and Washington Times. We further
4Unfiltered data with average response scores are also publicly available.
analyze the topic diversity of our dataset by collecting the tag information. Tag is the list of words representing the topic provided in Allsides headlines. For instance, tags for a topic, Supreme Court Sides With Google in Copyright Dispute Case, are Supreme Court, Copyright, Google, and Oracle. The number of unique tags is 276, while each topic includes an average of 3.6 tags. The most common issues are Elections, Donald Trump, and Coronavirus. The overall lists of presses and tags are presented in Appendix D.
Table 1 presents the overall statistics of ClaimDiff. ClaimDiff dataset provides 2,941 examples, extracted from 268 articles with 134 issues. Note that ClaimDiff-S and ClaimDiff-W consists of the same claim pairs with different labels. Since nonoverlapping claim pairs do not provide rationales, the number of rationales is less than the overall claim pairs. Each pair contains an average of 1.4 rationales with an average length of 13.2 tokens.5 In the train and test dataset, "strengthening" pairs are available to be found with more than 50% appearance. Finding the "weakening" claim pairs is more challenging, resulting in 7.50% weakening examples in the test-doc environment.

heading: 3.4 Dataset Analysis
text: This part analyzes claim pairs in ClaimDiff with respect to the class label. We first provide the subjectivity analysis over claims with positive labels in ClaimDiff-S and ClaimDiff-W. ClaimDiff includes both subjective and objective claims, indicating the proposed task is designed to predict a more general relation between each claim pair. We further present prediction results of the natural language inference (NLI) and fact verification (FEVER) models. The results indicate that models trained on the datasets are not suitable for understanding the nuances.6
Subjectivity Analysis To analyze the subjectivity of claims, we randomly sample 50 "strengthening" examples from ClaimDiff-S and 50 "weakening" examples from ClaimDiff-W test data. We manually label each claim in a pair with subjective or objective. Following Wiebe and Riloff (2005), we distinguished subjective and objective claims based on whether each claim includes at least one private state – opinions, evaluations, emotions, and speculations. We found that "strengthening" pairs
5We use spaCy tokenizer for tokenizing the rationales. 6Check the Appendix E for additional information.
have a high proportion of objective claims on both claim A and claim B (A: 72%, B: 80%). However, "weakening" examples include more than 40% of subjective claims (A: 44%, B: 48%), indicating more diverse patterns in "weakening" relations. We expect that ClaimDiff-W to be more challenging not only because of the skewed distribution but also because of the more diverse composition of claims.
Prediction Results of NLI / FEVER Model To compare the ClaimDiff with previous sentence pair classification tasks, we analyze prediction results of transformer-based models trained on NLI and FEVER. We use RoBERTa-large trained on MNLI (Williams et al., 2018) and FEVER (Thorne et al., 2018). Each model yielded 90.2 (MNLI) and 75.6 (FEVER) F1 scores, respectively. Among 614 "strengthening" and 239 "weakening" pairs, the MNLI model predicts 561 and 203 pairs as neutral. FEVER model predicts 532 (strengthen) and 226 (weaken) examples as not enough info. These failures might come from multiple reasons, including the domain shift in claims and the different goals of each task. However, we observe that "weakening" pairs contain a slightly higher ratio of contradiction (strengthen: 5.7% vs. weaken: 15%) and 0 entailment with the MNLI model. "Strengthening" pairs show the difference in FEVER, containing more support (FEVER, 9.0% vs. 1.6%) examples. MNLI and FEVER models might be able to distinguish weakening and strengthening examples from others, which can work like the prior knowledge for solving ClaimDiff in Section 4.

heading: 4 Task 1 - Relation Classification

heading: 4.1 Experimental Setup
text: In this section, We present the baselines with different learning strategies: (1) finetuning, (2) parameter-efficient finetuning, and (3) zero-shot baselines. Implementation details of each model are described in Appendix B.
Finetuning Baselines We finetune pre-trained language models for the sentence classification tasks. Each model takes a pair of claims as inputs and predicts whether the first claim strengthens/weakens the other. We finetune models with a weighted loss function, where the class weight is determined by the label distribution.7 We construct the development data by sampling 30% of
7Training without the weight is not possible due to the extremely skewed distribution of ClaimDiff-W, resulting 0 F1.
issues from training data. Following the test environment, issues in development data are exclusive. We train RoBERTa-base and RoBERTa-large (Liu et al., 2019) on ClaimDiff-S and ClaimDiff-W, respectively. We further present RoBERTa (FEVER) and RoBERTa (MNLI), RoBERTa-large initialized on MNLI and FEVER.8
Parameter-efficient finetuning As the number of training examples is not large enough, we further explore the parameter-efficient finetuning methods. Following Hu et al. (2022), we apply low-rank adaptation (LoRA) on pre-trained language models, which only finetune the task-specific low-rank matrices. We follow the same procedure as finetuning baselines for model selection. We compare the effect of LoRA on RoBERTa-large (355M) and DeBERTa-XXL (1.5B) (He et al., 2021).
Zero-shot Baselinse We present the zero-shot performance of large language models, T0 (Wei et al., 2021) and GPT-3 (Brown et al., 2020; Ouyang et al., 2022). Both models get a test pair
8We use the same models as in Section 3.4 for initialization.
following the prompt format and generate the answer directly. We consider the class token with maximum probability as the prediction results. The examples of the zero-shot prompts are presented in Appendix B.

heading: 4.2 Evaluation on ClaimDiff Test
text: Human Evaluation Following Rajpurkar et al. (2016), we evaluate human performance on ClaimDiff based on the human annotation results. As each example has at least three responses, we randomly sample one response as the human prediction. We obtain ground-truth labels using the remainder following the same procedures as in Section 3.2. The resulting human performance is shown in Table 2. Humans are capable of detecting both types of relation, resulting in a significant gap between human and model performance. Even for more challenging ClaimDiff-W, humans can detect more than half of the weakening nuances while maintaining 81.01% precision.
Main Results Figure 2 shows the precisionrecall curve of our baselines. Compared to the vanilla RoBERTa-large, initialization with FEVER worsen the performance while MNLI gives a significant gain in both ClaimDiff-S and ClaimDiff-W. Parameter-efficient finetuning (LoRA) is effective for ClaimDiff-W, even when using a same size model (RoBERTa-large). This is due to the small number of "weakening" examples, which makes finetuning the whole parameters more difficult. When recovering about 80% of "strengthening" examples (ClaimDiff-S), humans retain over 90% of precision, while the best working model retains 80%. The gap is more significant in ClaimDiff-W,
showing about a 30% of difference in precision. Zero-shot baselines, T0 and GPT-3, are worse than finetuning RoBERTa-large in both tasks. We use prompts asking about a single relation ("support" / "weaken") for zero-shot baselines, while the actual relations in ClaimDiff are more complex, which results low coverage of zero-shot models (i.e., GPT-3 predicts only 17% of examples as "strengthening").
The overall evaluation results are presented in Table 2. Note that we report AUROC except for zero-shot baselines, as zero-shot generation does not require any threshold. Align with previous observations, initialization by MNLI boost the performance on both tasks, obtaining the best AUROC in ClaimDiff-S. LoRA becomes more effective when the number of examples is small. LoRA enables training the 1.5B size model (DeBERTa-XXL) with only the hundreds of "weakening" examples, obtaining the best AUROC and F1 in ClaimDiff-W.
Error Analysis To further understand the challenges in ClaimDiff, we investigate the errors of finetuned RoBERTa-large. We randomly sample 25 false negatives (i.e., relationships that the model failed to detect) from each ClaimDiff-S and ClaimDiff-W.9 We manually categorize the errors and analyze them. The examples of each category and the ratio are provided in Appendix F. The errors in "strengthening" have relatively simple patterns (ex., 28% entailment or 20% coherent nuance). However, it is more difficult to capture these relationships in ClaimDiff as claims are collected from real-world news articles, resulting relatively low lexical overlap between two claims. On
9We also analyzed the false positives (FP) while the patterns of FP are too diverse to capture the common patterns.
the other hand, we find that patterns in ClaimDiffW are more challenging; we suspect that this is because there are diverse ways to weaken one’s argument. Finally, a common error (12%) in ClaimDiffS and ClaimDiff-W is due to the need for context information or background knowledge to understand the claims.

heading: 5 Task 2 - Rationale Extraction
text: We perform rationale extraction over "strengthening" and "weakening" pairs, which have positive labels for ClaimDiff-S and ClaimDiff-W. Because of the low appearance of "weakening" pairs, training individual models for each task is challenging. Therefore, unlike relation classification, we train a single model to extract rationales from both "strengthening" and "weakening" pairs. We experiment with extractive and generative models. The evaluation is also conducted on combined sets of ClaimDiff-S and ClaimDiff-W.

heading: 5.1 Models
text: Extractive Model In this work, we present machine reading comprehension (MRC) models as the extractive baselines. As shown in Figure 1, rationales are found as the phrases existing in input claims. Extracting the phrases from a given text is similar to the previous MRC (Rajpurkar et al., 2016; Trischler et al., 2017). Following Devlin et al. (2019), we finetune pre-trained language models with the output layer that predicts the start and end positions of given rationales. We train RoBERTabase and RoBERTa-large for rationale extraction.
Generative Model Generative baselines directly generate the rationales rather than extract it from input claims. Existing works (Narang et al., 2020; Lakhotia et al., 2021) show that generative models obtain a strong performance on rationale benchmark, ERASER (DeYoung et al., 2020). ERASER is designed to evaluate the reasoning ability of NLP models, containing 7 NLP tasks, including BoolQ (Clark et al., 2019), and Movie Reviews (Zaidan and Eisner, 2008). Following Narang et al. (2020), we finetune the T5 models to sequentially generate a list of rationales in a token-by-token fashion. We experiment with T5 and T5 with "strengthening" / "weakening" labels. The exact input formats are explained in Appendix B.

heading: 5.2 Evaluation Metrics
text: Perplexity We report the per-token perplexity of rationales that measures how well the language model predicts the tokens in each rationale. Perplexity is defined as the exponentiated average negative log-likelihood of a sequence. Note that perplexity is only for the generative baselines.
Token F1 (TF1) Following Lakhotia et al. (2021), we compute the Token-level F1 between ground-truth rationales and generated rationales. TF1 measures the number of overlapping tokens between two rationales. Following DeYoung et al. (2020), we use spaCy tokenizer10 to compute the F1 score.
Intersection over Union F1 (IOU F1) IOU F1, as used in DeYoung et al. (2020), computes the F1 on matched predictions. IOU F1 first checks whether predicted rationales match ground-truth rationales by calculating the intersection of union (IOU). IOU is computed as the number of overlapping tokens divided by the union of tokens. If IOU is larger than the threshold, the predicted explanation becomes a matched prediction. In this work, we set the threshold as 0.5.

heading: 5.3 Results
text: In Table 3, we show the results of finetuned RoBERTa and T5 trained on rationale extraction. Following DeYoung et al. (2020), we report extractive measures (TF1 and IOUF1), as the groundtruth rationales are extracted phrases from each claim. We further measure the generative score (perplexity) of output sequences for generative models. Note that we choose perplexity as the metric because rationales are in phrases rather than complete sentences. Although ClaimDiff contains multiple phrases as the ground-truth rationales, MRC models predict a single rationale for each claim pair. Since T5 is capable of generating multi-
10https://spacy.io/
ple phrases at once, even smaller T5-base obtains better performance than RoBERTa-large. T5-large consistently provides better results than T5-base regardless of whether the class labels are given or not. The injection of labels degrades the performance of T5-base, while the T5-large shows a slight improvement. The increasing number of parameters is also beneficial for incorporating additional label information.

heading: 6 Extension: Document-level ClaimDiff
text: Suppose we want to compare the articles with opposing views on contentious issues. For instance, there are two articles about a topic, “Will Gas Prices Come Down Soon or Stay High?”. One forecasts the increase in gas prices, while the other supports the prices have already reached a peak. A single stance label on the relation between the articles (i.e., whether one article supports or opposes the other) might not be enough to understand the complex relations of claims in the articles. ClaimDiff can be applied to provide a granular-level comparison between two articles. Document-level ClaimDiff enables to provide information about which arguments of the first article strengthen or weaken the views of the other. We provide a live demo for document-level extension with RoBERTalarge model.11 As an example, the demo result of the above topic is shown in Appendix G.
In order for the real-world document-level comparison scenario, we evaluate our baseline models on the test-doc dataset, which follows the realworld label distribution. Specifically, the testdoc dataset includes all non-overlapping sentence pairs, which were originally filtered out for the test dataset construction as described in Section 3.2.
11https://www.claimdiff.com
The results are shown in Table 4. Note that we do not provide human performance on the test-doc, as obtaining human annotation over the whole article is costly. Aligning with previous observations, RoBERTa (MNLI) and LoRA with the DeBERTaXXL achieves the best AUROC on documentlevel ClaimDiff-S and ClaimDiff-W, respectively. However, unlike previous results, T0 achieves the second-best F1 on test-doc ClaimDiff-S. One possible reason in that finetuned models have a high proportion of false positives in the full document setting due to distribution shift, whereas the zeroshot model seems to be more robust to it.
Although the fine-grained comparison is helpful for understanding contentious issues, looking over the whole article pair is costly. Future work includes providing summarized statistics from finegrained comparisons. For example, the ratio of strengthening / weakening pairs can represent how much the two articles oppose each other. We can further extend ClaimDiff to compare more than two articles with summarized results, and even compare between different presses.

heading: 7 Conclusion
text: This paper presents ClaimDiff , a new benchmark dataset of 2.9k annotation to compare claims in news articles on contentious issues. Unlike the previous fact verification, ClaimDiff focues on comparing the nuance between claim pairs from trusted sources, whether one claim strengthens or weakens the other. We experiment with pre-trained language models in finetuning, parameter-efficient finetuning, and zero-shot approaches. The results show a significant room for improvement with over 19% absolute gap between human and model performance. We further suggest document-level ClaimDiff as a real-world application and show its
potential by presenting the baseline performance on the test-doc dataset that follows the real-world distribution. We hope this initial study could pave the way for providing an analysis tool for article readers to obtain an unbiased understanding of contentious issues.
Limitations
First, most articles are crawled from the US and UK presses. This means the crawled data is Englishonly and regionally biased, limiting the scope and the diversity of issues. Extending our work to other languages and more regionally-diverse presses will be helpful for reducing such bias in our dataset.
Second, we suspect that there will be a nontrivial annotation bias in our dataset. We are concerned with the fact that all of our in-house annotators share the same cultural background and similar personal interest (given that the annotators volunteered to partcipate in this turking task). Furthermore, given that ClaimDiff-W is aiming to catch the subtle differences in the nuances of these professional news articles, it is very challenging for different annotators to have a common view, especially compared to ClaimDiff-S (which also explains why ClaimDiff-W human performance is much lower than that of ClaimDiff-S).
Third, since ClaimDiff is a sentence-level comparison task, it currently does not give information about the surrounding context of each sentence. This means inter-sentence dependency such as coreference often cannot be resolved. One way to work around this is to give an access to the full articles for each claim pair, but we have refrained from it in this work for simplicity (though we believe it will be interesting to see if the performance can be improved with such access).
Fourth, the size of ClaimDiff is relatively small compared to other fact verficiation datasets. This is mainly because its annotation process is quite challenging and requires a substantial amount of time. Future work includes expanding the size of ClaimDiff when additional budget is available.

heading: Acknowledgements
text: We thank Yongrae Jo, Joel Jang, Hyunji Lee, Hanseok Oh, Soyoung Yoon, Hwisang Jeon and Gangwoo Kim for the useful discussion and feedback on the paper. This work was partly supported by KAIST-NAVER Hypercreative AI Center (80%) and Institute of Information & communications
Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.20210-02068, Artificial Intelligence Innovation Hub, 20%).

heading: A Data Construction Process
text: This section describes the preprocessing step and the data construction process by human annotators. We construct ClaimDiff by 2 steps: (i) filtering non-overlapping claim pairs and (ii) annotating the pairs.
Removing Identifying Information We first preprocess the collected articles to remove identifying information, such as reporters’ contact information. We remove personal information in a two-step procedure. First, we use automatic ways—regular expressions and pre-trained language models—to classify whether a sentence contains personal information. Then, after the automatic removal step, we manually inspect each sentence again and remove the sentence if it contains personal information.
Data Filtering Process we use MTurk12 for filtering large amount of non-overlapping claims. Each worker is asked to solve Human Intelligent Tasks (HITs), which consist of 6 multiple-choice questions. Each HITs is composed of 1 quiz question to manage workers and 5 claim pairs extracted from news articles. The interface for a single question is presented in Figure 3. The reward for a single HIT is $0.18. We collect the responses from 3 different workers for a single example. If more than two workers choose the ’overlap’ or ’large overlap’, the pair are then considered as the ’overlapping’ pair. If more than two workers choose ’small or no overlap’, then the pair is considered as ’non-overlapping’. We process the annotation step for only the ’overlapping’ claim pairs.
Data Annotation Process For the second annotation step, we separately hire 15 in-house expert annotators. We held two training sessions for inhouse experts; one for providing guidelines and the other for solving example tasks. Each expert should pass the final quiz (15 out of 16 questions) after training sessions to start the main tasks. The interface for annotation is shown in Figure 4. Annotators are asked to choose the directional relation of a given pair and select the rationale that supports the relation. In the data construction process, we provide additional context information for a better understanding of the sentence. We offer $0.25 for a single example.13
12https://www.mturk.com 13We provide at least $7.5 per hour even if annotators submit
less than 30 responses.
B Implementation Details
For all experiments, we use PyTorch (Paszke et al., 2019) and Transformers (Wolf et al., 2020). Most of the experiments are conducted on 8 V100 GPUs except T5 for rationale extraction. The validation performance of each model is presented in Table 5.
B.1 RoBERTa (MNLI) / RoBERTa (FEVER)
RoBERTa (MNLI) and RoBERTa (FEVER) are finetuned RoBERTa-large models finetuned on MNLI and FEVER, respectively. We load the trained checkpoints for the MNLI model,14 while me manually finetune RoBERTa-large15 for FEVER. Following Nie et al. (2019), we convert FEVER into an NLI-style task, predicting only the labels among the given query and context. We train RoBERTa-large on NLI-style FEVER during 10 epochs with batch size of 32 and 100 warmup steps. The model is optimized by Adam(Kingma and Ba, 2015) optimizer with learning rate of 5e−5.
B.2 Relation Classification
Fientuned RoBERTa For finetuning experiments, we train models during 10 epochs with a batch size of 32 and 200 warmup steps. We find the best hyperparameters for each model using the
14https://huggingface.co/roberta-large-mnli 15https://huggingface.co/roberta-large
results of 3-fold cross-validation. We choose the best-working checkpoints and thresholds based on the validation F1 score. The search space for each hyperparameter is presented in Table 6. We use Adam optimizer for training. For ClaimDiff-S, we use learning rates of 5e−5 for RoBERTa-base and 5e−6 for others. ClaimDiff-W models are trained with learning rate of 1e−5 for RoBERTa (MNLI), and 5e−6 for other models.
LoRA We train RoBERTa-large and DeBERTaXXL16 with LoRA during 20 epochs with a batch size of 32, the learning rate of 1e−4, and 0.01 weight decay. We use the LoRA implementation released by the authors17. We use a linear scheduler for the learning rate schedule with a 0.1 warmup ratio. We set the rank as 8 with LoRA α of 16 for RoBERTa-base. For DeBERTa-XXL, rank and α are set to be 16 and 32.
T0 As the proposed tasks are binary classification tasks, T0 takes the input prompts and generates answers between (’yes’, ’no’) as prediction labels. We use pre-trained weights of T018 for zero-shot prediction. The input prompts for ClaimDiff-S and ClaimDiff-W are as follow: (i) ClaimDiff-S
Claim A: {claim_a} \n\n Claim B: {claim_b} \n\n Does Claim A support Claim B? yes or no?
(ii) ClaimDiff-W
Claim A: {claim_a} \n\n Claim B: {claim_b} \n\n Does Claim A weaken Claim B? yes or no?
GPT-3 Similar to T0, we consider predictions of GPT-3 as correct if GPT-3 generates ’Yes’ (when the label is 1) or ’No’ (when the label is 0) in the outputs. We use text-davinci-003 of the GPT-3 family in this work. The input prompts for ClaimDiffS and ClaimDiff-W are as follow: (i) ClaimDiff-S
Does A support B?: \n\n A: {claim_a} \n\n B: {claim_b} \n\n
(ii) ClaimDiff-W
Does A weaken B?: \n\n A: {claim_a} \n\n B: {claim_b} \n\n
16https://huggingface.co/microsoft/deberta-v2-xxlarge 17https://github.com/microsoft/LoRA 18https://huggingface.co/bigscience/T0
B.3 Rationale Extraction
MRC Models We train RoBERTa-base and RoBERTa-large with additional answer prediction layers during 3 epochs. Models are trained with Adam optimizer with a learning rate of 5e−5 and batch size of 32. We choose the best checkpoints based on validation TF1.
T5 We finetune T5-base19 and T5-large20 to directly generate a list of rationales. More formally, given claim pairs (c1, c2), we optimize models to obtain the list of rationale phrases. The model takes input as "explain claimdiff claim1: c1 claim2: c2", and is trained to generate the target sequence represented as "explanation: {rationale1} explanation: {rationale2} ···". For class label models, we additionally append "relation: r" to the input text as class information, where r is either one of strengthen or weaken.
We use T5 with a maximum input sequence length of 512 and a batch size of 8. All experiments are conducted on 4 Tesla M60 GPUs using ZeRO (Rajbhandari et al., 2019) stage-3 provided in DeepSpeed (Rasley et al., 2020) to reduce GPU memory usage. We train all models using the Adam optimizer with a constant learning rate of 1e−4. To obtain rationales, we perform beam search decoding using a beam size of 2.

heading: C ClaimDiff Statisics
text: Table 7 shows top-15 presses and tags with their occurrence. Tags in ClaimDiff have long-tailed distribution, indicating ClaimDiff do not concentrate of specific topic.

heading: D ClaimDiff Examples
text: The examples positive and negative examples of ClaimDiff are presented in Table 8. Note that the same pair can have different labels for ClaimDiffS and ClaimDiff-W.

heading: E Dataset Analysis
text: Figure 5 provides more detailed results of Section 3.4. For MNLI model, we gave the former claim as ’premise’ and the later claim as ’hypothesis. For FEVER model predictions, the second claim are given as ’claim’ and the former as ’evidence’.
19https://huggingface.co/t5-base 20https://huggingface.co/t5-large

heading: F Error Analysis
text: We randomly sample 25 false negatives of RoBERTa-large predictions from ClaimDiff-S and ClaimDiff-W, respectively. Table 9 and Table 10 show each error category and its corresponding example in false negative errors.

heading: G Live Demo Result
text: Figure 6 shows the screenshot of the demo and running results.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section name: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section name: Limitations

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1 (Abstract), Section (Introduction)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4, Section 5, Appendix B

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Appendix B
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? [MIT license] Dataset: FEVER-NLI dataset Model checkpoints: RoBERTa-large / RoBERTa-base / RoBERTa-large-mnli / DeBERTa-XXL [Apache 2.0] Model checkpoints: T0, t5-base, t5-large
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? All existing artifacts used in this paper are used in research purpose, which do not violate the intended use of CC BY-NC 4.0 license, MIT license, Apache 2.0 license.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Appendix A

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Limitations
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4, Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4, Appendix B

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix B
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3, Section 5, Appendix B
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3, Appendix A

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix A

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix A

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Limitations

heading: Future Work
text: Future work includes expanding the size of ClaimDiff when additional budget is available