SYNTHVERIFY: Unleashing the Power of LLMs for Zero-Shot Claim Verification via Step-by-Step Synthetic Data Generation
Authors: Rongwen Zhao, Jeffrey Flanigan
Accepted at: acl25

SYNTHVERIFY: Enhancing Zero-Shot Claim Verification through Step-by-Step Synthetic Data Generation

Rongwen Zhao and Jeffrey Flanigan
University of California, Santa Cruz
{rzha017, jmflanig}@ucsc.edu

Abstract

Claim verification is a fundamental task in natural language processing (NLP), involving the assessment of whether available evidence supports or refutes a given claim. While large language models (LLMs) have shown promise in this area, they continue to struggle with domain-specific knowledge. Synthetic data generation has emerged as an effective solution to this challenge. However, existing methods are often either inefficient to scale across multiple domains or overly reliant on external documents. We introduce SYNTHVERIFY, a novel step-by-step prompting-based synthetic data generation framework designed to enhance zero-shot claim verification. Our core insight is that guiding generation with domain-specific claim patterns and structured evidence plans can bridge LLMs’ knowledge gaps in specialized domains without requiring access to external corpora or sacrificing generalizability. Using SYNTHVERIFY, we construct a diverse synthetic dataset for zero-shot verification, enabling instruction fine-tuning tailored to the verification task. Empirical results across multiple specialized domains demonstrate significant accuracy improvements, including a 20.1-point gain on the Llama-3-8B model. Our results highlight the effectiveness of structured synthetic data generation in addressing the limitations of verification systems, particularly in domain-specific tasks.

1 Introduction

Claim verification, the task of determining whether a claim is supported by given evidence, has emerged as a crucial component in combating misinformation and ensuring information integrity (Litou et al., 2017; Hassan et al., 2017; Shu et al., 2017). While LLMs have shown impressive capabilities in various NLP tasks, LLMs face several critical challenges in claim verification (Zhang and Gao, 2023; Guan et al., 2024; Augenstein et al., 2024; Quelle and Bovet, 2024).

A primary concern is the lack of domain-specific knowledge during verification. When evaluating claims in specialized fields such as medicine, law, or scientific research, LLMs often fail to understand the relationships between these domain-specific terminologies (Vladika and Matthes, 2024). This limitation affects both the evidence analysis step, where models must identify relevant domain-specific information, and the logical reasoning step, where they need to apply domain-specific constraints.

Fine-tuning on domain-specific datasets can help LLMs mitigate these issues. While existing datasets like FEVER (Thorne et al., 2018), and VitaminC (Schuster et al., 2021) have driven progress in this field, they often focus on narrow domains (e.g., Wikipedia articles) or specific types of claims. This specialization leads to poor generalization when systems encounter claims from previously unseen domains (Zhu et al., 2022; Nan et al., 2022; Gu et al., 2023). Furthermore, the manual creation of claim verification datasets is both time-consuming and expensive, particularly when aiming to cover multiple knowledge domains with sufficient depth and diversity.

Synthetic data generation has emerged as a promising solution to address this challenge (Pan et al., 2021; Wright et al., 2022; Busotti et al., 2024). Despite its potential, existing approaches face notable limitations. Some methods lack scalability across diverse domains, making them impractical for broad deployment (Pan et al., 2021; Wright et al., 2022), while others rely heavily on external documents to guide data synthesis, introducing dependencies that may not always be available or reliable (Pan et al., 2021; Busotti et al., 2024). As a result, there is a growing need for more efficient and self-contained strategies that can generalize across multiple domains without compromising data quality or diversity.

To address these challenges, we propose SYN-
(1)

Knowledge Domain 1
Domain: Political News
Description: Political news is characterized by rapid developments, partisan language, and selective reporting, making fact verification challenging.
Properties: [Real-time data; Bias and spin; Official vs. independent sources]

Knowledge Domain 2
Domain: Health and Medicine
Description: Health news often includes emerging research, medical jargon, and conflicting expert opinions, complicating accurate interpretation and verification.
Properties: [Scientific uncertainty; Technical terminology; Expert disagreement]

Knowledge Domain n
Domain: Climate and Environment
Description: Climate reporting includes long-term trends, scientific projections, and politically charged interpretations, leading to disputes over data reliability and causality.
Properties: [Scientific modeling; Data interpretation; Policy framing]

Knowledge Domain Clustering

(2)

Knowledge Domain k
Domain: Political News
Description: Political news is characterized by rapid developments, partisan language, and selective reporting, making fact verification challenging.
Properties: [Real-time data; Bias and spin;....]

Evidence Plan
Introduction: [An Introduction that sets the description in context.]
Key Facts: [Key Facts/Elements that reflect the domain properties.]
Conclusion: [A Conclusion summarizing the key insight or challenge.]

Evidence Document
In a recent national debate, a prominent political figure claimed that public spending on infrastructure had increased by 12% over the last fiscal year. However, independent audits and official government records reveal that the actual increase was only 8%, largely due to changes in the definition of what qualifies as “infrastructure” budgeting. This notable gap between the reported figure and the verified data highlights concerns .......

(3)

Evidence Document
In a recent national debate, a prominent political figure claimed that public spending on infrastructure had increased by 12% over the last fiscal year .......

SUPPORTS claim: "Independent audits confirm that ..."
REFUTES claim: "The audits found significant discrepancies ..."
NEI claim: "The evidence suggests the ..."

Figure 1: Overview of our synthetic data generation framework for claim verification. The framework consists of three main stages: (1) Knowledge Domain Generation (\S3.2), (2) Plan-guided Evidence Synthesis (\S3.3), and (3) Aspect-Conditioned Claim Generation (\S3.4).

THVERIFY, a novel step-by-step prompting-based synthetic data generation framework. Our framework carefully controls the generation process to ensure data quality and verifiability. The core insight is that guiding generation with domain-specific claim patterns and structured evidence plans, without requiring access to external corpora or sacrificing generalizability.

First, we develop a structured domain specification stage that captures the nature of domain knowledge and verification requirements, providing a foundation for domain-aware claim verification. We then employ a simple but effective evidence synthesis method that provides clear verification signals, addressing the challenge of generating realistic and verifiable evidence. Building upon this stage, we implement an aspect-based claim generation approach that ensures both diversity in claim types and verifiability of generated claims.

Through extensive evaluation, we demonstrate that our synthetic data generation approach leads to prominent improvements on multiple claim verification tasks. Models trained on our synthetic dataset show improved zero-shot transfer capabilities, better handling of complex claims, and more robust verification across diverse knowledge domains. Our further analysis reveals that the structured nature of our generation process helps models learn generalizable verification strategies rather than superficial patterns specific to particular domains. Overall, our main contributions are:

• We propose a step-by-step prompting-based framework for generating synthetic claim verification data that enables zero-shot transfer across domains (\S3.1).

• We introduce a structured framework for specifying knowledge domains and claim patterns that ensures both coverage and diversity (\S3.2).

• We develop a simple but effective method for evidence synthesis (\S3.3) and claim generation (\S3.4) tailored to complex verification scenarios.

• We conduct extensive evaluations demonstrating that our approach significantly outperforms existing synthetic data baselines across multiple specialized domains (\S5).

2 Background and Problem Setup

Problem Setup Following the prior work (Thorne et al., 2018; Eisenschlos et al., 2021; Schuster et al., 2021; Wadden et al., 2020), given a sentence-level claim \( c \) and a set of sentence-level
evidences \( E = \{e_1, ..., e_{|E|}\} \), the task of claim verification is to determine whether \( E \) supports or refutes \( c \), or if there is insufficient information to make a determination. Here, we assume that each given claim \( c \) is check-worthy and can be fully verified based on the evidence set \( E \), without relying on the external context. Formally, given an LLM \( M \), we define a verification function \( f_M : \mathcal{C} \times \mathcal{E} \to \mathcal{Y} \), where \( \mathcal{C} \) represents the space of possible claims, \( \mathcal{E} \) denotes the space of evidence sets and \( \mathcal{Y} = \{\text{SUPPORTS}, \text{REFUTES}, \text{NOT_ENOUGH_INFO}\} \) is the set of verification labels.

For a given claim-evidence pair \( (c, E) \), the LLM \( M \) performs the verification through three main steps. First, \( M \) performs evidence analysis by processing each evidence \( e_i \in E \) to extract and understand the information relevant to the claim. Second, \( M \) performs logical reasoning to determine the relationship between the extracted evidence and the claim, applying its understanding of the domain knowledge. Finally, \( M \) assigns a verification label \( y \in \mathcal{Y} \) based on the aggregated evidence and reasoning outcome.

Challenges of LLM-based Claim Verification
LLM-based claim verification systems face two significant challenges in real-world applications. First, although LLMs acquire extensive domain knowledge during pre-training, they often demonstrate limited generalization capabilities in unseen domains (Pan et al., 2023b). Human-annotated, domain-specific datasets can substantially enhance performance in such contexts. However, the high costs associated with annotation, both in terms of time and resources, constrain the coverage, diversity, and scalability of these datasets. Second, LLMs are susceptible to reasoning errors and over-reliance on spurious correlations (Tang et al., 2023), which can lead to incorrect veracity assessments. This is particularly problematic when claims require complex reasoning or contextual understanding that spans multiple knowledge types. In this work, we introduce an innovative synthetic data generation approach designed to bolster the capabilities of LLMs in addressing these challenges effectively.

\footnotetext{1We acknowledge that retrieval is a critical component of real-world fact-checking pipelines. However, our evaluation setup assumes that claims are accompanied by evidences, focusing specifically on the claim verification stage rather than evidence retrieval.}

Domain: Climate Science
Description: A dynamic domain involving long-term alteration of temperature and typical weather patterns. Claims are challenging to verify due to reliance on complex climate models and long-term observational data.
Properties: [Temperature data, Weather patterns, Climate models]

Figure 2: An example of the generated knowledge domains. Our Knowledge Domain Generation module (\S3.2) generates a diverse set of knowledge domains, coupled with a clustering-based diversity enforcement.

3 Methodology: Zero-Shot Claim Verification Data Generation

In this section, we describe the details on how we design a structured approach to generate synthetic claim verification data that enables zero-shot transfer across diverse domains. Our goal is to construct a dataset \( \mathcal{D} = \{(D_i, C_i, E_i, y_i)\}_{i=1}^N \) of \( N \) instances, where \( D_i \) represents a specification of knowledge domain, \( C_i \) is a claim within that domain, \( E_i \) is the corresponding evidence, and \( y_i \in \{\text{SUPPORTS}, \text{REFUTES}, \text{NOT_ENOUGH_INFO}\} \) is the verification label.

3.1 Overview
As shown in Figure 1 , our synthetic data generation framework consists of three stages. First, the Knowledge Domain Generation stage defines the scope and constraints of the target domain, organizing knowledge hierarchically to ensure generated content remains focused and relevant. Second, the Plan-guided Evidence Synthesis stage synthesizes informative evidence documents by leveraging the knowledge domains from the previous step. Third, the Aspect-Conditioned Claim Generation stage generate three different types of claims (Supported/Refuted/Not Enough Info) conditioned on each generated evidence document.

3.2 Knowledge Domain Generation
First, to collect a diverse set of domains where evidence and claims are sampled, a set of knowledge domains are randomly sampled from an LLM as structured triplets. Each knowledge domain \( O_i \) is formally characterized by: \( O_i = (D_i, S_i, P_i) \), where \( D_i \) is the domain name, \( S_i \) is a basic description of the domain explaining why claims in this domain are challenging to verify. \( P_i = \{p_1, ..., p_k\} \)
is a set of domain properties. The knowledge domain \( O_i \) will be used to guide the generation of the evidence documents. The additional information \( S_i \) and \( P_i \) can lead to more non-trivial and domain-diverse generations compared to using \( D_i \) alone.

We prompt an LLM to generate a list of knowledge domains that require claim verification in the real-world. The detailed prompt and a sample of generated knowledge domain is shown in Appendix. However, naive LLM prompting often results in redundant or overlapped domains when generating full set of knowledge domains \( O_i \), leading to low coverage (Ding et al., 2023).

To ensure broad coverage across knowledge spaces, we employ an iterative clustering-based generation and filtering process. Let \( f(\cdot) \) be an embedding function that maps each domain \( D_i \) to a \( d \)-dimensional semantic space. For a batch of \( k \) generated knowledge domains, we compute their embeddings: \( \mathbf{E} = \{ \mathbf{e}_i = f(D_i) \}_{i=1}^k \). To identify and remove near-duplicate domains, we apply density-based clustering DBSCAN (Ester et al., 1996):

\[
\text{clusters} = \text{DBSCAN}(\mathbf{E}, \epsilon, \text{min\_samples})
\]

where \( \epsilon \) and min\_samples are empirically determined parameters. This process continues iteratively until we obtain \( n \) diverse domains.

3.3 Plan-guided Evidence Synthesis
In this stage, we generate an informative evidence document by leveraging the knowledge domain from the previous step. However, in a preliminary experiment, prompting LLMs to get the evidence document based on the knowledge domain resulted in more generic content that lack sufficient details to generate meaningful and check-worthy claims in the future stage. To address this issue, inspired by some prior studies in the story generation (Yang et al., 2022, 2023), we generate the evidence document from the knowledge domain in two steps.

The key idea is to leverage a knowledge domain to generate a structured plan, which is subsequently expanded into a coherent evidence document. This two-step process ensures that the final output adheres not only to the factual but also to the stylistic nuances of the target domain but also maintains a consistent academic structure.

Step 1: Evidence Plan Generation. We first prompt the LLM to generate a concise plan that serves as a blueprint for the evidence document. The plan is designed to capture a brief domain-dependent statement and several specific details drawn from the domain properties.

Step 2: Evidence Document Synthesis. Once the plan is generated, the next step is to expand it into a full evidence document. The model is instructed to integrate all plan components into a polished paragraph written in an information-preserving style. This expansion ensures that the evidence document is comprehensive, coherent, and reflective of the domain-specific characteristics.

Figure 3 provides an overview of the plan-guided evidence synthesis process. By grounding the generation process in a structured plan derived from the knowledge domain, our method achieves enhanced control over content structure, ensuring both consistency and domain-specific nuance. This approach has demonstrated improved quality in synthetic evidence, thereby contributing to more robust downstream claim verification across diverse domains.

3.4 Aspect-Conditioned Claim Generation
As shown in Figure 4, we generate three different types of claims (Supported/Refuted/Not Enough Info) conditioned on the generated evidence docu-
Evidence Document
In a recent national debate, a prominent political figure claimed that public spending on infrastructure had increased by 12% over the last fiscal year...

Aspect 1    Aspect 2    Aspect 3
Temporal    Causal    Contextual

Claim
SUPPORTS: "Independent audits confirm that ..."
REFUTES: "The audits found significant discrepancies ..."
NEI: "The evidence suggests the ..."

Figure 4: Our Aspect-Conditioned Claim Generation module (§3.4) takes in the generated evidence. It first generates three aspect features (e.g., temporal, causal and contextual) (middle), then it outputs the three claim (lower) based on the generated key aspects.

ment. Our approach increases the diversity and complexity of the generated claims through a aspect-based generation method that ensures both structural diversity and verifiability while maintaining semantic coherence within each source document. We first generate the supported claim based on the evidence document, then generate the other two claims conditioned on the supported claims.

Supported Claim Generation. We first extract several key aspects from the evidence document. These key aspects are concise descriptors that capture different dimensions or viewpoints present in the evidence (e.g., contextual background, numerical details, source credibility, or timing). By subsequently conditioning the claim generation on each identified aspect, the model produces multiple supported claims that, while all factually correct with respect to the evidence, emphasize different details or angles.

Given an evidence document and the associated knowledge domain, the model is prompted to generate a list of key aspects. Each key aspect is a short description that highlights a particular facet of the evidence. For example, one aspect might focus on the statistical discrepancy mentioned in the text, while another might highlight the reliability of the sources. This step ensures that the extracted aspects are aligned with the unique properties defined in the knowledge domain.

For each key aspect, a secondary prompt is created that conditions the model on both the original evidence and the specified aspect. The prompt instructs the LLM to generate a supported claim that is not only consistent with the evidence but also emphasizes the conditioned aspect. This encourages the model to explore multiple views of the evidence, thereby producing a diverse set of claim formulations.

Refuted Claim Generation. Starting from a supported claim, we generate refuted claims by applying a series of controlled perturbations designed to introduce discrepancies between the claim and the underlying evidence. These perturbations include: (1) Entity Substitution: Replacing critical entities with alternative, contextually plausible candidates. (2) Temporal Modification: Adjusting time-related references while preserving overall historical coherence. (3) Relationship Reversal: Inverting the relational dynamics between entities to alter the claim’s meaning. (4) Attribute Modification: Modifying specific properties or characteristics to create a divergence from the original details.

Not-Enough-Info (NEI) Claim Generation. Given a supported claim, the model is instructed to modify it by removing or replacing the identified key elements with more vague, qualitative, or uncertain language. For example, a claim stating “The study shows a 70% reduction in hospitalization rates” may be transformed into “The study suggests a significant reduction in hospitalization rates,” where the exact figure is omitted.

3.5 Human Evaluation

In order to investigate the accuracy and quality of the labels on our generated (evidence, claim) pairs. We randomly sample 50 examples from the generated dataset. Each example was annotated with a label by three NLP researchers. The examples are unedited outputs from the models, with no revisions made during annotation. The average Cohen’s \( \kappa \) score between annotators is 65.83%, indicating substantial agreement. Furthermore, a majority label (agreed upon by 2 out of 3 annotators) was obtained for 94% of examples, while all three annotators reached unanimous agreement on 72% of examples. The model achieved a high accuracy of 79.6% against the majority label and 88.5% against the unanimous label. The Cohen’s \( \kappa \) scores between the model’s labels and both the majority and unanimous labels are 69.73% and 82.46%, respectively.
Instruction: You are given a claim and evidence. Your task is to verify the claim based solely on the evidence provided. Analyze the claim and evaluate the evidence. Based on your evaluation, return one of the following labels: "supports", "refutes" or "not enough info".

Evidence: {EVIDENCE}
Claim: {CLAIM}
Label: {LABEL}

Figure 5: The instruction template that we use for compiling the final instruction tuning dataset to instruct tuning LLMs for the claim verification.

3.6 The Final Instruction Tuning Dataset

Using our proposed framework SYNTHVERIFY, we generate a domain-diverse synthetic claim verification dataset for training zero-shot verification models. We use the generated dataset to fine-tune the various LLMs (e.g., Llama-3.1-8B). To achieve this, we combine the predefined instruction template in Figure 5 and instance input into a single prompt and train the LLMs in a standard supervised learning setup to generate the corresponding instance output.

4 Experimental Setup

Evaluation Benchmarks. In this work, following the prior work (Pan et al., 2023b), we use a wide range of existing claim verification benchmarks from various domains: (1) Wikipedia domain: We use FEVER (Thorne et al., 2018), VitaminC (Schuster et al., 2021) and FoolMeTwice (Eisenschlos et al., 2021). These datasets were created based on Wikipedia documents; (2) Climate domain: We use Climate-FEVER (Diggelmann et al., 2020), which consists of real-world claims related to climate change; (3) Science domain: We use SciFact (Wadden et al., 2020); (4) Health domain: We use PubHealth (Kotonya and Toni, 2020); (5) Other domains: we also use Covid-Fact (Saakyan et al., 2021) from the forum domain and FAVIQ (Park et al., 2022) from the question domain in our evaluations. The statistics of these benchmarks are shown in Table 5. Please refer to Appendix A for more details.

Evaluation Metric. Following the prior work (Laban et al., 2022), we use the Balanced Accuracy (BAcc), the average of recall obtained on each label, as our evaluation metric. Here, \( \text{BAcc} = \frac{1}{|\mathcal{Y}|} \sum_{y \in \mathcal{Y}} \frac{\mathrm{TP}_y}{\mathrm{TP}_y + \mathrm{FN}_y} \), where \( \mathcal{Y} \) is the set of all class labels (e.g., supports, refutes, not_enough_info), \( \mathrm{TP}_y \) is the number of true positives for class \( y \), and \( \mathrm{FN}_y \) is the number of false negatives for class \( y \).

Baselines. (1) Zero-Shot: By default, we compare the model fine-tuned by our synthetic dataset to the zero-shot setting. We use the same prompt as shown in Figure 5. (2) Zero-Shot Chain-of-Thought (Kojima et al., 2022): We also compare to the proposed zero-shot CoT prompting method. We append the prompt “Let’s think step by step” to the instruction prompt in Figure 5. (3) We also compare our method to QACG (Pan et al., 2021), which generates synthetic claims conditioned on the Wikipedia data. We take their released dataset and fine-tune Llama-3 and InternLM2.5 models.

Implementation Details. We compare against various state-of-the-art instruction-tuned LLMs, including: Qwen 2.5 (Yang et al., 2024), Gemma 2 (Team et al., 2024), InternLM (Cai et al., 2024), Mistral (Jiang et al., 2023), and Llama-3 family (Dubey et al., 2024).

For the synthetic data generation, we use GPT-4o-mini with temperature 0.9 for all prompting generation. For domain similarity computation, we employ the all-MiniLM-L6-v2 sentence transformer, which provides 384-dimensional embeddings. The clustering parameters (\( \epsilon = 0.3, \min\_samples = 2 \)). The all prompts, dataset statistics and fine-tuning hyperparameters are available in Appendix.

5 Main Results

Table 1 presents a comprehensive evaluation of various language models on eight claim verification benchmarks. Several notable findings emerge from our experimental results.

First, our synthetic data framework demonstrates consistent and substantial improvements across all variants of the model. Most notably, InternLM2.5-7B-Chat with our method achieves the best overall performance with an average balanced accuracy of 62.8, surpassing its base version by a large margin of 13.2 absolute points (from 49.6 to 62.8). This improvement is particularly pronounced in other datasets like FEVER (+19.7 points) and VitaminC (+25.4 points).

Second, we observe that larger model size does not necessarily translate to better claim verification performance. For example, Gemma-2-2B-it, despite its relatively small size, achieves a com-
<table>
  <tr>
    <th>Model</th>
    <th>FEVER</th>
    <th>FM2</th>
    <th>VitaminC</th>
    <th>C-FEVER</th>
    <th>PubHealth</th>
    <th>SciFact</th>
    <th>CovidFact</th>
    <th>FAVIQ</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B-Instruct</td>
    <td>35.7</td>
    <td>50.0</td>
    <td>37.0</td>
    <td>32.5</td>
    <td>31.3</td>
    <td>38.1</td>
    <td>52.0</td>
    <td>50.0</td>
    <td>40.8</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B-Instruct</td>
    <td>34.0</td>
    <td>50.0</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>34.8</td>
    <td>50.0</td>
    <td>49.0</td>
    <td>39.7</td>
  </tr>
  <tr>
    <td>gemma-2-2b-it</td>
    <td>55.7</td>
    <td>78.5</td>
    <td>47.7</td>
    <td>49.6</td>
    <td>38.7</td>
    <td>52.9</td>
    <td>66.5</td>
    <td>57.5</td>
    <td>55.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B-Instruct</td>
    <td>49.7</td>
    <td>61.0</td>
    <td>38.0</td>
    <td>39.0</td>
    <td>32.7</td>
    <td>51.9</td>
    <td>60.5</td>
    <td>53.0</td>
    <td>48.2</td>
  </tr>
  <tr>
    <td>gemma-2-9b-it</td>
    <td>37.7</td>
    <td>51.0</td>
    <td>33.0</td>
    <td>38.2</td>
    <td>34.7</td>
    <td>34.8</td>
    <td>63.0</td>
    <td>55.5</td>
    <td>43.5</td>
  </tr>
  <tr>
    <td>Llama-2-13b-hf</td>
    <td>33.3</td>
    <td>50.0</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>33.7</td>
    <td>33.3</td>
    <td>50.0</td>
    <td>49.5</td>
    <td>39.6</td>
  </tr>
  <tr>
    <td>InternLM2-Chat-20B</td>
    <td>59.3</td>
    <td>69.5</td>
    <td>47.3</td>
    <td>43.1</td>
    <td>29.3</td>
    <td>54.3</td>
    <td>64.0</td>
    <td>54.0</td>
    <td>52.6</td>
  </tr>
  <tr>
    <td>InternLM2.5-20B-Chat</td>
    <td>46.0</td>
    <td>56.0</td>
    <td>47.3</td>
    <td>39.0</td>
    <td>36.3</td>
    <td>36.7</td>
    <td>57.0</td>
    <td>52.0</td>
    <td>46.3</td>
  </tr>
  <tr>
    <td>Llama-3.2-1B-Instruct</td>
    <td>43.7</td>
    <td>53.0</td>
    <td>31.3</td>
    <td>35.8</td>
    <td>35.7</td>
    <td>37.1</td>
    <td>52.5</td>
    <td>56.0</td>
    <td>43.1</td>
  </tr>
  <tr>
    <td>Llama-3.2-1B-Instruct (Ours)</td>
    <td>56.7</td>
    <td>64.0</td>
    <td>58.0</td>
    <td>53.7</td>
    <td>44.0</td>
    <td>48.1</td>
    <td>61.5</td>
    <td>60.0</td>
    <td><b>55.8</b></td>
  </tr>
  <tr>
    <td>Llama-3.2-3B-Instruct</td>
    <td>43.7</td>
    <td>63.0</td>
    <td>36.7</td>
    <td>48.8</td>
    <td>38.7</td>
    <td>54.8</td>
    <td>64.5</td>
    <td>57.0</td>
    <td>50.9</td>
  </tr>
  <tr>
    <td>Llama-3.2-3B-Instruct (Ours)</td>
    <td>61.0</td>
    <td>75.5</td>
    <td>56.0</td>
    <td>49.6</td>
    <td>51.2</td>
    <td>53.3</td>
    <td>64.5</td>
    <td>60.5</td>
    <td><b>59.0</b></td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct-v0.3</td>
    <td>46.0</td>
    <td>54.0</td>
    <td>36.0</td>
    <td>34.1</td>
    <td>36.3</td>
    <td>36.2</td>
    <td>50.5</td>
    <td>48.0</td>
    <td>42.6</td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct-v0.3 (Ours)</td>
    <td>59.0</td>
    <td>74.5</td>
    <td>54.0</td>
    <td>53.7</td>
    <td>52.3</td>
    <td>67.0</td>
    <td>55.5</td>
    <td><b>58.6</b></td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>37.0</td>
    <td>60.5</td>
    <td>32.7</td>
    <td>38.2</td>
    <td>36.3</td>
    <td>38.1</td>
    <td>54.5</td>
    <td>56.0</td>
    <td>44.2</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (Ours)</td>
    <td>60.3</td>
    <td>76.0</td>
    <td>56.7</td>
    <td>50.4</td>
    <td>45.3</td>
    <td>59.5</td>
    <td>68.0</td>
    <td>59.5</td>
    <td><b>59.5</b></td>
  </tr>
  <tr>
    <td>Llama-3-8B-Instruct</td>
    <td>33.3</td>
    <td>50.0</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>50.5</td>
    <td>51.0</td>
    <td>39.8</td>
  </tr>
  <tr>
    <td>+ Zero-Shot CoT</td>
    <td>43.7</td>
    <td>53.0</td>
    <td>47.3</td>
    <td>39.0</td>
    <td>36.3</td>
    <td>36.7</td>
    <td>52.5</td>
    <td>53.0</td>
    <td>42.7</td>
  </tr>
  <tr>
    <td>+ QACG (<a href="https://arxiv.org/abs/2110.09772">Pan et al., 2021</a>)</td>
    <td>37.7</td>
    <td>53.0</td>
    <td>38.0</td>
    <td>38.2</td>
    <td>35.7</td>
    <td>34.8</td>
    <td>52.5</td>
    <td>52.0</td>
    <td>42.7</td>
  </tr>
  <tr>
    <td>Llama-3-8B-Instruct (Ours)</td>
    <td><b>59.3</b></td>
    <td><b>80.5</b></td>
    <td><b>57.0</b></td>
    <td><b>49.6</b></td>
    <td><b>49.3</b></td>
    <td><b>55.7</b></td>
    <td><b>65.5</b></td>
    <td><b>62.5</b></td>
    <td><b>59.9</b></td>
  </tr>
  <tr>
    <td>InternLM2.5-7B-Chat</td>
    <td>44.3</td>
    <td>76.5</td>
    <td>33.3</td>
    <td>35.0</td>
    <td>34.0</td>
    <td>46.7</td>
    <td>67.0</td>
    <td>60.0</td>
    <td>49.6</td>
  </tr>
  <tr>
    <td>+ Zero-Shot CoT</td>
    <td>46.0</td>
    <td>76.5</td>
    <td>36.7</td>
    <td>35.8</td>
    <td>36.3</td>
    <td>48.1</td>
    <td>67.0</td>
    <td>60.5</td>
    <td>50.9</td>
  </tr>
  <tr>
    <td>+ QACG (<a href="https://arxiv.org/abs/2110.09772">Pan et al., 2021</a>)</td>
    <td>46.0</td>
    <td>78.5</td>
    <td>37.7</td>
    <td>42.3</td>
    <td>32.7</td>
    <td>51.0</td>
    <td>67.0</td>
    <td>60.0</td>
    <td>51.9</td>
  </tr>
  <tr>
    <td><b>InternLM2.5-7B-Chat (Ours)</b></td>
    <td><b>64.0</b></td>
    <td><b>83.5</b></td>
    <td><b>58.7</b></td>
    <td><b>53.7</b></td>
    <td><b>49.3</b></td>
    <td><b>57.1</b></td>
    <td><b>71.0</b></td>
    <td><b>65.0</b></td>
    <td><b>62.8</b></td>
  </tr>
</table>

Table 1: Zero-shot performance across different datasets, using Balanced Accuracy (BAcc). For Llama-3 and InternLM2.5 models, we bold the best model for each dataset. Otherwise, we bold the best average score.

petitive average accuracy of 55.9, outperforming several larger models such as InternLM2-Chat-20B (52.6) and InternLM2.5-20B-Chat (46.3). This suggests that verification performance is more closely tied to a model’s capabilities than its size.

Third, our method shows remarkable consistency in improving performance across different model architectures and scales. The enhancement is particularly evident in the Llama family: Llama-3.2-1B-Instruct improves from 43.1 to 55.8 (+12.7 points), Llama-3.2-3B-Instruct from 50.9 to 59.0 (+8.1 points), and Llama-3.1-8B-Instruct from 44.2 to 59.5 (+15.3 points). This consistent improvement pattern suggests the robustness and generalizability of our approach.

We also observe that our method is better than zero-shot chain-of-thought (CoT) prompting. Zero-shot CoT can usually get some improvements, but fine-tuning models on domain-relevant data would be more beneficial. Furthermore, our method is also better than QACG (<a href="https://arxiv.org/abs/2110.09772">Pan et al., 2021</a>). The major reason is that our method benefits from LLMs. LLMs store a lot of domain knowledge during the pre-training stage.

<table>
  <tr>
    <th>Synthesis Framework</th>
    <th>VitaminC</th>
    <th>SciFact</th>
    <th>FAVIQ</th>
  </tr>
  <tr>
    <td>No Training</td>
    <td>33.3</td>
    <td>46.7</td>
    <td>60.0</td>
  </tr>
  <tr>
    <td>Direct Prompting</td>
    <td>35.0</td>
    <td>34.8</td>
    <td>58.5</td>
  </tr>
  <tr>
    <td>R1</td>
    <td>57.0</td>
    <td><b>56.2</b></td>
    <td>62.5</td>
  </tr>
  <tr>
    <td>R2</td>
    <td>54.0</td>
    <td>51.0</td>
    <td>60.5</td>
  </tr>
  <tr>
    <td>R3</td>
    <td>56.0</td>
    <td>54.3</td>
    <td>62.5</td>
  </tr>
  <tr>
    <td>Full Framework</td>
    <td><b>57.3</b></td>
    <td><b>56.2</b></td>
    <td><b>64.0</b></td>
  </tr>
</table>

Table 2: The results of our ablation studies. R1 means removing knowledge domain specification. R2 represents that removing the evidence plan generation. R3 is removing the key aspect generation.

6 Further Analysis

In this section, we conduct more analysis studies, comparing our framework to a direct prompting method, analyzing the effect of training data size and the impact of domain knowledge.

6.1 Ablation Study

To understand the contributions of each component of our pipeline, we systematically evaluate our framework’s three main components. We use the same domains as our framework to generate 20k samples for each setting. We use InternLM2.5-7B-Chat as our base model and evaluate on VitaminC (<a href="https://arxiv.org/abs/2110.09772">Schuster et al., 2021</a>), SciFact (<a href="https://arxiv.org/abs/2110.09772">Wadden et al.,
2020), and FAVIQ (Park et al., 2022) datasets. The full prompt is shown in Appendix.

Direct Prompting  The most straightforward way is the direct prompting without any intermediate step. This method directly prompts the LLM to generate the claim-evidence pairs when given a domain label. Table 24 shows the prompt we use to perform direct prompting.

Removing knowledge domain Specification (R1)  When removed, we replace the structured knowledge domain with simple domain labels (e.g., climate science), eliminating the domain-specific constraints. This tests the impact of structured knowledge domain representation in our framework. The prompt is shown in Table 21.

Removing Evidence Plan Generation (R2)  We replace our evidence synthesis module with basic prompting generation, removing the evidence plan generation. This tests the value of the evidence plan in guiding the evidence document generation. The prompt is shown in Table 22.

Removing Key aspect Generation (R3)  Without this component, we use basic prompt-based generation without key aspects to generate the supported claim. This evaluates the benefit of our diversity-oriented claim generation approach over simple prompting. The prompt is shown in Table 23.

Table 2 shows the results of our ablation studies. The results demonstrate the effectiveness of our proposed framework. We can observe that performance decreases slightly when removing a specific component. The results also show consistent improvements over both the no-training baseline and direct prompting approaches. Specifically, our full framework achieves the highest performance across all datasets. The improvement is particularly pronounced for VitaminC, where our framework outperforms the no-training baseline by 24 points. Interestingly, direct prompting shows mixed results compared to no training, performing slightly better on VitaminC (+1.7 points) but worse on SciFact (-11.9 points). These findings strongly suggest that our designed framework contributes significantly to the framework’s overall effectiveness.

6.2 Impact of the Budget Control

Since our data synthesis framework uses more API calls compared to the direct prompting method.

<table>
  <tr>
    <th>Synthesis Framework</th>
    <th>VitaminC</th>
    <th>SciFact</th>
    <th>FAVIQ</th>
  </tr>
  <tr>
    <td>No Training</td>
    <td>33.3</td>
    <td>46.7</td>
    <td>60.0</td>
  </tr>
  <tr>
    <td>Direct Prompting (30k)</td>
    <td>38.0</td>
    <td>48.1</td>
    <td>60.0</td>
  </tr>
  <tr>
    <td>Full Framework (5k)</td>
    <td><b>56.0</b></td>
    <td><b>55.7</b></td>
    <td><b>62.5</b></td>
  </tr>
</table>

Table 3: Ablation study by considering the budget constraints.

![Bar chart showing the average performance of Llama-3.2-1B-Instruct and Llama-3-8B-Instruct models trained on different sizes of synthetic instruction tuning data.](page_1012_1012_388_246.png)

Figure 6: The results of how the performance of Llama-3.2-1B-Instruct and Llama-3-8B-Instruct models varies when trained on different sizes of synthetic instruction tuning data.

In order to investigate the benefit of the structure-aware data synthesis, we also compare our framework to the direct prompting method incorporating budget constraints. Since it is hard to control the synthesis budget directly, we conduct an additional experiment under the rough estimation. The table Table 3 compares a model fine-tuned with 30k samples from direct prompting against a model fine-tuned with only 5k samples generated using our framework. As we can see, our framework is still significantly better than the direct prompting even under the similar budget.

6.3 Effect of the Training Dataset Size

Here we study the effect of the size of the training dataset. In particular, we study how the performance of the Llama-3.2-1B-Instruct and Llama-3-8B-Instruct models varies when trained on different sizes of synthetic instruction tuning data. Figure 6 shows that training on more steps typically improves performance. We find that fine-tuned LLMs on all datasets reach the peak performance generally after 30,000 steps. It means training on more of the synthetic data may not be necessary.
<table>
  <tr>
    <th>Model</th>
    <th>C-FEVER</th>
    <th>SciFact</th>
    <th>CovidFact</th>
  </tr>
  <tr>
    <td>Llama-3.2-1B-Instruct</td>
    <td>35.8</td>
    <td>37.1</td>
    <td>52.5</td>
  </tr>
  <tr>
    <td>w/ FEVER</td>
    <td>38.2</td>
    <td>43.8</td>
    <td>54.5</td>
  </tr>
  <tr>
    <td>w/ (SynthVerify & FEVER)</td>
    <td>42.3</td>
    <td>47.7</td>
    <td>57.0</td>
  </tr>
  <tr>
    <td>Llama-3-8B-Instruct</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>50.5</td>
  </tr>
  <tr>
    <td>w/ FEVER</td>
    <td>40.3</td>
    <td>43.8</td>
    <td>61.0</td>
  </tr>
  <tr>
    <td>w/ (SynthVerify & FEVER)</td>
    <td>50.4</td>
    <td>58.6</td>
    <td>67.0</td>
  </tr>
</table>

Table 4: Performance comparison across C-FEVER, SciFact, and CovidFact datasets.

6.4 Impact of Domain Knowledge

Fine-tuning models on unseen domains can improve the verification performance (Pan et al., 2023a). We hypothesize that the synthetic data generated by SYNTHVERIFY can benefit models further from the general verification training. To investigate the effectiveness of SYNTHVERIFY in domain adaptation setting, we conduct experiments comparing the performance of two Llama models (3.2-1B and 3-8B) across three distinct verification datasets. We examine three training configurations for each model: the base instruction-tuned model, fine-tuning with FEVER data only, and fine-tuning with both FEVER and our synthetic data generated by SYNTHVERIFY.

The results in Table 4 demonstrate that SynthVerify consistently enhances verification performance across all domains. The improvement is particularly pronounced in the Llama-3-8B model, where the combination of SYNTHVERIFY and FEVER training leads to substantial gains, improving from 33.3 to 58.6 on SciFact. Notably, even the smaller 3.2-1B model shows consistent improvements with SynthVerify, suggesting that our approach effectively improves domain adaptation across model sizes.

7 Related Work

7.1 LLM-based Claim Verification

Numerous datasets have been compiled for fact verification across various domains, such as politics (Vlachos and Riedel, 2014), encyclopedias (Thorne et al., 2018, 2021; Eisenschlos et al., 2021), news (Pérez-Rosas et al., 2018), climate (Diggelmann et al., 2020), science (Wadden et al., 2020), and healthcare (Kotonya and Toni, 2020). Honovich et al. (2022) consolidated several datasets to evaluate the ability to measure input-output consistency. The statements in these datasets are typically single sentences, generated either by crawling specific websites (Vlachos and Riedel, 2014), manually altering factual sentences (Thorne et al., 2018), or reformulating QA pairs (Thorne et al., 2021). In this work, we study the problem of LLM-based zero-shot claim verification across multiple domains.

7.2 Synthetic Data Generation for Fact Verification

Recent advancements in synthetic data generation have significantly enhanced fact verification systems by reducing reliance on costly human annotations. Pan et al. (2021) introduced QACG, a framework that generates claims from Wikipedia by transforming question–answer pairs into supported, refuted, or unverifiable statements. Building upon this, Wright et al. (2022) proposed CLAIMGEN-BART and KBIN to automatically produce atomic scientific claims and their negations in the biomedical domain, enabling zero-shot fact checking with performance reaching up to 90% of fully supervised models. Extending to multimodal data, Busotti et al. (2024) developed UNOWN, a system that synthesizes training examples from both textual and tabular sources. Collectively, these works underscore the effectiveness of synthetic data in scaling fact verification across diverse domains and data modalities. In contrast to these works, we propose a step-by-step prompting-based generation framework that does not rely on external input documents and remains scalable across diverse domains.

8 Conclusion

We present SYNTHVERIFY, a novel synthetic data generation framework for synthesizing domain-diverse claim verification datasets. Our approach addresses the critical challenge of limited domain coverage in existing datasets through automated generation of claim-evidence pairs across novel domains. Experimental results demonstrate that models trained on SYNTHVERIFY-generated data significantly outperform baselines on multiple benchmarks. This success highlights that the step-by-step nature of our generation process helps models learn generalizable verification strategies.

Limitations

LLM Prompting Since our method is prompting an LLM to generate synthetic data. Some less powerfull LLMs may not be able to do this task.
We will explore how to self-improve using small LLMs.

Semantic Redundancy in Generated Data
While SYNTHVERIFY successfully generates valuable training datasets for zero-shot claim verification, it does not ensure that each synthesized claim is semantically unique. For zero-shot applications, this limitation is less problematic, as such models are designed to adapt to any provided claims and evidence to determine veracity. Addressing this redundancy would improve the applicability of SYNTHVERIFY for broader training scenarios.

Label Quality
Given the fully automated nature of our data synthesis process, some noise in the veracity labels is unavoidable. This noise likely underestimates the true impact of training data domain diversity on zero-shot claim verification, as models trained on SYNTHVERIFY are inadvertently exposed to, and learn to predict, this noise. Ideally, a dataset with the same level of domain diversity as SYNTHVERIFY, but with gold-standard veracity labels, would serve as the basis for our experiments. The lack of such a dataset underscores the motivation for our work.

Acknowledgments
We are thankful for the computing resources provided by the Pacific Research Platform’s Nautilus cluster, supported in part by National Science Foundation (NSF) awards CNS-1730158, ACI-1540112, ACI-1541349, OAC-1826967, OAC-2111267, CNS-2100237, CNS-2120019, the University of California Office of the President, and the University of California San Diego’s California Institute for Telecommunications and Information Technology/Qualcomm Institute. Thanks to CENIC for the 100Gbps networks. We also thank the reviewers for their valuable feedback.

References
Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, et al. 2024. Factuality challenges in the era of large language models and opportunities for fact-checking. Nature Machine Intelligence, 6(8):852–863.

Jean-Flavien Bussotti, Luca Ragazzi, Giacomo Frisoni, Gianluca Moro, and Paolo Papotti. 2024. Unknown claims: Generation of fact-checking training examples from unstructured and structured data. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12105–12122.

Zheng Cai, Maosong Cao, Haojiqiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Intermlm2 technical report. ArXiv preprint, abs/2403.17297.

Tri Dao. 2024. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Thomas Diggleman, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. Climate-fever: A dataset for verification of real-world climate claims. ArXiv preprint, abs/2012.00614.

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3029–3051, Singapore. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. ArXiv preprint, abs/2407.21783.

Julian Einschlos, Bhuwan Dhingra, Jannis Bulian, Benjamin Börschinger, and Jordan Boyd-Graber. 2021. Fool me twice: Entailment from Wikipedia gamification. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 352–365. Online. Association for Computational Linguistics.

Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226–231.

Jiawei Gu, Xuan Qian, Qian Zhang, Hongliang Zhang, and Fang Wu. 2023. Unsupervised domain adaptation for covid-19 classification based on balanced slice wasserstein distance. Computers in Biology and Medicine, page 107207.

Jian Guan, Jesse Dodge, David Wadden, Minlie Huang, and Hao Peng. 2024. Language models hallucinate, but may excel at fact verification. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1090–1111, Mexico City, Mexico. Association for Computational Linguistics.
Naeemul Hassan, Fatma Arslan, Chengkai Li, and Mark Tresayne. 2017. Toward automated fact-checking: Detecting check-worthy factual claims by claim-buster. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 - 17, 2017, pages 1803–1812. ACM.

Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering, pages 161–175, Dublin, Ireland. Association for Computational Linguistics.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. ArXiv preprint, abs/2310.06825.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu-taka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Neema Kotonya and Francesca Toni. 2020. Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7740–7754, Online. Association for Computational Linguistics.

Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. SummaC: Re-visiting NLL-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177.

Iouliana Litou, Vana Kalogeraki, Ioannis Katakis, and Dimitrios Gunopulos. 2017. Efficient and timely misinformation blocking under varying cost constraints. Online Social Networks and Media, 2:19–31.

Qiong Nan, Danding Wang, Yongchun Zhu, Qiang Sheng, Yuhui Shi, Juan Cao, and Jintao Li. 2022. Improving fake news detection of influential domain via domain- and instance-level transfer. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2834–2848, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.

Liangming Pan, Wenhui Chen, Wenhan Xiong, Min-Yen Kan, and William Yang Wang. 2021. Zero-shot fact verification by claim generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 476–483, Online. Association for Computational Linguistics.

Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min-Yen Kan, and Preslav Nakov. 2023a. Fact-checking complex claims with program-guided reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6981–7004, Toronto, Canada. Association for Computational Linguistics.

Liangming Pan, Yunxiang Zhang, and Min-Yen Kan. 2023b. Investigating zero- and few-shot generalization in fact verification. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 511–524, Nusa Dua, Bali. Association for Computational Linguistics.

Jungsoo Park, Sewon Min, Jaewoo Kang, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. FaVIQ: FAct verification from information-seeking questions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5154–5166, Dublin, Ireland. Association for Computational Linguistics.

Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2018. Automatic detection of fake news. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3391–3401, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Dorian Quelle and Alexandre Bovet. 2024. The perils and promises of fact-checking with large language models. Frontiers in Artificial Intelligence, 7:1341697.

Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. 2021. COVID-fact: Fact extraction and verification of real-world claims on COVID-19 pandemic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2116–2129, Online. Association for Computational Linguistics.

Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624–643, Online. Association for Computational Linguistics.
Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake news detection on social media: A data mining perspective. ACM SIGKDD explorations newsletter, 19(1):22–36.

Ruixiang Tang, Dehan Kong, Longtao Huang, and Hui Xue. 2023. Large language models can be lazy learners: Analyze shortcuts in in-context learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4645–4657, Toronto, Canada. Association for Computational Linguistics.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Husseidot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. ArXiv preprint, abs/2408.00118.

James Thorne, Max Glockner, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2021. Evidence-based verification for real world information needs. ArXiv preprint, abs/2104.00640.

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.

Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18–22, Baltimore, MD, USA. Association for Computational Linguistics.

Juraj Vladika and Florian Matthes. 2024. Comparing knowledge sources for open-domain scientific claim verification. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2103–2114, St. Julian’s, Malta. Association for Computational Linguistics.

David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association for Computational Linguistics.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierre Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

Dustin Wright, David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Isabelle Augenstein, and Lucy Lu Wang. 2022. Generating scientific claims for zero-shot scientific fact checking. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2448–2460, Dublin, Ireland. Association for Computational Linguistics.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. ArXiv preprint, abs/2407.10671.

Kevin Yang, Dan Klein, Nanyun Peng, and Yuandong Tian. 2023. DOC: Improving long story coherence with detailed outline control. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3378–3465, Toronto, Canada. Association for Computational Linguistics.

Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4393–4479, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Xuan Zhang and Wei Gao. 2023. Towards LLM-based fact verification on news claims with a hierarchical step-by-step prompting method. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 996–1011, Nusa Dua, Bali. Association for Computational Linguistics.

Yongchun Zhu, Qiang Sheng, Juan Cao, Qiong Nan, Kai Shu, Minghui Wu, Jindong Wang, and Fuzhen Zhuang. 2022. Memory-guided multi-view multi-domain fake news detection. IEEE Transactions on Knowledge and Data Engineering, 35(7):7178–7191.

A Appendix: Evaluation Benchmarks and Fine-Tuning Details

A.1 Dataset Statistics of the Evaluation Benchmarks

In this study, we used the following pre-processed datasets in Pan et al. (2023b). The statistics of these dataset are shown in Table 5. Claims are typically sentence-level statements, which can be either real-world natural claims sourced from websites, textbooks, and forums, or artificially gener-
<table>
  <tr>
    <th>Dataset</th>
    <th>Domain</th>
    <th>Claim</th>
    <th>Evidence</th>
    <th>Label</th>
    <th colspan="2"># Claims</th>
    <th colspan="2">Avg. # tokens</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>Train</th>
    <th>Test</th>
    <th>Claim</th>
    <th>Evid.</th>
  </tr>
  <tr>
    <td rowspan="3">I</td>
    <td>FEVER</td>
    <td>Wikipedia</td>
    <td>artificial</td>
    <td>sent-level</td>
    <td>S (52%), R (22%), N (26%)</td>
    <td>145,327</td>
    <td>19,972</td>
    <td>9.4</td>
    <td>35.9</td>
  </tr>
  <tr>
    <td>VitaminC</td>
    <td>Wikipedia</td>
    <td>artificial</td>
    <td>sent-level</td>
    <td>S (50%), R (35%), N (15%)</td>
    <td>370,653</td>
    <td>63,054</td>
    <td>12.6</td>
    <td>29.5</td>
  </tr>
  <tr>
    <td>FoolMeTwice</td>
    <td>Wikipedia</td>
    <td>artificial</td>
    <td>sent-level</td>
    <td>S (49%), R (51%)</td>
    <td>10,419</td>
    <td>1,169</td>
    <td>15.3</td>
    <td>37.0</td>
  </tr>
  <tr>
    <td rowspan="4">II</td>
    <td>ClimateFEVER</td>
    <td>Climate</td>
    <td>natural</td>
    <td>sent-level</td>
    <td>S (25%), R (11%), N (64%)</td>
    <td>6,140</td>
    <td>1,535</td>
    <td>22.8</td>
    <td>33.8</td>
  </tr>
  <tr>
    <td>Sci-Fact</td>
    <td>Science</td>
    <td>natural</td>
    <td>sent-level</td>
    <td>S (43%), R (22%), N (35%)</td>
    <td>868</td>
    <td>321</td>
    <td>13.8</td>
    <td>61.9</td>
  </tr>
  <tr>
    <td>PubHealth</td>
    <td>Health</td>
    <td>natural</td>
    <td>sent-level</td>
    <td>S (60%), R (36%), N (4%)</td>
    <td>8,370</td>
    <td>1,050</td>
    <td>15.7</td>
    <td>137.6</td>
  </tr>
  <tr>
    <td>COVID-Fact</td>
    <td>Forum</td>
    <td>natural</td>
    <td>sent-level</td>
    <td>S (32%), R (68%)</td>
    <td>3,268</td>
    <td>818</td>
    <td>12.4</td>
    <td>82.5</td>
  </tr>
  <tr>
    <td>FAVIQ</td>
    <td>Question</td>
    <td>natural</td>
    <td>doc-level</td>
    <td>S (50%), R (50%)</td>
    <td>17,008</td>
    <td>4,260</td>
    <td>15.2</td>
    <td>304.9</td>
  </tr>
</table>

Table 5: List of the 8 fact verification datasets for our study and their characteristics.

ated by crowd-workers. Evidence, the key information used to validate a claim, is often drawn from textual sources such as news articles, academic papers, and Wikipedia documents. Claim labels vary, with common formats being binary (supports/refutes) or three-class labels (supports/refutes/not enough info).

FEVER (Thorne et al., 2018) It treats Wikipedia as the primary evidence source and instructs crowd-workers to alter sentences from Wikipedia articles to create claims, which are then classified as supporting, refuting, or lacking sufficient information.

VitaminC (Schuster et al., 2021) It generates contrastive evidence pairs for each claim, where the pairs are nearly identical in wording and content, except that one supports the claim while the other does not.

FoolMeTwice (Eisenschlos et al., 2021) develops a multi-player game that encourages diverse strategies for constructing claims (e.g., temporal inference) based on Wikipedia, leading to more complex claims with reduced lexical overlap with the evidence.

Climate-FEVER (Diggelmann et al., 2020) It consists of 1,535 real-world claims related to climate change, collected from the Internet. The five most relevant sentences from Wikipedia are retrieved as evidence, and human annotators label each sentence as supporting, refuting, or providing insufficient information for verification.

Sci-Fact (Wadden et al., 2020) It includes 1.4K expert-generated scientific claims, each linked to abstracts containing supporting evidence, annotated with labels and sentence-level rationales.

PubHealth (Kotonya and Toni, 2020) It contains 11.8K claims accompanied by gold standard judgments crafted by journalists to support or refute the claims. The claims come from five fact checking platforms, news headlines, and news reviews. We pair each claim with its corresponding judgment text as evidence.

COVID-Fact (Saakyan et al., 2021) It consists of 4,086 claims related to the COVID-19 pandemic, collected from the r/COVID19 subreddit. We use the sentence-level evidence annotated by crowd-workers as the supporting evidence.

FAVIQ (Park et al., 2022) It contains 26K claims derived from naturally occurring ambiguous questions posed by real users. Each claim is paired with an answer-containing Wikipedia paragraph as its document-level evidence.

Since many of the original datasets do not provide a publicly available test set, we use their original dev splits as our evaluation sets. Furthermore, we standardize label naming conventions to supports, refutes, and not enough info.

A.2 Fine-Tuning Details

We use LoRA (Hu et al., 2022) and FlashAttention-2 (Dao, 2024) with a cosine learning rate schedule and an initial learning rate of \(1e-4\) to fine-tune LLMs. The maximum sequence length is 1024 and the batch size is 4. The LoRA rank is 16 for all models. We use the Huggingface Transformers (Wolf et al., 2020) library to train all models.

B Appendix: Prompts

This appendix provides the complete set of prompts used in our synthetic data generation pipeline, along with example outputs for each stage.
<table>
  <tr>
    <th>Prompt</th>
    <th>Table Reference</th>
  </tr>
  <tr>
    <td colspan="2"><b>Synthetic data generation</b></td>
  </tr>
  <tr>
    <td>knowledge domain generation</td>
    <td>Table 7</td>
  </tr>
  <tr>
    <td>Evidence plan generation</td>
    <td>Table 9</td>
  </tr>
  <tr>
    <td>Evidence document synthesis</td>
    <td>Table 11</td>
  </tr>
  <tr>
    <td>Key aspect extraction</td>
    <td>Table 13</td>
  </tr>
  <tr>
    <td>Supported claim generation</td>
    <td>Table 15</td>
  </tr>
  <tr>
    <td>Refuted claim generation</td>
    <td>Table 17</td>
  </tr>
  <tr>
    <td>Not-Enough-Info claim generation</td>
    <td>Table 18</td>
  </tr>
  <tr>
    <td colspan="2"><b>Model evaluation</b></td>
  </tr>
  <tr>
    <td>Zero-shot evaluation</td>
    <td>Table 19</td>
  </tr>
  <tr>
    <td>Zero-shot Chain-of-Thought evaluation</td>
    <td>Table 20</td>
  </tr>
  <tr>
    <td colspan="2"><b>Ablation studies</b></td>
  </tr>
  <tr>
    <td>Removing knowledge domain Specification</td>
    <td>Table 21</td>
  </tr>
  <tr>
    <td>Removing Evidence Plan Generation</td>
    <td>Table 22</td>
  </tr>
  <tr>
    <td>Removing Key Aspect Generation</td>
    <td>Table 23</td>
  </tr>
  <tr>
    <td>Direct prompting</td>
    <td>Table 24</td>
  </tr>
</table>

Table 6: A collection of all prompts used in synthetic data generation, model evaluation and ablation studies.

B.1  Knowledge Domain Generation Prompt

B.2  Prompts for Plan-guided Evidence Synthesis

B.3  Prompts for Aspect-Conditioned Claim Generation

B.4  Prompts for Model Evaluation

B.5  Prompts for Ablation Studies
Please generate a list of diverse domains that is highly relevant for claim verification tasks. For each domain, provide:

1. The domain name. Do not include “&” or “and” in the generated domain name. Do not combine two or more words together to form a single domain name.
2. A concise description (1–2 sentences) highlighting its unique linguistic and factual characteristics and explaining why claims in this domain are challenging to verify.
3. A list of several domain properties.

Format your response exactly as follows:
Domain: [Domain Name]
Description: [Your description here]
Properties: [Property 1:Property 2:Property 3]

Table 7: The prompt for knowledge domain generation.

Example Output:
Domain: Climate Science
Description: A dynamic domain involving long-term alteration of temperature and typical weather patterns. Claims are challenging to verify due to reliance on complex climate models and long-term observational data.
Properties: ["Temperature data", "Weather patterns", "Climate models"]

Table 8: The example output for knowledge domain generation.

You are an expert academic writer in the field of claim verification. Given the following knowledge domain:

Domain: [Domain Name]
Description: [A concise description (1–2 sentences) highlighting its unique linguistic and factual characteristics and explaining why claims in this domain are challenging to verify.]
Properties: [Property 1; Property 2; Property 3]

Please generate a concise bullet-point plan for a synthetic evidence document in this domain. Your plan should include:
• An Introduction that sets the description context.
• Key Factual Elements that reflect the domain properties.
• A Conclusion summarizing the key insight or challenge.

Table 9: The prompt for evidence plan generation.

Example Output:
Domain: Political News
Description: Political news is characterized by rapid developments, partisan language, and selective reporting, making fact verification challenging.
Properties: Real-time data; Bias and spin; Official vs. independent sources

Evidence plan:
• Introduction: Briefly introduce a recent political event or debate.
• Key Factual Elements:
  - Reported statistic or claim by a political figure.
  - Data from official sources (e.g., government records) versus independent audits.
  - Mention of bias or discrepancies in reporting.
• Conclusion: Summarize the discrepancy between the claimed figure and the verified data, emphasizing its impact on public trust.

Table 10: The example output for evidence plan generation.

You are a domain expert in claim verification. Expand the following evidence plan into a full evidence document suitable for a claim verification dataset. Ensure that the final document is coherent, detailed, and written in a formal academic style.

Evidence plan:
Insert the plan generated here.

Please generate the complete evidence document following this structure.

Table 11: The prompt for evidence document synthesis.
Example Output:
Evidence Document:
In a recent national debate, a prominent political figure claimed that public spending on infrastructure had increased by 12% over the last fiscal year. However, independent audits and official government records reveal that the actual increase was only 8%, largely due to delays in project implementation and discrepancies in budget reporting. This notable gap between the reported figure and the verified data highlights concerns over potential bias and spin in political reporting, which may ultimately undermine public trust in government accountability.

Table 12: The example output for evidence document synthesis.

You are a domain expert in claim verification. Given the following evidence document and its associated knowledge domain, please extract and list three key Aspects that capture different facets of the evidence. Each key aspect should be a concise description that highlights a distinct dimension (e.g., numerical details, source credibility, or temporal context) and aligns with the domain properties.

Evidence:
In a recent national debate, a prominent political figure claimed that public spending on infrastructure increased by 12% last fiscal year, while independent audits reported an actual increase of only 8%.

knowledge domain:
Domain: Political News
Description: Political news often features conflicting reports and varying interpretations due to partisan Aspects.
Properties: Real-time data; Bias and spin; Official vs. independent sources

Output Format:
1. [Key Aspect 1]
2. [Key Aspect 2]
3. [Key Aspect 3]

Table 13: The prompt for the key aspect extraction from the evidence document.

Example Output:
Statistical Aspect: Focuses on the numerical difference between the 12% claimed and the 8% audited increase.
Source Reliability Aspect: Emphasizes the contrast between the official claim and the independent audits.
Contextual Aspect: Highlights the influence of political debate and potential bias in reporting.

Table 14: The example output for the key aspect extraction from the evidence document.

You are a domain expert in claim verification. Based on the evidence document provided below and a specified key aspect, generate a supported claim that is factually consistent with the evidence while emphasizing the given aspect.

Evidence:
In a recent national debate, a prominent political figure claimed that public spending on infrastructure increased by 12% last fiscal year, while independent audits reported an actual increase of only 8%.

Key Aspect:
Statistical Aspect: Focuses on the numerical difference between the 12% claimed and the 8% audited increase.

Output Format:
Supported Claim: [Your claim here]

Table 15: The prompt for generating supported claims based on key aspects.

Example Output:
Supported Claim: Independent audits confirm that public spending on infrastructure actually increased by only 8% last fiscal year, highlighting a significant statistical discrepancy compared to the 12% claimed by the political figure.

Table 16: The example output for generating supported claims based on key aspects.
You are a highly skilled claim verification expert. Your task is to generate a refuted claim from a given supported claim by applying controlled perturbations that introduce discrepancies between the claim and the underlying evidence. In doing so, consider the following perturbation strategies:

1. Entity Substitution: Replace critical entities in the claim with alternative, contextually plausible candidates.
2. Temporal Modification: Adjust any time-related references, while maintaining overall historical coherence.
3. Relationship Reversal: Invert the relational dynamics between entities, altering the claim’s meaning.
4. Attribute Modification: Modify specific properties or characteristics to create a divergence from the original details.

Now, given the following information, generate a refuted claim.

Supported Claim: [Insert Supported Claim Here]
Evidence: [Insert Relevant Evidence Here]

Output your response in the following format:
Refuted Claim: [Your refuted claim here]

Table 17: The prompt for generating the refuted claim based on the supported claim.

You are a domain expert in claim verification. Given a supported claim that is fully detailed and verifiable based on the evidence, your task is to generate a “Not-Enough-Info” (NEI) claim by intentionally omitting or generalizing the key verifiable elements. Replace precise numerical figures and explicit qualifiers with vague, qualitative, or uncertain language, so that the claim no longer contains enough concrete information for conclusive verification.

For example:
Supported Claim: The study shows a 70% reduction in hospitalization rates.
Not-Enough-Info Claim: The study suggests a significant reduction in hospitalization rates.

Now, please generate a Not-Enough-Info claim for the following input:

Supported Claim: [Insert Supported Claim Here]
Evidence: [Insert Evidence Here]

Output your response in the following format:
Not-Enough-Info Claim: [Your generated Not-Enough-Info claim here]

Table 18: The prompt for generating the Not-Enough-Info claim.

You are a highly skilled claim verification expert. You are given a claim and evidence. Your task is to verify the claim based solely on the evidence provided. Analyze the claim and evaluate the evidence. Based on your evaluation, return one of the following labels: “supports”, “refutes” or “not enough info”.

Evidence: {EVIDENCE}
Claim: {CLAIM}
Label: {LABEL}

Table 19: The prompt used for zero-shot evaluation.

You are a highly skilled claim verification expert. You are given a claim and evidence. Your task is to verify the claim based solely on the evidence provided. Analyze the claim and evaluate the evidence. Based on your evaluation, return one of the following labels: “supports”, “refutes” or “not enough info”. Let’s think step by step.

Evidence: {EVIDENCE}
Claim: {CLAIM}
Label: {LABEL}

Table 20: The prompt used for zero-shot chain-of-thought evaluation.

You are an expert academic writer in the field of claim verification. Given the following domain:

Domain: [Domain Name]
Please generate a concise bullet-point plan for a synthetic evidence document in this domain.

Table 21: The prompt for the ablation study of removing knowledge domain Specification.
You are an expert academic writer. Generate an evidence document based on the following knowledge domain specification. Ensure that the final document is coherent, detailed, and written in a formal academic style.

Domain: [Domain Name]
Description: [A concise description (1–2 sentences) highlighting its unique linguistic and factual characteristics and explaining why claims in this domain are challenging to verify.]
Properties: [Property 1; Property 2; Property 3]
Please generate the complete evidence document following this structure.

Evidence Document:

Table 22: The prompt for the ablation study of removing evidence plan generation.

You are a domain expert in claim verification. Based on the evidence document provided below, generate a supported claim that is factually consistent with the evidence.

Evidence:[Insert the evidence document here.]

Output Format:
Supported Claim: [Your claim here]

Table 23: The prompt for the ablation study of removing key aspect extraction.

You are a domain expert in claim verification. Given a predefine domain, generate one evidence document and three claims about that domain so that their labels are “supports”, “refutes” and “not enough info”, respectively.

Domain: [Domain name]

Output Format:
Evidence Document: [Evidence Document]
Supported Claim: [Supported Claim]
Refuted Claim: [Refuted Claim]
Not-Enough-Info Claim: [Not-Enough-Info Claim]

Table 24: The prompt for the ablation study of the direct prompting method.

Reviewer paper summaries: This paper proposes SYNTHVERIFY, a synthetic data generation framework designed to improve large language models (LLMs) for zero-shot claim verification. The authors demonstrate how systematically generating diverse, structured synthetic data across multiple knowledge domains can significantly boost verification performance on various benchmark datasets. While the empirical results show notable improvements, particularly on zero-shot scenarios, the paper lacks substantial methodological novelty and primarily builds upon existing synthetic data generation strategies.

In this paper, the authors propose a pipeline based on Large Language Models to generate examples for training verifiers in fact-checking tasks. The generated examples are used to build a synthetic dataset targeted at domain-specific tasks where verifiers may face challenges. The pipeline, all LLM based, consists of three steps: Knowledge Domain Generation, Structured Claim Generation, Evidence Synthesis.

- synthetic data generation (using GPT4omini) for improving LLM claim verification in domain-specific settings
    - develop a structured domain specification framework that captures the hierarchical nature of knowledge and verification requirements
        - prompt an LLM to generate a list of knowledge domains (domain names, descriptions and their properties); cluster these domains using DBScan and remove near duplicates; iterate until they reach n diverse domains
    - employ an evidence synthesis method to provide clear verification signals; do it via a structured plan to ensure specificity in details / controllability 
        - first, generate a structured plan for an evidence document from a knowledge domain (includes a domain dependent statement and specific details from the domain’s properties)
        - then, expand that structured plan into an evidence passage 
    - then, implement an aspect-based claim generation approach that ensures diversity in claim types and verifiability of generated statements 
        - first, extract key aspects from the evidence document that capture different dimensions present in the evidence; generate supported claims conditioned on each aspect 
        - generate refuted claims from a supported claims via controlled perturbations (entity substitution, temporal modification, relationship reversal, attribute modification)
        - generated NEI claims replacing identified key elements with vague / qualitative / uncertain language 
- have human annotators validate the quality of 50 examples
- models trained on their data show consistent and considerable improvements in across several different benchmarks 
    - improved zero-shot transfer
    - better handling of complex claims
    - more robust verification across different domains
- additionally, include ablations that cover ever step of their data generation process to validate their use / value
    - in particular, show major gains over naive prompting, even when controlling for the prompting budget 
- explore limits of their synthetic data approach and show saturation after 30k steps
- finally, show that their synthetic data complements fine-tuning on domain-specific data
Summary of strengths: - The method clearly addresses an important and practical issue: the limited domain coverage and high annotation costs of existing datasets.

- Extensive experiments across multiple datasets demonstrate clear empirical gains.

- Thorough ablation studies provide useful insights into the contributions of individual components.

S1. The pipeline is clearly described, with the rationale behind each step well justified, ensuring a coherent and logical flow. The modules are designed to work seamlessly together. 

S2. The soundness has improved a lot with the removal of duplicate domains, bringing strength to the model. Good justification is provided on why synthetic datasets permit more generalization than human data (l160), and an ablation study is performed. More specifically, the ablation has improved significantly, with more steps being ablated, allowing the reader to grasp deeper insights into the important components. Additionally, the authors propose a cost study where they take into consideration the additional cost of their system and show results obtained with fewer training examples, making the system cost-equivalent to a simpler one. Human evaluation is performed, although missing details. SynthVerify is now compared to another synthetic dataset generation method, QACG. The addition of CoT is also interesting.

S3. Clarity: The new running example in 3.3, as well as the new way of generating the evidence, seems more coherent to me.

S4. Framework: The pipeline has been revisited: the evidence generation timeframe has shifted, and the derivation of negative and NEI claims from the positive one might provide the model with more contrastive examples during training.

1) This paper is well written and easy to follow.  It does a good job of setting the goal / context before getting into the details of each section.  The figures are clear and complement the writing.
2) The different aspects of the synthetic data generation process are well motivated and the experimental design (in particular, the ablations) validate the value of the different steps.  
3) The experimentation seems reasonably thorough in terms of substantiating their main claims, that is, that their method for generating synthetic data yields meaningful gains in claim verification.
4) Also appreciated the "budget control" experiments as a natural question is, given a fixed compute budget, what's the most we can expect to get out of synthetic data.
Summary of weaknesses: - The paper provides incremental contributions, largely adapting known synthetic data generation and prompting methods without fundamentally new technical innovations.

- Evaluation is extensive yet limited to standard benchmarks, without deeper analysis into how the method scales or generalizes to fully realistic scenarios beyond controlled environments.

- Minimal discussion on limitations related to synthetic data generation quality, noise, and the risk of biases introduced by LLMs.

W1. Where are the results for Section 3.5? There is no mention of the 50 examples that were manually checked. This would have been a nice follow-up to my previous comment: “The factuality of the evidence generated with this method cannot be guaranteed. While this may not be a critical issue for training data, as the model would still learn to be faithful to the evidence when trained on the synthetic dataset, it would be valuable to conduct a post-analysis of how factual the evidence is.”

W2. Even though QACG was added, having more fine-tuning baselines to compare against would improve the soundness. Also, I understand that your method doesn’t require input documents and that comparing it with models fine-tuned on domain-specific data might be unfair - however, such comparisons would help quantify the gap between your method and the best available alternatives.

W3. In the example from Section 3.3, there is only one evidence passage used to construct the claim - is that the case? In the previous submission, there were multiple evidence passages, which seemed more coherent.

1) I was a little surprised to see that models do not see further benefits in training beyond 30k synthetic examples.  That makes me think of this work as a kind of task-specific instruction tuning through synthetic data (teach a model the form of the claim verification task) instead of the underlying reasoning.  Part of what makes this difficult to assess is that the paper does not report what kind of performance the models achieve on held out examples of the synthetic corpus after tuning.  Do the converged models achieve near perfect performance on the synthetic claim data?  If not (and there's still considerable headroom), that would suggest that there's more to squeeze out of this kind of synthetic data through architectural improvements / test time compute / etc.  If they have, then perhaps this saturation result needs to be reflected in the framing of the paper in the introduction / its claims.
2) In line 417, the authors use the term "significant" but as far as I can tell, there was no significance testing performed.  Could the authors speak to this?  The paper would certainly benefit from its addition.
3) I appreciate the human analysis (3.5) of the synthetic data described in the paper but it seems like the authors only report the agreement among the human annotators and not how well the humans agree with the synthetic labels?  If the kappa number reported includes the model labels, could the authors make that clearer?  And also perhaps report standard classification numbers between the synthetic labels and the majority vote of the human annotators?  I think knowing this would be useful in thinking about the saturation dimensions I raise in question 1 -- ie, if the data is very high quality and there's headroom in task performance, that increases the value of this kind of approach.
4) I think a brief description of the evaluation datasets (or at least the domains they cover) should be included in the paper somewhere -- perhaps as a row in the main results table?  They are relevant for making sense of the results and can't be relegated in their entirety to the Appendix.  Additionally, the paper would be strengthened with some kind of summary analysis of the domains reflected in the synthetic data so that readers can understand what kinds of domains are feasible to generate / how well they transfer to improved performance in the downstream domains.
Comments, suggestions and typos: - **Enhance Methodological Novelty**: Clearly articulate the novel contributions of your work relative to existing synthetic data generation approaches. Highlight specifically how your approach improves upon existing prompting or synthetic generation methodologies.

- **Generalization & Realism**: Consider adding additional experiments or a brief discussion demonstrating SYNTHVERIFY’s applicability and effectiveness on completely unseen, complex, or real-world scenarios outside standard benchmarks.

- **Data Quality Analysis**: Provide qualitative examples illustrating high-quality versus problematic synthetic claims to give readers a clearer insight into the strengths and limitations of your synthetic data generation process.

- **Ethical Considerations**: Explicitly discuss potential ethical risks associated with generating synthetic claims (e.g., introducing biases or misinformation) and suggest possible mitigation strategies.

- **Complexity of Reasoning**: Add examples or discussions highlighting the framework’s performance on claims requiring complex or multi-hop reasoning, clearly outlining areas of strength and limitation.

There is a missing space on line 10.
	On line 212, I believe it should be a comma, not a period.
	Lines 935 to 941 seem incorrect — all the titles are merged together.
	The example on line 348 is a nice addition; however, it doesn’t seem correct. If the evidence mentions a 70% reduction, then both claims (one stating a significant reduction and the other mentioning a 70% reduction) should be labeled as SUPPORTS. The evidence would need to be changed to justify a different label.
	
	It also appears that the quality control process was replaced by human evaluation, but it is still mentioned in the introduction — this could be clarified.

- in a few places, the paper uses the term "hierarchical" but I actually don't find that the generated domains are hierarchical?  it left me a little confused as I found myself looking for the domains to be organized in some kind of tree structure
Overall assessment scores: 2.5

3.5

3.5
Metareview score: 4
Metareview: This paper introduces SYNTHVERIFY, a synthetic data generation framework designed to improve zero-shot claim verification for LLMs. The framework addresses the limitations of LLMs in handling domain-specific knowledge by generating structured, diverse claim-evidence pairs through a multi-stage pipeline. The pipeline includes steps like pattern-based claim generation, evidence synthesis, and systematic verification. The authors demonstrate that training LLMs on the generated synthetic data results in significant improvements in verification accuracy across specialized domains, showing the potential of synthetic data in enhancing fact-checking tasks.
Summary of reasons to publish: The paper clearly describes a structured, hierarchical framework for generating domain-diverse synthetic data. The multi-stage pipeline, including knowledge domain generation, structured claim generation, and plan-guided evidence synthesis, is reasonable for the task of claim verification.

The framework is validated through extensive experiments across multiple claim verification benchmarks (e.g., FEVER, SciFact, VitaminC) and diverse models (from Llama-1B to Llama-8B).

The paper addressed the concerns from last round.