Venue: NAACL 2024

# Evidence-Driven Retrieval Augmented Response Generation for Online Misinformation

Zhenrui Yue  Huimin Zeng  Yimeng Lu  Lanyu Shang

Yang Zhang  Dong Wang

University of Illinois Urbana-Champaign

{zhenrui3, huiminz3, yimengl4, lshang3, yzhangnd, dwang24}@illinois.edu

###### Abstract

The proliferation of online misinformation has posed significant threats to public interest. While numerous online users actively participate in the combat against misinformation, many of such responses can be characterized by the lack of politeness and supporting facts. As a solution, text generation approaches are proposed to automatically produce counter-misinformation responses. Nevertheless, existing methods are often trained end-to-end without leveraging external knowledge, resulting in subpar text quality and excessively repetitive responses. In this paper, we propose retrieval augmented response generation for online misinformation (RARG), which collects supporting evidence from scientific sources and generates counter-misinformation responses based on the evidences. In particular, our RARG consists of two stages: (1) evidence collection, where we design a retrieval pipeline to retrieve and rerank evidence documents using a database comprising over 1M academic articles; (2) response generation, in which we align large language models (LLMs) to generate evidence-based responses via reinforcement learning from human feedback (RLHF). We propose a reward function to maximize the utilization of the retrieved evidence while maintaining the quality of the generated text, which yields polite and factual responses that clearly refutes misinformation. To demonstrate the effectiveness of our method, we study the case of COVID-19 and perform extensive experiments with both in- and cross-domain datasets, where RARG consistently outperforms baselines by generating high-quality counter-misinformation responses.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 5628–5643

June 16-21, 2024 0204 Association for Computational Linguistics

## 1 Introduction

As social media discussions on trending topics continue to grow, increased presence of online misinformation has been observed on such sources Chen et al. (2022); Chen and Shu (2023b). Yet timely intervention against the spreading of misinformation is often lacking, resulting in potential threats to public interest Ball and Maxmen (2020); Roozenbeek et al. (2020). For instance, vaccine hesitancy is strongly associated with the exposure of antivaccine misinformation Pierri et al. (2022). Therefore, it is essential to identify and curb the spreading of misinformation before it leads to severe consequences Litou et al. (2017); Zhu et al. (2021). To identify text-based misinformation, various approaches have been proposed by leveraging language models Shu et al. (2020); Wu et al. (2022c); Shu et al. (2022); Yue et al. (2022, 2023); Chen and Shu (2023a); Liu et al. (2024). Upon the detection of misinformation, crowdsourcing approaches (e.g, users, fact-checkers) are designed to prevent the continued propagation of misinformation via counter-responses Veeriah (2021); Vraga and Bode (2021); Kou et al. (2022a); Seo et al. (2022); Kou et al. (2022b); Bozarth et al. (2023); Drolsbach and Prollochs (2023). Nevertheless, such responses to misinformation often lack politeness, direct refutation or fail to provide supporting evidence He et al. (2019).

Figure 1: The proposed retrieval augmented response generation (RARG), which collects scientific evidence to generate counter-misinformation responses.



2023; Tafur and Sarkar, 2023). Moreover, human efforts are often less effective when confronted with an overwhelming amount of online misinformation (Micallef et al., 2020, 2022).

To facilitate early intervention at scale, language generation methods are proposed to generate counter-responses against detected misinformation (Vo and Lee, 2019, 2020; He et al., 2023). For example, MisinfoCorrect adopts reinforcement learning to generate responses to refute misinformation (He et al., 2023). Despite their effectiveness, existing generation methods are trained end-to-end on a specific domain. As a result, the generated texts often highly resemble the training responses and demonstrate deteriorated quality upon domain shifts (as we show in Section 4). Additionally, such generation models are unaware of external knowledge and must be frequently updated to incorporate up-to-date domain knowledge (Lewis et al., 2020b; Izacard and Grave, 2021; Borgeaud et al., 2022; Asia et al., 2023). As such, we consider a retrieval augmented generation (RAG) setting that incorporates relevant scientific documents to utilize external evidence without retraining. That is, given input misinformation, our first objective is to retrieve evidence from an extensive collection of documents (e.g., academic publications). Then, we select the relevant documents as evidence to perform retrieval augmented generation, with the goal of extracting supporting facts to debunk misinformation. We illustrate an example of our framework in Figure 1, where evidence-backed counter-response is generated upon the identification of misinformation.

In this paper, we focus on counter-response generation against online misinformation and propose an evidence-driven retrieval augmented response generation (RARG) framework, which efficiently retrieves supporting documents and generates responses using the collected supporting facts. Specifically, RARG consists of two modules: (1) evidence collection, in which we design a two-stage retrieval pipeline based on a collection of over 1M academic articles we collected. For efficient retrieval, our pipeline first performs a coarse search over the data collection, followed by a reranking stage with improved relevance estimation of the retrieved documents; (2) evidence-based response generation, here, we align large language models (LLMs) to generate responses upon the collected evidence via reinforcement learning from human feedback (RLHF). In particular, we propose a reward design that maximizes the utilization of the retrieved evidence while maintaining the quality of the responses. As such, our RARG generates polite and factual responses that clearly refutes input misinformation. To the best of our knowledge, we are the first to introduce a retrieval augmented generation framework with two-stage retrieval and fine-grained RLHF. To demonstrate the effectiveness of RARG, we study the case of COVID-19, where we perform both in- and cross-domain experiments with comprehensive quantitative and qualitative analyses. Experimental results highlight the effectiveness of the proposed framework, where RARG consistently outperforms state-of-the-art baselines by generating high-quality evidence-based responses against online misinformation.

We summarize our contributions as follows:

1. We propose a retrieval augmented generation setting for counter-misinformation response generation. In this work, we focus on COVID-19 misinformation and collected over 1M academic publications as the source for evidence-based response generation.
2. We design RARG, a response generation framework against online misinformation. Our framework combines two-stage retrieval for evidence collection and RLHF-based LLM alignment, such that RARG is optimized to generate polite and factual counter-responses.
3. We show the effectiveness of RARG by experimenting on both in-domain and cross-domain COVID misinformation to validate the generalization of RARG. Both quantitative and qualitative results demonstrate that RARG can outperform state-of-the-art methods in generating high-quality responses.

## 2 Related Work

### Detecting and Countering Misinformation

Existing methods for detecting misinformation can be broadly categorized into: (1) content-based detection, in which machine learning models are trained upon input contents (i.e., claims) to perform classification (Yue et al., 2022; Zeng et al., 2022; Jiang et al., 2022; Yue et al., 2023; Chen and Shu, 2023a; Liu et al., 2023; Mendes et al., 2023; Zeng et al., 2024; Liu et al., 2024). Additional modalities such as image, video or propagation paths can also be used to improve detection performance (Shang et al., 2021; Santhosh et al., 

2022; Shang et al., 2022b; Wu et al., 2022c; Zhou et al., 2023); (2) evidence-based detection, where external knowledge can be collected as supporting evidence to verify input contents Brand et al. (2021); Kou et al. (2022a); Wu et al. (2022a); Yang et al. (2022); Shang et al. (2022c); Xu et al. (2022); Zhao et al. (2023). For example, knowledge graphs or retrieved document pieces can be processed to support or refute statements Kou et al. (2021); Hu et al. (2021b); Koloski et al. (2022); Kou et al. (2022b); Shang et al. (2022a); Wu et al. (2022b).

To curb the spreading of misinformation, the majority of existing approaches relies on social correction (e.g., comments, replies by users or experts Veeriah (2021); Vraga and Bode (2021); Seo et al. (2022); Bozarth et al. (2023). Despite their effectiveness, many of such responses can be characterized by the lack of politeness and supporting facts. Furthermore, user efforts often fall short when confronted with an overwhelming amount of misinformation Micallef et al. (2020, 2022); He et al. (2023). Recently, language-based methods are proposed to generate counter-responses / explanations Vo and Lee (2019, 2020); He et al. (2023); Wan et al. (2024). However, these methods concentrate on improving text quality, often neglecting the response factuality through the incorporation of evidence. Hence, we aim to design a counter-misinformation response generation framework via the collection of relevant documents, followed by the reasoning and generation of responses.

### Retrieval Augmented Generation

Recent progress in large language models (LLMs) has led to substantial improvements in both language understanding and generation Raffel et al. (2020); Brown et al. (2020); Wei et al. (2021); Ouyang et al. (2022); Chowdhery et al. (2022); Touvron et al. (2023); OpenAI (2023). Thanks to the extensive corpora used during pretraining, LLMs have the ability to embed world knowledge in their parameters, and thus achieve significant performance enhancements across various scenarios Touvron et al. (2023); OpenAI (2023); Penedo et al. (2023). Nevertheless, LLMs struggle to capture fine-grained knowledge and frequently exhibit instances of hallucination Sun et al. (2023); Peng et al. (2023). To incorporate up-to-date knowledge without expensive retraining, retrieval augmented generation is proposed to generate text conditioned on collected documents Lewis et al. (2020); Izacard and Grave (2021); Borgeaud et al. (2022); Izacard et al. (2022); Shi et al. (2023); Ram et al. (2023). For example, REALM retrieves from a large set of documents such as Wikipedia to solve conditional generation tasks like question answering Guu et al. (2020). Nevertheless, current retrieval augmented methods primarily study knowledge-intensive NLP tasks and have not been well researched for response generation against online misinformation, let alone the combination of fine-grained retrieval with RLHF-based alignment. As such, our work explores retrieval augmented response generation by collecting scientific evidence and performing RLHF alignment to generate evidence-based counter-responses.

## 3 Methodology

### Preliminary

We consider the following problem setup for counter-misinformation response generation: given input misleading claim \(x\), our objective is to: (1) collect a set of \(m\) scientific evidence \(\{e_{i}\}^{m}_{i=1}\) that are relevant to input \(x\) to be used as supporting evidence; and (2) generate response \(y\) upon input \(x\) and the retrieved evidence \(\{e_{i}\}^{m}_{i=1}\), which should demonstrate certain desirable properties (e.g., politeness and factuality), see example in Figure 1. We elaborate our framework in the following.

**Input & Output**: We denote the evidence collection model as \(f_{\mathrm{ret}}\) and the response generation model as \(f_{\mathrm{gen}}\). Formally, our research framework can be defined as two sub-problems of information retrieval (i.e., evidence retrieval) and retrieval augmented text generation (i.e., response generation), with each of the settings defined as follows:

* _Evidence Retrieval_: Given input misinformation \(x\), human annotated evidence \(e\) and a collection of \(n\) evidence documents \(\{e_{i}\}^{n}_{i=1}\) (with \(e\in\{e_{i}\}^{n}_{i=1}\)), the model \(f_{\mathrm{ret}}\) should ideally generate the highest relevance score for the claim-evidence pair (i.e., \(f_{\mathrm{ret}}(x,e)=\max\{f_{\mathrm{ret}}(x,e_{i})\}^{n}_{i=1}\)). During training, input \(x\) and \(e\) can be used to optimize \(f_{\mathrm{ret}}\). In inference, we collect a subset of \(m\) documents \(\{e_{i}\}^{m}_{i=1}\) for response generation, with \(m\ll n\).
* _Response Generation_: For response generation purpose, we incorporate both input claim \(x\) and collected evidence subset \(\{e_{i}\}^{m}_{i=1}\) from the previous step in the input prompt, while \(y\) represents the generated response from \(f_{\mathrm{gen}}\) (i.e., \(y=f_{\mathrm{gen}}(x,\{e_{i}\}^{m}_{i=1})\)). To learn the generation 

model, we first train \(f_{\rm gen}\) end-to-end, followed RLHF tuning. The inference stage is completed by collecting evidence documents and performing evidence-based response generation.

**Optimization:** The retrieval model \(f_{\rm ret}\) and generation model \(f_{\rm gen}\) are parameterized by \(\theta_{\rm ret}\) and \(\theta_{\rm gen}\). To learn \(f_{\rm ret}\), we maximize the relevance score of the label evidence \(e\) upon input \(x\). In other words, we minimize the expected loss of input-evidence pair \((x,e)\): \(\min_{\theta_{\rm ret}}\mathbb{E}_{(x,e)\sim\mathcal{X}_{\rm ret}}[\mathcal{L} (\theta_{\rm ret},(x,e))]\), with \(\mathcal{X}_{\rm ret}\) representing the evidence retrieval dataset. For \(f_{\rm gen}\), the optimization is two-fold: (1) \(\theta_{\rm gen}\) is first trained end-to-end by minimizing cross entropy loss on input-evidence-response triplets, where \(m\) can be empirically selected (i.e., \(\min_{\theta_{\rm gen}}\mathbb{E}_{(x,\{e_{i}\}^{m}_{i=1},y)\sim\mathcal{X}_{ \rm gen}}[\mathcal{L}(\theta_{\rm gen},(x,\{e_{i}\}^{m}_{i=1},y))]\)); (2) to improve the response quality, we leverage an additional reinforcement learning step to tune \(\theta_{\rm gen}\) towards generating responses of human preference. In particular, we design the reward by considering the following aspects that could help reducing the spread of misinformation Starbird et al. (2014); Chan et al. (2017); Tanaka and Hirayama (2019); Malhotra et al. (2022); He et al. (2023):

* _Refutation_: Refutation involves phrases within the response that explicitly and objectively expose the error or inaccuracy of input claims.
* _Factuality_: Factuality evaluates the correctness and supporting evidence of the response, which reflects the reliability of the generated text.
* _Politeness_: Politeness refers to using considerate and thoughtful language, and thus respectfully counters misinformation and avoids backfire.
* _Claim Relevance_: Response should establish a clear and direct connection to the input claim for improved coherence and comprehension.
* _Evidence Relevance_: The relevant evidence information should be included in the response to demonstrate the falsehood of input claim.

To summarize, we introduce a retrieval augmented response generation framework with two-stage evidence retrieval and fine-grained reinforcement learning, which learns to construct evidence-based counter-responses against misinformation.

### Two-Stage Evidence Retrieval

Existing retrieval augmented generation approaches adopts unsupervised sparse retrieval to collect relevant documents Lewis et al. (2020b); Izacard and Grave (2021); Izacard et al. (2022); Ram et al. (2023). An example of such algorithms include BM25, which computes query-document relevance by considering word frequency in both query and documents Robertson et al. (1995). Although BM25 is widely used, it does not yield satisfactory ranking performance for knowledge-intensive tasks such as question answering. Therefore, dense retrieval methods are proposed for improved text understanding and relevance estimation Karpukhin et al. (2020); Ren et al. (2021); Izacard et al. (2021); Wang et al. (2022, 2023b). Nevertheless, dense retrieval is known to be inefficient for massive data quantity and require large amounts of annotated data for training. This limitation renders dense retrieval to be less effective in retrieval augmented response generation, where only limited annotated data is available for training.

Unlike existing retrieval methods, we design a two-stage retrieval pipeline in RARG that performs coarse-to-fine ranking, which improves the computation efficiency and retrieval performance with limited data. Specifically, we include the following stages: (1) retrieve a smaller subset from the large collection of evidence documents via BM25; and (2) rerank the retrieved subset using a dense retriever fine-tuned on limited claim-evidence pairs. In the first retrieval stage, we adopt BM25 to efficiently generate a small subset \(\{e_{i}\}^{m}_{i=1}\) from a much large collection \(\{e_{i}\}^{n}_{i=1}\). While BM25 may occasionally retrieve less or ir-relevant documents, we observe that in most cases, the desired evidence can still be found in \(\{e_{i}\}^{m}_{i=1}\) with proper selection of \(m\). In our implementation, we adopt \(m=20\) as the subset size to balance the performance and efficiency of our retrieval pipeline. In the following stage, we fine-tune a dense retrieval model to perform fine-grained reranking and select the most relevant documents as supporting evidence. Compared to sparse retrieval, this additional stage yields notable performance improvements with minimal computational overhead. We illustrate the overall retrieval pipeline of RARG in Figure 2 (left).

For dense retriever \(f_{\rm den}\) within \(f_{\rm ret}\), fine-tuning often requires large amounts of annotated data for improved ranking performance and generalizability Karpukhin et al. (2020); Wang et al. (2022). Despite the insufficiency of data, the retrieval results from our first stage can be used as a coarse estimation of claim-evidence relevance. In other words, it is possible to leverage the BM25 scores 

from the first stage, combined with limited annotated examples, to optimize the dense retrieval model. Specifically for input-evidence pair \((x,e)\), we sample \(k\) positive documents \(\{e_{i}^{p}\}_{i=1}^{k}\) and \(k\) negative documents \(\{e_{i}^{n}\}_{i=1}^{k}\) using BM25 and inverse BM25 scores, which avoids introducing extensive noise in the training data. Based on the positive set \(\{e_{i}^{p}\}_{i=1}^{k}\), we design a ranking loss to enlarge the margin between evidence \(e\) and the highest ranked positive evidence in \(\{e_{i}^{p}\}_{i=1}^{k}\) (i.e., \(f_{\mathrm{den}}(x,e)-\max\{f_{\mathrm{den}}(x,e_{i}^{p})\}_{i=1}^{k}\))). Here, we maximize the relevance between input-evidence pairs by applying penalty only when the margin is below threshold \(\tau\). In addition to the ranking loss, our optimization goal contains a contrastive term based on InfoNCE loss Chen et al. (2020), which improves the relevance estimation between input-evidence pairs while 'pushing away' negative evidence. Formally, the overall optimization objective \(\mathcal{L}\) for \(f_{\mathrm{den}}\) can be formulated as:

\[\mathbb{E}_{(x,e)\sim X_{\mathrm{ret}}}[ \max(0,\max(\{f_{\mathrm{den}}(x,e_{i}^{p})\}_{i=1}^{k})-f_{ \mathrm{den}}(x,e)+\tau) \tag{1}\] \[-\lambda\frac{\exp(f_{\mathrm{den}}(x,e))}{\exp(f_{\mathrm{den}} (x,e))+\sum_{k}\exp(f_{\mathrm{den}}(x,e_{i}^{n}))}],\]

where \(\tau\) is the margin threshold for the ranking loss and \(\lambda\) is a scaling factor. For each pair of \(x\) and \(e\), the first term in Equation (1) is effective when \(f_{\mathrm{den}}(x,e)\) score is not \(\tau\) greater than the score of the highest ranked positive evidence. The next term, on the other hand, maximizes exponential score of the annotated input-evidence pair over the sum of those from the sampled negative evidence. We illustrate the optimization of the dense retriever in Figure 2 (right), where we differentiate the annotated evidence \(e\) from the positive evidence \(\{e_{i}^{p}\}_{i=1}^{k}\) and 'push away' the negative evidence \(\{e_{i}^{n}\}_{i=1}^{k}\).

For the evidence document collection, since we focus on the case of COVID-19, we limit the evidence documents to academic articles on COVID research from reliable sources. In particular, we collect all available articles from two large databases of COVID-related research: CORD Wang et al. (2020) and LitCovid Chen et al. (2021). In sum, we obtain 1,056,262 articles from CORD and 301,136 articles from LitCovid1. The collected articles are preprocessed to filter invalid and repeated items. As a result, 1,118,112 articles remain in our evidence database and we extract title and abstract information from each article as the evidence corpus. To fine-tune our dense retriever model, we utilize Check-COVID, a dataset with \(\sim\)350 evidence documents and \(\sim\)1k input-evidence pairs for training, with all documents collected from the CORD dataset Wang et al. (2023a). To improve the ranking performance over large evidence collections, we manually increase the document size of Check-COVID to 5k by sampling additional documents from CORD and LitCovid. We use \(k=4\) in \(\{e_{i}^{p}\}_{i=1}^{k}\) and \(\{e_{i}^{n}\}_{i=1}^{k}\), we also provide more data collection and processing details in Section 4 and Appendix A.

Footnote 1: CORD is maintained until June 2022, while LitCovid is active as of October 2023. Our LitCovid collection is updated with information available until August 2023.

### Response Generation

Despite recent progress in large language models (LLMs), it is frequently observed that LLMs demonstrate hallucination behavior due to the lack of knowledge and reasoning capabilities Sun et al. (2023); Peng et al. (2023). To improve the factuality of LLM-generated text, retrieval augmented text generation is used to incorperate external knowledge Lewis et al. (2020b); Izacard and Grave (2021); Borgeaud et al. (2022); Ram et al. (2023). Yet the training of such models heavily relies on the quality of the data and demands extensive tuning efforts to achieve substantial performance improvements. On the other hand, generating large amounts of responses with desired properties (e.g., factuality,

Figure 2: The proposed two-stage retrieval pipeline in RARG (left) and its optimization (right).

 

policteness) can be time-consuming and costly. In contrast, gathering basic human feedback such as preference ranking proves to be a notably simpler and more cost-effective approach for feedback collection (Ouyang et al., 2022). Therefore, we combine the advantages of LLMs, which can be easily instruction-tuned with small-size data, while exploiting the potential for further improvement through RLHF (Ouyang et al., 2022).

To optimize the generation model, we propose a RLHF-based approach to generate responses towards human preference. Specifically, \(f_{\mathrm{gen}}\) is first fine-tuned in a supervised fashion to generate counter-response conditioned on input claim and retrieved evidence. At the same time, a reward model is constructed by leveraging binary human feedback (e.g., whether responses are refuting, factual and polite) as training data to evaluate the response quality. The reward model is then applied in the reinforcement learning stage, where the fine-tuned \(f_{\mathrm{gen}}\) (i.e., actor model) is trained to further improve generation quality using proximal policy optimization (PPO) (Schulman et al., 2017). Upon deployment, only the actor model is required for inference, which generates counter-misinformation responses based on input claims and collected evidence. We provide an illustration for the response generation process and its optimization in Figure 3.

**Supervised Fine-Tuning.** Reinforcement learning is known for being unstable for complex tasks, therefore, we first perform supervised fine-tuning using a pretrained LLM (e.g., Llama 2) to learn initial capabilities in counter-misinformation response generation. Additionally, the fine-tuned model is used as a reference model (i.e., critic model in Figure 3) in the following reinforcement learning step to stabilize training. We denote the supervised finetuned model as \(\pi_{\mathrm{ref}}\) (i.e. reference model).

**Modeling Reward.** The purpose of reward modeling is to leverage human feedback for evaluating the text quality. Unlike methods that generate text pairs for preference evaluation (Ouyang et al., 2022), we train smaller LMs to model human feedback, as the swift convergence accelerates reward estimation in RLHF. For a dataset of \(\{x_{i},y_{i},r_{i}\}^{n}_{i=1}\) with \(r\in\{0,1\}\) representing the reward label from human feedback (i.e., refutation, factuality or politeness in binary form), the objective of modeling reward is to learn a modeling function \(f\) that minimizes the expected negative log-likelihood loss. The learnt reward model evaluates the quality of the generated responses. In our implementation, we adopt three BERT models to learn human feedback in refutation, factuality, politeness, and use the sum of the scores as reward. To further improve the coherence of the response w.r.t. input claim and retrieved evidence, we assess the relevance between claim-response pairs (i.e, \((x,y)\)) and evidence-response pairs (i.e, \((e,y)\)). To maximize parameter efficiency, we utilize \(f_{\mathrm{den}}\) from the previous retrieval module, since \(f_{\mathrm{den}}\) is specifically trained to model the evidence relevance. Thus the estimated reward \(\hat{r}\) is formulated as:

\[\begin{split}&\hat{r}=f_{\mathrm{reftuation}}(x,\{e_{i}\}^{m}_{ i=1},y)+f_{\mathrm{factuality}}(x,\{e_{i}\}^{m}_{i=1},y)\\ &+f_{\mathrm{politness}}(x,\{e_{i}\}^{m}_{i=1},y)+\alpha(f_{ \mathrm{den}}(x,y)\\ &+\max\{f_{\mathrm{den}}(e_{i},y)\}^{m}_{i=1}),\end{split} \tag{2}\]

where \(f_{\mathrm{reftuation}}\), \(f_{\mathrm{factuality}}\) and \(f_{\mathrm{politness}}\) model refutation, factuality and politeness respectively. \(f_{\mathrm{den}}(x,y)\) is used to evaluate claim-response relevance, and \(\max\{f_{\mathrm{den}}(e_{i},y)\}^{m}_{i=1}\) evaluates the best

Figure 3: The optimization of \(f_{\mathrm{gen}}\) in RARG. Upon input claim, the evidence documents are retrieved for response generation. Then, the reward model estimates the rewards and update the actor model with PPO-based RL.

 

matching evidence-response relevance. \(\alpha\) is a hyperparameter to scale the relevance reward.

**Reinforcement Learning.** For reinforcement learning with human feedback, we use Equation (2) to assess the quality of the generated responses. The objective is to maximize the expected reward w.r.t. the actor model under the constraint that the parameters in \(\pi_{\mathrm{act}}\) do not significantly deviate from \(\pi_{\mathrm{ref}}\). Formally, the learning is formulated as:

\[\max_{\pi_{\mathrm{act}}} \mathbb{E}_{(x,e)\sim(x_{i},c_{i})^{n}_{=1},y\sim\pi_{\mathrm{act} }(x,e)}[\hat{r}(x,e,y)] \tag{3}\] \[-\beta\mathcal{D}[\log\pi_{\mathrm{act}}(y|x,e)]|\log\pi_{\mathrm{ ref}}(y|x,e)],\]

where \(\beta\) is a hyperparameter to regularize the output difference between \(\pi_{\mathrm{act}}\) and \(\pi_{\mathrm{ref}}\), and \(\mathcal{D}\) stands for the KL divergence. Here, we aim to find the optimal policy \(\pi_{\mathrm{act}}\) that maximizes the expected reward \(\hat{r}\). The additional KL divergence term controls how far the actor model can travel from the reference model via the minimization of KL divergence. Therefore, penalizing the KL distance effectively prevents optimization instability or model collapse. During training, we initialize the actor model \(\pi_{\mathrm{act}}\) with reward head using the weights from \(\pi_{\mathrm{ref}}\) and leverage PPO as the learning algorithm Schulman et al. (2017); Ouyang et al. (2022). We provide an illustration of the training pipeline in Figure 3, where the upper subfigure demonstrates evidence retrieval and response generation. The lower subfigure explains our reinforcement learning scheme, in which the generated responses are used to estimate the rewards and compute the KL distances. Finally, the rewards and KL penalty are used the compute the training loss and update the actor model. After the reinforcement learning stage, the resulting actor model \(\pi_{\mathrm{act}}\) is used as our response generation model in RARG.

## 4 Experiments

### Experiment Design

**Evidence Retrieval.** Our retrieval module consists of BM25 and \(f_{\mathrm{den}}\) initialized with E5 Wang et al. (2022)). We adopt the Check-COVID dataset for evidence retrieval evaluation Wang et al. (2023a), the adopted metrics are NDCG and Recall (i.e., N@\(k\) and R@\(k\)) with \(k\in[1,3,5]\). For baselines, we adopt sparse algorithms TFIDF and BM25 Robertson et al. (1995). We also incorporate state-of-the-art dense retrievers DPR and E5 for comparison Karpukhin et al. (2020); Wang et al. (2022).

**Response Generation.** We adopt our collected scientific articles (see Section 3.2) and use the top-3 / top-5 documents from retrieval. Our \(f_{\mathrm{gen}}\) is based on Llama 2 (7B) Touvron et al. (2023), with the supervised fine-tuning variant denoted by RARG (S). We adopt MisinfoCorrect dataset for training He et al. (2023) and perform both in-domain and cross-domain evaluation (on Constraint and ANTiVax) Patwa et al. (2021); Hayawi et al. (2022). We follow the evaluation from He et al. (2023) using metrics refutation (R.), factuality (F.) and politeness (P.), we also evaluate the response relevance to input claim (C.) and evidence (E.) using our dense retriever (i.e., \(f_{\mathrm{den}}\)). Baseline methods include BART, DialoGPT, PARTNER, GODEL, MisinfoCorrect (MisinfoC.), Llama 2 w/ and w/o retrieval (denoted with Llama and Llama (R)) and GPT-3.5 Lewis et al. (2020); Zhang et al. (2020); Sharma et al. (2022); He et al. (2023); Touvron et al. (2023); Brown et al. (2020).

 

pared to the second best results. (2) In contrast to sparse retriever along, the additional dense retrieval significantly improves the ranking performance. For example, RARG achieves \(37.61\%\) performance improvement in Recall@5 compared to BM25. (3) The performance gains through our dense retriever increases as we narrow the size of the retried subset (i.e., top-\(k\)). For instance, the NDCG improvements compared to the best baseline rise from \(6.07\%\) to \(8.04\%\) with \(k\) decreasing from 5 to 3. (4) By leveraging the ranking margin and contrastive learning, our dense retriever successfully exploits noisy BM25 results and outperforms the second best baseline E5 by \(15.28\%\) in top-1 score, indicating strong reranking performance with RARG. Overall, we find the two-stage retrieval pipeline in RARG performs well even with limited training data, which demonstrates substantially improved retrieval performance.

We additionally study the effect of \(\lambda\) in the training objective of the retriever model, which regularizes the strength of contrastive loss. In particular, we vary the value of \(\lambda\) from 0 to 0.5 and evaluate the retrieval performance on NDCG and Recall with \(k\in[3,5]\). We focus on top-3 and top-5 scores as they are representative ranking scores and show consistent trends with changing \(\lambda\). We present the performance visually in Figure 4, with x-axis representing the \(\lambda\) values and y-axis representing the ranking scores. We observe consistent improvements by applying the contrastive loss, followed by minor changes with futher increasing \(\lambda\) values. The performance for all metrics remains robust even with the maximum value of 0.5. In sum, the proposed contrastive loss improves the performance of RARG retrieval, and performs quite robust regardless of hyperparameter selections.

### Response Generation Results

For response generation, we leverage the retrieved documents as part of the input prompt to generate the counter-responses. We report the in-domain response generation results in Table 2, with rows representing response generation models and columns representing different metrics. To evaluate the model generalization, we additionally perform response generation on cross-domain datasets Constraint and ANTiVax Patwa et al. (2021); Hayashi et al. (2022), with results presented in Table 3. We also demonstrate the generation quality on cross-domain data via qualitative examples in Table 6.

From both in- and cross-domain evaluation results we observe: (1) Both RARG versions perform well on in-domain data and achieve superior performance over baseline methods. For example, RARG outperforms the best baseline on the refutation metric with \(3.65\%\) relative improvement. (2) RLHF-tuned RARG can further improve generation quality, achieving an average \(2.40\%\) performance improvement on refutation, factuality and politeness metrics compared to RARG (S). (3) Despite significantly reduced size (i.e., 7B), RARG performs similarly or exceeds GPT-3.5 on all metrics. In particular, RARG significantly outperforms GPT-3.5 on refutation, which may be attributed to the misinterpretation of certain examples by GPT-3.5, as we demonstrate in Table 6. (4) In cross-domain experiments (Table 3), although most models show comparable performance on refutation, factuality and politeness metrics, RARG can generate responses of substantially enhanced coherence and quality. In contrast, baseline methods exhibit signs of overfitting, leading to responses that highly resemble the training data (see Table 6). (5) Llama 2 and GPT-3.5 perform well in cross-domain response generation (Table 3). Yet they show significantly reduced refutation scores, likely stemming from the frequent misclassification (i.e., as if responding to valid information). (6) Combined both quantitative and qualitative results on cross-domain data, RARG clearly outperforms baseline generation methods with significantly improved claim and evidence relevance as well as generation quality. Overall, we conclude that the RLHF-tuned RARG demonstrates enhanced claim understanding, evidence-based reasoning and response generation abilities. Furthermore, RARG exhibits superior performance in countering both in-domain and cross-domain mis

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{4}{c}{**MisinfoCorrect Dataset**} \\ \cline{2-6}  & R. \(\uparrow\) & F. \(\uparrow\) & P. \(\uparrow\) & C. \(\uparrow\) & E. \(\uparrow\) \\ \hline
**BART** & 0.824 & 0.623 & 0.824 & 0.799 & 0.685 \\
**DialoGPT** & 0.831 & 0.693 & 0.874 & 0.800 & 0.689 \\
**PARTNER** & 0.792 & 0.779 & 0.790 & 0.759 & 0.663 \\
**GDDEL** & 0.931 & 0.904 & 0.987 & 0.751 & 0.680 \\
**MisinfoC.** & 0.916 & 0.914 & 0.927 & 0.788 & 0.686 \\
**Llama** & 0.727 & 0.950 & 0.984 & 0.761 & 0.674 \\
**Llama** (R) & 0.761 & 0.942 & 0.984 & 0.776 & 0.694 \\
**GPT-3.5** & 0.857 & **0.971** & 0.987 & 0.805 & 0.690 \\ \hline
**RARG** (S) & 0.922 & 0.944 & 0.988 & 0.804 & 0.702 \\
**RARG** & **0.965** & 0.967 & **0.989** & **0.812** & **0.704** \\ \hline \hline \end{tabular}
\end{table}
Table 2: In-domain response generation results, with best results in bold and second best results underlined.

 

information, highlighting the potential of RARG in generating counter-misinformation responses across a wide range of real-world scenarios.

## 5 Conclusion

In this paper, we propose a novel evidence-driven retrieval augmented response generation framework RARG against online misinformation. To the best of our knowledge, RARG is the first to introduce evidence-backed response generation to counter misinformation. The proposed RARG comprises of: (1) evidence retrieval, where evidence documents are efficiently collected and reranked; and (2) response generation, in which RARG generates evidence-based counter-responses that are factual and polite. We demonstrate the effectiveness of RARG by performing extensive experiments on multiple datasets, where RARG can consistently generate evidence-based responses with improved quality over state-of-the-art baseline methods.

## 6 Limitations

Despite introducing RARG for evidence-based counter-response generation against online misinformation, we have not discussed the setting where test domains significantly differ from COVID (e.g., fake news), which may hinder the deployment of the proposed method for more generalized applications. In addition, we have not studied the case when the reliability of evidence sources can not be guaranteed (e.g., documents from unverifiable online sources), which may introduce inaccuracies and impact the validity of the generated responses. Due to the limited research scope and budgets, we have not conducted human evaluations to assess the overall response quality of RARG. Therefore, we plan to explore a more generalized and domain-adaptive solution with an improved evaluation protocol for counter-response generation in the future.

## Acknowledgment

This research is supported in part by the National Science Foundation under Grant No. IIS-2202481, CHE-2105032, IIS-2130263, CNS-2131622, CNS-2140999. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.

## References

* Asai et al. [2023] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. _arXiv preprint arXiv:2310.11511_.
* Ball and Maxmen [2020] Philip Ball and Amy Maxmen. 2020. The epic battle against coronavirus misinformation and conspiracy theories. _Nature_, 581(7809):371-375.
* Borgeaud et al. [2022] Sebastian Borggeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In _International conference on machine learning_, pages 2206-2240. PMLR.
* Bozarth et al. [2023] Lia Bozarth, Jane Im, Christopher Quarles, and Ceren Budak. 2023. Wisdom of two crowds: Misinformation moderation on reddit and how to improve this process--a case study of covid-19. _Proceedings of the ACM on Human-Computer Interaction_, 7(CSCW1):1-33.
* Bader et al. [2017]

\begin{table}
\begin{tabular}{l l c c c c|c c c c c} \hline \hline  & \multicolumn{4}{c}{**Constraint Dataset**} & \multicolumn{4}{c}{**I**} \\ \cline{2-13}  & R. \(\uparrow\) & F. \(\uparrow\) & P. \(\uparrow\) & C. \(\uparrow\) & E. \(\uparrow\) & I. & R. \(\uparrow\) & F. \(\uparrow\) & P. \(\uparrow\) & C. \(\uparrow\) & E. \(\uparrow\) \\ \hline
**BART** & 0.949 & **0.969** & **0.987** & 0.692 & 0.608 & I & 0.942 & 0.949 & 0.960 & 0.741 & 0.641 \\
**DialoGPT** & 0.953 & 0.953 & 0.968 & 0.636 & 0.584 & I & 0.952 & 0.952 & 0.966 & 0.748 & 0.641 \\
**PARTNER** & 0.952 & 0.953 & 0.968 & 0.633 & 0.585 & I & 0.887 & 0.882 & 0.901 & 0.732 & 0.632 \\
**GODEL** & 0.938 & 0.908 & 0.986 & 0.594 & 0.578 & I & 0.931 & 0.906 & 0.985 & 0.706 & 0.668 \\
**MisinfoC.** & **0.955** & 0.955 & 0.970 & 0.636 & 0.584 & I & 0.952 & 0.952 & 0.967 & 0.747 & 0.641 \\
**Llama** & 0.595 & 0.929 & 0.977 & 0.745 & 0.628 & I & 0.655 & 0.942 & 0.984 & 0.753 & 0.663 \\
**Liama** (R) & 0.708 & 0.930 & 0.976 & 0.750 & 0.634 & I & 0.722 & 0.943 & 0.984 & 0.775 & 0.689 \\
**GPT-3.5** & 0.855 & 0.933 & 0.979 & 0.795 & **0.684** & I & 0.909 & 0.948 & 0.987 & 0.780 & 0.689 \\ \hline
**RARG** (S) & 0.941 & 0.937 & 0.981 & 0.799 & 0.679 & I & 0.966 & 0.975 & 0.989 & 0.791 & 0.687 \\
**RARG** & 0.951 & 0.940 & 0.982 & **0.805** & 0.682 & I & **0.971** & **0.978** & **0.991** & **0.795** & **0.691** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Cross-domain response generation results, with best results in bold and second best results underlined.

 

Erik Brand, Kevin Roitero, Michael Soprano, and Gianluca Demartini. 2021. E-bart: Jointly predicting and explaining truthfulness. In _TTO_, pages 18-27.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Chan et al. (2017) Man-pui Sally Chan, Christopher R Jones, Kathleen Hall Jamieson, and Dolores Albarracin. 2017. Debunking: A meta-analysis of the psychological efficacy of messages countering misinformation. _Psychological science_, 28(11):1531-1546.
* Chen and Shu (2023a) Canyu Chen and Kai Shu. 2023a. Can llm-generated misinformation be detected? _arXiv preprint arXiv:2309.13788_.
* Chen and Shu (2023b) Canyu Chen and Kai Shu. 2023b. Combating misinformation in the age of llms: Opportunities and challenges. _arXiv preprint arXiv:2311.05656_.
* Chen et al. (2022) Canyu Chen, Haoran Wang, Matthew Shapiro, Yunyu Xiao, Fei Wang, and Kai Shu. 2022. Combating health misinformation in social media: Characterization, detection, intervention, and open issues. _arXiv preprint arXiv:2211.05289_.
* Chen et al. (2021) Qingyu Chen, Alexis Allot, and Zhiyong Lu. 2021. Litcovid: an open database of covid-19 literature. _Nucleic acids research_, 49(D1):D1534-D1540.
* Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In _International conference on machine learning_, pages 1597-1607. PMLR.
* Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. _arXiv preprint arXiv:2204.02311_.
* Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm. int8 (! 8-bit matrix multiplication for transformers at scale. _arXiv preprint arXiv:2208.07339_.
* Drolsbach and Prollochs (2023) Chiara Patricia Drolsbach and Nicolas Prollochs. 2023. Diffusion of community fact-checked misinformation on twitter. _Proceedings of the ACM on Human-Computer Interaction_, 7(CSCW2):1-22.
* Guu et al. (2020) Kelvin Guo, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In _International conference on machine learning_, pages 3929-3938. PMLR.
* Hayawi et al. (2022) Kadhim Hayashi, Sakib Shahriar, Mohamed Adel Serhani, Ikbal Taleb, and Sujith Samuel Mathew. 2022. Anti-vax: a novel twitter dataset for covid-19 vaccine misinformation detection. _Public health_, 203:23-30.
* He et al. (2023) Bing He, Mustaque Ahamad, and Srijan Kumar. 2023. Reinforcement learning-based counter-misinformation response generation: A case study of covid-19 vaccine misinformation. In _Proceedings of the ACM Web Conference 2023_, pages 2698-2709.
* Hu et al. (2021a) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021a. Lora: Low-rank adaptation of large language models. _arXiv preprint arXiv:2106.09685_.
* Hu et al. (2021b) Linmei Hu, Tianchi Yang, Luhao Zhang, Wanjun Zhong, Duyu Tang, Chuan Shi, Nan Duan, and Ming Zhou. 2021b. Compare to the knowledge: Graph neural fake news detection with external knowledge. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 754-763, Online. Association for Computational Linguistics.
* Izacard et al. (2021) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense information retrieval with contrastive learning. _arXiv preprint arXiv:2112.09118_.
* Izzacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 874-880, Online. Association for Computational Linguistics.
* Izacard et al. (2022) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. _arXiv preprint arXiv:2208.03299_.
* Jiang et al. (2022) Gongyao Jiang, Shuang Liu, Yu Zhao, Yueheng Sun, and Meishan Zhang. 2022. Fake news detection via knowledgeable prompt learning. _Information Processing & Management_, 59(5):103029.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781, Online. Association for Computational Linguistics.
* Koloski et al. (2022) Boshko Koloski, Timen Stepisnik Perdih, Marko Robnik-Sikconja, Senja Pollak, and Blaz Skrl. 2022. Knowledge graph informed fake news classification via heterogeneous representation ensembles. _Neurocomputing_.
* Kou et al. (2022a) Ziyi Kou, Lanyu Shang, Yang Zhang, and Dong Wang. 2022a. Hc-covid: A hierarchical crowdsource knowledge graph approach to explainable covid-19 misinformation detection. _Proceedings of the ACM on Human-Computer Interaction_, 6(GROUP):1-25 

Ziyi Kou, Lanyu Shang, Yang Zhang, Christina Youn, and Dong Wang. 2021. Fakesens: A social sensing approach to covid-19 misinformation detection on social media. In _2021 17th International Conference on Distributed Computing in Sensor Systems (DCOSS)_, pages 140-147. IEEE.
* Kou et al. [2022b] Ziyi Kou, Lanyu Shang, Yang Zhang, Zhenrui Yue, Huimin Zeng, and Dong Wang. 2022b. Crowd, expert & ai: A human-ai interactive approach towards natural language explanation based covid-19 misinformation detection. In _Proc. Int. Joint Conf. Artif. Intell.(IJCAI)_, pages 5087-5093.
* Lewis et al. [2020a] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Online. Association for Computational Linguistics.
* Lewis et al. [2020b] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktiaschel, et al. 2020b. Retrieval-augmented generation for knowledge-intensive nlp tasks. _Advances in Neural Information Processing Systems_, 33:9459-9474.
* Litou et al. [2017] Iouliana Litou, Vana Kalogeraki, Ioannis Katakis, and Dimitrios Gunopulos. 2017. Efficient and timely misinformation blocking under varying cost constraints. _Online Social Networks and Media_, 2:19-31.
* Liu et al. [2023] Hui Liu, Wenya Wang, and Haoliang Li. 2023. Interpretable multimodal misinformation detection with logic reasoning. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 9781-9796, Toronto, Canada. Association for Computational Linguistics.
* Liu et al. [2024] Hui Liu, Wenya Wang, Haoru Li, and Haoliang Li. 2024. Teller: A trustworthy framework for explainable, generalizable and controllable fake news detection. _arXiv preprint arXiv:2402.07776_.
* Malhotra et al. [2022] Pranav Malhotra, Kristina Scharp, and Lindsey Thomas. 2022. The meaning of misinformation and those who correct it: An extension of relational dialectics theory. _Journal of Social and Personal Relationships_, 39(5):1256-1276.
* Mendes et al. [2023] Ethan Mendes, Yang Chen, Wei Xu, and Alan Ritter. 2023. Human-in-the-loop evaluation for early misinformation detection: A case study of COVID-19 treatments. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15817-15835, Toronto, Canada. Association for Computational Linguistics.
* Micallef et al. [2020] Nicholas Micallef, Bing He, Srijam Kumar, Mustaque Ahamad, and Nasir Memon. 2020. The role of the crowd in countering misinformation: A case study of the covid-19 infodemic. In _2020 IEEE international Conference on big data (big data)_, pages 748-757. IEEE.
* Micallef et al. [2022] Nicholas Micallef, Marcelo Sandoval-Castaneda, Adi Cohen, Mustaque Ahamad, Srijan Kumar, and Nasir Memon. 2022. Cross-platform multimodal misinformation: Taxonomy, characteristics and detection for textual posts and videos. In _Proceedings of the International AAAI Conference on Web and Social Media_, volume 16, pages 651-662.
* OpenAI [2023] R OpenAI. 2023. Gpt-4 technical report. _arXiv_, pages 2303-08774.
* Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Patwa et al. [2021] Parth Patwa, Shivam Sharma, Srinivas Pykl, Vineeth Gupta, Gitianjali Kumari, Md Shaok Akhtar, Asif Ekbal, Amitava Das, and Tanmony Chakraborty. 2021. Fighting an infodemic: Covid-19 fake news dataset. In _International Workshop on Combating Online Mobile Posts in Regional Languages during Emergency Situation_, pages 21-29. Springer.
* Penedo et al. [2023] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. _arXiv preprint arXiv:2306.01116_.
* Peng et al. [2022] Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, and Jianfeng Gao. 2022. Godel: Large-scale pre-training for goal-directed dialog. _arXiv preprint arXiv:2206.11309_.
* Peng et al. [2023] Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. _arXiv preprint arXiv:2302.12813_.
* Pierri et al. [2022] Francesco Pierri, Brea L Perry, Matthew R DeVerna, Kai-Cheng Yang, Alessandro Flammini, Filippo Menczer, and John Bryden. 2022. Online misinformation is linked to early covid-19 vaccination hesitancy and refusal. _Scientific reports_, 12(1):5966.
* Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _The Journal of Machine Learning Research_, 21(1):5485-5551.
* Ram et al. [2023] Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. _arXiv preprint arXiv:2302.00083_ 

Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. PAIR: Leveraging passage-centric similarity relation for improving dense passage retrieval. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 2173-2183, Online. Association for Computational Linguistics.
* Robertson et al. (1995) Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. _Nist Special Publication Sp_, 109:109.
* Roozenbeek et al. (2020) Jon Roozenbeek, Claudia R Schneider, Sarah Dryhurst, John Kerr, Alexandra LJ Freeman, Gabriel Recchia, Anne Marthe Van Der Bles, and Sander Van Der Linden. 2020. Susceptibility to misinformation about covid-19 around the world. _Royal Society open science_, 7(10):201199.
* Santhosh et al. (2022) Nikita Mariam Santhosh, Jo Cheriyan, and Lekshmi S Nair. 2022. A multi-model intelligent approach for rumor detection in social networks. In _2022 International Conference on Computing, Communication, Security and Intelligent Systems (ICSISI)_, pages 1-5. IEEE.
* Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _arXiv preprint arXiv:1707.06347_.
* Seo et al. (2022) Haeseung Seo, Aiping Xiong, Gian Lee, and Dongwon Lee. 2022. If you have a reliable source, say something: effects of correction comments on covid-19 misinformation. In _Proceedings of the international AAAI conference on web and social media_, volume 16, pages 896-907.
* Shang et al. (2022a) Lanyu Shang, Ziyi Kou, Yang Zhang, Jin Chen, and Dong Wang. 2022a. A privacy-aware distributed knowledge graph approach to qois-driven covid-19 misinformation detection. In _2022 IEEE/ACM 30th International Symposium on Quality of Service (IWQoS)_, pages 1-10. IEEE.
* Shang et al. (2021) Lanyu Shang, Ziyi Kou, Yang Zhang, and Dong Wang. 2021. A multimodal misinformation detector for covid-19 short videos on itktok. In _2021 IEEE International Conference on Big Data (Big Data)_, pages 899-908. IEEE.
* Shang et al. (2022b) Lanyu Shang, Ziyi Kou, Yang Zhang, and Dong Wang. 2022b. A duo-generative approach to explainable multimodal covid-19 misinformation detection. In _Proceedings of the ACM Web Conference 2022_, pages 3623-3631.
* Shang et al. (2022c) Lanyu Shang, Yang Zhang, Zhenrui Yue, YeonJung Choi, Huimin Zeng, and Dong Wang. 2022c. A knowledge-driven domain adaptive approach to early misinformation detection in an emergent health domain on social media. In _2022 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)_, pages 34-41. IEEE.
* Sharma et al. (2021) Ashish Sharma, Inna W Lin, Adam S Miner, David C Atkins, and Tim Althoff. Towards facilitating empathic conversations in online mental health support: A reinforcement learning approach.
* Shi et al. (2023) Weijia Shi, Sewon Min, Michihiro Yasunaga, Mingoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Replug: Retrieval-augmented black-box language models. _arXiv preprint arXiv:2301.12652_.
* Shu et al. (2022) Kai Shu, Ahmadreza Mosallanezhad, and Huan Liu. 2022. Cross-domain fake news detection on social media: A context-aware adversarial approach. In _Frontiers in Fake Media Generation and Detection_, pages 215-232. Springer.
* Shu et al. (2020) Kai Shu, Suhang Wang, Dongwon Lee, and Huan Liu. 2020. _Disinformation, misinformation, and fake news in social media_. Springer.
* Starbird et al. (2014) Kate Starbird, Jim Maddock, Mania Orand, Peg Achterman, and Robert M Mason. 2014. Rumors, false flags, and digital vigilantes: Misinformation on twitter after the 2013 boston marathon bombing. _IConderence 2014 proceedings_.
* Sun et al. (2023) Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Lunna Dong. 2023. Head-to-tail: How knowledgeable are large language models (llm)? aka will lms replace knowledge graphs? _arXiv preprint arXiv:2308.10168_.
* Tafur and Sarkar (2023) Bruno Tafur and Advair Sarkar. 2023. User perceptions of automatic fake news detection: Can algorithms fight online misinformation? _arXiv preprint arXiv:2304.07926_.
* Tanaka and Hirayama (2019) Yuko Tanaka and Runti Hirayama. 2019. Exposure to counterring messages online: alleviating or strengthening false belief? _Cyberpsychology, Behavior, and Social Networking_, 22(11):742-746.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Veeriah (2021) Jeyasushima Veeriah. 2021. Young adults' ability to detect fake news and their new media literacy level in the wake of the covid-19 pandemic. _Journal of Content, Community and Communication_, 13(7):372-383.
* Vo and Lee (2019) Nguyen Vo and Kyumin Lee. 2019. Learning from fact-checkers: Analysis and generation of fact-checking language. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 335-344.
* Vo and Lee (2020) Nguyen Vo and Kyumin Lee. 2020. Standing on the shoulders of guardians: Novel methodologies to combat fake news. _Disinformation, Misinformation, and Fake News in Social Media: Emerging Research Challenges and Opportunities_, pages 183-210 

Emily K Vraga and Leticia Bode. 2021. Addressing covid-19 misinformation on social media preemptively and responsively. Emerging infectious diseases, 27(2):396.
* Wan et al. [2024] Herun Wan, Shangbin Feng, Zhaoxuan Tan, Heng Wang, Yulia Tsvetkov, and Minnan Luo. 2024. Dell: Generating reactions and explanations for llim-based misinformation detection. arXiv preprint arXiv:2402.10426.
* Wang et al. [2022a] Gengyu Wang, Kate Harwood, Lawrence Chillrud, Amith Ananthram, Melanie Subbiah, and Kathleen McKeown. 2022a. Check-COVID: Fact-checking COVID-19 news claims with scientific evidence. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada. Association for Computational Linguistics.
* Wang et al. [2022] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv e-prints, pages arXiv-2212.
* Wang et al. [2023b] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023b. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368.
* Lu Wang et al. [2020] Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Douglas Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Kinney, et al. 2020. Cord-19: The covid-19 open research dataset. ArXiv.
* Wei et al. [2021] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
* Wu et al. [2022a] Junfei Wu, Qiang Liu, Weizhi Xu, and Shu Wu. 2022a. Bias mitigation for evidence-aware fake news detection by causal intervention. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2308-2313.
* Wu et al. [2022b] Junfei Wu, Weizhi Xu, Qiang Liu, Shu Wu, and Liang Wang. 2022b. Adversarial contrastive learning for evidence-aware fake news detection with graph neural networks. arXiv preprint arXiv:2210.05498.
* Wu et al. [2022c] Xueqing Wu, Kung-Hsiang Huang, Yi Fung, and Heng Ji. 2022c. Cross-document misinformation detection based on event graph reasoning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States. Association for Computational Linguistics.
* Xu et al. [2022] Weizhi Xu, Junfei Wu, Qiang Liu, Shu Wu, and Liang Wang. 2022. Evidence-aware fake news detection with graph neural networks. In Proceedings of the ACM Web Conference 2022, pp. 2501-2510.
* Yang et al. [2022] Zhiwei Yang, Jing Ma, Hechang Chen, Hongzhan Lin, Ziyang Luo, and Yi Chang. 2022. A coarse-to-fine cascaded evidence-distillation neural network for explainable fake news detection. In Proceedings of the 29th International Conference on Computational Linguistics, Piscataway, USA, August 26-261, Atlanta, USA, June 10-12, New York, NY, USA, May 10-12, New York 



## Appendix A Experiment Details

### Evidence Retrieval

Our two-stage retrieval consists of BM25 and a fine-tuned dense retriever. The dense retriever is initialized with the pretrained embeddings from bidirectional encoder representations (E5) Wang et al. (2022). We describe the experimental design below.

* _Dataset_: For retrieval evaluation, we adopt the Check-COVID dataset that provides claim-evidence pairs Wang et al. (2023a). The dataset statistics of the training/test set is provided in Table 4.
* _Baseline_: For baseline models, we adopt unsupervised sparse retrieval algorithms TFIDF and BM25 Robertson et al. (1995). We also incorporate two state-of-the-art dense retrievers for comparison: dense passage retrieval (DPR) and E5 retriever Karpukhin et al. (2020); Wang et al. (2022).
* _Evaluation_: In our evaluation, we adopt the following evaluation ranking metrics: normalized discounted cumulative gain (NDCG@\(k\)) and recall (Recall@\(k\)) with \(k\in[1,3,5]\). We validate the model using the best NDCG@10 scores and rerank the retrieved results.

For the subsequent response generation experiments, we adopt our collected scientific articles as the evidence sources (see Section 3.2) and collect the top-3 / top-5 documents for each input claim.

### Response Generation

For experiments in response generation, the base model is the 7B version of Llama 2 Touvron et al. (2023). To improve parameter efficiency, RARG is trained with 8bit quantization and LoRA Hu et al. (2021); Dettmers et al. (2022), we report the experiment design below.

* _Dataset_: We adopt MisinfoCorrect for training He et al. (2023). All models are trained with the identical training set and evaluated on a non-overlapping test set for unbiased assessment. To evaluate model generalization, we additionally adopt Constraint and ANTiVax, with 500 sampled examples each, see Table 4 Patwa et al. (2021); Hayawi et al. (2022).
* _Baseline_: For baseline models, we adopt multiple state-of-the-art baselines. In particular, we adopt text generation-based BART, DialoGPT, GODEL Lewis et al. (2020); Zhang et al. (2020); Peng et al. (2022), reinforcement learning-based methods PARTNER and MisinfoCorrect Sharma et al. (2023), LLM-based Llama 2, Llama 2 with retrieval (with identical retrieval as in RARG, denoted with Llama (R)) and GPT-3.5 Brown et al. (2020).
* _Evaluation_: We follow the evaluation protocol from, with metrics in refutation, factuality and politeness using the supervised trained LMs in Section 3.3 He et al. (2023). We also evaluate the response relevance to input claims and the collected evidence using the relevance estimation model (i.e., dense retriever) as in Section 3.2.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Datasets** & **\#Eval.** & **\#Claim** & **\#Resp.** & **\#Leng.** \\ \hline
**Check-COVID** & 5.6k & 1,068/229 & N/A & 19.4 \\
**MisinfoC.** & N/A & 568/134 & 568 & 38.8 \\
**ANTiVax** & N/A & 500 & N/A & 37.2 \\
**Constraint** & N/A & 500 & N/A & 25.0 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Dataset statistics.

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **BA**\(\uparrow\) & **Acc.**\(\uparrow\) & **F1**\(\uparrow\) & **Prec.**\(\uparrow\) & **Rec.**\(\uparrow\) \\ \hline
**Refutation** & 0.835 & 0.800 & 0.873 & 0.978 & 0.788 \\
**Factuality** & 0.928 & 0.933 & 0.942 & 0.925 & 0.961 \\
**Politeness** & 0.849 & 0.881 & 0.923 & 0.941 & 0.905 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Reward modeling results.



Besides RARG, we also report results of the supervised fine-tuned version of RARG (i.e., RARG (S)). For reward modeling, we train three BERT models on the binary human feedback data w.r.t refutation, factuality and politeness. As the label distribution is imbalanced, we adopt stratified sampling to split train/test sets and apply class-balanced cross entropy loss for training. Besides accuracy, F1, precision and recall, we also use balanced accuracy (BA) to evaluate class-balanced performance on human feedback classification. The evaluation results for reward modeling are in Table 5. In summary, BERT models achieve well-balanced classification performance across all human feedback metrics, with average BA of 0.871 and accuracy of 0.871.

### Implementation

For retrieval baseline methods, we follow the original works for implementation and hyperparameter selection. For RARG retrieval pipeline, we use the E5-base version for further tuning and train with AdamW optimizer for 5 epochs. We search the learning rate from [1e-5, 2e-5, 3e-5] and select both \(\tau\) and \(\lambda\) from \([0.1,0.2,0.3,0.4,0.5]\). Training is performed with 100 warm up steps and cosine learning rate scheduler. For sampling evidence, 4 positive evidence documents and 4 negative documents are adopt for training. We align the rest settings with the original E5 work (Wang et al., 2022). For the generation pipeline, we adopt publicly available implementation from the original authors. To train our 7B version Llama 2, we adopt 8bit quantization and LoRA for parameter efficient tuning, which is less than \(0.2\%\) of the original parameters in Llama 2 (Hu et al., 2021; Dettmers et al., 2022). We use consistent LoRa settings with 16 as LoRA dimension, 32 as LoRA \(\alpha\) along with 0.05 for LoRA dropout. The LoRA target modules are the \(Q\) and \(V\) projection matrices. For supervised fine-tuning, we train 3 epochs with learning rate of 1e-4 and 0.03 percentage of the steps as warmup. For reinforcement tuning, we adopt PPO with 1e-5 learning rate and 0.2 as initial KL regularization. Training epochs is selected between 1 to 3, batch size is selected between 4 to 32, and we update parameters after 4 gradient accumulation steps. To construct the text input for instruction tuning, we prepend evidence as context, followed by an instruction that describes the task. We also incorporate the input misinformation in the prompt before the start of the response token. The resulting prompt template is:

```
%%Instruction {evidence};Basedontheaboveevidence,determineiftheclaimisvalidand explainwhy:{claim} %%#Response {response} whereevidenceandclaimarefilledbycollectedevidenceandclaim.Fortraining,theannotatedresponseofeachdataexamplefollows.Forinference,theresponsepositionisleftemptyforgeneration.
```

## Appendix B Qualitative Results

We provide additional qualitative examples to compare the response quality among different response generation methods, examples can be found in Table 6. Due to space limitations, we select GODEL, MisinfoCorrect, Llama 2 (R) and GPT-3.5 for comparison, as the generated responses from such models are of the highest quality. Since RARG (S) and RARG generate similar responses to most claims, we provide qualitative example with RLHF-tuned version of RARG. Overall, we notice high similarity in the responses from GODEL and MisinfoCorrect compared to the training data. For LLMs, we observe significantly increased response length by Llama 2 (R) and GPT-3.5, which may affect the non-redundancy and readability scores in the response quality evaluation. In addition, Llama 2 (R) and GPT-3.5 often demonstrate false responses (i.e., hallucination) by taking the input claim as valid. For example in the example 1, both Llama 2 and GPT-3.5 incorrectly identifies the claim as valid, and thus generates invalid responses to the original misinformation. Similarly in example 2, GPT-3.5 produces questionable rationales on the efficacy of masks in combating COVID, possibly due to the lack of claim understanding. Considering both the model size and efficiency, RARG shows improved response generation capabilities and can outperform baseline methods w.r.t. the response quality 



**Response Examples from Constraint**

\begin{table}
\begin{tabular}{l l} \hline  & \\
* allowing more poison into air._ \\ \hline
**GODEL** & The statement that the vaccine manipulates the DNA is incorrect. As verified by fact-checkers, the vaccine ... This is not correct. The vaccine is not related to COVID-19. The vaccine only uses mRNA for ... \\
**Llama** (R) & The claim made in the statement is **valid**. There have been several studies conducted that suggest a ... \\
** GPT-3.5** & The claim is **valid**. Studies have shown a correlation between air pollution and COVID-19 mortality rates ... \\
**RARG** & This is not correct. The air pollution has nothing to do with the mortality rate. The mortality rate is related to the virus. The air pollution is not the cause of the virus. Please stop this misinformation. \\ \hline
**Claim** & _Says a warning label on a box of disposable masks shows that they are ineffective at protecting against the spread of COVID-19._ \\ \hline
**MODEL** & The assertion is wrong. The vaccine is not the gene therapy and the nurse is not committing crime. The ... \\
**MisinfoC.** & The vaccine is not a vaccine. It is safe, and it is well tested on many people. Please stop spreading ... \\
**LDA** (R) & The claim that a warning label on a box of disposable masks shows that they are ineffective at protecting ... \\
**GPT-3.5** & It is **impossible to determine the validity** of this claim without seeing the actual warning label on the ... \\
**RARG** & The warning label is not correct. The masks are effective at protecting against the spread of COVID-19. \\ \hline  & \\ \end{tabular} 
\begin{tabular}{c c} \multicolumn{2}{c}{**Response Examples from ANTIVax**} \\ \hline
**Claim** & _People who don’t care about Covid: You can’t live in fear of the virus. Also people who don’t care about Covid: I’m not getting a vaccine. It’s experimental. We don’t know what’s in it. Me: So about living in fear?_ \\ \hline
**GODEL** & Your understanding about the experiment is inaccurate. The vaccine is not the gene therapy, and it does ... \\
**MisinfoC.** & This is not correct. The vaccine is not experimental, and it is not the experimental gene therapy. It only ... \\
**Llama** (R) & The claim that people who don’t care about COVID-19 are living in fear of the virus is not valid. It is ... \\
**GPT-3.5** & The claim is not valid. The statement highlights a contradiction in the attitudes of people who claim they ... \\
**RARG** & This is not correct. The vaccine is not experimental. It is well tested and safe. \\ \hline
**Claim** & _Firstly, how long have you worked there? 2nd If you have been there more than 3 years you will have more rights. 3rd If you don’t want this untested vaccine, find examples of people who have had heart attacks and died because of Vaccine. 4th Mention your human rights_ \\ \hline
**GODEL** & Your understanding is not correct. The vaccine is safe, and before we take it, it has been tested on many ... \\
**MisinfoC.** & Your claim is not correct. The vaccine is not untested and has not been found to cause heart problems ... \\
**Llama** (R) & The claim you provided is not valid and is based on a series of false assumptions and misleading statement ... \\
**GPT-3.5** & This claim is not valid. It contains several unrelated statements and does not present a clear argument ... \\
**RARG** & Please stop spreading misinformation. The vaccine is not untested. It has been tested on thousands of people. It is safe and effective. \\ \hline \end{tabular}
\end{table}
Table 6: Generated responses for baseline methods and RARG on Constraint and ANTiVax datasets.