Reinforcement Tuning for Detecting Stances and Debunking Rumors Jointly with Large Language Models (2024)

Abstract: Learning multi-task models for jointly detecting stance and verifying rumors poses challenges due to the need for training data of stance at post level and rumor veracity at claim level, which are difficult to obtain. To address this issue, we leverage large language models (LLMs) as the foundation annotators for the joint stance detection (SD) and rumor verification (RV) tasks, dubbed as JSDRV. We introduce a novel reinforcement tuning framework to enhance the joint predictive capabilities of LLM-based SD and RV components. Specifically, we devise a policy for selecting LLMannotated data at the two levels, employing a hybrid reward mechanism to choose highquality labels for effective LLM fine-tuning on both tasks. Results demonstrate that JSDRV improves the capabilities of LLMs in the joint tasks, not only outperforming state-of-the-art methods but also generalizing to non-LLMs accommodated as task models.

heading: 1 Introduction
text: Social media has transformed the ways people access information by facilitating rapid information sharing. However, it has become a fertile ground for nurturing rumors and misinformation due to its lack of systematic moderation (Vosoughi et al., 2018; Cheng et al., 2021). Their rampant spread has become a considerable global societal issue that can profoundly impact people’s beliefs and normal life (Roozenbeek and van der Linden, 2019).
In general, stance provides insights into the attitudes, opinions, and beliefs of individuals regarding a specific target (Küçük and Can, 2020; ALDayel and Magdy, 2021). Stance and rumor are deeply coupled since the stance expressed by social media posts toward a rumorous event, i.e., rumor stance, offers valuable cues for assessing the overall credibility of the target claim. Figure 1 shows
* Jing Ma is the corresponding author.
that the rumor stance helps in understanding the context and the way the rumor is perceived by different users in an intuitive and explainable manner.
Early research has found that detecting stances expressed in related conversation threads on social media is beneficial for rumor detection and monitoring (Qazvinian et al., 2011; Zhao et al., 2015; Zubiaga et al., 2016b). Nowadays, researchers are increasingly focusing on leveraging related stances for improving rumor detection and verification effectiveness (Dungs et al., 2018; Ma et al., 2020; Li et al., 2020; Tian et al., 2020; Yuan et al., 2021) as well as performing stance-rumor joint detection tasks through multi-task learning (Ma et al., 2018; Kochkina et al., 2018; Wei et al., 2019b; Li et al., 2019; Yu et al., 2020; Yang et al., 2022).
However, most existing approaches require extensive post-level stance labels and claim-level veracity labels for training stance detection and rumor verification models, respectively, which are very expensive to obtain. While some “for free” unsupervised (Kobbe et al., 2020; Allaway and McKeown, 2020a; Pick et al., 2022; Li et al., 2023) meth-
13423
ods were developed, they are only for the stance detection task and suffer from poor generalizability due to reliance on crafted features or specific models pre-trained on the data (e.g., online debate) unmatched with rumor stances from social media. More recently, a weakly supervised method (Yang et al., 2022) based on multiple instance learning (Dietterich et al., 1997; James Richard Foulds, 2010) is proposed to predict the stance of an individual post and the rumor veracity of the claim only being supervised with claim veracity labels. However, determining post stances via dispersing the veracity of a source claim down to stances of many individual posts, as illustrated by Figure 1, is intuitively much more challenging than inferring a source claim’s veracity via aggregating post stances in an opposite direction.
The emergence of disruptive technologies for Large Language Models (LLMs) such as ChatGPT (Brown et al., 2020) and smaller variants such as Llama (Touvron et al., 2023) and ALPACA (Taori et al., 2023) have shown performance and explainability on a par with human or even better in various NLP tasks. However, their abilities in rumor-stance related tasks within social media context are still understudied, especially for the joint prediction for detecting posts stances and debunking rumors from claims at the same time. On one hand, LLMs might suit such tasks thanks to their rich pre-trained knowledge and strong zeroand few-shot capabilities; on the other hand, their predictive power based on a large number of rumorrelated social media posts might be compromised by noise, unreliability, and lack of supervision.
To study and unleash the potential of LLMs for this joint task, we propose a reinforcement tuning framework for joint stance detection and rumor verification (JSDRV) based on LLMs1. The framework contains three complementary parts: the LLM stance detection (SD) network, the reinforcement label selector, and the LLM rumor verification (RV) network. Specifically, assuming merely a small set of seeding veracity labels at claim level, the reinforcement selector chooses high-quality examples for fine-tuning the SD and RV LLMs based on their generated labels and explanations. We present an end-to-end joint optimization mechanism to boost the integrated framework. Our contributions are summarized as follows:
1Code is released at https://anonymous.4open. science/r/JSDRV-F3CE/
• We propose a novel LLM-based reinforcement tuning framework to detect stance and verify rumor veracity jointly starting off with a small set of seed instances labeled by humans.
• Our JSDRV framework is generic, which can not only accommodate open or closed LLMs, but also non-LLM-based models as stance detection and rumor verification networks.
• Extensive experiments on multiple benchmark rumor stance datasets demonstrate that JSDRV outperforms a range of strong baselines, including pretrained language models and fully supervised models on both tasks.

heading: 2 Related Work
text: Rumor Verification. Early studies on rumor verification train supervised classifiers by utilizing content (Yang et al., 2012; Liu et al., 2015) or contextual features (Zhao et al., 2015; Zubiaga et al., 2017) extracted from claim and related posts from social media. Nowadays, most methods predominately focus on utilizing neural networks such as RNN (Ma et al., 2016), CNNs (Yu et al., 2017), tree-/graph-based (Lu and Li, 2020; Ma et al., 2020; Rosenfeld et al., 2020; Lin et al., 2021), transformer-based (Khoo et al., 2020; Ma and Gao, 2020; Yu et al., 2020) and adversarial contrastive (Ma et al., 2021; Lin et al., 2022) models. Researchers also find that stances towards the specific claim shared among social media users, can assist rumor verification by revealing crucial cues and dissemination patterns (Qazvinian et al., 2011; Zhao et al., 2015; Zubiaga et al., 2016b), leading to more recent stance-aware approaches for rumor verification (Dungs et al., 2018; Ma et al., 2020; Li et al., 2020; Tian et al., 2020; Yuan et al., 2021).
Recently, research has leveraged pre-trained language models (PLMs) including LLMs for misinformation-related tasks such as fact checking (Lee et al., 2021; Pan et al., 2023; Zeng and Gao, 2023; Zhang and Gao, 2023, 2024). Lin et al. (2023) proposed zero-shot prompt learning for rumor detection using a multilingual PLM addressing diverse languages and domains on social media. Little has been done on using LLMs for rumor stance detection and verification in social media contexts.
Stance Detection. Stance detection, initially relies on hand-crafted features (Lukasik et al., 2016; Zubiaga et al., 2018), and later has advanced to use
deep learning (Augenstein et al., 2016; Zhang et al., 2019; Liang et al., 2021). Subsequent research has explored incorporating propagation structure (Zubiaga et al., 2016a; Kochkina et al., 2017). Recent approaches delve into reinforcement learning (Wei et al., 2019a), contrastive learning (Liang et al., 2022) and teacher-student models (Li et al., 2023). Yet these methods require large annotated corpora for training. To address this limitation, unsupervised (Kobbe et al., 2020; Allaway and McKeown, 2020a; Pick et al., 2022; Ran and Jia, 2023) and weakly supervised model (Yang et al., 2022) have emerged, albeit with concerns about their weak detection efficacy and generalizability. PLMs elevate stance detection by utilizing variants of BERT (Devlin et al., 2019), setting new standards this task (Allaway and McKeown, 2020b; Li et al., 2021a). ChatGPT demonstrates its accuracy in stance detection (Aiyappa et al., 2023) via zero-shot and few-shot prompt engineering.
Rumor-Stance Dual Task. The rumor-stance dual task commenced from RumorEval shared task series (Derczynski et al., 2017; Gorrell et al., 2019) as a two-step pipeline, where rumor verification (subtask B) performs veracity prediction based on the claim and SDQC stances of the posts classified in stance detection (subtask A). Then, joint detection has been studied through multi-task learning (Ma et al., 2018; Kochkina et al., 2018; Wei et al., 2019b; Li et al., 2019; Yu et al., 2020). However, such approach is fully supervised by large training sets labeled for claim veracity and posts stance. Yang et al. (2022) proposed a weakly supervised neural model with multiple instance learning only using a full set of veracity-labeled claims for training. We assume only a small set of training examples at claim level as seeds for LLM-based annotation, and learn a policy to select high-quality annotations for fine-tuning SD and RV LLMs.

heading: 3 Problem Statement
text: We define a rumor dataset as C = {(ci, Xi)}|C|i=1, where each instance (ci, Xi) is a tuple consisting of a source claim ci and a conversation thread of posts responding to ci denoted as Xi = {xi,1, xi,2, · · · , xi,T }. The posts are presented in a chronological order while reply structure may exist via ‘@user’ symbol in the text. We define the dual tasks as follows:
• Stance Detection: The task is to determine the stance yi,j for each post xi,j ∈ Xi un-
der claim ci. That is, f : xi,1xi,2 · · ·xi,T → yi,1yi,2 · · · yi,T , where yi,j takes one of the Support (S), Deny (D), Question (Q) or Comment (C) stance labels.
• Rumor Verification: The task is to classify each claim ci together with the responding posts into one of the four veracity classes Yi: Non-Rumor (N), True Rumor (T), False Rumor (F), or Unverified Rumor (U). That is, g : (ci, Xi)→ Yi.
Traditionally, the ground-truth of yi,1yi,2 · · · yi,T and Yi of all training instances are assumed available for full supervision (Ma et al., 2018; Kochkina et al., 2018; Wei et al., 2019b; Li et al., 2019; Yu et al., 2020), or only Yi of each training instance is available for weak supervision (Yang et al., 2022). In contrast, we target a more challenging setting, where only a small set of seeding claims C′ ∈ C (|C′|≪ |C|) are provided with veracity labels, while no post stance is provided for training.

heading: 4 Our JSDRV Framework
text: Using zero-/few-shot prompting (Radford et al., 2019; Brown et al., 2020) and parameter-efficient fine-tuning (PEFT) (Hu et al., 2021; Houlsby et al., 2019; Lester et al., 2021) techniques, LLMs can perform on a par with or even better than traditional supervised models. However, using LLMs for the stance-rumor joint tasks faces two major issues: 1) Human labels, especially stance labels of posts in social conversation about a claim, are difficult to obtain. Even if they were abundant, it could be hard to fully utilize them for fine-tuning due to the limit of computing resources; 2) While one could use LLMs to annotate data in scale, LLM-provided labels may be unreliable, subject to further refinement for quality. Thus, how to label and select high-quality instances becomes paramount for effective prompting or fine-tuning.
We propose a reinforcement tuning framework to perform data annotation, selection, and model fine-tuning for the dual tasks, based on a small set of seeding claims with veracity labeled manually. By iteratively refining the selection policy with LLM’s annotations, the model prioritizes instances that align with the desired task objectives. With heterogeneous reward functions based on limited ground-truth data, JSDRV enhances the instance selection process, ultimately boosting the overall performance of both tasks.
SMU Classification: Restricted
Figure 2 illustrates our end-to-end framework, encompassing an SD network, a reinforcement selection policy model, and an RV network. We adopt Llama-2 7B version2 (Touvron et al., 2023), an open-source LLM for both SD and RV networks, which can be fine-tuned with PEFT techniques such as LoRA (Hu et al., 2021).

heading: 4.1 LLM Stance Detection Network
text: The purpose of the LLM-based SD network is to provide a stance label to any post in Xt of a given claim ct. Since the size of Xt can be very large, it is neither necessary nor feasible to use all the posts due to the potential high costs of interaction with LLMs or fine-tuning. We use an ϵ-greedy strategy to pre-select a subset of posts for LLM to annotate before learning our selector policy model (§ 4.3), aiming to facilitate the continuous iterative optimization of the stance LLM.
ϵ-greedy Controller for Post Pre-Selection We propose an ϵ greedy controller to judiciously regulate the quantity of posts submitted for LLM prediction, thereby balancing the exploitation, i.e., choosing posts from Xt by following the time order of posts, and exploration, i.e., selecting posts in Xt randomly. The intuition is that the model strikes to balance the earliness and time span for getting useful posts. Thus, a post x̃t,t′ is sampled at any
2While we opt to host local, open-sourced LLMs, our framework is general and can accommodate closed LLMs like GPT-3.5/4 from OpenAI.
step t′ based on the following trade-off:
x̃t,t′ ∼ {
Next(Xt) with prob ϵ Uniform(Xt) with prob 1- ϵ
(1)
where Xt is a temporally ordered list of posts as aforementioned, Next(.) is a function of choosing the next post in the list, and Uniform(.) is a uniform distribution over all posts in Xt. At each step t′, we sample a post without replacement from one of the two distributions and feed it into SD LLM for prompting and label generation.
Stance Prompt Learning We design a prompt as the instruction to make the SD LLM generate a stance label and a brief explanation for each sampled post, following a required format. An example prompt is shown in Figure 4 in Appendix A.1.
Pretraining For a better initial quality of labeling, we pre-train the SD LLM with P-stance dataset3 (Li et al., 2021b). The objective is to minimize the negative conditional language modeling for generating correct stance labels.

heading: 4.2 LLM Rumor Verification Network
text: Posts after pre-selection and annotation by SD network will be further selected by the policy model (§ 4.3) for determining whether they should be retained or discarded, considering how useful they
3P-stance is a general stance dataset with 3-category stances. We find it still helps SD LLM in 4-class stance classification in our task.
are to claim veracity prediction by the RV network. The RV network also needs to augment labeled instances at claim level, for which a pre-selection of claims is conducted using ϵ-greedy method. Since we have some human labeled claims, the sampling of a claim c̃t at step t strives to balance exploiting human labeled claims and exploring unlabeled claims both uniformly, which yields:
c̃t ∼ {
Uniform(C′) with prob ϵ Uniform(C − C′) with prob 1- ϵ (2)
where C′ denotes the labeled claim set. Given each pre-selected claim and its related posts that are retained by the selection policy model, RV LLM network is then prompted to generate veracity label for the claim and a brief explanation of the decision, considering the claim and posts content and stance. An example prompt is shown in Appendix A.2.
Pretraining Similarly, we also pretrain the RV LLM for better initial quality using the small manually labeled claim set C′. Without post stance labels, we just feed posts content with the claim into the RV LLM, which is trained to minimize the negative likelihood of predicted labels and ground truth.

heading: 4.3 Selector Policy
text: We design a selector policy to transform input states, i.e., annotated instances that are pre-selected, to their corresponding actions, i.e., decisions to discard or retain an instance. For optimizing the selector’s policy πθ with parameter θ, each step corresponds to sampling an annotated claim followed by a sequence of sub-steps sampling annotated posts, and receiving rewards from the RV network.
We formulate a two-level Markov Decision Process (MDP) with the following elements: (1) {St} and {st,t′} correspond to a sequence of states at claim and post level, respectively; (2) {At} and {at,t′} correspond to a sequence of actions for sampled claims and posts, respectively, for deciding whether to keep the current instance; (3) {Rt} and {rt,t′} are rewards received after taking an action for the respective level. This reward incorporates predictions based on instances with and without human labels. For easing presentation, we will use a unified notation to describe these elements at both levels, that is, ς , α, γ, and τ denote state, action, reward, and time step, respectively, but note that in practice each of them is separated into two versions corresponding to claim and post as mentioned.
State State ςτ denotes the current status at step τ after the previous instances, i.e., claims or posts, sampled up to τ − 1. It contains the representations of three parts: the current claim c̃τ , the selected instances thus far, and the prediction explanation for the current instance. This yields ςτ = [c̃τ ,Cτ−1,Eτ ], where c̃τ , Cτ−1 and Eτ denote the embedding of c̃τ , the context by averaging the embeddings of selected instances up to τ − 1, and the embedding of explanation for the current instance, respectively. The embeddings are obtained through RoBERTa (Liu et al., 2019).
Action An action ατ is sampled from πθ stochastically given the state ςτ . Specifically, the policy network will output a probability distribution over action space {discard, retain}, which yields:
πθ(ατ , ςτ ) = σ(w2 · ReLU(w1 · ςτ )) (3)
where θ = {w1, w2} are the weights of policy network and σ is the sigmoid activation function. Then, the action ατ is sampled according to the output probability: ατ ∼ πθ(ατ , ςτ ). Reward For a selected claim with human label, the reward considers congruence between the RV network prediction and its ground truth, since the contribution of each selected post under the claim can be reflected by claim veracity prediction; For a selected claim without ground truth, the reward considers how well the stance distribution of its sampled posts conforms to the posts stance distribution of those claims in C′ that have the same veracity label as the predicted label of the selected claim. This yields:
γτ = { E(cos(Ŷc̃τ ,Yc̃τ )) if c̃τ ∈ C′ E(cos(ŷc̃τ ,x̃τ,∗ , ŷc∈C′,xc,∗ |Ŷc̃τ = Yc)) otherwise
(4)
Here, Ŷc̃τ and Yc̃τ are respectively the predicted and ground-truth veracity distributions of sampled claim c̃τ ; ŷc̃τ ,x̃τ,∗ is the mean of stance distributions predicted on the selected posts x̃τ,∗ under c̃τ ; ŷc∈C′,xc,∗ is the mean of stance distributions predicted on all the posts xc,∗ over all the claims c ∈ C′, of which the veracity label is same as the predicted veracity of c̃τ ; and E(.) is a sign function that turns the cosine similarity of two distributions to -1, 0 or 1 depending on the sign of similarity. The reward encourages the model to retain the instances (i.e., claims and posts) that can help veracity prediction keep close to the human-labeled claims providing similar stance distribution.

heading: 4.4 Model Training
text: Our training is an end-to-end joint optimization process, which involves alternating training on policy network and LLM-based SD and RV networks in each epoch.
Policy Network We employ the widely used offline optimization method (Sutton and Barto, 2018) to maximize expected accumulative rewardRt:
(5)
Rt = 1
t
t∑
i=1
 Ri log(πθ(Ai, Si))
+ 1
t′
t′∑
j=1
ri,j log(πθ(ai,j , si,j))
 
where Ri and ri,j are calculated by Equation 4, and t and t′ respectively denote the current time step at claim and post levels. The policy network is updated after a claim and its posts are selected.
LLMs We fine-tune the LLM-based SD and RV networks using the annotations (human or machine labeled) of selected instances by minimizing standard Negative Log Likelihood loss in language model training (Kanamori, 2010). In each epoch, the two LLMs are fine-tuned only once following the selection process. Note that we can skip fine-tuning them but use the selected instances for few-shot in-context learning.
Training Procedure The training detail is depicted as Algorithm 1. JSDRV is trained with an end-to-end fashion by optimizing the two-level selector policy model for claim selection and post selection and the SD and RV LLMs.
Termination condition The ϵ greedy process can terminate automatically. For the termination condition, we utilize the reward function described in Equation 4. The selection process stops if the model receives reward γτ = 1 for N continuous steps, for which we set N as 100 tuned on the validation set.

heading: 5 Experiments and Results

heading: 5.1 Datasets and Setup
text: Datasets For model training, we utilize three public rumor verification benchmark datasets, Twitter15/16 (T15/16) (Ma et al., 2017) and PHEME
(PH)4, where only claim veracity labels are provided. Since both stance label and rumor veracity are required for testing, we resort to RumorEvalS5 (Yang et al., 2022) and SemEval-8 (Derczynski et al., 2017) datasets with labeled claim veracity and posts stance. Details of the datasets are provided in Table 4 and Table 5 in Appendix B.
Setup We hold out 20% instances of test sets as validation sets (Val), for training models that require stance labels and tuning hyper-parameters. We use micro-averaged, macro-averaged F1 score, and class-specific F-measure as evaluation metrics considering imbalanced class distributions (Zubiaga et al., 2016a). We use public codes for baselines or re-implement them if codes are not released. We use 50% claims in the training set as seeding claims for JSDRV for the main results and examine how the percentage affects its performance (see Appendix E.2). We set ϵ = 0.3 with validation data. The detail of other hyper-parameters setup is provided in Appendix C.

heading: 5.2 Stance Detection Performance
text: Since our SD network does not need post stance label, we choose the following unsupervised, supervised, and weakly supervised baselines, which are detailed in Appendix D.1: (1) TGA (Allaway and McKeown, 2020a); (2) BerTweet (Nguyen et al., 2020); (3) Llama 2-ST (Touvron et al., 2023); (4) Llama 2-MT (Touvron et al., 2023); (5) BiGRU (Augenstein et al., 2016); (6) BrLSTM (Kochkina et al., 2017); (7) MT-GRU (Ma et al., 2018); (8) JointCL (Liang et al., 2022); (9) SRLF (Yuan et al., 2021); (10) TD-MIL (Yang et al., 2022). We use model (DATASET) to denote model trained on DATASET6.
For the stance detection baselines, we use the original source code of TGA. The first group refers to unsupervised baselines, while BiGRU, BrLSTM, MT-GRU, and JointCL in the second group are four popular supervised stance detection baselines. SRLF and TD-MIL are weakly supervised models that do not need stance annotation. We use the validation dataset to train the JSDRV (Val) variant for fair comparison with the supervised models. This is because there is no stance annotations in the training set.
4https://figshare.com/articles/PHEME_dataset_ of_rumours_and_non-rumours/4010619.
5https://github.com/2302Jerry/Data-Repo 6We also train JSDRV on the validation set for fairly com-
pared with supervised methods that need stance labels.
Algorithm 1 The JSDRV Model Training 1: Input: A rumor dataset C, seeding claims C′ ∈ C, P-stance dataset, SD network, RV network, ϵ, πθ
policy network. 2: Pretrain SD network with P-stance dataset according to § 4.1, and RV network with seeding claims
according to § 4.2. 3: for t← 1 to |C| do 4: if termination condition meets then 5: break. 6: end if 7: Sample a claim c̃t according to Equation 2. 8: for t′ ← 1 to |Xc̃t | do 9: if termination condition meets then
From Table 1, we observe that: 1) In zero-shot models, TGA performs worst as it is pretrained on
specific topics and cannot generalize well to Twitter data; BerTweet, which is fine-tuned on enormous
Twitter datasets, outperforms TGA; Llama 2-ST and -MT outperform BerTweet, indicating that the LLM has promising zero-shot capability to detect stances. 2) In fully supervised baselines, Bi-GRU based on a sequential architecture performs worst; BrLSTM, benefiting from propagation structure, makes improvements, but is unable to detect deny stance as it is sensitive to such infrequent class in the training data; JointCL leveraging both contextand target-aware features outperforms MT-GRU which is sequential. 3) While both TD-MIL and SRL are weakly supervised only using claim labels, TD-MIL benefits from propagation information while SRLF cannot. 4) Trained on the validation set, JSDRV (Val) and its BERT-based variant are comparable to TD-MIL trained on the full training set, indicating JSDRV is less demanding on labeled data.
JSDRV outperforms all the baselines on the corresponding datasets. When LLM is replaced by BERT in JSDRV, there is a performance drop but JSDRV-Bert is still superior to baselines, suggesting it can be generalized to non-LLM task models. JSDRV (PH) performs the best, beating its counterparts trained on T15/16 datasets with large margins due to the much larger size of PH dataset.

heading: 5.3 Rumor Verification Performance
text: We compare to unsupervised, supervised, weakly supervised, and multi-task rumor verification baselines: (1) BerTweet (Nguyen et al., 2020); (2) Llama 2-ST (Touvron et al., 2023); (3) Llama 2-MT (Touvron et al., 2023); (4) GCAN (Lu and Li, 2020); (5) TD-RvNN (Ma et al., 2020); (6) PLAN (Khoo et al., 2020); (7) DDGCN (Sun et al., 2022); (8) SRLF (Yuan et al., 2021); (9) TD-MIL (Yang et al., 2022); (10) MTL2 (Kochkina et al., 2018); (11) MT-GRU (Ma et al., 2018). The details are depicted in Appendix D.2. In Table 2, we report the best rumor verification results obtained across different training and test datasets. MT-GRU and MTL2 require both rumor and stance labels for training. So, we train them on the validation set, which has both rumor and stance labels. We also compare JSDRV (Val) with MT-GRU and MTL2.
From Table 2, we observe a similar trend as the stance detection task. While Llama 2-ST and -MT outperform BerTweet, their performance is still on a par, indicating directly prompting Llama 2 to perform both tasks together is similar to doing them separately. Among supervised baselines,
GCAN performs worst because it only considers local structure while structure-aware models such as TD-RvNN, PLAN and DDGCN appear much better. MTL2 and MT-GRU are multi-task frameworks that require both labels, so we trained them on validation sets. JSDRV (Val), trained on the same dataset only using a small set of veracitylabeled claims, outperforms MT-GRU on MacF and class-level F1, indicating that JSDRV is effective in multi-task learning.
Similarly, JSDRV outperforms all the baselines on the corresponding datasets for the RV task, and is still well generalized to the non-LLM-based task models with the LLMs replaced by BERT. JSDRV (PH) outperforms all the baselines regardless of datasets used and gets 3.2%/17.1% MicF/MacF improvement over the best baseline TD-MIL (PH) on RumorEval-S.

heading: 5.4 Ablation Study
text: For the ablation, we separate the components in JSDRV. (RSS): reinforced stance selector; (RVS): reinforced veracity selector; (FTSD): fine-tuning SD; (FTRV): fine-tuning RV; (PTSD): pretraining SD; (PTRV): pretraining RV; (epSD): ϵ greedy control for SD; (epRV): ϵ greedy control for RV.
Table 3 shows that except removing epSD and epRV, removing other components decreases the performance, indicating they are useful and the pre-selection with ϵ greedy method can maintain performance with a little cost. Removing RSS and RVS drops the most, meaning that the reinforced selector is the most vial. Removing FTRV and FTSD drops more than removing PTSD and PTRV respectively, indicating that joint fine-tuning is more important.

heading: 5.5 Analysis
text: We plot posts with their stances on RumorEval-S dataset with tSNE (Van der Maaten and Hinton, 2008) to show the effect of post stance selection. We input each post as "[CLS] post [SEP] stance [SEP] reason" into RoBERTa-base and take [CLS] token representations to plot Figure 3. We observe that the selected stances are separated much better, indicating that JSDRV can differentiate stances and may help rumor verification.
We also plot example outputs of JSDRV in Figure 6 in Appendix E.1 and find that the selector policy can choose posts with high-quality stances annotated by stance LLM, which provides useful
cues to rumor verification. More analyses on the impact of seeding claims and ϵ are in Appendix E.2.

heading: 6 Conclusion and Future Work
text: We introduced a novel reinforcement tuning framework, JSDRV, for joint stance detection and rumor verification tasks based on LLMs. By leveraging LLMs as annotators and employing a reinforcement label selector, we effectively fine-tune the LLMs for both tasks supervised only with minimal human-labeled claims. Our experiments on benchmark datasets demonstrated the superiority of JSDRV over existing methods, suggesting its promising potential for addressing the challenges of joint stance detection and rumor verification in social media environments. Our future work aims to develop unsupervised methods for both tasks.
Limitations
JSDRV relies on the SD and RV networks to provide class probability distributions for calculating rewards. While some LLMs like Llama 2 can return word distribution, converting it to the distribution of a limited number of classes may compromise accuracy. Moreover, LLMs like ChatGPT, which do not return such distributions, could hinder the applicability and practical use of our method.
In JSDRV, frequent interactions with SD and RV networks, particularly when they rely on closed LLM services, can introduce overhead during train-
ing. This overhead may impact training efficiency. Despite demanding much less labeled data, JSDRV necessitates human-labeled claims, posing challenges in obtaining them, especially in certain domains such as healthcare and science. This constraint may limit its applicability in scenarios where labeled claims are scarce. Therefore, our future work aims to develop unsupervised methods for both tasks.

heading: Acknowledgements
text: This work is partially supported by National Natural Science Foundation of China Young Scientists Fund (No. 62206233) and Hong Kong RGC ECS (No. 22200722).
Ethical Considerations
While our research utilizes publicly accessible datasets, using social media conversations for debunking rumors and detecting user stances may raise privacy concerns. While our approach does not necessitate access to sensitive user data, we anonymized all social media posts, ensuring user information is invisible and unusable by others.
Both tasks studied in this paper hold social implications. Key considerations include system reliability and the potential for mislabeling information and misleading users. To address this, we will establish responsible policies for code dissemination, aligning with ethical standards.

heading: A Prompt Design
text: A.1 Stance Detection Prompt
For stance labeling, we use the following prompt: There is a claim [CLAIM] and I will give you its corre-
sponding conversation thread on Twitter. The conversation
thread consists of a sequence of posts and each post [POST]
is written by a user [USER]. Please decide the stance ex-
pressed by each post towards the claim and explain why, and
the stance can be one of the following labels: Support, Deny,
Question, and Comment. Please follow the format: Stance:
[stance], Reason:[reason].
The SD LLM will output a stance label and an explanation of labeling rationale for each input post. We show the details of stance detection prompt in Figure 4. The blue box represents all the context, the text in the orange box represents the stance and reason output from the stance LLM.
A.2 Rumor Verification Prompt
Given a claim and its related posts that are retained by the selection policy model, RV LLM network is prompted to generate veracity label for the claim and a brief explanation of the decision, considering the claim and posts content and stance. The prompt is constructed as follows:
There is a claim [CLAIM], I will give you its related posts,
each expressing a stance toward this claim. Please deter-
mine the veracity of the claim, categorizing it as ’True Rumor,’
’False Rumor,’ ’Unverified Rumor,’ or ’Non-Rumor,’ and ex-
plain your reasoning. Please follows the format: Veracity:
[veracity], Reason: [reason].
The RV LLM will output a veracity label and an explanation of labeling rationale for any input claim. We show the details of claim veracity classification prompt in Figure 5. The blue box represents all the context, the text in the orange box represents the outputs of claim veracity and the reason.

heading: B Dataset Statistics
text: We provide the statistics of training and testing dataset in Table 4 and Table 5, respectively.

heading: C Hyper-parameter Setting
text: As for Llama 2 model, We download the version with 7 billion parameters(Llama-2-7b) on August, 20, 2023 to obtain its responses. we then employ
HuggingFace AutoTokenizer7 and classes to run the model. The following arguments are setup during fine-tuning stage:
“model_name": “Llama-2-7b"; “learning_rate": 1e-4; “num_train_epochs": 6; “max_seq_length": 4096; “load_in_4bit": True; “lr_scheduler_type": “linear"; “temperature": 0 As for selector policy model, we set the hyper-
parameters as the following: “learning rate": 5e-5 “optimizer": Adam “warm-up rate": 0.1 “batch size": 4 “maximum epoch": 50

heading: D Details of Baseline Models
text: D.1 Stance Detection Baselines
Since our SD network only requires claim veracity label but not post stance label, we choose unsupervised, supervised, and weakly supervised baselines: (1) TGA (Allaway and McKeown, 2020a): A topicgrouped attention network for zero-shot stance detection. (2) BerTweet (Nguyen et al., 2020): A language model pre-trained on 850M tweets, which is fine-tuned on validataion dataset to adapt to stance detection task. (3) Llama 2-ST (Touvron et al., 2023): A pre-trained large language model developed by Meta for only stance detection task. (4) Llama 2-MT (Touvron et al., 2023): A pre-trained
7https://huggingface.co/docs/transformers/ v4.33.0/en/model_doc/auto#transformers. AutoTokenizer
large language model prompted to perform multitask of stance detection and rumor verification together. (5) BiGRU (Augenstein et al., 2016): A bidirectional GRU-based stance detection model. (6) BrLSTM (Kochkina et al., 2017): An LSTMbased model that models the structured conversational thread to detect stance. (7) MT-GRU (Ma et al., 2018): A RNN-based multi-task learning model to jointly detect rumors and stances. (8) JointCL (Liang et al., 2022): A zero-shot stance detection model based on contrastive learning. (9) SRLF (Yuan et al., 2021): A stance-aware reinforced framework for stance detection.8 (10) TDMIL (Yang et al., 2022): A weakly supervised model for stance detection and rumor verification based on top-down tree structure. Additionally, JSDRV (DATASET) is our LLM-based reinforcement tuning method applied for stance detection.
D.2 Rumor Verification Baselines
We collect unsupervised, supervised, weakly supervised baselines for rumor verification. Since multi-task learning also can enhance rumor detection, we also introduce two multi-task baselines: (1) BerTweet (Nguyen et al., 2020): A pre-trained language model with 850M tweets, and we finetune it for rumor verification here. (2) Llama 2- ST (Touvron et al., 2023): A large language model pre-trained on various domains myriad corpus, we use it for single rumor verification task. (3) Llama 2-MT (Touvron et al., 2023): A pre-trained large language model prompted to perform multi-task of stance detection and rumor verification together. (4) GCAN (Lu and Li, 2020): A graph-aware coattention model utilizing retweet to detect rumor veracity. (5) TD-RvNN (Ma et al., 2020): A topdown tree-structured attention networks for rumor verification. (6) PLAN (Khoo et al., 2020): A transformer based rumor verification model utilizing interactions between users. (7) DDGCN (Sun et al., 2022): A rumor verification model utilizing the dynamic propagation texts and external knowledge. (8) SRLF (Yuan et al., 2021): A stance-aware refinforced framework for rumor verification. (9) TD-MIL (Yang et al., 2022): A top-down tree propagation model for joint detection tasks on stance type and rumor veracity. (10) MTL2 (Kochkina et al., 2018): A sequential model utilizing a number of task-specific layers for stance detection and ru-
8The original method just outputs rumor veracity result, we put a Softmax layer behind stance embeddings to get stance output here.
mor verification. (11) MT-GRU (Ma et al., 2018): A multi-task learning approach to jointly detect rumors and stances by capturing both shared and taskspecific features. Additionally, JSDRV(DATASET) is our reinforcement tuning method applied for rumor verification.

heading: E Analysis
text: E.1 Case Study
We also plot example outputs of JSDRV in Figure 6, where the red line represents the stance detection process, the gray texts denote discarded posts, and the blue line indicates the rumor verification process during inference. We observe that the selector policy can choose posts with high-quality stances annotated by stance LLM, which provides useful cues to rumor verification LLM.
E.2 Sensitivity Study
We conduct a sensitivity study to see the impact of the size of seeding claims. Using JSDRV (PH) model, we show the variation of micF score on stance detection and rumor verification tasks with the increase of the proportion of seeding claims. We also show the performance of the strongest baseline TD-MIL trained on the same sets of seeding claims. Figure 7 indicates that (1) With more seeding claims, the performance of TD-MIL (PH), JSDRV (PH), and RSDRV-Bert (PH) all gets improved; (2) The performance of JSDRV tends to stabilize more quickly than TD-MIL. When the ratio reaches 50%, the performance of JSDRV becomes
saturated at a much higher level, while TD-MIL cannot catch up even using up all the training data. (3) JSDRV performs much better than TD-MIL using a same proportion of training data.
To get an intuitive understanding of the greedy controller, we conduct a study to assess the performance of JSDRV-Bert (PH) and JSDRV (PH) across varying ϵ rates. We use the same ϵ to control both post and claim sampling. The MicF scores for stance detection and rumor verification on the RumorEval-S dataset are shown in Figure 8. Our observations include: (1) The performance of both models improves as ϵ increases. (2) Model performance stabilizes as ϵ reaches 0.3, indicating that our greedy controller can achieve comparable results with limited instances labels compared to abundant instances labels.
E.3 User Study
We conduct a user study to evaluate the quality of the model output. We sample 120 samples from RumorEval-S and present them in two forms: Baseline (claim, posts) and JSDRV (claim, selected poststance pairs, reasons). We then ask 6 users to label the articles and give their confidence in a 5-point Likert Scale (Joshi et al., 2015), and each person is
given only one form to avoid cross influence.
Table 6 shows that 1) users determine the rumors more accurately with JSDRV, 2) users spent 75% less time identifying rumors, and 3) users show higher confidence with the results of JSDRV, suggesting that users tend to be more sure about their decision when stance and related reasons have been provided.

heading: Future Work
text: Our future work aims to develop unsupervised methods for both tasks. Limitations
JSDRV relies on the SD and RV networks to provide class probability distributions for calculating rewards. While some LLMs like Llama 2 can return word distribution, converting it to the distribution of a limited number of classes may compromise accuracy. Moreover, LLMs like ChatGPT, which do not return such distributions, could hinder the applicability and practical use of our method. In JSDRV, frequent interactions with SD and RV networks, particularly when they rely on closed LLM services, can introduce overhead during train-
ing. This overhead may impact training efficiency. Despite demanding much less labeled data, JSDRV necessitates human-labeled claims, posing challenges in obtaining them, especially in certain domains such as healthcare and science. This constraint may limit its applicability in scenarios where labeled claims are scarce. Therefore, our future work aims to develop unsupervised methods for both tasks