Faking Fake News for Real Fake News Detection: Propaganda-Loaded Training Data Generation
Year: 2023

Abstract:
Despite recent advances in detecting fake news generated by neural models, their results are not readily applicable to effective detection of human-written disinformation. What limits the successful transfer between them is the sizable gap between machine-generated fake news and human-authored ones, including the notable differences in terms of style and underlying intent. With this in mind, we propose a novel framework for generating training examples that are informed by the known styles and strategies of human-authored propaganda. Specifically, we perform self-critical sequence training guided by natural language inference to ensure the validity of the generated articles, while also incorporating propaganda techniques, such as appeal to authority and loaded language. In particular, we create a new training dataset, PROPANEWS, with 2,256 examples, which we release for future use. Our experimental results show that fake news detectors trained on PROPANEWS are better at detecting human-written disinformation by 3.62– 7.69% F1 score on two public datasets.

heading: 1 Introduction
text: The dissemination of false information online can cause chaos, hatred, and trust issues, and can eventually hinder the development of society as a whole (Dewatana and Adillah, 2021; Wasserman and Madrid-Morales, 2019). In particular, humanwritten disinformation2 is often used to manipulate certain populations and reportedly already had a catastrophic impact on multiple events, such as Brexit (Bastos and Mercea, 2019), the COVID-19 pandemic (van Der Linden et al., 2020), and the 2022 Russian assault on Ukraine.
1The code and data are released on GitHub: https:// github.com/khuangaf/FakingFakeNews
2There are many types and definitions of fake news, but here we focus on text-only disinformation. Yet, we will also use the less accurate term fake news as it is more common.
Hence, there is an urgent need for a defense mechanism against human-written disinformation.3 For this, we need a substantial amount of training data to build detectors. A naïve solution is to collect human-written news articles that contain inaccurate information by crawling untrustworthy news media. However, news articles published by suspicious sources do not necessarily contain false information, which means that annotators are required to fact-check every claim in each untrustworthy article. Moreover, articles containing false claims are often removed shortly after posting. While some work collected human-written fake news from factchecking websites (Shu et al., 2018; Nguyen et al., 2020), the size of these datasets is limited. The curation process of these websites also requires a lot of manual efforts. Hence, such a solution is neither scalable nor reliable. Thus, an alternative direction complementing the existing efforts would be to generate training data automatically in a way that avoids these issues.
Our goal is to enhance disinformation detection by generating training examples that are better informed by the known styles and strategies of human-authored disinformation. We started by collecting human-written articles from untrustworthy sites4, and we analyzed around 40 of them that spread false claims. Throughout our analysis, we found two characteristics of this human-written disinformation. First, about 33% of the articles used propaganda techniques to convince the audience that the fake information was actually authentic, and these techniques often involve the use of emotion-triggering language or logical fallacies (Da San Martino et al., 2019) to increase the impact on the reader. Statistics about the propaganda techniques are given in Appendix A.
3WARNING: This paper contains disinformation that may be sensitive or offensive in nature.
4These news sources are rated low for their factuality of reporting by mediabiasfactcheck.com.
14571
salient sentence, which it then replaces with a plausible but disinformative sentence that is coherent to the context. Finally, it generates a propaganda sentence to make the article resemble human-written fake news.
Second, more than 55% of the articles that we analyzed contained inaccurate information mixed with correct information: in fact, all claims, except for one or two, in these disinformation articles were factual, which makes the few false claims in these articles even more believable.
Prior work has made significant progress in generating fake news using large pre-trained sequenceto-sequence (seq2seq) models (Zellers et al., 2019; Fung et al., 2021; Shu et al., 2021). However, the articles generated by these approaches contain an overwhelmingly large proportion of false information and do not explicitly use propaganda.
To address these issues, here we propose a novel generation method. Given an authentic news article, we replace a salient sentence with a plausible but fake piece of information using a seq2seq model. As the generated texts can often be entailed by the original contexts, we incorporate a self-critical sequence training objective (Rennie et al., 2017) that incorporates a natural language inference (NLI) model into the loss function. Additionally, we use the NLI model to filter out generated sentences that can be inferred from the replaced ones. Then, we add propaganda techniques to mimic how humans craft disinformation. In particular, we automate two commonly used propaganda techniques, appeal to authority and loaded language, (Da San Martino et al., 2019) to add propaganda into the faked sentences.
Subsequently, we use the silver-standard training data generated from these two steps to train a detector. An example is shown in Table 1. We further recruited crowdsourcing workers to validate that some generated texts were indeed fake, so that we could construct a gold-standard training dataset.
Comparing our method to state-of-the-art fake news generation approaches, the evaluation results on two human-written fake news datasets show that detectors are substantially better at spotting human-written disinformation when trained on our generated fake news dataset.
Our ablation studies confirm the effectiveness of incorporating propaganda into the generated articles for producing better training data.
Our contributions can be summarized as follows: • We propose an effective method to automat-
ically generate more realistic disinformation compared to previous work. • We develop the first automatic methods to generate specific propaganda techniques such that the generated articles are closer to disinformation written by humans. • We demonstrate that detectors trained on our generated data, compared to generated articles using other methods, are better at detecting human-written disinformation. • We release PROPANEWS, a dataset for disinformation detection containing 2.2K articles generated by our approach and validated by humans.

heading: 2 Training Data Generation
text: Our process of generating training data for propaganda-loaded disinformation consists of two main steps: disinformation generation (§2.1) and propaganda generation (§2.2). Below, we describe each of these steps in detail.

heading: 2.1 Disinformation Generation
text: Our disinformation generation approach aims at two sub-goals: (i) replacing a salient sentence in the given article with a sequence of generated coherent texts that looks plausible, and (ii) ensuring that the generated information cannot be entailed by the original masked-out sentence; otherwise, the generated texts will not be disinformative. To achieve the first sub-goal, we first identify salient sentences using extractive summarization, and we then perform mask-infilling with BART (Lewis et al., 2020). We achieve the second sub-goal using self-critical sequence training (Rennie et al., 2017) with an NLI component, which we use as a reward function for generation.
Salient Sentence Identification A salient sentence is critical for the overall semantics of the article. When it is manipulated or replaced, the complex events described in the article may be drastically changed. Yet, there is no salient sentence identification dataset publicly available. Motivated by the fact that sentences included in an extractive summary are often of higher importance, we take the scores computed by an extractive summarization model (Liu and Lapata, 2019), which predicts how likely each sentence is to belong to the summary, to estimate its saliency. We found that this yields reasonably good sentence saliency estimation. For each news outlet, we replaced one sentence that had the highest extractive summarization score with our generated disinformation.
Mask Infilling with BART To perform infilling, we took an approach similar to that of Donahue et al. (2020), but we used BART (Lewis et al., 2020). At training time, we randomly masked out a sentence y∗ from a given article x. The bidirectional encoder first produces contextualized representations he = Encoder(x̃) given the article with a masked-out sentence x̃ = x− y∗. Then, the autoregressive decoder learns a maximum likelihood estimation that aims to maximize the probability of generating the next token y∗t at time step t given all tokens in previous time steps {y∗0 , ..., y∗t−1} and the encoder hidden states he by minimizing the negative log probability of generating y∗t as follows:
Lm = − T
∑ t=1 logP (y∗t ∣y∗0 , ..., y∗t−1,he). (1) During inference time, rather than using random masking, x̃ is formed by masking out the sentence with the highest score computed by the extractive summarization model given the original document x, as discussed in the previous paragraph.
Self-critical Sequence Training BART optimized via maximum likelihood estimation alone is capable of generating coherent texts. However, although the generated texts y′ may be very different from the originally masked out sentence y∗, there is no guarantee that y′ contains incorrect information. If the generated texts y′ can be entailed by the masked out sentence y∗, then y′ is actually not disinformative. An example is shown in Figure 2. Here, except for the lack of details, the generated sentence y′ delivers the same message as the masked out sentence y∗. To reduce the probability that y′ can be entailed by y∗, we leverage self-critical sequence training (Rennie et al., 2017; Bosselut et al., 2018) that rewards the model for generating sequences that cannot be entailed by the masked-out sentences. Self-critical sequence training (SCST) is a form of the REINFORCE algorithm (Williams, 1992) that allows direct optimization on non-differentiable functions. Using a baseline output y′′ of the model to normalize the rewards, SCST avoids the challenge of directly estimating the reward signal or estimating normalization (Rennie et al., 2017). Since our goal is to avoid entailment from y∗ to y′, we define the reward as the negative entailment probability computed by a ROBERTA-based (Liu et al., 2019) NLI model
fine-tuned on Multi-NLI (Williams et al., 2018)5,
r(y′) = −Pnli(y∗, y′), (2) where r(y′) is the reward of the sequence sam-
pled from the current policy y′, and Pnli(y∗, y′) is the probability that y∗ entails y′. To generate y′, we use Nucleus Sampling (Holtzman et al., 2020) with p = 0.96, as this sampling method has shown advantages in open-ended generation (Holtzman et al., 2020; Zellers et al., 2019).
We generate the baseline output y′′ using greedy decoding, then obtain the entailment probabilities between y′ and y′′ from the NLI model. We then compute the self-critical sequence training loss:
Ls = −(r(y′) − r(y′′)) T∑ t=1 logP (y′t∣y′0, .., y′t−1,he). (3) Here r(y′′) is a baseline reward, and r(y′)−r(y′′) is a normalized reward. This loss function encourages BART to generate y′ when r(y′) > r(y′′), whereas it suppresses the probability of decoding y ′ when r(y′) < r(y′′). An overview of SCST is shown in Figure 1. The final objective function to minimize is a weighted sum of Equation (1) and Equation (3),
Lfinal = αLm + βLs, (4)
where α and β are the weights for each loss.6
Post-processing To further ensure the quality of the disinformation generated, we reuse the NLI model discussed in the previous paragraph to filter out invalid outputs y′ that can be entailed from the masked-out sentence y∗, as demonstrated in Figure 2. We found that incorporating the SCST loss (Equation (3)) into the training objective successfully reduces the invalid rate from 7.8% to 3.2%.

heading: 2.2 Propaganda Generation
text: After generating inaccurate information, we incorporate propaganda into each generated article. We chose two representative propaganda techniques of each type: emotional versus non-emotional. Loaded language is an emotional technique and it is also by far the most frequent propaganda technique as shown in Table 5 of (Da San Martino et al., 2019) and Table 2 of (Dimitrov et al., 2021).
5We use the fine-tuned NLI model from https:// huggingface.co/roberta-large-mnli. Its accuracy is 90.2% on the dev set of MNLI, which is on par with state-of-the-art methods.
6Empirically, we set α = 1 and β = 0.01.
Based on these two tables, we also see that appeal to authority is among the most frequent nonemotional techniques.
Appeal to Authority Appeal to authority is a propaganda technique that aims to strengthen or to invalidate an argument by referring to a statement made by authorities or experts (Da San Martino et al., 2019). We first collect experts from various domains, such as economics and immunology, from Wikidata.7 In particular, we specify the occupation (P108) of each expert and we filter out entities that were born before 1940 to ensure recency. To consider only impactful entities, we rank all candidates based on the number of corresponding outcoming statements (i.e., connected concepts in Wikidata), inspired by PageRank (Page et al., 1999), and we add the top 100 entities for each occupation into the candidate list Z. Then, we include the person named entities extracted by a name tagger,8 which are more relevant to the local context. This makes sense as we found that more than 73% of the news articles contain authorities. More details on how authority candidates Z are collected can be found in Appendix E.
Once we collect a candidate list Z, we then generate fake arguments made by each zi ∈ Z with the BART model that has already been fine-tuned in §2.1. In particular, a <mask> token is inserted right after the filled-in sentence y′ in the input article to BART so that it knows where to perform infilling. To inform BART that it should generate a statement made by an authority, we prefix the decoder with the template [zi confirmed that “], where zi ∈ Z is the name of the authority.
The prefix ends with an opening quotation mark to indicate that it should be followed by a statement by authority zi. To increase the diversity of the generated statements, we devise a variety of templates, as detailed in Appendix E. Finally, the best sequence s∗ is selected with the lowest perplexity s ∗ = argminsi Perplexity(si), where si denotes the generated sequence using zi as the authority.
Loaded Language Loaded language is another propaganda technique that uses emotion-triggering terms or phrases to influence the opinions of the audience (Da San Martino et al., 2019; Dimitrov et al., 2021). Often, loaded language uses sensational adverbs or adjectives to exaggerate a statement.
7https://query.wikidata.org/ 8https://stanfordnlp.github.io/stanza
Based on this observation, we utilize the propaganda dataset released by Da San Martino et al. (2019) where propaganda techniques are annotated at the fragment level (i.e. span level). The dataset contains 2,547 loaded language instances. Yet, not every instance contains adjectives or adverbs that are emotion-triggering. To create valid training data for loaded language generation, we first use SpaCy to perform part-of-speech tagging and dependency parsing, and then keep the examples where there exists an adverb pointing to a verb or an adjective pointing to a noun through dependency parsing edges. This results in 1,017 samples of valid loaded language instances. Examples of the generated appeal to authority and loaded language are shown in Table 2.
Upon collecting the training data to generate loaded language, we fine-tune another BART on this dataset. Naïvely, we can take the articles with emotion-triggering adverbs or adjectives removed as input to BART and using the original article as the decoding target. However, we found that around 25% of the time BART does not exactly reproduce the unmasked texts due to hallucination. This observation is consistent with Donahue et al. (2020)’s findings. To this end, we propose a twostep generation approach. First, we train BART to insert a <mask> token into the target sentence in the input document marked with special tokens. Then, BART learns to infill the <mask> with an approach similar to what is discussed in §2.1 but without the SCST objective. Empirically, we found that this approach successfully reduces the chance of failure in generating the exact unmasked contexts to around 2%.

heading: 2.3 Intermediate Pre-training
text: As the size of TIMELINE17 (Tran et al., 2013) and the propaganda dataset (Da San Martino et al., 2019) are relatively small, we perform intermediate pre-training (IPT) on the news articles from CNN/DM, a large news summarization dataset (Hermann et al., 2015), for domain adaptation. Details of IPT can be found in Appendix F.

heading: 3 Our PROPANEWS Dataset

heading: 3.1 Data Source
text: When selecting the source of data, we considered two criteria. First, the news articles must have high trustworthiness. This ensures that, except for our manipulated sentences, the rest is genuine. Second, the news events described in the articles must be important. Motivated by these two criteria, we repurposed the TIMELINE17 dataset (Tran et al., 2013) as our source of data. It contains 17 timelines, each of which corresponds to a news event. Each timeline is associated with a series of news articles that span across a wide time span, implying the high importance and impact of these events. Moreover, the articles come from trustworthy media. In total, there are 4,535 news articles in TIMELINE17.

heading: 3.2 Crowdsourcing for Data Curation
text: We use Amazon’s Mechanical Turk (AMT) to verify the quality and the correctness of the generated disinformation. In total, there are around 400 unique crowdsourcing workers contributing to approximately 2,000 Human Intelligence Tasks (HITs). For each HIT, the annotators were tasked to look for supporting evidence from trustworthy news media to determine whether the sentences generated are indeed inaccurate. Only those labeled as inaccurate were included in PROPANEWS, while the accurate counterparts were discarded. Appendix H gives additional details.
To measure the inter-annotator agreement (IAA), we use the Worker Agreement With Aggregate (WAWA) score (Ning et al., 2020; Sheng et al., 2021), which compares each annotator’s answer to the aggregated answer obtained via majority votes and micro-averages the results across all samples.9 The resulting WAWA precision, recall, and F1 are 80.01%, 78.94%, and 79.47%, which indicates moderate to high agreement.
9We did not use other IAA metrics, such as Cohen’s Kappa (Cohen, 1960), as we expect the vast majority of our generated disinformation to be inaccurate. WAWA provides a better approximation for inter-annotator agreement in our scenario.

heading: 4 Disinformation Detection
text: The disinformation detection task challenges detectors to determine whether a given input article contains inaccurate information or not. We experiment on four detectors, including HDSF (Karimi and Tang, 2019), GROVER (Zellers et al., 2019), BERT (Devlin et al., 2019), and ROBERTA (Liu et al., 2019). HDSF leverages the hierarchical structures of discourse-level features, such as dependency trees, to predict the veracity of a news article. GROVER is an unidirectional seq2seq model pre-trained on news documents. We use the discriminative version for detection which is adapted from its generative version by feeding the [CLS] token representations to a multi-layer perceptron. Similarly, BERT and ROBERTA take in the entire article as input and feed the representations of the first token to a classification head to determine the veracity of each article. In addition, all models are optimized using cross entropy. For fair comparison, we set the maximum sequence length to 512 and we use the LARGE variants for all models. More details can be found in Appendix J.

heading: 5 Experiments
text: In our experiments, we aim (1) to analyze the performance of different models on our new PROPANEWS dataset, (2) to examine the effect of various training data sets, and (3) to investigate how much silver-standard data is equivalent to goldstandard data.

heading: 5.1 Data
text: PROPANEWS The PROPANEWS dataset consists of 2,256 distinct articles, with a balanced number of fake and real documents. Within the fake articles, 30% use appeal to authority, another 30% include loaded language, and the remaining 40% simply contains inaccurate information. We split the data into 1,256:500:500 for training, validation, and testing.
Evaluation Data We use two sets of humanwritten articles released by Nguyen et al. (2020) and Shu et al. (2018) to evaluate the effectiveness of our approach. The articles in each dataset are collected from two fact-checking websites, SNOPES and POLITIFACT, respectively. Articles no longer accessible via the given URL are removed. Statistics about both datasets are shown in Appendix I.
Other generated training data We compare PROPANEWS to the following approaches. GROVER-GEN (Zellers et al., 2019) generates headlines which condition on the original body texts, followed by body text generation conditioning on the generated headlines. FACTGEN (Shu et al., 2021) enhances the factual consistency of the generated article with a fact retriever that fetches supporting information from external corpora. FAKEEVENT (Wu et al., 2022) generates sentences sequentially with condition on the manipulated knowledge elements of each sentence. Also, we form the PN-SILVER dataset by resampling our generated data but disregarding the annotator validation. Furthermore, we construct additional training sets by replacing the salient sentence in each article with one sentence generated by each baseline method, as indicated by -1SENT. To ensure fair comparisons, all generators take in the same set of authentic articles as inputs.

heading: 5.2 Results and Discussion
text: Human-written disinformation detection To study the effectiveness of human-written disinformation detection, we train GROVER-LARGE and ROBERTA-LARGE on different training datasets and evaluate them on the SNOPES and POLITIFACT datasets, as shown in Table 3. Both models perform best when trained on PROPANEWS, compared to training on other datasets. Consider ablating human validation, detectors trained on PN-SILVER still outperform their counterparts trained on other datasets. This shows that our generative method produces articles that are more similar to humanwritten disinformation. To further verify this finding, we measure the similarity between articles generated by different approaches and disinformative articles in the POLITIFACT dataset using the MAUVE metric (Pillutla et al., 2021). MAUVE computes the similarity between two text distributions by adding the areas under a divergence curve, and has been shown to produce better approximations than other metrics such as JS divergence (Martins et al., 2020). We find that the MAUVE score with POLITIFACT for PROPANEWS and GROVER-GEN is 17.1% and 13.7%, respectively, suggesting that the generated documents in PROPANEWS are closer to human-written disinformation. These results confirm that the advantage of our generated articles in defending against human-written disinformation is resulted from the closer gap between them.
Comparing each baseline method and its counterpart that only generates one sentence to be substituted for the salient sentence (i.e., -1SENT), we found significant performance drops on GROVERGEN and FACTGEN when only generating one sentence. This is likely caused by the incoherence between the right context and the sentence generated by these approaches due to the left-to-right fashion of text generation. While FAKEEVENT does not see the right context, it additionally conditions on knowledge elements corresponding to the sentence, which discourages it from producing topically irrelevant content and thus does not lead to huge performance drop.
In Table 4, we show two examples of disinformative articles from POLITIFACT where ROBERTA is able to classify them as inaccurate when trained on PN-SILVER, but fails when trained on GROVERGEN. Both articles contain propaganda, which is incorporated into PN-SILVER but not into GROVERGEN. This demonstrates that detectors trained on our generated data perform better at detecting human-written disinformation that has such properties.
Is propaganda generation helpful for disinformation detection? We further conduct an ablation study to analyze the contributions of each propaganda technique. As shown in the bottom of Table 3, both appeal to authority and loaded language prove beneficial in enhancing models’ abilities to detect human-written disinformation.
We can further see in Table 3, when comparing PROPANEWS WITHOUT AA& LL to other generation approaches, that both models trained on our generated data, even without the incorporation of propaganda techniques, still outperform their counterparts trained on other datasets. This illustrates that our generated disinformation texts are closer to news articles written by humans.
How good is the generation quality? To evaluate the quality of our generation approach, we asked Amazon Mechanical Turk (AMT) workers to rate the plausibility of 100 generated articles from PROPANEWS and to determine the degree by which their answer to this question is influenced by the generated propaganda. Each article was rated by three different AMT workers. For comparison, we also asked the AMT workers to rate the plausibility of 100 generated articles from GROVER-GEN. The average plausibility scores for PROPANEWS and GROVER-GEN were 2.25 and 2.15 (out of 3), respectively. indicating that our generation approach has a slight advantage over GROVER-GEN in terms of plausibility. Moreover, among the articles in PROPANEWS that are rated highly plausible, 29.2% of the workers think that the generated propaganda highly affects their response (i.e. rated 3 out of 3) that the generated article is plausible. This demonstrates the effectiveness of our propaganda techniques in increasing the plausibility of generated articles. Survey details and score distributions are discussed in Appendix K.

heading: 6 Related Work
text: Fake News Generation and Detection There has been a focus in prior research on using neural networks to automatically generate fake news as a means of defending against the proliferation of machine-generated fake news. Zellers et al. (2019) pre-trained a generator with the GPT-2 architecture (Radford et al., 2019) on a large-scale news corpus and demonstrated that it was effective in detecting neural fake news. More recently, Fung et al. (2021) improved the controllability of the generated fake news by conditioning the generator on knowledge elements, such as entities, relations and events, extracted from the original news article. Shu et al. (2021) enhanced the factuality of the generated article by introducing a fact retriever that fetches relevant information from external corpora. Mosallanezhad et al. (2021) used adversarial reinforcement learning to generate topic-preserving articles. These studies developed methods for generating fake news that is hard to distinguish from real news to humans. Nevertheless, due to the overwhelming amount of inaccurate information introduced and the lack of propaganda techniques in the generated texts, these approaches are suboptimal for detecting human-written fake news, as shown in §5.2. In contrast, we generate fake news by incorporating propaganda techniques and preserving the majority of the correct information. Hence, our approach is more suitable for studying defense against human-written fake news. Also, since our dataset is annotated with the exact offset of the disinformative passages, it enables research on interpretable detection of fake news.
Propaganda Generation and Detection There is little previous work on propaganda generation. Zellers et al. (2019) is the only relevant work, and it studied the generation of propaganda to communicate targeted disinformation. In contrast, we generate propaganda techniques to bring the generated articles closer to human-written fake news.
To the best of our knowledge, we are the first to study the incorporation of specific propaganda techniques into generated articles. Prior work on propaganda detection mainly focused on documentlevel detection. Early work collected propaganda datasets using distant supervision (Rashkin et al., 2017) by assigning the same propaganda label to each news outlet under the same source based on the news-media-level label of corresponding news source listed on trustworthy sites. However, classifiers trained on such datasets may only learn to recognize the bias of each news source instead of propaganda (Martino et al., 2020). Our dataset avoids such issues by explicitly incorporating propaganda into each generated article. Furthermore, Da San Martino et al. (2019) presented a fragmentlevel propaganda detection dataset, where specific propaganda techniques were labeled onto spans of text instead of each document. Recent approaches for detecting these propaganda techniques rely on pre-trained transformers (Morishita et al., 2020; Feng et al., 2021). In contrast, we focus on detecting disinformative articles with propaganda signals.

heading: 7 Conclusions and Future Work
text: We have proposed a novel method for generating disinformation that is closer to human-written fake news. Evaluation on two human-written fake news datasets, POLITIFACT and SNOPES, demonstrated the effectiveness of our generated data PROPANEWS in enabling better detection performance on human-written fake news. We hope that the dataset presented in this work, PROPANEWS, can serve as an enabling resource for detecting human-written fake news and encouraging future research in this direction.
In future work, we plan to extend our approach to other languages and to cover more propaganda techniques. We are also interested in studying other aspects of fake news generation, such as novelty and elaboration, as well as engaging linguistic style.

heading: 8 Limitations
text: To understand the gap between our automatic data generation method and fake news written by humans, we expanded PN-SILVER to different sizes and compared the performance of ROBERTALARGE when trained on these generated datasets and the human-written fake news dataset, SNOPES. Note that since the TIMELINE17 dataset only contains around 4K samples, we additionally crawled New York Times news articles as an input to our generator for the “5 times” to “10 times” experiments. The results are shown in Figure 3. Although the detector performance at first improves as we add more silver training data, it reaches a plateau after the size is increased five-fold. This illustrates that while our approach is more effective compared to baseline generation methods, there is still a clear gap between our generated articles and human-crafted fake news, likely in aspects such as style (as discussed in §5.2), intent (i.e., limited modeling of propaganda techniques), and falsehood (i.e., the generated content is 100% false).
Despite the advantages of our generation approach, as compared to previous methods, it is uncapable of generating other propaganda techniques covered in (Da San Martino et al., 2019), such as straw man. Thus, our method is not generic enough to handle all types of propaganda techniques within a unified framework. Moreover, our approach is limited to generating English-only news articles, and cannot be applied to other languages.

heading: 9 Ethical Statement and Broader Impact
text: Our objective for developing a generative approach that produces more realistic news articles is to advance the field of disinformation detection and to bring awareness that the current approaches for generating training data for fake news detection are sub-optimal.
We acknowledge that our generator may produce toxic text as it was fine-tuned on propagandistic datasets. We also understand the dual-use concerns for such a generation framework. One potential concern is the possibility of using the generator to produce fake news for political gain or to sow social discord. Another concern is the potential for the generator to be used to generate fake news that could cause harm, such as false medical information or misleading financial advice. Additionally, the generator might be used to create false evidence or to fabricate information to support false allegations in legal or regulatory proceedings.
Therefore, to contribute to future studies on human-written disinformation detection, we decided to release the codebase for only the detectors used in the experiments as well as the generated data but not the generator.
We highlight some scenarios that illustrate appropriate and inappropriate uses of our generator:
• Appropriate: Researchers can use our framework to produce more challenging training data for learning stronger detectors.
• Inappropriate: The method should not be used to intentionally create or propagate false information.
• Inappropriate: The propaganda generation technique should not be used for political campaigns or any malicious purposes.
Both inappropriate uses could lead to harmful consequences, such as undermining trust in the media and causing social unrest.

heading: Acknowledgement
text: This research is based upon work supported by U.S. DARPA SemaFor Program No. HR001120C0123 and DARPA MIPs Program No. HR00112290105. The views and the conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and to distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

heading: A Distribution of Propaganda
text: Figure 4 shows the distribution of the propaganda techniques used in the human-written fake news we collected and analyzed in §1. Note that one article may contain multiple propaganda techniques.

heading: B Additional Research Questions
text: Q1: Is the detector learning to distinguish between fake/real news articles or simply learning to detect the use of propaganda techniques? In Table 3, PROPANEWS w/o AA & LL is the variant of our proposed dataset with both propaganda techniques removed. By training detectors on this version of the proposed dataset, the model is still effective in identifying human-written articles containing false information. Therefore, the detectors trained on our generated data have learned to distinguish between fake and real articles instead of exploiting propaganda information only. On the other hand, comparing the detectors trained on PROPANEWS and their counterparts trained on PROPANEWS w/o AA & LL in Table 3, we see that propaganda can help improve the detection of real human-written fake news. We further want to emphasize that fake news detection is an extremely challenging task that requires both factual and stylistic analysis as demonstrated by our experiments and by the relatively low performance of prior SOTA models.
Q2: Do real articles make use of propaganda techniques, such as appeal to authority and loaded language? The similarity between our generated text and the real articles in PolitiFact is 7.3% as per the MAUVE measure, which is much lower than the similarity between the generated text and the fake news articles, as discussed in §5.2. It is possible that some real news articles can contain propaganda. However, according to MAUVE, the real articles in POLITIFACT do not contain much loaded language or appeal to authority.

heading: C Further Analysis
text: C.1 Remaining Challenges
To better understand the remaining disinformative articles that the detectors failed to identify, we conducted additional analysis by comparing the ROBERTA predictions and the labels. As a result, we identified the following three major modeling capabilities required for successful detection:
Static knowledge enrichment About 30% of misclassification is due to the lack of static knowledge that can be found in public databases, such as law dictionaries. For example, in this article,10 Alexandria Ocasio-Cortez falsely states that the U.S. Immigration Customs Enforcement (ICE) is required to fill 34,000 beds every day. According to the Appropriations Act of 2016,11, ICE is only required to detain 34,000 available beds. Therefore, to detect such kind of misinformation, the detector needs to be enriched with static knowledge bases.
Dynamic knowledge acquisition Around 48% of the misclassified human-written disinformation is due to the inability to acquire dynamic knowledge from new news sources. For instance, COVID19-related articles are usually published after 2020, while ROBERTA was pre-trained on news articles released before 2019. It is very challenging for ROBERTA to detect disinformation of such topics unless the detector is equipped with the capability to acquire dynamic knowledge from news articles. Particularly, ROBERTA achieves an accuracy of 69.0% on detecting fake articles published before 2019, but its accuracy drops to 51.9% when testing on articles published after 2019.
Multi-document reasoning The rest of the incorrect detection is caused by the lack of multidocument reasoning ability. For instance, a news article12 wrongly associates Hillary Clinton with a flawed immigration policy of the former government, and strengthens such a statement by referring to a Senate report and relevant news articles. However, the cited report does not mention Clinton, and the other news articles contain disinformation. To correctly detect this piece of disinformation, detectors should reason across multiple documents.
10https://tinyurl.com/static-knowledge 11https://www.congress.gov/114/bills/
hr2029/BILLS-114hr2029enr.pdf 12https://tinyurl.com/multi-doc

heading: D Qualitative Examples of Generated Articles
text: In Table 8, we show a comparison of generated articles given the same input data across different generative methods. Our approach produces articles with a small fraction of inaccurate information, which matches a property of human-written fake news discussed in §1.

heading: E Appeal to Authority Details
text: To recap, we first gather a list of authorities Z for each article from Wikidata and the corresponding context. The best appeal to authority sequence s∗ is selected, i.e., the one with the lowest perplexity s ∗ = argminsi PPL(si), where si denotes the generated sequence using zi as the authority. However, this process results in every sequence s∗ containing the substring “confirms that”, which makes it trivial for detectors to classify these generated documents as fake by simply detecting such substrings. Therefore, we devise an algorithm to diversify the templates so that these generated articles are not easily detectable.
First, we define a set of verbs V that can be swapped with “confirms”: V = {said, concluded, confirmed, emphasized, stated, argued}. Then, we diversify the generated structure of the generated sentence s∗ by reordering the subject, the verb, and the object. Next, we swap the verb with another verb from V . Finally, in order to diversify the context, we append a preposition from the preposition set PP = {on, at, in} to the output of the previous step, and then we feed the sequence to BART to generate the context. An example of this process is given in Table 6.
F Intermediate Pre-training Details
For domain adaptation, we perform intermediate pre-training (IPT) on the CNN/DM dataset, a large summarization corpus containing more than 280K news articles from CNN and Daily Mail. The IPT objectives for disinformation generation and propaganda generation are mostly the same as described in the previous sections, but with some minor changes due to different goals in the IPT phase. When performing IPT for disinformation generation, we removed Ls from the final loss function (Equation (4)) as the goal for IPT is only to learn to generate coherent sentences, and thus IPT is not needed.
Moreover, in order to create training samples for loaded language IPT, we gather all the appearances of adjectives pointing to a noun or adverbs pointing to a verb via dependency parsing graphs without considering whether the samples actually contain loaded terms since the goal here is to enable BART to identify where properly to insert which adjectives or adverbs.

heading: G Benchmarking Detectors
text: The performance of various detectors on the PROPANEWS dataset is shown in Table 5. We find that ROBERTA and GROVER demonstrate advantages over BERT. This could be explained by the fact that ROBERTA and GROVER are pre-trained on news domain corpora, whereas BERT has no access to such domains during pre-training. In addition, we find that HDSF performs much worse than the other three models. This reflects that largescale pre-training of language models brings more benefit to detection performance than explicit modeling of discourse-level features.

heading: H Human Validation Details
text: Next, we describe the details of human validation, where AMT workers were tasked to validate whether the generated sentences contained inaccurate information. We recruited AMT workers from USA and Canada. To ensure the annotation quality, only workers who had an acceptance rate greater than 95% and more than 100 accepted HITs in the past were allowed to work on our annotation task. This greatly reduced the chances of collecting annotations from scammers. Each HIT was designed such that the annotators were rewarded $12-$15 per hour, which complies with the ethical research standards outlined by AMT (Salehi et al., 2015). In each HIT, the annotators were presented an article with the generated part marked in boldface. The questions and the guidelines are given below. (Note that we only use the annotators’ response for Q1 to validate our generated data. The annotations for the other questions will be used for future research.)
sequence to the decoder in yellow, and the generated sequence in blue. To increase the diversity of the generated sequences, step 2 to 4 are each performed 50% of the time.
Q1: Is the generated text in boldface Accurate or Inaccurate? (If you cannot find any supporting evidence, please select Inaccurate.) Note that a statement (in quotation marks) made by a person is only accurate if this person actually made the exact same statement. If the statement in quotation marks is just a paraphrase of what the person actually said, then the statement is inaccurate.
- Inaccurate: Any false information presented in the generated text makes it inaccurate.
- Accurate: All the information in the generated text must be accurate.
Q2: Enter the URL of the news article you found that supports your decision in the previous response in the below box. Put down “from context" if the evidence can be found in the context.
Q3: Does the generated text in boldface deliver the same sentiment as the rest of the article?
- False: The sentiment of the generated text is NOT the same as the rest of the article.
- True: The sentiment of the generated text is the same as the rest of the article.
Q4: Is the discourse of the generated text in boldface consistent with the rest of the article?
- False: The discourse of the generated text is NOT consistent with the rest of the article.
- True: The discourse of the generated text is consistent with the rest of the article.
Q5: If there is any grammatical error or inconsistent discourse, please rewrite and correct generated text and put it in the below box. Just put down the corrected generated text in bold is enough. For example, “Harry is a boy. He likes go to school.” Please put in “He likes to go to school.” in the box below.

heading: I Statistics about the Evaluation Datasets
text: In Table 7, we give some statistics about the two evaluation datasets used in our experiments. The reported numbers are not the same as those in the original papers (Nguyen et al., 2020; Shu et al., 2018) since some of the articles were no longer accessible via the provided URLs.

heading: J Detector Implementation Details
text: For our experiments with BERT and ROBERTA, we used AdamW (Loshchilov and Hutter, 2019) with a batch size of 2 and gradient accumulation steps of 8. We set the learning rate and the weight decay to 5e-5 and 1e-5 for the parameters that have been pre-trained, and 1e-3 and 1e-3 for the other parameters. For the GROVER detector, we follow the original detection setting. GROVER is trained using Adam (Kingma and Ba, 2015) with a learning rate of 2e-5 and a batch size of 64.
Similarly, we follow the original recipe to train HDSF, which is optimized with Adam with a learning rate of 1e-2. All detectors are fine-tuned for at most 20 epochs where the best model is determined by the accuracy on the development set.
All experiments are conducted on an Ubuntu 18.04 machine with NVIDIA Tesla V100. We use PyTorch 1.10.0 and Transformers 4.3.0 for constructing all models and loading pre-trained weights, except for GROVER, which operates on Tensorflow 1.13.1. The training time for BERT and ROBERTA, each of which has 340M parameters, is around 2-3 hours, while for GROVER, which contains 355M parameters, it is about 1 hour.

heading: K Human Evaluation Details
text: In this section, we describe the survey we did with AMT workers for evaluating the quality of the generated articles. The annotators were presented with a generated article and were asked to answer a few questions regarding its quality. Q2 is only applicable for evaluating generated articles from PROPANEWS, in which we show the sentence that contains propaganda. The low, the medium, and the high ratings in the response correspond to the 1, 2, and 3 scores described in §5.2. The questions and the guidelines we gave were as follows:
Q1: How plausible do you think the article above is?
- Low: It likely contains inaccurate information.
- Medium: Not sure.
- High: It is unlikely to contain inaccurate information.
Q2: How much does this sentence in the article affect your decision for the previous answer?
- Low: This sentence does not affect my answer for the previous question.
- Medium: This sentence somehow affects my answer for the previous question.
- High: This sentence largely affects my answer for the previous question.
The score distribution for Q1 is shown in Figure 5. We can see that our approach yields higherquality fake news compared to GROVER-GEN.

heading: L Scientific Artifacts
text: We re-purposed the Timeline17 dataset (Tran et al., 2013) to produce our PROPANEWS dataset.
Among the software we use, SpaCy (“en_core_web_md”) is licensed under the MIT License, Transformers and Stanza are licensed under the Apache License 2.0, and PyTorch is released under the Modified BSD License. In terms of models, both BART and the NLI model we used are licensed under the MIT License.
In summary, all artifacts are produced with software and models released under open-source licenses. We will release the PROPANEWS dataset as well as the detectors with licenses compatible with the software discussed above such that they are consistent with their intended use.
orange indicate disinformation, and the texts in blue denote propaganda. We see that other approaches generate a large amount of inaccurate information, which contrasts with the property of human-written fake news mentioned in §1. We also note that the article generated using FACTGEN appears to be low-quality. This is likely caused by the fact that the checkpoints reported in the paper were not released and we trained FACTGEN from scratch by closely following the recipe described in Shu et al. (2021). It is possible that some details about the training process of FACTGEN were missing from their paper, which in turn affected our training, and resulted in low generation quality.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 9.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract & Section 1.

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: Grammarly is used to fix grammar errors throughout all sections of the paper.
B 3 Did you use or create scientific artifacts?

heading: Appendix L.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Appendix L.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix L.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix L.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No personal/sensitive information is collected.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix L and K.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5.1.
C 3 Did you run computational experiments?

heading: Appendix J.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix J.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix J.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5 and Table 3.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix J and L.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3 and Appendix H.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix H.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3 and Appendix H.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? We do not curate annotators’ personal data.

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: IRB 22841

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section 3 and Appendix H.

heading: Future Work
text: The annotations for the other questions will be used for future research. )
sequence to the decoder in yellow, and the generated sequence in blue. To increase the diversity of the generated sequences, step 2 to 4 are each performed 50% of the time. Q1: Is the generated text in boldface Accurate or Inaccurate? (If you cannot find any supporting evidence, please select Inaccurate. ) Note that a statement (in quotation marks) made by a person is only accurate if this person actually made the exact same statement. If the statement in quotation marks is just a paraphrase of what the person actually said, then the statement is inaccurate. - Inaccurate: Any false information presented in the generated text makes it inaccurate. - Accurate: All the information in the generated text must be accurate. Q2: Enter the URL of the news article you found that supports your decision in the previous response in the below box. Put down “from context" if the evidence can be found in the context. Q3: Does the generated text in boldface deliver the same sentiment as the rest of the article?
- False: The sentiment of the generated text is NOT the same as the rest of the article. - True: The sentiment of the generated text is the same as the rest of the article. Q4: Is the discourse of the generated text in boldface consistent with the rest of the article?
- False: The discourse of the generated text is NOT consistent with the rest of the article. - True: The discourse of the generated text is consistent with the rest of the article. Q5: If there is any grammatical error or inconsistent discourse, please rewrite and correct generated text and put it in the below box. Just put down the corrected generated text in bold is enough. For example, “Harry is a boy. He likes go to school. ” Please put in “He likes to go to school. ” in the box below