Analysing State-Backed Propaganda Websites: a New Dataset and Linguistic Study
Authors: Freddy Heppell
Kalina Bontcheva
Carolina Scarton
Venue: EMNLP 2023 Main (2023)
Keywords: misinformation
disinformation
social media
dataset
social science

Abstract: This paper analyses two hitherto unstudied sites sharing state-backed disinformation, Reliable Recent News (rrn.world) and WarOnFakes (waronfakes.com), which publish content in Arabic, Chinese, English, French, German, and Spanish. We describe our content acquisition methodology and perform cross-site unsupervised topic clustering on the resulting multilingual dataset. We also perform linguistic and temporal analysis of the web page translations and topics over time, and investigate articles with false publication dates. We make publicly available this new dataset of 14,053 articles, annotated with each language version, and additional metadata such as links and images. The main contribution of this paper for the NLP community is in the novel dataset which enables studies of disinformation networks, and the training of NLP tools for disinformation detection.

Analysing State-Backed Propaganda Websites: a New Dataset and Linguistic Study
Freddy Heppell, Kalina Bontcheva and Carolina Scarton Department of Computer Science, University of Sheffield, Sheffield, UK (frheppel11, k. bontccee,,, c. coscarton)@sheef wew.eell.............. ac.uk
Abstract
2022; Roache, 2022), and publicised by the Min- istry of Foreign Affairs of Russia's official Twitter account'. We focus on these two "news" sources due to their links to the Doppelganger network, their potential to deceive unsuspecting citizens (compared better known propaganda sources such as Russia Today), and their prior exposure as disinformation spreaders (see Appendix A),
This paper analyses two hitherto unstudied sites sharing state-backed disinformation, Reliable Recent News (rrn. (rrn.world) and WarOnFakes waronfakess com), which publish content in Arabic, Chinese, English, French, German, and Spanish. We describe our content acquisition methodology and perform cross-site unsuper- vised topic clustering on the resulting multilin- gual dataset. We also perform linguistic and temporal analysis of the web page translations and topics over time, and investigate articles with false publication dates. We make publicly available this new dataset of 14,053 articles, notated with each language version, and addi- tional metadata such as links and images. The main contribution of this paper for the NLP community is in the novel dataset which en- ables studies of disinformation networks, and the training of NLP tools for disinformation detection.
Backovic and Walter (2023) investigated the ownership of WarOnFakes, and stated it was op- crated by Russian journalist Timofey Vasiliev, known affiliate of Russian propaganda groups, due to the presence of his name, email and phone num- ber on the website. However, they do not state precisely how they found this information, and do not attempt to establish a link between Vasiliev and RRN or the Doppelganger operation.
Hanley et al. (2022) included selected articles from WarOnFakes and nine other disinformation websites in an analysis of narratives spread Red- dit. In contrast, our dataset includes all WarOn- Fakes posts and extracts the full article content.
1 Introduction
Coordinated, state-backed disinformation opera- tions have become increasing problem in recent years, particularly surrounding the war in Ukraine (Morkūnas, 2022). In September 2022, a sophisti- cated network of doppelganger websites (imperson- ating genuine news sites from across Europe) was discovered by EUDisinfoLab (Alaphilippe et al., 2022) and later expanded on in report from Meta (Nimmo and Torrey, 2022). Among these was also small number of conventional false news sites. The focus of this study is on two related dis- information sites in particular: Reliable Recent News! (RRN) and War On Fakes? (WoF). Both sites are multilingual, publishing in Arabic, Chi- nese, English, French, German, and Spanish, and RRN additionally in Italian They have been promoted by Russian government sources, includ- ing being shared by Russian embassies (Maitland,
Propaganda is defined as content that intention- ally influences opinion to advance its creators' goals (Bolsover and Howard, 2017). Numerous propaganda datasets have previously been cre- ated, with both document-level (Rashkin al., 2017: Barrón-Cedeño et al., 2019) and span-level (Da San Martino al., 2019b) technique annota- tions, using articles collected from multiple dis- information sites. At article-level, classifiers us- ing combinations of multiple linguistic representa- tions based on style and readability outperform con- tent representation (Barrón-Cedeñño whereas content-based transformer models such BERT have seen use at span-level (Da San Martino et al., 2019a). Detectors are often evaluated on sin- gle datasets, prompting concerns on generalisation (Martino et al., 2020).
Cedeño et al., 2019),
We are not aware of any prior work including RRN, of any work which has released com-
'ht /ttpp:://rr.... world, formerly called Reliable Rusxia News using wrwssianews.comm htap::://wwooaaaaeee wwwroonaakke... com
phtpps:/ / /tt///twtttt wtwwitte...... wtom///farrussaastatuus 9302941487107
WoF also has a separate Russian-language site

---

plete dataset of disinformation operation, includ- ing a detailed linguistic analysis.
lows for embeddings not be included in cluster, preventing overly broad clusters by forcing nearby but unrelated sentences in. expected that this produces a large number of outliers, since is nat- ural that many of the sentences in the articles will have meanings unrelated to any other. A minimum cluster size of 25 is set to prevent too many small clusters from being generated.
Thus the contributions of this paper are: i) a new publicly available" dataset of content from two state-backed disinformation websites; ii) a linguis- tic, topic, and temporal analysis of their articles; and iii) our open- toolkit for processing site data and extraction of translations
Keyword representations are generated by creat- ing bag-of-words vector the unigrams and bi- grams of each topic (excluding English stopwords) which is Ll-normalized to account for cluster size. An adapted Mlass-based TF-IDF is used to calculate the most significant words in each cluster. This representation is then fine-tuned by selecting key- words with high Maximal Marginal Relevance, in order to maximise their diversity. The di diversity parameter was set to 0.5. The top 3 most significant keywords are used to name the cluster.
2 Methodology
2.1 Data Collection
In March 2023 we used the WordPress REST AP17 obtain all posts from Wol' and RRN. Each post parsed extract its text, removing non-article content (such as figure captions). The webpage of each post was then analysed extract the dif- ferent translations from the language picker. Our extraction tool supports the specific markup used by these two sites, but can be easily extended to support others. An example of an extracted article is shown in Appendix B.
Each article is then labelled with the unique set of clusters assigned to its sentences.
Publication and modification times, which provided in GMT by the API, were also converted Moscow local time for analysis, since be- lieved that at least one the sites is based in Russia (Backovic and Walter, 2023).
2.3 Article Backdating
In WordPress, article publication dates can be set to any given date, however this does not affect the auto-incrementing IDs which are generated in the order of article creation. Thus backdated articles can be detected based their IDs being higher than that of their following articles, when they are ordered by supposed publication date.
2.2 Topic Analysis
The articles were clustered using BERTopic (Groo- tendorst, 2022). We assume that whilst each ar- ticle may discuss many topics, each sentence of an article is likely to discuss a single topic. Ar- ticles were split using spaCy's dependency-paree based sentenceizer, and sentences with less than 5 tokens were removed. The remaining sentences embedded with Sentence Transformers MP- NETS (Reimers and Gurevych, 2019; Song al., 2020). The dimensionality of each embedding was reduced using UMAP (McInnes et al., 2020) from 768 to 5, whilst keeping the structure of the higher- dimensional space. This is necessary to avoid the 'curse of dimensionality"",
2.4 n-gram Frequency
Frequent 2-4-grams were extracted using NLTK, after tokenisation, lowercasing, and stopword and punctuation removal. N-gram frequency was cal- culated monthly, and the most frequent 10 n-grams per month were selected, excluding the phrase "armed forces"10 and n-grams which are part of another, longer n-gram of equal frequency (e.g. re- moving "ukrainian armed" in favour of "ukrainian armed forces"). We include ties for 10th place.
Analysis
The 5d embeddings were clustered with HDB-
SCAN (Campello al., 2013), which notably al-
3.1 Dataset Size and Coverage
Our dataset contains 14,053 translations of 3,447 articles posted between 4 Mar 2022 and 6 Mar 2023. Table shows the number of articles per
sitee
xtractor http:://ddeeeooperewwrrrrrrresse wtw.rrrrreeees.rrreeetaaaaa.
'nttps://huggingfaase wtw.reeeieeeeeeeeeeeeeeeeeeeerrr
a1--mpet-bbsee-v2 bd44305
Th term is highly frequent, but is ambiguous as includes both Russian and Ukrainian armed forces, which appear as separate highly frequent trigrams.
"Clustering is difficult in higher dimensional spaces as
distance is less mcaningful (Aggarwal et al., 2001)

---

RRN
WoF
Language
200
ar
de
150
Count
en
CS
100
fr
it
50
zh
May
Jul
Sep
Nov
2023
Mar
May
Jul
Sep
Nov
2023
Mar
Date
Date
Figure
Monthly post counts across both sites, by reported article date, Partial March 2023 excluded.
Asticle Count
Mean #larticle
the lowest activity in the sites' history, with only 60 articles published RRN and 19 on WoF. For comparison, the mean in other weeks is 200 (RRN) and 66 (WoF).
Language
RRN
WoF
Total
Toks.
Sents.
Arahic (ur)
509
324
833
201.89
10.44
German (de)
2032
473
2505
339.90
15.91
English (en)
2265
864
3129
341.31
15.84
25 identical articles were published on both sites predominantly in March 2022, and in all but case they were WoF-style debunk. They were not published simultaneously on the two sites, nor is it consistent which site published first.
Spanish (es)
1229
468
1697
345.22
14.99
French (fr)
1968
683
2651
386.91
15.42
Italian (it)
1288
1288
429.89
19.03
Chinese (2h)
1220
730
1950
261.47
13.41
All
10511
3542
14053
338.89
15.37
3.3
Language Coverage
Table I: Number of articles per site, language
Only small minority of posts (~9.1%% not available in English, and the majority of these do not have any translations at all, suggesting they are likely 'orphaned' translations. The mean number of available languages for post is 4.1 + 1.5 (1 std)
site and language, and mean token and sentence 'l
3.2 Article Frequency
Figure I shows the proportion of each language over time for each site. The first WoF article is pub- lished on the 4th March 2022, and the first RRN article on the 11th. WarOnFakes has an unusual pattern publication its first few days, publish- ing sixty articles on the first day, and an average of 34 articles/day over the first 7 days, whereas RRN published only 7 articles on day one and an average of 21 articles/day over the first week.
All site-language pairs continued to be published until the end of the collection period, except Ara- bic and Spanish WoF and Chinese RRN, which stopped in July and October 2022 respec- tively. Spanish posts resumed in December 2022.
3.4 Topics
Amongst the 45,991 English sentences in the En- glish articles, 24,800 were considered outliers and 21,191 were assigned one of 144 topics. These top- ics ranged from broad, recurrent themes (c.g. #0, the donation of arms and aid to Ukraine) to more specific, time-limited ones (e.g. #139, the burning of the Quran by far-right activist Rasmus Paludan).
Generally, posts are published on weekdays, with only 9.5% of posts having publication dates and 7.0% having modification dates on a Saturday or Sunday. The week beginning 2nd January 2023, much of which is public holidays in Russia 12 has
"Calculated using spaCy tokeniser rule-based sen- tenciser. with added delimiters. Chinese segmented with PKUSEG web model (Luo et al.. 2022). Arabic support limited.
The mean number of topics assigned per arti- cle 4.33 2.66 std). the first week of the war in Ukraine (beginning 28th Feb 2022), the vast majority of articles are categorised as #2 (russian
Phttps:// httt:::/wwwwwbb cbr wttp.ru/eeng/tthe/holl..ie

---

military, ukranian telegram, telegram channels, ac- cording [to] ukranian). These articles are all from WarOnFakes (since RRN did not start publishing until the following week) and are claiming that various evidence from the war in Ukraine is fake.
Mctric
RRN
WoF
NYT
Tone
27.71 4
15.06 L
43.61
Affective Processes
4.67 1
3.97 f
3.82
Positive Emotion
2.12 4
1.23 L
2.32
Negative Emotion
2.49 f
2.721 f
1.45
Of the 144 topics we identified, 126 were as- signed articles from both RRN and WoF, and only 18 were assigned to posts from just one of the two sites. This demonstrates the significant topi- cal overlap between the sites. Further details and figures are provided in Appendix C.
Anger
1.01 1
0.98 f
0.47
Past Focus
3.67 4
3.77 4
4.09
Present Focus
6.42 1
6.40 f
5.14
Future Focus
1.12
1.00 1
0.8
Table 2: Comparison of selected LIWC2015 attributes, compared the New York Times
3.5 LIWC Analysis
We use LIWC2015 (Pennebaker et al., 2015) to compare the linguistic properties of English RRN and WoF posts against the metrics for genuine New York Times (NYT) articles provided by Pennebaker et al. (see Table and Appendix C.I).
RRN
WoF
Metric
Metric
Conjunctions
0.311
Colons
0.487
Pos. Emotions
0.310
Interrogatives
0.373
Pers. Pronouns
0.277
Impers. Pronouns
0.293
Emotional tone, which is on scale of 0-100 (negative to positive), shows that RRN and WoF written more negatively than real news, with WoF being even more negative than RRN. This is con- firmed by the values for Affective Processes, which show that both sites use more emotion-laden words than NYT. The sub-metrics show this is skewed towards negativity, particularly anger (where both sites have over double the proportion of anger- indicating words than NYT).
Discrepancies
0.271
See
0.263
Time
0.270
Leisure
0.189
Table Top correlated LIWC values. Bold values above strength threshold.
nouns (i.e. one, you, they) as it tends to discuss groups, such as the Russian and Ukrainian armed forces (see Table Appendix C).
All three sources focus most commonly on the present (e.g. words like "today", "is", "now"), how- ever RRN and WoF do so at a higher rate than the NYT. RRN and WoF also use more future focus terms (e.g. "may"" "will", "soon") compared to the NYT, and past focus terms (e.g. "ago", "did", "talked") less frequently. This suggests that the content of RRN and Wolf comments more spec- ulative as compared to reputable journalism and is more focused on covering current events than past ones.
3.6 Article Backdating
Both sites tend to backdate non-English posts (by as much as 136 days in two cases, see Appendix C, Table 5), in order to make translations appear pub- lished at similar time. The two most backdated articles are Spanish and Chinese translations of an English article, which were actually published 136 days later.
Our hypothesis for the backdating is due to lim- ited resources articles were only translated into a given language when that became necessary for a particular disinformation campaign. In order to convey timeliness, the translations were then back- dated to the date of the original.
Table shows the top LIWC categories with the strongest correlation for each of the two sites. The strong correlation of colons and interrogatives for WoF is unsurprising, given its repeated use of the phrase "What's really going on:". RRN's cor- relation with conjunctions suggests it tends to use complex sentences. The remaining attributes are below the 0.3 threshold of strong correlation. However RRN is weakly correlated personal pronouns which due its tendency to cover in- dividual politicians (see Table 6 in Appendix C),
3.7 n-gram Analysis
Tables 6 and 7 in Appendix C show the top occuring n-grams per month for the respective websites. The most frequent "really going" n-gram on WoF is part of the phrase "What's really going on", which appears in all of its fact-check-styy style articles. The n-gram also appears frequently in the first month of RRN data, due to the articles copied from WoF.
while WoF weakly correlated to impersonal pro-

---

Category
RRN
WoF
e.g. the name of Russian organisation.
Unclear: We were unable to determine why the characters were used.
Accidental Cyrillic
58
34
Forgotten Cyrillic
15
25
Given that both RRN and WoF had forgotten Russian text in all languages, hypothesise that all articles were originally written in Russian. Two Arabic articles on RRN contain the phrases "the franslation is too long" and "save translation" Russian, likely copied from a machine translation tool's UI, although we were not able to determine the specific tool used. Although this was only found in one language on one of the sites, it sug- gests it is more likely the articles machine than human translated.
Intentional
36
10
Unclear
0
2
Table 4: Frequency of Cyrillic usage reasons
On WoF. the most frequent n-grams typically relate directly to the war in Ukraine itself ("rus- sian troops", "ukranian armed forces"), whereas on RRN they relate to the consequences of the conflict for the rest of the world ("umited states", "russian gas"). Consequently, the most frequent n-grams on WoF are relatively constant across the differ- ent months, whereas RRN's n-grams change from one month to the next as they tend to be connected to current affairs. For example, the bigram "anti- russian sanctions" enters the top 10 in June 2022, and remains the second most used bigram from July to September, and refers to the damage allegedly caused to Western economies. Other terms demon- strate that RRN also covers some genuine news, e.g. "elizabeth ii" in September 2022 and "world cup" November and December 2022.
4 Future Work
There is much additional work which could be per- formed on this dataset. Although we identify the subject of articles via topic clustering and n-grams, we do attempt to identify stance towards it. More complex topic analysis, such as identifying commonly co-occuring topics, would also be pos- sible. Given the mixture true and false posts on the sites, this dataset may be a useful resource for automated fact-checking, although this would require human annotation and ground-truth may be difficult to establish in the complex information environment of the war Ukraine.
Even though to much lesser degree, WoF still responds to specific highly controversial events from the conflict. For example in August 2022, in response to Ukraine and Russia blaming each other for the shelling of the Zaporizhzhia nuclear power station! the n-grams "nuclear power" and "nuclear power plant" both appear with high fre- quency in WoF articles that promote the Russian perspective on these events.
5 Conclusion
This paper presented an analysis of the Russian disinformation sites Recent Reliable News and WarOnFakes, including analysis of the articles' topies, publication times, and linguistic properties. We show that the sites cover diverse range of topics, and that their linguistic propertics differ from those of reputable media. We analysed the presence of Cyrillic characters due to site opera- tor errors, and their practice of backdating articles, showing that significant proportion of translations falsely dated. This new multilingual dataset will facilitate further research in disinformation analy- sis and promote repeatability.
3.8 Presence of Cyrillic Characters
178 of the articles were found to contain characters in the Cyrillic codepoint range (Table 4), which were manually examined determine the reason. Accidental Cyrillic: Incorrect usage of Cyrillic characters instcad of the intended character in the Latin alphabet. For example, 11 times the "c" in Robert Habeck, a German politician, is actually the identical-looking lowercase Cyrillic 14
Forgotten Cyrillic: Issues with translation where a Russian sentence was left in the article, with or without the target language translation.
Limitations
Although our work provides complete collection of Wol and RRN, since these two websites seem be highly related, it is unsurprising that they tend to publish similar types of content. Therefore this dataset cannot be considered fully representative
Intentional: Expected usage of Cyrillic characters
Phttps://reut rs/46KWVTS
"https: //en. www.pedia.a.cc.mmm httpe/aaa..eeiiiiiseSCctririll

---

of all kinds of Russian disinformation. Neverthe- less, it is complementary to overtly Russian state media, such as Sputnik and Russia Today. Unfor- tunately, due to the ban on accessing their content from the FUU, we could not supplement the dataset from those sources or compare against them.
viewing distressing media; iii) ensuring familiarity with supporting resources for researchers working with potentially disturbing content.
As the websites in question are not legitímate news websites, they do not have a terms of use to allow or prohibit the acquisition of their content. We consider the collection and distribution of their articles in the public interest, due to the prominence of their disinformation and the harm that results from it. not feasible to contact them to obtain permission, as they have previously been unrespon- sive to enquiries 15 The dataset does not include images, as in many cases they appear to have been taken from stock agencies. This is a commonly used tactic by disinformation websites.
Our topic analysis model has not been formally validated, for example by comparing topies those assigned by human or expert annotators. Some small scale manual validation was performed in order to find good hyperparameters, however this consisted of inspecting small random sample of some of the categoríes. A particular area warrant- ing validation in future work is examining the texts not assigned categories. These are only very small number, however as we aggregate sentence classifications article level, which means that an article can be assigned the correct topics even some of its sentences may not be.
We have checked that the dataset does not con- tain personally identifiable information in the user data files, as all users have either generic (e.g. "Ad- min") or random (e.g. "UiXnZyvH")) names. No user comments were available to collect.
In our LIWC analysis, we compare to the New York Times data provided by Pennebaker et al. (2015). Although this is the closest source out of the provided LIWC baselines, the New York Times represents more formal style of journalism than many online media. future work we plan to compare these two disinformation sites against official state-affiliated news sources such as Russia Today.
It is possible that the process of creating disin- formation dataset increases the spread and promi- nence of the disinformation. We would argue that is not the case with this dataset as we: i) are only focusing on content from disinformation web- sites, the low credibility of which has already been widely publicised (see Appendix A); iii) are not in- creasing the longevity of disinformation narratives by preserving them after they have being taken down, since the two independent websites that are publishing them are still publicly accessible via all common search engines.
Finally, we did not analyse the separate Russian- language edition of WarOnFakes. As is separate site in Russian only, there is no reliable way to con- nect its articles to their similar versions (if such are published). Analysing the Rus- sian Wol website is planned for future work, as it requires adaptation of the analysis to be bilingual, which is out of scope for this paper.
Some articles make reference to individuals, al- beit only public figures to our knowledge, and many contain narratives which are hateful towards indi- viduals and groups. We encourage researchers who use this dataset to do so responsibly, and in par- ticular to avoid highlighting specific individuals and to ensure that the disinformation narratives are presented alongside authoritative evidence of their untrue nature. We would like to specifically discourage the use of this dataset for training gen- erative models that are capable of creating new disinformation. The dataset is released under a license which prohibits commercial activity.
Ethics
The data collection was carried out in accordance with our institutional ethics policy.
Collection was via the Wordpress API, followed by automated processing and a limited amount of manual analysis by the authors. No external vol- unteers or crowd-workers were recruited. Due to the disinformation nature of these two websites, the data may contain content which is disturbing distressing. Therefore we limited the possibil- ity of harm during analysis by: i) minimising the number of individual articles studied by the authors as much as possible; ii) where necessary, viewing only the text of articles, to avoid the possibility of
A Acknowledgments
This work is partially supported by the UK's innovation agency (InnovateUKK) grant number
NNewsGuarrd attempted to contact them as part of their review process (Maitland, 2022; Roache, 2022)

---

10039039 (approved under the Horizon Europe Pro- gramme as VIGILANT, EU grant agreement num- ber 101073921). 16 Freddy Heppell supported by a University of Sheffield Faculty of Engineering PGR Prize Scholarship.
Hans W. A. Hanley, Deepak Kumar, and Zakir Du- rumeric. 2022. Happenstance: Utilizing semantic search to track russian state media narratives about the russo-ukrainian war on reddit. Computing Re- search Repository, arXiv: 14484v2. Version 2.
Ruixuan Luo, Jingjing Xu, Yi Zhang. Zhiyuan Zhang, Xuancheng Ren, and Xu Sun. 2022. Pkuseg: A toolkit for multi-domain chinese word segmentation. Computing Research Repository, arXiv:1906. 1455. Version 3.
References
Charu C. Aggarwal, Alexander Hinneburg, and Daniel A. Keim. 2001. On the surprising behavior of distance metrics in high dimensional spaces. In International Conference on Datahase Theory.
Eva Maitland. 2022. world nutrition label. Tech- nical report, NewsGuard.
Alexandre Alaphilippe, Gary Machado, Raquel Miguel, and Francesco Poldi. 2022. Doppelganger media clones serving russian propaganda. Technical report, EU DisinfoLab.
Giovanni Da San Martino, Stefano Cresci, Alberto Barron-Cedeno, Seunghak Yu, Roberto Di Pietro, and Preslav Nakov. 2020. A survey on computa- tional propaganda detection. Computing Research Repository, arXiv:2007.08024
Nick Backovic and Kyle Walter. 2023. Logically in- vestigations: Russian propaganda disguised as fact checking. Technical report, Logically.
Leland McInnes, John Healy, and James Melville. 2020. UMAP: Uniform manifold approximation and projec- tion for dimension reduction. Computing Research Repository, arXiv: 1802.03426.
Alberto Barrón-Cedeion Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organiz- ing the news based on their propagandistic content. Information Processing & Management, 36(5):1849- 1864.
Mangirdas Morkunas. 2022. Russian disinformation the baltics: Does it really work? Public Integrity, pages 1-15.
Gillian Bolsover and Philip Howard. 2017. Compu- tational propaganda and political big data: Moving toward more critical research agenda. Big Data, 5(4):273-276.
Ben Nimmo and Mike Torrey. 2022. Taking down coor- dinated inauthentic behavior from russia and china. Technical report, Meta.
Ricardo J. G. B. Campello, Davoud Moulavi, and Jo- erg Sander. 2013. Density-basee clustering based on hierarchical density estimates. In Advances in Knowledge Discovery and Data Mining, pages 160- 172, Berlin, Heidelberg. Springer Berlin Hcidelberg.
James W Pemnebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psycho- metric properties LIWC2015.
Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and po- litical fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2931-2937, Copenhagen. Denmark. Association for Computational Linguis- tics.
Giovanni Da San Martino, Alberto Barrón-Cedeño, and Preslav Nakov. 90. Findings of the NLP4IF-2019 shared task on fine-graines propaganda detection. In Proceedings of the Second Workshop on Natural Lan- guage P Processing for Internet Freedom: Censorship. Disinformation, and Propaganda. pages 162-170. Hong Kong, China. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP--JCNILP), pages 3982-3992 Hong Kong, China. Association for Com- putational Linguistics.
Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedefio, Rostislav Petrov, and Preslav Nakov. 2019b. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP) pages 5636-5646, Hong Kong, China. Association for Com- putational Linguistics.
Madeline Roache. 2022. WarOnFakes.com nutrition label. Technical report, NewsGuard.
Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with class-based TF-IDF procedure. Computing Research Repository, arXiv:2203.05794. Version
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. MPNet: Masked and permuted pre- training for language understanding. Computing Re- search Repository, MxXiv22044009299 Version 2.
"httts://wwwwviillantttpp www.:wigiigaaaprjec...omm

---

Defence's Telegram channel. BBC Monitoring, the specialist media source analysis division of BBC News, states "Some of its fact-checks are genuine but most content is Russian talking points on the invasion which do not stand up to scrutiny"
600
Number of articles
400
The site has also been covered by EUvsDis- info DFRLab the European Digital Media Ob- servatory and Media Bias/Fact Check
200
RRN has received comparatively less attention from fact checkers, however was described as disin- formation by NewsGuard (Maitland, 2022), which additionally claims that they reuse content from WarOnFakes, and EU Disinfo Lab have noted a connection in the hosting infrastructure of the two sites (A (Alaphilippe et al., 2022). It is therefore prob- able that the apparent state-backing of WarOnFakes also applies to RRN.
5
10
15
Number of topics
Figure 2: Number of unique topics assigned per article
Language
Backdated
Mean
Max
ar
39.4%
-8d
-34d
es
11.5%
-9d
-
136d
de
11.1%
-7d
-14dd
B Data Example
fr
10.2%
-9d
-09d
zh 8.4%
-15d
136d
Figure shows an example of an article published on Wo in English, French, Spanish, Chinese and Arabic. Full texts are omitted for languages other than English. This story was judged to be fake by fact-checkers Usage of guillemets (< x) quote marks is reproduced returned by the WordPress API, but this appears to be normalised when the page is rendered.
6.9%
.4d
-6dd
en 0.2%
-1d
-2d
Table Backdating per language both sites.
A Evidence of Disinformation
For WarOnFakes, there is a substantial number of articles and fact-checks establishing it as disin- formation source. PolitiFact undertook review of over 380 of their fact-checks and found a sig- nificant number of falsehoods!?, In an article by AFP via France24, Roman Osadchuk, from the Atlantic Council's Digital Forensic Research Lab (DFRLab), quoted as saying "Since Russia's in- vasion, the 'War On Fakes' initiative has become a powerhouse of spreading false debunks" and "It is an effective tool of state propaganda and disinfor- mation" 18, The Institute of Network Cultures de- scribes it as "Kremlin-Sponsoree Particpatory Pro- 19 and highlights connections between the Russian state and the website, including promo- tion from organisations under the Russian Ministry Foreign Affairs, and on the Russian Ministry of
Detailed Dataset Statisties
Figure 4 shows a weekly chart of the 10 most com- mon topics the site. In general, there is no clear variation between these topics, with the exception of the initial popularity of the topic #2 due the majority of posts that week being from WarOn- Fakes. The significant dip in January 2023 is due to the Russian public holidays discussed section 3.2.
Figure 2 shows the distribution of sentence-level topic counts aggregated for each article. 78 posts were not assigned any topic, the majority of articles
htittps::/mmmmitrring ing.tt..iiiniooeeee bbc. co. ok/pproduct/c283aq Phttos::/euwelliiiffoee/ui Rattess//mmddunncoo//r/////////s
d- dddisnnformatiio--cass493335555
http://ppwwp..aaaatoa/oaaaaaa/// http://saaapaaaaaaaooaaoooaoaa// 88 hhhowwwwrr-aakes-ueeee fact-cchekinggspreeeee russia/ 16- oef.aaeeeratcccceee---eeeeeeerer
nda fed.eruiiigiiiiiii.aaaaacccttttn
m- the edmo taskortaskforce/////o/ooo
-bias/
e-in-war 19 com/ -pponsored-particcptttt propaaaadd/
Vatton-struck-emmterrttt---uuuuu wtwp--hoooppalllltt----tttttt-- and-children!

---

have 1-3 topics, however in the extreme some have many as 21 this is an article from WarOnFakes "What happened in Bucha? A full analysis of the Ukrainian provocation", a long article supposedly explaining the truth about many elements of the Bucha massacre.
Table 5 shows the proportion of backdated ar- ticles per language, and the mean and maximum backdating period for each. For English, small number of posts are backdated after a short period of time. It is likely this caused by posts that have been forward-dates (i.e. set to be published in the future) by one or two days, resulting in subsequent posts appearing to be backdated until the publica- tion date catches up. However, for other languages, backdates are for much longer period.
C.1 Complete LIWC2015 Data
The complete listing of LIWC2015 is included in Table 8, in the hope it can be used for comparison in future work.

---

ENGLISH
Fake: Russian aircraft attacked a maternity hospital with mothers and children inside
What is fake about:
Information that Russia launched an airstrike on maternity hospital in Mariupol is being spread online. Ukrainian President Volodymyr Zelensky called it wan atrocity and said that women and children remained under the rubble.
The fact
Despite the fact that information about the strike appeared in the middle of the day March 8, no single patient was visible on numerous videos and photos. The footage of pregnant women appeared on the Internet much later in the evening of March 9. However, it immediately was circulated by all news agencics, social media, popular communities and bloggers, which may be the result of a preplanned campaign. Moreover, was happening despite the fact that the locals themselves claimed that there were no patients or members of the staff in the maternity hospital.
This story rather dubious. It logical to assume that there really had been patients then the rescue service officers and eyewitnesses who arrived the scene would immediately have taken photos of the accident scene with their phones, without waiting for a well-known photographer. However, so happened that the well-known Ukrainian propaganda activist Evgeniy Maloletka was the first to prepare and publish the photographs.
Today we received indisputable confirmation that the aphotos of pregnant women" were staged. The Ukrainians used a model called Marianna who comes from Mariupol for the most striking photos (there three in total). It is notable that she played roles of two different pregnant women at the same time: she even had to change clothes and the color of her hair, which, however, is not surprising: in fact, Marianna is well-known beauty blogger in the region. It's worth noting that the girl is indeed pregnant, but she just could not have been in the maternity hospital: the Azov militants had used the medical facility for several days fortified stronghold that does not function heroine of this hoax has already been caught in the spotlight. In the comment section of her Instagram account there are already more than 500 comments under her last post written by real users condemning the girl for participating in information manipulations.
maternity hospital any longer. The main
FRENCH
L'infox: les forces aériennes russes ont bombardé la maternité. Les femmes et les enfants ont été ciblés
SPANISH
Fake: La aviación rusa atacó al hospital materno-infantil con madres bebés
CHINESE
ARABIC
الأطفال الأمهات بها للادة الو مستشفي هاجم الروسي ان الطير مزيفن خبر
Figure example of an article from WoF. Article text omitted for non-EN languages space.

---

000 military aid vehicles bundeswehr,
vehicles_bundeswhh___nn tanks ukraine = 001 truth think jell situation conclusions drawn
002 mussian military_ akrunnan telegrum selegnam channels according ukrainian
telegram_telegram
yladimir president zelensky volodymyr ukrminian president
european union, european leaders commission
005 german government berlin german foreign_ republic germany 006 russian gas price ceiling russian energy rubles
007 winter lighting firewoed 19 degress = 008 joejjo fbi white house
009 ukrainian telegram_reportel ukrainian reported ukrainian media
010 china china_jinping jinpinggnnpe
100
Number of articles
80
40
20
Mar 2022
May 2022
Jul 2022
Sep 2022
Nov 2022
Jan 2023
Week
Figure 4: Weekly frequency of top 10 clusters. Partial data for March 2023 excluded.

---

Mar 22
Apr
May
Jun
Jul
Aug
united states
united states
united states
united states
prime minister
prime minister
russion gas federation
russian foderation
special operation
curopean union
anti-russial sanctions
anti-russian sanctions
russian military
prime minister
anti russian
european union
really going
special operation
ruesian militury
prime minister
nord stream
energy crisis
russian military
ukrainian arned
ukrainian armed
sanctions russia
ukrainian refugees
russian
european countries
ukrainian armed forces
ukrainian military
white house
ukrainian amy von der leyen european commission german govemment united states
united states
prime minister
pen
western countries
joe biden
per cent
ministry defense
russian troops russian gos
forces
ukrainian refugees
nuclear power
people republic
ukrannian soldiers joe biden
europeun commission
energy prices nord stream
ukrainian nationalists
olaf scholz
russian oil
ukranian crisis
Sep
Oct
Nov
Dec
Jan
prime minister
prime minister
united
united states
united states
united
anti-russian sanctions
united states
hiden
world cup
prime minister
white house
nord stream
liz truss
prime minister
white house
ukrainian amyy
joe biden
energy crisis
naclear power
white house
joe biden
ukrainian arned
prime minister
united states
vladimir putin
world cup
prime minister
ukrainian arned Forces
viadimir zelensky
european commission
foreign
ukrainiun army
white
military aid
vladimir putin
sanctions
elon musk
vladinir putin
foreign minister
nord
elizabeth
crimean bridge
vladimir putio
emmanuel macron
joe biden
ukrainian armed forces
muclear power
energy crisis
rishi sunak
price cup
foreign policy
last year
liz truss
white house
anti-russian sanctions
clon musk
olaf scholz
ukrainian army
gas
new year ordinary people
world war
Table 6: Top n-grams for RRN
Mar
Apr
May
Jan
hil
Aug
relly poing
russian troops
really going
really going
really going
really going
fike mesagg
telegrim churnels
welegamm chamnes
telegram chemnele
telggam churnele
ukrsinsm armal firee
telegram channels
forces
military
telegram channels
ukrainian
telegram
miliory
nows
akmnnnian telegom
shopping center
power plant
foree tkmise
semed fores ukraise
ukoiinia telegma chances
ukraisia anned foces
fike russian
fake ressian
anned foces ukrinz
take message
rassian woops
fake rustaa
uka ainias telegrum channels
ukezinian telegrami
ninistry defense
soldiess
forces
powe:
federstion
really going
ukzainian
uismation
military
ukzzniaa telegram channels
skmimion relegram channels
russian military
ukrainin
fake acconding
western media
naclear power
rusian anned
ukrainian telegram channels
russian amndd torces
fake dkramian
ukrainia side
rassian amed
nosion amnrd fonces
ukminan
ummed fonoes
skraisiin malis
ulesinian
wally pring
really going
really poing
going
really going
really going
tologram chranek
telegram charnes
forees
vates
tolegrats channel
skraninn amed lorces
telegract charnels
ukcninan ixmed torees
ukraition anny
telogram channels
milliary operation
channels ikmimion nelegram
russin armed tinces
ukmiman telegrmm
nuscunt armed forces
ukrainin telegrim
special miltary operation
federation
ukrainian
channels
channels
utrainiat
Soderution
channels really going
zelencky
telegmim
telegne
clemnel really going
ukrainis ielegren chemnels
russian foneos
nuviuon foleration
sir defence
telegram chanziss maily poing
maiia anned foeces
vladair puta
ukeainia media
telogrann channel
russian a med forces ulrainian aemel freces ulrainias poopagands.
unded states
selegram channels really
fikte ruseian
air defence
prssidest viulimir fakes
need steam
selggam channels really going
ulcainias aray
chanaels going
puta
Table 7: Top n-grams for WoF

---

Category
RRN
WoF
All
Category
RRN
WoF
All
Word count (mean)
313.80
249.24
295.74
Perceptual processes
1.72
2.03
1.81
Summary Variables
See
0.65
1.32
0.84
Analytic
94.37
95.12
94.58
Hear
0.63
0.42
0.57
Clout
63.62
56.75
61.70
Feel
0.34
0.22
0.30
Authentic
23.13
22.95
23.08
Biological processes
1.24
1.06
1.19
Emotional Tone
27.71
15.06
24.17
Body
0,40
0.36
0.39
Health
0.56
0.57
0.57
Language Metries
Sexual
0.04
0.03
0.04
Words/sentence
19.92
20.15
19.98
Ingestion
0.27
0.14
0.24
Words > 6 letters
27.97
29.12
28.30
Drives
8.41
6.95
8.00
Dictionary words
76.81
75.33
76.40
Affiliation
1.57
1.25
1.48
Function Words
45.94
46.86
46.20
Achievement
1.54
1.06
1.40
Total pronouns
6.30
6.20
6.27
Power
4.60
4.05
4.44
Personal pronouns
2.70
1.54
2.38
Reward
0.77
0.51
0.70
1st pers singular
0.16
0.06
0.14
Risk
0.97
0.64
0.88
1st pers plural
0.50
0.38
0.46
Time orientations
2nd person
0.16
0.09
0.14
Past focus
3.67
3.77
3.70
3rd pers singular
0.95
0.41
0.80
Present focus
6.42
6.40
6.42
3rd pers plural
0.94
0.60
0.84
Future focus
1.12
1.00
1.09
Impersonal pronouns
3.59
4.66
3.89
Relativity
13.77
13.05
13.57
Articles
10.30
11.06
10.51
Motion
1.70
1.78
1.72
Prepositions
15.81
16.09
15.89
Space
7.73
8.02
7.81
Auxiliary verbs
6.53
7.21
6.72
Time
4.39
3.21
4.06
Adverbs
3.23
3.83
3.40
Personal Cocnerns
Conjunctions
4.44
3.37
4.14
Work
3.77
3.32
3.64
Negations
1.17
1.16
1.17
Leisure
0.76
1.29
0.91
Other Grammar
Home
0.34
0.31
0.33
Common verbs
11.01
11.03
11.02
Money
1.48
0.67
1.25
Common adiectives
3.97
3.62
3.87
Religion
0.38
0.35
0.37
Comparisons
2.02
Death
0.38
0.43
0.39
Interrogatives
1.13
Informal Language
0,23
0.19
0.22
Numher
2.13
2.02
Swear words
0.01
0.01
0.01
Quantifiers
1.62
1.24
1,51
Netspeak
0,06
0.09
0.07
Psychological Processes
Assent
0.04
0.04
0.04
Affective processes
4.67
3.97
4.47
Nonfluencies
0,08
0.06
0.07
Positive emotion
2.12
1.23
1.87
Fillers
0.02
0.00
0.02
Negative emotion
2.49
2.72
2,55
Punctuation
Anxiety
0.38
0.22
0.34
Total Punctuation
15.09
13.87
14.75
Anger
1.01
0.98
1.00
Periods
5.26
5.31
5.28
Sadness
0.37
0.21
0.33
Commas
4.91
4.14
4.70
Social processes
6.82
4.84
6.26
Colons
0.36
1.08
0.56
Family
0.15
0.09
0.13
Semicolons
0.05
0.03
0.04
Friends
0.16
0.09
0.14
Question marks
0.13
0.03
0.11
Female references
0.34
0.19
0.30
Exclamation marks
0.09
0.01
0.07
Male references
0.85
0.41
0.72
Dashes
0.69
0.70
0.69
Cognitive processes
8.26
7.69
8.10
Quotation marks
2.09
1.52
1.93
Insight
1.52
1.36
1.48
Apostrophes
0.97
0.57
0.86
Causation
1.72
1.91
1.77
Parentheses
0.25
0.37
0.28
Discrepancy
0.96
0.47
0.82
Other punctuation
0.28
0.12
0.23
Tentative
1.36
1.13
1.30
Certainty
1.11
1.02
1.09
Differentiation
2.34
2.26
2.32
Table 8: Complete LIWC2015 listings
continued...


Reviews:
[{"review_id": "9BGdyTKRqf", "reviewer": "Reviewer_Bm7V", "summary": "", "questions": "[A] Which tokenizers did you use (for which languages)? Was it Spacy (as line 90 seems to indicate)? \n\n[B] Can you provide a gist of the factuality for the collected articles (though I understand this is only a short paper)? Do you have a rough idea (from your manual annotations) what kind of articles are present with respect to factuality (e.g., mis-citing reliable sources vs making up claims)? Maybe some automated fact checking systems could provide some initial insights [1]. \n\nSuggestions:\n\n* The number of sentences per article would also be interesting to add (e.g., in Table 8).", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Manual annotations of the articles may provide interesting further analysis, but this is out of the scope for this paper and has prohibitive costs given the large volume of articles. \n\nThank you for your suggestions and corrections. We agree that the number of sentences is an interesting metric, we will add this if the paper is accepted.\n\n---\n\n**A**: The mean token count statistics for all languages used the Spacy tokenizer. We will ensure this is stated explicitly.\n\n**B**: From the fact-checks published on the two sites, and our impressions from working with the content, it appears that the sites publish a mixture of outright false, misleading (e.g. mislabelling media), and some (particularly on RRN) selective true content that is advantageous to their agenda. The sites appear to follow a ‘quantity over quality’ approach, as PolitiFact identified instances of mutually conflicting stories.\n\nWe agree that trialling this data on automated fact-checking systems would be interesting, but our initial impressions are that it may struggle with this data. Given the subject matter, identifying the truth is difficult and likely requires extensive domain knowledge and investigation, and as the true content appears to be written in the same style as the false content, approaches that look at credibility signals will be ineffective. Due to these difficulties, we believe a proper evaluation would be overly complex and out-of-scope for this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents an interesting corpus and analysis of articles crawled from propaganda websites. Upon publication, the presented corpus would offer a wide range of research possibilities.", "reasons_to_reject": "Some manual annotations (for instance, on the factuality of the articles) would have been interesting; however, the presented work fits the short paper track very well.", "questions_to_authors": "[A] Which tokenizers did you use (for which languages)? Was it Spacy (as line 90 seems to indicate)? \n\n[B] Can you provide a gist of the factuality for the collected articles (though I understand this is only a short paper)? Do you have a rough idea (from your manual annotations) what kind of articles are present with respect to factuality (e.g., mis-citing reliable sources vs making up claims)? Maybe some automated fact checking systems could provide some initial insights [1]. \n\nSuggestions:\n\n* The number of sentences per article would also be interesting to add (e.g., in Table 8).", "paper_topic_and_main_contributions": "This paper presents a novel corpus with ~3,5k unique articles collected from Reliable Recent News and War On Fakes (between March 22-23) and their translations across seven languages (14k in total). After pre-processing, the authors provide analysis with respect to the present topics and frequent n-grams; then proceed with analyzing the publication dates as well as respective patterns. Finally, they conduct an analysis on a psycho-linguistic level using LIWC2015.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Guo, Zhijiang, Michael Schlichtkrull, and Andreas Vlachos. \"A survey on automated fact-checking.\" Transactions of the Association for Computational Linguistics 10 (2022): 178-206.", "Typos_Grammar_Style_and_Presentation_Improvements": "* line 148: add (Appendix) to Figure 2 to improve navigation."}, "extra_text": {}}, {"review_id": "03YFWWAWuf", "reviewer": "Reviewer_Rv3e", "summary": "", "questions": "Question A: Other than backdating, did you work with other metadata like author names, categories and tags, or titles?\n\nQuestion B: You mention holidays in section 3.2, did you witness other overlapping holidays?\n\nQuestion C: The article lacks a section on future work, yet other text analyses are possible, do you have experience with language models for example?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "It’s unclear which part of the content analysis the reviewer is referring to, but we believe that our analysis techniques are generalisable to other sites (e.g. the n-grams and topics). The only part that may be difficult to generalise is the article backdating detection, which requires the site to use a platform that in some way ‘leaks’ the creation order of articles. However, this will work on any site that is powered by the WordPress content management system, which is widely used in general, and appears popular for disinformation sites due to its ease of setup. Our tool to extract translations currently supports the internationalisation plugin used by the sites, but is designed to be easily extended to others. The Cyrillic detection is the part that is most specific to these sites due to their place of origin, but similar ‘unicode traps’ exist for other languages.\n\nIn our experience, the topic modelling process was in fact relatively straightforward. The library is well documented and in our experience did not require significant tuning of hyperparameters, particularly as the HDBSCAN algorithm automatically determines the number of clusters to make. It would be, however, difficult to compare topics across datasets, as the topics that form are dependent on a vast number of factors, and thus different and potentially incomparable topics would be produced for different datasets.\n\nWe agree that other methods of textual analysis would be interesting. Due to space limitations, we had to select only some of the analysis we performed, and due to the volume of data some methods had to be avoided due to the labour required to properly analyse them. For example, we considered NER to identify sentiment towards actors in the war in Ukraine, and whether entities mentioned correlated with languages available, but it was prohibitively time-consuming to properly annotate the nations and groups each entity belonged to. We chose LIWC over other sentiment detection methods because it provides extremely fine-grained results and can serve as a linguistic profile for comparison to other datasets, and is unaffected by document length.\n\nThank you for the suggestion to use Scattertext. We don’t believe there is sufficient space to incorporate it in this paper, but will consider it in future work.\n\nThank you for your corrections and suggestions for clarification, we will address these.\n\n---\n\n**A**: We did explore the categories and tags of the articles, but it appears little care is taken by the sites’ operators in using them, and as a result they are quite noisy. The author information of the sites is limited, using either generic (‘Admin’) or random (‘UiXnZyvH’) names and no useful profile data. Closer analysis of author data was conducted on a preliminary scrape - we hypothesised there may be a variation in post language or posting times for each author, but none was found. All metadata is included in the dataset, permitting future analysis by others.\n\n**B**: There appear to be no other periods of time with significant reductions in activity, as there was for the New Year and Christmas period. We suspect this is because this period is 8 continuous days, whereas other public holidays are single, isolated days.\n\n**C**: We will add a section on additional potential future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Scientifically sound article, relevant cues on disinformation, interesting discussion on the methods and their limitations, as well as on ethical issues.\n\nDecisions made on the tools are clearly described and quite valuable.", "reasons_to_reject": "Part of content analysis is specific to the websites of interest and cannot be generalized easily.\n\nBeyond topic analysis which involves a great deal of fine-tuning and may not be easy to reproduce on other datasets, the article lacks more refined textual analysis, i.e. proper sentiment detection other than word lists (section 3.5) and better word grouping or filtering methods than n-gram analysis (section 3.7).", "questions_to_authors": "Question A: Other than backdating, did you work with other metadata like author names, categories and tags, or titles?\n\nQuestion B: You mention holidays in section 3.2, did you witness other overlapping holidays?\n\nQuestion C: The article lacks a section on future work, yet other text analyses are possible, do you have experience with language models for example?", "paper_topic_and_main_contributions": "The article introduces a corpus on state-backed disinformation, provides a quantitative overview of the resource, and evaluates different detection approaches.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "It could be interesting to mention or use Scattertext (Kessler 2017) to compare the sources."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 099: you could explain the curse of dimensionality for the sake of clarity.\n\nLine 217: sub metrics → sub-metrics?\n\nThe sections on backdating (2.3 and 3.6) could be merged into a single one. Information in Table 5 on article backdating could be added to the article if there is enough room.\n\nGenerally speaking, sections on article text and article metadata could come in groups to make the results easier to follow."}}, {"review_id": "YD3fDMVeKn", "reviewer": "Reviewer_q5Fz", "summary": "", "questions": "A) Something that remains unclear to me is how are the articles labelled with topic, as it is the sentences the ones which are clustered. The article mentions that “Each article is then labelled with the unique set of clusters assigned to its sentences.”, so I am assuming that each article has a different number of labels and that some articles might actually cover a high number of topics.\n\nB) The dataset contains 3,447 articles, which, translated into different languages, make 14,053. Those original 3,447 articles, are they in the same language? I can’t find that figure in Table 1, so I understand they are originally written in different languages, is that right? Do these sites have an official or primary language and translations in others or articles can be written, originally, in different languages?\n\nC) I feel the topic detection experiment design might have some issues, if more than half of the sentences of a dataset are considered outliers. Could you explain this situation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We acknowledge that the background information does not include analysis of the propaganda detection field. Given limited space, we decided to focus on analysing prior work on these specific sites and comparing our work to it. Although there are prior datasets for propaganda detection, we are not aware of any that are parallel multilingual, and thus limiting direct comparisons. We believe a survey of propaganda detection techniques would not be out of this paper's scope, as it does not itself propose a method. However, we agree that the paper would be improved with an overview of applicable problems, such as propaganda detection, which would include an explicit comparison to prior datasets. We will add this to the paper.\n\nWe will add an example of an article to better illustrate the type of content.\n\nThank you for your grammar corrections and suggestions for clarification, we will address these.\n\n---\n\n**A**: This is correct, a given article’s labels (0 or more) are the combined set of its sentences labels (1 per sentence). Most articles had 1-3 topics assigned, but a small number were assigned many more  (see figure 1) -- typically very long articles that appear to have been comprised of previously published articles. We will clarify this in the paper.\n\n**B**: Around 60% of the 3,447 articles were first published in English, and around 30% were published in English as a subsequent language. Of the ~10% that were never published in English, they almost all have no translations set, some were manually checked and the translations do exist but were mistakenly not linked by the site operators.\n\nThe fact that nearly all articles are available in English and that it is the default language on both sites, suggests that they intend English to be the primary language. However, given the presence of Cyrillic characters in some English articles that are most likely to be caused by mistakes when translating, it appears that at least some of the English content is originally written in Russian.\n\nPlease note that, whilst reviewing statistics for this answer, we identified an error in the original version of the paper. On line 172 it states 2.4% of posts are not available in English. This figure was calculated incorrectly, 2.4% refers to the percentage of the 14,053 translations that do not have an English version. It should be the percentage of the 3,447 articles that do not have an English version, which is 9.1%. We will correct this in the paper, as well as add the above clarifications.\n\n**C**: Only very short sentences are removed before clustering so some of the sentences will be irrelevant or contain little useful information, and some sentences may be on specific topics that are too distinct to cross the 25 size threshold to form a cluster. Whilst it is possible to assign outliers to nearby clusters, or further tweak clustering parameters to minimise them, this may force outliers into irrelevant topics or cause the creation of too many extremely specific small topics and a reduction of broader, more useful topics.\n\nAs we aggregate sentence-level topics at article level, the presence of outliers at sentence level is less significant. Intuitively, an article on a given topic will have many ‘chances’ to be assigned correctly, as it will have many sentences on that topic. It is better for a sentence to incorrectly be considered an outlier (because another sentence in the same article will likely be correctly classified) than to be assigned to the wrong cluster (because the whole article’s labels will include this incorrect cluster).\n\nWhen aggregated, only 78 articles were outliers (i.e. all sentences within the article were outliers). We manually reviewed these articles: many are extremely short texts where the article was primarily a video or image, require external knowledge to interpret, or are on specific one-off news stories.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an interesting paper. I think the dataset can be very valuable not only for the NLP community, but also for analysis from other disciplines, such as discourse analysis, sociology, politics or journalism.  I found the analysis interesting, although more from a sociological, cultural perspective than from an NLP point of view.", "reasons_to_reject": "There is no background analysis. There is much work done in propaganda detection, both in the publication of linguistic resources, such as datasets, and in the development of models to detect it. In addition, propaganda has been studied for many years from other disciplines. All this background needs to be accounted for in a paper that presents a new resource for automatic propaganda detection.\n\nExamples of the dataset would be very helpful in a paper that introduces a new dataset, as the reader learns about your work, but is not able to see it.", "questions_to_authors": "A) Something that remains unclear to me is how are the articles labelled with topic, as it is the sentences the ones which are clustered. The article mentions that “Each article is then labelled with the unique set of clusters assigned to its sentences.”, so I am assuming that each article has a different number of labels and that some articles might actually cover a high number of topics.\n\nB) The dataset contains 3,447 articles, which, translated into different languages, make 14,053. Those original 3,447 articles, are they in the same language? I can’t find that figure in Table 1, so I understand they are originally written in different languages, is that right? Do these sites have an official or primary language and translations in others or articles can be written, originally, in different languages?\n\nC) I feel the topic detection experiment design might have some issues, if more than half of the sentences of a dataset are considered outliers. Could you explain this situation?", "paper_topic_and_main_contributions": "This paper treats an interesting topic and introduces a valuable, very timely resource, namely a publicly available dataset of state-backed propaganda, enriched with linguistic, topic and temporal analysis. They also share an open-source toolkit for processing information from the source site and translations, such as n-gram frequency, distribution of languages in which the articles are translated or articles shared by the two sites.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "There is no background analysis. There is much work done in propaganda detection, both in the publication of linguistic resources, such as datasets and in the development of models to detect it. In addition, propaganda has been studied for many years in other disciplines. All this background needs to be accounted for in a paper that presents a new resource for automatic propaganda detection.", "Typos_Grammar_Style_and_Presentation_Improvements": "Some small issues I detected that you might want to consider:\n\n- Line 98-99: This is necessary to avoid the ‘curse of dimensionality’. → Needs citation and explanation\n- Table 1: Which are the languages? Not all abbreviations are easy to understand for me. \n- 2.4. n-gram Frequency → I assume this is for each website, but it is not specified.\n- Section 3.2. refers to Figure 2, which is not included in the paper, but in the annex. Same with Table 6. \n- 2.3 and 3.6. are both “Article backdating”"}}]

Author rebuttal:

Meta-review: This submission offers a straightforward short dataset paper, which collects 3,447 articles from two state-backed fake news websites (WarOnFakes and Reliable Recent News) using the WordPress API. The authors also perform rudimentary exploratory analysis on the dataset, by analyzing topics and article frequencies. Additionally, they analyze the dataset by comparing the distributions of LIWC words on the fake news sites to the New York Times, and counting the most frequent n-grams by month. 

All reviewers offered positive assessments of the paper, although the reviews were fairly brief (perhaps because the paper was very straightforward). All three reviewers awarded an excitement score of 4, and seemed enthusiastic about the work. One noted that the submission offered an “interesting corpus” which would “offer a wide range of research possibilities.” They also pointed out that the paper “fits the short paper track very well.” Another suggested that “I think the dataset can be very valuable” and that they “found the analysis interesting, although more from a sociological, cultural perspective than from an NLP point of view.”

Because all reviewers were excited about the contribution, the only questions surrounding this submission have to do with soundness. One reviewer awarded a soundness score of 4, and the others awarded a soundness score of 3.  However, looking more closely, the reviews did not surface any major issues surrounding soundness. Therefore, there do not appear to be concerns about the soundness of this work.
 
- Reviewer Bm7V initially awarded a soundness score of 3, but changed it to 4 after the authors answered a question about tokenization during the rebuttal period.
- Reviewer q5Fz awarded a 3 for soundness, but did not list any concerns about soundness in their reasons to reject. Instead, they suggested that the paper might be improved if the authors include a discussion of prior scholarly study of propaganda. 
- Finally, reviewer Rv3e awarded a soundness score of 3, and said they were concerned about fine-tuning during the neural topic modeling analysis. However, based on their comment, it seemed as if they actually may have been concerned about how the authors performed hyperparameter tuning. The authors replied to Rv3e with information about hyperparameter selection, which seemed to satisfy Rv3e. Rv3e thanked the authors for the additional details but did not change their score.
Decision: Accept-Main