Multilingual Coarse Political Stance Classification of Media. The Editorial Line of a ChatGPT and Bard Newspaper
Authors: Cristina España-Bonet
Venue: EMNLP 2023 Findings (2023)
Keywords: political bias
news
stance classification
instruction-following language models
chatGPT
Bard

Abstract: Neutrality is difficult to achieve and, in politics, subjective. Traditional media typically adopt an editorial line that can be used by their potential readers as an indicator of the media bias. Several platforms currently rate news outlets according to their political bias. The editorial line and the ratings help readers in gathering a balanced view of news. But in the advent of instruction-following language models, tasks such as writing a newspaper article can be delegated to computers. Without imposing a biased persona, where would an AI-based news outlet lie within the bias ratings? In this work, we use the ratings of authentic news outlets to create a multilingual corpus of news with coarse stance annotations (Left and Right) along with automatically extracted topic annotations. We show that classifiers trained on this data are able to identify the editorial line of most unseen newspapers in English, German, Spanish and Catalan. We then apply the classifiers to 101 newspaper-like articles written by ChatGPT and Bard in the 4 languages at different time periods. We observe that, similarly to traditional newspapers, ChatGPT editorial line evolves with time and, being a data-driven system, the stance of the generated articles differs among languages.

Multilingual Coarse Political Stance Classification of Media.
The Editorial Line of ChatGPT and Bard Newspaper
Cristina España-Bonet
DFKI GmbH, Saarland Informatics Campus
Saarbrüken, Germany cristinaeddfki.cmm
Abstract
information if their outputs are not fact checked. A second big issue with systems based on language models (LM) the fact that they might repro- duce the biases present in the training data (Nav- igli et al., 2023). Biases range from cultural miss- representation due to data imbalance to offensive behaviour reproduced from written texts. LMs are finetuned into ILMs either in supervised way ing input-output pairs and an instruction (Wei al., 2022; Wang al., 2022, 2023b) or with reinforce- ment learning from human feedback (Ouyang al.. 2022; Nakano et al., 2021). In both cases, the fine- tuning should help removing bias. But neutrality is something very difficult to achieve, also for the humans that generate the supervisory data. The finetuning phase might therefore over correct the original biases introduce new ones. For meth- ods that generate the supervision data with the LM itself, the original biases might be inherited.
Neutrality is difficult to achieve and, in politics, subjective. Traditional media typically adopt an editorial line that can be used by their po- tential readers as an indicator of the media bias. Several platforms currently rate news outlets according to their political bias. The editorial line and the ratings help readers in gathering a balanced view of news. But in the advent of cerruutttoo-ollowiwii language models, tasks such writing newspaper article be dele- gated to computers. Without imposing a biased persona, where would an Al-based news outlet lie within the bias ratings? In this work, we use the ratings of authentic news outlets to cre- ate multilingual corpus of news with coarse stance annotations (Left and Right) along with automatically extracted topic annotations. We show that classifiers trained on this data able to identify the editorial line of most un- seen newspapers English, German, Spanish and Catalan. We then apply the classifiers to 101 newspaper-like articles written by Chat- GPT and Bard in the languages different time periods. We observe that, similarly to traditional newspapers, ChatGPT editorial line evolves with time and, being data-driven sys- tem, the stance of the generated articles differs among languages.
We focus on a specific use of ILMs: the writing of newspaper articles. Journals and newspapers follow editorial line which is in general know to the reader. Besides, sites such AllSides, Media Bias Fact Check² (MB/FC), or Ad Fontes Media provide ratings about the political bias of (mostly USA) media sources and their quality with respect to factual information. With these ratings, consci- entious readers can make informed decisions about which media outlets to choose in order to get a balanced perspective. But what happens when jour- nalists use systems such as ChatGPT Bard to aid in their writing? As said above, humans also have biases, the danger lies in being unaware them, as they might affect the user ssea''/reader's perspective (Jakesch et al., 2023; Carroll et al., 2023). Chat- GPT already warns its users about misinformation. However, the political bias, if any, is not known apart from the subjective perception that user has.
1 Introduction
Instruction-folloiing language models (ILMs) are omnipresent. Their use not so extended as that of search engines yet, but due to the availability and high quality of systems and models such as Alpaca (Taori et al., 2023), Bard (Google, 2023), BLOOMZ and mTO (Muennighoff et al., 2023), ChatGPT (OpenAI, 2023), Llama 2-chat (Touvron et al., 2023), Koala (Geng et al., 2023), their use is expected to be more common in the near future. These models face several problems being the most relevant the lack of trustworthiness (van Dis et al., 2023; Huang et al., 2023; Wang et al., 2023a). They are not ready to be used as a source of reliable
hhttps://ww/ httt::://www.allsddee com/
com/ hnttpss//aaffontemmdia.cmm

---

We address the question above for articles gen- crated by ChatGPT and Bard in four languages: English, German, Spanish and Catalan. We do this in an automatic and systematic way with almost no human intervention so that the method can be eas- ily extended to new languages and other ILMs with few effort. We do not aim at classifying individ- ual articles with their specific bias, but to classify the media source (an ILM in this case) as Left or Right-oriented in a similar way as the media bias sites do for newspapers and other media outlets.
Spain and Catalonia. With the list of newspapers. their URL, and their stance, we use OSCAR, a multilingual corpus obtained by filtering the Com- mon Crawl (Ortiz Suárez et al., 2019; Abadji et al., 2021), to retrieve the articles. Appendix A lists the sources used in this work: 47 USA newspapers with 742,691 articles, 12 German with 143,200, 38 Spanish with 301,825 and 19 Catalan with 70,496. Topic Modelling. Not all articles have a bias, some topics are more prone than others. While the Sports section of newspaper is usually less prone to reflect political biases, the opposite hap- pens with the International section. We therefore use topics to select a subset of relevant training data for our binary classification. We do topic mod- elling on the articles extracted from OSCAR using Mallet (McCallum, 2002) which applies LDA with Gibbs sampling. We cluster the data in both 10 and 15 groups per language, roughly correspond- ing to the number of sections a newspaper has. The keywords extracted for each topic are listed in Appendix B. We choose articles that fall under the topics we label as International, Government, Law & Justice, Economy, Live Science/Ecology, and specific language-dependent topics such as Im- migration and Violence for English, Nazism for German, and Social for Spanish. The selection is done after the inspection of the keywords. For the final dataset, we do the union of the selected ar- ticles clustered to 10 and 15 topics. The process filters out 49% of the Spanish articles, 39% of the German and 31 % of the English ones.
2 Corpora Compilation
We approach our task as a classification problem with two classes: Left (IL) and Right (IR) political orientations. This is simplification of the real problem, where articles can also be neutral and there might be different degrees of biases. Previ- ous work relied on or 5 classes, always includ- ing the neutral option (Baly et al., 2020; Aksenov et al., 2021). In these works, data was manually annotated creating high quality training data but also limiting a lot the scope of the work in terms of languages and countries covered. When using the fine-grained classification scale, the authors ac- knowledge bad generalisation of the classifiers to new sources. On the other hand, Gareía-Díaz et al. (2022) and Russo et al. (2023) exclude the neutral class and work with a binary or multiclass Left- Right classifications of tweets from Spanish and Italian politicians respectively, but their work does not include longer texts. The binary classifica- tion might be justified as they worked with tweets, a genre where people tend to be more visceral and therefore probably more polarised. In our case, we need to be sure that the classifier generalises well to unseen sources and we stick to the 2-class task while minimising the number of neutral articles in training (see below).
Preprocessing and Cleaning. We discard ar- ticles with more than 2000 or less than 20 words before cleaning. Afterwards, remove headers, footers and any boilerplate text detected. This text has the potential to mislead neural classifier, as might encourage the classifier to learn to distin- guish among newspapers rather than focusing on their political stance. We select a newspaper per language and stance for testing and clean manually their articles. To create a balanced training corpus for each language, we randomly select a similar number of Left and Right-oriented articles from the remaining collection. This balanced dataset is divided into training and validation as shown in Table (top rows).
Distant Supervision. As far as we know, only a manually annotated newspaper corpus in En- glish (Baly et al., 2020) and another one in Ger- man (Aksenov et al., 2021) are available. We fol- low a different approach in the spirit of Kulkamn et al. (2018) and Kiesel et al. (2019). We do not manually annotate any article, but we trust All- Sides, MB/FC, Political Watch and Wikipedia (the latter only in cases where the information is not available in the previous sites) with their classi- fication of a newspaper bias. We extract this in- formation for newspapers from USA, Germany,
ChatGPT/Bard Corpus. We create multilin- gual dataset with 101 articles. For this, we define
"This implics selecting all the articles that are under a domain name of a news outlet, whether they are news or not.

---

English (USA)
German (Germany)
Spanish (Spain)
Catalan (Catalonia)
IL
R
Training
182,056 (756)
178,463 (768)
31,445 (550)
30,745 (384)
70,384 (874)
67,888 (806)
Validation
1,503 (723)
1,497 (678)
1,528 (570)
1,472 (374)
1.539 (878)
1,461 (842)
Newspaper
298 (731)
413 (487)
623 (278)
276 (734)
350 (844s)
518 (731) 2,105 (1,152)
800 (538)
CharGPTv02
101 (337)
101 (346)
ChaiGPTv03
101 (299)
ChatGPTv0S
101 (585)
101 (436)
101 (553)
101 (496)
ChaiGPTvUS
101 (x08:::770008h:4677)
101 (+08s:3211008::324)
101 (4008:45440088044444
101 (v08::378808h::655)
Bardv08
101 (v08::437708b::07))
101 (x08a:268/088::66)
101 (v08s:33Hv008:325)
101 (v08::331/008b:245)
Table 1: Number of articles (average word count in parentheses) divided as articles belonging a newspaper with Left (IL) and Right orientation (R). For testing, we use newspapers not seen in training or validation: Slate (L) and The National Pulse (IR for USA, My Heimat (L) and die Preußiscke Allgemeine Zeitung (R) for Germany, Mundo Obrero (L) and EI Diestro (R) for Spain and Vilaweb (L) and Diari de Tarragona (R) for Catalonia.
101 subjects including housing prices, abortion, to- bacco. Barak Obama, etc. and translate them man- ually into the 4 languages (see Appendix D). The subjects consider topics prone to have political stance such as those related to feminism, capital- ism, ecologism, technology, etc. We also include proper names of people in the 4 countries being considered, whose biography may differ depending on the political stance of the writer. These subjects are inserted into the template prompt (and its trans- lations into German, Spanish and Catalan): Write a newspaper article on [SUBJECT]en
of the network and the hyperparameter exploration per model are reported in Appendix F.
The Models. We train 4 models: 3 monolingual finetunings with the English, German and Spanish data, plus a multilingual one with the shuffled con- catenation of the data. All models are based on mul- tilingual embeddings (RoBERTa) finetuned either monolingually or multilingually. Notice that we do not train any model for Catalan. With this, we want to compare the performance of mono- and multi- lingual finetunings and explore the possibility of using multilingual models for zero-shot language transfer.
We prompt ChatGPT (GPT-3.5-Turbo) five times using the same subjects across four time periods. We generate the dataset with ChatGPT versions of Feb 13 (v02), Mar 23 (v03), May 24 (v05) and Aug 3 (v08); we cover the 4 languages simultaneously only with the last two. ChatGPTv05 generates sig- nificantly longer texts than the other ones with an article-oriented structure with slots to be filled with the name of the author, date and/or city. Multi- lingual Bard was available later, and we prompt it twice during the same period ChatGPTv8, Table 1 shows the statistics for this corpus.
Coarse Classification with Newspaper Arti- cles. Table summarises the results. All the mod- cls achieve more than 95% accuracy on the valida- tion set which is extracted from the same distribu- tion as the training data. In order to see how the models behave with unseen data, we calculate the percentage of articles that are classified as Left (LL and Right (R) in the test newspapers of Table 1. We perform bootstrap resampling of the test sets with 1000 bootstraps to obtain confidence intervals at 95% level. We do not expect all the articles of a newspaper leaning towards the Left to show clear characteristics of the Left, but given that there is no neutral class, we expect the majority of them to be classified as Left. A good result is not necessarily 100%-0% as this would not be realistic either. We consider that a newspaper has been classified having Left/Right political stance if more than 50% of its articles have been classified as such.
3 Political Stance Classification
The Network. We finetune XLMMRooERTa large (Conneau et al., 2020), a multilingual transformer basse masked LM trained on 100 lan- guages including the 4 we consider. The details
*More specific prompts did not lead to different styles for the first versions of ChaiGPT, for the last one added more information such as w..whhhout subheaders. to avoid excesive subsectioning and/or bullet points. Neither ChatGPT nor Bard did always follow properly the instruction. The dataset we provide includes the prompts we used.
These cases are boldfaced in Table
This is the behaviour we obtain for all the test newspapers but for the German Right-oriented newspaper: die Preußische Allgemeine Zeitung (PAZ). The German model is trained only on 12
"Prompted 14-21 August 2023 from Berlin for English and German and from Barcclona for Spanish and Catalan as, contrary to ChatGPT, the generation depends on the location.

---

English
German
Spanish
Catalan Muliilingua
Monolingual
Multilingual
Monolingual
Multilingual
Monolingual
Multilingual
R
L
R
L
R
L
R
R
IL
R
R
Val. Acc (%)
97.9
96.9
99.2
96.9
95.9
96.9
Classification (% of articles per stance)
Newspaper II.
82+-5
1844
81+5
19+4
87+-3
13+-2
65 1-4
35-44
55+-5
45+-5
61+5
39+-5
65--2
35+-2
Newspaper R
11::3
89:13
7=2
93:13
71:16
2916
65:16
35:55
1243
88:33
19 =3
81:4
1312
87:-2
ChatGPTv02
75:19
25:1:8
93:15
7::5
65:10
35:10
53:100
47:10
ChasGPTv03
97-14
34-3
69:99
31-9
CharGPTv0s
26:1:9
74:9
40:19
60:19
96:25
4::3
65:19
35:29
25:19
75:
26:1:9
74:88
71:9
29:19
ChaGGTTV88a
54::10
46:4
85-88
15:16
99-33
100::2
50:+10
50++
40:+10
60::10
50:+10
504.9
ChaiGPTvosh
85
1-5+6
100+-2
100+-2
51+-10
49-+10
36++10
644-9
474-10
53+110
Bardv08a
11
43-
75499
25:-8
82-18
1847
82488
18=7
74:99
26::8
35::9
654-9
66:99
344.9
Bardv08b
61=10
39:10
82:18
1817
81=8
19:7
90:17
10:55
74:99
26
44:10
56::10
68:99
3249
Table 2: (top) Accuracy of the finetuned models on the comesponning validation sets. (bottom) Percentage of articles classified as having a Left (I.) and Right (I) orientation (columns) for the test newspapers and the Bard/ChatGPT generated articles at four different time periods (rows). The majority stance is boldfaced.
newspapers to be compared to the 47 in English and 38 in Spanish. The incorrect classification might be an indication that diversity is key aspect for the final model performance. Multilinguality does not help and 65% of the PAZ articles are still classified as Left oriented. We also assess the effectiveness of the English model on the German data, two close languages. We acknowledge that the topics of the USA and German newspapers might differ lot, but the high diversity of the English training data could potentially compensate for this. The English model is able to correctly classify the German My Heimat as a Left-oriented newspaper (I.: 67.3%) and PAZ as Right-oriented one (R:: 58+5%)). We again attribute the difference to the German model being trained on a corpus lacking diversity. When we use the multilingual system, the dominant fac- tor distinguishing the outputs is the language itself rather than the stance. The addition of English data is insufficient alter the classification significantly. When use the English system, the language does not play a role any more and only the stance fea- tures are considered. When we apply the English model to the Catalan newspapers we do not obtain satisfactory results though 195+1% for the Left but 16.1.3% for the Right newspaper) showing that the relatedness across languages is important. The mul- tilingual model however properly detects the stance of the Catalan newspapers probably because it has been trained with heterogeneous corpus that in- cludes a related language (Spanish). We are able to perform zero-shot language transfer classification when we deal with close related languages.
We first focus on the English and Spanish models as the German one did not properly classify our test newspapers. The most relevant aspect no- tice in ChatGPT is the strong change in political stance between February (v02) and May (v05) fol- lowed by a movement towards neutrality in August (v08). We checked that this polarity change is not an effect of the length of the outputs -the major shallow change in the generated articles. The train- ing data in English has 5,730 L-6,988 R articles with 584<<length((ee (words) <224 <224 (similar to ChatG- PTv05 length) and 4,563 L-7,127 IR articles with 331 <lengthh<37 (similar ChatGPTv02). In both cases the number of articles is larger for the Right stance, but the prediction for ChatGPTv02 clearly points towards the Left, rejecting the hy- pothesis that length plays a role in the classifica- tion. A similar thing happens for Spanish. Ac- cording to our models, the May 24th version of ChatGPT in English and Spanish would have editorial line close the Right ideology, which differs from the ideology of the previous versions. Notably, this period corresponds to the time when ChatGPT experienced a performance drop in sev- eral tasks according to Chen et al. (2023). The German and Catalan outputs would still show an imprint from the Left ideology also in v05 but more diverse training data would be needed to confirm this with our monolingual models. It is interesting to notice that if we use the English monolingual model for German and Catalan, we still get the Left imprint (60+ 10% for German and 87+7% for Catalan). So we have indications that the political stance of ChatGPT depends on the language, which is not surprising in a data-driven system. The last
Coarse Classification with ILM--gnnrrtedd Ar- ticles. The bottom part of Table details the results.

---

There is only a version available for multilingual version, ChatGPTVO8, produces the most neutral texts, with only German clearly leaning towards the Left. The two generations, v08a and v08b, show that results are robust and are not tied to a particular generation.
Policy Research" reported that ChatGPT tended to give responses typical of Left-of-center politi- cal viewpoints for English (Rozado, 2023). The authors administered 15 political orientation tests to the ChatGPT version of Jan 9. Their results are consistent with our evaluation of the Feb 13 model. Finally, Motoki et al. (2023) performed a battery of tests based on PC to show that ChatGPT is strongly biased towards the Left. The authors do not state the version they use, but the work was submitted on March 2023, All these results are therefore before the move to the Right we detected in May.
Bard that covers our time frame. ? The variation between generations is larger for Bard than for ChatGPT but, comparing v08 versions, Bard points towards the Left in a more consistent way across languages. Bard's political orientation can also be determined by its answers to political test or quiz questions. The Political Compass (PC) site defines 62 propositions identify the political ide- ology wwith an Suropean/Westeni view- in two axes: economic policy (ight) and social pol- icy auathortarian-LLieeraaaaaa both in the range [-10,10]. Each proposition is followed by 4 alterna- tives: strongly agree, agree, disagree and strongly disagree. When prompted with the questionnaire. Bard's scores are (-6.50, -4.77) for English, (-8.00, -7.13) for German, (-5.75, -4.15) for Spanish and (-6.75, 4.56) for Catalan, where the first number corresponds to the economic policy and the second to the social policy. The results are in concordance with Table 2 and give an indirect validation of our method which does not rely on direct questions. 10
4 Summary and Conclusions
Media sources have an editorial line and an associ- ated bias. Getting rid of political biases is difficult for humans, but being aware of them helps us get- ting global view of news. Biases are sometimes clear and/or appear in form of harmful text, but sometimes are subtle and difficult to detect. These subtle hidden biases are potentially dangerous and lead to manipulation whenever we are not aware of them. In this work, we systematically studied the subtle political biases behind ChatGPT and Bard, those that appear without assigning any persona role (Deshpande et al., 2023). We showed that ChatGPT's orientation changes with time and it is different across languages. Between Feb and Aug 2023, ChatGPT transitioned from a Left to Neutral political orientation, with a Right-leaning period in the middle for English and Spanish. The evolution for Bard cannot be studied yet. Its cur- rent version of Aug 2023 consistently shows Left-leaning for the 4 languages under study. This bias is independent on the factual mistakes that the model generates, and should also be considered by its users. We provide models to regularly check the bias in text generations for USA, Germany and Spain, as well as in closely related political con- texts and languages using a zero-shot approach.
This kind of analysis is not possible with Chat- GPT any more as it refrains from expressing opin- ions and preferences, demonstrating the relevance of an approach that detects the leaning in a more indirect way. Also notice that these questionnaires are well-known and public, so it would be easy to instruct a LM to avoid the questions or react to its propositions in a neutral manner. Previous work used only political tests and questionnaires estimate CharGPT's orientation. Hartmann et al. (2023) used PC, 38 political statements from the voting advice application Wahl-O-Mat (Germany) and 30 from StemWijzer (the Netherlands) to con- clude that ChatGPT's ideology in its version of Dec 15 2022 was pro-environmental and left-libertarian study conducted by the Manhattan Institute for
As by-product of our analysis, created multilingual corpus of 1.2M newspaper articles with coarse annotations of political stance and topic. We show that distant supervision allows us to build meaningful models for coarse political stance clas- sification as long as the corpus is diverse. We make available this data together with the LMS genera- tions and our code through Zenodo (España-Boned 2023) and Github. 12
"Notice that the version we use does not officially support Catalan, but native speakers confirmed that generations are mostly correct and fluent with few grammatical mistakes.
"ht //www/ ciitcalclopssss. org/test (accessed between 13th and 20th August 2023)
"The Spanish questionnaire was translated into Catalan, as the questionnaire was not available.
10 "Even though, similarly to people, it is possible for an ILM to say one thing (chose an option for proposition) and act
11 conservative think tank according to Wikipedia. ht Ppttps:/ / thub. comm http//ristinee/edee ransfoorrrrerrrrrrrrrrr
(write a text) in an inconsistent way.

---

Limitations
References
Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Ro- mary, and Benoit Sagot. 2021. Ungoliant: An op- timized pipeline for the generation of very large- scale multilingual web corpus. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9) 2021. Limerick, 12 July 2021 (Online-Event), pages 1-9, Mannheim. Leibniz- Institut für Deutsche Sprache.
We are assuming that All media sources have ed- itorial line and an associated bias, and we treat the ILM as any other media source. We do not consider the possibility of a ChatGPT or Bard article being unbiased. This is related to the distant supervision method used to gather the data that currently al- lows for a binary political stance annotation. Since manually annotating hundreds of thousands of ar- ticles with political biases in truly multilingual setting seems not possible in the foreseeable future, we decided to implement a completely data-based method and study its language and culture transfer capabilities.
Dmitri Aksenov, Peter Bourgonje, Karolina Zaczyn- ska, Malte Ostendorff, Julian Georg Rehm. 2021. Fine-graine classification of political bias German news: A data set and initial experiments. In Proceedings of the 5th Workshop Online Abuse and Harms (WOAH 2021), pages 121-131, Online. Association for Computational Lin- guistics.
and
Using distant supervision for detecting the po- litical stance at article level is delicate topic though. First, because the same newspaper can change ideology over time. Second, and this is more related to the content of an individual article, non-controversial subjects might not have a bias. Even cases where bias exists, there is a spec- trum ranging from the extreme Left to the extreme Right, rather than a clear-cut division between the two ideologies.
Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020, We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4982-4991 Online. Association for Computa- tional Linguisties.
Micah Carroll, Alan Chan, Henry Ashton, and David Krueger. 2023. Characterizing Manipulation from Al Systems. arXiv preprint arXiv:2303.09338
In order to quantify and if possible mitigate the current limitations, we plan to conduct stylis- tic analysis of the annotatdd corpora (Baly et al., 2020; Aksenov et al., 2021) and compare to our attit annotated corpus. As a follow-up of this work, we will perform a stylistic analysis of the ILM -LM-generated texts too as a similar style between the training data and these texts is needed to ensure good generalisation and transfer capabilities.
Lingjiao Chen, Matei Zaharia, and James Zou, 2023, How is ChatGPT's behavior changing over time? arXiv preprint arXiv:2307.000 0.9099.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross lingual representation learning scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Ethics Statement
Ameet Deshpande, Vishvak Murahari, Tanmay Rajpuro- hit, Ashwin Kalyan, and Karthik Narasimhan. 2023, Toxicity in ChaiGPT: Analyzing Persona-assigned Language Models. arXiv preprint wwXiv:2304.05335
We use generative language models, ChatGPT and Bard, to create our test data. Since we deal with several controversial subjects (death penalty, sexual harassment, drugs, etc.) the automatic generation might produce harmful text. The data presented here has not undergone any human revision. We analyse and provide the corpus as it was generated, along with the indication of the systems version used.
Cristina España-Bonet, Bonet. 2023. Multilingual Coarse Po- litical Stance Classification of Media. The Editorial Line a ChatGPT and Bard Newspaper. Dataset on Zenodo, vl.0.
José Antonio Garefa-Díaz, Ricardo Colomo-Palacios, and Rafael Valencia-García. 2022. Psychographic Traits Identification Based on Political Ideology: An Author Analysis Study on Spanish Politicians' Tweets Posted in 2020. Future Gener, Comput. Syst., 130(C):59-74.
Acknowledgements
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wal- lace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: Dialogue Model for Academic Re- search. Blog post.
The author thanks the anonymous reviewers for insightful comments and discussion. Eran dos if's

---

Google, 2023. Bard [Instruction- toolloictionfollooiing large lan- guage model].
et al. 2021. WebGPT: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:21
Jochen Hartmann, Jasper Schwenzow, and Maximil- ian Witte. 2023. The political ideology of conversa- tional AI: Converging evidence on ChatGPT's pro- environmental, left- libertarian orientation. arXiv preprint arXiv:2301.01 00768..
Roberto Navigli, Simone Conia, and Bjöm Ross. 2023. Biases in Large Language Models: Origins, Inven- tory, and Discussion. J. Data and Information Qual- ity, 15(2).
Xiaowei Huang, Wenjie Ruan, Wei Huang, Gaojie Jin, Yi Dong, Changshun Wu, Saddek Bensalem, Ronghui Mu, Yi Qi. Xingyu Zhao. al. 2023, A Survey of Safety and Trustworthiness of Large Lan- guage Models through the Lens of Verification and Validation. arXiv preprint 2305.
OpenAl. 2023. ChatGPT [Instruction-folloving large language model].
Pedro Javier Ortiz Suárez, Benoit Sagot, and Lau- rent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low re- source infrastructures. Proceedings of the Work- shop Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9-16, Mannheim. Leibniz-Institut für Deutsche Sprache.
Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023. Co-writing with opinionated language models affects users' views. In Proceedings of the 2023 CHI Conference on Hu- man Factors in Computing Systems, pages 1-15, New York, NY, USA. Association for Computing Machin- ery.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages Curran Associates, Inc.
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Em- manuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. SemEval- 2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 829-839, Minneapolis, Minnesota, USA. Association for Computational Lin- guistics.
David Rozado. 2023, Danger in the Machine: The Per- ils of Political and Demographic Biases Embedded in AT System. Manhatan Institue, Issue Brief, pages 1-16.
Vivek Kulkarni, Junting Ye, Steve Skiena, and William Yang Wang. 2018 Multi-view models for political idcology detection of news articles. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3518- 3527, Brussels, Belgium. Association for Computa- tional Linguistics.
Daniel Russo, Salud María Jiménez-Zafra, José Antonio Garefa- Tommaso Caselli, L. Alfonso Ureña- López, and Rafael Galencia-Garche 2023. PoliticIT at EVALITA 2023: Overview of the Political Ideol- ogy Detection in Italian Texts Task. In Eighth Eval- uation Campaign of Natural Language Processing and Speech Tools for Italian 2023, number 3473 in CEUR Workshop Proceedings, Aachen.
Andrew Kachites McCallum. 2002. MALLET: A Machine Learning for Language Toolkit. http://mallet.cs.uaasseee
Fabio Motoki, Valdemar Pinho Neto, and Victor Ro- drigues. 2023. More human than human: Measuring ChatGPT political bias. Public Choice.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang. and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Inslloctionnfollooinng LLaMAA model. https: //github. wtt cow.ub.boomttttt--laeee Lab/ http//stffordaaappee
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hai- ley SchoeTkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji. Khalid Almuharak, Samuel Al- banie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generaliza- tion through multitask finetuning. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991-16111 Toronto, Canada. Association for Computational Linguistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu. Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller. Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly-
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders,

---

bog, Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Niyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.092:
Eva M. van Dis, Johan Bollen, Willem Zuidema, Robert van Rooij, and Claudi I.. Bockting. 2023. ChatGPT: five priorities for research. Nature, 61447947):244-266
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong. Ritik Dutta, Rylan Schaeffer, al. 2023a. DecodingTrune Trust: Comprehensive Assessment of Trustworthiness GPT Models. arXiv preprint arXiv:2306.11698.
Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah Smith, Daniel Khashabi, Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1 133881 Toronto, Canada. Association for Computational Linguistics.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phami Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Huper-Naturallnstrueree tion via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference Empiri- cal Methods in Natural Language Processing, pages 5085-5 109, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Generaliza-
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022. Finetuned language mod- els zero-shor learners. In International Confer- ence on Learning Representations.

---

A
Newspapers in OSCAR 22.01
Language
Leaning
Newspaper
URL
#Articles
Word average
L
ABC News
abenews.ge
thenews.gn.om
28101
659
L
AlterNet
www.alterett.ogg
1228
1680
L
Associated Press News
apnews.com
47090
679
L
Axios
www.axios.com
2119
733
L
Buzzfeed News
www.buzzeednnews.com
11025
885
CBS News
Www.chsnews.com
19761
534
L
CNN
www.cnn.com
15273
712
edition.cnn.com
19204
772
e7
Democracy Now!
www.demmocracyoow.org
1481
1213
HuffPost
www.huffpost.com
45721
961
en
preview. www.hfffost.com
181
1097
C7
staging. www.hufffost.coo
266
1142
en
Insider
www.insider.com
3705
1047
en
Mother Jones
www.motterrjones.co
12332
em
MSNBC News
www.msnbe.com
en
NBC News
www.nbcnews.com
38620
em
NPR
www.npr.org
45517
793
em
archive.nytimes.co
524
913
em
Politico
www.politico.com
30842
67
Slate
www.slate.com com
674
761
C76
L
The Atlantic
www.theatlantic.com
41816
750
C7
L
The Daily Beast
www.theduilybeasst
www
27771
806
C76
L
The Economist
Ww
www.economtist.com
38672
822
e7
L
The Intercept
theintercept.com
17766
1328
L
The New Yorker
I
www.newyorker.ccm
18698
688
e7
L
The New York Times
www.nytimes.com
2382
446
L.
The Washington Post
www.wasshiggtonpost.com
11944
1009
L
USA Today
www.usatoday.com
4526
1308
L
Vox
www.vox.comt
6746
934
R
American Greatness
amgreatnesss.omm
2405
849
R
American Thinker
www.americantlinker.com
9891
805
R
Breitbart News Network
www.breitbart.com
18762
651
R
ConservativeHQ
www.conserratticchg.com
1685
455
Fox News
www.foxnews.com
25074
770
R.
Info Wars
www.infowars.com
127
574
Life Action
www.livcaction.org
3095
740
R.
National Review
www.nationalreview.com
7251
946
Reason
reason.com
23553
489
R.
The American Conservative
www.
www.theameericanconeevaiiieeeeee
1228
935
I.
The Blaze
www.theblaze.com
34
1291
R.
The Daily Caller
dailycaller.com
26125
449
R.
delmeclllclll amp.dail
61
540
The Daily Wire
www.dailywire.com
7143
603
The Epoch Times
www.thecpochhimes.com
27553
695
The Federalist
thefederallis.com
7759
835
em
Gateway Pundit
www.thegatewaayandi..cm
5232
National Pulse
thenationalpuuse..oo
481
en
Washington Beacon
freebeacon.com com
7310
en
Washington Times
www.washingtontiies.com
19486
en
Spectator
spectator.org
14903
en
Washington Examiner
m.
www.ashiggonexxminee.comm
en
WND
www.wnd.com
21099
de
L
Die Zeit
www.zeit.de
12
4211
de
Die Tageszcitung
taz.dc
40253
de
blogs taz.de
1336
de
genossenschaft.taz.de
482
News
www.dw.com
2290
1075
Junge Welt
www.jungewelt.de
7386
My Heimat
www.staz.de
2499
Neues Deutschland
www.nd-aktuull..t
6608
Süddeutsche Zeitung
www.sucddecutsche.d
www.bild.de
24459
Frankfurter Allgemeine Zeitung
www.faz.net
12340
Continued on next page

---

Language
Leaning
Newspaper
URL
#Articles
Word average
de
R
Die Welt
www.welt.de jungefreiheii.de
9546
553
de
R
Junge Freiheit
8379
496
de
R
Preußische Allgemeine Zeitung
paz.de
385
945
L
Cuarto poder
www.cuarlopoder..s
3105
1353
L
De Verdad digital
deverdaddigital.com
3422
1071
L
Diario progresista
www.dlariopnoresistt.es
217
599
L
Digital Sevilla
digitalsevilla.com
751
494
L
clcomunista.net
elcomunista.net
736
1974
L
elDiario.es
www.eldiario.es
17846
1887
L
EI Obrero
elobrero.es
2856
1094
L
El país
elpais.com
49923
903
periódico
www.clperiodico.com
26940
647
L
plural
www.elplural.com
$70
759
siglo de Europa
elsiglodeuropa.er
207
1596
HuffPost
www.hurfingtomposs..s
9967
1097
La República
larepublica.es
125
842
Los Replicantes
www.losreplicantes.cmm
5398
68
Mundiario
www.mundiario.com
16299
Mundo Obrero
www.mundnohrero.es
527
764
68
Postdigital
postdigital.es
209
68
Público
www.pablico.es
9126
1097
68
Adelante España
adelantecsspnnacco
178
731
68
Altavoz sucesos
altavozdesucesos.er
136
68
Disidentia
disidentia.com
167
967
68
R
Confidencial
www
www.clconfidencial.co
23560
1037
68
corren Andalucía
elcorrooweh.es
4940
68
R
correo de España diestro
elooreodeesppna.comm
1896
1343
68
RR
www
www.eldiestrro.es
3046
1321
68
R
Imparcial
www.elimparciill.s
ww
4777
518
68
RR
independiente
www.elindependiente.com
7061
806
68
R
EI Mundo
www.clmundo.cs
24828
901
68
R
Hispanidad
www.hispanidad.com
2021
1414
68
RR
Información
www.informacion.cs
1432
768
68
RR
La Gaceta
gaccta.es
326
986
68
RR
La Razón
www.larazon.ce
22283
619
68
R.
La Vanguardia
www.
www.lavanguardia.com
26533
777
R
La voz de Galicia
www.lavozdegaliche
6709
935
R
Libertad digital
www
www.liierraddggaallccoom
12591
561
R
OK Diario
okdiario.com
1636
780
R
Periodista digital
www.perrodistadiginne.
9386
1690
R
Voz libre
vozlibre.com
329
1441
00
Ara
www.ara.cam
14317
664
60
Ara Balears
ww www.arabalears.cam
3743
626
00
CatalunyaPress
www.catalunyapres.cat
2669
411
Crític
www.elcritic.co cat
447
1516
Diari d' l'Andorra
www.diariandee www.diariandorra.ad
5036
444
Diari de Girona
www.diaridegironacoo
4706
456
Diari Segre
www.segre.com
1196
945
El 9 Nou
e19nou.cat
287
684
sslub.e9noo..ca
11
334
EI Nacional
www.elnacional.. cat
1400
575
EI Punt Avui
www.elpuntavui. cat
11571
666
c-Noticies
www.e-noticies.cam
20
240
La República
www.larepublica.cm
1137
662
La Nació digital
www.naciodiiittl.com
14183
602
Regió 7
www.regio7.ccm
2728
476
L
El Periodico
www.elperiodico.ca
3095
792
L
jornal.cat
www.jomal.cat
106
648
L
Público
www.publico.es
85
1745
VilaWeb
www.vilaweb.catt
2955
2247
Diari de Tarragona
www.diaridctaaragnna..om
800
540
Table 3: List of newspapers available in the OSCAR corpus (version 22.01) (1) for the four languages used in this work.
B Topics

---

ID
%
Label
Keywords
10:0
0.12
economy
percent year guvernment lax money billion ecunomic econumy years pay financial federal workers u.s johs people companies market business spending energy plan climate prices work policy trade bank time budget
10:1
0.16
culture
time people life years work book story love family good day film music man feel women great told young year lot movic kind long art friends woman times thought york
10:2
0.10
international
u.s military governmeut war president united sccurity china people minister country foreign isracl iran iraq international american countries russia chinese officials forees nuclear political attack news north party intereept group
10:3
0.15
government
trump president house democrats campaign republican obama election senate democratic republicuns party white vote clinton political biden voters presidential donald news people time congress conservative sen gop washington support national company companies business facebook data technology online market news media internet work google social people users time digital twitter service tech site products including content industry cco free customers apple
10:4
0.08
technology
10:5
0.13
law & justice
court law federal case department u.s justice investigation government attomny told report legal news officials public judge office fbi security administration general supreme trump border president statement house criminal evidence
10:6
0.13
education
people school students women black american education rights children university america schools public white years political work religious social community americans college time life men country church history parents free
10:7
0.08
covid
health people covid coronavinus care medical vaccine pandemic share cases children patients news abortion virus discase study drug public hospital time percent risk email women doctors university cancer deaths treatment
10:8
0.10
hotchpotch I
police game city team time gun told shooting scason people year-old year man officers games shot night players killed news sports day left officer football sunday death week county play water food people city years space year air time area day climate local national miles travel fight land small island oil south park coust scientists north long sea california change
10:9
0.09
hotchpotch II
15:0
0.09
educution
school people students women children health university work study time education purents schools life kids percent college years care medical child high family student young cancer good feel day patients immigration border u.s people immigrants mexico govermment illegul country united president americun marijuana drug migranis children year years security work democracy administration america news mexican texas refugees legal today enforcement
15:1
0.04
immigration
15:2
0.07
government
trump news president u.s intelligenne media report house investigation security fbi department govern- ment white intereept campaign officials committee clinton told story national trump's justice washington russian administration member documents donald
15:3
0.14
hotchpotch
time people life years love film book story work music family day good year movic man great women told young kind foel lot york series long woman books night covid health people coronavirus pandemic vaccine share cases virus york news public medical cure officials u.s week vaccinated discuse workers deuths vaccines reported hospital city twitter email patients told facebook
15:4
0.06
covid
15:5
0.11
violence
police people told city news death man officens gun county shooting family year-old black killed violence officer case vidco school reported charges prison department crime years shot arrested time authoritics china chinese government countrics minister north curopcan united u.s south party country international president russia korea forcign curope prime russian british political trade germany year years french leader global britain
15:6
0.07
international
15:7
0.13
government II
trump president house democrats republican campaign obama scnate election democratic republicuns party vote biden voters white presidential politicul clinton sen gop percent congress support donald candidate people time washington national
15:8
0.13
hotchpotch
people american political america president black time years white war conservative media fact americans left history conservatives movement good country church power speech book social public free life religious man
15:9
0.10
cconomy
percent tax year money government billion economic coonomy pay financial years federal jobs workers people spending market companies business prices bank u.s budget insurance rate plan debt health growth costs
15:10
0.08
ccologism
water climate food years energy space people city year oil change time cavironmental uir gas area power national natural miles day land scientists small science local emissions carbon earth weather military isruel iran iraq government security forces afghanistan united syria attack people president israeli islamic attacks killed american troops nuclear army obuma country muslim group teroristt officials weapons
15:11
0.05
international II
15:12
0.03
sports
game team season games players sports foothall play time win enach league year nil points player won teams left lield week good fans yards sunday night final big college played
15:13
0.09
law & justice
court law federral rights case supreme legal abortion justice public government judge decision laws action u.s amendment department order attorney lawsuit news ban cases ruling issue general filed texas policy company companics business facebook technology market data online internet google media users products digital work tech people service industry news site time social content customers ceo free apple firm year
15:14
0.07
technology
Table 4: Topics (with 10 and 15 clusters) obtained with Mallet on OSCAR's English newspaper documents. Clusters boldfaced and colored in blue are used to build the training data.

---

ID
Label
Keywords
10:0
0.14
culture
film leben frau bild welt musik schen crzählt fraucn mutter kunst geschichte berlin paar vater weiß familic liebe steht künstler zeigt bühne kinder leute spielt bilder eher publikum männer band welt deutschen deutschland gesellschaft buch leben geschichte deutsche beitrag frage frauen politik junge politischen politische berlin medien kirche juden thema wissen krieg freiheit steht cher leute kultur staat arbeit sprache
10:1
0.15
hotchpotch
10:2
0.11
law & justice
polizci gericht fall polizisten laut staatanwallschaft opfer deutschland bild berlin gewalt frau täter jährige richter crmittlungen urteil verletzt prozess behörden foto personen dpa verfahren angaben verurteilt offenbar beamten berliner männer
10:3
0.11
government
spd partei edu grönen merkel afd berlin deutschland fåp union politik bundostag linke csu parteicn wahl grtine koalition regicrung angela frage linken bundesregicrung stimmen kanzlerin berliner mehrheit müsse kritik steht
10:4
0.10
economy
curo millionen unternehmen deutschland milliurden geld deutschen wirtschaft deutsche zahlen berlin kosten dollar bank banken welt kunden europa folgen zukunft europaischen arbeit laut usa china land firmen bild konzern markt
10:5
0.05
sport
bayern spiel trainer spieler futiball spielen saison bild deutschen mannschaft fans platz spiele team sieg sport for münchen league liga deutsche minuten leizten dortmund stcht bundesliga verein ball teilen datensicherhcit
10:6
0.07
technology
daten inhalte finden informationen internet facebook vergleich inbalt google twitter zustimmung person- eabezogene artikel nutzer angezeigt übermittelt cinverstanden brauchen laden gerät anzeige forscher netz sozialen bild untemehmen curo kunden produkte app kinder deutschland frauen eltem zahl schulen schule leben pandemic schüler patienten laut kindem arbeit wocbe liegt bayern berlin zahlen kind corona arbeiten bekommen deutschen flüchtiing studie gilt personen wochen millionen
10:7
0.09
covid
10:8
0.10
international
regierung usa trump priisident türkei russland europa staaten deutschland soldaten china israel präsidenten syrien afghanistan landes welt krieg iran flichtlinge frankreich grenze ausland donald millionca russischen hauptstadt armee obama stadt curo wasser münchen meter straße stcht ort kilometer stchen haus leben liegt quelle auto platz fahren berlin augsburg gebäude münchner bild straßen ticre besucher paar unterwegs projekt millionen sicht
10:9
0.10
local
15:0
0.05
covid
bayem pandemie corona münchen landkreis deutschland uugsburg oktober zahl junge freitag woche virus montag welt covid coronavirus mittwoch stadt patienten november dienstag donnerstag liegt maßnahmen sonntag bayerischen geimpft feiern wochen
15:1
0.06
local
stadt euro vergleich berlin auto wohnungen straße autos bahn münchen fahren meter gebäude hamburg quelle hau platz kilometer berliner prijett haus finden kosten straßien steht stehen gehaut millionen wohnen bauen
15:2
0.08
social
frauen kinder deutschland cltem schule arbcit berlin schulen schüller loben männer kindemn arbeiten kind frau ftüchtiinge studic deutschen bekommen universitär lchrer zahl thema familien familic jungen laut berliner curo stellen
15:3
0.11
government
spd edu grünen partei merkel afd berlin fop deutschland union bundestag esu grtine linke koalition parteien politik wahl angela bild kanzlerin regierung bundestegirungg geben berliner linken kritik müsse sechofer dpu
15:4
0.06
live science
wasser tiere deutschland forscher natur erde wissenschaftlr millionen grud meter welt lehen klimawandel studie pllanzen wald liegt landwirsschat umwelt essen weltweit fleisch tonnen bauemn tier kilometer kürper bild gesundheit laut
15:5
0.06
Navism
deulschen herlin geschichte kirche juden buch dår deulsche berliner seite deutschland leben krieg hücher jüdischen museum stadt tod ausstellung papst weltkricg kultur hitler nazis judische literatur polen verlag schriftsteller welt
15:6
0.18
family
leben frau paar leute bild weil familic steht kinder erzählt mutter welt vater frauen sehen männer junge eltern geld wissen sicht stehen bisschen liebe haus kind kopf hause schnell jahrige
15:7
0.19
hotchpotch
welt deutschland frage gesellschaft politik beitrag deutschen politischen politische leben eher wissen buch fragen deutsche freiheit art europa staat medien steht leute thema problem geschichte sehen rolle demokratie all
15:8
0.05
sport
hayern trainer spiel spieler fubballl spielen saison manoschaft deutschen fans platz team sieg sport bild spiele tor liga league deutsche minulen münchen dorimund leizien bundesliga verein hall stadion steht minute
15:9
0.12
law & justice
polizei gericht fall polizisten shaassanwal laut opfer täter richter jährige ermittlungen urteil gewalt bild verietzt berlin prozess deutschland behörden verfahren dpa personen angaben foto frau verurteilt beamten offenbar hamburg montag film musik kunst künstler welt berlin band bühne sehen geschichte bild filme publikum zeigt spielt ausstellung theater bilder album regisseur lehen roman art schauspieler kino musiker berliner york werk erzählt
15:10
0.07
culture
15:11
0.06
technology
daten inhalte artikel facebook internet inhult informationen finden twitter google medien netz: bild zus- timmung personenbezugene laden nutzer angezeigi einverstanden ühermittell sovialen seitung wahrheit digitalen journalismus brauchen soxiale gerit app unternehmen trump usa regierung präsident land china donald partei präsidenten ohamna frankreich staaten wahl europa washington parlament großbritannien us präsident europäischen brisssel trumps stimmen well landes italien york biden macron london mehrheit
15:12
0.07
international
15:13
0.10
economy
euro millionen unternehmen deutschland milliurden geld deutschen wirtschaft deutsche zahlen dollar kosten bank banken kunden berlin europäischen europa länder bundesregierung griechenland firmen konzem folgen laut krise mitarbeiter insgesamt land markt
15:14
0.07
international II
türkei russland regierung deutschland land israel flüchtinnge syrien sokdaten afghanistan iran prosident krieg grenze europa bundeswehr russischen putin ukraine armee türkischen deutsche irak tirkische stanten taliban moskau russische stadt
Table 5: Topics (with 10 and 15 clusters) obtained with Mallet on OSCAR's 's German newspaper documents. Clusters colored in blue are used to build the training data.

---

ID
Label
Keywords
10:0
0.14
botchpotch
mensaje españa sociedad años opinión vida mundo denunciar política mujeres gente personas historia privado educación social articulo iglesia redactar realidad país citar politicos forma enviar libertad derecho espuñoles leer papa
10:1
0.16
economy
millones euros españu año economía empresas años gobiemn crisis mereado secter empresa pais banco social países oconómica información compañia dinero politica europea precio trabajadores sistema caso deuda servicios mes meses
10:2
0.16
culture
años vida mundo historia casa año película cine libro obra mujer familia música serie madre gente padre the arte programa españa premio televisión director hijo novela amor hombre joven noche gobierno sánchez psoc partido presideute elocciones congreso política pedro iglesias ciudadanos esрaña rajoy madrid pablo cataluña vex partidos electoral lider lcy socialista moneloa díaz izquierda politico votos dipurados constitución euros
10:3
0.12
government
10:4
0.15
local
burcelona madrid ciudad años centro ayuntamiento covid catalunya personas españa zona visto comunidad periódico año calle metros relacionadas noticias local proyecto palma galicia vecinos hora plaza euros comentado nacional semana
10:5
0.11
science
años forma personas salud mundo tipo estudio tecnologia vida productos datos sistema información agua explica internet caso usuarios año investigación importante calidad universidad mejores permite cambio consumo españa problema enfermedad
10:6
0.06
covid
españa coronavirus casos sociedad covid mapas evolución gobierno daros vacunación gráficos política mundo personas socios canarias madrid Ieido años pandemia contagios sanidad variante ómicron salud avanza bazte enlace copiar vacuna
10:7
0.11
law & justice
años caso madrid policía comentarios juez tribunal vox páblico espuña díaz hombre sociedad ayuso justicia ley fiscalia investigación comunidad casado nacional judicial yolanda sentencia delito publicidad prisión audiencia civil noticias
10:8
0.07
sport
madrid real partido equipo fútbol años españa club liga temporada barcelona jugador jugadores barça bulón año juego carrera mundial puntos partidos minutos jomada messi españiol gol jugar historia campo catrenador
10:9
0.10
international
país presidente gobierno años países unidos internacional guerra trump china ecuu personas ministro curopa rusia información mundo diciembre seguridad méxico política militar nacional noviembre francia curopea elecciones actualidad millones frente
15:0
0.07
live science
salud años estudio personas agua vida enfermedad forma investigación pacientes cáncer virus hospital tipo riesgo alimentos explica productos tratamiento enfermedades cientificos mundo caso animales año médicos consumo médico investigadores niños
15:1
0.05
Catalonia
burcelona catalunya generalitat catalán covid periódico cataluña puigdemont catalana relacionadas noticias visto comentado temas mossos pasaporte lee sant govern minutes erc años contiar jordi pandemia centro linguística parlament coronavirus directo
15:2
0.14
government
gobiemn partido psoc presideate sánchez clocciones política cspaña congreso ciudadanos madrid rajoy ley partidos electoral pedro líder izquierda votos vox politico socialista pablo pais diputados ejecutivo debate iglesias portavoz comunidad
15:3
0.11
law & justice
años caso policía juez comentarios madrid tribunal público sociedad justicia investigación fiscalía nacional espuña comunidad euros judicial ley civil sentencia delito prisión audiencia ayuso mujer fiscal recuerda juicio guardia juzgado
15:4
0.05
covid
españa coronavirus casos sociedad covid mapus evolución gobierno datos vacunación gráficos mundo politica socios canarias personas leido madrid contagios pandemia años variante ómicron avanza hazte sanidad salud enlace copiar economia
15:5
0.04
botchpotch
españa corentar accede archivado leidas galicia portadu suciedad madrid diciembre liitbol gobierno alerta años lecr navidad economía mahana grados famosos historia gallego voz antonio juan marruccos covid josé máxima carlos
15:6
0.12
local
madrid ciudad zona años ayuntamiento centro personas metros año proyecto palma san comunidad vecinos calle mar agua información kilómetros obras capital zonas plaza isla volcán sevilla locul edificio municipal burrio
15:7
0.10
international
pufs presidente gobierno años unidos paises internacional guerra trump china eeuu rusia ministro personas diciembre europa méxico seguridad información mundo militar noviembre francia venezuela nacional millones actualidad europea reino política
15:8
0.18
social
mujeres años mundo vida españa personas sociedad politica gente opinión social educación realidad país farna historia sociales universidad caso niños libertad violencia hombres problema mujer derecho autor género sentido padres
15:9
0.14
economy
millones curos españa año economía años empresas gobiemno crisis sector mercado banco país países empresa económicu social curopea trabajadores deuda dinero precio crecimiente meses medidas mes plan inversión europa información
15:10
0.05
sport
madrid equipo real partido fútbol club años liga barcelona temporadu jugador españa jugadores burça pupa año balón mundial carrera partidos español puntos minutos juego selección gol entrenador mundo jornada messi
15:11
0.02
government II
sinchez iglesias gobierno pedro euros millones moncloa psae jues periodista denuncia rey ayuso puhlo robles margarita oloma dinero calle leído ocultan video comunicación congreso articulo venezuela año telecinco
15:12
0.03
botchpotch II
mensaje denunciar privado redactar publicidad citar españi opinión enviar años articulos vida sociedad cconomia corrco díaz vox yolanda favor hombre articulo virales quieres deja gracias coche casado anteriores problema polémica
15:13
0.09
technology
compuñía tecnología usuarios internet datos privacidad editorial empresa forma información s.1 espuña titania euros cookies reservados politica mundo red comscore auditado digital transparencia web google condiciones loteria recomienda tipo móvil
15:14
0.16
culture
años vida mundo historia casa cine pelicula año lariila libro ohra mujer música madre serie padre the premio gente programa director horbbre hijo españi televisión novela amor joven arte nache
Table 6: Topics (with 10 and 15 clusters) obtained with Mallet on OSCAR's Spanish newspaper documents. Clusters colored in blue are used to build the training data.

---

Table Number of articles per newspaper (row) and topic (column) for the German subset OSCAR. See Table 5 for the definition of the topics. Topics boldfaced and in blue are used for training the classifier after balancing II vs R.
450
1239
3
530
817
592
677
15:13
2923
212
1171
28
688
2885
1776
1121
523
10
15:12
2/91
253
888
508
747
521
372
15:11
27
68
0
187
108
60
0
15:10
27
12
57
36
13
14
3921
84
921
374
757
2653
3089
582
567
1634
2
10
0
3
5
15:7
97
299
14
697
1191
905
440
1752
11
15:6
19
26
45
221
74
66
43
15:5
142
357
65
426
925
46.5
407
382
328
219
160
71
252
8/6
593
1046
41
86
531
46
766
2345
1910
712
1677
541
34
60
5
121
185
57
52
180
15:1
13
37
11
135
377
82
122
19
5
377
25
6
108
17
12
0
109
1099
101
91
169
168
1377
331
552
42
14
10:8
5070
713
2035
3
972
2701
1229
1406
1024
880
10:7
24
88
11
63
106
107
91
51
12
817
395
262
488
12
3
10.3
20
2
38
36
17
12
10:4
274
1205
961
1896
1272
504
C Distribution of Topics per Newspaper
121
632
73
1041
972
883
2071
10:2
103
945
299
728
584
550
1652
5
10:1
246
1035
39
969
1292
741
2100
17
69
2
264
1,13
110
16
9
Prnnffrrerr Allgemeine Zeilung
R Preufosche Allgemeine Zeitung
Sidddeuscche Zeitung
Neves Deatschlaxl
Tageszeitug
Junge Freiheit
DW News
Junge Wek
My Heimat
Newspajer
Zoit
Die Welt
Bild
L
Rt

---

Table 8: Number of articles per newspaper (row) and topic (column) for the English subset of OSCAR, See Table 4 for the definition of the topics. Topics boldfaced and in blue are used for training the classifier after balancing L vs R.
15:14
249
26
50
87
54
0
113
201
32
30
171
95
130
46
586
21
29
13
3
16
48
3
0
25
131
0
7M
17
1901
9
20
53
0
15:13
64
1840
815
502
700
1938
41
848
1100
149
1551
1336
551
408
303
265
64
666
213
120
112
360
839
108
7
2325
762
4174
3
1403
446
100
218
24
641
1375
178
2
2025
15:12
43
11
25
25
24
4
15
41
5
13
18
10
15:11
60
1923
320
1025
3592
10
1300
26
617
719
2648
2729
11.30
2300
1500
2374
4849
4.9
1042
20
20
1386
1835
62
483
980
5
1200
354
1274
312
1079
1744
3
1940
15:10
111
5759
353
311
1963
5865
3924
405
1522
400
5708
5334
136
1176
2353
284
1361
250
1040
302
195
29
338
393
205
913
1098
101
2340
90
80
1036
709
15:9
103
1866
245
1218
988
70
1259
955
2511
2394
2016
3952
708
9748
721
532
162
978
301
319
97
875
M12
52
9
1339
3179
2
1744
309
2763
305
280
25
507
1412
569
7
1119
15.8
366
333
232
206
665
23
1003
71.
502
1098
768
5504
1907
1994
984
945
95
463
124
847
2889
1558
745
1613
15
51
1504
3085
177
957
626
329
31
320
928
1009S
5308
15:7
3021
236
1201
2005
4
22
8500
5817
3942
15297
3404
1582
M51
225
1787
412
714
321
1306
3403
343
952
2201
7
5071
1137
1451
1163
124
1664
2965
917
10
2070
15:66
19
2632
264
531
2312
1548
34
294
1859
1918
732
1526
783
10722
62
479
156
516
55
137
25
368
1675
12
1675
259
551
672
127
3440
113
46
328
1067
89
15:55
5007
2365
5742
5743
9
542
565
1941
6737
834
4530
327
578
487
153
885
465
191
195
437
2522
23
15
156
2385
11300
3373
349
871
332
1884
2036
15:4
296
44
166
164
3
31
162
218
120
605
60
149
60
13
17
24
21
14
185
58
68
150
125
450
46
17
118
0
76
15:3
68
61
46
20
82
28
176
125
132
188
213
69
12
121
12
40
31
65
62
112
156
27
19
12
26
33
58
15:2
37
1006
11288
863
678
28
860
2044
1235
1410
3/42
1357
1910
151
7509
489
55
950
162
191
247
458
1094
99
1024
185
668
38
2054
551
638
985
132
1528
893
169
1500
15:1
518
1224
467
503
665
1418
616
23
279
1657
901
312
425
778
710
622
145
231
113
58
205
1421
28
951
0
109
515
2
823
232
551
141
8
160
608
411
15.0
58
31
66
65
2
42
58
103
129
47
242
30
270
18
43
16
11
22
12
73
0
30
11
40
10
27
0
31
109
21013
83
3933
344
2028
6177
3409
405
1214
302
5929
5357
306
3648
1152
2MI1
1261
2012
3/2
159
24
283
367
1665
0
152
841
0
873
95
2660
91
60
1005
139
0
10:8
2900
1098
2315
1911
5
842
532
242
1179
4588
1823
339
526
3148
133
379
199
376
242
102
196
1211
1014
2
1578
595
2006
135
469
138
1022
0
941
10.77
1.5
362
88
220
242
17
143
349
166
188
131
110
93
33
65
34
13
65
179
128
2174
71
548
225
89
253
39
11
64
153
66
2
226
10/6
661
327
721
434
434
933
1244
4405
51
250
2057
1336
660
4209
1580
1473
1986
670
581
204
285
681
2405
1820
180
15
167
1398
2918
448
8
1946
1070
776
362
408
1074
1049
10:5
91
4299
2600
2136
2267
15
2345
178
1656
2253
3376
3438
3975
2220
2771
990
2334
594
1605
499
228
379
736
2903
199
10
40
651
5019
5
4252
1031
2300
1333
46
1524
2901
236
6
3033
10.4
243
175
133
333
2
49
84
243
231
268
124
138
1421
39
106
17
55
14
32
108
8
24
241
0
181
26
167
70
22
63
71
19
0
315
10:3
169
3365
1561
2371
2255
3740
34
2212
97.59
4244
4374
16959
6298
4666
1813
1752
11334
252
2036
478
516
1863
4273
951
7
24
1265
28
6
6448
1555
1628
1221
1393
214
2377
3233
58666
10
2054
16:2
90
4642
729
6031
181
2361
59
819
1156
4327
1870
4008
2472
8R27
935
335
1653
151
226
158
1914
3707
98
755
1-899
288
1500
533
4184
445
48
1476
2841
3
10.1
516
22
180
124
416
640
26
314
583
241
1239
668
557
23
656
48
129
26
27
50
206
190
5
0
160
687
0
406
1.59
220
62
2
61
122
0
442
16:0
133
2676
347
268
1402
1067
90
1065
2387
2992
2664
7K3
14775
751
670
200
1127
2:30
402
141
1048
1173
49
10
1533
3568
2211
367
3699
231
544
1731
5
1315
The Washington Free Deacom
Breithat News Network
Newopppper
News
L. AlteeNet
L. Anociated Press News
L. Buzzlee News
CBS News
CNN
L. Democracy Now!
HuttPoss
L. Insider
Mocher
L MSNBC News
L. NBC News NPR
L. Politio
L. The Atlantic
L The Daily Beant
T. The Rosnomiss
The Intercegs
L. The New Yorker
Times
L The Wastington Pod
USA Today
R. American Greamess
American Thinker
R ConserativeH
InfoWWrs
Life Life-Actio Action
National Review
Roaoon
The American
R. The Blaze
The Daily Caller
The Daily Wize
The Epoch Times
The Federalist
B The Gateway Pundit
The National Polse
Thee Wastington lines
The Spectator
R Washingonn Exuminer
wND
The
Vox
Fox,
R
8
R.
R
8
R

---

Table 9: Number of articles per newspaper (row) and topic (column) for the Spanish subset of OSCAR. See Table 6 for the definition of the topics. Topics holdfaced and in blue are used for training the classifier after balancing IL
15:
92
31
53
77
69
25
42
12
15
67
0
27
59
45
0
32
15:13
10
22
574
103
21
171
21
108S
14
97
352
13
99
49
56
96
15:12
0
3700
26
28
10
36
53
0
5335
0
15:10
2
39
3
0
29
13
16
45
2
31
0
264
1674
396
379
7715
1443
80
1642
39
10
671
41
3
4349
520
69
445
1358
2977
288
129
7
2280
837
534
526
8
15:8
924
414
35
494
965
5305
1217
75
18
126
2458
191
238
50
160
846
306
717
711
154
2065
46
106
43
957
358
1248
22
373
0
15:7
212
505
26
81
734
216
361
8212
1150
20
163
1587
95
365
25
16
1477
38
39
444
254
2801
16
26
189
1273
1870
273
731
410
49
15.6
11
599
86
29
74
0
28
154
21
64
243
0
41
46
201
23
43
21
0
15:3
0
1
140
12
15
0
0
0
0
0
11
1
0
24
0
7
3
0
2
8
0
0
0
152
448
47
3760
884
87
429
2924
543
16
146
5400
14
33
1743
438
155
334
1052
2020
8
92
363
8600
15
84
921
509
1112
389
5049
1191
126
340
1036
109
40
1017
24
0
2737
364
146
1242
1709
2528
63
51
2
556
1697
20
58
15:1
3
47
750
0
0
5
63
8
18
15
0
29
200
33
0
15:0
2
14
130
12
0
0
42
0
0
23
0
0
0
5
0
20
38
0
4
0
10
0
4
18
10:9
268
578
79
734
255
420
9477
1298
11
24
198
1740
116
476
27
18
1644
25
491
319
3014
11
36
190
1573
2215
321
870
396
71
10:3
44
3
0
27
0
0
16
0
0
6
10
3
23
0
141
23
36
0
10:7
61
349
57
3536
789
80
2851
503
17
148
5412
12
39
1611
315
972
1641
93
14
1354
1237
300
826
10
65
10:6
489
85
26
4
18
52
0
0
26
0
41
19
28
27
0
31
65
28
0
10:5
14
30
448
94
95
225
3
40
255
25
18
23
189
71
55
43
51
2
R.
8
2
0
6
2
509
568
3
0
28
99
3
0
47
64
23
102
289
0
31
0
540
310
45
0
2
0
10:3
810
434
80
943
386
4163
1549
119
802
325
919
69
880
18
44
3038
2
1238
1564
2000
48
56
1954
451
1.596
19
5377
67
10:2
30
249
630
132
8
343
14
0
52
0
119
79
27
237
0
13
0
256
114
2
82
10:1
57
67
455
619
9109
1568
75
67
461
141
2024
64
18
811
46
5347
688
49
495
1613
3517
544
185
11
3205
822
651
24
701
11
10:0
873
297
0
284
419
3541
763
1716
170
3
158
50
7
1.58
417
2000
763
886
6228
104
2273
671
1.88
1121
17
295
E1 corren de Andalucia
Newspuper
Cuaro poder
De Verdad digital
Darrio muggessista
Digital Sevilla
dicommnnstta.mo
elD Diaario.ss
EI Obrero
EI pais
EI peróódico
El plural
El siglo de Europa
HuffPPod
La RepúNica
Replicantes
Mundiario
L Mundo Obrero
Prottfggaal
L Publico
Ade latte lispañia
R Altevoz de sucesos
Disidentia
El Confidencia
EI diestro
EI Imparcial
R EI independiente
R IEI Mumdo
R Hispundlad
Información
La Gaoeta
Razón
La Vanguardia
Galicia
Libertad digital
OK Diario
Periedistt digital
Viz libre
El
La
L
R
R

---

D Subjects for the ChatGPT and Bard Article Generation
English
German
Spanish
Catalan
teloworking
Telearbeit
el teletrabajo
el telerchalll
labor conflicts
Arheitskoofiktte
los conflictos laboraes el uráico por la madiau precie la vivionda
confiictes laborals el vänsit al mati el preu h'haiitagge
meemnngg truflic
Murgenwethehr
housing prices
Wohnungspreise
housing construction
Wohnungshau
construcción de viviendas venta ambulanic
la construcció d l'habiiatges la venda ambulant
street vending
StraBenveckaf
the disemborkation of illegal bouts
die Ausschillung von illegation Booten
el deserbburco de paleras los nctores
el desembureament de posteres als actors
actres
Schauspieler
soap operas
Seifenopem
telenovelas
telenovel-les
10
tolevision
Femsschnn
la tolovisión
la tolovisió
11
lase shows
Late Night-Show
los late shows
als late shows els diuris digitals la policia
12
digital newspupers
digitale Zeitungen
los periódicus digitales policia
15
the policc
die Polizei
14
the army
die Armoe
el cjéreito
l'exèrest
15
termriss
Terroiismus
el terrorismo
el rerrorisme
robberies
Raubüberfälle
los robos
robatoris
17
murder
Mord
el asesinato
I'assassinar
18
death penalty
Todesstrafe
de muerte
la pena de mort les eloccions el progrmmai Pegasus la importaciia la ciència la tecnologia
19
elections
Wahlkn
las clocciones el software Pegasus
20
Pegasus software
Pegasus-Software Software
the importance science
die Bedlutung der Wissenschuft
importancia ciencia la tecnologia
22
technology
Technologie
23
the metaverse
das Metaversum
metaverso
el metavers
24
augmented reality
Augmented Reality
la realidad aumentada
la realinar augmentada els mobils
25
cell phones
Tundys
los móviles
26
electric cars
Hlektromatos
Ios ooches eléctricos
els cotxes eléetrics el consum de cam l'agricultura ecologica els superafizents bosses de plastic ol reciclatge la desforestacid els bosces
27
meat consumptiio
Fleischksonsum
el consumo de came
28
organic farming
Ökolngischer Landbau
agricultura ecológica los superalimenios bolsas plásticu of reciclaje
29
superfood
Superfood
30
plastic bugs
Plastkktüten
31
recycling
Recyeling
32
deforestation
Enrwaldung
desforestación
33
forests
Wälder
los bosques
34
hird farms
Vegelfarmen
las granjas de aves ciclistas
les granges d' l'mss ciclistes
35
cyclists
Radfahrer
36
naclear energy
Kemenergic
la energia nuckar
l'energia nuckar
37
companies
Mineraäälumteenchurea
petroloras
petrolieres la contaminació els abrics de pell els diamants la cap de l'empresa
38
pollution
Umwweltve Unwelrerschmmizungg
la conraminac ión
39
fur coats
Pelzmintel
los abrigos los diamantes la jefa de la empresa matrimonio casarse de blanco
40
diamonds
Diamanten
41
the female head of company
die weihliche Leiterin eines Untemehmens Heirat
42
murriage
45
marrying in white
Heiraten in Weiß
essar-se de blanc l'avortameot l'assetjament sexual
44
abortion
Abcrcibung.
aborto
45
sexual haraesment
sexuelle Relissigung das Aller der Mitter
acoso sexual
46
the of mothers
edai de lus madres
l'edal de les mares solteres
47
single mothers
alleinerziehende Mürter
madres solteras
48
career
Karriere
la carrera profesional
la camera professional
49
job stress
Stress Arbeitsplatz
estrés Inboral el abuso de poder depresión
l'estèss luboral
50
abuse of power
Machrmisstrumch
I'ahús de poder la depressió
51
depressioo
Depression
52
layuffs
Emtlassungen
el despido
"acomiadament
53
private schools
Privatschuken
las escuclas privadas
les escoles privades
54
private universities
Privanuniversitttenn
universidades privadas
universitats privades
55
extracurriularr activities
suBerschullichee Aktivitälen
las actividades extraescolares
les activilats estrussoolars trcball infantil els diners
56
child labor
Kinderarbeit
trabajo infantil
57
money
Geld
dinero
58
capitalism
Kapisalismus
capitalismo
el capisalisme
59
the stock market
der Aktienmarkt
bobsa
la bonsa
60
othical banking
ethischen Banken
la hanca érica los
la banca òtica
bunks
Banken
bancs
62
alcobol
Alkohol
el alcobol
I'alcobol
63
tahacco
Tahak
tahaco
al rahac
64
cannabis
Canmabis
el cannubis
el cinnabis
65
drugs
Drogen
las drogas
les drogues
66
bealth care
Gesuuddheiss/rrsorgge
sanidad
la sanitat
67
diet
Ditr
la dieta
la dieta
68
rivalry sport
Rivalita Sport
rivalidad el deporte
divalital l'espori
69
Saturday's 's game
Samstagrspiel
of partido del sábado
el partit de dissabes
70
sports
Sportwagen
los coches deportivos
els exportius
71
the olympic games
die olympischen Spiele
Jos juegos olimpicos
els jocs olimpics
72
Qatar World Cup
Weltmeistorschaft in Katar
el Mundial de Qatar
el Mundial de Qatar
73
China
China
China
Xina
74 Turkey
Türkei
Turquía
Turquia
75
United States
die Vereinigte Staaten
Estados Unidos
Units
76
the lntest iPhone model
das neueste iPhone- Modell
al ditimo modelo de iPhone
el darrer model d'iPhone
77
CharGPT
CharGPT
ChaGPT
ChaiGPT
78
Netflx
Nettlix
Nottlix
Nettlx
79
Amazon
Amazon
Amazon
Amazon
80
Google
Google
Google
Google
Continued on next page

---

English
German
Spanish
Catalan
TikTok
TikTok
TikTok
82
Margaret Tharcher
Margaret Thatcher
Margaret Thancher
Margaret Thatcher
Donald Trump
Trump
Denald Trump
Dosald Trump
84
Barak Ohama
Barak Chama
Barak Ohama
Barak: Ohamn
85
Karmla Harris
Kamala Haris
Kamala Haris
Kamala Harris
86
Nelson Mondela
Nelson Mandela
Nelson Mandela
Nelson Mandela
87
Angela Merkel
Angela Merkel
Angela Merkel
Angela Markel
José
89
Francicco Franco
Fruncisco Franco
Fruncisco Franco
Francisco Franco
Julian Assange
Julian Assange
Julian Assange
Assenge
91
Greta Thunbery
Greta Thunherg
Greta Thunherg
Greta Thunherg
92
Claudia Schaller
Clandia Schiller
Clandia Schiller
Clandia Schiller
93
Angelina Jolie
Angelina
Angelina Jolie
Angelina
94
Richard Gere
Richard Gere
Richard Gere
Richard Gere
Bono
96
Plácido Demingg
Plácido Domingo
Plácido Domingo
Plácido Domingo
97
Polé
Pelé
Pell
98
Magic Johnson
Magic Johnson
Magic Johnson
Magic Johnson
19
Rata Nacal
Rafa Nadal
Rafa Nadal
Rafa Nadal
101
Joan Antoni Sumarunch
Joan Antoni Samaranch
Joun Antoni Samaranch
Joun Antoni Samaranch
Table 10: List subjects used to generate newspaper-like articles with ChatGPT and Bard.

---

E
Stance Classification at Article Level
English
German
Spanish
Catalan
Subject
Mono
Multi
Mono
Mono
Multi
Multi
teleworking
R
R
R
R
L
L
RR
R
R
R
R
labor conflicts
R
L
R
I
IL
L
L
L
L
morning traffic housing prices
H.
R
L
II
IL
L
L
L
housing construction
H.
R
vending
L
IR
disembarkation of illegal boats
IR
I.
L.
actors
soap operas
IR.
L
television
IR.
shows
R
IR.
R
L
12
digital newspapers
L
II.
IR.
the police the army
R
IR
R
L
II.
15
terrorism
IR
IL
L
robberies
II.
R
R
17
murder
L
I
IL
IL
L
R
L
18
death penalty
IR
IR.
19
elections
L
IR.
20
Pegasus software
I.
IR
IR.
21
the importance of science
R
IR
IR
IL
22
technology
IR
IR
IR.
R
23
the metaverse
I.
IR
IR
IR.
IR
R
24
augmented reality
R
IL
R
25
cell phones
I.
IR
D.
L
HI
IR.
R
26
electric cars
R
IL
IR
R
27
meat consumption
R
R.
I
IL
L
L
28
organic farming
R
IL
R
29
superfood
R
IR
IL
IR
R
30
plastic bags
R
IR
31
recycling
R
IL
I.
32
deforestation
R
33
forests
R
34
bird farms
35
cyclists
36
nuclear energy
R
IR
oil companies
R
L
38
pollution
R
fur coats
diamonds
the female head of company
IR
R
marriage
marrying white
I
abortion
sexual harassment
IR
the age mothers
IR
L
single mothers career job stress abuse power depression
R
L
R
I.
R
R
IR
R
I.
I.
II
L
I.
52
layoffs
L
LL
IL
L
L
private schools
IR
L.
II
L
HI
L
R
I.
private universities
R
IR
R
R
IL
L
R
R
L
55
extracurricular activities
R
IR
L
IR
IL
II
L
R
R
child labor
R
L
R
L
IL II
R
57
money
R
IR
L
I
R
R
R
R
58
capitalism
R
IR
R
RR
II
L
II
L
II.
IR
II
R
I.
59
the stock market
L
L
L
L
II
L
R
IR
R
R
R
60
ethical hanking
I.
L
I.
L
II
L
II
L
II.
L
II
L
R
L
61
banks
L
L
L
L
L
L
II
L
R
L
R
R
L
R
Continued on next page

---

English
German
Spanish
Catalan
Subject
Mono
Multi
Mono
Multi
Mono
Multi
Multi
62
alcohol
L
L
L
L
IL
L
L
R
R
L
63
tohacco
IR
D.
11
L
11
1
1
11
1
R
L
64
cannabis
IR
R
L
11
II
L
L
II
R
65
drugs
R
L
E.
L
H.
L
H.
L
H.
L
H.
IR
I.
L
66
health care
R
L
L
L
IL
L
IL
L
IL
L
R
R
R
R
67
diet
L
IR
L.
IR
II
L
H.
I.
L
R
IR
R
L
68
rivairy in sport
L
L
L
IL
IL
IL
IR
I.
L
69
Saturday's game
R
L
IL
L
70
sports cars
I.
II
H.
IR.
I
71
the olympic games
L
R.
I.
IL
II
72
Qatar World Cup
IR
I.
H.
73
China
IL
IL
R
R
74
Turkey
II
75
United States
R.
IL
L
76
the latest iPhone model
R
77
ChatGPT
Netflix
R
Amazon
80
Google
R
TikTok
Margaret Thatcher
Donald Trump
Barak Obama
L
Kamala Harris
L
L
R
Nelson Mandela
LLLL
L
R
R
Angela Merkel
L
L
R
R
LLLL
José María
IR
L
R
L
Francisco Franco
R
IR
IR
Julian Assange
L
II
R
Greta Thunberg
I. I I I IR
II
L
R
R
Claudia Schiffer
II.
L
IR
II
I.
93
Angelina Jolie
IL
II
L
L
L
94
Richard Gere
L
IL
L
IR
R
R
Bono
IL
IL
L
L
L
96
Plácido Domingo
R
R
L
IL
II
L
IR
I
L
I
97
Pelé
R
IR
R
II
L
IL
L
R
IR
R
R
R
IR
Magic Johnson
R
IR
L
L
L
II
L
I.
R
L
99
Rafa Nadal
IL
IR
L
II
L
II
L
R
L
R
R
R
R
100
Alexia Putellas
R
IR
I.
II
L
II
L
R
IR
R
R
R
IR
101
Joan Antoni Samaranch
L
II
II
L
II
L
IR
II
R
Table 11: Class obtained by the classifiers on the 101 articles generated by CharGPTv08a
(8) and Bardv08a
Mono refers any of the monolingual models (finetuned with either English, German or Spanish) and Multi refers to the model finetuned will all the data.

---

F
Training Details
F. I/R Classifier
We finetune L vs. classification as schematised Figure classifier small network top RoBERTa that first performs dropout with probability 0.1 on RoBERTa's [CLS] token, followed by a linear layer a tanh. pass trough another dropout layer with probability 0.1 layer projects into the two classes. The whole architecture finetuned.
large (Conneau al., 2020) for
final linear
Stance (I/R)
dropout (0.1)
lanh
linear (1024 1024)
[CLS]
dropout (0.1)
XLM-RoBERTa (lurge)
Newspaper article
Figure 1: Finetuning architecture.
We use cross-entropy loss, AdamW optimiser and learn ing rate decreases linearly. We tune the batch size, the learning rate, warmup period the number epochs. The best values per language and model are summarised in Ta- bie 12.
Parameter
en
de
68
en+do+es
batch
8
learning rate
5e-6
5e-6
5c-6
5e-6
epochs
4
6
6
4
step best Acceal
146000
23000
93000
142000
best Acceal (%)
97.9
99.2
95.9
96.9
Table 12: Main hyperparameters used and their perfor- mance the three monolingual finetunings (en, de and es) and the multilingual one (en+tde+es).
All trainings are performed using single NVIDIA Tesla V100 Volta GPU with 32GB.
F.2 Topic Modelling
We Mallet (MeCallum, 2002) perform LDA the corpus after removing the stopwords, with the hyperparameter optimization option activated and done every 10 iterations. Other parameters the defaults. We do run per language topies and another run with 15 topics. We tag the corpus with both labels.


Reviews:
[{"review_id": "VaLHojtdJp", "reviewer": "Reviewer_2C5A", "summary": "", "questions": "A: The average word counts of the data suggests a significant amount of it exceeds the maximum token length of the classification model. Was any consideration given to this? Whilst in newspaper articles, the most significant information is typically at the beginning, there is no guarantee that ChatGPT can replicate this structure.\n\nB: During preprocessing, were self-references to the newspapers removed, as these would also allow classifiers to learn sources? e.g. \"[SOURCE] was unable to reach x for comment\", \"x, speaking exclusively to [SOURCE]\" etc. \n\nC: What motivated the choice of languages? Particularly, why was it decided to use Catalan as zero-shot when Spanish is included, was trialling a less-related language as zero-shot considered?\n\n--\n\n_Thank you for your answers to these questions which I am satisfied with_", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your detailed review. We will of course address the comments regarding presentation improvements, thank you. Below some comments and answers to the other issues (Reasons to Reject with prefix RX, Questions: QX).\n\n**R1**: We completely agree that considering the problem as a binary (L/R) classification is a simplification of the problem and we discussed the issue in the Limitations section. We think it is a limitation both for short texts (García-Díaz et al.) and long texts (us), so that reference is not intended to be a justification but just to show what has been done before. Our motivation for the simplification is to be able to do an analysis that goes beyond US English text. The annotation of data is not only expensive in time and resources, it also needs a specific training for the annotators. To give an example, a specialist site as Ad Fontes [1] has 60 analysts which receive a 30h initial training plus an additional training of 40h/year. Besides, for every article, there must be the same number of annotators with self-identified bias from the Left, Right and Centre to get reliable results according to the AllSides site [2]. Doing this is not feasible for most research groups and, as a consequence, we mostly have analyses with US data. In our case, we try to simplify the problem to be able to cover more countries and languages in almost an automatic way. We show that the results of our binary classification in 4 languages are reliable, and it is easy to extend to other languages with few language knowledge, but the same method would not work with a 5 degrees scale. We believe that being able to give a coarse indication, just L vs R, for many languages can be useful. In our work, we wouldn't have been able to analyse ChatGPT in Spanish and Catalan otherwise for example.\n\n**R2**: Heuristics for cleaning. We manually inspected the data to detect patterns mainly for newspaper-specific headers and footers. We eliminated these lines in the articles with regular expressions, but this is not part of the main code. We will add the regular expressions and the patterns in the github together with the main code so that all the process can be reproduced.\n\n**R3**: ChatGPT prompts. The stylistic output of chatGPT is in fact a very interesting topic. We tried different prompts with no significant differences in the style, length or content generated, so we went with the most simple one. It might be that chatGPT was/is generating blog-like text and this deserves a follow-up work where we analyse and compare the stylistic features of real blogs, news and chatGPT when asked to generate blogs and news. The goal of this short paper is only the political leaning. I would consider the style also a characteristic of the leaning though, so if chatGPT style resembles more to left or right-oriented ways to talk we think this is also a valid feature to classify the generated text. Unfortunately, we cannot show and prove the following with a reference, but for the sake of the argument, in an independent work we did some simple stylistic analysis of the data available in the Manifesto Project [3] (political parties’ election manifestos) and obtain that right-leaning texts are less complex, include less numbers, etc. confirming that the style of the text is also a characteristic of the leaning.\n\n\n**QA**: Yes, a significant amount of real newspaper articles are above the 512 limit, but we expect articles to have the most relevant information at the beginning. You are right and this might not be the case for chatGPT but the generated articles are in general shorter: only the last version of chatGPT for Spanish and English is slightly longer. To be sure that the length is not relevant and the training data we use has the relevant information at the beginning, we implemented a version of the classifier that averages the embeddings of the document at sentence level before classification so that all the document is considered (--split_documents option in the code). We did not find significant differences on the experiments we performed but training time was significantly slower so we decided not to include this in the paper.\n\n**QB**: Not explicitly, but the patterns for the heuristics included strings such as \"The AP news staff was not involved in its creation\" or \"data provided by Refinitiv Lipper.\" Our current corpus has 1.2M articles in 4 languages so it might be the case that some self-references are still there. We plan to provide another version of the corpus covering more countries and with another round of cleaning but 100% removal of these expressions multilingually will not be possible as this would imply going though the million(s) of articles manually.\n\n**QC**: We chose the languages for which we could get help from native speakers in our lab. We used these native speakers to manually translate from English the 101 subjects we use, and to check the adequacy of the prompts we provide to chatGPT. We also asked them feedback about the newspapers we are using for testing the classifier. We acknowledge that with our models zero-shot for Catalan will perform better than zero-shot for Icelandic and we tried to make it clear in the paper by specifying that the good performance is on close languages.\n\nFinally, we would like to notify to the 3 reviewers that during the reviewing period 2 very interesting things happened: 1) a new version of chatGPT was deployed and 2) a multilingual Bard is now available. With this we can extend our results (Table 2) with 4 more rows: 2 generations with the newest version of chatGPT (named v08a and v08b in the table below) and 2 generations with Bard. Our aim is to reflect the variability (or lack of!) among generations with a same chatGPT/Bard version in a systematic manner (it was just a footnote in the submitted version as noticed by one reviewer) and, to the best of our knowledge, we are the first ones to analyse Bard's bias multilingualy.\n\nA very interesting thing to notice in the new results is that the period of time when chatGPT produced more right-oriented content corresponds to the time when chatGPT experienced a performance drop in several tasks according to [4]. The old and new results can be seen in the table below.\n\n|                   | English     |||| German       |||| Spanish     |||| Catalan  ||\n|-------------------|------------------------|-------------------------|------------------------|-------------------------|------------------------|-------------------------|-------------------------|----------|--------|---------|--------|---------|--------|---------|\n|                   | Mono || Multi || Mono || Multi ||Mono || Multi || Multi ||\n| (\\% of articles)  | L                 | R                 | L                 | R                 | L                 | R                 | L                  | R  | L | R | L | R | L | R |\n| chatGPT v02       | **75&plusmn;9**         | 25&plusmn;8                | **93&plusmn;5**         | 7&plusmn;5                | --                     | --                      | --                      | -- | **65&plusmn;10**   | 35&plusmn;10              | **53&plusmn;10**         | 47&plusmn;10              | --                      | --                     |\n| chatGPT v03       | --                     | --                      | --                     | --                      | **97&plusmn;4**         | 3&plusmn;3                 | **69&plusmn;9**          | 31&plusmn;9 | --     | --      | --     | --      | --     | --      |\n| chatGPT v05       | 26&plusmn;9               | **74&plusmn;9**          | 40&plusmn;9               | **60&plusmn;9**          | **96&plusmn;5**         | 4&plusmn;3                 | **65&plusmn;9**          | 35&plusmn;9     | 25&plusmn;9               | **75&plusmn;9**          | 26&plusmn;9               | **74&plusmn;8**          | **71&plusmn;9**         | 29&plusmn;9                |\n| chatGPT v08a      | **54&plusmn;10**        | 46&plusmn;10               | **85&plusmn;8**         | 15&plusmn;6                | **99&plusmn;3**         | 1&plusmn;1                 | **100&plusmn;2**         | 0&plusmn;0                    | 50&plusmn;10              | 50&plusmn;10               | 40&plusmn;10              | **60&plusmn;10**         | 50&plusmn;10              | 50&plusmn;9                |\n| chatGPT v08b      | **52&plusmn;10**        | 48&plusmn;10               | **85&plusmn;8**         | 15&plusmn;6                | **100&plusmn;2**        | 0&plusmn;0                 | **100&plusmn;2**         | 0&plusmn;0                  | **51&plusmn;10**        | 49&plusmn;10               | 36&plusmn;10              | **64&plusmn;9**          | 47&plusmn;10              | **53&plusmn;10**         |\n| bard v08a         | **57&plusmn;11**        | 43&plusmn;10               | **75&plusmn;9**         | 25&plusmn;8                | **82&plusmn;8**         | 18&plusmn;7                | **82&plusmn;8**          | 18&plusmn;7                   | **74&plusmn;9**         | 26&plusmn;8                | 35&plusmn;9               | **65&plusmn;9**          | **66&plusmn;9**         | 34&plusmn;9                |\n| bard v08b         | **61&plusmn;10**        | 39&plusmn;10               | **82&plusmn;8**         | 18&plusmn;7                | **81&plusmn;8**         | 19&plusmn;7                | **90&plusmn;7**          | 10&plusmn;5 | **74&plusmn;9**         | 26&plusmn;8                | 44&plusmn;10              | **56&plusmn;10**         | **68&plusmn;9**         | 32&plusmn;9                |\n\n\n[1] https://adfontesmedia.com/\n\n[2] https://www.allsides.com/unbiased-balanced-news\n\n[3] https://manifesto-project.wzb.eu/information/documents/information\n\n[4] Chen, Lingjiao, Matei Zaharia, and James Zou. \"How is ChatGPT's behavior changing over time?.\" arXiv preprint arXiv:2307.09009 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper provides an interesting analysis of the political bias of ChatGPT's output. This has been a contentious topic, with notable public figures criticising OpenAI's approach to reducing bias and improving safety, claiming they are inserting political biases into the model. This work demonstrates that the news output of ChatGPT is notably politically biased, and this bias has (in English, for example) flipped between the oldest and newest versions.\n\nParticular strengths:\n- Considering which news topics are most likely to be politically divisive, both in general and for specific cultures\n- Evaluating multiple versions of ChatGPT, with consideration that other factors (article length) are not responsible for polarity shift. _This is furthered by data provided in the author response, which includes subsequent versions of ChatGPT and results for Bard_\n- I found the methodology simple and easy to follow", "reasons_to_reject": "This paper only considers binary Left/Right political stance, a simplified view of a highly complex problem. The authors note that the prior work of García-Díaz et al. using the same label schema is on social media posts, but do not discuss the implications of using the same simple labels on long-form journalistic texts. _Given the rebuttal I accept that this simplification is necessary. I still think that the paper would benefit from acknowledging this simplification is likely to be more significant for these long-form texts_\n\nNot enough information is provided to reproduce the heuristic cleaning in the preprocessing stage - the code in the supplementary information does not appear to include this. _The authors have stated they will add this to the code, so I believe this is no longer an issue_\n\nThe prompt used in ChatGPT is very simplistic. Since it is instructed to create articles on a topic, rather than an event as real news would be, I am concerned about the stylistic differences this will cause. Reading just the first few articles in English, it appears many of the generated articles resemble 'blogspam' more than news (although there are some articles where it has created a fictional event to write about). It is a possibility that the linguistic properties of the generated articles (which may also change over time as the model is updated) are more similar to one leaning of news or the other, which the classifier could be using instead of the actual content of the articles.\n\n _Following the subsequent author comment I am satisfied that this is in fact a property of both the real and ChatGPT-generated corpora, so is less likely to be problematic_", "questions_to_authors": "A: The average word counts of the data suggests a significant amount of it exceeds the maximum token length of the classification model. Was any consideration given to this? Whilst in newspaper articles, the most significant information is typically at the beginning, there is no guarantee that ChatGPT can replicate this structure.\n\nB: During preprocessing, were self-references to the newspapers removed, as these would also allow classifiers to learn sources? e.g. \"[SOURCE] was unable to reach x for comment\", \"x, speaking exclusively to [SOURCE]\" etc. \n\nC: What motivated the choice of languages? Particularly, why was it decided to use Catalan as zero-shot when Spanish is included, was trialling a less-related language as zero-shot considered?\n\n--\n\n_Thank you for your answers to these questions which I am satisfied with_", "paper_topic_and_main_contributions": "This paper analyses whether a hypothetical ChatGPT-authored news source would be considered left or right-leaning, if media observers assessed it. It covers English, German, Spanish and Catalan.\n\nIts main contributions are:\n- Showing it is possible to determine the L/R stance of unseen real media outlets with a classifier trained on real news articles labelled with the L/R stance of their source. This trialled both monolingual and multilingual models, and zero-shot performance.\n- A dataset of ChatGPT-authored news articles on contentious topics in the four languages, generated using multiple versions of ChatGPT\n- Analysis of the classifier on the dataset, showing that all considered versions of ChatGPT exhibit strong bias. It shows that the bias varies across languages, and that this bias has flipped between different versions.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I found the explanation of article length (lines 264-268) confusing initially. I think it would be clearer to represent the comparison in length distribution as a graph. Also, it discusses token length when table 1 used word count - I think this should be consistent.\n\nI found table 2 hard to understand initially. Perhaps it would be clearer if the text \"Classification (% of articles)\" ran across the top of the data, to make it clearer the L and R columns are the predicted labels."}}, {"review_id": "3ynCrC1MgV", "reviewer": "Reviewer_75sN", "summary": "", "questions": "1- Instead of using LDA, simple filtering based on a category assigned by the source could be used (I know it is tricky for many sources, but you do not need many sources for the aims of this study). A simple classifier could be a straightforward tool for filtering as well.\n\n2- Topic detection may be affected by the ideology of the source. What were the parameters of the topic modeling, how many times did it run, etc.? \n\n3- How much of the data was discarded using LDA-based filtering?\n\n4- Is the data open (copyright, etc.)? Or do you plan to release at least the URLs of the articles?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all thanks for all your comments. Let us clarify some concepts and answer your questions below (we use QX for questions and PX for presentation improvements). We will extend the appendix with more statistics of the corpus to cover some of your comments. If there is the opportunity during the rebuttal, it would be very useful for us if you could specify the gaps in the report (reason to reject) or if those are related to the LDA questions which we address here. It would be also useful to know which are the dark points in the evaluation, so that we can clarify them (basically we use accuracy to evaluate our classifiers).\n\nSome of the comments and questions can be explained by the nature of the training data, which has been extracted from OSCAR [1] an open source project which does an additional cleaning to Common Crawl, a repository of web crawled data [2].\n\n**Q1**: We use OSCAR because it contains data in 166 languages and the data there is open; given that, anyone can easily extend the current analysis to more languages. We extracted 1,258,212 articles from 116 newspapers (Sec.2), but being an extensive web crawl, there is no information about the category of each article. We find important to have as many sources (newspapers) as possible in order to improve generalisation. Training a classifier to detect the category implies pre-specifying the categories and using some annotated data for training (which we don't have in general). Even if we think this is the way to go when dealing in depth with one language, the aim of our method is to go multilingual, cross-cultural and facilitate the analysis in any language. That includes languages for which no annotated data with categories is freely available. Also notice that with LDA, topics differ across languages (German has the topic we label as Nazism and Immigration appears in English but not in the others for example). We can capture this only with unsupervised methods, a newspaper would never categorise an article this way.\n\n**Q2**: Do you mean that the topics are different for the left and for the right newspapers? We studied this when creating the corpus to be sure that the corpus is balanced or at least that all classes are well represented. We will add the tables in the new appendix. To summarise: what happens is that the variation in newspapers across a same leaning is larger than the difference between L and R, although it is true that there exist differences. For instance, for the left in English we consider among others The Economist (mostly talking about \"International\" and \"Economy\") and Politico (mostly talking about \"Government\"), but when considering all the sources, all topics are covered. For this reason it is important to have as many sources as possible. Another example, in average, the largest difference of topic in Spanish newspapers is given for topic \"Law and Justice\" (15,024 articles L vs 10,068 R), so even if there is a difference, both leanings are well represented. Finally, let us point out that we expect the classifier not to learn only the content but also the style of L and R, we do the selection by topic only to minimise the number of neutral articles. When we generate the chatGPT outputs, we use more topics such as those related to Sport or Science which have not been seen during training but could have a different style depending on the leaning of the writer.\n\nWe use Mallet to perform LDA on the corpus after removing the stopwords, with the hyperparameter optimization option activated and done every 10 iterations. Other parameters are the defaults. We do a run per language with 10 topics and another run with 15 topics. We tag the corpus with both labels.\n\n\n**Q3**: The percentage depends on the language. The method filters out 49% of the Spanish articles, 39% of the German ones and 31% of the English ones. We will add this information per source and topic in the new appendix. \n\n**Q4**: The data is a processed subset of OSCAR, so it will be freely available in the same terms as OSCAR. For every article, we will provide the OSCAR ID, the link to the article as given by OSCAR, the class we assign to the article according to its source (L/R), our cleaned text, and our 2 topic classes.\t\n\n**P3**: We manually inspected OSCAR's articles to detect patterns for newspaper-specific headers and footers. We eliminated these lines in the articles with regular expressions. The regular expressions and the patterns will be available in the Github together with the main code so that all the process can be reproduced.\n\n**P4**: We defined 101 subjects in English to prompt chatGPT (left column of Table 7). Then, we asked native speakers of German, Spanish and Catalan to manually translate them. This process is bias free, as only the concept itself (i.e., \"technology\") is translated.\n\n**P5**: We tested 3 versions of chatGPT (v02, v03 and v05 in Tables 1 and 2) but only v05 is simultaneously tested for the 4 languages. This changed now, as during the reviewing period a new version appeared which has been also tested for the 4 languages (see v08 below).\n\n**P6**: For German, the result was exactly the same (L=97&plusmn;4, R=3&plusmn;3 with both generations); for Catalan, the result lied within 95% confidence intervals (generation1: L=47&plusmn;10, R=53&plusmn;9 vs generation 2: L=50&plusmn;10, R=50&plusmn;9). We will remove this footnote and add the results for 2 generations in 4 languages with the new chatGPT version (see v08 below).\n\n**P7**: This is what Table 2 shows, but there is probably a misunderstanding between training and test which we will clarify. For training, a source (i.e., The Economist) is either L or R at 100%. For The Economist all the articles are labelled as Left articles in our training data. This is the assumption of our method, and the main reason why we try to filter out articles prone to be neutral in a newspaper. At testing time, we classify every article independently and look at the percentage in the test newspaper or chatGPT generations. This is Table 2. There cannot be an average across newspapers: we only use a newspaper for testing as the main goal is chatGPT, the newspaper we test is only to see whether the classifier is able to generalise to unseen newspapers or not as this has been shown to be a hard task [3]. We could average chatGPT versions, but one of our conclusions is precisely that it changes with time.\n\nFinally, we would like to notify to the 3 reviewers that during the reviewing period 2 very interesting things happened: 1) a new version of chatGPT was deployed as we said above and 2) a multilingual Bard is now available. With this we can extend our results (Table 2) with 4 more rows: 2 generations with the newest version of chatGPT (named v08a and v08b in the table below) and 2 generations with Bard. Our aim is to reflect the variability (or lack of!) among generations with a same chatGPT/Bard version in a systematic manner and substitute the footnote in the submitted paper (P6 improvement above) by the complete results. Also, and to the best of our knowledge, we will be the first ones to analyse Bard's bias multilingually. This experiment will enlarge our corpus of multilingual AI-generated newspaper articles that can be used for further research.\n\nA very interesting thing to notice in the new results is that the period of time when chatGPT produced more right-oriented content corresponds to the time when chatGPT experienced a performance drop in several tasks according to [4]. The old and new results can be seen in the table below.\n\n|                   | English     |||| German       |||| Spanish     |||| Catalan  ||\n|-------------------|------------------------|-------------------------|------------------------|-------------------------|------------------------|-------------------------|-------------------------|----------|--------|---------|--------|---------|--------|---------|\n|                   | Mono || Multi || Mono || Multi ||Mono || Multi || Multi ||\n| (\\% of articles)  | L                 | R                 | L                 | R                 | L                 | R                 | L                  | R  | L | R | L | R | L | R |\n| chatGPT v02       | **75&plusmn;9**         | 25&plusmn;8                | **93&plusmn;5**         | 7&plusmn;5                | --                     | --                      | --                      | -- | **65&plusmn;10**   | 35&plusmn;10              | **53&plusmn;10**         | 47&plusmn;10              | --                      | --                     |\n| chatGPT v03       | --                     | --                      | --                     | --                      | **97&plusmn;4**         | 3&plusmn;3                 | **69&plusmn;9**          | 31&plusmn;9 | --     | --      | --     | --      | --     | --      |\n| chatGPT v05       | 26&plusmn;9               | **74&plusmn;9**          | 40&plusmn;9               | **60&plusmn;9**          | **96&plusmn;5**         | 4&plusmn;3                 | **65&plusmn;9**          | 35&plusmn;9     | 25&plusmn;9               | **75&plusmn;9**          | 26&plusmn;9               | **74&plusmn;8**          | **71&plusmn;9**         | 29&plusmn;9                |\n| chatGPT v08a      | **54&plusmn;10**        | 46&plusmn;10               | **85&plusmn;8**         | 15&plusmn;6                | **99&plusmn;3**         | 1&plusmn;1                 | **100&plusmn;2**         | 0&plusmn;0                    | 50&plusmn;10              | 50&plusmn;10               | 40&plusmn;10              | **60&plusmn;10**         | 50&plusmn;10              | 50&plusmn;9                |\n| chatGPT v08b      | **52&plusmn;10**        | 48&plusmn;10               | **85&plusmn;8**         | 15&plusmn;6                | **100&plusmn;2**        | 0&plusmn;0                 | **100&plusmn;2**         | 0&plusmn;0                  | **51&plusmn;10**        | 49&plusmn;10               | 36&plusmn;10              | **64&plusmn;9**          | 47&plusmn;10              | **53&plusmn;10**         |\n| bard v08a         | **57&plusmn;11**        | 43&plusmn;10               | **75&plusmn;9**         | 25&plusmn;8                | **82&plusmn;8**         | 18&plusmn;7                | **82&plusmn;8**          | 18&plusmn;7                   | **74&plusmn;9**         | 26&plusmn;8                | 35&plusmn;9               | **65&plusmn;9**          | **66&plusmn;9**         | 34&plusmn;9                |\n| bard v08b         | **61&plusmn;10**        | 39&plusmn;10               | **82&plusmn;8**         | 18&plusmn;7                | **81&plusmn;8**         | 19&plusmn;7                | **90&plusmn;7**          | 10&plusmn;5 | **74&plusmn;9**         | 26&plusmn;8                | 44&plusmn;10              | **56&plusmn;10**         | **68&plusmn;9**         | 32&plusmn;9                |\n\n[1] https://oscar-project.org/\n\n[2] https://commoncrawl.org/\n\n[3] [We Can Detect Your Bias: Predicting the Political Ideology of News Articles](https://aclanthology.org/2020.emnlp-main.404) (Baly et al., EMNLP 2020)\n\n[4] Chen, Lingjiao, Matei Zaharia, and James Zou. \"How is ChatGPT's behavior changing over time?.\" arXiv preprint arXiv:2307.09009 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1- The research question is interesting.\n\n2- The methodology chosen is a proper one.\n\n3- The newspaper corpus is valuable. It is multilingual. The ChatGPT-generated news articles are valuable as well.", "reasons_to_reject": "1- It is hard to understand the paper. There are many gaps in the report.\n\n2- The evaluation is not clear.", "questions_to_authors": "1- Instead of using LDA, simple filtering based on a category assigned by the source could be used (I know it is tricky for many sources, but you do not need many sources for the aims of this study). A simple classifier could be a straightforward tool for filtering as well.\n\n2- Topic detection may be affected by the ideology of the source. What were the parameters of the topic modeling, how many times did it run, etc.? \n\n3- How much of the data was discarded using LDA-based filtering?\n\n4- Is the data open (copyright, etc.)? Or do you plan to release at least the URLs of the articles?", "paper_topic_and_main_contributions": "The authors measure political stance (left vs right) of ChatGPT over a couple of periods. A corpus of newspapers is collected based on its political bias (left vs right) in multiple languages and refined using LDA. The corpus is utilized to develop several classifiers for political stances. This classifier is used to determine stance of news articles created by ChatGPT in a couple of periods.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1- formatting of the numbers, e.g. 742691 -> 742,691\n\n2- comma's between words in Table 4 could improve readability.\n \n3- \"We apply heuristics to automatise the task.\" -> what do you mean?\n\n4- \"and translate them into the 4 languages\"? What was translated and how (manual vs automatic). Does this process introduce any bias?\n\n5- \"but only the last one is available for the 4 languages.\" I could not understand this.\n\n6- \"we prompt the same ChatGPT model twice to check that multiple generations do not lead to significantly different results.\" what was the result of this check?\n\n7- \"if more than 50% of its articles have been classified as such.\", good, so what was the average across all sources? min and max and distribution of the averages per source?"}}, {"review_id": "R8pqZD0tZn", "reviewer": "Reviewer_3VUk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments. We would like to first clarify two points of the \"Paper Topic And Main Contributions\" that do not correspond to what we reflect in the paper:\n\n- \"filtered for politically divisive articles (abortion, the death penalty, etc.)\"\n\nAbortion, the death penalty, but also Olympic games, Richard Gere, etc. are the subjects we use to generate articles with chatGPT (Table 7), not for filtering the training data (Sec2-ChatGPT Corpus). For filtering, we apply LDA on the data and obtain broad topics that we label as \"International\", \"Government\", \"Law & Justice\", \"Sports\"... (Tables 4-6, and Sec2-Topic Modelling), then, we select a subset of the broad topics for training as explained in Section 2.\n\n- \"The results show a left political bias across all languages in the test data.\"\n\nOur main conclusion already summarised in the abstract is \"ChatGPT editorial line evolves with time and, being a data-driven system, the stance of the generated articles differs among languages.\" The analysis Section 3 with the results in Table 2 show that, for instance, the last chatGPT version we analyse (v05) has a right-leaning in English and Spanish. In the Conclusion we state \"We show that ChatGPT writes newspaper articles with a Right-oriented perspective for English and Spanish, and results seem to indicate a Left orientation for German and Catalan.\"\n\nThese misunderstandings are related to the reasons to reject the paper we would like to comment:\n\n**1a)** We are not reproducing previous results, as we don't claim that chatGPT is biased towards the left, but that the leaning depends on the language and the period. We are providing a classifier to easily check the stance of a new version when needed. \n\n**1b)** Still in the same paragraph, we agree that LMs reproduce the biases of the training data, but in the case of chatGPT we don't know the training data (and even less the human annotations for the reinforcement learning from human feedback phase), so, under our point of view, it makes sense to study the biases it has.\n \n**2a)** We agree that a filtering that only selects papers about immigration or death penalty would introduce a bias but, as explained above, this is not what we are doing: we test, not filter, on those topics (and others such as \"technology\", \"television\", \"the Olympic games\"...). In the whole, the training corpus is balanced in the number of articles left vs right.\n\n**2b)** \"When they do appear, neutral language will be employed. The opposite is likely true for a left-leaning publication, for whom the existence of the death penalty is a problem: the articles will occur with more frequency and will employ less neutral language.\" Assuming that articles about death penalty appear in the training data, the model could hopefully learn as you say that if the language is not neutral it is a left-oriented article and if the language is neutral is right-oriented. And this is what we want. Now, if we use the classifier on chatGPT we will know if chatGPT's language is more or less neutral.\n\n**2c)** \"which was annotated via distant supervision using political slant tags assigned by Wikipedia\". This is not what we say in the paper, in Section 2 we write \"We do not manually annotate any article, but we trust AllSides, MB/FC, Political Watch and Wikipedia (in cases where the information is not available in the previous sites) with their classification of a newspaper bias.\" Wikipedia is a backup for the sites that do bias annotation in a professional manner, and that we used in 4 sources and double checked with natives, as Germany is mostly not covered by those sites. We can add this information (where the leaning of every source comes from) in Table 3. We would like to stress that our main sources are AllSides, MB/FC and Political Watch. \n\nFinally, we would like to notify to the 3 reviewers that during the reviewing period 2 very interesting things happened: 1) a new version of chatGPT was deployed and 2) a multilingual Bard is now available. With this we can extend our results (Table 2) with 4 more rows: 2 generations with the newest version of chatGPT (named v08a and v08b in the table below) and 2 generations with Bard. Our aim is to reflect the variability (or lack of!) among generations with a same chatGPT/Bard version in a systematic manner (it was just a footnote in the submitted version as noticed by one reviewer) and, to the best of our knowledge, we are the first ones to analyse Bard's bias multilingualy.\n\nA very interesting thing to notice in the new results is that the period of time when chatGPT produced more right-oriented content corresponds to the time when chatGPT experienced a performance drop in several tasks according to [1]. The old and new results can be seen in the table below.\n\n|                   | English     |||| German       |||| Spanish     |||| Catalan  ||\n|-------------------|------------------------|-------------------------|------------------------|-------------------------|------------------------|-------------------------|-------------------------|----------|--------|---------|--------|---------|--------|---------|\n|                   | Mono || Multi || Mono || Multi ||Mono || Multi || Multi ||\n| (\\% of articles)  | L                 | R                 | L                 | R                 | L                 | R                 | L                  | R  | L | R | L | R | L | R |\n| chatGPT v02       | **75&plusmn;9**         | 25&plusmn;8                | **93&plusmn;5**         | 7&plusmn;5                | --                     | --                      | --                      | -- | **65&plusmn;10**   | 35&plusmn;10              | **53&plusmn;10**         | 47&plusmn;10              | --                      | --                     |\n| chatGPT v03       | --                     | --                      | --                     | --                      | **97&plusmn;4**         | 3&plusmn;3                 | **69&plusmn;9**          | 31&plusmn;9 | --     | --      | --     | --      | --     | --      |\n| chatGPT v05       | 26&plusmn;9               | **74&plusmn;9**          | 40&plusmn;9               | **60&plusmn;9**          | **96&plusmn;5**         | 4&plusmn;3                 | **65&plusmn;9**          | 35&plusmn;9     | 25&plusmn;9               | **75&plusmn;9**          | 26&plusmn;9               | **74&plusmn;8**          | **71&plusmn;9**         | 29&plusmn;9                |\n| chatGPT v08a      | **54&plusmn;10**        | 46&plusmn;10               | **85&plusmn;8**         | 15&plusmn;6                | **99&plusmn;3**         | 1&plusmn;1                 | **100&plusmn;2**         | 0&plusmn;0                    | 50&plusmn;10              | 50&plusmn;10               | 40&plusmn;10              | **60&plusmn;10**         | 50&plusmn;10              | 50&plusmn;9                |\n| chatGPT v08b      | **52&plusmn;10**        | 48&plusmn;10               | **85&plusmn;8**         | 15&plusmn;6                | **100&plusmn;2**        | 0&plusmn;0                 | **100&plusmn;2**         | 0&plusmn;0                  | **51&plusmn;10**        | 49&plusmn;10               | 36&plusmn;10              | **64&plusmn;9**          | 47&plusmn;10              | **53&plusmn;10**         |\n| bard v08a         | **57&plusmn;11**        | 43&plusmn;10               | **75&plusmn;9**         | 25&plusmn;8                | **82&plusmn;8**         | 18&plusmn;7                | **82&plusmn;8**          | 18&plusmn;7                   | **74&plusmn;9**         | 26&plusmn;8                | 35&plusmn;9               | **65&plusmn;9**          | **66&plusmn;9**         | 34&plusmn;9                |\n| bard v08b         | **61&plusmn;10**        | 39&plusmn;10               | **82&plusmn;8**         | 18&plusmn;7                | **81&plusmn;8**         | 19&plusmn;7                | **90&plusmn;7**          | 10&plusmn;5 | **74&plusmn;9**         | 26&plusmn;8                | 44&plusmn;10              | **56&plusmn;10**         | **68&plusmn;9**         | 32&plusmn;9                |\n\n\n\n[1] Chen, Lingjiao, Matei Zaharia, and James Zou. \"How is ChatGPT's behavior changing over time?.\" arXiv preprint arXiv:2307.09009 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Diagnosing bias in LLMs, particularly ChatGPT, is a timely research topic and the authors have made an interesting attempt at providing a different approach to quantifying the bias in ChatGPT that has been reported extensively in the social science literature.\n\n- The paper is overall well-written and sensibly organized", "reasons_to_reject": "- The authors have reproduced a well-known result in the literature--left political bias in ChatGPT and in LLMs in general--using the \"coarse\" (their description) methodology of passing a binary stance classifier over ChatGPT's output. The observation that language models reproduce the biases of the corpora on which they're trained has been made at each step of the evolution of these models, from word2vec to BERT to ChatGPT, and so it's unclear why this observation needs to once again be made using the authors \"coarse\" methodology. \n\n- The authors' decision to filter by divisive topic using introduces an unacknowledged prior: for most of the topics given in the appendix (immigration, the death penalty, etc.), the choice to bother writing about the topic itself introduces bias. This choice will be reflected in both the frequency with which such articles appear and in the language used in those articles. The existence of the death penalty, for example, might be considered unproblematic on the right and, for that reason, one will see fewer articles on the subject in right-leaning newspapers.  When they do appear, neutral language will be employed. The opposite is likely true for a left-leaning publication, for whom the existence of the death penalty is a problem: the articles will occur with more frequency and will employ less neutral language. For these reasons, it's not surprising that the authors' classifier, which was annotated via distant supervision using political slant tags assigned by Wikipedia, will tend to score most content generated by an LLM as left-leaning.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors present the results of a series of political stance classification experiments with the goal of identifying political bias in ChatGPT.  A corpus of multilingual newspaper articles tagged using distant supervision and then filtered for politically divisive articles (abortion, the death penalty, etc.) is used to train a binary political stance classifier, which achieves 95% accuracy on a held-out test set. The classifier is then passed over a series of ChatGPT-authored articles on politically divisive issues. The results show a left political bias across all languages in the test data.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree that this paper is sound and moderately exciting, adding to existing work on understanding of political bias in LLMs. Reviewers raised some concerns about clarity and particular methodological choices; authors provided an extensive set of rebuttals including updated empirical results with newer multilingual models and satisfied many reviewer questions and concerns. 

Reviewers noted several issues that could be clarified in any final version of the paper, including: a) that binary left/right political stance is a simplification and potential limitation with long-form text, b) that the corpus includes non-news articles, c) some further explanation on the heuristics used for data cleanliness, and so on. We ask the authors to integrate these changes and clarifications following reviewer feedback.
Decision: Accept-Findings