Introduction
In a survey across 38 countries, the Pew Research Center reported that the global public opposed partisanship in news media BIBREF0 . It is, however, challenging to assess the partisanship of news articles on a large scale. We thus made an effort to create a dataset of articles annotated with political partisanship so that content analysis systems can benefit from it.
To construct a dataset of news articles labeled with partisanship, it is required that some annotators read each article and decide whether it is partisan. This is an expensive annotation process. Another way to derive a label for an article is by using the partisanship of the publisher of the article. Previous work has used this method BIBREF1 , BIBREF2 , BIBREF3 . This labeling paradigm is premised on that partisan publishers publish more partisan articles and non-partisan publishers publish more non-partisan articles. Although there would be non-partisan articles published by partisan publishers (and vice versa), and thus labeled wrongly, the assumption ensures more information than noise. Once the partisanship of a publisher is known, the labels of all its articles are known, which is fast and cheap. We created a dataset of two parts. The first part contains a large number of articles that were labeled using the partisanship of publishers. The second part contains a few hundreds of articles that were annotated by readers who were asked to read each article and answer survey questions. In the following sections, we describe the collection and annotation of both parts of the dataset.
Dataset description
DpgMedia2019 is a Dutch dataset that was collected from the publications within DPG Media. We took 11 publishers in the Netherlands for the dataset. These publishers include 4 national publishers, Algemeen Dagblad (AD), de Volkskrant (VK), Trouw, and Het Parool, and 7 regional publishers, de Gelderlander, Tubantia, Brabants Dagblad, Eindhovens Dagblad, BN/De Stem PZC, and de Stentor. The regional publishers are collectively called Algemeen Dagblad Regionaal (ADR). A summary of the dataset is shown in Table TABREF3 .
Publisher-level data
We used an internal database that stores all articles written by journalists and ready to be published to collect the articles. From the database, we queried all articles that were published between 2017 and 2019. We filtered articles to be non-advertisement. We also filtered on the main sections so that the articles were not published under the sports and entertainment sections, which we assumed to be less political. After collecting, we found that a lot of the articles were published by several publishers, especially a large overlap existed between AD and ADR. To deal with the problem without losing many articles, we decided that articles that appeared in both AD and its regional publications belonged to AD. Therefore, articles were processed in the following steps:
Remove any article that was published by more than one national publisher (VK, AD, Trouw, and Het Parool). This gave us a list of unique articles from the largest 4 publishers.
Remove any article from ADR that overlapped with the articles from national publishers.
Remove any article that was published by more than one regional publisher (ADR).
The process assured that most of the articles are unique to one publisher. The only exceptions were the AD articles, of which some were also published by ADR. This is not ideal but acceptable as we show in the section UID8 that AD and ADR publishers would have the same partisanship labels. In the end, we have 103,812 articles.
To our knowledge, there is no comprehensive research about the partisanship of Dutch publishers. We thus adopted the audience-based method to decide the partisanship of publishers. Within the survey that will be explained in section SECREF11 , we asked the annotators to rate their political leanings. The question asked an annotator to report his or her political standpoints to be extreme-left, left, neutral, right, or extreme-right. We mapped extreme-left to -2, left to -1, center to 0, right to 1, extremely-right to 2, and assigned the value to each annotator. Since each annotator is subscribed to one of the publishers in our survey, we calculated the partisanship score of a publisher by averaging the scores of all annotators that subscribed to the publisher. The final score of the 11 publishers are listed in Table TABREF9 , sorted from the most left-leaning to the most right-leaning.
We decided to treat VK, Trouw, and Het Parool as partisan publishers and the rest non-partisan. This result largely accords with that from the news media report from the Pew Research Center in 2018 BIBREF4 , which found that VK is left-leaning and partisan while AD is less partisan.
Table TABREF10 shows the final publisher-level dataset of dpgMedia2019, with the number of articles and class distribution.
Article-level data
To collect article-level labels, we utilized a platform in the company that has been used by the market research team to collect surveys from the subscribers of different news publishers. The survey works as follows: The user is first presented with a set of selected pages (usually 4 pages and around 20 articles) from the print paper the day before. The user can select an article each time that he or she has read, and answer some questions about it. We added 3 questions to the existing survey that asked the level of partisanship, the polarity of partisanship, and which pro- or anti- entities the article presents. We also asked the political standpoint of the user. The complete survey can be found in Appendices.
The reason for using this platform was two-fold. First, the platform provided us with annotators with a higher probability to be competent with the task. Since the survey was distributed to subscribers that pay for reading news, it's more likely that they regularly read newspapers and are more familiar with the political issues and parties in the Netherlands. On the other hand, if we use crowdsourcing platforms, we need to design process to select suitable annotators, for example by nationality or anchor questions to test the annotator's ability. Second, the platform gave us more confidence that an annotator had read the article before answering questions. Since the annotators could choose which articles to annotate, it is more likely that they would rate an article that they had read and had some opinions about.
The annotation task ran for around two months in February to April 2019. We collected annotations for 1,536 articles from 3,926 annotators.
For the first question, where we asked about the intensity of partisanship, more than half of the annotations were non-partisan. About 1% of the annotation indicated an extreme partisanship, as shown in Table TABREF13 . For the polarity of partisanship, most of the annotators found it not applicable or difficult to decide, as shown in Table TABREF14 . For annotations that indicated a polarity, the highest percentage was given to progressive. Progressive and conservative seemed to be more relevant terms in the Netherlands as they are used more than their counterparts, left and right, respectively.
As for the self-rated political standpoint of the annotators, nearly half of the annotators identified themselves as left-leaning, while only around 20% were right-leaning. This is interesting because when deciding the polarity of articles, left and progressive ratings were given much more often than right and conservative ones. This shows that these left-leaning annotators were able to identify their partisanship and rate the articles accordingly.
We suspected that the annotators would induce bias in ratings based on their political leaning and we might want to normalize it. To check whether this was the case, we grouped annotators based on their political leaning and calculate the percentage of each option being annotated. In Figure FIGREF16 , we grouped options and color-coded political leanings to compare whether there are differences in the annotation between the groups. We observe that the "extreme-right" group used less "somewhat partisan", "partisan", and "extremely-partisan" annotations. This might mean that articles that were considered partisan by other groups were considered "non-partisan" or "impossible to decide" by this group. We didn't observe a significant difference between the groups. Figure FIGREF17 shows the same for the second question. Interestingly, the "extreme-right" group gave a lot more "right" and slightly more "progressive" ratings than other groups. In the end, we decided to use the raw ratings. How to scale the ratings based on self-identified political leaning needs more investigation.
The main question that we are interested in is the first question in our survey. In addition to the 5-point Likert scale that an annotator could choose from (non-partisan to extremely partisan), we also provided the option to choose "impossible to decide" because the articles could be about non-political topics. When computing inter-rater agreement, this option was ignored. The remaining 5 ratings were treated as ordinal ratings. The initial Krippendorff's alpha was 0.142, using the interval metric. To perform quality control, we devised some filtering steps based on the information we had. These steps are as follows:
Remove uninterested annotators: we assumed that annotators that provided no information were not interested in participating in the task. These annotators always rated "not possible to decide" for Q1, 'not applicable' or "unknown" for Q2, and provide no textual comment for Q3. There were in total 117 uninterested annotators and their answers were discarded.
Remove unreliable annotators: as we didn't have "gold data" to evaluate reliability, we used the free text that an annotator provided in Q3 to compute a reliability score. The assumption was that if an annotator was able to provide texts with meaningful partisanship description, he or she was more reliable in performing the task. To do this, we collected the text given by each annotator. We filtered out text that didn't answer the question, such as symbols, 'no idea', 'see above', etc. Then we calculated the reliability score of annotator INLINEFORM0 with equation EQREF21 , where INLINEFORM1 is the number of clean texts that annotator INLINEFORM2 provided in total and INLINEFORM3 is the number of articles that annotator INLINEFORM4 rated. DISPLAYFORM0
We added one to INLINEFORM0 so that annotators that gave no clean texts would not all end up with a zero score but would have different scores based on how many articles they rated. In other words, if an annotator only rated one article and didn't give textual information, we considered he or she reliable since we had little information. However, an annotator that rated ten articles but never gave useful textual information was more likely to be unreliable. The reliability score was used to filter out annotators that rarely gave meaningful text. The threshold of the filtering was decided by the Krippendorff's alpha that would be achieved after discarding the annotators with a score below the threshold.
Remove articles with too few annotations: articles with less than 3 annotations were discarded because we were not confident with a label that was derived from less than 3 annotations.
Remove unreliable articles: if at least half of the annotations of an article were "impossible to decide", we assumed that the article was not about issues of which partisanship could be decided.
Finally, we mapped ratings of 1 and 2 to non-partisan, and 3 to 5 to partisan. A majority vote was used to derive the final label. Articles with no majority were discarded. In the end, 766 articles remained, of which 201 were partisan. Table TABREF24 shows the number of articles and the percentage of partisan articles per publisher. The final alpha value is 0.180.
Analysis of the datasets
In this section, we analyze the properties and relationship of the two parts (publisher-level and article-level) of the datasets. In Table TABREF25 , we listed the length of articles of the two parts. The reason that this is important is to check whether there are apparent differences between the articles in the two parts of the dataset. We see that the lengths are comparable, which is desired.
The second analysis is the relationship between publisher and article partisanship. We want to check whether the assumption of partisan publishers publish more partisan articles is valid for our dataset. To do this, we used the article-level labels and calculated the percentage of partisan articles for each publisher. This value was then compared with the publisher partisanship. We calculated Spearsman's correlation between the publisher partisanship derived from the audience and article content. We take the absolute value of the partisanship in table TABREF9 and that in table TABREF24 . The correlation is 0.21. This low correlation resulted from the nature of the task and publishers that were considered. The partisan publishers in DPG Media publish news articles that are reviewed by professional editors. The publishers are often partisan only on a portion of the articles and on certain topics.
Limitations
We identified some limitations during the process, which we describe in this section.
When deciding publisher partisanship, the number of people from whom we computed the score was small. For example, de Stentor is estimated to reach 275K readers each day on its official website. Deciding the audience leaning from 55 samples was subject to sampling bias. Besides, the scores differ very little between publishers. None of the publishers had an absolute score higher than 1, meaning that even the most partisan publisher was only slightly partisan. Deciding which publishers we consider as partisan and which not is thus not very reliable.
The article-level annotation task was not as well-defined as on a crowdsourcing platform. We included the questions as part of an existing survey and didn't want to create much burden to the annotators. Therefore, we did not provide long descriptive text that explained how a person should annotate an article. We thus run under the risk of annotator bias. This is one of the reasons for a low inter-rater agreement.
Dataset Application
This dataset is aimed to contribute to developing a partisan news detector. There are several ways that the dataset can be used to devise the system. For example, it is possible to train the detector using publisher-level labels and test with article-level labels. It is also possible to use semi-supervised learning and treat the publisher-level part as unsupervised, or use only the article-level part. We also released the raw survey data so that new mechanisms to decide the article-level labels can be devised.
Acknowledgements
We would like to thank Johannes Kiesel and the colleagues from Factmata for providing us with the annotation questions they used when creating a hyperpartisan news dataset. We would also like to thank Jaron Harambam, Judith Möller for helping us in asking the right questions for our annotations and Nava Tintarev for sharing her insights in the domain.
We list the questions we asked in the partisanship annotation survey, in the original Dutch language and an English translation.
Translated

Question: What limitations are mentioned?
Answer: deciding publisher partisanship, risk annotator bias because of short description text provided to annotators

Question: What examples of applications are mentioned?
Answer: partisan news detector

Question: Did they crowdsource the annotations?
Answer: Yes