Crossing the Aisle: Unveiling Partisan and Counter-Partisan Events in News Reporting
Authors: Kaijian Zou
Xinliang Frederick Zhang
Winston Wu
Nicholas Beauchamp
Lu Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: partisan event
media bias

Abstract: News media is expected to uphold unbiased reporting. Yet they may still affect public opinion by selectively including or omitting events that support or contradict their ideological positions. 

Prior work in NLP has only studied media bias via linguistic style and word usage. 

In this paper, we study to which degree media balances news reporting and affects consumers through event inclusion or omission. 
We first introduce the task of detecting both partisan and counter-partisan events: events that support or oppose the author's political ideology. 

To conduct our study, we annotate a high-quality dataset, PAC, containing $8,511$ (counter-)partisan event annotations in $304$ news articles from ideologically diverse media outlets. 

We benchmark PAC to highlight the challenges of this task. 

Our findings highlight both the ways in which the news subtly shapes opinion and the need for large language models that better understand events within a broader context. Our dataset can be found at https://github.com/launchnlp/Partisan-Event-Dataset.

Crossing the Aisle: Unveiling Partisan and Counter-Partisan Events in News Reporting
Kaijian Zou't
Xinliang Frederick Zhang'
Winston Wu²
Nick Beauchamp" Lu Wang'
'Computer Science and Engineering, University of Michigan, Ann Arbor, MI DDepartment of Computer Science, University of Hawaii Hilo, Hilo, HI "Department of Political Science, Northeastern University, Boston, MA 'zzkjzou, xlfzhang, wangluxy)@umichh edu 2 wwswuhawaiii. eddu, 2n. beauchamm@nrrtheasteeeee edu
Abstract
Story Title: Texas Governor Signs 'Heartheat Bill' Ban- ning Abortion
News media is expected to uphold unbiased re- porting. Yet they may still affect public opinion by selectively including or omitting events that support or contradict their ideological positions. Prior work in NLP has only studied media bias via linguistic style and word usage. In this pa- per, we study which degree media balances news reporting and affects consumers through event inclusion or omission. We first introduce the task of detecting both partisan and counter- partisan events: events that support oppose the author's 's political ideology. To conduct our study, annotate high-quality dataset, PAC, containing 8, 511 (counter-)partisan event an- notations in 304 news articles from ideolog- ically diverse media outlets. We benchmark PAC to highlight the challenges of this task. Our findings highlight both the ways in which the news subtly shapes opinion and the need for large language models that better under- stand events within broader context. Our dataset can be found at htttps://gi //gtthb.....././////gg thub. com/ launchnlp/Pan San-Event Saaeeeen---tttereeeeeeee
National Review (Right):
Texas Governor Greg Abbott signed a bill Wednesday barring abortions "Our creator endowed with the right to life and yet millions of children lose their right to life every year because of abortion." Abbott, Republican. said during a bill signing ceremony.
Event #1 sign pos-steartheat Bill
Event #2 lose ): pos pearHbeat Bill neg aab-oaiotioon
Guardian (Left):
The Texas Republican governor Greg Abbott has signed into law one of the most extreme six seek abortion bans in the US, despite strong opposition from the medical and legal communities "TTis bill ensures that every unbom child who has heartheat will be saved from the ravages of abortion," said Abbott
Event #3 signed ): neg->lleartbeut Bill
Event #4 opposition ): neg->llearbbaa Bill
Event #2 saved ): pos->Heartbea Bill neg->abortion
Figure 1: Excerpt from news story reporting on Heart- beat Bill. Blue indicate events favoring left-leaning entities and disfavoring right-leaning entities; vice versa for Red Although both media outlets report the event Greg Abbott signed Heartbeat Bill, The Guardian select the additional event opposition to attack Heartbeat Bill. Interestingly, both media outlets include quotes from Greg Abbott but for different purposes: for support- ing the bill and the other for balanced reporting.
1 Introduction
Political opinion and behavior are significantly af- fected by the news that individuals consume. There is now extensive literature examining how journal- ists and media outlets promote their ideologies via moral or political language, tone, or issue framing (de Vreese, 2004; Vella and Gentzkow, 2009; Shen et al., 2014; Perse and Lambe, 2016). How- ever, in addition to this more overt and superficial presentation bias, even neutrally written, broadly- framed news reporting, which appears both "ob- jective" and relatively moderate, may shape pub- lic opinion through a more invisible process of selection bias, where factual elements that are in- cluded or omitted themselves have ideological ef- fects (Gentzkow and Shapiro, 2006; D' Alessio and
Allen, 2006; Groeling, 2013).
Existing work in NLP has only studied bias at the token- or sentence-level, particularly examin- ing how language is phrased (Greene and Resnik, 2009; Yano et al., 2010; Recasens et al., 2013; Lim et al., 2020; Spinde et al., 2021). This type of bias does not rely on the context outside of any individ- ual sentence, and can be altered simply by using different words and sentence structures. Only a few studies have focused on bias that depends on broader contexts within a news article (Fan et al.,

---

2019; van den Berg and Markert, 2020) or across articles on the same newsworthy event (Liu al.. 2022; Qiu et al., 2022). However, these studies are limited to token- or span-level bias, which is less structured, and fail to consider the more complex interactions among news entities.
et al., 2013) and sentence-level (Yano et al., 2010; Lim al., 2020; Spinde et al., 2021). Similar to our work is informational bias (Fan et al., 2019), which is defined as "tangential, speculative, or background information that tries to sway readers' opinions towards entities in the news." However, they focus on span-level bias, which does not nec- essarily contain any salient events. In contrast, our work considers bias the event level, which is neither "tangential" news, nor the token level. Importantly, we examine both partisan and counter- partisan events in order to study how these core, higher-level units produce ideological effects while maintaining an appearance of objectivity.
To understand more complex content selection and organization within news articles, we scrutinize how media outlets include and organize the funda- mental unit news-events to subtly reflect their ideology while maintaining seemingly balanced reporting. Events are the foundational unit in the storytelling process (Schank and Abelson, 1977), and the way they are selected and arranged affects how the audience perceives the news story (Shen al., 2014; Entman, 2007). Inspired by previous work on selection bias and presentation bias (Groel- ing, 2013; D' Alessio and Allen, 2006), we study two types of events.
Our work is also in line with broad range of research on framing (Entmann 1993; Card et al., 2015), in which news media select and emphasize some aspects of a subject to promote a particular interpretation of the subject. Partisan events should be considered as one type of framing that focuses on fine-graines content selection phenomenon, as writers include and present specific "facts" to sup- port their preferred idcology. Moreover, our work relates research on the selection or omission of news items that explicitly favor one party over the other (Entman, 2007; Gentzkow and Shapiro, 2006; Prat and Strömberg, 2013), or selection for items that create more memorable stories (Mullainathan and Shleifer, 2005; van Dalen, 2012). In contrast, we focus on core news events, those that may not explicitly favor side, but which nevertheless ideological in their effect.
(i) Partisan events, which we define as events that are purposefully included to advance the media outlet's 's ideological allies' interests or suppress the beliefs of its ideological enemies. (ii) Counter- paatter-arttiann events, which we define events purposefully included to mitigate the intended bias or create a story acceptable the media industry's 's market. Figure shows examples of partisan events and counter-pante cunterrpartisan events.
To support our study, we first collect and label PAC, a dataset of 8,511 PArtisan and Counter- partisan events in 304 news articles. Focusing on the partisan nature of media, PAC is built from 152 sets of news stories, each containing two articles with distinct ideologies. Analysis on PAC reveals that partisan entities tend to receive more positive sentiments and vice versa for count-partisan enti- ties. We further propose and test three hypothe- ses to explain the inclusion of counter-partisan events, considering factors of newsworthiness, mar- ket breadth, and emotional engagement.
Finally, our research is most similar to another recent study on partisan event detection (Liu et al., 2023), but they only investigate partisan events and focus on developing computational tools to detect such events. In contrast, our work also incorporates counter-partisan events, enabling broader and deeper understanding of how media tries to balance impartial news coverage and promoting their own stances. We also construct a significantly larger dataset than the evaluation set curated in Liu et al. (2023), enhancing its utility for model training.
We then investigate the challenges of partisan event detection by experimenting on PAC. Results show that even using carefully constructed prompts with demonstrations, ChatGPT performs only bet- ter than random bascline, demonstrating the diffi- culty of the task and suggesting future directions on enabling models better understand the broader context of the news stories.
3 Partisan Event Annotation
PAC contains articles from two sources. We first sample 57 sets of news stories published between 2012-2022 from SEESAW (Zhang et al., 2022). Each news story set contains three articles on the same story from outlets with different idcologies. Here we take out the articles labeled with a Cen-
2 Related Work
Prior work has studied media bias primarily at the word-level (Greene and Resnik, 2009; Recasens

---

Tran
0.25
voris
86 orEvams
0.20
0.35
1,377
of 201.2 2021
1.357
2,079
0.10
2002
2002.00
2022.00 2003
2013 2023
Table 1: Statistics of PAC.
0.05
ter ideology and only keep stories with two news articles from opposite ideologies. To increase the diversity of topics our dataset, we further collect 95 sets of news stories from www. www.allsides com, covering topics such as abortion, gun control, cli- mate change, etc. We manually inspect each story and keep the ones where the two articles are la- beled with left and right ideologies. Next, fol- low the definition of events from TimeML. (Puste- jovsky al., 2003), i.c., a cover term for situations that happen or occur, and train RoBERTa-Large model on MATRES (Ning et al., 2018) for event detection. Our event detector achieves an Fl score of 89.31, which is run on PAC to extract events.
0.00
Right
Article Ideology
Partisan
Counter-Purtisan
Figure 2: Average percentage of partisan and counter- partisan events reported across articles for media with different ideologies. More moderate news outlets tend to include a more equal mix events.
Sentiment Distribution on ideological entities
1.4
NEG Out-group Entity
POS In-group Entity
NEG In-group Entity
Sentiment Distribution
POS Out-group Entity
1.0
0.6
Next, the partisan events are annotated based on the following process. For each pair of articles in story, annotator is asked to first read both articles get balanced view of the story. Then, at the article level, the annotator determines the relative ideological ordering, i.e., which article falls more on the left (and the other article more right) on the political spectrum. Then, the annotator estimates each article's absolute ideology on 5-point scale, with being far left and 5 as far right.
0.0
Loft
Right
Article Ideplogy
Distribution of positive and negative portray of left and right entities. News events tend to positive toward in-group entities and negative toward guougroups.
Figure 3:
many more partisan events than counter-partisan events, whereas more moderate news outlets tend to include more equal mix.
For each event in an article, annotators first iden- tify its participating entities, i.e., who enables the action and who affected by the events, and assign them an entity ideology when appropriate, and es- timate the sentiments they receive any. Using the story context, the article's ideology, and the information of the participating entities, annotators label each event as partisan, counter-partisant neutral relative to the article's ideology, based the definitions given in the introduction. If an event is labeled as non-neutral, annotators further mark its intensity indicate how strongly the event supports the article's ideology. A complete annota- tion guideline can be found Appendix A. The an- notation quality control process and inter-annotator agrecment are described in Appendix B. We also discuss disagreement resolution in Appendix C. The final dataset statistics are listed in Table 1.
Partisan sentiment. News media also reveal their idcology in the partisan entities they discuss, via the sentiments associated with those entities, where partisan entities tend to have positive as- sociations and vice versa for count-partisan enti- ties (Groeling, 2013; Zhang et al., 2022). In Fig- ure 3, we find support for this expectation. We also find that left entities generally receive more exposure articles from both sides.
Partisan event placement. Figure 4 shows that for both left and right media outlets, partisan events appear bit earlier in news articles. For counter- partisan events, left-leaning articles also place more counter-partisse events at the beginning, while right-leaning articles place more counter-partisan events towards the end. This asymmetry suggests that right-leaning outlets are more sensitive to driv- ing away readers with counter partisan events, thus placing them at the end of articles to avoid that.
4 Descriptive Analysis
Partisan event selection effect. As shown Fig- ure more ideologically extreme outlets include

---

Partisan Counter-Partisan Events Position
Media Outlet Traffic (log) vs. Partisan Ratio
50%
Fartban Exenta
Narcaan bennts
#Fartisan @Counter-Partts
25
20
20%
10%
1036
-0.35
096
08
04
01
122
Left
Right
65% CI
Figure 4: Distribution of partisan and counter-partisan events in each quartile per news article. Shaded area shows 95% confidence level for both left and right.
5
2 3
Website Traffic (In millions) log scale
Probability
Figure 6: The average ratio of partisan counter- partisan events by media outlets versus logged website traffic. The size of dots represents media's 's article num- ber in PAC. Larger media tends to be more balanced.
Eyert hubaiity
0.02
0.15
Evert
0.02
Partlsan
0.33
Neutral
0.00
Partican
Partican
Neuttall
ideelcdyy
Gounter i-Partisan
Meutrall
Countte-Farttaan
Figure 5: The probability of each type of event being common left and right articles. If an event is counter- partisan, it will more likely be common event.
-04
5 Explaining Partisan and Counter-Partisan Event Usage
-0.3 Ld0
Hiolooy
Figure 7: The average sentiment intensity of each type of event measured by VADER. Blue indicates posi- tive sentiments, and red indicates negative sentiments. Both partisan and counter-partin are associated with stronger sentiments.
In this section, we investigate a number of hypothe- ses about why media outlets include both partisan and counter-partisan events. It is intuitive to un- derstand why partisan events are incorporated into the news storytelling processing, yet unclear why cunter--arrisan events that portray members of one's own group negatively or members an- other group favorably are reported. Specifically, we establish and test three hypotheses for why an outlet would include counter- paatterraattiann news, simi- lar to some of the theories articulated in Groeling (2013): (1) newsworthiness, (2) market breadth, and (3) emotional engagement.
5.2 Hypothesis 2: Market Breadth
Our second hypothesis is that media may seek to preserve a reputation of moderation, potentially in order not to drive away a large segment of its potential audience (Hamilton, 2006). One implica- tion of this hypothesis is that larger media either grew through putting this into practice, or seek maintain their size by not losing audience, while smaller media can focus on more narrowly partisan audiences. To test this implication, we collected the monthly website traffic' of cach media outlet with more than one news article in our dataset and computed the average ratio of partisan to counter- partisan events, calculated per article and then av- eraged over each outlet. In Figure 6, we plot the average partisan ratio against the logged monthly website traffic. The correlation coefficient of -0.35 supports the hypothesis that larger outlets produce a more bipartisan account of news stories.
5.1 Hypothesis 1: Newsworthiness
This hypothesis suggests that primary goal of mainstream media is to report newsworthy content, even if it is counter-partisan. In Figure 5, we find that counter-partisan events are more likely to be reported by both sides (which is not tautological because the ideology of events is not simply in- ferred from article ideology). However, we find striking asymmetry, where the left appears to re- port mainly counter-partisan events that were also reported on by the right, but the counter-partisan events reported by the right are not as common on the left. This suggests that the left may be moti- vated by newsworthiness more.
5.3 Hypothesis 3: Emotional Engagement
Our third hypothesis is that outlets will include counter-partisan content if its benefits in terms
We collected data from www. s simi larweb.com

---

of emotional audience engagement outweigh its idcological costs (Gentzkow and Shapiro, 2006). This implies that the emotional intensity of counter- partisan events should be higher than that of parti- san events (since higher intensity is required to off- set ideological costs). We employ VADER (Hutto and Gilbert, 2014), a lexicon and rule-based sen- timent analysis tool each event to compute its sentiment intensity. Figure 7 shows that both par- tisan and counter-partisan events have stronger sentiments than non non-partisan events, but we find no strong support for our hypothesis that counter- partisan events will be strongest. If anything, right- leaning events are more intense when reported on by either left or right media, but this difference is not statistically significant.
Partisan
Counter- Partisan
Neutral
Combined
Random RoBERTa
24.74
7.83
67.50
33.36
18,40
15.78
80.61
48,26
+joint
47.48
17.05
80.52
48.35
POLITICS +joint
48.72
19.88
80.09
49.56
47.09
19.55
81.28
49.31
ChatGPT 31.42
9.99
71.75
37.62
Table 2: Model performance on partisan event detection task measured by macro with the best results in bold. Average of 5 random seeds.
training yields further improvement for POLITICS on ideology prediction and a slight improvement for RoBERTa event detection. Moreover, it is worth noting that partisan events are more easily de- tected than counter- paatter- ones, which are usually implicitly signaled in the text and thus require more complex reasoning to uncover. Finally, though ChatGPT model has obtained impressive perfor- mance on many natural language understanding tasks, its performance is only better than random baseline. This suggests that large language models still fall short of reasoning over political relations and idcology analysis that require the understand- ing of implicit sentiments and broader context.
6 Experiments
We experiment on PAC for two tasks. Partisan Event Detection:
Given all events a news article, classify an event as partisan, counter-partisan, or neutral. Ideology Prediction: Predict the political leaning of a news article into left or right.
6.1 Models
We experiment with the following models for the tasks. We first compare with a random base- line, which assigns an article's ideology and an event's partisan class based on their distribution in the training set. Next, we compare to RoBERTa- base (Liu et al., 2019) and POLITICS (Liu et al., 2022), a RoBERTa-base model adapted to polit- ical text, continually trained on 3 million news articles with a triplet loss objective. We further design joint models that are trained to predict both partisan events and article ideology. Finally, see- ing an emerging research area of using large lan- guage models (LLMs), we further prompt Chat- GPT to detect events with a five-sentence context size. Appendix F contains an analysis of experi- ments with different context sizes and number of shots for prompting ChatGPT.
6.3 Error Analysis
We further conduct an error analysis on event pre- dictions by RoBERTa model. We discover that it fails to predict events with implicit sentiments and cannot distinguish the differences between partisan and counter- partissrpatisnnn events. To solve these two problems, future works may consider broader context from the article, other articles on the same story, and the media itself, and leverage entity coref- erence and entity knowledge in general. More de- tails on error analysis can be found at Appendix E.
Conclusion
We conducted novel study on partisan and counter-partisan event reporting in news articles across ideologically varied media outlets. Our newly annotated dataset, PAC, illustrates clear par- tisan bias in event selection even among ostensibly mainstream news outlets, where counter-partisan event inclusion appears to be due to a combination of newsworthiness, market breadth, and emotional engagement. Experiments partisan event detec- tion with various models demonstrate the task's difficulty and that contextual information impor- tant for models to understand media bias.
6.2 Results
For Partisan Event Detection task, we report macro Fl each category of partisan events and on all categories in Table 2. For Ideology Predic- tion task, we use macro Fl score on both the left and the right ideologies, as reported in Table First, both RoBERTa and POLITICS improve per- formance over the random baseline, where joint

---

Acknowledgments
This work is supported in part by the National Sci- ence Foundation under grant III-2127747 and by the Air Force Office of Scientific Research through grant FA9550-22-1- --5009 We would like to thank the anonymous reviewers for their helpful com- ments and feedback.
8 Limitations
Our study only focuses on American politics and the unidimensional left-right ideological spectrum, but other ideological differences may operate out- side of this linear spectrum. Although our dataset already contains diverse set of topics, other top- ies may become important in the future, and will need to update our dataset. The conclusion we draw from the dataset may not be generalizable to other news media outlets. the future work, we plan to apply our annotated dataset to infer events in larger corpus of articles for better generaliz- ability. The event detection model does not have perfect performance and may falsely classify bi- ased content without any justifications, which can cause harm people trust the model blindly. We encourage people to consider these aspects when using our dataset and models.

---

References
Sora Lim, Adam Jatowt, Michael Färber, and Masatoshi Yoshikawa. 2020. Annotating and analyzing biased sentences in news articles using crowdsourcing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1478-1484, Marseille, France. European Language Resources Association.
Dallas Card, Amber E. Boydstum, Justin H. Gross, Philip Resnik, and Noah A. Smith. 2015. The media frames corpus: Annotations of frames actoss issues. In Pro- ceedings of the 53rd Annual Meeting of the Asso- ciation for Computational Linguistics and the 7th International Joini Conference Natural Language Processing (Volume 2: Short Papers), pages 438- 444, Beijing, China. Association for Computational Linguistics.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1
Dave D' Alessio and Mike Allen. 2006. Media Bias in Presidential Elections: Meta-Analyen Journal of Communication, 50(4): 133-156.
Yujian Liu, Xinliang Frederick Zhang, David Wegsman, Nicholas Beauchamp, and Lu Wang. 2022. POLI- TICS: Pretraining with stor-story article comparison for idcology prediction and stance detection. In Find- ings of the Association for Computational Linguis- tics: NAACL 2022, pages 1354-1374, Seatlle, United States. Association for Computational Linguistics.
Claes de Vreese, 2004, The effects of strategic news on political cynicism, issue evaluations, and policy sup- port: A two-wave experiment. Mass Communication and Society, 7(2):191-214,
Stefano VellaVigna and Matthew Gentzkow. 2009. Per- suasion: Empirical evidence. Working Paper 15298, National Bureau of Economic Research.
Yujian Liu, Xinliang Frederick Zhang, Kaijian Zou, Rui- hong Huang, Nick Beauchamp. and Lu Wang. 2023. All things considered: Detecting partisan events from news media with article comparison. In Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, Association for Computational Linguistics.
Robert M. Entman. 1993. Framing:
Toward clarification of fractured paradigm. Journal of Communication, 43(4):51-58.
Robert M. Entman. 2007. Framing bias: Media the distribution of power. Journal of Communication, 57(1):163-17 173.
Sendhil Mullainathan and Andrei Shleifer. 2005. The market for news. American Economic Review, 95(4)::031-1053
Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Pra- fulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019. In plain sight: Media bias through the lens of factual reporting. In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6343-6349, Hong Kong, China, Association for Computational I Linguistics.
Qiang Ning, Hao Wu, and Dan Roth. 2018. A multi- axis annotation scheme for event temporal relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1318-1328, Melbourne, Aus- tralia. Association for Computational Linguistics.
Matthew Gentzkow and Jesse M. Shapiro. 2006. Media bias and reputation. The Journal of political econ- omy, 114(2):280-3)
Elizabeth M. Perse and Jennifer Lambe. 2016. Media Effects and Society. Routledge.
Andrea Prat and David Strömberg. 2013. The Political Economy of Mass Media, volume 2 of Economet- ric Society Monographs, page 135-187. Cambridge University Press.
Stephan Greene and Philip Resnik. 2009. More than words: Syntactic packaging and implicit sentiment. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North Ameri- can Chapter of the Association for Computational Linguistics, pages 503-511, Boulder, Colorado. As- sociation for Computational Linguistics.
James Pustejovsky, José M. Castaño, Robert Ingria, Roser Saurf, Robert J. Gaizauskas, Andrea Sct- zer, Graham Katz, and Dragomir R. Radev. 2003. Timeml: Robust specification of event and temporal expressions in text. In New Directions in Question Answering.
Tim Groeling. 2013. Media bias by the numbers: Chal- lenges and opportunities in the empirical study of partisan news. Annual Review of Political Science, 12:1151.
Changyuan Qiu, Winston Wu, Xinliang Frederick Zhang, and Lu Wang. 2022. Late fusion with triplet margin objective for multimodal idcology prediction and analysis. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pages 9720-9736. Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
James Hamilton. 2006. All The News That's 's Fit to Sell: How the Market Transforms Information Into News.
C. Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. Proceedings of the International AAAI Conference on Web and Social Media, 8(1:211--255

---

Marta Recasens, Cristian Danescu-Niculesee Dan Jurafsky. 2013. Linguistic models for analyz- ing and detecting biased language. In Proceedings of the 51st Anmual Meeting the Association for Computational Linguistics (Volume I: Long Papers), pages 1650-1659, Sofia, Bulgaria, Association for Computational Linguistics.
and
Roger Schank and Robert Abelson. 1977. Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Psychology Press.
Fuyuan Shen, Lee Ahern, and Michelle Baker. 2014. Stories that count: Influence of news narratives on issue attitudes. Journalism & Mass Communication Quarterly, 91(1)998-117.
Timo Spinde, Manuel Plank, Jan-David Krieger, Tenry Ruas, Bela Gipp, and Akiko Aizawa. 2021. Neu- ral media bias detection using distant supervision with BABE bias annotations by experts. In Find- ings of the Association for Computational Linguis- rics: EMNLP 2021, pages 117, Punta Cana, Dominican Republic. Association for Computational Linguisties.
Arjen van Dalen. 2012. Structural bias in cross-national perspective: How political systems and journalism cultures influence government dominance in the news. The International Journal of Press/Politics, 17(1):32- 55.
Esther van den Berg and Katja Markert. 2020. Context in informational bias detection. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6315-6326, Barcelona, Spain (On- line). International Committee Computational Lin- guistics.
Tae Yano, Philip Resnik, and Noah A. Smith. 2010. Shedding (a thousand points of) light on biased lan- guage. In Proceedingy Workshop on Creating Speech and Language Data with Amazon's ' Mechanicul Turk, pages 152-158, Los Angeles. Association for Computational Linguistics.
the NAACI. HIT 2010
Xinliang Frederick Zhang, Nick Beauchamp. and Lu Wang. 2022. Generative entity-to- to-entity stance de- tection with knowledge graph augmentation. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 9950- 9969, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

---

A Annotation Guidelines
We hired six college students who major in polit- ical science, communication and media, and related fields to annotate our dataset. Three are native En- glish speakers from the US, and the other three are international students with high English pro- ficiency who have lived in the US for more than five years. All annotators were highly familiar with American politics. To further ensure the quality of the annotation, before the process began, we hosted two-week training session, which required each annotator to complete pilot annotations for eight news articles and revise them based on feed- back. After the training session, we held individual weekly meetings with each annotator to provide further personalized feedback and revise annota- tion guidelines if there was ambiguity. Each article is annotated by two students.
Below we include the full instructions for the an- notators. A Javascript annotation interface is used to aid annotators during the process.
Given pair of news articles, you need to first read two news articles and label their relative ide- ologies and absolute ideologies. Then, for each event, you need to follow these steps to label its partisanship (partisan, counter-partisan, or neural):
Identify the agent entity and the patient en- tity for each event and other salient entities, These entities can be a political group, politi- cians, bills, legislation, political movements, or anything related to the topic of the article.
Label each entity as left, neutral, or right based on the article context or additional in- formation online.
We calculate inter-annotator agreement (IAA) levels on the articles' relative ideologies, their ab- solute idcology, and the events. The IAA on the articles' relative ideologies between two annotators was 90%, while the agreement on the articles' ab- solute idcologies was 84%. The higher agreement on the articles' relative ideologies demonstrates the usefulness of treating a story as one unit for annota- tion. For stories with conflicting relative ideologies or articles with a difference greater than in their absolute ideologies, a third annotator resolves all conflicts and corrects any mistakes. Despite the subjective nature of this task and the large number of events in each article, the Cohen's i Kappa event labels is 0.32, which indicates fair agree- ment is achieved. When calculating agreement on whether a sentence contains partisan or counter- partisan event when one exists, the score increases to 0.43, which moderate agreement.
Estimate sentiments the author tries to convey toward each entity by reporting the events.
Based on each entity, its ideology, and senti- ments, you can decide whether an event sup- ports or opposes the article's idcology. If it supports, label it as partisan. Otherwise, label it as counter-partisan. For example, in right article, if left entity is attacked or right entity is praised by the author, you should label the event as partisan event. If a left entity praised or a right entity is attacked by the author, you should label the event as counter-partisan.
B Annotation Quality
We collect stories from Allsides, a website present- ing news stories from different media outlets. Its editorial team inspects news articles from differ- ent sources and pairs them together a triplet. One of the authors, with sufficient background in American polities, manually inspected each story by following the steps below
Our dataset covers diverse topics, including but not limited to immigration, abortion, guns, elec- tions, healthcare, racism, energy, climate change, tax, federal budget, and LGBT.
C Annotation Disagreement
In total, the dataset contains 304 news articles cov- ering 152 news stories. All news stories are an- notated by at least two annotators: 5 stories are annotated by one annotator and revised by another to add any missing labels and correct mistakes, while 147 stories are annotated by two annotators. Out of news stories annotated by two people, a third annotator manually merges 54 news articles to correct errors and resolve any conflicts. For the
Read the summary from the Allsides, which includes the story context and comments from the editors on how each news article covers the story differently.
Read each article carefully and compare them.
Pick the two news articles with significant differences in their ideologies.

---

P
R
F1
and cannot distinguish the differences between par- tisan and counicr-partisan events.
Random
51.71
51.72
51.54
RoBERTa
60.64
58.86
57.43
+joint
51.58
54.29
53.36
E.1 Events with Implicit Sentiments
POLITICS
76.37
74.00
73.17
The first example in Figure is from a news arti- cle about the climate emergency declared by Joe Biden after Congress failed the negotiation. The model fails to predict "give" a partisan event. This is primarily because the term itself does not exhibit explicit sentiment and the model does not link "him" Joe Biden. However, when contex- tualized within the broader scope of the article, it becomes evident that the author includes this event to bolster the argument for a climate emergency by highlighting its positive impact. To predict this type of events correctly, the model needs to under- stand the context surrounding the event and how each entity is portrayed and linked.
+joint
75.72
74.86
74.65
Table 3: Model performance on ideology prediction with the best results in bold. Average of 5 random seeds.
rest of the news stories, we combine annotations from two annotators and have third annotator re- solving only conflicting labels. During the merging process, we also discover three types of common annotation disagreements:
Events with very weak intensity: some events are only annotated by one annotator, typically, these events have low intensity in their par- tisanship or are not relevant enough, so the other annotator skips them.
E.2 Counter-partisse Events
The second example in Figure is from right news article about the lawsuit by Martha's 's Vineyard migrants against Ron DeSantis. The model incor- rectly categorizes the event "horrified" as partisan due to the strong sentiment conveyed in the text. However, when placed in the broader context of the article, which defends Ron DeSantis and criticizes Democrats for politicizing migrants, this event should be more accurately classified as a counter- partisan event. The author includes specifically to showcase the response from Democrats. The model seems to have limited capability of differentiating between partisan and counter counter-partisan events, pos- sibly because of the similar language used to ex- press partisan and counter- parttrnn events and the difficulty of recognizing the overall slant of news articles.
Label different events within the same sen- tence: this happened the most frequently be- cause when news articles report an event, they describe it with cluster of smaller and re- lated events. Two annotators may perceive differently which event(s) is partisan.
Events are perceived differently by two an- notators, one may think it is partisan, and the other may think it is counter counter-partisan. Usually, both interpretations are valid, and we have a third annotator to decide which interpretation should be kept.
Ideology Prediction
Table shows the idcology prediction performance of different models.
ChatGPT Prompts
E Error Analysis
We use five different context sizes for our ChatGPT prompt: story with two articles, single article, 10 sentences, 5 sentences, and 3 sentences. An example prompt with sentences as context can be viewed in Table
We perform a detailed examination of 100 event predictions generated by our RoBERTa model. We discover sentiments' intensity closely corre- lates with the model's performance. Specifically, when the model classifies events as either partisan or counter-partisan 70% of these events feature strong/explicit event triggers like "opposing" or "deceived". The remaining events use more neutral triggers such "said" or "passed". Our model demonstrates higher accuracy in predicting events that contain strong or explicit sentiments. However, it fails to predict events with implicit sentiments
Context Size vs. Number of Shots. Since the context window size of ChatGPT is fixed, ex- plore prompts with different context window sizes and investigate the trade-off between context win- dow size and the number of shots. We try out five window sizes on our development set: 3 sen- tences, sentences, 10 sentences, a single article,

---

Article Title: Biden ready to invoke "domestic mobiliza- tion' against climate crisis after Congress failed
The Washington Times (Left):
Declaring climate emergency would give him the power implement significant changes energy produc- tion and consumption.
Prediction: Neutral
Gold Label: Partisan
Article Title: DeSantis Admin Slams Lawsuit Filed by Martha's Vineyard Migrants
New York Post (Right):
... Newsom wrote on Thursday: Like millions of Amer- icans, have been horrified at the images migrants being shipped on buses and planes across the country to be used as political props. ....
Prediction: Partisan
Gold Label: Counter-Partisan
Figure 8: Two errors made by the RoBERTa model.
Partisan
Counter- Partisan
Neutral
Combined
Story
33.29
12.16
52.04
32.50
+ shot
31.51
8.42
54.89
32.61
Article
33.53
11.96
53.59
33.03
+ 2 shots
34.66
10.48
57.53
34.29
Sentence
sentences
20.95
11.05
70.54
34.18
+34 shots
28.67
3.57
79.33
37.19
sentences
21.78
12.29
70.38
34.82
+22 shots
31.80
10.47
72.45
38 24
10 sentences
27.99
12.10
66.06
35.38
+11 shots
35.71
10.74
64.17
36.87
Table Fl scores of different ChatGPT prompts on development with the best results in bold. We use the maximum number of shots that can fit into each context window size. For each text imput, we randomly sample demonstrations from the train set until the maximum number shots can be fit. Average of random seeds.
and a story with two articles of different ideolo- gies. As shown in Table 4, as the context window size increases, ChatGPT performs worse neu- tral and counternpartisan events but improves its performance on partisan events. The larger context size gives ChatGPT more information about the article ideology, and its event detection results may be more biased toward the article ideology. We use the sentence prompt with 22 shots as our ChatGPT model for the test set.

---

Prompt
Text
Sentence Prompt
Imagine you are a human annotator a research study about news narratives. You will be asked to annotate partisan events and counter-partisan events in the news articles. "Partisan Events" further the interests, goals, and values of left or right. Authors reveal their ideology when including partisan events that benefit their idcology foil their opposite idcology. Shounter-Partiseee Events' are events that counter authors' ideology support their opposite idcology create a more memorable story or make articles seem unbiased. "Neutral Events" are events that authors include and present fairly and objectively.
Given sentences from a news article, follow these instructions: Read the sentences. 2. For each event <eid> in the article, perform the following tasks: Identify the agent entities, patient entities, and salient entities in the event. b. Identify the ideologies of all entities from step receive d. Identify whether the event is partisan, counter-partisan, or neutral based on its entities, ideologies, and sentiments from a, b,
Identify the sentiment all entities from step
Demostrations>>
<Sentences from news article>
Constraint: You should return your annotations in the following format without entities, without sentiments, without explanation: Partisan Events: eid, cid, eid, eid,
Counter-Partisan Events: eid, eid, eid, eid, ...
Neutral Events: eid, eid, eid, eid,
Table 5: ChatGPT Prompt for sentences.


Reviews:
[{"review_id": "lopURGm6td", "reviewer": "Reviewer_ZkKA", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback. We are glad you found our task “new and interesting”. We would like to highlight that our contributions are threefold.\n\n1. We propose ***a new task of detecting partisan and counter-partisan events*** and provide multiple baselines of different model families, including fine-tuned PLM and recent LLM (ChatGPT), to show the difficulty of the proposed task. As we show in the paper, this task is extremely challenging due to ***the subtle language usage in partisan content and the media's tendency to obfuscate their biases***, especially for how events are selectively reported to promote certain media ideologies.\n\n2. For this task, we collect and label ***the first dataset of 206 news articles with 6,162 partisan and counter-partisan events***, which span ***more than 10 representative and controversial topics*** such as abortion and gun control.\n\n3. Our analyses of these models and data ***shed more light*** on the content selection effect and how partisan and counter-partisan events are used in the news by media representing different ideologies.\n\nTo the best of our knowledge, we are the ***first*** to propose and define the task of detecting partisan and counter-partisan events.  Studying this task gives us a deeper understanding of media bias within news articles and how news articles select and organize content in the production process to sway readers’ opinions tacitly. \n\nWhile the models we evaluate are not new, as you mention, our experiments and analyses of multiple strong baselines on our novel task and dataset reveal ***the limitations of current computational tools for studying the content selection effect at scale*** and point to future work for better understanding this phenomenon. We are making our dataset available to the community as a resource that could help pave the way for new insights into this important issue.\n\nWe hope we have tackled all your concerns about our work and hope you can reconsider your evaluation. If there are any other issues, please let us know by updating your review, and we would be happy to resolve them in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The proposed partisan and counter-partisan event detection task is new and interesting \n(2) A task-corresponding dataset is built for research purpose \n(3) The paper is well written and easy to follow\n", "reasons_to_reject": "(1) The novelty of the paper is limited. \n(2) It is mainly a case-study paper. No new methods and techniques are proposed. The experiments only show the difficulty of the proposed task. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper studied the effects of partisan and counter-partisan events in news reporting across different media outlets. A newly annotated dataset (PARTISAN EVENTS) is provided. Experiments on partisan event detection with a variety of models demonstrate the difficulty of the proposed task.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4WJ8SxwMk5", "reviewer": "Reviewer_wiNC", "summary": "", "questions": "- L134: \"We manually inspect each story [...] differ\": Please provide more details. Who inspected each story, which were the annotation guidelines/the inspectors' background, IAA, etc.\n\n- L136: Which is the definition of \"events from TimeML\"?\n\n- L138: How was the 89.31 F1 score calculated? On which test set? Is there a performance drop expected when applied on the PARTISAN EVENTS dataset?\n\n- L174: \"we held individual weekly meetings [...] if there was ambiguity\": could you provide an example use case in an appendix possibly? Was this part of the training or part of the actual annotation task? If the latter is the case, could this introduce bias in the annotations?\n\n- §3.2: Clarify early on this section that each annotation was performed by two students and not by all of them.\n\n- §3.2: You mention that the IAA on \"*stories'* relative ideology\" is 91%; but on §3.1, you mention that \"At the *article* level, the annotator determines the relative ideological ordering\". Do these two quoted pieces of text refer to the same annotation? If so, please be consistent on your terminology (i.e., \"story\" vs \"article level\" annotation).\n\n- L186: \"a significant difference in their absolute ideologies\": please provide the exact absolute value used.\n\n- §5: Is Task A a two- or a five-class prediction task?\n\n- Results section: it is unclear to my why you have omitted the neutral class. Surely the other two classes are more interesting to your task, but it is very important for the reader to see the overall picture of the results.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and suggestions.\n\n***Re: Lack of error and qualitative analysis***\n\nTo address your concerns regarding error analysis, we performed ***a detailed examination of 100 event predictions*** generated by our top-performing model, the joint RoBERTa model. We discovered ***sentiments' intensity closely correlates with the model's performance***. Specifically, when the model classifies events as either partisan or counter-partisan, 70% of these events feature strong/explicit event triggers like \"opposing\" or \"deceived.\" The remaining events use more neutral triggers such as \"said\" or \"passed.\" Our model demonstrates higher accuracy in predicting events that contain strong or explicit sentiments.\n\n\n1. Regarding errors made by the model, the first category of errors is ***the failure to detect events with implicit sentiments.*** \n> Declaring a climate emergency would ***give*** him the power to implement significant changes to energy production and consumption.\n\n   The above sentence is from a news article about the climate emergency declared by Joe Biden after Congress failed the negotiation. The model fails to predict “give” as a partisan event. This is primarily because the term itself doesn't exhibit explicit sentiment and the model may not link \"him\" to Joe Biden. However, when contextualized within the broader scope of the article, it becomes evident that the author includes this event to bolster the argument for a climate emergency by highlighting its positive impact. To predict this type of events correctly, the model needs to ***understand the context surrounding the event and how each entity is portrayed and linked.*** \n\n2. The second category of errors by the model, which is also the most challenging case, is ***the failure to distinguish between partisan and counter-partisan events.***\n> Newsom wrote on Thursday: Like millions of Americans, I have been ***horrified*** at the images of migrants being shipped on buses and planes across the country to be used as political props.\n\n   The above sentence is from a right news article about the lawsuit by Martha's Vineyard migrants against Ron DeSantis. Our model incorrectly categorizes the event \"horrified\" as partisan due to the strong sentiment conveyed in the text. However, when placed in the broader context of the article, which defends Ron DeSantis and criticizes Democrats for politicizing migrants, this event should be more accurately classified as a counter-partisan event. The author includes it specifically to showcase the response from Democrats. The model seems to have limited capability of differentiating between partisan and counter-partisan events, possibly because of the similar language used to express partisan and counter-partisan events and the difficulty of recognizing the overall slant of news articles. \n\nTo solve these two problems, future works may consider ***a broader context, use cross-document information, and utilize entity coreference and entity knowledge.*** We will include this error analysis in the camera-ready version.\n\n***Re: Questions For The Authors***\n\n***L134***: We collect stories from Allsides, a website presenting news stories from different media outlets. Its editorial team inspects news articles from different sources and pairs them together as a triplet. One of the authors, with sufficient background in American politics, manually inspected each story by \n1. Read the summary from the Allsides, which includes the story context and comments from the editors on how each news article covers the story differently. \n2. Read each article carefully and compare them \n3. Pick the two news articles with significant differences in their ideologies\n\n***L136***: In the TimeML annotation guidelines, events are defined as ***“a cover term for situations that happen or occur.”***\n\n***L138***: We train a Roberta-Large model to detect event triggers with the training set from MATRES. Then, we evaluate the model on the test set from MATRES. The performance drop is ***minimal*** since both MATRES and the PARTISAN EVENTS dataset consist of news articles.\n\n***L174***: One example of ambiguity is when the news article attacks a liberal politician for being too conservative on a specific issue. Although the left entity receive negative sentiment, the overall ideology of the news article is still left. The individual weekly meetings are held after the training. To ***minimize the bias***, we make the meeting mainly as a discussion to help annotators understand the annotation guidelines and clarify any confusion.\n\n***3.2***: Each story is annotated by ***at least two people***. More details can be found in Appendix B. \n\n***3.2***: Yes, they refer to the same annotation, which is the relative ideology. 91% of the news articles have the same relative ideology annotated by two annotators.\n\n***L186***: The value is 2. For example, if a news article is annotated Left by one person and Center by the other, a third person will annotate the news article again to resolve conflicts and correct any mistakes.\n\n***§5***: Task A is a two-class prediction task.\n\n***Results section***: Omitting the neutral class was mainly to save space. We will include the performance of the neutral class in the camera-ready version.\n\n***Presentation improvements***: We will incorporate them in the camera-ready version.\n\nWe hope we have tackled all your concerns about our work and hope you can reconsider your evaluation. If there are any other issues, please let us know by updating your review, and we would be happy to resolve them in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The introduction of a novel dataset.", "reasons_to_reject": "- Lack of error and qualitative analysis of the results, which can set up the future research directions on this task.\n- A few missing details/clarification points are needed, primarily in section 3 (annotation process) which is the most important.", "questions_to_authors": "- L134: \"We manually inspect each story [...] differ\": Please provide more details. Who inspected each story, which were the annotation guidelines/the inspectors' background, IAA, etc.\n\n- L136: Which is the definition of \"events from TimeML\"?\n\n- L138: How was the 89.31 F1 score calculated? On which test set? Is there a performance drop expected when applied on the PARTISAN EVENTS dataset?\n\n- L174: \"we held individual weekly meetings [...] if there was ambiguity\": could you provide an example use case in an appendix possibly? Was this part of the training or part of the actual annotation task? If the latter is the case, could this introduce bias in the annotations?\n\n- §3.2: Clarify early on this section that each annotation was performed by two students and not by all of them.\n\n- §3.2: You mention that the IAA on \"*stories'* relative ideology\" is 91%; but on §3.1, you mention that \"At the *article* level, the annotator determines the relative ideological ordering\". Do these two quoted pieces of text refer to the same annotation? If so, please be consistent on your terminology (i.e., \"story\" vs \"article level\" annotation).\n\n- L186: \"a significant difference in their absolute ideologies\": please provide the exact absolute value used.\n\n- §5: Is Task A a two- or a five-class prediction task?\n\n- Results section: it is unclear to my why you have omitted the neutral class. Surely the other two classes are more interesting to your task, but it is very important for the reader to see the overall picture of the results.", "paper_topic_and_main_contributions": "This work focuses on the task of detecting partisan vs counter-partisan events in news reporting. The authors aggregate 103 news stories, each with 2 news articles from opposing ideologies, end extract events from them using a RoBERTa-based model. They then annotate (a) each article as ideologically leaning towards left/right and on a Likert scale (1-5, left-right) and (b) each identified event in a news article as partisan/counter-partisan/neutral. After a qualitative analysis of their annotations, they test two RoBERTa-based models on two tasks: (A) article-level ideology prediction and (B) partisan event detection -- for (B) they also try ChatGPT. \n\nThe paper is well-written, albeit with a few clarification points needed primarily on the annotation process (§3). The major contribution is the introduction of a novel dataset, which can be useful for future research in this area. My major concern though is the lack of error and qualitative analysis of the results, which is quite important for this type of work: which are the most challenging cases for the best-performing model? Which cases are more easily captured? Is there a correlation between qualitative characteristics (e.g., sentiment) and the accuracy of the models? Incorporating a section on this type of analysis would greatly benefit the paper, as it can guide future work on this task/dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L240: research vs reach?\n- Table 3: add \".\" at the end\n- L253: \"hurts the model performance\" for Task A"}, "extra_text": {}}, {"review_id": "9nIcwKQ0aO", "reviewer": "Reviewer_5vXQ", "summary": "", "questions": "[Q1] What are the high-level discussion topics in the news articles studied in this paper?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and suggestions. We will address your questions in the revision.\n\n***Re: R1***\n\nAs outlined in our paper, our approach extends beyond the ***token-level*** spans of informational bias (Fan et al., 2019) to focus on ***event-level*** granularity. This allows for a more nuanced capture of the complex interactions among entities within news stories. In our revision, we intend to ***add comparisons against subframes*** (Roy and Goldwasser, 2020), which primarily concentrate on ***token-level*** bias owing to their reliance on ***manually curated lexicons*** for each topic. While their study covers only three topics, our dataset covers more than 10 controversial topics. It contains more extensive information (e.g., sentiments towards a participatory entity) that allows us to study how bias is subtly signaled in the news.\n\nAdditionally, we introduce ***the novel concept of counter-partisan events***. This enables us to investigate not only the events that align with the media's stance but also those that run counter to it. By studying partisan and counter-partisan events collectively, we gain a more comprehensive understanding of media bias within news articles. Specifically, we emphasize the subtle ***content selection effect***, often overlooked by readers yet proven effective in persuasion (Groeling, 2013). Given the current computational limitations in studying this effect at scale, we are making our dataset publicly available to foster new insights into this critical area.\n\n***Re: R2***\n\nAs shown in Table 1, our dataset contains ***5,720 sentences, 4,612 annotations of partisan events, and 1,550 annotations of counter-partisan partisan events***. It contains ***more than 10 controversial political topics***. This comprehensive nature—boasting a high sentence count, rich annotations, and topic diversity—enables our dataset to offer a meaningful snapshot of news media trends. In comparison to similar datasets, BABE (Spinde et al., 2021) annotates 3,700 sentences; the sentence dataset (Lim et al., 2020) annotates 46 news articles, which contain 966 sentences; BASIL (Fan et al.,2019) consists of 300 news articles annotated with 1,727 spans. We are confident that our dataset is a more extensive resource for analyzing media bias.\n\n***Re: Q1***\n\nOur dataset covers diverse topics, including ***immigration, abortion, guns, elections, healthcare, racism, national defense, climate change, federal budget, and Congress***. On average, each topic has about 20 news articles. \n\n\nWe hope we have tackled all your concerns about our work and hope you can reconsider your evaluation. If there are any other issues, please let us know by updating your review, and we would be happy to resolve them in the revision.\n\n**References**\n\n[1] Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Prafulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019. In Plain Sight: Media Bias Through the Lens of Factual Reporting. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6343–6349, Hong Kong, China. Association for Computational Linguistics.\n\n[2] Shamik Roy and Dan Goldwasser. 2020. Weakly Supervised Learning of Nuanced Frames for Analyzing Polarization in News Media. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7698–7716, Online. Association for Computational Linguistics.\n\n[3] Groeling, T. (2013). Media Bias by the Numbers: Challenges and Opportunities in the Empirical Study of Partisan News. Annual Review of Political Science, 16(1), 129–151. https://doi.org/10.1146/annurev-polisci-040811-115123\n\n[4] Timo Spinde, Manuel Plank, Jan-David Krieger, Terry Ruas, Bela Gipp, and Akiko Aizawa. 2021. Neural Media Bias Detection Using Distant Supervision With BABE - Bias Annotations By Experts. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1166–1177, Punta Cana, Dominican Republic. Association for Computational Linguistics.\n\n[5] Sora Lim, Adam Jatowt, Michael Färber, and Masatoshi Yoshikawa. 2020. Annotating and Analyzing Biased Sentences in News Articles using Crowdsourcing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1478–1484, Marseille, France. European Language Resources Association.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "[A1] This work proposes a novel dataset for analyzing partisan news media reporting of events.\n\n[A2] The human annotation framework is well-described and it seems easy to replicate.", "reasons_to_reject": "[R1] The concept of the partisan event as it is reported in the paper seems conceptually very similar to previously studied information bias (Fan et al., 2019) (discussed by the authors in the related works section) and subframes (Roy and Goldwasser, 2020) (not discussed by the authors in the paper). Hence, I am not totally convinced about the novelty of the work rather it seems like an incremental extension of the previous studies.\n\n[R2] The number of news articles annotated is very few (only 206). Hence, it is hard to say that any trends found in this small amount of data are generalizable to the whole news media.", "questions_to_authors": "[Q1] What are the high-level discussion topics in the news articles studied in this paper?", "paper_topic_and_main_contributions": "This paper proposes a news article dataset consisting of 206 news articles annotated for partisan and counter-partisan events in the articles. Partisan events are defined as events reporting of which benefit the stance of the news article and counter-partisan events are events that are reported with partisan events to make the news article seem neutral.\n\nThe authors proposed a human annotation framework for annotating such events by focusing on reported entities and sentiment toward them. The authors identified the ideologies of the news articles and the partisanship of the events using a few off-the-shelf classifiers and ChatGPT. The authors also presented some qualitative analysis of the partisan events and reported how different polarities use such events. For example, in the right-biased media, counter-partisan events appear towards the end of the news articles.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a newswire dataset with an annotation framework for labeling partisan and counter-partisan events. 
Overall strengths include the novel dataset and annotation framework that are clearly presented. On the other hand, weaknesses include a lack of clear definition of partisan news, lack of error analysis or any insights on results, and the limited size of the dataset.
Decision: Accept-Findings