All Things Considered: Detecting Partisan Events from News Media with Cross-Article Comparison
Authors: Yujian Liu
Xinliang Frederick Zhang
Kaijian Zou
Ruihong Huang
Nicholas Beauchamp
Lu Wang
Venue: EMNLP 2023 Main (2023)
Keywords: Partisan event detection
media bias

Abstract: Public opinion is shaped by the information news media provide, and that information in turn may be shaped by the ideological preferences of media outlets. But while much attention has been devoted to media bias via overt ideological language or topic selection, a more unobtrusive way in which the media shape opinion is via the strategic inclusion or omission of $\textit{partisan events}$ that may $\textit{support}$ one side or the other. We develop a latent variable-based framework to predict the ideology of news articles by comparing multiple articles on the same story and identifying partisan events whose inclusion or omission reveals ideology. Our experiments first validate the existence of partisan event selection, and then show that article alignment and cross-document comparison detect partisan events and article ideology better than competitive baselines. 
Our results reveal the high-level form of media bias, which is present even among mainstream media with strong norms of objectivity and nonpartisanship. Our codebase and dataset are available at https://github.com/launchnlp/ATC.

All Things Considered: Detecting Partisan Events from News Media with Cross-Article Comparison
Yujian Liu' Xinliang Frederick Zhang² Kaijian Zou2 Lu Wang?
Ruihong Huang" Nick Beauchamp"
'Computer Science, UC Santa Barbara, Santa Barbara, CA
2 Computer Science and Engineering, University of Michigan, Ann Arbor, MI Computer Science and Engineering, Texas A&M University, College Station, TX DDepartment of Political Science, Northeastern University, Boston, MA yuliaan iu@ucsb. edu, "[x1fzhang, zkjzou, wangluxy)@@michh edu hhangrhhtamu edu, *n. beauchamp@hortheasteer edu
Abstract
News Story: Biden pushes for gun legislation after visiting Uvalde.
Public opinion is shaped by the information news media provide, and that information in turn may be shaped by the idcological prefer- ences of media outlets. But while much atten- tion has been devoted to media bias via overt ideological language or topic selection, more unobtrusive way in which the media shape opin- ion is via the strategic inclusion omission of partisan events that may support one side or the other. We develop a latent variable-based framework to predict the ideology of news arti- cles by comparing multiple articles on the same story and identifying partisan events whose in- clusion or omission reveals ideology. Our ex- periments first validate the existence of parti- san event selection, and then show that article alignment and cross-document comparison de- tect partisan events and article ideology better than competitive baselines. Our results reveal the high-level form of media bias, which is present even among mainstream media with strong norms of objectivity and nonpartisan- ship. Our codehase and dataset are available at https:/ thub. com/ /AAC..
The Washington Post (left):
[suid]psa [he asked the president: "Could you please make our schools safer and send more police, plcase?"
El: [Jaydien]aRoo)
E2: [Biden]aua ... [noting]pms "[You couldn't 't buy can- non when the Sccond Amendment was passed] Awnn."
New York Post (right):
BI: [oU couldn't [buy lped [a cannonlaRGI when the Second Amendment was passed. E2: Biden has made that claim before, .... and they have been repeatedly Ideclared IfalselaRai (b facts checkers]aaco.
Figure I: Article snippets by different media the same story. Events are represented by triplets of (ARGO, predicate, ARGI). Events favoring left and right sides highlighted in blue and red. Events in black are reported by both media and not considered as partisan.
in the report, has recently become focus of study in political science. Numerous studies point out that media selectively report information that is flattering to particular political party or ideology, which may consequently shift audience beliefs and attitudes (Broockman and Kalla, 2022; Baum and Groeling, 2008; Grossman et al., 2022; D' Alessio and Allen, 2006). However, most existing work either requires manual inspection of reported con- tent (Broockman and Kalla, 2022), or relies on sim- ple tools for coarse analyses, such as overall slant and topic emphasis (Baum and Groeling, 2008; Grossman et al., 2022). As a result, these studies are either limited to short time period, or are un- able to provide a detailed understanding of content selection bias. Thus there remains strong need for automatic tools that can analyze and detect how more complex content is selectively reported.
1 Introduction
News media play a critical role in society not merely by supplying information, but also by select- ing and shaping the content they report (de Vreese, 2004; Vegna and Kaplan, 2007: Della Vigna and Gentzkow, 2009; Perse and Lambe, 2016). To understand how media bias affects media con- sumers (Gentzkow and Shapiro, 2006; Gentzkow et al., 2015), we must understand not just how me- dia idcology affects the presentation of news sto- ries on surface level, such the usage of partisan phrases or opinions, but also the less obvious pro- cess of content selection (Fan et al., 2019; Enke, 2020). Content selection, such as what events that are related to the main story and should be included
Rather than focusing on more superficial biases such as word, topic, or entity selection, we inves- tigate here how media idcology affects their selee- tion of which events to include for news reporting.

---

Events are the fundamental high-level components of the storytelling process (Prince, 2012), and their inclusion or omíssion shapes how a news story is perceived. In line with previous analysis of par- tisan selection bias in the literature (Broockman and Kalla, 2022), we define partisan events as selectively reported events that are favorable to media organization's co-partisans or unfavorable counter-partisans When there are many poten- tially relevant events, which subset are included in an article fundamentally affects how readers inter- pret the story, and can reveal a media outlet's s stance on that topic and their ideology (Mullainathan and Shleifer, 2005; McCombs and Reynolds, 2008; Lint- man, 2007). One example of event-selection bias is shown in Fig. 1, where a Washington Post article includes survivor's 's request to impose gun control (pro-gun control), whereas New York Post article claims Biden's 's statement as false (pro-gun rights).
randomly selected events.
For the second goal, the most critical challenge in developing computational tools to identify par- tisan events is the lack of annotation, where man- ually labeling a large-scale dataset requires do- main expertise and is highly time-consuming. For that reason, we use latent variables to represent whether an event is partisan or not, and propose to jointly infer partisan events and predict an article's idcology. Our models are trained using article-level ideology labels only, which are easier to obtain, and they do not require any human annotation of par- tisan events. We compare two approaches (Chen et al., 2018; Yu et al., 2019) to train latent variable models and explore two methods for further im- provement: (1) steering the model toward events that are selected only by one side, which are more likely to be partisan, and (2) providing prior ide- ology knowledge with pretrained event representa- tions.
This paper has two major goals: (1) examining the relation between event selection and media ide- ology, and (2) formulating a task for partisan event detection in news articles and developing computa- tional methods to automate the process. For the first goal, we verify the existence of partisan event selec- tion by measuring how event selection affects the performance of media ideology prediction. Specifi- cally, we represent articles using triplets of (ARGO, predicate, ARGI), denoting the set of events they report with participating entities (e.g., in Fig. 1). This representation shown be effective in narra- tive understanding (Chambers and Jurafsky, 2008; Mostafazadeh et al., 2016). We conduct two studies. First, we compare article-level ideology prediction performance by using events within single arti- cle vs. contextualizing them with events in other news articles on the same story but reported by media with different ideologies, inspired by the ob- servation that biased content should be evaluated against other media (Larcinese al., 2011). We show that the latter setup yields significantly higher Fl scores, suggesting that cross-article comparison can identify partisan events and thereby produce more accurate ideology prediction. Second, we annotate evaluation dataset of 50 articles that focus on two recent political issues, where in total we manually label 828 partisan events out of 1867 sentences from all articles. Testing on this dataset, we show that removing partisan events from the articles hurts idcology prediction performance sig- nificantly more than removing similar amounts of
We conduct experiments on two existing news ar- ticle datasets (Liu al., 2022; Fan et al., 2019) and our newly annotated data with partisan events (test only). Results indicate that latent variable mod- els outperform all competitive baselines on both partisan event detection and ideology prediction, where cross-article event comparison is shown to be critical for both tasks. Analysis of the extracted partisan events reveals key challenges in detecting implicit nuanced sentiments and discerning event relations (c.g., main vs. background events), sug- gesting future research directions.
To the best of our knowledge, this is the first time that computational methods are developed for studying media bias at the event selection level. It is also the first time that automatic models are investigated to detect partisan events. Our results provide new insights into a high-level form of me- dia bias that may be present even in apparently nonpartisan news, enabling a new understanding of how news media content is produced and shaped.
2 Related Work
Media Bias Understanding and Detection. Re- search in political science, economics, and com- munication has extensively demonstrated the lationship between news media and ideological bias (Mullainathan and Shleifer, 2005; Gentzkow et al., 2014). According to Broockman and Kalla (2022), there are three common strategies news media use to affect readers: Agenda setting (Mc-

---

Combs and Shaw, 1972) refers to when the pub- lic's perception of topic's overall significance is shaped by the amount of news coverage spent on that topic (Field et al., 2018; Grimmer, 2010; Quinn et al., 2010; Kim et al., 2014). Framing concerns how media highlight some aspects of the same reality to make them more salient to the pub- lic (Entman, 1993; Tsur et al., 2015; Baumer et al., 2015; Card et al., 2015; Liu et al., 2019a). Finally, partisan coverage filtering used by media to se- lectively report content that is flattering to their co- partisans or unflattering to opponents. While there is a certain amount of conceptual overlap among these three categories, this work focuses primarily on the third: the selection of which events relevant to the main stories to report, and how that reveals media outlet's s ideology and stance. Compared previous work in agenda setting, which mainly focuses on the topics of news articles (Field et al., 2018; Kim et al., 2014), our partisan event study focuses on a more thoughtful process for informa- tion filtering. Event selection is also subtler than framing, since framing examines how perspective is evoked through particular phrases (Card et al., 2015), whereas partisan event detection requires both event extraction and cross-article comparison.
Our work differs in two important aspects: First, their "informational bias" can occur in any text span, and detecting speculative information often requires complex inference and also depends on specific wording. By contrast, by focusing on the presence or absence of events, we target concrete units of potentially partisan information, which can be more easily validated and understood by readers, Second, they train supervised models on annotated biased content, while our latent variable models do not need any labels on partisan events.
Ideology Prediction with Text. Many computa- tional models have been developed to predict ide- ology using textual data (Gentzkow and Shapiro, 2010; Gerrish and Blei, 2011; Ahmed and Xing. 2010; Nguyen et al., 2013). Recent work, for in- stance, leverages neural networks to incorporate phrase-level ideology (lyyer et al., 2014), exter- nal knowledge from social media (Kulkarni et al., 2018; Li and Goldwasser, 2019), and large-scale language model pretraining (Liu et al., 2022; Baly et al., 2020). However, most of this computational work focuses directly on ideology prediction, with little attention the higher-level processes under- lying media bias. In particular, ideology prediction may fail for many mainstream media outlets who eschew overtly ideological language, and instead may bias readers only via a more sophisticated in- formation selection procedure at the event level. We demonstrate that incorporating story-level con- text enables global content comparison over po- litical spectrum, and benefits both partisan event detection and ideology prediction.
While partisan coverage filtering has been stud- ied in political science, detecting it requires human efforts review all news content (Broockman and Kalla, 2022; Baum and Groeling. 2008), making these methods unscalable and only applicable to short time periods. Grossman et al. (2022) auto- mate the process, but use predefined lists of phrases and simple topic models to determine the over- all slant and topic of news report, which cannot capture more tactful content selection like events. Most recently, a contemporaneous work (Zou et al.. 2023) also explores the partisan events within news articles, but they mainly curate a larger-scaled an- notated dataset to support fine-tuning models on the labeled events. Compared to these works, we operate with more nuanced factual details than phrases and topics, and we treat partisan events latent variables and automatically detect them from news articles with methods that are scalable large quantities of news.
Event Selection Effect Study
In this section, we verify the existence of parti- san event selection by examining partisan events' influence on ideology prediction. Using events extracted from articles, we design a model that pre- diets ideology with single- or multi-article context 1)3 based on the assumption that comparing events included by different media may reveal their idcological leanings. We then manually annotate dataset with partisan events in news stories ($3.2). Using this dataset and two existing corpora, we show that cross-article content comparison can re- veal potential partisan events and removing partisan events hurts ideology prediction ($3.3).
Another line of work that is similar to ours is the detection of informational bias (Fan et al., 2019; van den Berg and Markert, 2020), defined as "tan- gential speculative, or background information that sways readers' opinions" (Fan et al., 2019).

---

3.1 Ideology Prediction with Events
AllSides
Basil
PEvent (ours)
We build on the narrative embedding model in Wilner et al. (2021) and extend it to include story level context by adding article segment, event frequency, and event position embeddings. This allows gauge the effect of partisan events" presence or absence on ideology prediction.
stories
2,221
67
25
articles
5,361
134
50
events detected per urtilee
66.82
48.71
Table I: Statistics for AllSides training set, Basil (test only), and PartisanEvent (test only). AllSides test set contains 1,416 articles.
Event Extraction. We follow prior work (Zhang et al., 2021) to train event extractor on the MA- TRES dataset (Ning et al., 2018). Our extractor achieves an Fl score of 89.53, which is on par with the state-of-the-art performance (90.5) (Zhang et al., 2021). See details in Appendix B.I.
tails in Appendix B.2). Note that Eq. describes the model with story level context as it includes all events in all articles. We also experiment with models that only use events in one article. Finally, the model predicts article's s ideology using average representation of all events in the article.
Ideology Prediction. Given N articles d1 d1.--., NNN that report on the same news story, we denote events in article as z ,...,,L''" where L is the number of events in article a;. We first use a Distil- RoBERTa model (Sanh et al., 2019) to get the em- bedding for an event.' Concretely, we input the sentence that contains the event to DistilRoBERTa and get the embeddings epred: CargOs Cargl for pred- icate, ARGO, and ARGI by taking the average of last-layer token embeddings. If a sentence has multiple events, we mask out other events' tokens when encoding one event, so that the information in one event does not leak to others. We then get
(()
3.2 Partisan Event Dataset Annotation
Since there no dataset with partisan event an- notations for news articles, we manually label a Partisan Event (PEvent) dataset with 50 articles (1867 sentences) covering two controversial events happened in the U.S. in 2022: mass shooting in Texas, and the overturn of Roe Wade. Note that PEvent contains articles from separate and later time than the training data with ideology predic- tion objective. PEvent is only used for evaluation purposes on the task of partisan event detection.
W (predd CargU; eargi], where means con- catenation and W is learnable. We then input all events in one article or all articles on the same story to another transformer encoder (Vaswani et al., 2017) to get contextualized e for each event:
Since labeling partisan events is costly, which requires both domain knowledge and news anno- tation experience, we only focus on two broad high-profin topics where the partisanship of all constituent events is already known to coders expe- rienced with US politics. We acknowledge that a dataset with diverse topics would useful, but will leave this for the future work. We collect articles from AllSides Headline Roundups section, where groups of three articles that report the same news story are carefully selected by editors to demon- strate "how opposite sides of the media are dis- cussing or framing subject". For each story, we discard the center ideology article due to lack of consensus of what constitutes center ideology by the community. The remaining two articles, to- gether with extracted events, are provided to two college students who have prior news article anno- tation experience and have gone through careful training of the annotation tasks. They are instructed to first label article idcology, 4 and then partisan
[c (0) .......c (N) = Transformer (le)
(1)
where Transformer is a standard transformer en- coder trained from scratch (details in Appendix B.2) and contains three types of embeddings: Article embeddings distinguish the source by as- sociating the index of the article with its events, with a maximum of three articles per story. Fre- quency embeddings highlight the prevalence of events by signaling if event appears in only one article, more than one but not all articles, all articles that report the same story. We train one em- bedding for each category and use lexical matching to determine common events. Finally, position em- beddings represent the relative position of an event in the article, e.g., partisan events may appear later in the reports. All embeddings are learnable (de-
Phttps: htttps://www..e /twp -ww.te--bllannee-neeeeeeeee---e-
use Distil due to computational constraints. We use a zero vector if ARGO or ARGI does not exist.
"We intentionally annotate articles' idcology rather than using media-level idcology to ensure accurate idcology labels.

---

AllSides
Basil
PEvent
Single-articl
61.10 + 3.51
56.08 +: 6.01
41.37 =: 2.60
+ppss.
64.37 1: 0.75
54.78
a: 2.38
46.77 3.44
Multi-artilee
79.52 +: 1.32
64.91
1.78
76,61 3.16
83
Log grobabllity
tart.
58.61 + 0.84
67.30
2.45
85.19
+: 2.28
-0.50
tort. fre.
88.64
68.05
1.33
83.00 +: 1.67
Macro
82
tart. fre. pos.
2.07
1.67
-0.55
Table 2: Macro Fl scores for article ideology predic- tion (average of runs). Best results are in bold and second best are underlined. art., fre., and pos, refer to article, frequency, and position embeddings in $..1.
-0.65
0.0
0.5
1.0
0.5 Drop rate
1.0
Drop rate
Partisan Events
Random Events
events. During annotation, we only annotate left partisan events for left articles and vice versa. Fi- nally, a third annotator compares the annotations and resolves conflicts. Appendix C contains the full annotation guideline.
Figure 2: Performance (10 runs) after removing the number of partisan events and random events. Per- formance drops for both settings. but removing partisan events leads to more severe performance regression.
In total, 828 partisan events are annotated out 3035 events detected by our tool from 1867 sen- tences. Inter-annotator agreement calculated using Cohen's x (Cohen, 1960) is 0.83 for article-level ideology. For partisan event labeling, two annota- tors achieve k = 0.43, which is substantial agree- ment. After discussing with the annotators, we find that disagreement often occurs when one annotator is insufficiently confident and thus ends up labeling an event as non-partisan. Therefore during the dis- agreement resolution stage, an event is frequently deemed partisan if is labeled by at least one anno- tator. This again highlights the subtlety of partisan event usage by media. On average, 16.56 (27 .88% events are annotated as partisan events per article. Among all partisan events reported by left-leaning media, 98.41% % are chosen only by the left side, and 95.09% for the right media. We further check where partisan events are included in the articles, and find that they occur more frequently in the later parts of articles written by right-leaning media (dis- played in Fig. 4 in the Appendix). These findings validate our design in $.
is manually annotated similar to $3.2. Likewise, we remove articles of the center ideology. Table 8 presents the statistics for datasets used in this study.
We experiment with multi- and single-article variants of the model, depending on whether the transformer in Eq. has access to events in all or one article. As shown in Table 2, multi-article models that allow content comparison across arti- cles written by different media significantly outper- form single-article models, demonstrating the ben- efits of adding story-level context to reveal partisan events that improve ideology prediction. Among multi-article models, article embeddings lead to the largest gain since it supports cross-artie cross-article compari- son. For experiments the rest of this paper, we add position embedding for single-artich models and all three embeddings for multi-article models.
Effects of Removing Partisan Events. Next, we investigate how would removing partisan events affect model's prediction on ideology. Intuitively, when having access to fewer partisan events, the model will be less confident in predicting correct ideologies. Concretely, we run the multi-article model on PÊvent, We drop m% of partisan events. where m = 25, 50, 75, 100. We also run the same model and remove the same number of events ran- domly (random events). We then measure the macro Fl and log probability of true classes.
3.3 Results for Ideology Prediction
We first compare ideology prediction performance using different model variants in $3.1 and then pick two to study the effect of removing partisan events. Effects of Cross-Artich Event Comparison. We train models AllSides dataset collected in Liu al. (2022), where media outlets' ideology is used articles' ideology. We use articles before 2020 (inclusive) as training and dev data and articles after 2020 (exclusive) as test data. We also evaluate models on Basil (Fan et al., 2019), where ideology
As shown in Fig. 2, removing partisan events hurts the performance more compared removing random events. Moreover, the more partisan events are removed, the larger the performance gap is, which confirms that models exploit the presence of partisan events to discern ideology.

---

4
Latent Variable Models for Partisan Event Detection
tations to logits, from which sample k% of them following the subset sampling method in Xie and Ermon (2019)-a differentiable sampling method that allows us to train the whole system end-to-end At inference time, we select the top k% of events with the largest logits by the extractor. For the pre- dictor, we again get event embeddings but we input m @ to the transformer encoder so that it only sees the sampled subset of events.
The general idea of our latent variable models for partisan event detection is that the detected partisan events should be indicative of article's ideology, the removal of which would lower models' prediction confidence, according to our study in $3. We adopt two methods that are originally developed to extract rationales of model predictions ($4.2) for our task and further improve them by adding constraints on the usage of common events and adding prior knowledge of event-level ideology ($4.3).
Three-Player Model. Among all events in the arti- cle, some may have spurious correlation with the ideology. For instance, the event "a CNN reporter contribute to this article" can almost perfectly re- veal article's ideology. To prevent models from focusing on these shortcuts, we further investigate the method in Yu et al. (2019). Concretely, they propose a three-player model where third com- plement predictor of predicts ideology using the complement of partisan events, i.c., (1 m) O x. The goal for both predictors is to correctly predict the ideology. i.e., maximize 4og4((v m @ x) and log ( (1 m) @ x). The objective for the extractor is to select 4% of events that can predict while the remaining events cannot as in Eq. 3:
4.1 Task Overview
We assume our data comes in the form of (a, v), where y is the ideology for article a. We extract events (zi,...., z1. from article a where L is the number of events in the article. We define latent random variable m, E {0,1} } for each event Tú, and m, = means zy is a partisan event. The ideology prediction task aims at predicting y using x. The partisan event detection task focuses on predicting partisan indicators m = (m....., m.).
4.2 Latent Variable Models
max Σ
Em-e [log (v m @ x)
Two-Player Model. We adopt methods in ra- tionale extraction, where rationale is defined as part of inputs that justifies model's prediction (Lei al., 2016). We use the formulation in Chen et al. (2018), which tackles the rationale (partisan events in our model) extraction task from an information- theoretic perspective. In details, suppose a posi- tive number k is given, the goal is to extract 1% of events that have the highest mutual informa- tion with label y and treat them as partisan events. In other words, our partisan indicator m satisfies m| 4% L. Since optimizing mutual informa- tion is intractable, Chen al. (2018) provides variational lower bound as the objective instead:
(X.9)E
(3)
-log ( (1 m) C @x)
Intuitively, the extractor and the complement pre- dictor play an adversarial game, and Eq. 3 drives the extractor to identify partisan events as compre- hensive as possible so that the complement predic- tor cannot perform well. In fact, Yu et al. (2019) uses an explicit objective to penalize E; m; when it deviates from k% * L, but we find this objective does not work well with Eq. 3, leading to an extrac- tor that either selects all events partisan events or detects nothing. We thus modify it with the subset sampling method (Xie and Ermon, 2019) again. At inference time, we use da for ideology prediction. Both two-player and three-player models can have the single- and multi multi-article variants, depend- ing on whether the extractor and predictors can access all events in a story or just from a single article. Appendix D details the training process.
max Eo:26
Σ (X.))ED
Em--
(o
x]]
where Eg is an extractor that models the distribution of m given x, qa is predictor that predicts y given partisan events, D is the training set, and @ is the wlement-wise multiplication.
4.3 Improving Partisan Event Detection
We parameterize both Eg and qo using the same model in $3.1. For the extractor, we first get the embedding e for all events and then pass it to the transformer encoder to get contextualized event rep- resentations. linear layer converts these represen-
Restricting Models from Picking Common Events. As shown in Fig. 1, common background events and main events should not be considered
as partisan events. We therefore explicitly prohibit models from selecting these events. Precisely, we

---

use the same lexical matching method as in $3.1 to find common events in the story. During training, we add an auxiliary objective that minimizes the probability of the extractor to predict events that ap- pear in both left and right articles as partisan events, thus driving models to prefer events reported by only one side. We only apply this constraint to multi-article models since it requires story-level context to locate common events.
Idoology Prediction
Event
AllSides
Basil
PFivent
Pfivent
Random
49.881
501.99.3.0
Event-prior Non-lutent-attn
63.3940.00
61 3700000
55.44_2000
30.68
88. 48.407
68 50.2or
83.59=107
20.90
-pri.
89. 2...3
69. 19-1.3
83. 83.58=a22
30.38+1.40
Non latent pert -pri.
88. 49.4
68.50_2er
81.39
31.17
89.83100.
69 2..35
31.50)
Single-article Models Two player
66.75
58.28_449
28.79 48.4
prr. Thire-playee
81. $1.50
68 65-2.1
70.87=280
31.53aa.0
66. 66.8722.82
00 1522,50
48. 7...s
29.72+2.00
Pretraining to Add Event Ideology Priori. Prior knowledge, especially the media's stance on con- troversial topics, plays an important role in par- tisan content detection. Given that the AllSides training set is relatively small, it is unlikely for the model to gain such knowledge on a broad range of topics. We therefore pretrain a model on HIGNEWSALIGH dataset in Liu et al. (2022) to acquire prior knowledge at the event level.
-ri..
81. 81.06ga.. as
65 80.0.
70.57=261
30.70
Multi-article Models Two-player
86, 15
69 98
82.36g3.85
33.27
-res.
85.6840.35
68. 21.0
82.38
pri..
91.00
71.27 71.27_1.34 1.34
84.31 25.5
33.32
res. + prit.
91,58
71.43_
89.16
33.99
Three-player
85.48
65. 35.08_1.57
10.10
33.20
res.
85.8 86.8140.19
66 88-81-2.
80.36
31.68+1.1
pri.
88, 23:
70. 26.2..4
74.02=400
33.01+1.02
res. + pei.
88.02 s1.40
69 6m
80.92
31.46
Table 3: Fl scores (avg. of 5 runs) for ideology predic- tion and partisan event detection. res.: restrict models to prefer events selected by only one side; pri.: prior knowledge with pretrained event representations. Mod- els that do cross-article comparison yield better perfor- mance on both tasks. Adding prior knowledge helps in almost all cases. Non-latent models have the same ide- ology prediction scores since they are the same model. Best results are in bold and second best are underlined.
BIGNEWSALIGN is dataset with 1 million po- litical news stories, and each story contains about articles that report the same main event. We extract events in these articles and train a DistilRoBERTa model as in $3.1 to predict the ideology of each event, where we use article's ideology as event's s ideology. Note that this model takes each event as input and does not consider any context informa- tion. Intuitively, it counts the reporting frequency of each event: If an event is reported more by left media, it has higher probability of being left and vise versa. We use this pretrained model for initial- ization in the extractor and the predictor.
take the majority vote among partisan events as article's 's ideology. Intuitively, this baseline utilizes the prior knowledge of event idcology to detect par- tisan events. (3) Non-latent is the best performing multi-article model in $3.3, which does not contain latent variables. Built upon this method, we create two variants for partisan event detection. The first is attention-basee method, which is shown effec- tive at finding input words that trigger the sentiment prediction (Wang et al., 2016). We use our trained model and consider the top 30% of events with the largest attention weights (sum over all heads and positions) as partisan events. The second method is perturbation- (Li et al., 2016), where we use the non-latent model and iteratively remove one event a time and choose 30% of events that lead to the largest output change as partisan events. We also report the performance of the aattile model initialized from the pretrained event encoder in $4.3 for fair comparison with our latent models.
5 Experiments
Tasks and Datasets. Similar to $3.3, we train all models solely on AllSides and evaluate on All- Sides test set, Basil, and our partisan event dataset (PEvent). We measure idcology prediction per- formance on all three datasets and partisan event detection performance on PEvent.
Evaluation Metrics. For idcology prediction, we measure the macro Fl score at the article level. For partisan event detection, we measure the Fl score for the positive class, i.e., partisan event. Baselines. We consider the following baselines: (1) We randomly predict partisan events with a 0.3 probability, and randomly predict ideology for the article. (2) Event-prior is the pretrained event model with ideology priori in $4.3. We run to get the probability of each event being left and right. We then consider the 30% of events with the most skewed distribution as partisan events. Finally, we
Results. Table presents the results when k 30, since 27. 27.28% of events in PEvent dataset are par- tisan. We explore the influence of l in $6. Un- surprisingly, we observe that arttile models

---

Ideology Prediction/Partisan Event Detection F1
multi-article models: base two-player, two-player with restriction and prior knowledge, and basc three-player models. We train these models with = 20, 30, 40, 50 and plot the performance partisan event detection and ideology prediction on PEvent in Fig. 3. For event detection, the model improves k increases, but the improvement is moderate when & 30. For ideology prediction, the performance plateaus at k 20 except for the two-player model with restriction and prior knowl- edge, which peaks at k 30. This suggests that only a subset of events reflect the article's 's ideology. and it is enough to make predictions based on them.
90
Ideology
80
70
10
15
20
25
30
35
40
45
50
F2
40
30
20
10
15
20
25
30
35
40
45
50
Two-player -B- Two-player Res. Pri. <B. Throe-player
Error Analysis. Table and Table 10 in Appendix present predictions by the two-player model on the multi-article setup with one-sided restriction and prior knowledge for events. Two major types of er- rors are observed. First, the model struggles when an article attacks statement from the opposite side with an implicit sentiment. For instance, "threw," "continue," and "had" in Table are events or state- ments from the right, but the author reports them with an implicit negative sentiment (c.g., "not a thing!"), making the event flatter to the left. Future models need to (1) have an enhanced understanding of implicit sentiment along with the involving en- tities (Deng and Wiebe, 2015; Zhang et al., 2022), and (2) acquire knowledge of entity ideologies and their relations. Second, the model still frequently selects main events as partisan content, as shown by the "delivered" event in Table 10. For this example, is because the main event should be included as necessary context for ideology prediction, i.e., the training objective. For other examples, some main events also carry sentiment towards ideological en- tities, thus indeed should be labeled partisan events according to our definition. Future work should investigate whether the selective usage of partisan events are different when the main sto- ries already support a certain ideology compared to when they disfavor the same ideology.
Figure 3: Idcology prediction and partisan event detec- tion performance with different & values (average of 5 runs). Error bars show the standard deviation. Perfor- mance variance is small for partisan event detection.
outperform single-article models on both tasks, em- phasizing the importance of story-level context for cross dossmett event comparison.
On partisan event detection (last column of Table 3), latent variable models outperform all baselines, showing the effectiveness of training with article ideology labels. Note that the three-player mod- els do not outperform the two-player models, indi- cating that the spurious correlation may not be significant issue PEvent, and partisan events notated on PEvent, as standalone events, cannot be directly associated with specific ideological lean- ing. Moreover, restricting models from selecting common events improves partisan event detection for two-player models, which validates that com- mon events are less likely to be partisan. Providing prior knowledge of event ideology further boosts on both tasks, especially for single-article models, illustrating the benefits of prior knowledge when the context is limited. Finally, combining the two improvements, the two-player model on the multi- article setup achieves the best performance. It is also important to point out that this model only uses 30% of events to predict ideology, but still outper- forms the model that sees full articles in the story. which suggests that a good modeling of events in the article could be more helpful than raw text rep- resentations when predicting ideology.
Usage of the Latent Variable Model and Future Directions. The latent variable model can be used as a stance analyzer, which would come in handy in practice as well, especially for generating ratio- nales out-of-the-box in different settings. Firstly, being trained as an ideology predictor, it has the potential for future extensions to multi-modal ide- ology analysis, as suggested Qiu et al. (2022). Secondly, working on articles on the same topics, it enables exploration how different media outlets
6 Further Analyses and Discussions
Effect of Varying k. We now explore the ef- fect of k's values. We experiment with three

---

Title:
At the NRA Convention, People Blame Mass Shoo- tings on Everything But Guns
Title: Biden calls for assault weapons ban, making gun manufacturers liable for shootings
The nation has been plunged into despair and mourning ... in Houston, the National Rifle Association still threw -w w w w g_party ... Two messages emerged from the assembled throngs the doting politicians allendance, just 300 miles from Uvalde: 1) People must continue to enjoy the right to sequire any damn firearm they choose, with- out meddling from the state; and 2) the masseree had absolutely. not a thing!-to do with the untramme- led commerce in guns
President Biden on Thursday made an emotional appeal for ambitious new gun laws, including a han on military-style rifles ... On the other side of the aisle, Republicans bristled at Democrats' equating support for the Second A- mendment with tolerating mass murder. "You think we don't have hearts," said Rep. Louis Gohmert, Texas Republican
Idcology label: night
Prediction: left
Ideology label: left
Prediction: left
Table 5: Article snippets where the extractor detects right event, but the predictor predicts the article as left.
Table 4: Article snippets of model predictions (multi- article two-player model with both improvements) and annotations. Colored spans denote events, with the predicate bolded. Blue: model predictions; red: human annotations; purple: annotations and predictions.
Acknowledgments
This work is supported in part by the National Sci- ence Foundation under grant III-2127747 and by the Air Force Office of Scientific Research through grant FF95550-2-1-0099, We would like thank the anonymous reviewers for their helpful com- ments and feedback.
convey stances on shared topics by highlighting the inclusion or omission of partisan events. Lastly, our tool has the potential to complement entity-focused stance analysis. For instance, in the context of a left-leaning entity slashing right-leaning entity, existing datasets only reveal a stance label between the entities with no further rationale provided the users (Deng and Wiebe, 2015; Zhang al., 2022). In contrast, our analyzer can take an anno- tated stance as input and then identify the partisan event that influences readers' interpretation.
8 Limitations
We investigate the impact of event selection on models' ideology prediction performance, to ver- ify the existence of event selection in news media. The results, however, do not state a causal relation between media ideology and reported events.
We analyze the model output and discuss in details two major limitations of our latent vari- able models in $6. Apart from those errors, we also observe that events detected by the model as partisan may not align with the model's pre- diction of the article's ideology. In other words, the model could identify right-leaning events as partisan events while predicting the article as left- leaning (Table 5). Although the methods we adopt in this paper identify events that are indicative of ideology (Chen al., 2018; Yu et al., 2019), they do not provide further justifications for how these events interact reflect the ideology. For instance, the extractor could detect a right event and several left events that attack it. To further understand the event selection effect, future work may consider incorporating event-level ideology to model the interplay among events.
7 Conclusion
Partisan event selection is an important form of media bias which may exist in even the most ap- parently nonpartisan news, but which is especially hard to detect without extensive cross-artes cross-article com- parison. We first verify the existence of partisan event selection by inspecting the impact of parti- san events on the performance of ideology predic- tion. We then jointly detect partisan events and predict article's ideology using latent variable mod- els. Experiments show that detected partisan events reasonably align with human judgement, and our models using cross-article event context outper- forms the counterpart that only uses single-article context on both two tasks. Our analysis also sug- gests future directions for identifying interactions among entities in an article and for resolving event coreference across articles.
A Although our models that include cross-article context can be extended to any number of articles without modification, they may be restricted by the GPU memory limit in practice. Particularly, the Transformer encoder that contextualizes all events

---

in a story requires computational resources to scale quadratically with the number of events, which is infeasible for stories that contain many articles. Future work may consider designing novel mecha- nisms to address this issue, e.g.. by using special attention patterns based on the discourse role of each event in the article (van Dijk, 1988: Choubey et al., 2020).
wrong ideology for an article. They may cause mis- perception and misunderstanding of an event. For vulnerable populations (e.g., people who maybe not have the specific knowledge to make the right judgements), the harm could be amplified if they blindly trust the machine outputs.
Biases. The training dataset is roughly balanced in the number of left and right articles, so the model is not trained to encode bias. However, the dataset relatively small and does not cover all possible political topics. Particularly, most of the news arti- cles in the training set are related to U.S. politics, thus the model is not directly applicable to other areas in the world.
Finally, due to the cost of manual labeling, we only evaluate our partisan event detection models on a dataset that covers two specific political issues. It remains to be seen whether methods introduced in this paper can be generalized to broader range of issues. We call for the community's s attention to design and evaluate partisan event detection models on more diverse topics.
Misuse Potential. Users may mistakenly take the model outputs as ground truth. We recommend any usage of our model displaying "use with caution" message to encourage users to cross-check the in- formation from different sources and not blindly trust a single source.
9 Ethical Considerations
9.1 Dataset Collection and Usage
Partisan Event Dataset Collection. We conform with the terms of use of the source websites and the intellectual property and privacy rights of the original authors of the texts when collecting articles. We not collect any sensitive information that can reveal original author's 's identity. We also consult Section 107$ of the U.S. Copyright Act and ensure that our collection action fall under the fair use category.
Datasets Usage. Except the partisan event dataset collected this work, we get access to the Basil dataset by direct download. For AllSides, we con- tact with the authors and obtain the data by agreeing that we will not further distribute it.
9.2 Usage in Application
Intended Use. The model developed in this work has the potential assist the public better under- stand and detect media bias news articles. The experiments in $5 show that our model is able to identify partisan events on two controversial issues that moderately align with human judgement. The detected events can be presented to show different perspectives from both ends of the political spec- trum, thus providing readers with more complete view of political issues.
Failure Modes. Our model fails when it mistak- enly predicts a non-partisan event as a partisan event, misses out the partisan events, or predicts the
http::/wwww.ogrrgggggggg/////e/e html#107 107

---

References
Prafulla Kumar Choubey, Aaron Lee, Ruihong Huang, and Lu Wang. 2020. Discourse as a function of event: Profiling discourse structure in news articles around the main event. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5374-5386, Online. Association for Computational Linguistics.
Amr Ahmed and Eric Xing. 2010. Staying informed: Supervised and semi-supervised multi-view topical analysis of ideological perspective. In Proceedings of the 2010 Conference on Empirical Methods in Nat- ural Language Processing. pages 1140-1150, Cam- bridge, MA. Association for Computational Linguis- tics.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Mea- surement, 20 10)37377-6..
Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020. We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4982-4991, Online. Association for Computa- tional Linguistics.
Dave D' D'Alessio and Mike Allen. 2006. Media Bias in Presidential Elections: A Meta--nalysis. Journal of Communication, 50(4):133-156,
Claes de Vreese. 2004. The effects of strategic news on political cynicism, issue evaluations, and policy sup- port: two-wave experiment. Mass Communication and Society, 1(2):111-214
Matthew A. Baum and Tim Groeling. 2008. New media and the polarization of american political discourse. Political Communicution, 25(4):345-365.
Stefano DellaVigna and Matthew Gentzkow. 2009. Per- suasion: Empirical evidence. Working Paper 15298, National Bureau of Economic Research.
Eric Baumer, Elisha Elovic, Ying Qin, Francesca Pol- letta, and Geri Gay. 2015. Testing and comparing computational approaches for identifying the lan- guage of framing in political news. In Proceedings of the 2015 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, pages 1472-144 1482, Denver, Colorado. Association for Computational Linguistics.
Stefano DellaVigna and Ethan Kaplan, 2007. The Fox News Effect: Media Bias and Voting*. The Quarterly Journal of Economics, 122(3):1187-122
Lingjaa Deng and Janyee Wiebe. 2015. MPQA 3.0: An entitylevent-lew sentiment corpus. In Proceedings of the 2015 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1323-1328, Denver, Colorado. Association for Computational Linguisties.
Steven Bird, Ewan Klein, and Fedward Loper. 2009. Nut- ural language processing with Python: analyzing text with the natural language toolkit. O'Reilly Media, Inc.".
Benjamin Enke. 2020. What you see is all there is. The Quarterly Journal of Economics, 135(3):1363-1398
David Broockman and Joshua Kalla, 2022. The mani- fold effects of partisan media viewers' helief' and attitudes: A field experiment with fox news viewers.
Robert M. Entman. 1993. Framing:
Toward clarification of a fractured paradigm. Journal of Communication, 43(4):51-58.
Dallas Card, Amber Boydstun, Justin H. Gross, Philip Resnik, and Noah A. Smith. 2015. The media frames corpus: Annotations of frames across issues. In Pro- ceedings of the 53rd Annual Meeting of the Asso- ciation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume Short Papers), pages 438 444, Beijing, China. Association for Computational Linguistics.
Robert M. Entman. 2007. Framing bias: Media in the distribution of power. Journal of Communication. 57(1):163-173
Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Pra- fulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019. In plain sight: Media bias through the lens of factual reporting. In Proceedings of the 2019 Confer- ence Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNILP), pages 6343 6339, Hong Kong, China. Association for Computational Linguistics.
Nathanael Chambers and Dan Jurafsky. 2008 Unsuper- vised learning of narrative event chains. In Proceed- ings of ACL-08: HLT. pages 789-797, Columbus, Ohio. Association for Computational Linguistics.
Jianbo Chen, Le Song, Martin Wainwright, and Michael I. Jordan. 2018. Learning to explain: An intommation-thoorete perspective on model interpre- tation. In Proceedings of the 35th International Con- ference on Machine Learning, ICML. 2018, Stock- holmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Re- search, pages 882-891. PMLR.
Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018. Fram- ing and agenda-setting in Russian news: computa- tional analysis of intricate political strategies. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3570- 3580, Brussels, Belgium. Association for Computa- tional Linguistics.

---

Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord. Pradeep Dasigi, Nelson F. Liu, Matthew Pe- ters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language pro- cessing platform. In Proceedings of Workshop for NIP Open Source Software (NI.P-OSS), pages 1-6, Melhourne, Australia. Association for Computational Linguistics.
Vivek Kulkarni, Junting Ye, Steve Skiena, and William Yang Wang. 2018. Multi-view models for political ideology detection of news articles. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3518- 3527, Brussels, Belgium. Association for Computa- tional Linguistics.
Valentino Larcinese, Riccardo Puglisi, and James M Snyder Jr. 2011. Partisan bias in economic news: Evidence on the agenda-setting behavior of us news- papers. Journal of public Economics, 95(9-10):1178. 1189.
Matthew Gentzkow and Jesse M. Shapiro. 2006. Media bias and reputation. The Journal of political econ- omy, I 1142)-280-3..
Matthew Gentzkow and Jesse M. Shapiro. 2010. What drives media slant? evidence from u.s. daily newspa- pers. Econometrica, 78(1):35-71.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 107-117, Austin, Texas, Association for Computational Linguistics.
Matthew Gentzkow, M Shapiro, and Michael Sink- inson. 2014. Competition and idcological diversity:
Historical evidence from us newspapers. American Economic Review, 104( 13)33333333311
Chang Li and Dan Goldwasser. 2019. Encoding so-
cial information with graph convolutional networks forPolitical perspective detection in news media. Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2594- 2604, Florence, Italy. Association for Computational Linguistics.
Matthew Gentzkow, Jesse M. Shapiro, and Daniel F. Stone. 2015. Chapter 14- media bias in the market- place: Theory. In Simon Anderson, Joel Wald- Togel, and David Strömberg, editors, Handbook of Media Economics, volume of Handbook of Media Economics, pages 623-645. North-Holland.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Un- derstanding neural networks through representation erasure.
Sean M. Gerrish and David M. Blei. 2011. Predicting legislative roll calls from text. In Proceedings of the 28th International Conference on International Conference on Machine Learning. ICML'11, page 489-490 496, Madison, WI, USA. Omnipress.
Siyi Liu, Lei Guo, Kate Mays. Margrit Betke, and Derry Tanti Wijaya. 2019a. Detecting frames in news headlines and its application analyzing news framing trends surrounding U.S. gun violence. In Proceedings of the 23rd Conference on Computa- tional Natural Language Learning (CoNI..), pages 504-514, Hong Kong, China. Association for Com- putational Linguistics.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative ad- versarial nets. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining ap- proach.
Justin Grimmer. 2010. A bayesian hierarchical topic model for political texts: Measuring expressed agen- das in senate press releases. Political Analysis, 18(1):1-35.
Guy Grossman, Yotam Margalit, and Tamar Mitts. 2022. How the ultrarich use media ownership as a politi- cal investment. The Journal of Politics, 84(4):1913- 1931.
Yujian Liu, Xinliang Frederick Zhang. David Wegsman, Nicholas Beauchamp, and Lu Wang. 2022. POLI- TICS: Pretraining with same-story article comparison for ideology prediction and stance detection. In Find- ings of the Association for Computational Linguis- tics: NAACL 2022, pages 1354-1374, Seattle, United States. Association for Computational Linguistics.
Mohit lyyer, Peter Enns, Jordan Boyd-Graber and Philip Resnik. 2014. Political ideology detection us- ing recursive neural networks. In Proceedings of the 52nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1113-1122, Baltimore, Maryland. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter, 2019, Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Maxwell McCombs and Amy Reynolds. 2008 How the news shapes our civic agenda. In Media Biffects, pages 17-32 Routledge.
Yevoul Kim, Suin Kim, Alejandro Jaimes, and Alice Oh. 2014. A computational analysis of agenda setting. In Proceedings of the 23rd International Conference on World Wide Web, WWW '14 Companion, page 323-324, New York, NY, USA. Association for Com- puting Machinery.
Maxwell E. McCombs and Donald L. Shaw. 1972. The agenda-setting function of mass media. The Public Opinion Quarterly, 36(2): (6(2):7776 76-1877

---

Nasrin Mostafazadeh. Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages $39-849, San Diego, California. Association for Computational Linguis- tics.
Changyuan Qiu, Winston Wu, Xinliang Frederick Zhang, and Lu Wang. 2022. Late fusion with triplet margin objective for multimodal ideology prediction and analysis. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pages 9720-9736, Abu Dhabi, United Arah Emirates. Association for Computational Lin- guistics.
Kevin M. Quinn, Burt L.. Monroe, Michael Colaresi, Michael H. Crespin, and Dragomir R. Radev. 2010. How to analyze political attention with minimal as- sumptions and costs. American Journal of Political Science, 54(1))209-228.
Sendhil Mullainathan and Andrei Shleifer. 2005. The market for news. American Economic Review, 95(4): 1031-13331
Viet-An Nguyen, Jordan Ying, and Philip Resnik. 2013. Lexical and hierarchical topic regression. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.
Qiang Ning. Sanjay Subramanian, and Dan Roth. 2019. An improved neural baseline for temporal relation extraction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing (EMNLP-IJCNLP), pages 6203-628 6209, Hong Kong, China. Association for Com- putational Linguistics.
Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling, ArXiv, abs/1904.05225
Oren Tsur, Dan Calacci, and David Lazer. 2015. A frame of mind: Using statistical models for detec- tion of framing and agenda setting campaigns. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 1629- 1638, Beijing, China. Association for Computational Linguistics.
Qiang Ning, Hao Wu, and Dan Roth. 2018. A multi- axis annotation scheme for event temporal relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Pupers), pages 1318-1328, Melbourne, Aus- tralia. Association for Computational Linguistics.
Esther van den Berg and Katja Markert. 2020. Context in informational bias detection. In Proceedings of the 28th International Conference on Computational Linguistics. pages 6315-6326, Barcelona, Spain (On- line). International Committee on Computational Lin- guistics.
Adam Paszke, Sam Gross. Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Yang, Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Im- perative Style, Pergh-Pefformance Deep Learning Li- brary. Curran Associates Inc., Red Hook, NY, USA.
Teun Adrianus van Dijk. 1988. News as discourse. L. Erlbaum Associates.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, & ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, Curran Associates, Inc.
Elizabeth M. Perse and Jennifer Lambe. 2016. Media Effects and Society. Routledge.
Gerald Prince. 2012. A grammar of stories. In A Gram- mar of Stories. De Gruyter Mouton.
Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-basel LSTM for aspect- level sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 606-615, Austin, Texas. Association for Computational Linguisties.
James Pustejovsky, José M. Castaño, Robert Ingria, Roser Saurí, Robert J. Gaizauskas, Andrea Set- zer, Graham Katz, and Dragomir R. Radev. 2003. Timeml: Robust specification of event and temporal expressions in text. In New Directions in Question Answering.
Sean Wilner, Daniel Woolridge, and Madeleine Glick. 2021. Narrative embedding: Re- Re-Contextualization through attention. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 1393-1405, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Peng Qi. Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations.

---

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of taate-of-th---ee the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 48845, Online. Association for Computational Linguistics.
Winston Wu and David Yarowsky. 2020. Computational etymology and word emergence. In Proceedings of the 12th Language Resources and Evaluation Confer- ence, pages 3252-3259, Marseille, France. European Language Resources Association.
Sang Michael Xie and Stefano Ermon. 2019. Repa- rameterizable subset sampling via continuous relax- ations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI' 19, page 3919-3925. AAAI Press.
Mo Yu, Shiyu Chang. Yang Zhang, and Tommi Jolk 2019. Rethinking cooperative rationalization: In- trosecttvve extraction and complement control. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNIPP) pages 4094- 4103, Hong Kong, China. Association for Computa- tional Linguistics.
Shuaicheng Zhang, Lifu Huang, and Qiang Ning- 2021. Extracting temporal event relation with syntactic- guided temporal graph transformer.
Xinliang Frederick Zhang, Nick Beauchamp, and Lu Wang. 2022. Generative entity-to-entity stance de- tection with knowledge graph augmentation. In Pro- ceedings of the 2022 Conference Empirical Meth- ods in Natural Language Processing, pages 9950- 9969, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Kaijian Zou, Xinliang Frederick Zhang, Winston Wu, Nick Beauchamp, and Lu Wang. 2023. Crossing the aisle: Unveiling partisan and counter-partisan events reporting. In Findings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing. Association for Computational Linguis- tics.

---

Appendix A Implementation Details
Hyperparameter
Value
number of epochs
20
For all experiments in this paper, our imple- mentation is based on Pytorch (Paszke et al., 2019) and Hugginglaace transformers (Wolf et al.. 2020) library, and preprocess all articles us- ing Stanza (Qi et al., 2020). All experiments conducted on NVIDIA RTX A6000 GPUs.
patience
4
maximum learning rate
3e.5
leamning rate scheduler
lincur docay with warmup
warmup percentage
6%
optimizer
Adam (Loshchilov and Hutter, 2019)
weight decay
Se.5
Appendix B
Event-based Ideology
FENN layer
Prediction Models
hidden layer dimension in PPNN
768
dropout FFNN
0.1
B.1 Event Extraction
Table 6: Hyperparameters used for the event extraction model.
We follow the scheme in TimeML which defines events as "situations that happen or occur" (Puste- jovsky et al., 2003). We train an event extraction model on the MATRES data (Ning et al., 2018), its event annotation is not limited to predefined event types, and thus is applicable to the open do- main scenario. We use RoBERTa-large (Liu et al., 2019b) that predicts binary label for each word, deciding whether the word is an event predicate or not. To provide surrounding context, we split arti- cles into groups of 4 sentences and process sen- tences together. We follow previous work on using TimeBank and AQUAINT sections in MATRES as training set and Platinum section as test set (Ning et al., 2019). Table 6 shows the hyperparameters for model architecture and training process. On the same train and test split, our model achieves an Fl score of 89.53, which is on par with the state-of- the-art performance of 90.5 FI score (Zhang et al.,
B.2 Contextualized Event Representation
Here we detail our model that uses cross-article context for idcology prediction. As described $3.1, we first input the sentence that contains the event to a DistilRoBERTa model (Sanh et al., 2019) to get event representation This representation is then passed to Transformer encoder (Vaswani et al., 2017) with three embeddings to obtaín texualied event representation c:
Article embedding indicates the index of the ar- ticle that contains the event, with one embedding per article index. The datasets we experiment with in this paper have at most articles in each story. During training, we randomly shuffle the articles in each story.
Frequency embedding informs the model whether the event appears in only one article, 2021). As verbs and nouns account for 96.8% of event predicates in MATRES dataset, we extract arguments 0 and for verb and noun predicates us- ing semantic role labeling tools (Shi and Lin, 2019; Gardner al., 2018) 6 and we only keep predicates that match our event extraction results.
at least two but not all articles, or all articles in the story. We have one embedding per cate- gory. We find common events through lexical matching. Concretely, we use dictionary that contains derivational morphology mappings (Wu and Yarowsky, 2020) to get the base form the event predicate. We then construct a set of words for the predicate by including the syn- onyms for the base form and original form (Bird et al., 2009). Finally, two events considered the same if their predicate both of their ARGO and ARGI have a high word overlap (a threshold of 0.4,7 calculated by over-
Multiple events can exist in one sentence with overlapping predicates and arguments. We hence remove the shorter event if there is an overlap, as we find that shorter events tend to be less infor- mative. For example, it is easier to determine the partisanship of the event "the leak of a draft opin- ion would mark stunning betrayal of the Court's s process" than shorter one "the leak of draft opinion." Therefore, remove event if its predicate is covered by another event's arguments.
overlap and
"We search threshold values from 0.2 to 0,5 by manually inspecting identified common events in 6 articles. A value of 0).4 can identify common events accurately while still allowing variations such as variants of mentions (c.g.. president vs.
Sttph/ub..oo//ooooon/sssS-L---- for nouns.

---

Hyperparameter
Value
AllSides
Basil
PEvent (ours)
number epochs
stories
2,221
67
25
maximum learning rate
5c-5
articles
5,361
134
50
learning rate scheduler
linear decay with warmup
events detected per uricle
48.71
Table 8: Statistics for AllSides training set, Basil (test only), and PartisanEvent (test only). AllSides test set contains 1,416 articles.
warmup percentage
6%
optimizer
AdamW
weight decay
le-4
transformer hidden dimension
768
extracted events (predicate, ARGO, and ARGI) to annotators, without revealing the media source. The annotators are asked to first finish reading two articles on the same story but written by media of left and right leanings. They will then follow the steps below:
transformer
# transformer layer
# FENN layer
hidden layer dimension FFNN
768
dropout in FENN
0.1
Table 7: Hyperparameters used for the event-based ide-
Sort articles by their ideological position (left right) in this story.
ology prediction model.
lap coefficient, without stop words).
Identify the main entities or pronouns in ARGO and ARGI of the event. The main entities can be the name of political groups/figures, bills/legislation, political movements or anything related to the topic of each article. If ARGO and ARGI are empty, identify the main entities pronouns within the same sentence. Based the context, try to resolve what event or entity each pronoun refers to.
Position embedding represents the relative po- sition of the event in the article. We multiply the relative position of the event (a real number in [0, 1]) with learnable embedding.
We further train [SEP] token that separates the events from different articles. Finally, average representation all events in an article is used to predict the article's ' ideology. Table 7 includes the hyperparameters of the model.
Estimate the sentiment toward each entity in the event. Sentiments can be reflected in words, quotations, and the relations between entities.
The entire model contains 106M parameters. On average, the training takes 25 minutes single NVIDIA RTX A6000 GPU.
Use entities and sentiments to decide whether the event is sided with the article's ideology. If it does, label as a partisan event. Ex. Label an event as partisan in the left article, only if its "left" entity has positive sentiment, or its "right" entity has negative sentiment. Also, it may be possible for events to be purely factual, which means there is no strong sentiment toward entities in events. For these kinds of events, try your best to estimate whether these events indi- rectly present any sentiment toward entities in the article.
Appendix C Partisan Event Annotation
Data Collection. We manually collect stories, each with three articles from AllSides that relate to the mass shooting in Texas and the overturn of Roe v. Wade. We extract events from each article and only keep the left and right article in each story. We mask out the name of the media (e.g.. "CNN" and "Fox News") in the article before annotation avoid bias.
Annotation Process. We hire three college stu- dents proficient English and familiar with dis- cerning ideology under the context of U.S. political spectrum. We present each story, together with
Two annotators label all 50 articles, and a third annotator compares their annotations and resolve conflicts. We calculate inter-annotator agreement on all 50 articles and numbers can be found in $3.2.
president Biden).
"ht //www www..allslin allsides. con/
aabiased-balancedd
Partisan Events Distribution. We further investi- gate the distribution position of partisan events in the article. Fig. 4 shows the percentage of parti-
"Each story on AllSides contains three articles from left, center, and right respectively. We only include the left and right articles in our datasct.

---

Partisan Events Position Distribution
# articles
# events
00%
Left
128, 481
6, 280, 732
Right
123, 380
1, 98, 165
everts in quartle
30%
Table 9: Statistics for the BIGNEWSALIGN pretraining dataset.
20%
10%
Title: "Enough":
Biden Exhorts Congress To Pass Gun
Control Laws
Partisan by
by Right
President Joe Biden delivered the second evening address of his presidency on Thursday night, almost begging Congress to
Figure 4: Distribution of partisan events found in each quartile of article, in terms of spatiality. Shaded area shows the 95% confidence interval.
control legislation Biden cited former Supreme Court Justice Antonin Scalia- conservative icon-who had declared that the Second Am- endment "not unlimited."
However,
events that belong to each quartile of an article. As can be observed, right articles have more par- tisan events that appear later parts of an article, whereas partisan events in left articles are evenly distributed in the article.
Ideology label: left
Prediction: left
Table 10: Article snippets of human annotations and model predictions (multi-article two-player model with both improvements). Highlighted spans denote events. with the predicate bolded. Blue: model predictions; purple: human annotations and predictions.
Appendix D Latent Variable Models
Implementation Details. For both extractor and predictors, we use the same model architecture as in $B.2 with hyperparameters listed Table 7. For the three-player model, we follow the training pro- cess Generative Adversarial Nets training (Good- fellow et al., 2014).
livered" as a partisan event, which is the main event in the story. The constraint introduced in $4.3 fails in this case because the other article describes this event differently (i.e., "Biden made an emotional appeal"), thus suggesting future research direction that leverages cross cross-document event coreference.
The two-player model contains 213M parame- ters, and the three-player model contains 320M parameters. On average, the training takes 50 min- utes for the two-player model and 1.5 hours for the three-plae pirre--payer model on single NVIDIA RIX A6000 GPU.
Pretrained Model for Event Representation. use the BIGNEWSALIGN dataset (Liu et al., 2022) to pretrain a model with prior event ideology knowl- edge. We remove stories in the dataset that contain duplicate articles and downsample articles in each story that the number of left and right articles are balanced. Table 9 shows the statistics of the pre- training dataset. We then train Bista model that takes each event as input and predicts the event's idcology, where we use the article ide- ology as the event's ideology. We train this model on BIGNEWSALIGN for 2 epochs and use it to initialize our latent variable models.
Appendix E
Additional Error Analysis
Table 10 this section is supplementary to the Er- ror Analysis section in $6. The model detects "de-


Reviews:
[{"review_id": "QNc3k6svW6", "reviewer": "Reviewer_bwkA", "summary": "", "questions": "See reasons to reject above -- Happy to hear what the authors think and any clarifications that they might have. I will adjust my ratings accordingly.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to first thank reviewer#bwkA for your time and thoughtful input. Please find our responses to your questions below.\n\n**Q1.** The event extraction is not part of our partisan event detection task (section 4). Concretely, we assume the inputs to our ideology prediction model in section 3 and latent variable model in section 4 are **events** instead of articles. These events are extracted by an off-the-shelf event extraction model (Appendix B.1) in the format of triplets of  <ARG0, predicate, ARG1>. The event representations are based on contextual representations derived from the articles.\n\nTo annotate partisan events in the PEvent dataset, we use the same event extraction model to extract events from an article in the first place. Annotators are then asked to label partisan events **among the extracted events**. We will make sure this is clearly stated in the revision\n\n\n**Q2.** Thanks for the suggestion. We will add a task definition section to clarify task input/output format at the beginning of section 3.\n\n\n**Q3.** We would like to point out that we only annotate partisan events on the PEvent dataset, which is only used for the **evaluation purpose**. Our training/validation sets **do not contain labeled partisan events**. Instead, we train the latent variable models using the article-level ideology labels, and the identified partisan events are what the model “believes” to be the most indicative of the article-level ideology.\n\nThe partisan event detection F1 score for the random baseline reported in the paper (28.93%) is the average performance of 5 runs. The expected (i.e., average of infinite numbers of runs) F1 score for the random baseline should be 28.58% (precision of 27.28% and recall of 30.00%).\n\n\n**Presentation improvements.** Thanks for the suggestions. We will incorporate them in the revision.\n\n\nWe hope we have addressed all the concerns you raised about our work and hope you can reconsider your evaluation. If there are any other issues, please let us know by updating your review, and we would be happy to resolve them in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The parisan event detection task could be an interesting reserach topic for the community,\nThe proposed dataset could facilitate future research and evaluation on the task. \nThe drop rate analysis for ideology prediction after removing partisan events is interesting, and supports the motivation of the study.  \nThe paper uses a latent variable model to demonstrate the benefit of using partisan event detection to improve ideology prediction.", "reasons_to_reject": "In the paper's current form, many of the experimental settings are underspecified, and could be greatly improved by another round of revision. Specifically, here are some comments and suggestions --\n\n1. Personally I'm having a hard time understanding the task format of event detection. During evaluation, are the models expected to extract the events, or if the gold events are given as input for each document? Given how the authors describe the \"evaluation metrics\", I'm guessing the gold events are given? However, section 4.1 \"task overview\" mentions \"we extract events\" from document, which seem to suggest that event extraction is a part of the task?? \n \n2. Along the previous point, It's probably worth having a separate section for task definition, perhaps at the beginning of section 3.  What Section 4.1 \"Task Overview\" offers is a description of the inputs/outputs of latent variable models, but the precise definition of task or the format in which the task is evaluated is still unclear. \n\n3. Ideally, please include the % of \"partisan\" events in train/test/dev split, as understanding whether the model is trained in similar label distritbution is important for readers to understand your baseline performance in context. The random baseline is getting 28.93%, which seems to suggest there are 28.93% in test set, while the paper mentions \"27.28% of events in PEvent dataset are partisan\".  So I would've guessed that all splits have similar distribution. (A reader of your paper shouldn't really be guessing for such things!)\n\nWith the evaluation + experimental settings clarified, I would tend to rate it as a solid paper. But in its current form, presentation needs to be improved.", "questions_to_authors": "See reasons to reject above -- Happy to hear what the authors think and any clarifications that they might have. I will adjust my ratings accordingly.", "paper_topic_and_main_contributions": "The paper studies the problem of detecting patisan events, i.e. selectively reported events that are flattering to a media organization’s copartisans or unfavorable to opponents. The paper defines patisan events detection as a NLP task, where given an article, a model is expected to extract all events in the form of predicate-argument triples, and identify the set of partisan events. To facillitate evaluation on the task, the paper introduces a benchmark dataset. The paper offers a latent variable model as baseline for the task, and conduct analysis to demonstrate the effectiveness and weakness of the model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Here are a few structral improvements that I would suggest for the paper -- \n1. Have a \"task definition\" section and describe the evaluation format of the task without considering the solution. This can be in the beginning of section 3.\n2. Dedicate more space for result analysis, so that people understand (1) What baseline models can do currently, and (2) If people are going to work on your problem, where can potential improvements come from.\n3. Maybe revise figure 1, so that the task setting is clear from the get-go. In its current form, it looks like the task requires the model to extract the event as well.  (I still have no clue what the setting is up to now.)"}}, {"review_id": "m7Ycv8FWQj", "reviewer": "Reviewer_MVtQ", "summary": "", "questions": "A. It is unclear why nonpartisan news content is mentioned in the following sentence: “providing new insights into high-level media bias even in seemingly nonpartisan news content.” Can you elaborate more on this point?  \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks reviewer#MVtQ for your constructive feedback and assessment of our work.\n\n**R1.** Our current version might not have clearly articulated the difference among these concepts. We will polish them in our revised version as follows. Prior work on agenda setting is more on **topics-level**, e.g., whether abortion is something significant to talk about, and existing work on framing is more on **linguistic choices** by highlighting one particular aspect , e.g., pro-life frame v.s. Pro-choice frame. In contrast,  our work on partisan coverage filtering is on the subtle way by which the media sway their readers through **conscious inclusion or omission of some certain events or subevents**. Eventually, this fundamentally affects how readers will interpret the stories through salience estimation and causal relation understanding.\n\nAs for the significance of event-level selection bias to the media bias analysis, we will elaborate it more in the revision. Below is our brief response:\n\nWe would like to first stress the **difficulty of locating a reported biased event**. This is mainly due to the fact that an event evinces partisanship only under certain circumstances even if an event itself might not be biased. Detecting such a partisanship is non-trivial since partisan events are meticulously and consciously selected by the writer in a subtle way to make the news content maintain apparent objectivity (see our response to R4). Our work is the first that advances effective partisan event extraction out of a set out seemingly neutral events, which would in turn allow people to spot out where the media bias emerges.\n\nFurther, traditional techniques tackling media bias challenges are only able to predict article or media-level ideological slant. In contrast, our model enhances models’ **reliability and interpretability** by explicitly informing the audiences of potentially biased events (i.e., detected partisan events).\n\nAnother merit of our approach is that it **does not rely on annotated partisan event labels**, which makes it easier to adapt to unseen topics since article-level ideology labels are cheaper to get. \n\n**R2.** We admit that the evaluation results might only adequately reveal our models’ performances on two representative topics in 2022. However, it is worth noting that the two topics in our annotated PEvent are only used for evaluation, while our models are trained on AllSides, which covers a disjoint set of topics. The improved performance on the cross-topic setting **demonstrates the generalizability of our model**. Therefore, we conjecture that models would follow a similar trend when performing on other common yet controversial topics. We will leave a larger-scale evaluation to the future study. \n\n**R3.** As validated in Broockman and Kalla (2022), researchers in the social science community have unveiled the existence of partisan event selection bias, which has a persuasion effect to sway audience beliefs and attitudes. Our work, motivated by their survey-based empirical finding, **develops computational tools to verify the existence** in the following two aspects:\n\n(1) We inspect the impact of partisan events on the performance of ideology prediction to indirectly validate the existence (section 3). (2) Due to the absence of annotated partisan events in the literature, we curate PEvent by directly annotating partisan events. The PEvent dataset illuminates that **partisan events are ubiquitous in news and account for around 30% of reported events** (section 4). We hope that our tools and findings can be used in future studies of assessing the belief change and attitude change by the public at scale.\n\n**R4.** We acknowledge that the best model achieves incremental improvement. However, considering the subjectivity and difficulty of the task where human annotators can only achieve 52.83 F1 score (using one annotator's labels as ground truth and another's as prediction), the improvement (33.99 vs 28.93) is non-trivial.\n\n**Additional Questions.** In journalism, journalistic objectivity is a foundational principle that instructs journalists to report news content factually and non-partisanly. However, the actual news reporting is inevitably influenced by ideological slant of media outlets or article writers. In order to maintain a balance between objectivity principle and ideological slant, writers selectively include or omit some certain events (i.e., partisan events) without the usage of partisan words, which in turn still sways people’s interpretation. However, readers might falsely deem such an article non-partisan, though, it is not as we and other works argue, because of the absence of partisan words, so we call articles of this kind **seemingly non-partisan**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A1. Investigating media bias is an important societal challenge, and the study proposes new computational methods for studying media bias at the event selection level.\n\nA2. The study formulates a new NLP task for partisan event detection in news articles and provides a new annotated dataset of 828 partisan events from 50 articles that focus on two political issues.\n\nA3. It proposes latent variable models to detect partisan events, and these models outperform competitive baselines in both partisan event detection and ideology prediction tasks.\n\nA4. The paper is very well written, and the methods and analyses sound strong.\n", "reasons_to_reject": "R1. In the manuscript, the authors mention that the 'partisan coverage filtering' has a conceptual overlap with coverage bias (agenda setting) or framing bias. Unfortunately, the merit of this approach is not well elucidated. It would greatly enhance the clarity of the study if the authors could provide a more comprehensive explanation of how event-level selection bias contributes to the analysis of media bias.\n\nR2. The authors' efforts in creating a new annotated dataset of partisan news events are appreciated. However, it is important to note that these events are derived from only two political issues, which may limit the generalizability of the study's findings.\n\nR3. The study indirectly verifies the existence of partisan event selection bias by predicting the ideology of news articles. Nonetheless, it remains unclear whether those partisan events hold enough significance to sway audience beliefs and attitudes.\n\nR4. The proposed model's performance for partisan event detection seems to show marginal improvement, with approximately a 5 percent point increase in the f1 score compared to the random model's score of 28.93.\n", "questions_to_authors": "A. It is unclear why nonpartisan news content is mentioned in the following sentence: “providing new insights into high-level media bias even in seemingly nonpartisan news content.” Can you elaborate more on this point?  \n", "paper_topic_and_main_contributions": "This paper addresses the gap in understanding media bias at the event selection level and introduces computational methods for detecting partisan events in news articles. By focusing on event selection rather than superficial biases, the study investigates how media ideology influences the choice of events to include in a news story. The authors define partisan events as selectively reported events that favor a media organization's partisans or are unfavorable to their opponents. The paper first shows the existence of partisan event selection by 1) article-level ideology prediction in single vs cross- article comparison setup and 2) removing partisan events from the articles for article-level ideology prediction. Then, the paper proposes the use of latent variables to represent whether an event is partisan or not, allowing for the development of computational tools without the need for manual annotation. This work sheds light on how news media content is produced and shaped, contributing to a deeper understanding of media bias's impact on society.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WCgh5hMyg4", "reviewer": "Reviewer_9YKN", "summary": "", "questions": "Are there larger-scale studies which you will perform to use the stance predictor, or is the goal purely technological in a machine learning sense?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks reviwer#9YKN for your constructive evaluation and comments.\n\n**Q1.** We acknowledge that a larger dataset with diverse topics would be beneficial. However, we also want to point out that the annotation is costly as it requires annotators with political domain background, and the annotation is done at the fine-grained event level. Compared to the existing informational bias dataset BASIL, our dataset contains **828 partisan events**, whereas theirs contains **1,249 informational bias spans**. However, all of our annotated partisan events are used for the **evaluation purpose**, whereas they use the informational spans for both **supervised training and evaluation**, resulting in a smaller test set of **62 sentences**.\n\nAs you've noticed that we leverage other datasets for a meaningful evaluation, indeed we also report the ideology prediction performance on AllSides and BASIL (Table 4). Our hypothesis is that if the model can correctly detect partisan events, which by definition, will improve ideology prediction, which is validated by our experiments. We will leave the creation of a larger dataset for the future work.\n\n**Q2.** We acknowledge that the performance variance of multi-article models is small. However, our focus in this paper is not to develop a SOTA algorithm that outperforms existing methods, though our best model does end up with superior performance compared to baselines. Our goal is  to study the partisan events in news articles. We **verify their impacts** on the article's ideology, and we explore methods to detect partisan events **without supervised training** on large-scale datasets. The experiment results also verify that **cross-article event comparison** is important for partisan event detection and ideology prediction.\n\n**Q3 and additional questions.** Thanks for your interest in the use case of our model (i.e., stance analyzer). We would love to bring to your attention that the focus of this work is not only improving article-level ideology/stance prediction, but also providing rationales to the predicted stance so that readers would know **where the stance arises from**. For example, as presented in Fig 1, one may easily tell an article written by New York Post is right-leaning, while our analyzer further points readers to the specific right-leaning event (i.e., claims Biden’s statement as false) that emphasizes pro-gun rights.\n\nOn top of that, our stance analyzer can be **employed in multiple settings**: (1) **Article-level ideology/stance prediction**, which is explored in this work (section 4). (2) **Topic-specific stances clustering** where we can investigate how different media express stances on the same topics by highlighting the inclusion or omission of partisan events. (3) **Entity-level stance analysis**: for instance, in the context of a left-leaning entity slashing a right-leaning entity, an existing dataset [1] only reveals a stance label between the two entities with no further rationale provided to the users. In contrast, our analyzer can take an annotated stance as input and then identify the partisan event that plays a role in influencing the reader's interpretation.\n\n[1] Generative Entity-to-Entity Stance Detection with Knowledge Graph Augmentation. (Zhang et al., EMNLP 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n(1) The idea of models for stance identification using the presence and absence of events is appealing, and apparently understudied.\n(2) Thorough evaluation of a range of different models for interpreting partisan events.\n(3) Carefully and clearly written, a solid piece of work.", "reasons_to_reject": "(1) The level of annotation (50 articles on only two topics) appears too small for a meaningful evaluation, although they leverage other existing data sets to get a meaninful evaluation\n\n(2) Generally speaking, the performance of all the variants of the multiarticle models were pretty close, even though one model did win on all of the datasets.\n\n(3) There was nothing \"done\" with the resulting stance analyzer other than validate its performance.   ideally one would build such an analyzer to conduct a study of news sources more than seek better F1 scores.", "questions_to_authors": "Are there larger-scale studies which you will perform to use the stance predictor, or is the goal purely technological in a machine learning sense?", "paper_topic_and_main_contributions": "This paper presents an approach to identifying the partisan stance of news articles through an analysis of partisan events.   The key observation is that events which are missing in an article (or located at the bottom of an article) can be highly predictive of stance given a notion of which events are more likely to appear in liberal or conservative sources.   The authors develop a series of models of increasing sophistication to exploit detections of events.\n\nThe main contributions are\n\n(1) Introducing the use of partisan event detection for media ideology detection\n(2) Comparing article level event analysis vs. contextualizing them through analysis of other articles on the topic from other sources.\n(3) Preparing a modest annotated corpus on partisan events for evaluation and future studies.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is very clearly and carefully written."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Quality: The paper presents an interesting and relevant task. The authors have conducted experiments to back up their claims, however the experiments are lacking details in the current version of the work. Clarifications that were provided in the rebuttals should be added in to the final version of the paper. 

Clarity: The paper is grammatically sound and well written. However, some reviewers suggested better organization to make the task definition more clear and easier to follow. 

Originality: The paper proposes partisan event detection in news articles, which can be seen as a novel task since most works focus on sentence or document level ideology classification. Additionally, a joint latent model is introduced for solving this task. All the reviewers liked the idea and motivation of the work. 

Significance: Computational political science researchers in NLP will be interested in this work. Detection of media bias is also incredibly important in today's society. However, the lack of experimental details, low inter-annotator agreement on a small dataset (~800 events, 2 political issues), marginal improvement over a minimal baseline, and limited discussion of the application of this work towards understanding framing and bias in the media are considerable drawbacks. 

Pros: (1) Proposing an important societal task of partisan event detection, which can be seen as novel compared to typical sentence-level (or higher) ideology detection. 
(2) A latent model for event detection, which can also be leveraged in an unsupervised setting. 
(3) A new dataset of 50 articles on 2 political issues, with annotation provided for 828 events.

Cons: (1) Underspecified experimental settings which led to reviewer confusion about the task and evaluation. While this was clarified to the reviewers' (bwkA & MVtQ) satisfaction in rebuttal, this missing information is critical and should be included in the final version of the paper. 
(2) The weak performance of the model and lack of further application is a concern to the reviewers. The random baseline performance is quite low and the latent model has marginal improvement. 
(3) The dataset is very small. Of 50 articles, 3035 events were extracted. Of these, 828 events were labeled and used in this work. Also, the reported inter-annotator agreement of 0.43 (and 52.83 in rebuttal) is too low. 
(4) Lack of discussion on the link between ideology, agenda setting, and framing bias; specifically more explanation is needed for how event selection bias contributes to the analysis of media bias. Inclusion of rebuttal points to Reviewer MVtQ could strengthen this area. (5) In the author's rebuttals to Reviewer MVtQ & 9YKN: Please note that the Broockman and Kalla term partisan coverage filtering is essentially framing (here, at the event level). Also, the proposed task of topic-specific stance clustering is again framing of events. The meta-reviewer recommends further reading into the interplay of political ideology, framing, and media bias to strengthen the contributions of this work. 

Overall, the reviewers really like the idea and motivation of the proposed task. Inclusion of the details that were provided in the author's rebuttals would greatly strengthen the paper.
Decision: Accept-Main