Disentangling Structure and Style: Political Bias Detection in News by Inducing Document Hierarchy
Authors: Jiwoo Hong
Yejin Cho
Jiyoung Han
Jaemin Jung
James Thorne
Venue: EMNLP 2023 Findings (2023)
Keywords: Natural Language Processing
NLP Applications
Political Bias
Journalism
TL;DR: We propose a new method to detect political bias in news articles by capturing sentence-level semantics and document-level structure.

Abstract: We address an important gap in detecting political bias in news articles. Previous works that perform document classification can be influenced by the writing style of each news outlet, leading to overfitting and limited generalizability. Our approach overcomes this limitation by considering both the sentence-level semantics and the document-level rhetorical structure, resulting in a more robust and style-agnostic approach to detecting political bias in news articles. We introduce a novel multi-head hierarchical attention model that effectively encodes the structure of long documents through a diverse ensemble of attention heads. While journalism follows a formalized rhetorical structure, the writing style may vary by news outlet. We demonstrate that our method overcomes this domain dependency and outperforms previous approaches for robustness and accuracy. Further analysis and human evaluation demonstrate the ability of our model to capture common discourse structures in journalism.

Disentangling Structure and Style:
Political Bias Detection in News by Inducing Document Hierarchy
Jiwoo Hong', Yejin Cho', Jaemin Jung". Jiyoung Han?, and James Thorne'
KAIST AI
"KAIST Moon Soul Graduate School of Future Strategy (jiwoo_hong, yejin_cho, nettong, jiyoung. han, thorne)@kais ac.
Abstract
We address an important gap in detecting polit- ical bias in news articles. Previous works that perform document classification can be influ- enced by the writing style of each news outlet, leading to overfiting and limited generalizabil- ity. Our approach overcomes this limitation by considering both the sentence-level seman- ties and the document-level rhetorical structure, resulting in a more robust and style-agnostic approach to detecting political bias in news arti- cles. We introduce novel multi-head hierarchi- cal attention model that effectively encodes the structure of long documents through a diverse ensemble of attention heads. While journal- ism follows formalized rhetorical structure, the writing style may vary by news outlet. We demonstrate that our method overcomes this do- main dependency and oulperforms previous ap- proaches for robustness and accuracy. Further analysis and human evaluation demonstrate the ability of our model to capture common dis- course structures in journalism."
Bob Burr- "A" & For "Amustyy And "Amuoiza" (8y)
Head RD
Thooughot tisoyy ochinann cislliatton, wherever there
population. thee ware bocinn Stong bunkes homelands/g.. I[ee immigation bill does not make it ensi: for 2Ndnng innignass
good neightsss alss,
Sun lave to wotk and puose the Amnctiiin dreans miso oppoeusit wab fi. lliuall imnigration once and for Aml. Like eloitial cunest, immigaaion tolons the patt As dasgenaes and eapensise ilqually enaring Unise Stadee people sinple becasse of buressatti
issill may be the caics mode completino huilt
wetth the Se neimsally hooal caueryy
sptama... We alse
inssiaaaiion procese
(inclading workkfince
Our model identifies main sentences in an arti- cle (yellow) and the supporting sentences (blue) through the use of hierarchical multi-head attention based on their utility for the document-level classification task.
Figure 1:
Iraq war (Luther and Miller, 2005), and climate change (Feldman et al., 2012). This disparity in media coverage has significant impacts on shap- ing people's 's perceptions of the issue (Levendusky, 2013) and their voting behavior (Della Vigna and Kaplan, 2007). Even the COVID-19 pandemic, global health crisis, has been covered differently across the conservative and liberal political spec- trum (Motta et al., 2020). Consequently, Ameri- cans have displayed a deep partisan divide, with Republicans showing less concern about personal COVID-19 risks and the severity of the pandemic than Democrats (Allcott et al., 2020). This led to Republicans being less willing to adhere stay-at- home orders and engage in social distancing (Clin- ton et al., 2021), resulting in higher COVID-19 mortality rates among this group (Bendix, 2022).
1 Introduction
One of the primary reasons people consume news is for social cognition: to stay informed about events and developments and make informed decisions. To fulfill this purpose, news outlets must provide citizens with information from diverse sources and perspectives. The Pew Research Center (Mitchell et al., 2018) reports that the majority of Ameri- can respondents (78%) indicated that news orga- nizations should refrain from showing favoritism towards any political party in their reporting, How- ever, more than half of the respondents (52%) ex- pressed dissatisfaction with the media's ability to report on political issues fairly and unbiasedly. Extensive research has revealed that partisan bias exists in various social issues, such as the 2016 US presidential election (Benkler et al., 2018), the
The role of news media in shaping public dis- course and perceptions of social issues cannot be overstated. The media plays crucial role in dis- seminating information, framing issues, and setting the agenda for public debate. ultimately leading public policy-making. Mapping the political land- scape of news media is an important task. It helps news consumers evaluate the credibility of the news
'Our code is available at: httpss//giite thwbb com/ Tractlab/emmlp222 Docra/-DDDnnttt-eeee -Hiurrnht

---

they are exposed to and more easily interpret and contextualize the information at hand. Moreover, news consumers who are aware of news outlets' political leanings can seek out other viewpoints to balance their understanding of an issue. This is par- ticularly important in a socicty where media outlets are increasingly polarized along parties (Jurkowitz and Mitchell, 2020; Mitchell and Jurkowitz, 2021). It is more crucial than ever for news consumers be discerning and critical in their news media consumption. However, previous works mainly fo- cus on a document-level classification, making challenging to assess why an article is biased.
Hahn, 2009). However, recent web tracking tech- nologies have revealed that significant amount of traffic crosses ideological lines (Dubois and Blank, 2018; Flaxman et al., 2016; Gentzkow and Shapiro, 2011) This conflicting evidence suggests the audi- ence makeup of a news outlet may no longer be stable in highly competitive media environment. Content- methods typically rely on stu- dent coders (Feldman et al., 2012; Luther and Miller, 2005) or workers from crowdsourcing plat- forms (Budak et al., 2016) to measure progres- sive/conservative media bias. The introduction of computer-assisted content analysis techniques allows researchers to compare the linguistic pat- terns adopted by partisan media on large scale (Gentzkow and Shapiro, 2010; Luther and Miller, 2005). Nonetheless, these methods are generally confined to specific news topics, such as the fed- eral tax on inherited assets (Gentzkow and Shapiro, 2010), and may not be generalizable to other news topics or entire news channels. Recently, facial recognition algorithms have been used to quantify the visibility of politicians in broadcast news pro- grams (Kim et al., 2022). This technique detects whether a program displays liberal or conserva- tive bias if it features more actors from the left (or right) for a longer duration. However, it is re- stricted to analyzing news visuals only and cannot be applied to other types of news content.
To validate the necessity of discourse structural analysis in this task, we analyze the drawbacks of political bias classifiers that disregard the dis- course structure by questioning their credibility. We formulate this issue as a domain dependency problem: instability of the model to which news articles model is trained or tested on. And we pro- pose an approach for biased context detection in news articles which uses multi-head attention mechanism to propagate document-level labels to prominent subsets of sentences, which helps make the model more reliable and explainable, illustrated in Figure 1 and Appendix G.
Our paper offers four contributions: (1) we ad- dress the problem of domain dependencies in the political bias detection task, which is essential but understudied task; (2) we propose a new ap- proach for biased context detection news articles based on multi-head attention and a hierarchical model; (3) we evaluate the effectiveness of under- standing the special discourse structure in the jour- nalism domain in comparison to the syntactic hier- archy; and (4) we analyze the structures our model captures with respect to journalistic writing styles.
Structural Properties The writing style in the journalism domain plays crucial role in shaping public discourse and perceptions of social issues. Sentences in the articles have unique roles accord- ing to their position in the article (Van Dijk, 1985), typically following an inverted pyramid structure (Pöttker, 2003). Structural analysis for detecting political bias in news articles (Van Dijk, 2009; Gan- gula et al., 2019) has shown that analysis of how in- formation is presented can be used to determine the political bias of the author or news outlet. There- fore, specialized structural analysis of journalism is important in developing NLP-based approaches for political bias detection.
2 Background
Measuring bias There have been continuous ef- forts to determine the political bias of news out- lets, with two main approaches: audience- -ased and content-based analyses. Audience-based meth- ods assume that a news outlet's political stance can be inferred from the political preferences of its primary audience (Gramlich, 2020). This is because news outlets are expected to cater to the inclinations of their users to retain their viewership or readership. Studies have shown that partisans tend to choose news from politically congruent outlets (Davis and Dunaway, 2016; Iyengar and
3 Related Works
Detecting political ideologies and biases There are two common approaches in the content-based political bias detection task in NLP: either article- level (Kulkarni et al., 2018; Baly et al., 2020; Liu et al., 2022), sentence-level (Chen et al., 2020; Fan et al., 2019; Spinde et al., 2021; Lei et al.,

---

Mult-HHad Article Hieraday Building (V different Conteat Closers)
27 Prodictions Final Prodiction
Semantic Analysis
Sentemce Defcction
Context Closter Embedding
L
Tille (5c)
DERT
Muliti Maliti-Hodd Hood Arteatoo
Sentusee
Depenienny Scoue:
Sentence (8))
BERT
1
BtL STM
Cluster
Scatence (5.)
BERT
Closalfier
Sentemer a (S,)
BERT
Suggening
Figure 2: The model architecture. The 'Sentence Type Detection' and 'Context Cluster Embedding" sections are held independently by heads. The final prediction is an average of cach head's bias prediction.
2022). For sentence-level bias detection, previous work has centered around labeling additional data: Spinde et al. (2021) developed a process for la- beling data at a sentence-level with binary labels and Fan et al. (2019) defined informational bias and lexical bias while providing sentence-level and word-level bias labels for news articles. With these datasets, Lei et al. (2022) used discourse structures that inform the model about the role of sentence detect sentence-level bias.
structure trees on fake news detection tasks. Also, for political ideology classification, Kulkarni al. (2018) proposed a multi-view model that hierar- chically encodes the article with word-level and sentence-level embeddings. Devatine et al. (2022) also hierarchically encoded the article and applied ad adversarial adaptation for political bias classifica- tion. While hierarchical features have been used for document-level classification, propose prop- agating relevance through the document hierarchy.
Research in article-level bias centers around us- ing external information (Baly et al., 2020; Zhang et al., 2022; Feng al., 2022). For example, Baly al. (2020) incorporate additional external infor- mation such as Twitter bios and Wikipedia pages related to each news outlet, and Zhang et (2022) and Feng et al. (2022) implemented a knowledge graph of external facts to enhance the political bias detection performance. In related works, con- trastive learning with additional data preprocessing (e.g. curating article triplets) has also helped build robust models learning from annotations of left, centrist, and right viewpoints on the same event (Kim and Johnson, 2022; Liu et al., 2022).
Relevance Propagation Relevance propagation methods are widely used in many domains to im- prove the explainability of black box models (Nam et al., 2019; Binder et al., 2016; Bach et al., 2015). Specifically for NLP, Arras et al. (2017) propagate model weights to highlight relevant tokens for pre- dictions. Our method widens the range of relevance propagation in NLP by considering sentence-level relevance in document understanding.
Article-level political bias datasets For article- level bias detection, many previous works pub- lished labeled datasets with news articles collected from Allsides. com (Kiesel et al., 2019; Baly et al., 2020; Chen et al., 2020: Liu et al., 2022). Models will likely memorize aspects irrelevant political bias when predicting the label. Publisher (Kiesel et al., 2019), news outlet (Baly et al., 2020), and topic (Chen et al., 2020) have shown confound- ing effects. While Baly et al. (2020) reported this discrepancy between seen news outlets for training and unseen news outlets during testing, they did not analyze the cause of this problem.
Hierarchical attention Hierarchical attention (Yang et al., 2016) allows model to encode longer texts by combining independently encoded sen- tences or phrases (Shen et al., 2020; Zhang et al., 2020; Han et al., 2022; Kulkarni al., 2018; Karimi and Tang, 2019; Devatine et al., 2022). In the legal domain, Shen al. (2020) hierarchically formulated legal events and arguments for legal event extraction. In the medical domain, Zhang et al. (2020) used hierarchical attention in medi- cal ontology to embed medical terminologies with respect to both low- and high-level concepts.
4 Model Architecture
We have three modeling objectives: (1) induce sentence-level information from a document-level label, (2) to understand the discourse structure the news article, and (3) to reduce the effect
For journalism, Karimi and Tang (2019) uti- lized hierarchical attention for building discourse

---

spurious correlation between the media outlet and the bias label. It critical to model the complex relation between sentences the news article when predicting its political ideology, to this end, we apply hierarchical modeling to propagate document information to sentences.
multi-head attention mechanism of the transformer, we do concatenate the results. We instead per- form the sentence type detection independently for each of the N attention heads to propagate the polit- ical bias distribution to the independently captured N different main contexts.
We propose a bias prediction model that uses two-layer tree with sentence-level embeddings to better understand the news article's s semanties and discourse structure. Our pipeline consists of three key stages that can be jointly trained as single unified model: semantic analysis, sentence type detection, and context clustering. Each component is described in detail in the following sections, and the model architecture is illustrated in Figure 2.
4.2 Multi-Head Sentence Type Detection
This step is computed independently for each atten- tion head HІ. For simplicity, we omit the subscript defining the head k. We use the attended repre- sentation to predict the importance of each of the sentences in the article with respect to the headline. Each sentence is assigned one of two roles: main or supporting. We compute these roles by compar- ing dot product similarity between the sentences h and the headline hg. Supporting sentences are assigned
4.1 Semantic Understanding
The first stage of the model builds representation vector for each sentence comprising the semantics, positional information, and discourse relation.
= maii (S:
exp h ho
Semantics A news article contains headline So and a sequence of many sentences: (S)..... S,)). The semantics of each sentence is independently captured by using a large language model to gener- ate a sentence embedding S; = (S)) {0. {O....,,.) n}. We use S-BERT (Reimers and Gurevych, 2019), as it is trained specifically to rep- resent the semantic similarity between sentences.
Pinain (St]So) ai E exp h ho
(2)
We created weighted embeddings of sentences so that the sentences which are likely to be main sentences have high norms. These weighted em- beddings undergo further encoding with a sin- gle feed-forward layer with GELU unit: u, = Gelu FFN (agh:)). Similarly, we make encod- ings for sentences from the SUPPORTING perspec- tive:
Position The positional information captures the role of sentence with respect to its location in the article. We chose to add the positional informa- tion to the sentence embedding through BiLSTM (Hochreiter and Schmidhuber, 1997) over encoded sentences. As the sentences in the news articles have intricate relationships more than a simple se- quential relationship, chose to use BiLSTM instead of positional encoding as desire condi- tional representation that cannot be achieved with positional encoding alone (Wang et al., 2019).
Gelu(F (FFN((1 a;)h:)). Note that the headline acts main sentence with on 1.
4.3 Context Cluster Embedding
The generated perspective vectors are used for each attention head to predict the hierarchical relation between the main and supporting sentences. We use the dot product similarity to capture this dis- course dependency. The dependency scorer (Fig- ure 2) returns the proportion of focus between the main sentence S; and the supporting sentence Sy:
h: LLST M (s., 10) LST smatt)) (1)
Discourse To capture the discourse relation between multiple sentences in different view- points, we apply multi-head attention (Vaswani et al., 2017) over the BiLSTM encodings: I [bo...., hn]. Each attention head, H₃ (QW kW,(k) vW(")) independently models the relationships between all document sen- tences and headline with scaled dot product atten-
exp v
Paep(S
i i
(3)
Eブ= exp v
Context cluster embeddings are created by sum- ming the main sentence representation u, (m))) with the weighted sum of the supporting sentence repre- sentations. We weight this sum by the dependency score so that unrelated sentences contribute less to the context cluster embedding.
tion. We set Q = K V H. In contrast to the

---

Dataset
Media-based Split (Baly et al., 2020)
Augmented Media-based Split (Ours)
Type
Train
Valid
Test
Train
Test Set I
Test Set 2
Count
Data
Outlets
Data
Outlets
Data
Media
Data
Outlets
Data
Outlets
Data
Outlets
Left
8,861
61
1,640
14
402
7
7,300
16
200
240
4
Center
7,488
29
618
14
299
4
7,300
10
200
4
240
4
Right
10,241
69
13
599
18
7,300
200
240
4
Table I: Statistics for the two datasets. The augmented version is split after merging every data in the original dataset. The news outlets in each set are mutually exclusive and randomly selected. (c.g. Daily Kos (left), CNN (left), BBC News (center), Wall Street Journal (center), Fox News (right), National Review (right))
news outlets for each class to make our test sets. As the number of articles varied by news outlets, we selected 50 and 60 articles from each news outlet for Test Set I and 2, respectively, We sampled 7,300 articles from each side regardless of the news outlets to preserve the train data size for the train set. Dataset statistics and the list news outlets for each set are provided Table and Appendix
R Σ
C2
U4
(dep(S)1S.) vy J=
(4)
For each attention head, we create a single em- bedding by summing each of the context cluster embeddings с Layer Niverroon Norm c.). We apply LayerNorm (Ba et al., 2016) to mitigate the effect of having multiple depending sentences. We then predict the distribution of bias label for the news article distribution with linear classifier over this head's embedding: Yx = De//mmarFFFNN During training, we treat each attention head as component in mixture model, averaging the class probabilities to predict the bias: y = > At test time, we propagate the document-level label to single main sentence and corresponding sup- porting sentences (i.e. context clusters) for each head according arg max P Pmain (S:
5.2 Baseline
To compare against hierarchical model, we also fine-tuned BERT (Devlin al., 2019) model using the HuggingFace bert-base-uncased implemen- tation (Wolf et al., 2019). We use a single classifica- tion layer over the CLS token from the final hidden layer. Since news articles are lengthy documents, the articles were truncated by 512 tokens.
Ei= ýk-
Experiments
5 Experimental Settings
We compare model against BERT baseline classifier for the news article classification task with experiments to (I) compare our model against the BERT baseline, which is representative of previ- ous work: (2) compare the accuracy of the model on two disjoint test sets (3) evaluate sensitivity to the content of training data and (4) evaluate sensitivity to the number of training data. In all experiments, report AUROC and macro Fi scores. The hy- perparameters for training are in Appendix A.
5.1 Datasets
We train and evaluate the model on the media bias detection dataset from Baly et (2020). Unlike Baly et al. (2020), we build two test sets to test the performance gap upon two test sets rather than pur- suing the higher accuracy in a single test set. We use the media-based split to evaluate the general- ization between news outlets to which the model is trained and tested with two modifications:?
Comparison against baseline We compare against BERT classifier, which performs on par with the model from (Baly al., 2020). For both the baseline and model, we train 20 versions of the model with different random seeds. We report the average and standard deviation of the results for each model, respectively.
First, we deleted 425 news outlets with less than 50 articles ensure that the model is not memo- rizing the writing style of certain mediums. For the remaining mediums, we merged news outlets from the same company, such as "CNN" and "CNN (Web News)" to prevent overlapping between the train set and the others.
Second, made balanced dataset with respect both data size and news outlets. We selected four
Sensitivity to test data We evaluate whether the models depend on the news articles they are tested on. We use the two test sets, which were pairwise disjoint by a news outlet (i.e. a news outlet from
BBecaus the dataset reported in the paper significantly dif- fers from the published files, we re-bat balance the data ourselves.

---

BERT
Ours
Ratio of Var. (F-test)
AUROC
Macro F
AUROC
Macro Fi
Test Set I
0.5267 (0.0410) 0.3633 (0.0341)
0.6017 (0.0190) 0.4227 (0.0185)
4.6697***
1,000
Test Set 2
0.6297 (0.0216)
0.4639 (0.0189)
0.5703 (0.0298)
0.3971 (0.0250)
0.5249(ns)
JSD (t-test)
0.0377
0.0289
5,000
Test Set 1
0.5378 (0.0231)
0.3831 (0.0310)
0.6446 (0.0154) 0.4706 (0.0172)
2.2289*
Test Set 2
0.6557 (0.0250)
0.5042 (0.0219)
0.6507 (0.0150)
0.4805 (0.0143)
2.7930
JSD (t-test)
0.0313
0.0163(ns)
10,000
Test Set 1
0.5743 (0.0244) 0.4123 (0.0215)
0.6529 (0.0081)
0.4747 (0.0090)
9.1247***
Test Set 2
0.6588 (0.0173)
0.5079 (0.0127)
0.6783 (0.0094)
0.5089 (0.0122)
3,3934
JSD (t-test)
0.0165***
0.0099***
Full
Test Set 1
0,5903 (0.0240) 0,4165 (0.0146)
0.6548 (0.0106)
0.4751 (0.0111)
5.1633***
Test Set 2 JSD (t-test)
0.6659 (0.0134) 0.5128 (0.0107)
0.6905 (0.0072)
0.5247 (0.0082)
3.4710**
0.0145***
0.0097***
Table 2: Evaluation results for BERT and ours. The first and second rows of each data size refer the sample mean and the sample standard deviation (in the brackets) of 20 trials. The 'JSD (I-test)' row shows the ISD and the significance of the t-test. And the 'Ratio of Var. (F-test)' column shows the ratio of variance (BERT over ours) and the significance of the F-test. *, 44, *** refers to .5,, .0,,.01..00 .001 respectively, and (ns) refers to p .05.
in Test Set does not have any articles in Test Set 2). If the model is robust to the writing style of the news outlets in evaluation, the distribution of the tested result will be invariant between test sets. We apply a two-sided t-test over AUROC and also report divergence (ISD) between the distributions.
indicate that our approach (1) outperforms a con- ventional LM-based classifier, (2) is resilient to domain shift between train and test (3) uncovers structural properties which follow the general prac- tice and theoretical background in journalism.
7.1 Comparison against baseline
Sensitivity training data We study if the mod- els depend on the news articles they are trained on. We compute the learning curve by train- ing the model on increasing subsets of data from (N=1000,5000,, 10000,5000,1000000 Full., For each sample size, we report the variance based on training on 20 dif- ferent subsets of the training data. We first apply the Shapiro-Will Wilk test (Shapiro and Wilk, 1965) to validate whether the AUROC distribution of each model on the test set is normal. Then, we con- ducted the F-test and checked the variance ratio to validate which model was more sensitive to the training data. We report JSD to quantify the dis- crepancy between the results in the two test sets for both the BERT baseline and our model.
To compare the data efficiency of the BERT base- line and our model, we evaluate model accuracy with varying numbers of training data in Table 2. For Test Set 1, our model outperformed BERT for both AUROC and macro F1 score in every data size setting. For Test Set 2, our model outperformed BERT in the subsets of 10,000 articles and the full train set but not in the subsets of 1,000 articles and 5,000 articles. BERT outperformed when the models were trained to the small subsets and tested on Test Set 2. Otherwise, our model outperformed BERT in any setting. The highest AUROC of our model on both test sets was 0.6733 and 0.7071, 1, out- performing BERT (AUROC of 0.6384 and 0.6977). We further explored the reason why BERT out- performed in two specific cases with the model's 's sensitivity to test data in the following subsection.
Learning Curve We trained the model to the random subsets of 1,000, 5,000, and 10,000 arti- cles and the full train set (21,900 articles) for the experiments studying sensitivity to training data.
7.2 Sensitivity to test data
To measure the discrepancy of each model's results from Test Sets and 2, we used the two-sided test and JSD. The null hypothesis and alternative hypothesis for the t-test are listed in Appendix B.I. Except for our model trained with the subsets of
7 Result
We report Macro F1 and AUROC for the document- level bias prediction task in Table Our results

---

5,000 articles, it was evident reject Ho. This means that both models have discrepancies in AU- ROC from Test Set 1 and Test Set However, BERT always had a higher discrepancy measured by JSD. This indicated that BERT always showed distant results in Test Set and Test Set 2 in ev- ery case. This provides evidence that our model is more robust to test data than BERT (further re- ported in Figure 3 and Table 2).
Lexical bias refers the bias from the word choice of the journalist, such as using polarized words (e.g. Donald Trump is investing more in conspiracy theories about President Obama's 's birth certificate as he explores his bid for the presidency.) Infor- mational bias refers to the biased elaboration of certain events or facts, which includes using selec- tive quotations to strengthen their viewpoint. (c.g. The Arizona group said the call from Mr. Trump on Wednesday came unexpectedly, and the group had spent much of the day Thursday scurrying to make travel arrangements to New York.)
Teat Set
Teat Set
600
8.1 Main Sentences Extractive Summary
We conducted machine and human evaluations assess whether the main sentences identified by the model were informative. Our working assumption is that the concatenation of main sentences from our model acts as an extractive summary.
6.1
Figure 3: The learning curve of the models. Normality is assumed according to Shapiro- Wilk test (a = 0.05).
Machine Evaluation We used BART-large (Lewis et al., 2019) pretrained on CNN/Daily Mail dataset (Nallapati et al., 2016) to generate the sum- maries. We report BERTScore (Zhang al., 2019) between abstractive summarizer and our model an extractive summarizer in Table
7.3 Sensitivity to training data To compare the variance of AUROC, we used the F-test. The null hypothesis Ho. alternative hypoth- esis H1 and the ratio of variance fo listed in Appendix B.2. For both models, the AUROC vari- ance in Test Set and Test Set 2 tends to decrease as the subset size increases. In all but I case, the variance of BERT was consistently higher than our model. Evaluating on Test Set 1, fo was always higher than 10.05,19.19, so it was evident to reject Ho in every case. Evaluating Test Set 2, So linearly increased as the size of the subset increased. How- ever, for 1,000 training data, we could not reject Hp. This shows that our model is more invariant to the data was trained on than the BERT classifier.
Precision
Recall
Fl Score
vS BART-Large
Lex. Only
0.5741
0.5784
0,5761
Info. Only
0.7922
0.8144
0,8021
Biased Sent.
0.8253
0.8641
0,8440
Our Model
0.8739
0.8828
0.8781
Table BERTScore between BART-Large and ours. 'Lex. Only' and 'Info. Only.' refers to the set of lexically and informationally biased sentences in the news article. And 'Biased Sent.' refers the union of 'Lex. Only' and 'Inf, Only' sentences.
8 Structural Analysis
In this section, we analyze the structural properties news articles using the main sentences identified by the model. To do so, we collect the predicted main sentences and assess if they capture the for- malized discourse structures commonly used in journalism. By validating from summarization and structure, we ensure the reliability of using multiple attention heads as an explanation mechanism.
Human Evaluation To evaluate whether the main sentences selected by the model were infor- mative, two annotators (K = 0.113 recorded pref- erence of summary from either 1) the lead, 2) ran- domly selected sentences, 3) main sentences from our model. Without knowing the system, annota- tors were instructed to select which summary best explains the concept and biased contents of twenty news articles sampled from BASIL. Annotators se- lect our model's main sentences in 75% of cases. Detailed instructions are provided in Appendix D.
We use the BASIL dataset, which contains sentence-level annotations for two types of biases (Fan et al., 2019): lexical and informational bias.

---

1.5
density
1.0
0.0
0.2
Sentence Location
Cluster
Straight Uh) Striight News (Beyand)
Interpretive News
Figure 4: Density plot of the selected main sentences by clusters. The x-axis denotes the relative location of sentences, 0.0 and 1.0 indicating the first and last sentences, respectively, The rug plot below each density plot shows the actual location of the selected main sentence.
8.2 Structure of documents
8.2.1 Cluster Straight News with the Inverted Pyramid Structure
Our classifier identified that the main sentences cluster were located in the first quarter of the article, indicating the use of the inverted pyramid structure (Missouri Group, 2013). In this structure, the lead paragraph contains the core information of the news story, while subsequent paragraphs are ranged in decreasing order of importance (Van Dijk, 1985). By presenting the essential details the be- ginning, readers can rapidly comprehend the main points of the article, resulting in shorter article (Pöttker, 2003). These characteristics of straight news using the inverted pyramid structure are re- flected in the lower article length and fewer biased sentences, which is shown in Table 4.
To assess whether the main sentences identified by our model capture the structural properties of news articles, we cluster articles from the BASIL dataset with respect their distribution of contexts using Dynamic Time Warping (Tormene al., 2008, DTW), which captures the temporal elements without a depen- dency on the document length. The BASIL dataset consists of 100 political news stories, each com- prising three articles about the same main events, sourced from the New York Times, Fox News, and the Huffington Post. Articles containing less than 200 words or exceeding 1,000 words and op-eds excluded from the dataset.
K-Mcans. For the distance metrie,
8.2.2 Cluster 2 - Straight News Beyond the Inverted Pyramid
Our analysis of the BASIL dataset, including po- litical news longer than 200 words, identified three distinct clusters that differ in narrative structure. Clusters and are characterized straight news, while cluster is classified interpretive news. The main sentence locations and statistics are listed in Table and visualized Figure Sample arti- cles are provided in Appendix F, Furthermore, we validate the clustering results in Appendix E.
Cluster comprises news articles that are of similar length to those in cluster and account for 4.3% of the whole dataset. Unlike cluster 1, which follows the inverted pyramid structure, the articles in clus- ter present the main information at the beginning and the article's first and second quarters. This structure still employs the inverted pyramid nar- rative but incorporates distinct feature bridge sentences. These sentences connect anecdotes examples to the news story's 's broader theme and help tie seemingly unrelated information together, leading toward cohesive narrative thread. Interest- ingly, articles cluster exhibit the least amount of lexical and informational bias.
Size (%)
Avg. Words
Lex.
Info.
92.0
627.38
1.39
3.91
4.3
739.92
1.15
3.54
3.7
1107.00
2.27
6.73
Table 4: Statistics of clusters. Lex. and Info., refer to the average number of sentences with lexical and informational biases per article annotated in BASIL.
8.2.3 Cluster Interpretive News
Cluster 3 is distinguished by main sentences that are typically located the first and third quarters

---

the article, with the latter sentences often reflecting the reporter's interpretation of the event (Van Dijk, 1985). This type of reporting aligns with the trend toward Interpretive news, which seeks to uncover the meaning of news beyond the facts and state- ments of sources (Schudson, 1982; Patterson, 1994; Salgado and Strömbäck, 2012). As a result, inter- pretive news tends to be longer than straight news and often includes analysis and commentary along- side the reporting of events (Barnhurst and Mutz, 1997). Our findings are consistent with the obser- vations made by Chen et al. (2020), who found that any political bias in news stories, if present, tends to appear in the later part of the article.
Program(KAIST), Artificial Intelligence Industrial Convergence Cluster Development project funded by the Ministry of Science and ICT(MSIT, Korea) & Gwangju Metropolitan City, and by the National Research Foundation of Korea (NRF) grant funded by MSIT (Grant No. RS- RS-2023-00252535)
References
Hunt Allcott, Levi Boxell, Jacob Conway, Matthew Gentzkow, Michael Thaler, and David Yang. 2020. Polarization and public health: Partisan differences in social distancing during the coronavirus pandemic. Journal of Public Economics, 191:104254.
Leila Arras, Franziska Horn, Grégoire Montavon, Klaus- Robert Müller, and Wojciech Samek. 2017. "what is relevant in a text document?"; An interpretable ma- chine learning approach. PloS one, 12(8):e01 81 142.
9 Conclusions
Our multi-head attention model leverages sentence- level information to capture the narrative structure. Our model outperforms conventional document- level classifiers by mitigating the domain depen- dency constraints of traditional classifiers and gen- erating more robust and precise document-level representations. We validated our model's effec- tiveness in capturing journalism's 's rhetorical struc- tures and writing styles.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hin- ton. 2016. Layer normalization.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wo- jeiech Samek. 2015. On pixel-wise explanations for non- linear classifier decisions by ayer-wise rele- vance propagation. PLOS ONE, 10(7): 1-46.
Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020. We can detect your bias: Predicting the political idcology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNLP), pages 4982 4991, Online. Association for Computa- tional Linguisties.
Limitations
While news plays a vital rolc in every country and language, our method only applies to English news articles. Although the dataset contains articles from some international news outlets (e.g. Al Jazeera), most news outlets are from Western countries. As political bias in news articles closely relates to cul- tural background, we would like to expand our work to journalism in non- non-Engee -onglghlihh languages or based in other non-western contexts.
Kevin G Barnhurst and Diana Mutz. 1997. American journalism and the decline in event-centered report- ing, Journal of Communication, 47(4)227-53
Aria Bendix. 2022. Covid death rates are higher among republicans than democrats, mounting evi- dence shows. NBC News.
Our work is limited by the types of biases cap- tured in the dataset. There are many approaches to bias, and the datasets available only reflect limited range of discretized political bias. Future work from the community could focus on introduc- ing better datasets and resources for modeling the many dimensions and nuances of bias.
Yochai Benkler, Robert Faris, and Hal Roberts. 2018 Network propaganda: Manipulation, disinformation, and radicalization in American politics. Oxford Uni- versity Press.
Alexander Binder, Grégoire Montavon, Sebastian Bach, Klaus-Robert Müller, and Wojciech Samek. 2016. Laser-wise relevance propagation for neural net- works with local renormalization layers. CORR. abs/1604.0088
Acknowledgements
Jiyoung Han and James Thome are the correspond- ing authors. This work was supported by the Insti- tute of Information and Communications Technol- ogy Planning& & Evaluation (ITTP) grant funded by the Korean government (MSIT; Grant No. 2019- 0-00075, Artificial Intelligence Graduate School
Ceren Budak, Sharad Goel, and Justin M. Rao. 2016. Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis. Public Opinion Quarterly, 800$1)2200-211
Wei-Fan Chen, Khalid Al Khatib, Henning Wachsmuth, and Benno Stein. 2020. Analyzing political bias

---

and unfairness in news articles at different levels of granularity. In Proceedings of the Fourth Workshop on Natural Longuage Processing and Computational Social Science, pages 149-154, Online. Association for Computational Linguistics.
Seth Flaxman, Sharad Goel, and Justin M. Rao. 2016. Filter Bubbles, Echo Chambers, and On- line News Consumption. Public Opinion Quarterly, 80281):
Rama Rohit Reddy Gangula, Suma Reddy Duggenpudi, and Radhika Mamidi. 2019. Detecting political bias in news articles using headline attention. In Pro- ceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Nenworks for NLP, pages 77-84, Florence, Italy. Association for Com- putational Linguistics.
J. Clinton, J. Cohen, J. Lapinski, and M. Trussler. 2021. Partisan pandemic: How partisanship and public health concerns affect individuals&#*x219; social mobility during covid-19 19. Science Advances, 7(2)eabd7204.
Nicholas T. Davis and Johanna L. Dunaway. 2016. Party Polarization, Media Choice, and Mass Partisan- Ideological Sorting. Public Opinion Quarterly. 80($1))272-297.
Matthew Gentzkow and Jesse M. Shapiro. 2010. What drives media slant? evidence from u.s. daily newspa- pers. Econometrica,, 78(1)331-71
Matthew Gentzkow and Jesse M. Shapiro. 2011. Ideo- logical Segregation Online and Offline The Quar- terly Journal of Economicx,
Stefano DellaVigna and Ethan Kaplan, 2007. The Fox News Effect: Media Bias and Voting*. The Quarterly Journal of Economicx, 122(3):1187) 183-
John Gramlich. Q&a: How pew research center eval- uated americans' trust in 30 news sources [online]. 2020.
Nicolas Devatine, Philippee Muller, and Chloé Braud. 2022. Predicting political orientation in news with latent discourse structure to improve bias understand- ing. In Proceedings of the 3rd Workshop Com- putational Approaches to Discourse. pages 77-85, Gyeongju, Republic of Korea and Online. Interna- tional Conference on Computational Linguisties.
Sooji Han, Rui Mao, and Erik Cambria. 2022. Hierar- chical attention network for explainable depression detection on Twitter aided by metaphor concept map- pings. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 94-104, Gyeongju, Republic of Korea. International Commit- tee on Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-tern memory. Neural computation, 988::1335- 1780.
Shanto Iyengar and Kyu S Hahn. 2009. Red Media, Blue Media: Evidence of Ideological Selectivity in Media Use. Journal of Communication, 59(1):19-39,
Elizabeth Dubois and Grant Blank. 2018 The echo
Mark Jurkowitz and Amy Mitchell. About one-fifth of democrats and republicans get political news in a kind of media bubble [online]. 2020.
chamber is overstated: the moderating effect of po- litical interest and diverse media. Information, Com- munication & Society, 2155)2229-745
Hamid Karimi and Jiliang Tang. 2019. Learning hier- archical discourse structure for fake news de- tection. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 3432-3442, Minneapolis, Minnesota. Association for Computational Linguistics.
Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Pra- fulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019. In plain sight: Media bias through the lens of factual reporting. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Lnguage Processing (EMNLP-IJCNILP), pages 6343-6344 Hong Kong, China. Association for Computational Linguistics.
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Em- manuel Vincent, Payam Adineh, David Comey, Benno Stein, and Martin Potthast. 2019. SemEval- 2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 829 839, Minneapolis. Minnesota, USA. Association for Computational Lin- guistics.
Lauren Feldman, Edward W. Maibach, Connie Roser- Renouf, and Anthony Leiserowitz. 2012. Climate on cable: The nature and impact of global warming coverage on fox news, enn, and msnbe. The Interna- tional Journal of Press/Politics, 17(1)3-31..
Shangbin Feng, Zilong Chen, Wenqian Zhang, Qingyao Li, Qinghua Zheng, Xiaojun Chang, and Minnan Luo. 2022. Kgap: Knowledge graph augmented political perspective detection in news media.
Eunji Kim, Yphtach Lelkes, and Joshua McCrain. 2022. Measuring dynamic media bias. Pro- ceedings of the National Academy of Sciences. 119622)e22021191199

---

Michelle YoungJin Kim and Kristen Marie Johnson. 2022. CLoSE: Contrastive learning of subframe em- beddings for political bias classification of news me- dia. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 2780- 2793, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016. Abstrac- tive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Lan- guage Learning, pages 280-290, Berlin, Germany. Association for Computational Linguisties.
Vivek Kulkarni, Junting Ye, Steve Skiena, and William Yang Wang. 2018. Multi-view models for political ideology detection of news articles. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3518- 3527. Brussels, Belgium. Association for Computa- tional Linguisties.
Woo-Jeoung Nam, Jaesik Choi, and Scong- Whan Lec. 2019. Relative attributing propagation: Interpreting the comparative contributions of individual units in deep neural networks. CoRR,
T.E. Patterson. 1994. Out of Order: An incisive and boldly original critique of the news media's 's domina- tion of America's political process. Politics: Media studies. Knopf Doubleday Publishing Group.
Yuanyuan Lei, Ruihong Huang. Lu Wang, and Nick Beauchamp. 2022. Sentence-level media bias analy- sis informed by discourse structures. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10000, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Horst Pöttker. 2003. News and its communicative qual- ity: the inverted pyramid when and why did it ap- pear? Journalism Studies, 4(4):501-51 -111.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLA) pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Matthew S. Levendusky. 2013. Why do partisan media polarize viewers? American Journal of Political Science, 57(3)6611-623.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. BART: denoising sequence-to-sequencce pre-training for natural language generation, translation, and com- prehension. CoRR, abs/1910.13466.
Peter J. Rousseeuw, 1987, Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53-65. 65.
Yujian Liu, Xinliang Frederick Zhang, David Wegsman, Nicholas Beauchamp, and Lu Wang. 2022. POL.I TICS: Pretraining with same-story article comparison for ideology prediction and stance detection. In Find- ings of the Association for Computational Linguis- tics: NAACL 2022, pages 1354-1374, Seattle, United States. Association for Computational Linguistics.
Susana Sal Salgado and Jesper Strömbäck. 2012. Interpre- tive journalism: A review of concepts, operationaliza- tions and key findings. Journalism, 13(2): 161.
Michael Schudson. 1982. The politics of narrative form: The emergence of news conventions in print and tele- vision. Daedalus, 1 111(4):9--12.
Catherine A. Luther and M. Mark Miller. 2005. Fram- ing of the 2003 u.s.-iraq war demonstrations: An analysis of news and partisan texts. Journalism & Mass Commmicattin Quarterly, 82(1):78-96.
S. S. Shapiro and M. B. Wilk. 1965. An analysis of variance test for normality (complete samples). Biometrika, 52(3-4):591-611
Shirong Shen, Guilin Qi, Zhen Li, Sheng Bi, and Lusheng Wang. 2020. Hierarchical Chinese le- gal event extraction via pedal attention mechanism. In Proceedings of the 28th International Confer- ence Computational Linguistics, pages 100-113, 13, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Missouri Group. 2013. News Reporting and Writing. Bedford/St. Martin's. ss
Amy Mitchell and Mark Jurkowitz. How americans navigated the news in 2020: A tumultuous year in review [online]. 2021.
Amy Mitchell, Katie Simmons, Katerina Eva Matsa, and Laura Silver. Publics globally want unbiased news coverage, but are divided on whether their news media deliver [online]. 2018.
Leslie Smith. 2015. No more pesky learning rate guessing games. CoRR, abs/ 186.
Timo Spinde, Manuel Plank, Jan- Krieger, Terry Ruas, Bela Gipp, and Akiko Aizawa. 2021. Neu- ral media bias detection using distant supervision with BABE bias annotations by experts. Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages I 1166-1177, Punta Cana, Dominican Republic, Association for Computational Linguistics.
Matt Motta, Dominik Stecula, and Christina Farhart. 2020. How right-leaning media coverage of covid- 19 facilitated the spread of misinformation in the early stages of the pandemic in the u.s. Canadian Journal of Political Science/Revue canadienne de science politique, 53(2):335-342.

---

Paolo Tormene, Toni Giorgino, Silvana Quaglini, and Mario Stefanelli. 2008. Matching incomplete time series with dynamic time warping: An algorithm and an application to post-stroke rehabilitation. Artificial Intelligence in Medicine, 45(1):11-38
Teun A Van Dijk. 1985. Structures of news in the press. Discourse and communication: New approaches to the analysis of mass media discourse and communi- cation, 10:69.
Teun A Van Dijk. 2009. News, discourse, and ideology.
In The handbook of journalism studies, pages 2112 1- 224. Routledge.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Chenguang Wang. Mu Li, and Alexander J. Smola. 2019.
Language models with transformers.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's ss transformers: State-of-the-ar natural language processing. CoRR, abs/1910.03771.
Zichao Yang. Diyi Yang, Chris Dyer, Xiaodong He. Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter the Association for Computa- tional Linguistics: Human Language Technologies, pages 1480-1489, San Diego, California. Associa- tion for Computational Linguistics.
Muhan Zhang, Christopher King, Michael S. Avi- dan, and Yixin Chen. 2020. Hierarchical attention propagation for healthcare representation learning. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with BERT. CoRR, abs/1904.09675.
Wenqian Zhang. Shangbin Feng, Zilong Chen, Zhenyu Lei, Jundong Li, and Minnan Luo. 2022. KCD: Knowledge walks and textual cues enhanced political perspective detection news media. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 4129 4140, Seattle, United States. Association for Computational Linguistics.

---

Appendix
This appendix contains the following contents: (1) Hyperparameters used for training the baseline models and our models (A); (2) Notations for hypothesis test in Section 7.2 and Section 7.3 (B); (3) List of news outlets in each data set of augmented Media-based Split in Table 1 (C); (4) Human evaluation template and details used in Section 8.1 (D); (5) Sample news articles for each cluster in Section 8 (F).
A Hyperparameters
For our model, we used AdamW as an optimizer with a weight decay of le-05 for both models. Both models were trained for 25 epochs. For BERT, we used a constant learning rate of 2e-05. For our model, we used a two-layered BiLSTM with size of 512. There parameters of the S-BERT sentence encoder were frozen and not updated during training. The number of attention head was fixed to 8. And we used cycle learning rate scheduler (Smith, 2015) with maximum learning rate of 5e-05. The maximum learning rate was reached after 10 percent of the training was finished. We used single Nvidia RTX A6000 train the model: BERT took 9 minutes, and our model took 22 minutes per epoch for training with the full data. Hyperparameters were selected optimizing our model for the original dataset from (Baly et al., 2020) and then applied to our augmented datasets without modification.
Hypothesis Tests
This section introduces the null hypothesis Ho, alternative hypothesis H1, and ratio of variance fo used in Section 7.2 and Section 7.3 compare the robustness of baseline and model.
B.1 Sensitivity to test data
Ho psetl MSet2
(5)
H1 1Set + 1l
B.2 Sensitivity to training data
Hn o2 BERT Qurs
2
(6)
H1 0 BERT
0 Oure
22
fo BERT
(7)
22
Ours

---

C
List of News Outlets in Dataset
Train
Associated Press, Chicago Sun Times, Christian Science Monitor, BBC News, Time Magazine, Washington Times, New York Magazine, CBS News, Fox News, USA TODAY, American Spectator, Democracy Now, New York Times, MarkerWatch, Reuters, Mother Jones, Pew Research Center, NPR Online News, Daily Beast, ProPublica Vanity Fair, CNN, BuzzFeed News, The Guardian, The Flip Side, Vox, Slate, Politico, Breithart News, Reason, Wall Street Journal, New York Post, Newsmax
Test Set 1
The Atlantic, Yahoo! The 360, Bloomberg, Media Matters, HotAir, Salon, The Daily Caller, Victor Hanson, The Week, FiveThirtyEight, ABC News, CBN
Test Set 2
Townhall, National Review, The Hill, Business Insider, TheBlaze.com, Daily Kos, Axios, Washington Post, The Daily Wire, Vice, CNBC, Al Jazeera
Table The list of news outlets contained in the train set, Test Set 1, and Test Set the augmented media-based split dataset. The color of each news outlet is blue, black, and red if they are left-sided, centered, right-sided respectively. (ProPublica contains both left-sided and centered news articles.)
Human Evaluation Template
We hired two annotators who are not experts in journalism to select the option that best follows the instructions given in Table 6, The three options were given to the annotator in randomized order: 1) the lead of the news article, 2) set of randomly selected sentences, 3) the main sentences selected by our model. For the set of randomly selected articles, we first calculated the mean and variance of the main sentences selected by our model. Then we sampled twenty random numbers from the normal distribution with calculated mean and variance, which is used as a number of sentences to be randomly selected. After both annotations were finished, the authors measured Cohen's 's k and Fi scores after counting the common and uncommon annotations.
Instruction
These three different sets of sentences extracted from the article. Which of the options best explains 1) the main contents 2) the biased context of the news article?
Article ID
Article #1
Option (I)
Lead of Article #1
Option (2) Set of Sentences Randomly Selected from Article #1
Option (3)
Main Sentences that Our Model Selected from Article #1
Table 6: The example template of instruction and question given to the evaluators.
E
Validity of Clustering Results
We report the Silhouette score (Rousseeuw, 1987) and the additional results with changing the number of clusters. Also, we analyze the validity of Cluster in Section 8.
1. Silhouette Score: With the three clusters reported in Table 4, the Silhouette score was 0.8988. And the score consistently decreased as we increase the number of clusters.
2. Increasing Number of Clusters: By clustering the news articles into three, four, and five clusters, the biggest cluster had 92% of items in K 3, 91.33% in K = 4, and 88% in 5.
3. Clustering within Cluster By re-clustering the news articles Cluster I into minimum two to maximum five clusters, the biggest cluster had 97.6% of items in 2, 90.9% in K 3, 87.3%
K = 4, and 86.2% in = 5.

---

F
Sample News Articles from Each Cluster
We sampled two articles from each cluster and show their contents. The main sentences identified by the model are italicized bold sentences. The news outlet is mentioned in the bracket next to the headline. We report BERTScore (Zhang et al., 2019) between the set of selected main sentences and a summary from a BART model (Lewis et al., 2019) pre-trained on the CNN/Daily Mail dataset (Nallapati et al., 2016).
F.1
Cluster 1 Inverted Pyramid Structure
Headline
Texas Rep. Blake Farenthold resigns from Congress (Fox News)
U.S. Representative Blake Farenthold resigned from Congress on Friday, following multiple allegations of sexual harassment, misconduct and inappropriate behavior. Farenthold, R-Texas, had previously announced that he would not seek re-election in the 2018 midterm election, when reports of sexual misconduct first surfaced. "While planned on serving out the remainder of my term in Congress. know in my heart it's Farenthold said in statement Friday evening, noting he sent a letter Texas Gov. Greg Abbott to tell him about his resignation, effective at p.m. "It's been an honor and privilege to serve the constituents Texas' 27th Congressional District. would like to thank my staff both Washington and Texas for all of their hard work behalf of constituents. I would also like to thank my family for their unwavering support and most importantly the people that elected me," Farenthold said. "Leaving my service the House, I'm able look back the entirety of my career in public office and say that it was well worthwhile." Farenthold was sued by his former aide, Lauren Greene, in 201 4, alleging hostile work environment, gender discrimination and retaliation. Among other things, Greene claimed Farenthold asked her for threesome. She sued him and was paid $84,000 from a public fund behalf of Farenthold for sexual harassment claim. Farenthold pledged to repay the $84,000 in taxpayer money spent to settle the claim. Farenthold's 's former communications director, Michael Rekola, also described detail the congressman's alleged abusive behavior toward staff members, which ranged from making sexually graphic jokes to verbally abusing his aides. Other staffers accused Farenthold of routinely commenting on the size of women's breasts and making jokes about being on "redhead patrol" because he was attracted women with red hair. BERTScore 0.9337
Body
to move along and look new ways to serve." "
Table 7: Example of an inverted pyramid structure article from cluster 1.

---

Headline
Jesse Jackson Jr. Pleads Guilty in Campaign Money Case (New York Times)
Body
WASHINGTON - Jesse L. Jackson Jr., the former Democratic representative from Illinois, pleaded guilty on Wednesday one felony fraud count in connection with his use of $750,000 in campaign money to pay for living expenses and buy items like stuffed animals, elk heads and fur capes. As part of a plea agreement, prosecutors recommended that Mr. Jackson receive sentence of 46 to 57 months in prison. The federal judge oversecing the case, Robert L. Wilkins, is scheduled to sentence Mr. Jackson on June 28. "For years lived off my campaign." Mr. Jackson, 47, said in response to questions from the judge about the plea. "I used money I shouldn't 't have used for personal purposes." At one point during the hearing, the judge stopped his questioning of Mr. Jackson, who was crying, so that he could he given tissue. "Guilty, Your Honor - misled the American people." Mr. Jackson said when asked whether he would accept the plea deal. Mr. Jackson's father, the Rev. Jesse L. Jackson, his mother and several brothers and sisters accompanied him to the hearing. Mr. Jackson's 's wife, Sandi, also accompanied him, and later in the day she pleaded guilty to a charge that she filed false income tax statements during the time that Mr. Jackson was dipping into his campaign treasury. Prosecutors said they would seek to have her sentenced to 18 to 24 months. Mr. Jackson's 's plea was yel another chapter in the downward spiral of his career. Elected to Congress in 1995 at the age of 30 from district that includes part of the South Side of Chicago, Mr. Jackson was once one of the most prominent young black politicians in the country, working on issues related to health care and education for the poor. But as the federal authorities investigated Gov. Rod R. Blagojevich of Illinois over his efforts to sell the Senate seat that President Obama vacated in 2008, they uncovered evidence that one of Mr. Jackson's 's friends had offered to make a contribution to Mr. Blagojevich's 's campaign in exchange for the seat. Since then, Mr. Jackson, who has said he had no knowledge of the offer, has been dogged by questions about his ethics. Last summer, Mr. Jackson took a medical leave from Congress and was later treated for bipolar disorder. After winning re-election in November, he resigned, citing his health and the federal investigation into his use of campaign money. After the hearing, Mr. Jackson's 's lawyer, Reid H. Weingarten, said his client had "come to terms with his misconduct." Mr. Weingarten said that Mr. Jackson had serious health issues that "directly related" to his conduct. "That's not an excuse, it's just a fact," Mr. Weingarten said. Court papers released by federal prosecutors on Wednesday provided new details about how Mr. Jackson and his wife used the $750,000 in campaign money to finance their lavish lifestyle. From 2007 to 2011 1. Mr. Jackson bought $10,977.74 74 worth of televisions, DVD players and DVDs at Best Buy, according to the documents. In 2008 Mr. Jackson used the money for things like a $466.30 dinner at CityZen in the Mandarin Oriental in Washington and a $5,587.7 .75 vacation al the Martha's 's Vineyard Holistic Retreat, the document said, On at least two instances, Mr. Jackson and his wife used campaign money at Build-A-Bcar Workshop, store where patrons can create stuffed animals. From December 2007 through December 2008, the Jacksons spent $3 $3.8.89 on "stuffed animals and accessories for stuffed animals" from Build-A BBaar,, according to the documents. One of the more exotie items they bought was an elk head from taxidermist in Montana. According the documents. Mr. Jackson arranged in March 2011 to have $7,000 paid to the taxidermist, with much of the money coming from a campaign account, and was shipped a month later to Mr. Jackson's Congressional office. A year later, Mr. Jackson's wife, knowing that the elk head had been bought with campaign money, had it moved from Washington to Chicago, and she asked a Congressional staff member to sell it, the documents say. In August 2012, the staff member sold the elk head for $5,300 to an interior designer and had the money wired to one of Mr. Jackson's 's accounts. What the staff member did not know was that the interior designer was actually an undercover F.B. I. employee who was investigating the Jacksons, the documents say. Documents relcased on Friday showed how Mr. Jackson used his campaign money buy items like fur capes, celebrity memorabilia and expensive furniture. Among those items were a $5,000 football signed by American presidents and two hats that once belonged to Michael Jackson, including a $4.600 fedora.
BERTScore 0.9195
Table 8: Example of an inverted pyramid structure article from cluster 1.

---

F.2
Cluster 2 Straight Reporting
Headline
Republicans challenge Clinton claims on budget cuts, Benghazi cable (Fox News)
Body
Republicans are challenging host of statements made by Secretary of State Hillary Clinton and Democratic allies during Wednesday's 's heated Libya testimony claiming that complaints about a lack of funding are bogus and questioning the secretary's 's insistence she never saw urgent cables warning about the danger of attack. The questions come as the Senate Foreign Relations Committee begins its confirmation hearing for Sen. John Kerry, D-Mass., who was tapped to replace Clinton at the department. One issue that may come up is the department's funding. Assertions that State Department posts are left vulnerable because Congress has decided not to fully fund security requests pervaded Wednesday's hearings. "Shame on the House for failing to adequately fund the administration's 's request, Rep. Gregory Meeks, D-N.Y., said Democratic New York Rep. Eliot Engel repeatedly said Congress had "slashed" diplomatic security requests. Clinton, in turn, affirmed their claims, saying budget issues are a "bipartisan problem." Budget numbers, though, actually show the overall diplomatic security budget has ballooned over the past decade. Democrats point to modest decreases in funding in recent years, and the fact that Congress has approved less than was requested. But Congress often scales back the administration' 's requests, and not just for the State Department. And the complaints tend to overlook the fact that the overall security budget has more than doubled since fiscal 2004, For that year, the budget was $640 million. It steadily climbed to $1.6 6.6 billion in fiscal 2010. It dipped to $1.5 billion the following year and roughly $1.35 billion in fiscal 2012 - still more than it was decade ago. Slightly more has been requested for fiscal 2013. It's difficult to tell how much was specifically allocated for Benghazi. Tripoli was the only post mentioned in the department's ' fiscal 2013 request - -qquest-fudding for that location did slip, from $11.5 million in fiscal 2011 to $10.1 million the following year. Slightly more has been requested for fiscal 2013. Still, then-Deputy Assistant Secretary for Diplomatic Security Charlene Lamb testified in October that the size of the attack - and not the money - was the issue. Asked if there was any budget consideration that led her not to increase the security force, she said: "No." " She added: "This was an unprecedented attack in size." Asked again about budget issues, Lamb said: "Sir, if it's a volatile situation, we will move assets to cover that." " Asked Wednesday about Lamb's 's testimony, Clinton noted that the review board that examined the Libya attack found budget issues have played role. "That's why you have independent group like an (Accountabilityy Review Board); that's why it was created to look at everything." Clinton said. But Rep. Dana Rohrabacher, R- R-Calif., said "any suggestion that this is a budget issue is off base, or political." " Other lawmakers further complained that the State Department has spent millions on lower-priority projects that could have been spent on security. Another pivotal issue Wednesday dealt with an Aug. 16 cable. That cable summarized an emergency meeting the day before by the U.S. Mission in Benghazi and warned the consulate could not defend against a "ccordinatedd attack." " That cable is seen as one of the vital warnings sent out of Libya in the months leading up to the attack. But, to the dismay of lawmakers, Clinton repeatedly said she never saw it. "That cable did not come to my attention. have made it very clear that the security cables did not come to my atlention or above the assistant secretary level." Clinton said. "I'm not aware of anyone within my office, within the secretary's ' office, having seen the cable." Rep. Michael McCaul, R-Texas, said "somebody within your office should have seen this cable." " "An emergency meeting was held and cable sent out on Aug. 16 by the ambassador himself, warning of what could happen. And this meant this cable went unnoticed by your office. That's the bottom line." he said. Clinton said it was "very disappointing" that "inadequacies" were found in the "responsiveness of our team here in Washington," and said "it's something we're 're fixing and intend to put into place protocols and systems to make sure it doesn't 't happen again." The secretary tried to explain that "1.43 million cables" come through the department every year. They are addressed her but in many cases do not go to her. Rather, they go through "the bureaucracy." " Republicans argue the Aug. 16 cable was rather high priority. As Sen. Rand Paul, Ky., put it, "Libya has to have been one of the hottest of hot spots around the world." " He claimed that not knowing about their security requests "really, I think, cost these people their lives." "Had I been president at the time, and I found that you did not read the cables from Benghazi, you did not read the cables from Ambassador Stevens, would have relieved you of your post. think it's inexcusable." " Paul said.
BERTScore 0.8849
Table 9: Example of straight reporting article from cluster 2.

---

Headline 2020 Candidates Demand Full and Immediate Release of Mueller Report (New York Times)
Body
CHARLESTON, S.C. Democratic presidential candidates wasted no time Friday evening demanding the immediate public release of the long-awaited report from Robert S. Mueller III, with several saying that Americans deserved to know any findings about President Trump, Russia and the 2016 election in order to form judgments about Mr. Trump and the 2020 race. Former Representative Beto O'Rourke, campaigning in South Carolina Friday night, told reporters that "those facts, that truth, needs to be laid out for all Americans to be able to make informed decisions going forward, whether at the ballot box" or their discussions with their senators and representatives. Mr. O'Rourke, asked if he supported impeaching Mr. Trump, said he believed the president and his 2016 campaign "at least sought to collude with the Russian government to undermine our democracy" and that Mr. Trump "sought to obstruct justice" once in office. think those grounds enough for members of the House to bring up the issue of impeachment." he said. "But whether they do or not, this will ultimately be decided by the American public at the ballot box in South Carolina and in every state of the union." " No voters brought up the Mueller report to Mr. O' O'Rourke he spoke to them Friday. Indeed, in the hours after the announcement of the report, Democratic candidates reacted over Twitter or in remarks al events rather than in back-and-forth conversations with voters. Several candidates, calling for the swift release of the report, also sought to gather new supporters and their email addresses by putting out "petitions" calling for complete transparency from the Justice Department. "The Trump Administration shouldn't 't get to lock up Robert Mueller's 's report and throw away the key," Senator Cory Booker argued on Twitter, asking people to sign a petition and provide their names and emails. Such information is often used for future fund-raising solicitations. Within hours of Mr. Mueller's completing his investigation, Senator Elizabeth Warren and the campaign arm of House Democrats were already placing ads Facebook demanding the full report's 's release, with the Democratic Congressional Campaign Committee seeking 100,000 signatures for its online petition. With no detailed information available about the report, Ms. Warren and Mr. Booker as well Senators Kirsten Gillibrand and Kamala Harris sought focus attention and pressure on how quickly Attorney General William P. Barr would release the full report. "Attorney General Barr - release the Mueller report to the American public. Now," Ms. Warren wrote on Twitter. Ms. Gillibrand made similar demands and also retwected the news of the report along with three words: "See you Sunday." That is when Ms. Gillibrand plans to formally kick off her 2020 campaign in front of Trump International Tower in New York. Ms. Harris, in addition to calling for the report to be released "immediately." called on Mr. Barr to "publicly testify under oath about the investigation and its findings." Ms. Harris, Ms. Warren and Ms. Gillibrand also joined Mr. Booker and Senator Bernie Sanders in asking supporters to sign their petitions calling for the report's immediate release. Four other candidates Senators Amy Klobuchar of Minnesota; Gov. Jay Inslee of Washington; former Representative John Delaney; and Julián Castro - also called for the release of the full report. "As Donald Trump said, 'Let it come out," Mr. Sanders wrote on Twitter. "I call on the Trump administration to make Special Counsel Mueller's 's full report public as soon as possible. No one, including the president, is above the law."
from
BERTScore 0.8585
Table 10: Example of straight reporting article from cluster 2.

---

F33
Cluster 3 Interpretive Reporting
Headline
Supreme Court Questions Claims in Sex Bias Suit Against Wal-Mart (Fox News) pending class action lawsuit against Wal-Mart that would be the largest of its kind in U.S. history may soon be dismissed, considering the tenor of oral arguments before the Supreme Court Tuesday. Although she may ultimately side with the plaintiffs, even Justice Ruth Bader Ginsburg, an ardent defender of women's 's rights, expressed some concerns about the particulars of the sex discrimination lawsuit that covers 1.5 million women and could cost the world's largest retailer billions of dollars. The key vote for Wal-Mart victory could belong to Justice Anthony Kennedy, who said he was troubled by what he called inconsistency in the women's 's lawsuit. "Number one, you said this is culture where Arkansas knows, the headquarters knows, everything that's going on." Kennedy told lawyer Joseph Sellers who represents the women. "Then in the next breath, you say, well, now these (local) supervisors have too much discretion." " The lawsuit alleges that the company's 's corporate culture, described in court Tuesday as the "Wal-Mart Way," fosters the advancement of male workers over their femalc counterparts. It also claims that despite a company policy expressly prohibiting discrimination, local store managers are given too much flexibility in determining salary hikes and job promotions that invariably favor men over women. This dual argument that confounded Justice Kennedy also drew the ire of Justice Antonin Scalia who said he was whipsawed by the claims. "If somebody tells you how exercise discretion, you don't have discretion." he said. Chief Justice John Roberts also offered his doubts about the merits of lawsuit, suggesting that any discriminatory acts at Wal-Mart are no worse than anywhere else. "Is it true that Wal-Mart's pay disparity across the company was less than the national average?" he asked. Sellers said that wasn't a fair comparison because Wal-Man nart has an obligation under federal law to make sure its managers do not discriminate. The case started a decade ago when Wal-Mart worker Betty Dukes said the management at her Pittsburg, Calif., store bypassing her for promotions. "I could see the men going forth and the women in the store stayed in the basic positions they were always in," Dukes once told an interviewer. Her diserimination claims were folded into a class action lawsuit covering all current female Wal-Mart employees and any who worked for the company going back to late 1998. Two lower courts said the case could move toward trial. Wal-Mart's 's appeal is asking the Supreme Court to stop the case from ever getting to trial judge. On Tuesday, Dukes walked out of the courthouse full of confidence and poise saying she feels no anger toward her bosses or anyone else. "Wal-Mart may be a big company and that is no doubt. But they are not big enough where they can't he challenged in a court of law. you do wrong, then you should be held accountable. From the least of us to the greatest of us." " Dukes's 's case has drawn the attention of the larger business community who fear that if the justices allow the case to proceed, it will open the doors to more class action lawsuits. The U.S. Chamber of Commerce and major corporations including Bank of America, General Electric and Microsoft submitted briefss in the case supporting Wal-Mart Mart. Much of the hour-long argument delved into the tedious details of class action law and if the Wal-Mart women could properly certify their claims into single case. Wal-Mart lawyer Theodore Boutrous argued that every member of the class couldn't 't possibly meet a standard of commonality to justify the lawsuit. "Our expert's 's report and testimony showed that at 90 percent of the stores, there was no pay disparity," Boutrous told the court. "And that's the kind of and even putting that aside, the plaintiff's needed to come forward with something that showed that there was this miraculous recurrence at every decision across every store of stereotyping, and the evidence simply doesn't t show that." Another technical concern that appears to work against the women covers the different types of remedies they are seeking. In addition to the punitive damages they want injunction that would force Wal-Mart to adopt more stringent anti-discrimination policies. But those two remedies require different standards for class certification and Justice Ruth Bader Ginsburg said was "a very serious problem" in the case to try and sue for punitive damages after only obtaining certification under the lower threshold required for the injunction. It's possible that instead of an outright victory for Wal- Mart, the justices could issue split decision of sorts and allow only the injunction part of the lawsuit to move forward. That potential ruling would get Wal-Mart off the hook for any financial damages.
Body
BERTScore 0.8650
Table II: Example of interpretive reporting article from cluster 3.

---

This sample article is divided into three pages.
Headline
Trump Declares a National Emergency, and Provokes Constitutional Clash (New York Times) WASHINGTON - President Trump declared national emergency the border with Mexico on Friday in order to access billions of dollars that Congress refused to give him to build a wall there, transforming a highly charged policy dispute into a confrontation over the separation of powers outlined in the Constitution. Trying regain momentum after losing grinding two-month battle with lawmakers over funding the wall, Mr. Trump asserted that the flow of drugs, criminals and illegal immigrants from Mexico constituted a profound threat national security that justified unilateral action. "We're re going to confront the national security crisis on our southern border, and we're going to do it one way or the other," he said in a televised statement in the Rose Garden barely 13 hours after Congress passed spending measure without the money he had sought. "It's an invasion," he added. "We have an invasion of drugs and criminals coming into our country." But with illegal border crossings already down and critics accusing him of manufacturing a crisis, he may have undercut his own argument that the border situation was so urgent that it required emergency action. "I didn't 't need to do this, but I'd rather do it much faster," he said. "I just want to get it done faster, that's 's all." The president's decision incited instant condemnation from Democrats, who called it an unconstitutional abuse of his authority and vowed to try to overturn with the support of Republicans who also objected to the move. "This is plainly a power grab by disappointed president, who has gone outside the bounds of the law to try to get what he failed to achieve in the constitutional legislative process." Speaker Nancy Pelosi of California and Senator Chuck Schumer of New York, the Democratic leader, said in a joint statement. Mr. Trump's 's announcement came during freewheeling, 50-minute appearance which he ping-ponged from topic to topic, touching the economy, China trade talks and his coming summit meeting with North Korea's leader, Kim Jong-un. un. The president again suggested that he should win the Nobel Peace Prize, and he reviewed which conservative commentators had been supportive of him, while dismissing Ann Coulter, who has not Sounding alternately defensive and aggrieved, Mr. Trump explained his failure to secure wall funding during his first two years in office when Republicans controlled both houses of Congress by saying, "I was a little new to the job." He blamed "certain people, a particular one, for not having pushed this faster," clear reference to former Speaker Paul D. Ryan of Wisconsin, a Republican. Mr. Trump's 's assertions were replete with misinformation and, when challenged by reporters, he refused to accept statistics produced by his own government that conflicted with his narrative. "The numbers that you gave are wrong," he told one reporter. "It's a fake question." On point after point, the president insisted that he would be proved correct. "People said, 'Trump is crazy,"" "M said at one point, discussing his outreach to Mr. Kim. "And you know what ended up being? A very good relationship." Mr. Trump acknowledged that his declaration of a national emergency would be litigated in the courts and even predicted a rough road for his side. "Look, expect to be sued," he said, launching into mocking riff about how he anticipated lower court rulings against him. "And we'll 'Ill win in the Supreme Court," he predicted. Indeed, Public Citizen, an advocacy group, filed suit by the end of the day on behalf three Texas landowners whose property might taken for a barrier. California and New York likewise announced that they will sue over what Gov. Gavin Newsom of California called the president's "vanity project," and a roster of other groups lined up do the same. "Fortunately, Donald Trump is not the last word," said Mr. Newsom, a Democrat. "The courts will he the last word." Among those predicting flurry of judicial decisions against Mr. Trump was George T. Conway I11, a conservative lawyer and the husband of Kellyanne Conway, the president's counselor. "If he knows he is going to lose," Mr. Conway, a vocal critic of Mr. Trump, wrote on Twitter, "then he knows he is violating the Constitution and laws he has sworn to uphold." The House Judiciary Committee announced Friday that it would investigate the president's emergency claim, while House Democrats plan to introduce legislation block it. That measure could pass both houses of Congress if it wins the votes of the half-dozen Republican senators who have criticized the declaration, forcing Mr. Trump to issue the first veto of his presidency. The emergency declaration, according White House officials, enables the president to divert $3.6 billion from military construction projects to the wall. (Continued)
Body(Cont.)

---

Body(Cont.)
Mr. Trump will also use more traditional presidential discretion tap $2.5 billion from counternar- cotics programs and $600 million from a Treasury Department asset forfeiture fund, Combined with $1.375 billion authorized for fencing in the spending package passed on Thursday night, Mr. Trump would have about $8 billion in all for barriers, significantly more than the S5.7 billion he unsuccessfully demanded from Congress. The president opted not to tap hurricanc relief money from Texas or Puerto Rico, an idea that had generated angry complaints from Republicans. But he expressed no concern that diverting military construction money would delay projects benefiting the troops like base housing, schools and gyms. "It didn't I sound too important to me," he said. Neither the White House nor the Pentagon had yet identified which projects may be shelved as a result, but Pentagon lawyers and other officials planned to work over the weekend to identify which construction funds would be diverted. The declaration also provided that land may be transferred the Defense Department from other f agencies or from privately purchased or condemned land. The next step would be to secure those lands, where the Pentagon would erect barriers. The declaration gives Patrick Shanahan, the acting defense secretary, broad latitude to earry out this process. Most Americans oppose Mr. Trump's 's emergency declaration, according to polls. One released this week by Fox News found 56 percent against it, including 20 percent of Republicans. Mr. Trump's 's desire for approval by Fox and other conservative news outlets was on display when he identified various pundits supporters, naming Sean Hannity. Laura Ingraham, Tucker Carlson and Rush Limbaugh, although he insisted that "they don't 't decide policy." But Ms. Coulter, who has viscerally attacked Mr. Trump for caving on the wall, has clearly gotten under his skin. "I don't 't know her," he said before quickly correcting himself. "I hardly know her. I haven't 't spoken to her in way over a year." He noted, though, that she was an early predictor of his election victory. "So I like her, but she's 's off the reservation." " he said. "But anybody that knows her understands that." Ms. Coulter fired back shortly afterward. "The only national emergency is that our president is an idiot," she said on KABC radio in Los Angeles. White House officials rejected criticism from across the ideological spectrum that Mr. Trump was creating a precedent that future presidents could use to ignore the will of Congress. Republicans have expressed concern that a Democratic commander in chief could cite Mr. Trump's 's move to declare a national emergency over gun violence or climate change without legislation from Congress. "It actually creates zero precedent," Mick Mulvaney, the acting White House chief of staff, told reporters. "This is authority given to the president in law already. It's not as he just didn't 't get what wanted so he's waving magic wand and taking a bunch of money." " Presidents have declared national emergencies under a 1970s-era law about five dozen times, and 31 those prior emergencies remain active. But most of them dealt with foreign crises and involved freezing property, blocking trade or exports or taking other actions against national adversaries. not redirecting money without explicit congressional authorization. White House officials cited only two times that such emergency declarations were used by presidents to spend money without legislative approval - once by President George Bush in 1990 during the run-up to the Persian Gulf war, and again by his son, President George W. Bush, in 2001 after the terrorist attacks in New York, Washington and Pennsylvania. In both of those cases, the presidents were responding to new events - the Iraqi invasion of Kuwait and Al Qaeda's 's assault on the United States - and were moving military funds around for military purposes. Neither was taking action specifically rejected by Congress. In Mr. Trump's 's case, he is defining a longstanding problem at the border as an emergency even though border apprehensions have actually fallen in recent years, to 400,000 in the last fiscal year from peak of 1.6 million in the 2000 fiscal year. And unlike either of the Bushes, he is taking action after failing to persuade lawmakers to go along with his plans through the regular appropriations process. The spending package passed Thursday by Congress included none of the $5.7 billion that Mr. Trump demanded for 234 miles of steel wall. Instead, it provided $1.375 billion for about 55 miles of fencing. Mr. Trump signed the package into law on Friday anyway to avoid a second govermment shutdown after the impasse over border wall funding closed the doors of many federal agencies for 35 days and left 800,000 workers without pay. For weeks, Republicans led by Senator Mitch McConnell of Kentucky urged Mr. Trump not to declare national emergency, but the president opted to go ahead anyway to find way out of the political corner he had put himself in with the failed effort to force Congress to finance the wall. Mr. McConnell privately told the president that would support the move despite his own reservations, but warned Mr. Trump that he had about two weeks to win over critical Republicans to avoid having Congress vote to reject the declaration. Mr. Trump was among those Republicans who criticized President Barack Ohama for using his executive authority to spare millions of illegal immigrants from deportation after failing to persuade Congress to do (Continued)

---

Body (Cont.)
"Repubs must not allow Pres Obama to subvert the Constitution of the US for his own benefit & because he is unable negotiate w/ Congress," " Mr. Trump tweeted in 2014. But Mr. Trump sought to drive home the personal toll of illegal immigration, inviting to the Rose Garden several relatives country without authorization. Some
of Americans killed by people as "angel moms," stood up holding pictures of loved ones who had died. "Matthew's 's death was preventable and should have heen prevented," one of the women, Maureen Maloney, said in an interview after the event. Her son Matthew Denice, 23, was killed 2011 a motoreycle accident in Massachusetts after colliding with an automobile driven by undocumented immigrant. "He should have never been here in the first place," she said. "If he wasn't It here, it wouldn' I have happened."
relatives, known
BERTScore 0.8576
Table 12: Example of interpretive reporting article from cluster 3.

---

G
Example Model Output
641
Bob Barr "A" Is For "Amnesty" -- -Andd "Amnesia" (Sg)
Head #0
Throughout the history of human civilization, wherever there established cultures and populations, there were borders(Sy) ... Strong borders make good neighbors; also, safe homelands(Sg).. ... If the immigration bill does not make easier law law-abiding immigrants to here work and pursue the American dream, miss opportunity to truly fix illegal immigration once and for at)())).. Like an electrical current, immigration follows the path of least resistance S2z). As dangerous and expensive illegally entering the United States be, it still may be the easiest route for many people, simply because of the bureaucratic complextites built into immigration system(S.)).... We also must ensure manage the immigration process to our cconomic benefit(S...... Our system of visas and work permits fineluding student visas) must reformed so is more rationally based on our counrty's workforce needs(S..)...
21.6%
20.8%
26.7%
Bob Barr- "A" Is For " "Amnesty" And " "Amnesia" (Sp)
Head #4
Throughout the history of human civilization, wherever there were established cultures and populations, there were borders(Sy) ... Strong borders make good neighbors; also, safe homelands(5,)-- ... Meaningful immigration reform must include address why so many people immigrate to country illegally(S.). Only by tackling illegal immigration at the foundation can America's border security be made effective(S,)) Once the balance immigration shifted back toward legal pathways. Congress can address the question of what to do with the millions undocumented peoples within our border(S,,). The first is finanxial incentives "cut line" through the legal immigration process by sneaking into the country(S,,.. If foreigners they can break the law, enter the United States illegally, and still receive public aid, there is no incentive to adhere to the legal process(Sp))....
comprchensive steps
12.7%
11.1%
11.2%
Figure 5: Our model identifies main sentences in article (yellow) and the supporting sentences (blue) through the use of hicrarchical multi-head attention based on their utility for the document-leve level classification task.


Reviews:
[{"review_id": "3gfmTmGXXo", "reviewer": "Reviewer_Nv3C", "summary": "", "questions": "For semantic analysis component (4.1), did you try to use a standard transformer model in place of your \"BiLSTM + Multi-Head Attn\" since the transformer model has positional embedding in nature and implements the attn mechanism? If your concern is the concatenation of multi-heads results, you could manually decompose it.\n\nLine 452: In order to use figure 3 to draw the conclusion that your model is more robust, would it be better to have a another figure in which you could directly show the performance difference between test set 1 and 2, and you could still put your green region and orange region side-by-side. This way, readers will be more easily to visualize the sensitivity to test data of different models.  \n\nIn table 3, what are the meanings of \"Lex only\", \"Info only\", \"biased sentence\". It's strongly suggested to provide the meaning of each baseline either in text or in captions.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we appreciate your detailed reviews and kind suggestions for the overall content of our paper. With additional statistical analysis and experimental results, we provide our response to the ‘Questions For the Authors’ section and the ‘Reasons To Reject’ section sequentially.\n\n**[Q1: Positional Encoding and BiLSTM]**\n\nThank you for posing such an insightful question. The main purpose of using the BiLSTM layer and Multi-Head Attention blocks sequentially was to encode both contextual flow and discourse-level relations. We do agree that the positional encoding in conventional Transformers is made for a similar reason. However, positional encoding only indicates the sequential order of the sentences. While a lot of news articles typically follow an inverse pyramid structure, it does not mean that the sentences given in the news article are sequentially related to each other. For example, the lead of a news article usually contains a high-level description of the topic but the supporting points might stay far from each other. This pattern is a common occurrence in lengthy news articles. Our purpose in adopting BiLSTM was to capture these intricate contextual relationships between sentences that are more complex than simple sequential ordering that positional will encode. This finding can also be supported by established analysis on the limitation of positional encoding (Language Models with Transformers (Wang et al., 2019)).\n\n\n**[Q2: Suggestion for Figure 3]**\n\nThank you for the suggestion, we can work to improve the clarity of the diagram.  The intent of this diagram is to show the comparison in learning curves between BERT and our model. We will make this graph more clearly show that our model has lower variance and higher accuracy compared to BERT. \n\n\n**[Q3: Suggestion for Table 3 Caption]**\n\nThank you for suggesting to make the caption of Table 3 clearer. In the final version of the paper, will change the caption of the table to be more clear. As either lexically or informationally biased sentences tend to contain the writing style of the journalist or the news outlets, we reported the BERTScore between the set of biased sentences and the abstractive summaries generated by BART-Large along with the comparison with our paper. We will reiterate the definitions of ‘lexical bias’ and ‘informational bias’ from Fan et al., 2019 and briefly introduce the insight that we had from the table after modifying the caption.\n\n**[R1: Comparison with Other Models]**\n\nThank you for highlighting the importance of a comprehensive comparison with models other than BERT. We strongly concur that considering models that are specialized for political bias detection, can be helpful for fair comparison. During our research, however, we had difficulty finding relevant models suitable for comparison.  While there are specialized models for political bias detection like POLITICS (Liu et al., 2022) it is impossible to test them in our dataset setting as they require additional information or pre-processing from the plain text of the news articles: external knowledge (Baly et la., 2020, Zhang et al., 2022, Feng et al., 2022), specifically knowledge triplets (Liu et al., 2022, Kim and Johnson et al, 2022). In contrast, our method emphasizes content-based classification without the use of knowledge augmentation, ensuring easy reproducibility and independence from external knowledge sources. We view it as a significant strength of our model and for its distinctness, it is difficult to find models with a similar methodology for a direct comparison.\nThough we couldn't find directly comparable state-of-the-art models for a fair comparison, we conducted additional experiments to overcome the limitation. In follow-up experiments based on your comments we further found that when compared to other language models of different sizes(e.g., GPT-2, Bert-Large, Roberta-Base, Roberta-Large), our methodology consistently ensures robustness.\n\nWe experimented BERT-Large to empirically show that the variance of AUROC and F1 Score should be higher as the number of model’s parameters increase. Also, we experimented GPT2 to empirically show that same phenomena that we focus on also happens in stronger language model with more parameters and more input token limits. The below table shows that GPT2 got almost 10% point of difference in Macro F1 score in both 1,000 and 10,000 instances setting. Also, BERT-Large got more than 7% point difference in Macro F1 score in both 1,000 and 10,000 instances setting. It is notable that bigger and longer model like GPT2 and BERT-Large is still more variant than our model to the training and test sets.\n\n\n|        |            | GPT2                |                     | BERT-LARGE      |                 | Ours                |                     |\n|--------|------------|---------------------|---------------------|-----------------|-----------------|---------------------|---------------------|\n|        |            |        AUROC        |       Macro F1      |      AUROC      | Macro F1        | AUROC               | Macro F1            |\n| 1,000  | Test Set 1 | 0.5050 (0.0260)     | 0.3214 (0.0145)     | 0.5229 (0.0279) | 0.3180 (0.0585) | **0.6017 (0.0190)** | **0.4227 (0.0185)** |\n|        | Test Set 2 | **0.6185 (0.0174)** | **0.4353 (0.0428)** | 0.5858 (0.0532) | 0.3871 (0.0849) | 0.5703 (0.0298)     | 0.3971 (0.0250)     |\n| 10,000 | Test Set 1 | 0.5756 (0.0122)     | 0.3883 (0.0425)     | 0.5596 (0.0158) | 0.3841 (0.0131) | **0.6529 (0.0081)** | **0.4747 (0.0090)** |\n|        | Test Set 2 | 0.6564 (0.0134)     | 0.4833 (0.0269)     | 0.6559 (0.0231) | 0.4912 (0.0183) | **0.6783 (0.0094)** | **0.5089 (0.0122)** |\n\n\n\n**[R2: Statistical Analysis in Section 7.3]** \n\nIn the study \"We Can Detect Your Bias: Predicting the Political Ideology of News Articles\" by Baly et al. (2020), inconsistencies were underscored when classifying political bias in newspaper articles, especially when models are exposed to unfamiliar news outlets. Echoing their findings, our research aimed to create a model that remains consistent in its classifications, irrespective of the outlet's unique writing style. We noticed challenges, marked by fluctuating AUROC scores when testing on test data consists purely of new sources. Our hypothesis posited that our model would demonstrate a tighter standard deviation in AUROC scores compared to the BERT classifier, even when subjected to varied datasets or different training data volumes. ANOVA would have been a good method if we wanted to show that the performance of the model is invariant to the size of the training set. However, we would like to clarify that we intended to share the robustness of the model by comparing BERT and our model with a randomly sampled (by seed) training set in a fixed number of samples. In this manner, we couldn't apply ANOVA to our experiment. We acknowledge that the intention of our experiments lacks understandability, and we will try to explain our intention as clearly as possible.\n\n**[R3: Validity of K-Means Clustering]**\n\nThank you for your compliment on our discourse analysis in Section 8, and we would like to provide further support for Section 8 including the aggregated score. We ran clustering 5 times by changing seeds, and we got the exact same result in 4 cases. In one case, a single data point originally in Cluster 3 was allocated to Cluster 1. As a result, we got the exact same clustering result with the aggregated score. For further justification, we would like to share some statistics to validify our analysis:\n\n1) Silhouette Score: Silhouette score (1~-1) with three clusters was 0.8988, and it linearly decreased as the number of clusters increased.\n\n2) More than Three Clusters: We tried clustering into three to five clusters, and the biggest clusters took 92% (result in our paper, K=3),  91.33% (K=4), and 88% (K=5) of the whole set.\n\n3) Clustering within Cluster 1: We tried re-clustering the news articles in Cluster 1 into two to five clusters, and the biggest clusters took 97.6% (K=2), 90.9% (K=3), 87.3% (K=4), and 86.2% (K=5) of news articles in Cluster 1.\n\n \nWe believe that these results further clarify that the majority of news articles in Cluster 1 share a strong similarity, and Cluster 2 and 3 have distinct features that clearly separate them from the ones in Cluster 1 in terms of the location of the main sentences.\n\n\n**[R4: Clarification in Label Propagation]**\n\nAlong with your kind suggestions in Q2 and Q3, thank you again for suggesting the clarification of notations and writings. We will take steps to address the observations you’ve made. For example, we would be happy to clarify the writing in lines 332 to 335 and the notation in Equation 4 in the final manuscript. We will also remind the reader of this in Section 8 when we perform the analysis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is well written and easy to follow (except for section 4; see my questions below). The author also presents thorough related work. \n\nThe author conducts extensive experiments, and employ several statistical tools to validate their results and findings. Most of their experimental results seem to be statistically sound (see my concern of section 7.3 conclusion in \"Reasons To Reject\").\n\nThe experimental results are insightful. For example, interesting structural analysis are presented in section 8, which uncovers structural properties which follows the general practice and theoretical background in journalism. The unsupervised learning approach used here is also interesting, and opens up new possible directions for detecting sentence-level bias.", "reasons_to_reject": "Comparison is not comprehensive. One of your cited work, POLITICS (liu et al, 2022), developed a new LM trained on news articles with a special training objective. To draw a fairer comparison, models more powerful than BERT need to be included.\n\nSection 7.3: I don't think rejecting the null hypothesis could validate the claim \"This shows that our model is more invariant to the data it was trained on than the BERT classifier\". Instead, rejecting the null hypotheses in almost all but 1 case shows that *your model is more invariant to the random seeds*. In order to show that your model is invariant to the training data, your null hypothesis should be something like \"mu_1000 = mu_5000 = mu_10000 = mu_full\", i.e., multiple mean difference comparison. To this end, you may consult ANOVA. \n\nDespite the interestingness of section 8, I am concerned about the validity. In other words, if you run your k-means algorithm again, you may end up with a different cluster? It's advised to run k-means multiple times and try to determine the cluster id of an article based on its \"aggregated score\".\n\nSome writings in section 4 are not clear. For example, you should bold u_i in equation 4, and you are advised to explicitly say that \"the propagation is for structual analysis\" in line 332-335, otherwise people might get confused why you want to propagate the label to sentence-level.", "questions_to_authors": "For semantic analysis component (4.1), did you try to use a standard transformer model in place of your \"BiLSTM + Multi-Head Attn\" since the transformer model has positional embedding in nature and implements the attn mechanism? If your concern is the concatenation of multi-heads results, you could manually decompose it.\n\nLine 452: In order to use figure 3 to draw the conclusion that your model is more robust, would it be better to have a another figure in which you could directly show the performance difference between test set 1 and 2, and you could still put your green region and orange region side-by-side. This way, readers will be more easily to visualize the sensitivity to test data of different models.  \n\nIn table 3, what are the meanings of \"Lex only\", \"Info only\", \"biased sentence\". It's strongly suggested to provide the meaning of each baseline either in text or in captions.", "paper_topic_and_main_contributions": "The author study document-level bias detection, in the format of predicting among Left, Center and Right on the basis of US political spectrum. To this end, the author proposes a multi-stage model in which the model first capture semantic meaning of each sentence, then predict the sentence type (i.e., main sentence v.s. supporting sentence), and finally aggregate representations for document-level bias detection by computing the dependency scores between each pair of sentence. The author further conducts an unsupervised-style study where the document-level label is propagated to sentence-level in order to identify main and supporting sentences. Through this propagation, the author is able to uncover some interesting writing patterns commonly employed in journalism, e.g., reversed pyramid.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Late Fusion with Triplet Margin Objective for Multimodal Ideology Prediction and Analysis (echo line 142, this work also attempts to detect political bias using visuals/images)", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 198: to build"}, "extra_text": {"Justification_for_Ethical_Concerns": "No ethical consideration section found in this paper. Considering this paper is tackling the political bias detection problem which is full of sensitive data and information, there is a pressing need to discuss some ethical perspectives (e.g., copyright of news data, failure mode of your bias detection model, its impact to vulnerable populations)."}}, {"review_id": "jUMyWPDTIz", "reviewer": "Reviewer_b9Zn", "summary": "", "questions": "(1) If you repeat the the k-means analysis on just the large class 1 docuemnts, do you get a meaningful partition of this class into subclasses?   (Delete class 2 and 3 documents before doing this analysis)", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedbacks on our paper and suggestions that can improve the clarity of our paper. We would like to provide the responses to your points starting with ‘Reasons to Reject’ section, and continue with ‘Question to the Authors’ section.\n\n**[R1: Clarity of Writing]**\n\nThank you for the feedback. We will work to make sure the contents of the paper are accessible for a wider range of readers from practitioners in journalism as well as NLP.  We will provide a brief response to the two observations you made regarding the clarity of our work and we hope that this will provide clearer insights into our contributions. \n\n*Q. How does our research differ from prior studies in this field?*\n\n- We contribute a new modeling formulation that examines and addresses the domain dependency issue in political bias detection models, particularly the challenge of \"Predicting political bias for unfamiliar news sources that arise post-training.\"\n\n\n- Earlier models for detecting political bias required external resources, such as dataset-adjusted triplets (Liu et al., 2022, Kim and Johnson et al, 2022) or external knowledge (Baly et la., 2020, Zhang et al., 2022, Feng et al., 2022). In contrast, our model leverages multi-head attention and a hierarchical structure, relying solely on the article and its structure. This ensures simplicity in reproduction and eliminates dependence on external knowledge sources or specific datasets that make reproduction challenging. \n\n\n- We also incorporated journalism-specific features in discourse structure to verify our model's alignment with authentic knowledge. Specifically, our contributions in section 8 examine whether our model's structural analysis aligns with journalistic writing styles of straight reporting and opinion pieces. This is an emergent property of our model helping to validate our findings. \n\n*Q. Why did we opt for our dataset formulation?*\n\n- The Baly et al., 2020 dataset we use posed certain challenges which we mitigate. The dataset available on GitHub contains a contamination of news articles between the train, validation and test splits. This makes it difficult to analyze a model on the resilience to unseen writing styles.  \n- Recognizing this, we re-partitioned the dataset so that it has two distinct test sets that pairwise disjoint between the test and training sets. We created test set 1 and 2. Each of these sets, along with the training set, contains a mutually exclusive set of news outlets with perfectly balanced labels. This addition not only the contamination issue of Baly et al., 2020, it also rectifies a further imbalance issue and aids in a more comprehensive assessment of our model's performance across varied news sources. \n\n- To aid reproducibility of our work, we will release our data splits as well as our code\n\n**[R2: Validity of Structural Analysis]**\n\nWe also agree that the size of Cluster 1 is dominant. Also, we deeply understand that it might look like an insignificant result without an explanation of the specialty in the journalism domain. In addition to the results we reported in Q1, we report the Silhouette score as a metric for within-cluster similarity and inter-cluster dissimilarity, and the clustering results with different numbers of clusters. \n\n**1) Silhouette Score:** Silhouette score in three cluster settings was 0.8988 where the score near 1 indicates the strong similarity within the clusters and strong dissimilarity between the clusters. \n\n**2) The Cluster Size by Differing the Number of Clusters:** We report the size of the biggest cluster by differing the number of clusters from three to five. The biggest clusters in each setting took 92% (result in our paper, K=3),  91.33% (K=4), and 88% (K=5) of the whole set of news articles in the BASIL dataset.\n\n\n**[R3: Baseline Outperforming Proposed Model]**\n\nThe motivation and the most interesting point that we found in political bias detection as a downstream task in the field of NLP were that they were critical to both the training and test sets. So, we wanted to empirically validate the presence of the observed phenomena, making it essential to conduct experiments across various settings. In the scenario involving 1,000 instances, BERT might show higher accuracy in Test Set 1, but its performance significantly fell in Test Set 2. We believe the accuracy of the model is reliable if it is invariant to both the training and test sets. In that sense, we would like to note that the learning curve of BERT is distinct on Test Set 1 case and Test Set 2 case as shown in Figure 3. \n\tFurthermore, we report the results of GPT2 and BERT-Large trained and tested to our dataset to show that domain dependency problem occurs even in the models with more parameters and more input token limits.\n\n**[Q1: Re-Clustering in Cluster 1]** \n\nAlong with the response to R2, we report the results of clustering on top of the news articles in Cluster 1 to answer your question. We conducted clustering with a minimum of two to a maximum of five different clusters, and the biggest clusters took 97.6% (K=2), 90.9% (K=3), 87.3% (K=4), and 86.2% (K=5) of news articles in original Cluster 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) Their approach avoids the problem of recognizing writing styles of sources instead of principled analysis of bias, which seems an interesting idea\n\n(2) The machine learning models and statistical significance analysis are sophisticated\n\n(3) there is an effort to identify distinct article structures using k-means clustering", "reasons_to_reject": "\n(1) among the multiple papers I have refereed on stance prediction in this cycle, this is the one that I somehow found most difficult to understand.   See my comments on writing below.\n\n(2) The analysis of structure type of Section 8 was not really convincing to me.   The k-means analysis showed that 92% of the articles reflected the inverted pyramid. with the other two clusters so small as to seem insignificant.\n\n(3) There are some weird results where the baseline model out-performed their model for smaller test sizes, although I applaud the authors for the honesty of including the non-flattering data in the paper.", "questions_to_authors": "(1) If you repeat the the k-means analysis on just the large class 1 docuemnts, do you get a meaningful partition of this class into subclasses?   (Delete class 2 and 3 documents before doing this analysis)", "paper_topic_and_main_contributions": "This paper describes a method for political stance prediction using rhetorical structure, which avoids problems of source prediction by style matching.   They propose a new model of heirarchical analysis with multi-headed attention, and apply this model to cluster articles by rhetorical structure.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None I know"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper seems well written on the surface, yet I had more difficulty understanding things than other papers on the topic which I received.  Part of it may be the assumption of certain terms which need to be defined.    for example, the domain dependency problem is defined in terms of an \"instability\" of the model.\n\nSimilarly, the reasons for the augmented media-based splits were not clear to me.  My sense is that you had good reasons, but the explanations of motivation were not really adequate."}}, {"review_id": "hxSypqUzVV", "reviewer": "Reviewer_twSB", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for the helpful reviews and detailed recommendations for related works that you gave. We are happy to provide additional experiments and analysis on both the existing and additionally reported experimental results. Additionally, we've included a brief overview of our objectives and purpose. We trust this will enhance the clarity of our manuscript.\n\n**[R1: Comparison with Other Models]**\n\nThank you for highlighting the importance of a comprehensive comparison with models other than BERT. We strongly concur that considering models that are specialized for political bias detection, can be helpful for fair comparison. During our research, however, we had difficulty finding relevant models suitable for comparison.  While there are specialized models for political bias detection like POLITICS (Liu et al., 2022) it is impossible to test them in our dataset setting as they require additional information or pre-processing from the plain text of the news articles: external knowledge (Baly et la., 2020, Zhang et al., 2022, Feng et al., 2022), specifically knowledge triplets (Liu et al., 2022, Kim and Johnson et al, 2022). In contrast, our method emphasizes content-based classification without the use of knowledge augmentation, ensuring easy reproducibility and independence from external knowledge sources. We view it as a significant strength of our model and for its distinctness, it is difficult to find models with a similar methodology for a direct comparison.\n\nThough we couldn't find directly comparable state-of-the-art models for a fair comparison, we tried to compare the performance of our model with that of the recently proposed political bias detection models which uses external knowledge. However, it is worth noting that most works in the realm of political bias detection, even those recommended by the reviewer, refrain from releasing their code and dataset for public access (e.g. Li and Golwasser et al., 2019 and Li and Golwasser et al., 2021 did not open code, Zhang et al., 2022 did not open data pre-processing code). Given this context, we chose to showcase our model's superior performance within the same dataset configuration as other models. Subsequently, our aim was to underscore that even robust language models can be affected by the composition of their training and test sets. Through these endeavors, our objective was to emphasize the significant dependence of political bias detection models on the specific news articles used in their training and testing phases. In that sense, we report the results on GPT2 as a language model with a longer input token limit and BERT-Large as a stronger encoder model compared to the BERT-base. \n\n**[R2: Baseline Outperforming Proposed Models]**\n\n We were also aware of the phenomena you’ve mentioned, and we would like to provide further explanation on the low performance in low-training set setting. The underlying motivation of the paper was that capability of the model which can be shown through accuracy, F1 score, or AUROC is no longer credible if it is highly variant according to the training and test sets. In that sense, BERT as a political bias classifier is relatively harder to rely as Jansen-Shannon Distance is consistently higher than our proposed method. As our model have more parameters compared to BERT, it is natural for our model to be more variant in low-training set setting. However, our model is robust even in low-training set setting. We believe our joint analysis on model’s performance metric (e.g. AUROC, F1 Score) and robustness can question the performance of BERT even though it shows higher performance in a few low-training set cases.\n\nTo add, we report the experiments done with BERT-Large and GPT2. We experimented BERT-Large to empirically show that the variance of AUROC and F1 Score should be higher as the number of model’s parameters increase. Also, we experimented GPT2 to empirically show that same phenomena that we focus on also happens in stronger language model with more parameters and more input token limits. We believe these experimental results will further support the insights given in Section 7 of our paper.\n\n\n\n|        |            | GPT2                |                     | BERT-LARGE      |                 | Ours                |                     |\n|--------|------------|---------------------|---------------------|-----------------|-----------------|---------------------|---------------------|\n|        |            |        AUROC        |       Macro F1      |      AUROC      | Macro F1        | AUROC               | Macro F1            |\n| 1,000  | Test Set 1 | 0.5050 (0.0260)     | 0.3214 (0.0145)     | 0.5229 (0.0279) | 0.3180 (0.0585) | **0.6017 (0.0190)** | **0.4227 (0.0185)** |\n|        | Test Set 2 | **0.6185 (0.0174)** | **0.4353 (0.0428)** | 0.5858 (0.0532) | 0.3871 (0.0849) | 0.5703 (0.0298)     | 0.3971 (0.0250)     |\n| 10,000 | Test Set 1 | 0.5756 (0.0122)     | 0.3883 (0.0425)     | 0.5596 (0.0158) | 0.3841 (0.0131) | **0.6529 (0.0081)** | **0.4747 (0.0090)** |\n|        | Test Set 2 | 0.6564 (0.0134)     | 0.4833 (0.0269)     | 0.6559 (0.0231) | 0.4912 (0.0183) | **0.6783 (0.0094)** | **0.5089 (0.0122)** |\n\n**[Overview]**\n\n*Q. How does our research differ from prior studies in this field?*\n\n- We contribute a new modeling formulation that examines and addresses the domain dependency issue in political bias detection models, particularly the challenge of \"Predicting political bias for unfamiliar news sources that arise post-training.\"\n\n\n- Earlier models for detecting political bias required external resources, such as dataset-adjusted triplets (Liu et al., 2022, Kim and Johnson et al, 2022) or external knowledge (Baly et la., 2020, Zhang et al., 2022, Feng et al., 2022). In contrast, our model leverages multi-head attention and a hierarchical structure, relying solely on the article and its structure. This ensures simplicity in reproduction and eliminates dependence on external knowledge sources or specific datasets that make reproduction challenging. \n\n\n- We also incorporated journalism-specific features in discourse structure to verify our model's alignment with authentic knowledge. Specifically, our contributions in section 8 examine whether our model's structural analysis aligns with journalistic writing styles of straight reporting and opinion pieces. This is an emergent property of our model helping to validate our findings. \n\n*Q. Why did we opt for our dataset formulation?*\n\n- The Baly et al., 2020 dataset we use posed certain challenges which we mitigate. The dataset available on GitHub contains a contamination of news articles between the train, validation and test splits. This makes it difficult to analyze a model on the resilience to unseen writing styles.  \n- Recognizing this, we re-partitioned the dataset so that it has two distinct test sets that pairwise disjoint between the test and training sets. We created test set 1 and 2. Each of these sets, along with the training set, contains a mutually exclusive set of news outlets with perfectly balanced labels. This addition not only the contamination issue of Baly et al., 2020, it also rectifies a further imbalance issue and aids in a more comprehensive assessment of our model's performance across varied news sources. \n\n- To aid reproducibility of our work, we will release our data splits as well as our code", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "[A1] The proposed model for bias detection seems novel and easy to replicate which allows it to be used by researchers in future studies.\n\n[A2] The experiments done in the paper seems sound and the qualitative experiments are interesting. ", "reasons_to_reject": "[R1] Bias detection is a well-studied task at this point and the authors sidelined a lot of works and models previously proposed for bias detection (refer to missing citations). As a result, the authors compared their model's performance only with BERT, which is not a state-of-the-art model when it comes to political bias detection in news articles. Because BERT is not a good encoder when it comes to encoding long contexts such as news articles. Different approaches have been proposed to perform political bias detection in news articles that are directly applicable to the setting of this paper. Without comparison with those models, it is difficult to understand the effectiveness of the proposed approach in this paper.\n\n[R2] The proposed model even underperforms BERT in the low-training set and balanced test data setting (refer to Table 2). However, I did not find a good justification for this low performance in the paper. Does it mean the proposed model requires more training data and has some bias toward a particular class label? Per-class classification scores would be really helpful to evaluate that.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors of this paper studied political bias detection in news articles. For that, the authors proposed a model that takes into account both sentence-level semantics and document-level discourse structure of the articles for bias detection. The authors compared their model's performance against BERT and performed a set of qualitative evaluations of their model's prediction. The authors performed structural analysis of the news articles centered around -- in what part the main sentence appears in the news articles. And they found that the findings using their proposed model correspond to previous findings, indicating the correctness of their approach.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The literature gap is large in the paper. The authors should have cited and compared their model's performance with other existing models. A few of them are listed below.\n\n- Encoding social information with graph convolutional networks for political perspective detection in news media. (Li and Goldwasser, 2019)\n- Using social and linguistic information to adapt pretrained representations for political perspective identification. (Li and Goldwasser, 2021)\n- CLoSE: Contrastive Learning of Subframe Embeddings for Political Bias Classification of News Media. (Kim and Johnson, 2022)\n- KCD: Knowledge Walks and Textual Cues Enhanced Political Perspective Detection in News Media. (Zhang et al., 2022)\n- KGAP: Knowledge Graph Augmented Political Perspective Detection in News Media. (Feng et al, 2022)\n\nPlease have a look at the papers citing these papers and more models for this task might be found."}}]

Author rebuttal:

Meta-review: The authors proposed a new modeling formulation that examines and addresses the domain dependency issue in political bias detection models. The authors tried to remove dependency on external knowledge sources. Reviewers have concerns regarding comparison with baselines. 

All the reviewers agree that the paper provides sufficient support for the major claims.  

However, the contributions are marginal. Comparisons are not comprehensive (Reviewers Nv3C, twSB). Some terms need to be defined. An additional revision would be helpful to make the paper more readable (Reviewer Nv3C, b9Zn).
Decision: Accept-Findings