Mitigating Framing Bias with Polarity Minimization Loss
Authors: Yejin Bang
Nayeon Lee
Pascale Fung
Venue: EMNLP 2023 Findings (2023)
Keywords: framing bias

Abstract: Framing bias plays a significant role in exacerbating political polarization by distorting the perception of actual events.
Media outlets with divergent political stances often use polarized language in their reporting of the same event.
We propose a new loss function that encourages the model to minimize the polarity difference between the polarized input articles to reduce framing bias. Specifically, our loss is designed to jointly optimize the model to map polarity ends bidirectionally.
Our experimental results demonstrate that incorporating the proposed polarity minimization loss leads to a substantial reduction in framing bias when compared to a BART-based multi-document summarization model. Notably, we find that the effectiveness of this approach is most pronounced when the model is trained to minimize the polarity loss associated with informational framing bias (i.e., skewed selection of information to report).

Mitigating Framing Bias with Polarity Minimization Loss
Yejin Bang Nayeon Lee Pascale Fung
Centre for Artificial Intelligence Research (CAiRE) The Hong Kong University of Science and Technology yjbang@connect. ust.hk
Abstract
Multi-Doc. Summarization
Polarity Minimization
Polarized Articles
Officers fire tear gas peaceful protesters to clear the for Trump's photo op
Framing bias plays a significant in exac- erbating political polarization by distorting the perception of actual events. Media outlets with divergent political stances often use polarized language in their reporting of the same event. We propose new loss function that encourages the model minimize the polarity difference between the polarized input articles to reduce framing bias. Specifically, our loss is designed jointly optimize the model map polarity ends bidirectionally. Our experimental results demonstrate incorporating the proposed po- larity minimization loss leads to substantial reduction in framing bias when compared to a BART-based multi-document summarization model. Notably, find that the effectiveness this approach most pronounced when the model is trained to minimize the polarity loss associated with informational framing bias (i.e., skewed selection of information to report).
x
x
X
Y
Trump Pays Homage To Church Burned In Riots
Neutral Summary
With Bible Hand
MDS
Polic
I
Training Inference
Neut
x
Trump Visits Church Burned George Floyd Protests
x
X
Model
Figure 1: Illustration of training and inference with proposed polarity minimization loss for reducing framing bias.
1 Introduction
illustrated in Fig. 1. Intuitively, the model is forced to learn and focus the shared aspect between contrasting polarities from two opposite ends.
Framing bias has become pervasive problem in modern media, mislcading the understanding of what really happened via a skewed selection of information and language (Entmann 2007, 2010; Gentzkow and Shapiro, 2006). The most notable impact of framing bias is the amplified polarity between conflicting political parties and media out- lets. Mitigating framing bias is critical to promote accurate and objective delivery information.
In this work, we demonstrate the effectiveness of our proposed loss function by minimizing polar- ity in different dimensions of framing bias lex- ical and informational (Entman, 2002). Lexical polarization results from the choice of words with different valence and arousal explain the same in- formation (e.g., "protest" vs "riot"). Informational polarization results from a differing selection information to cover, often including unnecessary or unrelated information related to the issue being covered. Our investigation suggests that learning the opposite polarities that are distinct in the infor- mational dimension enables the model to acquire better ability to focus common ground and minimize biases in the polarized input articles. UI- timately, our proposed loss enables the removal of bias- Mlasinnuuiing information and the generation more neutral language choices.
One promising mitigation paradigm is to gener- ate a neutralized version of a news article by synthe- sizing multiple views from biased source articles (Sides, 8018; Lee et al., 2022). To more effectively achieve news neutralization, we introduce polar- ity minimization loss that leverages inductive bias that encourages the model to prefer generation with minimized polarity difference. Our proposed loss trains the model to be simultaneously good at map- ping articles from one end the polarity spectrum to another end of the spectrum and vice versa as

---

2 Related Work
political bias, the goal is generate neu- tral article summary Y. where dataset is D (2 X,), Y). The neutral summary Y should (i) retain salient information and (ii) min- imize as much framing bias as possible from the input articles.
Framing Bias Framing bias is a well- documented phenomenon in the field of media studies (Wright and Goodwin, 2002; Entman, 2002, 2010, 2007; Gentzkow and Shapiro, 2006;
Gentzkow al., 2015; Beratšová et al., 2016).
According to Gentzkow and Shapiro (2006). framing bias occurs when journalists and media outlets selectively emphasize certain aspects of a story while downplaying or ignoring others (informational) with biased use of languages (lexical). This can result in distorted perception of events among the public, particularly in cases where the framing is done to serve particulan ar agenda ideology (Kahneman and Tversky, 2013; Goffman, 1974). The impact of framing bias is especially evident in the political arena, where media outlets and political parties often engage in polarizing discourse that is designed to appeal to their respective bases (Scheufele, 2000; Chong and Druckman, 2007).
3.2 Polarity Regularized Model
To equip a model for neutral summarization, we for- mulate it as a conditional generation with given po- larized articles Xpolarized (X?, x?.... x*), where superseripts represent arbitrary polarization directions based in a certain criteria dimension. In this work, it is based on political ideologies (i.c.. left (X*), right (X*),, center (X²)). The model can be defined in a conditional auto-regressive man- ner as defined eq. 2. Input X, defined in eq. 1, is yielded to vector representations by the encoder. Then the encoded input polarized articles are used to generate target neutral summary sequentially using the decoder.
Automatic Mitigation Efforts To mitigate that, there have been various automatic media bias mit- igation efforts (Fan et al., 2019; Hamborg et al., 2019; Morstatter et al., 2018; Laban and Hearst,
X concat((X' shuffle([K))).
(1)
where shuffle(I) denotes the random permutation operation on a given list I.
2017; Hamborg al., 2017; Zhang et al., 2019b; den Berg and Markert, 2020; Lee al., 2022). A similar line of work is ideology prediction (Liu al., 2022) (if they are left-, right-, or center- leaning) or stance prediction (Baly et al., 2020) - which is polarity detection. On the other hand, our work focuses on generating a neural article from polarized articles. Given that framing bias often happens very subtle, Morstatter et al. (2018) learns the pattern of framing bias in a sentence and at- tempts to detect automatically. Another common mitigation attempt display multiple viewpoints in an automatic way (Hamborg al., 2017; Park et al., 2009). Lee al. (2022) took a further step to make a summary out of the polarized articles to provide multiple perspectives automatically one single summary. Our work aligns with the vision of previous works, but we focus on the more gen- eral way to mitigate framing bias by studying the polarity minimization loss.
Po (o(( = II po (oooleee X),
where V denotes the length of the article summary Y. The model parameterized by 0 is optimized through the objective maximum likelihood of target tokens the neutral article Y. Precisely, the loss function for training this model is as follows:
LMDS
Σ X.YEB
log log
where is a batch of input and target pairs (X, Y). However, LADS does not explicitly optimize for polarity distance.
Polarity Minimization Loss Framing bias re- sults from the polarized portrayal of the same event or issue. Motivated by this, we propose to train the model with an additional polarity mini- mization loss, Lpolars which requires learning from both directions between arbitrary polarities (e.g.. X Given there two po-
3 Approach
X X larity ends, an article with arbitrary source polarity is denoted as X" while an opposite polarity is a target article, X The polarity minimization loss
3.1 Problem Statement
Given a set of multiple polarized news articles XI..K with varying degrees and orientations of

---

Avg. Framing Bias Metrie
Salient Info 1
4rousal
Arousal-
Arousalaem
BLEU
BERTS-FF -1
All Source Iaput
6.76
3.64
10.4
8.27
64.00%
PEGASUSMULII
5.12
2.39
7.51
6.12
61.44%
BARTMULTI
5.94
2.66
8.61
4.24
61.06%
PEGASUSNEUSFT
2.18
3.30
11.26
68.41%
BARTNEUSFT
1.86
2,85
11.67
70.10%
BARTNEUSFT-T
1.69
0.83
2.53
12.05
70.50%
+LR VALENCE
1.57
0.91
2.48
10.62
69.67%
+LR- -L--AROUAAL
1.18
0.62
1.80
8.84
69.49%
LR-INFO
1.08
0.70
1.78
9.31
70.19%
ARC-SAL
1.22
0.75
1.97
9.66
69.94%
LRC-INFO
1.25
0.72
1.97
10.18
70.24%
Table I: Experimental results for ALILSSDES test set. For framing bias metric, lower number is the better (L). For other scores, the higher number is the better (t).
obtained with negative log-likelihood between one polarity X to the opposite X is as follows:
reducing framing bias is evaluated by adopting Arousal scores (Lee et al., 2022), which is based on the Malence-Arousal-Dominne (VAD) lexi- cons (Mohammad, 2018). We report all positive- valence arousal scores (Arousal, ) negative- valence arousal scores (Arousal_), and the com- bined arousal scores (Arousal It is ideal for models achieve lower Arousal scores. For salient info, we mainly evaluate with BERTSCORE- FI (Zhang* et al., 2020), which is a representative embedding-based metric, instead of lexicon-basee reference metrics because we expect the model to rewrite biased uses of language in a neutral way, thus, the embedding- metric is more appro- priate than the lexicon-based. For comparison with previous work, still report BLEU (Papineni al., 2002) score. Lastly, we conduct human- -valuuaia to validate the result on reducing framing bias.
I Σ
polaar
(x
(4)
X
where the represents a batch consisting of two ar- ticles of opposing polarities X* and X Note that s, & are not bound to specific polarities but indicate source and target ends (i.c., X E x and X X2,,X_..... } (X*) within batch). In summary, the model parameters are opti- mized with Lneuts which is composed of a multi- article summarization objective (LMDS) and a po- larity minimization term Lpolar.
Tmeut MDSS polar
(5)
where À > 0 and is hyperparameter that assigns a relative weight to the polarity minimization loss with respect to the multi-document summarization.
4.2 Models
Baselines We compare with off-the-shelf multi- document summarization (MDS) models trained on Multi-news dataset (Fabbri et al., 2019) (BARTMULTI (Lewis al., 2019) and PEGASUS- MULTI (Zhang et al., 2019a)) as baselines. Those models have achieved high performance in MDS, which can also be applied in summarizing polar- ized articles. However, these models do not have any learning about framing bias removal or neutral writing. We also compare with the statc-of-the. models (BARTNEUSIF and BARTNEUSFT-I) (Lee et al., 2022) that fine-tuned with ALL- SIDES dataset. BARTNEUSFT fine-tuned only with articles meanwhile BARTNEUSFT-TT addition- ally leverages titles of each article. We additionally
4
Experiments
4.1 Setup
Dataset ALLSIDES dataset (D) (Lee et al., 2022) is from Allsides.com designed for neutral multi- article summarization. consists of 3066 triplets from left, right, and center American publishers on the same event, xx'x"", (xx),, ), and an expert- written neutral summary (target) of the articles, Y. Note that "center" idcology contains relatively less bias, but still tends to contain framing bias (all, 2021) (Refer to Appendix B for examples).
Metric We evaluate with a suite of metries in- troduced in the benchmark. The effectiveness of

---

x
Officers fire tear gas peaceful protesters to clear the way for Trump's photo op
Input (X)
Trump Pays Homage To Church Burned In Riots With Bible Hand
Tear gas, threats for protesters before Trump visits church
Target (Y)
Controversy Surrounds Trump Church Visit, Dispersal of Protesters
[LRC-AROUSAL]
Trump Visits Church Burned in Riols.
[LR-INFO (best) 'Trupp Visits Church Burned in George Floyd Protests.
Generations
'L Trump Pays Homage to Church Burned Riots.
R to L' Trump Speaks on George Floyd Protests.
Table 2: Musstrationn of sample generations. Our best model LR-INFO could successfully generate the neutral summary of the inputs X X " - "L to R' denotes the ablation study of the effect of subtracting 'L to R' direction (i.e., having only R to L, not both) with LR-INFO.
report PEGASUSNEUSFT. Simply fine-tuning may not be effective enough to learn about framing bias. Thus, we will demonstrate how the polarity mini- mization loss can effectively mitigate framing bias compared to baseline and SOTA models.
of the number of unique tokens from each article pair. For instance, given an article pair (X*, X we calculated as below:
UniqueNum( (Xmiamm((,,,_ xx)x = |((TToen(X") U Token x²))
- -(Token(X*)) n Token (x
Proposed models with polarity minimization loss The degrees of polarization vary among sets articles and the polarization is expressed through lexical and/or informational bias. In our task, X X is in {X4, x" x²), where I., R, and C denote left- and right-wing center political ideolo- gies respectively. For the experiment, we used BART as the backbone model, thus our proposed models are denoted with '+' signs prepended to specific polarity minimization loss variation Ta- ble 1. We investigate which criteria would be help- ful to construct the most distinct (X*, X Thus, variations are explained as follows:
where Token(X) refers to set of tokens of ticle X separated by blank spaces. LR-INFO: construct with {X-,, R LRC-INFO: construct from {X4, XR. Xx).
4.3 Results
Effectiveness of polarity minimization loss in reducing framing bias As illustrated in Table I, all variations of polarization minimization loss could successfully reduce overall arousal score Arousalasn in comparison to the baseline models. LR-INFO achieves reduction up 8.62 absolute value compared the bias score of All source arti- cles (All Source), which 82.89% reduction. Com- pared to baselines, our models could keep overall salient information relatively well in terms of both BLEU and BERTSCORE-F1.. We investigate that loss based on informational polarization are more effective keep the salient info than lexical po- larization. For instance, Arousalsum similar for LR- LR-AROUSAL (1.80) and LR-INFO shows higher BLEU and BERTSCORE- Fl- by 0.47 and 0.7% respectively.
1) Lexical: LR- VALENEE Learning from the set of two extreme political ideologies articles that have high difference in valence scores, using VAD lexicons. After picking pairs of 1X4,X articles with high differences in valence scores, we teach models about polarity shift from one another (i.e., X + X X + X)). LR-AROUSAL. Similar to LR- VALENCE, but based on the high differences in arousal scores. LRC-AROUSAL: We construct set with all three ideologies.' To pick pairs of articles that have high differences in arousal score from (XL, X x²), calculate differences of all combinations, which are (X!. RR). (X4, X*), (XR, X²).
(1.78), but
Compared to the previous SOTA model BARTNEUSFT-T, our proposed polarity mini- mized models could successfully reduce overall arousal scores. Specifically, LR-INFF achives 7% of reduction. It could be achieved with relatively more reduction in Arousal 36% of reduc- tion from SOTA). However, there a bit of trade- off in salient information compared to the SOTA model 0.883 ~ 0.31% 31% in BERTSCORE-F1),
2) Informational: This focuses on polarization derived from "what" information to be covered. To learn the informational polarization, we select the article pairs that have high differences in informa- tion. We calculate the difference through the sum Here, X is also a biased article as explained in $.

---

Laleeceece
Arousll
LRo--rousal
Avg. Framing Bias Metric
Salient Info
LRC-Info
Framing Bias Score
2.75
Arous.+
Arous..
Arous.um
BERTS-FI
2.50
LR-INFO
1,081
0.698
1,778
70.24%
(Arousal
to
1.498
0.830
2.329
70.06%
2.25
to 1.597
0.869
2,466
70.61%
Table Ablation study: Effect of having only single- directional polarity minimization with LR-INFO model.
0.7
Weight L_polar (A)
Figure 2: Illustration of performances in terms of fram- ing bias score (Arousalsum) depends on varying rela- tive weights for polarity minimization loss Lpolar
of the models. In general, models remove fram- ing bias most effectively when Lpolar weighs the same (A = 1) or slightly less 0.7) with respect to LMDS The variant indicates there exist opti- mal weights for each model instead of a universal optimal weight.
Effective learning with extreme polarities We investigate that polarity minimization between ex- treme ends (left, right) is more effective than the mixture with a center media outlet. This is be- cause left and right-wing ideologies are the oppo- site ends that can train models more effectively about extreme ends than center media outlets al- though center media is not completely free of bias. Qualitative analysis results align with the quantita- tive measures. For instance, as illustrated in Table 2, the polarity minimized models LR-INFO and LRC-AROUSAL both could summarize with the essential information out of polarized input articles. Especially LR-INFO, the lowest biased model, it could even use more neutral choice of word (e.g.. "protests" instead of "riots" same to target Y).
Ablation Study: Polarity shift in uni-direction Our polarity minimization loss forces the model learn polarity shifts bi-bidirectionally at the same time to aid in reducing framing bias. We conduct an ablation experiment of the training model with loss optimizing polarity in uni-direction by sub- tracting one direction each from LR-INFO (i.e., by subtracting "R to L", explore the effect of learning X X only). The results support the effectiveness of our proposed bi-directional minimization loss (Table 3). This is because uni- directional learning does not lead the model to a "neutral" state of writing but to the oppositely biased polarity. In Table 2, about the issue of Trump's 's church visit, one direction mapping could not generate the neutral and essential information, but merely remove information that does not ex- ist in the opposite. For instance, when subtracting mapping from L to (i.e., "L. R"), generation copies most from X except, instead of minimiz- ing the polarity difference. On the contrary, bi- directional polarity minimization loss (LR-INFO) could obtain salient information of "church visit" and neutral choice of word.
Human evaluation We conducted a human eval- uation on LR-INFO against BARTNEUSFT-IT a model without polarity minimization loss, by ask- ing which article is more biased. The generations from the model with LR-INFO are annotated to be less biased similarly neutral 76.63% of the time (win: 53.3%; draw: 23.33%). Annotators agreed moderately, with Cohen's к 0.427 on average. According to analysis, ppsar shows its strength to remove information that is implemented to frame in certain ways in polarized input articles. Details and examples are in Appendix A.
5 Conclusion
Framing bias is a pervasive problem in modern media, which can lead to a distorted understanding of events and an amplification of polarization. To tackle this, we introduce polarity minimization loss that reduces framing bias in generation. Our experimental results demonstrate that incorporating the proposed polarity minimization loss is effective in the reduction of biased uses of language and in removing biased information from the source input, which ultimately mitigates framing bias.
4.4 Analysis
Variant of weights of Lpotar (A) X is a hyper- parameter that assigns weight to the polarity min- imization loss (Lpolar) with respect to the multi- document summarization loss (LMns). À also indi- cates the intensity of the signal for learning oppo- site polarity. As illustrated Fig. 2, the framing bias scores increase when polarity minimization loss overtakes MDS loss (i.e., x = 1.5) for most

---

Limitations
online users access more depolarized information online.
The study is limited by its adherence to the bench- mark's English-based task setup. The analysis is constrained to political idcologies in the United States and the English language. Additionally, the BART model's 1024 sub-token input limit restricts the number of biased source articles that can be included as an input. It important to note that these limitations, while potentially impacting the scope of the study's findings, are not uncommon in natural language processing research. Nonethe- less, future research may benefit from addressing these limitations by exploring alternative methods for broader range of political ideologies (Non- U.S. political ideologies) and languages, as well as incorporating longer input texts capture a more comprehensive range source articles.
References
2021. Center what does "center" media bias rating mean?
Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov, 2020. We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4982-4991, Online, Association for Computa- tional Linguistics.
Adriana Beratšová, Kristína Krchová, Nikola Gahová, and Michal Jirásek. 2016. Framing and bias: a lit- erature review of recent findings. Central European journal of management, 3(2).
Dennis Chong and James N Druckman. 2007. Framing theory, Annu. Rev. Polit. Sci., 10:103-126.
Ethics Statement
The issue of biased articles with framing has been extensively studied, as it can lead to polarization by influencing readers' opinions toward a certain person, group, or topic. To address this problem, our research focuses on introducing a loss function that can be incorporated to enable the model to reduce framing bias in the generated summary.
Robert M Entman. 2002. Framing: Towards elarifica-
tion of fractured paradigm. McQuail's 's Reader in Mass Communication Theory. London, Califorria and New Delhi: Sage.
Robert M Entman. 2007. Framing bias: Media in the distribution of power. Journal of communication, 57(1))
However, it is important to recogmize that auto- matic technologies can also have unintended neg- ative consequences if not developed with careful consideration of their broader impacts. For exam- ple, machine learning models can introduce bias their output, replacing known source bias with another form of bias (Lee et al., 2022). To mitigate this risk, Lee et al. (2022) have suggested includ- ing explicit mention of the source articles alongside automatically generated neutral summaries. Fur- thermore, while our work aims to remove framing bias in tial for hallucination in the generation, which is well- well-known problem of generative models (Ji et al., 2023). Thus, it is important to equip guardrail (c.g.. a provision of source reference) such au- tomatic technology is implemented for actual use cases.
Robert M Entman. 2010. Media framing biases and po- litical power: Explaining slant in news of campaign 2008. Journalism, (144:3899446)
Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. arXiv preprint 100600900.0/.0
Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Pra- fulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019. In plain sight: Media bias through the lens of factual reporting. arXiv preprint arXiv: 1909: 00990006700
articles, there is the poten-
Matthew Gentzkow and Jesse Shapiro. 2006. Media bias and reputation. Journal of political Economy. 114(2):200-316
Matthew Gentzkow, Jesse M Shapiro, and Daniel F Stone. 2015, Media bias in the marketplace: Theory. In Handbook of media economics, volume 1, pages 623-645. Elsevier.
Despite these challenges, our research can con- tribute to the effort of mitigating human-generated framing bias in order reduce polarization in so- ciety. One of the use cases can be aid human experts the process of providing multi-view syn- thesized articles without framing bias. In terms of broader societal impact, we hope our work can help
Erving Goffman. 1974. Frame analysis: An essay the organization of experience. Harvard University Press.
Felix Hamhorg, Karsten Donnay, and Bela Gipp. 2019. Automated identification of media bias in news arti- cles: an interdisciplinary literature review. Interna- tional Journal on Digital Libraries, 2014):391-455.

---

Felix Hamborg, Norman Meuschke, and Bela Gipp. 2017. Matrix-based news aggregation: exploring dif- ferent news perspectives. In 2017 ACM/IEEF Joint Conference on Digital Libraries (JCDL), pages 1-10.
aspects of news to mitigate media bias. In Proceed- ings of the SIGCHI conference on human factorsctors in computing systems, pages 443-452.
IEEE
Dietram A Scheufele. 2000. Agenda-setting, priming, and framing revisited: Another look at cognitive ef- fects of political communication. Mass communica- fion & society, 3(2-3):297-316.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023, Survey of halluci- nation in natural language generation. ACM Comput. Surv. 55())).
All Sides. 2018. Media bias ratings. Allsides.com.
Timo Spinde, Christina Kreuter, Wolfgang Gaissmaier, Felix Hamborg, Bela Gipp, and Helge Giese. 2021. Do you think it's biased? how to ask for the percep- tion of media bias. In 2021 ACM/IEEE Joint Con- ference on Digital Libraries (JCDL), pages 61-69. IEEE.
Daniel Kahneman and Amos Tversky. 2013. Prospect theory: An analysis of decision under risk. In Hand- book of the fundamentals of financial decision mak- ing: Part I. pages 99-127. World Scientific.
Philippe Laban and Marti A Hearst. 2017. newslens: building and visualizing long-ranging news stories. Proceedings of the Events and Stories in the News Workshop, pages 1-9.
Esther van den Berg and Katja Markert. 2020. Context in informational bias detection. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6315-63166 6326, Barcelona, Spain (On- line). International Committee on Computational Lin- guistics.
Nayeon Lee, Yejin Bang. Tiezheng Yu, Andrea Madotto, and Pascale Fung. 2022. NeuS: Neutral nulti-news summarization for mitigating framing bias. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3131-3148. Seattle, United States. Association for Computational Linguistics.
George Wright and Paul Goodwin. 2002. Eliminating a framing bias by using simple instructions to 'think harder' and respondents with managerial experience: Comment on 'breaking the frame', Strategic man- agement journal, 23 (0:11))059-107
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter J. Liu. 2019a. Pegasus: Pre-training with ex- tracted gap-sentences for abstractive summarization.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019, Bart: De- noising squencc-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.15 13461.
Tianyi Zhang Varsha Kishore*, Felix Wu*, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Yujian Liu, Xinliang Frederick Zhang, David Wegsman, Nicholas Beauchamp, and Lu Wang. 2022. POL.-- TICS: Pretraining with same-story article comparison for ideology prediction and stance detection. In Find- ings of the Association for Computational Linguis- ties: NAACI 2022, pages 1354-1374, Seattle, United States. Association for Computational Linguistics.
Yifan Zhang. Giovanni Da San Martino, Alberto Barrón- Cedeno, Salvatore Romeo, Jisun An, Haewoon Kwak, Todor Staykovski, Israa Jaradat, Georgi Karadzhov, Ramy Baly, et al. 2019b. Tanbih: Get know what you are reading. EMNLP-IJCNLP 2019, page 223.
Saif Mohammad. 2018. Obtaining reliable human rat- ings of valence, arousal, and dominance for 20,000 english words. Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 174-184.
A Experimental Details
Training Details the henchmark for the fair comparison: 3 learn- ing rate, batch size of 16 and 10 epochs with the early stopping of the patience of 3. Training takes around 20 ~ 30 minutes per epoch. All experi- ments were with NVIDIA GTX 1080Ti GPU de- vice. We do a hyper-parameter search for weights of the polarity minimization loss Lpolar with respect to LDDS in the range of X E {0.3, 0.7, 1, 1.5).
Hyperparameters setup follows
Fred Morstatter, Liang Wu, Uraz Yavanoglu, Stephen Corman, and Huan Liu. 2018. Identifying framing bias in online news. ACM Transactions on Social Computing, 1(2):1118.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th anmual meeting of the Association for Computa- tional Linguistics, pages 311-318.
BERTSCORE-FI
For assessing salient informa- tion, we adopted token-embedding based met- ric BERTSCORE-FI. We used the pre-trained seecrosootudebeert--laegeeee version provided
Souneil Park, Seungwoo Kang, Sangyoung Chung, and Junehwa Song. 2009. Newscube: delivering multiple

---

by (Zhang* et al., 2020) as the state-of-the-art checkpoint.
Issue: Trump to Speak CPAC
[I.R-INFO] Former President Donald Trump will address the Conservative Political Action Conference (CPAC) on Sunday, his first public speaking engagement since leaving office.
A.1 Human Evaluation
[HART TBURTTEUSFT-T J Former President Donald Trump will give a keynote address at the Conservative Political Action Conference (CPAC) on Sunday, his first speaking engagement since leaving office. Trump is expected to attack President Joe Biden's immigration policies.
We conduct A/B testing to evaluate if our proposed method could actually aid in reducing framing bias. We compare generations from the model with our proposed polarity minimization loss Lpolar LR- INFO, against the generations from the state-of-the- art model, a BART-large model fine-tuned on ALL- SIDES dataset without the loss (BARTNEUSFT-T) We got the generation by rumning the publicly avail- able checkpoint from BARTNRUSFT-T (Lee et al., 2022).
Table Human Evaluation Example.
A.2 Full Experimental Result over varying weights of polar
We investigated the effect of the weights of the polarity minimization loss Lpoaar with respect LMDS the range of À (0.3, 0.7, 1, 1.5). Ta- ble shows the full result for pour proposed mod- els with polarity minimization losses with varying weights A.
We conducted the evaluation with 30 randomly selected samples. We provide two articles from the two models (in random order) along with the issue sentence that describes what the articles are about. Then, the annotator is asked answer the question "Which article is more biased?", following Spinde et al. (2021): Lee et (2022). We get three annota- tions for each sample and select the majority voting. Since many of the test samples are closely related to U.S. politics, we recruited three non-U.S. citi- zens/natiooaals/resideniss to minimize any political bias personal preference involved in the evalu- ation. All three annotators claimed themselves as moderate in political leaning and they are qualified conduct the evaluation English (they all have received their tertiary education in English).
To verify that the selection of which is bi- ased in the pairs is not random, binomial test is conducted after obtaining the evaluation results. The null hypothesis was "The selection of articles generated from LR-INFO (our proposed method) to be less biased is random". Then, we obtained a p-value of 0.019, which rejected the null hypothe- sis (p 0.05). Therefore, the selection of articles generated from LR-INFO to be less biased is not random.
When the model is trained with polarity mini- mization loss, it can learn to remove bias-inducing information while BARTNEUSFT-T suffers. As illustrated in Table 4, our model LR-INFO could remove bias-inducing information "Trump ex- pected attack President Joe Biden's immigra- tion policies" from the summary about the issue of "Trump to speak at CPAC" while BART-- T failed to remove it.

---

Weight (A)
Avg. Framing Bias Metric
Salient Info
Arousal.
Arousal_
Arousal.sm
BLEU
BERTSCORE-FF
0.3
1.583
0.91
2,493
10.9
69.96%
0.7
1.688
0.919
2.607
10.86
69.10%
LR- VALENEE
1
1.572
0.913
2,486
10.62
69.67%
1.5
1.48
0.856
2.336
10.9
69.83%
0.3
1.342
0.788
2.13
10.23
70.00%
0.7
1.468
2.345
11.13
70.19%
I.R-AROUAL.
1
1.183
0.615
1,798
8.84
69 49%
1.5
1.341
0.831
2.171
10.92
70.56%
0.3
1.347
0.791
2.138
10.24
69.81%%
0.7
1.081
0.698
1.778
9.31
70.20%
LR-INFO
1
1.436
0.775
2.212
10.44
70.13%
1.5
1.529
0.86
2.388
10.6
69. 69.60%
0.3
1.664
2.626
69,
0.7
1.412
0.834
2.246
10.65
69.93%
LRC-AROUSAL
1
1,218
0.747
1,966
9.66
69.94%
1.5
1,536
0.87
2.406
10.88
70.10%
0.3
1.413
2.257
11.23
70.90%
0.7
1.248
0.722
1,97
10.18
70.24%
LRC-INFO
1
1.369
0.857
2.226
10.73
70.48%
1.5
1.352
0.789
2.141
11.78
70.74%
Table 5: Experimental results for models with proposed polarity minimization loss, LR-VALENCE, LR- AROUSAL, LR-INFO, LRC-AROUSAL, LRC-INFO, with varying weights (A). For framing bias metric, the lower number is the better (J). For other scores, the higher number the better minimization loss (those denote with +) reported with best λ. Full exploration of À is available in Appendix and Fig.
(t). The results of our models with polarity
B Generation Results
Table 7, 6, We provide examples of generations from our model LR-INFO and LRC-AROUSAL for better understanding of the effectiveness our pro- posed polarity minimization loss Lpolar- Addition- ally, we also provide the corresponding example generations from ablation study described in Sub- section 4.3.

---

Issue
Trump Returns to Campaigning; Doctor Says He is 'No Longer Conta- gious'
X Trump returns to public campaigning, falsely claiming that the
Inputs
virus that infected him "disappearing'
x
Trump no longer risk of spreading COVID-19, doctor says
x
Doctor Says Trump Isn't Transmission Risk After President Holds
Public Event
[R->L]
Trump Holds First Public Event Since Covid- 19. [[.>R] Trump Holds First Public Event Since Covid-19.
Generations
[LRC-A A.RC-AROUSAL] Trump Returns to Public Campaigning.
[LR-INFO (best)] Trump Holds First Public Event Since Contracting
COVID-19.
Issue
Mueller Issues Indictments Against Russians
X
What Mueller's new Russia indictments mean and don't 't mean
Inputs
X
Latest Mueller indietment complicates Russian collusion narrative
X
C
Five key takeaways from the Russian indictments
[R>LL] Mueller Indicts 13 Russians.
[[->>]]
New Russia Indictments.
Generations
[LRC-AROUSAL] Mueller Indicts 13 Russians.
[LR-INFO (best)) Mueller Indicts Russians.
Issue
What to Watch for: 2020 Gubernatorial Races
X
The 3 biggest governor races to watch in 2020
Inputs
X R Meet the Highest-Polling Libertarian Gubernatorial Candidate
the Country
X 11 States Are Choosing Their Governor On Tuesday. Here Are
Races To Watch
[R>>L] 11 States Are Choosing Their Next Governor.
[L->R] Governor Races to Watch in 2020.
Generations
I.RC-AROUSAL, Governor Races to Watch in 2020.
[LR-INFO (best)) States Choose Their Next Governor.
Issue
What to Watch for: 2020 Gubernatorial Races
X
The biggest governor races to watch in 2020
Inputs
R Meet the ftighest-Pollingg Libertarian Gubernatorial Candidate
the Country
X 11 States Are Choosing Their Governor On Tuesday. Here Are
Races To Watch
[R>>L] 11 States Are Choosing Their Next Governor.
[L->R] Governor Races to Watch in 2020.
Generations
[RC-AROUSAL] Governor Races to Watch in 2020.
[LR-INFO (best)) 11 States Choose Their Next Governor.
Table 6: Illustration of example generations. X 8 X denote polarized input articles from left-, right-, center- leaning media outlets. -R-INNO is generation from the best model with our proposed polarity minimization Lpolar A->B denotes when the model learns one direction of polarization from A to B, which is equivalent to "B in our ablation study.

---

Issue
2018 Midterms: Will There Be a Blue Wave?
X Get Over Your Election-Needle P.T.S.D.: The Blue Wave Is Real,
Inputs
and It's a Monster
X Don't Get Too Excited about Election Day Yet, Democrats
x Midwest Abandons Trump, Fueling Democratic Advantage For
Control Of Congress
[R->L]
The Blue Wave Real, and It's Monster.
[L->R] Midwesterners Are More Likely to Vote for Democrats.
Generations
[RC-AROUSAL] Polls Suggests Democrats Will Win Control of Congress.
[LR-INFO (best)) Midterm elections the U.S. Election.
Issue
Off-Yeau State Elections Prompt 2020 Speculation
X Polls close in Kentucky, Mississippi to follow, in elections testing
Inputs
Trump's political power
x
GOP's Bevin trailing in Kentucky guhernatorial race, Trump
calls for 'angry majority' to rise
x Polls close as off-year election results offer clues to 2020
[R->L] Polls Close in Kentucky, Mississippi. New Jersey and Virginia.
[L->R] Polls Close in Kentucky, Mississippi, New Jersey and Virginia.
Generations
[LRC-AROUSAL] Polls Close in Kentucky, Mississippi, New Jersey and Virginia Governor's 's Races.
[LR-INFO (best)) Off- Year Midterm Elections in Kentucky, Mississippi.
New Jersey and Virginia Offer Clues to 2020.
Issue
GOP Lawmakers Consider Gun Control
X Some GOP open discussing Democrats' proposal ban device
Inputs
used in Las Vegas attack xR Republicans Get Behind Gun Control in Wake of Las Vegas Shoot-
ing
c
GOP Lawmakers Consider Gun-Control Measure
[R>>L] Republicans Are Open to Banning Bump Stocks.
[L->R] GOP Lawmakers Consider Gun-Control Measure.
Generations
I.U-AA Republicans Consider Gun Control in Wake of Las Vegas Shooting.
[LR-INFO (best)) GOP Lawmakers Consider Gun Control Measures
Wake of Las Vegas Shooting.
Table 7: Illustaation of example generations 2. X=_ X X denote polarized input articles from left-, right-, center-leaning media outlets. LR-INFO is generation from the best model with our proposed polarity minimization loss Epolar. A->B denotes when the model learus one direction of polarization from A to B, which is equivalent to =B in our ablation study.


Reviews:
[{"review_id": "1vJPeeslaR", "reviewer": "Reviewer_X96S", "summary": "", "questions": "See above in Reasons To Reject", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the effort you have taken to provide feedback. Please find our answers and clarification to your questions below:\n\n**1. Clarification on names of baseline proposed models and metrics:**\n\n- **1.1 All Source:** All Source is **not** a result of generation from a model, but evaluation on the concatenation of all input sources (X^L, X^R, X^C). We report it to show a lower bound for framing bias metric as we assumed that source input contains bias. Also, it can indicate an idea of the upper bound for salient information. We will update it as “All source input” to remove confusion caused by its name.\n\n- **1.2 What is +LR-XYZ?** Those are **not** based on either of the All Source or BARTMULTI. All experiments with +LR(XYZ) are done with the BART-base model using the Allsides dataset. We named each model “+” to show how L_polar (polarity minimization term) is calculated on top of the multi-document summarization loss L_MDS.  We acknowledge that it causes misunderstanding, thus we will remove the + sign and also make it clear experimental results are based on the BART-base model with the Allside dataset. \n\n- **1.3 What does \"Arousal+/-\" mean?** They are not related to political leanings but they are based on valence (positive or negative) based on VAD lexicons (Mohammad, 2018). As defined by Lee et al., (2022), Arousal + / - stands for positive-valence and negative-valence arousal scores respectively. For instance, a word “waste” has a high arousal score with positive valence. \n\n**2. Regarding Figure 2**\n\nThank you for pointing out. We will add the following clarification about Figure 2 (regarding $\\lambda$) in the final manuscript.\n\n- **2.1. Line 43:** Yes, line 043 should be referring to Figure 1. Figure 2. Illustrates the effect of $\\lambda$ (weight of polarity minimization loss L_polar) on framing bias scores. We will clarify them in the paper.\n\n- **2.2. Regarding $\\lambda$:** In Table 1, we reported the best setting for each model. For completeness, we shared all results for all lambda variants we tried in Appendix and Figure 2. In the final version, we will explicitly state “The results are reported with best $\\lambda$.” in the caption of Table 1. \n\n- **2.3. Analysis on a variant of $\\lambda$:** $\\lambda$ is a hyperparameter that assigns weight to the polarity minimization loss (L_polar) with respect to the multi-document summarization loss (L_MDS). $\\lambda$ also indicates the intensity of the signal for learning opposite polarity. The framing bias scores increase when polarity minimization loss overtakes MDS loss (i.e.,  $\\lambda$ == 1.5) for most of the models, except for +LR-Valence model. In general, models remove framing bias most effectively when L_polar weighs the same ($\\lambda$=1) or slightly less ($\\lambda$=0.7) in respect to L_MDS. The variant indicates there exist optimal weights for each model instead of a universal optimal weight.\n\n**3. Binomial test on human annotation:**\n\nWe conducted a binomial test with a null hypothesis “The selection of articles generated from LR-INFO (proposed method) to be less biased is random”. Then, we obtained a p-value of 0.019, which rejected the null hypothesis (p < 0.05). Therefore, the selection of articles generated from LR-INFO to be less biased is not random.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose a polarity minimization loss incorporating into the causalLM loss and show the benefits of addressing framing bias. \n2. They provide a complete experiment, including the comparison of the baseline methods, ablation study, and generation analysis. \n3. The paper structure is clear and it is easy to follow. \n", "reasons_to_reject": "1. More explanation in the experiment part is needed. \n1.1. What is the \"All Source\" model in Table 1? Is the baseline model trained on the ALLSIDE dataset? And what is the model used here?  \n1.2. Similarly, In Table 1, what is the model used in the experiments of \"+LR...\"? Is it adopted from \"All Source\"  or \"BARTMULTI\"?  \n1.3. What does the \"Arousal+/-\" mean? Assuming it is corresponding to the party left and right. But which one corresponds to which? \n2. The discussion of Figure 2 is confusing to me. \n2.1. The only part that mentions Fig 2 is in line 043 but that would be Figure 1 instead?  \n2.2. And resultly, what is the selection of \\lambda for the experiment in Table 1 \"+LR...\"? Would that be the best for every setting or the same setting(\\lambda) for every experiment?  \n2.3. Also, What is causing the variant in the selection of \\lambda? Could you please provide some analysis on that?   \n\n3. I'm slightly skeptical about the IAA in line 235 for human evaluation. Could you please also provide the result of the binomial test showing that the selection of which one is biased in the pairs is not random? ", "questions_to_authors": "See above in Reasons To Reject", "paper_topic_and_main_contributions": "This paper proposes a polarity minimization loss to reduce the framing bias in the generation. Their experiment results show the effectiveness of the addition of the proposed loss. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In footnote 1, the index of the Appendix is missing. \n2. Please make sure the precision of all the results is the same across the experiment tables. E.g. Table 3, 0.83 -> 0.830.   "}, "extra_text": {}}, {"review_id": "D2tWS153Gl", "reviewer": "Reviewer_xt1k", "summary": "", "questions": "Following weakness 1 about the intuition, could you elaborate more on this point? As pointed out in line 039-045, you seem to prime the model to focus on the common ground by forcing the model to generate both same-side and opposite opinions. In this case, would it make more sense to compute losses only on the \"common phrases/words/sequences\" instead of the full sequence of opposite opinions?\n\nI don't quite follow how you construct {X_S, X_T} (line 177-192). For example, 1) for the LRC-arousal approach, what do you mean by \"compose the set with tuples of the highest arousal score and the lowest arousal score\"? My imagination is that, similar to LR-arousal, you would first pick a pair of L and R articles that have a high difference in valence score, but then how do you include the C article? Also, why do you want to include tuples that have the lowest arousal score? 2) for the informational approach, what do you mean by \"high differences between the numbers of unique tokens\"? For example, which pair would you choose: pair 1: L and R each has 100 unique tokens, and pair 2: L and R each has 10 unique tokens? In both cases, the difference is 0 but I guess you would choose pair 1 since more \"information bias\" is observed. Similarly, how do you include C article.\n\nFrom my reading of your manuscript, it's unclear to me whether PEGASUSMULTI and BARTMULTI are fine-tuned on AllSides since you said \"compare with off-the-shelf multi-document summarization models\" (line 155-156). Please correct me if I am wrong. However, if not fine-tuned, it's advised to include results for fine-tuned BART and PEGASUS. Meanwhile, which underlying model are you using when you report \"+LR\" results in the bottom panel in Table 1.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback and your effort. We would like to answer your questions and clarify some misunderstandings as below. \n\n**1. About \"polarity minimization loss\"** \n\nPolarity minimization loss is for explicitly teaching how politically polarized articles are mapped to each other in both directions (e.g., both L→ R, and R→ L simultaneously), thus the model can learn how an issue is represented in opposite polarities. Since our  L_polar allows mapping from each other, it will ultimately force the model to learn to generate the common ground (“common phrases/words/sequences”) between two extreme political views (i.e., polarized ends). \n\nTo respectfully explain the distinction of our work, the previous works [3,4] focus on ideology prediction (if they are left-, right-, or center-leaning) or stance prediction -- which is polarity detection. On the other hand, our work focuses on **generating** a neural article from polarized articles. Although it is an important task to detect political ideology (classification) [3,4], it is also important to generate a neutral summary (generation) from political ideologies to mitigate polarization. Thank you for introducing [3,4] works. As advised, we will move the Related Work section to the main body with an additional page and similarities/differences between our work and existing work, including [3,4] (political ideology/stance detection works).\n\n[3] We Can Detect Your Bias: Predicting the Political Ideology of News Articles (Baly et al., EMNLP 2020)\n\n[4] POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection(Liu et al., Findings 2022)\n\n**2. Construction of {X_S, X_T} (line 177-192)**\n\nWe would like to clarify the misunderstanding caused by our writing as follows with some examples: \n\n- **2-1) LRC-Arousal:** Your understanding about LR-arousal is correct. Similarly, the same logic applies to LRC-Arousal. Yet, unlike LR-arousal, we utilize all three of X^L, X^R, X^C source articles (Here, X^C is also a biased article as explained in lines 135-137). In other words, to pick pairs of articles that have high differences in arousal score, we calculate differences of all combinations, which are {L, R}, {L, C}, {R, C} whereas LR-only calculator {L, R}.\n\n- **2-2) Informational approach:** We counted the number of unique tokens between two pairs and selected those that have the highest count combined. We should modify our writing to “highest difference in information based on the sum of unique tokens from each article from the pair.” \n\n**3. Data for Baselines & Additional Baselines**\n\n- **3-1) PEGASUSMULTI, BARTMULTI baselines:** are finetuned on the multi-news dataset (Fabbri et al., 2019) as stated in line 159, thus, we called it off-the-shelf models. \n\n-  **3-2) Results of fine-tuned BART & PEGASUS:** As advised, we would like to share the results of the fine-tuned BART, and fine-tuned PEGASUS as follows: \n|  Model  |  Arousal+  | Arousal- | Arousal_sum | BLEU | Bertscore |\n|--|:--:|:--:|:--:|:--:|:--:|\n|  BART-finetuned    | 1.86   | 1.00   |  2.85  |  11.57  |  70.10%  |\n|  PEGASUS-finetuned  |  2.18   |  1.12  |  3.3   | 11.26   |  68.41%  |\n| + LR-VALENCE (Ours)  | 1.57 |  0.91 |  2.48 |  10.62 |  69.67% |\n| + LR-AROUSAL (Ours)  |  1.18 |  0.62 | 1.80 |  8.84 |  69.49% |\n| + LR-INFO (Ours)  |  1.08 |  0.70 |  1.78 |  9.31 |  70.19% |\nWe will update our manuscript with the results.\n\n- **3-3) the base model for Table 1:** We utilize the BART-large model for Table 1. We will remove the “+” sign to remove the confusion, instead, we will add BART_{LR-XYZ} to make it clear about the base model.\n\n- **3-4) PRIMERA result:**\nAdditionally, as suggested by the reviewer, we also share the suggested the result on PRIMERA model:\n|  Model  |  Arousal+  | Arousal- | Arousal_sum | BLEU | Bertscore |\n|--|:--:|:--:|:--:|:--:|:--:|\n|  PRIMERA    | 3.99   | 1.95   |  5.94  |  8.31  |  65.32%  |\n\n[Reference] \nMulti-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model (Fabbri et al., ACL 2019)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The motivation behind \"polarity minimization loss\" is interesting which can boil down to the two aspects of framing bias: lexical and information bias (though it would be good to cite some recent NLP working along this line, e.g., \"In Plain Sight: Media Bias Through the Lens of Factual Reporting\", \"Context in Informational Bias Detection\").\n\nThe proposed method outperforms all baselines in terms of both Framing Bias metric (assessing how much bias is reduced) and Salient Info metric (assessing how much info is preserved in the generated summary, e.g., BLEU).\n\nInteresting ablation study: Polarity shift in uni-direction. This ablation study shows that it's important to consider both directions in framing bias mitigation, and the sample generation in Table 2 well echoes this point. It would be nice to see more examples of such kind.", "reasons_to_reject": "The intuition behind the proposed \"polarity minimization loss\" is not well articulated. Similar \"polarity minimization\" idea is seen in [3][4], though they are not working on summarization tasks, where they train the model to minimize the representation difference between two articles of similar ideology but maximize between articles of opposite ideologies. Their approaches seem to be more **explicit** by forcing the model to directly learn the common ground; whereas this paper seems to **indirectly** learn the common ground by forcing the model to generate R article using L article, which doesn't seem to be sound. Also, see \"Questions For The Authors\" where I suggest another more explicit way to learn the common ground, which should have been included as a baseline.\n\nExperimental design can be improved: 1) include more recent multi-doc summarization technique (e.g., PRIMERA), 2) include fine-tuned BART and PEGASUS (see details in Questions For The Authors)\n\nSome texts are hard to follow (e.g., line 177-192), and see details in Questions For The Authors. The way the author constructs the pairs/tuples for polarity minimization loss is unclear, which makes it hard to judge the soundness of the model design. \n\nWriting can be improved since there are some minor typos (see Typos Grammar Style And Presentation Improvements).", "questions_to_authors": "Following weakness 1 about the intuition, could you elaborate more on this point? As pointed out in line 039-045, you seem to prime the model to focus on the common ground by forcing the model to generate both same-side and opposite opinions. In this case, would it make more sense to compute losses only on the \"common phrases/words/sequences\" instead of the full sequence of opposite opinions?\n\nI don't quite follow how you construct {X_S, X_T} (line 177-192). For example, 1) for the LRC-arousal approach, what do you mean by \"compose the set with tuples of the highest arousal score and the lowest arousal score\"? My imagination is that, similar to LR-arousal, you would first pick a pair of L and R articles that have a high difference in valence score, but then how do you include the C article? Also, why do you want to include tuples that have the lowest arousal score? 2) for the informational approach, what do you mean by \"high differences between the numbers of unique tokens\"? For example, which pair would you choose: pair 1: L and R each has 100 unique tokens, and pair 2: L and R each has 10 unique tokens? In both cases, the difference is 0 but I guess you would choose pair 1 since more \"information bias\" is observed. Similarly, how do you include C article.\n\nFrom my reading of your manuscript, it's unclear to me whether PEGASUSMULTI and BARTMULTI are fine-tuned on AllSides since you said \"compare with off-the-shelf multi-document summarization models\" (line 155-156). Please correct me if I am wrong. However, if not fine-tuned, it's advised to include results for fine-tuned BART and PEGASUS. Meanwhile, which underlying model are you using when you report \"+LR\" results in the bottom panel in Table 1.", "paper_topic_and_main_contributions": "This paper identify the framing bias issue in media outlets such that different media outlets have a skewed selection of language and information when reporting on the same news event. Inspired by lexical and informational bias in Framing Bias, the author proposes Polarity Regularized Model by designing Polarity minimization Loss which is to train a summarization model to generate articles from one end of the political spectrum to another end (i.e., generating an article of opposite ideology). The author conducts experiments to show the effectiveness of their model by evaluating on two aspects: bias mitigation and information preservation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] In Plain Sight: Media Bias Through the Lens of Factual Reporting\n\n[2] Context in Informational Bias Detection\n\n[3] We Can Detect Your Bias: Predicting the Political Ideology of News Articles\n\n[4] POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection\n\n[5] PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 043: \"in Fig 1\" not \"Fig 2\"\n\nLine 77: with polarized articles (no \"a\")\n\nLine 83: \"defined in eq. 1\"\n\nLine 116: I suppose you mean \"bound to\"\n\nLine 124: remove the comma at the beginning. \n\nIf accepted, please move Appendix A (related work) to the main content, and include missing references mentioned above as well as discuss the similarities/differences between your work and existing work."}}, {"review_id": "eEEptz95yB", "reviewer": "Reviewer_rWMe", "summary": "", "questions": "(A) Why did you compare the model only to general-purpose summarisation models rather than to the other neutralisation model for the benchmarks, given yours is not the first? Why did you conduct human evaluation compared to a model you do not provide benchmark results for?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback and will improve the writing based on your feedback.  \n\nTo answer your question, we compared with general-purpose summarization models to show the importance of polarization minimization loss in removing framing bias. Following your suggestion, we have added the comparison to the other neutralization model as follows:\n\n|  Model  |  Arousal+  | Arousal- | Arousal_sum | BLEU | Bertscore |\n|--|:--:|:--:|:--:|:--:|:--:|\n|  BART-finetuned    | 1.86   | 1.00   |  2.85  |  11.57  |  70.10%  |\n\nAs advised, we will add the results from the benchmark for the complete comparison in the final manuscript with an additional page.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well motivated with a very clear introduction, setting out the problem and introducing key concepts to readers unfamiliar with the topic. \n\nThe authors use existing benchmarks to measure improved neutrality and key information retention, and supplement this with a human evaluation that demonstrates that perceived polarity has been reduced. Showing improvements across a range of metrics supports their claims of the effectiveness of their approach. The ablation experiments provide further evidence for their claim that learning bidirectional polarity shifts are crucial for achieving neutrality. \n\n", "reasons_to_reject": "A concern I have is the selective comparison to general-purpose summarisation models. In Related Works in Appendix A, the authors mention a model produced by Lee et al. (2022) -  but give no reason why this is not considered as a baseline for comparison. Another model from Lee at al. (2022) (BARTfinetuned) is used for human evaluation, but not for the benchmark comparison. A glance at this paper suggests comparable results, if not improved, so the reason for omissions is unclear. \n", "questions_to_authors": "(A) Why did you compare the model only to general-purpose summarisation models rather than to the other neutralisation model for the benchmarks, given yours is not the first? Why did you conduct human evaluation compared to a model you do not provide benchmark results for?", "paper_topic_and_main_contributions": "The authors present the case for the need for depolarising news summarisation to improve access to neutral news information. They propose a loss function with two components; the first aims to generate a summary article conditioned on polarised source articles; the second aims to optimise the (bi-directional) mapping between opposed articles (in order to identify the “common ground”). The authors show variations of their methodology outperform general-purpose summarisation models with regards to neutralising valence and retaining salient information.  They also demonstrate that their methodology can outperform a summariser finetuned on sets of polarised articles and neutral summaries, with regards to subjective polarity, suggesting their technique is effective. \n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None that I know of, besides my point above about choice of baselines. "}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 2 - I think this section would benefit from some accompanying “plain English” to make the purpose of each element of the loss function clearer. \n\nLine 095 - erroneous capitalisation \n\nLine 203 - quite much → relatively well \n\nLine 211 - I found the paragraph “Effective learning with extreme polarities” hard to follow. The first sentence - “we could investigate” - could imply that you did not then investigate this topic, which disrupts the flow a little. The main argument seems to be that learning is effective from extreme polarities, but one of the examples you give is the LRC-arousal model which has a three-way comparison. \n\nLine 515 - citations given with no date. \n\nFigure 2 - as this is not referenced in the discussion, I would suggest moving this to the Appendix to give yourself more room in the main body of the paper if needed.  \n\n"}}]

Author rebuttal:

Meta-review: This paper presents a method for obtaining neutral articles from a multi-document summarization system by introducing a polarity minimization loss into the overall loss.

Like at least one of the reviewers, I found the intuition for the specific loss function neither well-described nor well-motivated, nor did I find the authors' rebuttal helpful in further elucidating this part.  This is the crux of the work, so this was rather important.

Nevertheless, this paper still presents a worthwhile problem with a novel loss function and good—if sometimes hard-to-follow—analysis.
Decision: Accept-Findings