SlowGuess/NAACL_2022_OCR huggingface dataset:
Ideology Prediction	Stance Detection	All avg
YT (cmt.)	CongS	HP	AllS	YT (user)	TW	Ideo. avg	SEval (seen)	SEval (unseen)	Basil (sent.)	VAST	Basil (art.)	Stan. avg
Baselines
SVM	65.34	71.31	61.25	52.51	66.49	42.85	59.96	51.18	32.89	51.08	39.54	30.77	41.09	51.38
BERT	64.64	65.88	48.42	60.88	65.24	44.20	58.21	65.07	40.39	62.81	70.53	45.61	56.88	57.61
RoBERTa	66.72	67.25	60.43	74.75	67.98	48.90	64.34	70.15	63.08	68.16	76.25	41.36	63.80	64.09
Baly et al. (2020)
with Original Data	65.42	66.74	58.37	72.89	70.47	44.95	63.14	68.66	56.29	61.30	75.57	37.98	59.96	61.69
with BIGNewsBLN	68.57	70.39	71.24	76.47	74.74	47.38	68.13	65.84	49.54	60.60	75.03	41.84	58.57	63.79
Our models with triplet loss objective only
Ideology Obj.	66.20	68.18	64.15	76.52	68.15	42.66	64.31	68.78	59.61	64.18	76.03	44.94	62.71	63.58
Story Obj.	66.09	69.11	56.70	74.59	68.89	46.53	63.65	69.02	63.54	67.21	76.66	53.16	65.92	64.68
Ideology Obj. + Story Obj.	68.91	69.10	63.08	76.23	77.58	48.98	67.31	69.66	63.17	64.37	76.18	47.01	64.08	65.84
Our models with masked language model objective only
Random	67.82	70.32	60.59	73.54	70.77	44.62	64.61	69.16	60.39	69.94	77.11	39.16	63.15	63.95
Upsamp. Ent.	69.06	70.32	60.09	70.89	71.40	47.16	64.82	69.81	63.08	69.49	76.76	46.46	65.12	64.96
Upsamp. Sentiment	67.41	70.03	56.05	72.35	74.93	48.15	64.82	70.09	60.81	71.28	76.61	44.42	64.64	64.74
Upsamp. Ent. + Sentiment	68.31	71.42	58.02	71.90	71.04	47.31	64.67	69.25	62.84	69.23	77.10	43.16	64.32	64.51
POLITICS	67.83*	70.86	70.25*	74.93	78.73*	48.92	68.59	69.41	61.26	73.41*	76.73*	51.94*	66.55	67.66
Table 3: Macro F1 scores on 11 evaluation tasks (average of 5 runs). Tasks are sorted by text length, short to long, within each group. "All avg" is the average of all 11 tasks. Best results are in bold and second best are underlined. Our models with triplet-loss objectives that outperform RoBERTa are in blue. Our models with specialized sampling methods that outperform the one with vanilla MLM (Random) are in green. POLITICS uses Ideology + Story Obj. and Upsamp. Ent. + Sentiment. Results where POLITICS outperforms all baselines are in red, with * indicating statistical significance (Mann-Whitney U test; Mann and Whitney, 1947,  $p \leq 0.05$ ). Standard deviations (std) are reported in Table A11. The range of std over tasks is [0.31, 3.42] for POLITICS, and [0.48, 7.35] for RoBERTa.
(positive, negative, and neutral) except for BASIL (sent.) that labels positive or negative.
- BASIL (Fan et al., 2019) contains news articles with annotations on authors' stances towards entities. BASIL (sent.) and BASIL (art.) are prediction tasks at sentence and article-levels.
- VAST (Allaway and McKeown, 2020) collects online comments from "Room for Debate", with stances labeled towards the debate topic.
- SemEval (Mohammad et al., 2016a) is a shared task on detecting stances in tweets. We consider two setups to predict on seen, i.e. SEval (seen), and unseen, i.e., SEval (unseen), entities.
We consider three baselines. First, we train a linear SVM using unigram and bigram features for each task, since it is a common baseline in political science (Yu et al., 2008; Diermeier et al., 2012). Hyperparameters and feature selection are described in Table A8. We further compare with BERT and RoBERTa, following the standard fine-tuning process for ideology prediction tasks and using the prompt described in §5.4 for stance detection.
5.3 Model Variants
We consider several variants of POLITICS. First, using triplet loss objective only, we experiment on
models trained with ideology objective (Ideology Obj.), story objective (Story Obj.), or both.
Next, we continue pretaining RoBERTa with MLM objective only, using vanilla MLM objective (Random), entity focused objective (Upsamp. Ent.), sentiment focused objective (Upsamp. Sentiment), or upsampling both entity and sentiment.
5.4 Fine-tuning Procedure
We fine-tune each neural model for up to 10 epochs, with early stopping enabled. We select the best finetuned model on validation sets using F1. Details of experimental setups are in Table A7.
Ideology Prediction. We follow common practice of using the [CLS] token for standard finetuning (Devlin et al., 2019). For Twitter and YouTube User data, we encode them using sliding windows and aggregate by mean pooling.
Stance Detection. We follow Schick and Schütze (2021) on using prompts to fine-tune models for stance detection. We curate 11 prompts (in Table A6) and choose the best one based on the average F1 by RoBERTa on all stance detection tasks:
$$
p [ \text {S E P} ] \text {T h e s t a n c e t o w a r d s} \{\text {t a r g e t} \} i s [ \text {M A S K} ].
$$
The model is trained to predict [MASK] for stance, conditioned on the input  $p$  and {target}.
5.5 Main Results
Table 3 presents F1 scores on all tasks. POLITICS achieves the best overall average F1 score across the board,  $3.6\%$  better than the strongest baseline, RoBERTa. More importantly, POLITICS alone outperforms all three baselines listed in §5.2 on 8 out of 11 tasks, including more than  $10\%$  of improvement for ideology labeling on Hyperpartisan and Youtube user-level. We attribute the performance gain to our proposed ideology-driven pretraining objective, which helps capture partisan content. Note that, on some tasks, other model variants lead POLITICS by a small margin, and this may be of interest to practitioners performing specific tasks.
We further compare with the model proposed by Baly et al. (2020), which also leverages triplet loss as pretraining objective but on articles of the same topics. We implement two versions of their model, using the original data released by Baly et al.  $(2020)^{7}$  and our BIGNEWSBLN. First, pretraining on our BIGNEWSBLN yields better results on ideology prediction tasks than using the original data, indicating the value of BIGNEWSBLN. Second, using the triple construction method by Baly et al. (2020) with BIGNEWSBLN does not generalize well on the stance detection task, compared to POLITICS and its variants. This highlights the advantage of our objectives that enable content comparison among articles of the same stories.
Moreover, our ideology-driven objectives help acquire knowledge needed to discern ideology as well as stance detection. When equipping the RoBERTa model with ideology and story objectives but no MLM objective, it achieves the second best overall performance.
Next, focusing on entities better identifies stance. Simply continuing training RoBERTa with vanilla MLM objective (Random) does not yield performance gain on stance detection, while our upsampling methods make a difference, i.e., increasing sampling ratios of entities improves F1 by  $2\%$ .
Comparisons with Previous State-of-the-arts. Using the original binary prediction setup (i.e., hyperpartisan or not) on Hyperpartisan data (Kiesel et al., 2019), POLITICS obtains an accuracy of 85.2, leading previous state-of-the-art results by at least 3 points, as shown in Table 4.
POLITICS achieves an F1 of 77.0 on the origi-
Models	Hyperpartisan
Acc.	Precision	Recall	F1
Jiang et al., 2019 (ELMo+CNN)	82.2	87.1	75.5	80.9
Srivastava et al., 2019 (Logistic Regression)	82.0	81.5	82.8	82.1
Hanawa et al., 2019 (BERT)	80.9	82.3	78.7	80.5
Isbister and Johansson, 2019 (SVM)	80.6	85.8	73.2	79.0
Yeh et al., 2019 (ULMFiT)	80.3	79.3	81.8	80.6
RoBERTa	84.3	87.2	80.6	83.7
POLITICS	85.2	86.3	83.7	84.9
Table 4: Comparison with the previous state-of-the-art models on Hyperpartisan using the original binary prediction setup. Best results are in bold. POLITICS obtains the best accuracy, recall and F1. Note, precision, recall, and F1 are measured for the hyperpartisan class.
Figure 3: Macro F1 aggregated over tasks of different formality, training size, document length and aggregation method (single post vs. posts by each user). POLITICS performs better on handling formal language, small training sets, and longer text.
nal VAST data where the previous state-of-the-art model obtained 69.2 (Jayaram and Allaway, 2021). On SemEval, POLITICS yields an F1 of 71.3 where the best performance is 76.5 by Al-Ghadir et al. (2021). Notably, we adopt one single classifier in our setup, while they include separate models for different prediction targets, which have been shown to outperform one single classifier (Mohammad et al., 2016a). Full comparisons with previous methods are included in Appendix F. For other tasks, there is no direct comparison as the datasets are either originally used for different prediction tasks (e.g., Basil is used for detecting media bias spans) or newly collected by this work.
On Texts of Different Characteristics. Based on Table 2, we further study the model's performance on data of different properties: language formality, training size, document length, and aggregation level. As shown in Figure 3, with each property (concrete criterion in Appendix E), we divide tasks into two categories. POLITICS yields greater improvements on more formal and longer text, since pretraining is done on news articles. POLITICS is also more robust to training sets with
$^{7}$ https://github.com/ramybaly/ Article-Bias-Prediction.
Ideology Prediction	Stance Detection	All avg
YT (cmt.)	CongS	HP	AllS	YT (user)	TW	Ideo. avg	SEval (seen)	SEval (unseen)	Basil (sent.)	VAST	Basil (art.)	Stan. avg
POLITICS	67.83	70.86	70.25	74.93	78.73	48.92	68.59	69.41	61.26	73.41	76.73	51.94	66.55	67.66
No Ideology Obj.	-3.78	-2.17	-16.35	-3.28	-12.54	-3.43	-6.93	-0.38	-0.83	-4.22	-0.45	-16.01	-4.38	-5.77
No Story Obj.	+1.98	+0.64	-0.72	+0.70	+0.29	-1.78	+0.19	-1.23	+2.94	-3.36	-0.87	-10.75	-2.66	-1.11
No Upsamp. Ent.	+0.18	-0.65	-0.05	+0.55	-0.29	-1.20	-0.24	+0.62	-0.67	-3.74	-0.55	-1.20	-1.11	-0.64
No Upsamp. Sentiment	+0.75	-0.28	+0.22	-1.27	-0.11	-1.40	-0.35	-0.84	+1.67	-3.91	-1.10	+1.44	-0.55	-0.44
POLITICS + Ideo. Pred.	+1.46	+1.10	-1.01	+4.72	+2.02	-3.96	+0.72	+0.41	-0.52	-3.82	+0.12	-3.10	-1.38	-0.23
Table 5: Ablation study results on POLITICS. POLITICS + Ideo. Pred.: triplet-loss objective is replaced with a hard label prediction objective on ideology of articles (left vs. right). Best results are in bold. Darker red shows greater improvements. Darker blue indicates larger performance drop. The ideology objective contributes the most to POLITICS, followed by the story objective.
Figure 4: Average of ideology prediction and stance detection performances with few-shot learning. POLITICS uniformly outperforms RoBERTa and it being continued pretrained with vanilla MLM (Random).
small sizes, showing the potential effectiveness in few-shot learning, which is echoed in §6.1.
6 Further Analyses
6.1 Few-shot Learning
We first fine-tune all PLMs on small numbers of samples. POLITICS consistently outperforms the two counterparts on both tasks, using small training sets (Figure 4). More importantly, naively training RoBERTa on the large BIGNEWSBLN does not help ideology prediction. By contrast, our ideology-driven objective can better capture ideology than the baselines, even when using only 16 samples for fine-tuning on the ideology tasks.
6.2 Ablation Study on POLITICS
We show the impact of removing each ideology-driven pretraining objective and upsampling strategy from POLITICS in Table 5. First, removing the ideology objective results in the most loss on both tasks. This again demonstrates the effective
ness of our triplet-loss formulation over same-story articles. Removing the story objective also hurts the overall performance by  $1\%$  but improves the ideology prediction marginally. This shows that the story objective functions as an auxiliary constraint to avoid over-fitting on the "shortcuts" for discerning ideologies. Moreover, removing upsampling strategies generally weakens POLITICS's performance, but only to a limited extent.
We also experiment with a setup with hard-ideology learning (i.e., directly predicting the ideology of each article without using triplet-loss objectives). Not surprisingly, this variant (POLITICS +Ideo. Pred.) outperforms POLITICS on ideology prediction since it can directly learn ideology from the annotated labels. However, it has been overfitted to ideology prediction tasks and lacks generalizability, thus yields worse performance on stance detection.
6.3 Visualizing Attentions
On the Hyperpartisan task, we visualize the last layer's attention weights between the [CLS] token and all other tokens by POLITICS and RoBERTa pretrained with vanilla MLM on BIGNEWSBLN (Random). We randomly sample 20 test articles, and for 13 of them, POLITICS is able to capture salient entities, events, and sentiments in the text whereas Random cannot. We present one example in Figure 5 where POLITICS captures "Ashley Judd", "the worst", and "Trump". More examples are given in Appendix G. This finding confirms that our ideology-driven objective and upsampling strategies can help the model focus more on entities of political interest as well as better recognize sentiments.
Figure 5: Last layer attention scores between [CLS] token and other input tokens (aggregated over all heads). POLITICS captures "Ashley Judd", "worst", and "Trump".
Figure 6: Model perplexities on 30K validation articles in BIGNEWSBLN. Perplexities do not drop sharply on POLITICS compared with RoBERTa being continued pretrained with MLM objective (Random), suggesting POLITICS can yield superior predictive performance while not overfitting with ideological languages.
6.4 POLITICS on Different Ideologies
Finally, we measure whether PLMs would acquire ideological bias as measured by whether they fit with languages used by a specific ideology. Concretely, we follow Salazar et al. (2020) to evaluate PLMs on 30K held-out articles of different ideologies from BIGNewsBLN with pseudoperplexity. For efficiency, we estimate the pseudo log-likelihood based on 200 random tokens in each article as used by Wang and Cho (2019). As illustrated in Figure 6, while MLM objective (Random) is effective at fitting a corpus, i.e., having the lowest perplexities, triplet-loss objectives act as regularizers during pretraining, shown by the higher perplexity of POLITICS compared to Random. Interestingly, we find center and right articles have lower perplexity than that of left articles. We hypothesize that it relates to political science findings that, over recent periods of political polarization in US, Republicans have become somewhat more coherent and similar than Democrats (Grossmann and Hopkins, 2016; Benkler et al., 2018), and are thus easier to predict.
We study the problem of training general-purpose tools for ideology content understanding and prediction. We present POLITICS, trained with novel ideology-driven pretraining objectives based on the comparisons of same-story articles written by media outlets of different ideologies. To facilitate model training, we also collect a large-scale dataset, BIGNews, consisting of news articles of different ideological leanings. Experiments on diverse datasets for ideology prediction and stance detection tasks show that POLITICS outperforms strong baselines, even with a limited amount of labeled samples for training, and state-of-the-art models.
This work is supported in part through National Science Foundation under grants IIS-2100885 and IIS-2127747, and computational resources and services provided by Advanced Research Computing (ARC), a division of Information and Technology Services (ITS) at the University of Michigan, Ann Arbor. We appreciate the anonymous reviewers for their helpful comments. We thank the members of the LAUNCH group at the University of Michigan for discussions and suggestions. We also thank Changyuan Qiu for helping collect the AllSides articles, Siqi Wu for giving us access to the Youtube comments, and Daniel Preotiuc-Pietro for sharing the Twitter user accounts used in the original study to allow the collection of the corresponding users' tweets in this work.
8.1 BIGNEWS Collection
All news articles were collected in a manner consistent with the terms of use of the original sources
as well as the intellectual property and the privacy rights of the original authors of the texts, i.e., source owners. During data collection, the authors honored privacy rights of content creators, thus did not collect any sensitive information that can reveal their identities. All participants involved in the process have completed human subjects research training at their affiliated institutions. We also consulted Section  $107^{8}$  of the U.S. Copyright Act and ensured that our collection action fell under the fair use category.
8.2 Dataset Usage
All of the newly collected datasets in this work will be made available upon request. Pretraining corpus details are included in Section 3. The other seven datasets used for downstream evaluation are obtained in the following ways. CongS, HP, BASIL, VAST and SEval are acquired by direct download. CongS is released under the ODC-BY 1.0 license (free to share, create, and adapt). HP and SEval are developed in shared tasks by the NLP community, which allow the use of copyrighted material without permission from the copyright holder for research purposes (Escartín et al., 2017). For VAST, the author explicitly states "We make our dataset and models available for use". BASIL is developed by the last author and her collaborators. For YT and TW, we consult with the corresponding authors and obtain the datasets by agreeing that we will not further distribute them. Dataset details are listed in Section 5.1 and Appendix D.
8.3 Benefit and Potential Misuse
Intended use. The models developed in this work can assist the general public to measure and understand ideological language used in diverse genres of texts. For example, POLITICS can help the general public know where their representatives stand on key issues. Our experiments in Section 5 demonstrate how POLITICS would be deployed in real life when handling applications in both ideology prediction and stance detection. We deem that our extensive experiments have covered the major usage of POLITICS.
Failure mode is defined as situations where POLITICS fails to correctly predict the ideology of an individual or a given text. In such cases, POLITICS might deliver misinformation or cause mis
understanding towards a political figure or a policy. For vulnerable populations (e.g., people who maybe not be able to make the right judgements), the harm could be tremendously magnified when they fail to interpret the model outputs or blindly trust machine responses. Ideally, the interpretation of our model's predictions should be carried out within the broader context of the source text.
Misuse potential. Users may mistakenly take the machine prediction as a golden rule or a fact. We would recommend any politics-related machine learning models, including ours, put up an "use with caution" message to encourage users to check more sources or consult political science experts to reduce the risk of being misled by single source. Moreover, POLITICS might also be misused to label people with a specific political leaning that they do not want to be associated with. We suggest that when in use the tools should be accompanied with descriptions about their limitations and imperfect performance, as well as allow users to opt out from being the subjects of measurement.
Potential limitation. Although multiple genres are considered, the genre coverage is not exhaustive, and does not include other trending media or content of different modalities for expressing opinions, such as TV transcripts, images, and videos. Thus, the predictive performance of POLITICS may still be under investigated. Further, in downstream evaluation, POLITICS is only trained and tested in the same domain, so its cross-genre ability needs further evaluation.
Bias Mitigation. During data preprocessing, we create BIGNEWSBLN to ensure that all ideologies have almost equal presence to minimize potential bias. POLITICS is not designed to encode bias. In Figure 6, the discrepancy in perplexities among different ideologies is more related to the greater coherence among Republicans than Democrats, rather than POLITICS encoding biased knowledge.
In conclusion, there is no greater than minimal risk/harm introduced by either BIGNEWSBLN or POLITICS. However, to discourage the misuse, we will always warn users that model predictions are for informational purpose only and users should always resort to the broader context to reduce the risk of absorbing biased information.
$^{8}$ https://www copyright.gov/title17/92chap1.html#107.
Abdulrahman I. Al-Ghadir, Aqil M. Azmi, and Amir Hussain. 2021. A novel approach to stance detection in social media tweets by fusing ranked lists and sentiments. Inf. Fusion, 67:29-40.
Emily Allaway and Kathleen McKeown. 2020. Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8913-8931, Online. Association for Computational Linguistics.
Stephen Ansolabehere, Jonathan Rodden, and James M Snyder. 2008. The strength of issues: Using multiple measures to gauge preference stability, ideological constraint, and issue voting. American Political Science Review, 102(2):215-232.
Samin Aref and Zachary P Neal. 2021. Identifying hidden coalitions in the us house of representatives by optimally partitioning signed networks based on generalized balance. Scientific reports, 11(1):1-9.
Isabelle Augenstein, Tim Roktaschel, Andreas Vlachos, and Kalina Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 876-885. The Association for Computational Linguistics.
Ramy Baly, Giovanni Da San Martino, James R. Glass, and Preslav Nakov. 2020. We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4982-4991. Association for Computational Linguistics.
Pablo Barbera, John T. Jost, Jonathan Nagler, Joshua A. Tucker, and Richard Bonneau. 2015. Tweeting from left to right: Is online political communication more than an echo chamber? Psychological Science, 26(10):1531-1542.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3613-3618. Association for Computational Linguistics.
Yochai Benkler, Robert Faris, and Hal Roberts. 2018. Network propaganda: Manipulation, disinformation, and radicalization in American politics. Oxford University Press.
Adam Bonica. 2013. Mapping the ideological marketplace. ERN: Models of Political Processes: Rent-Seeking.