SlowGuess/NAACL_2022_OCR huggingface dataset:
The model is trained with binary cross-entropy as  $\mathcal{L}^{(\mathrm{pred})}$  and Adam (Kingma and Ba, 2015) as the optimizer. Since  $\mathcal{L}^{(\mathrm{reg})}$  is non-differentiable, we use proximal gradient descent (Parikh and Boyd, 2013). We approximate the weighted proximal operator of the  $\ell_1 / \ell_2$  norm using the Newton-Raphson algorithm (Deleu and Bengio, 2021). We use area under the curve (AUC) for model evaluation. We refer to our model as SF-SGAE (Salience/Framing Sparse Graph Auto-Encoder).
Intrinsic evaluation. We compare SF-SGAE against three ablated models: one where we use only salience, i.e.,  $\psi_l = s$  (S-SGAE), one where
Model	2013	2014	2015	2016	2017	2018	2019	μ ± σ
SF-SGAE	.890	.895	.895	.923	.937	.908	.934	.912±.018
S-SGAE	.886	.890	.853	.875	.894	.864	.925	.884±.022
F-SGAE	.875	.893	.878	.885	.905	.875	.917	.890±.015
SF-SLAE	.653	.810	.754	.781	.764	.729	.752	.749±.046
SF-GAE	.829	.797	.871	.916	.898	.866	.933	.873±.044
Table 2: Test performance (AUC). SF-SGAE outperforms S-SGAE, F-SGAE, and SF-SLAE. It performs similarly to or better than SF-GAE despite using only a fraction of concepts. Best score per column in gray. See Appendix A.4 for dev performance.
we use only framing, i.e.,  $\psi_l = f$  (F-SGAE), and one where we use both types of information but replace the graph convolutions with linear layers (SF-SLAE). Furthermore, we implement a model that is identical to SF-SGAE but does not use sparsity, i.e.,  $|\mathcal{C}|$  is not reduced (SF-GAE).
SF-SGAE clearly—and substantially on some years—outperforms the ablated models (Table 2). This shows that jointly modeling salience and framing captures polarization better than only modeling one of the two. Between S-SGAE and F-SGAE, there is no clear winner, although F-SGAE performs slightly better overall. SF-SLAE performs substantially worse than all other models, which indicates that the Laplacian smoothing in the form of graph convolutions is a crucial component of the model. SF-SGAE also outperforms SF-GAE on test, suggesting that  $\mathcal{C}^*$  allows for a more robust generalization than the larger but noisier  $\mathcal{C}$ .
How does the sparsity threshold  $\theta_{|\mathcal{C}|}$  impact model performance? The answer to this question indicates how many concepts are required to capture the central ideological divides in the data. We vary  $0 \leq \theta_{|\mathcal{C}|} \leq 1000$  and measure the performance (AUC) of the four sparsifying models on dev (Figure 3). First, we find that for the models using graph convolutions, reducing  $|\mathcal{C}|$  to approximately 200 concepts does not hurt performance. For the model without graph convolution, on the other hand, performance starts to drop already around 400 concepts. This makes intuitive sense: given that the graph convolutions act as a form of smoothing, less concepts are needed for a reliable feature vector for each subreddit. Second, the advantage of SF-SGAE lies not only in its higher performance in the sparse regime but also in its ability to reduce  $|\mathcal{C}|$  much further than any of the other models given a performance threshold. This again demonstrates that a joint model of salience and framing results in richer information, making it possible to reduce the number of concepts further.
Figure 3: Impact of sparsity threshold  $\theta_{|\mathcal{C}|}$  on performance (AUC) on dev for 2016. SF-SGAE performs better than any other model in the sparse regime ( $\theta_{|\mathcal{C}|} \leq 200$ ), showing that it better captures polarization. Plots for all years are provided in Appendix A.5.
Extrinsic evaluation. The fact that SLAP4SLIP is a minimally supervised framework makes it challenging to evaluate the correctness of our model. While the performance on link prediction indicates how well  $\mathcal{C}^*$  captures the polarized structure of the social network, it is not a direct measure of ideological polarization. There is also no ground-truth dataset against which  $\mathcal{C}^*$  could be compared. We therefore devise an alternative extrinsic evaluation method. Specifically, we use DW-NOMINATE (Poole and Rosenthal, 1985, 1997), a quantitative measure of the ideological polarization of members of the US Congress based on their roll-call voting behavior. Recently, a large dataset of DW-NOMINATE scores has been made publicly available (Lewis et al., 2021).
We first create a dataset with all comments from subreddits dedicated to US state-level politics (e.g., TexasPolitics) in 2018. We discard subreddits with less than 250 comments, resulting in a set of 28 subreddits. For each state, we then compute the average DW-NOMINATE score of its representatives in the lower house of the 116th US Congress (elected in November 2018). The average DW-NOMINATE is a continuous measure of the ideological leaning of a state and ranges between  $-0.399$  for Massachusetts (very liberal) and 0.467 for Idaho (very conservative). Notice that this score reflects the state-level voting shares to a certain extent (since it is averaged over the representatives elected by a state) while at the same time being more fine-grained (since representatives of the same party can differ ideologically). Finally, for each state-level subreddit  $v_{i}$ , we extract  $s(v_{i},c_{j})$  for (i) the  $d$  concepts  $c_{j}$  from  $\mathcal{C}^*$  with the highest frequency across all state-level subreddits and (ii)
<sup>7</sup>We choose the larger subreddit in the case of multiple state-level subreddits. We retrieve the subreddits from the Pushshift Reddit Dataset (Baumgartner et al., 2020).
Figure 4: Performance on ideology prediction. The figure shows the distribution of accuracies for 100 models trained with relative frequencies of the concepts from  $\mathcal{C}^*$  versus the concepts from  $\mathcal{C} \setminus \mathcal{C}^*$ . The concepts from  $\mathcal{C}^*$  result in overall much higher accuracies, indicating that they better capture ideological polarization.
$d$  frequency-matched concepts  $c_{j}$  sampled from  $\mathcal{C} \setminus \mathcal{C}^{*}$ .<sup>8</sup> We set  $d = 5$ .<sup>9</sup> If the concepts from  $\mathcal{C}^{*}$  are better predictors of the average DW-NOMINATE scores than the concepts from  $\mathcal{C} \setminus \mathcal{C}^{*}$ , this indicates that the model has learned a correct split into more versus less polarized concepts.
To test this empirically, we compute the absolute value of Pearson's  $r$  between  $s(v_{i},c_{j})$  and the DW-NOMINATE scores. We find a higher correlation for the concepts from  $\mathcal{C}^*$  ( $\mu = 0.285$ ,  $\sigma = 0.062$ ) than for the concepts from  $\mathcal{C} \setminus \mathcal{C}^*$  ( $\mu = 0.126$ ,  $\sigma = 0.121$ ), a difference that is shown to be significant ( $p < 0.05$ ) by a two-tailed  $t$ -test. This indicates that the concepts in  $\mathcal{C}^*$  reflect the polarization of US politics better than the concepts in  $\mathcal{C} \setminus \mathcal{C}^*$ .
Furthermore, we try whether it is possible to predict the DW-NOMINATE scores from the relative concept frequencies. Specifically, we binarize the DW-NOMINATE scores by dividing them into the upper and lower half, thus resulting in a balanced dataset of more conservative and more liberal subreddits. We then train  $\ell_{2}$ -regularized logistic regression classifiers using the relative frequencies of the concepts from  $\mathcal{C}^*$  and  $\mathcal{C} \setminus \mathcal{C}^*$  as features. Since the dataset is small, we train 100 models on different random (label-stratified) splits of the subreddits into  $50\%$  training and  $50\%$  test. The models based on the concepts from  $\mathcal{C}^*$  have substantially higher accuracies ( $\mu = 0.657$ ,  $\sigma = 0.122$ ) than the models based on the concepts from  $\mathcal{C} \setminus \mathcal{C}^*$  ( $\mu = 0.491$ ,  $\sigma = 0.109$ ), a difference that is again shown to be significant ( $p < 0.01$ ) by a two-tailed  $t$ -test (Figure 4). We interpret this as further evidence that the concepts in  $\mathcal{C}^*$  (as opposed to the concepts in  $\mathcal{C} \setminus \mathcal{C}^*$ ) capture ideological polarization.
Year	α(cj) = 0	0 < α(cj) < 1	α(cj) = 1
2013	aca (l/b)	deregulation (l/b)	gay marriage
bush (a/s)	fox news (f/c)	gerrymandering
tax (c/h)	gun control (l/b)	surveillance
2016	julian (l/b)	cuba (a/s)	collusion
russian (s/d)	GOP (s/d)	fake news
trump voters (c/h)	nationalism (l/b)	reagan
2019	fact (a/s)	congress (a/s)	donald
illegal (a/s)	white (s/d)	fascist
mainstream (s/d)	women (c/h)	lefties
Table 3: Example concepts with  $\alpha^{(c_j)}$  values of 1, 0, and in between. For  $\alpha^{(c_j)} < 1$ , we also provide the moral foundation  $m_k$  with maximum  $\pi_k^{(c_j)}$ . c/h: care/harm; f/c: fairness/cheating; l/b: loyalty/betrayal; a/s: authority/subversion; s/d: sanctity/degradation. aca stands for Affordable Care Act (also known as Obama-care). julian refers to Julian Assange.
Qualitative analysis. We analyze which concepts are selected by SF-SGAE (Table 3). Many concepts in  $\mathcal{C}^*$  are names of politicians (e.g., bush, donald) and designations of parties and political orientations (e.g.,GOP,lefties). Furthermore,  $\mathcal{C}^*$  contains concepts related to contested political issues. While many of these issues (e.g., gay marriage,gun control) have been shown to be characterized by polarized online discussions before (Lai et al., 2015; Demszky et al., 2019), others (e.g.,deregulation,mainstream) have been in the focus to a lesser degree, highlighting SLAP4SLIP's potential as an exploratory framework.
The design of our model also allows us to analyze in what way the concepts are polarized. To do so, we first examine the weight distribution of  $\alpha^{(c_j)}$  for all  $c_{j}\in \mathcal{C}^{*}$ . We notice that for the majority of concepts (roughly  $80\%$  )  $\alpha^{(c_j)} = 1$ , i.e., the model uses only information about salience. Concepts with  $\alpha^{(c_j)} = 1$  tend to be of immediate relevance for certain ideologies, leading to higher frequencies in relevant network regions. For example, in communist subreddits, discussion often revolves around fascism as the central opposing ideology, leading to higher frequencies of fascist than in other parts of the network (Figure 1a).
For concepts with  $\alpha^{(c_j)}\neq 1$  , we can analyze which moral foundation has the largest  $\pi_k^{(c_j)}$  . This moral foundation constitutes the basis for intersubreddit differences in highlighting certain aspects of the concepts, which can be measured by  $|p_{k}(v_{i},c_{j})|$  , i.e., the absolute value of the projection of the concept embedding onto the  $m_{k}$  subspace. For example, within the sanctity/degradation sub
For  $\mathcal{C}^*$ , we only consider concepts for which  $\alpha^{(c_j)} = 1$ , i.e., the polarization is captured by  $s(v_i, c_j)$  alone.
9Results are robust with respect to the exact selection of  $d$
Concept cj	Small value of |pk(vi, cj)|	Large value of |pk(vi, cj)|
Subreddit vi	Example	Subreddit vi	Example
bush (2013, a/s)	Freethought	This reminds me of what I read about the way the Bush administration worked religious quotes into military briefings.	Anarchy101	What's stopping from murderers becoming presidents? Oh wait... US has Obama, previously had Bush.
trump voters (2016, c/h)	Conservative	Trump voters, and people on the right in general, believe this is a grand country with little institutional racism left.	socialism	Trump voters have a hate boner for the Clinton that they've maintained since their 92 campaign.
mainstream (2019, s/d)	Kamala	She's good at making progressive ideas sound like reasonable mainstream policies, which is the best of both worlds.	TheNewRight	I think mainstream media has infected your brain with such rot that it effects your emotions.
Table 4: Polarization in framing. The table provides contexts for three concepts with  $\alpha^{(c_j)} = 0$ , both for subreddits with weak framing ( $|p_k(v_i, c_j)|$  small) and subreddits with strong framing ( $|p_k(v_i, c_j)|$  large) in the relevant moral subspace. c/h: care/harm; a/s: authority/subversion; s/d: sanctity/degradation.
space (the subspace with maximal  $\pi_k^{(c_j)}$ ), many subreddits frame the concept mainstream in neutral terms. Right-wing subreddits, on the other hand, frame it as something degenerate, particularly in the context of media (Figure 1b, Table 4), reflecting appeals to discredit mainstream media reporting of political news (Lee and Hosam, 2020).
To get a more global picture of which moral subspaces are most important for the polarized framing, we examine the learned values of  $\pi_k^{(c_j)}$  (Section 4) for all concepts with  $\alpha^{(c_j)} \neq 1$ . The three moral foundations that most frequently have the highest  $\pi_k^{(c_j)}$  value are loyalty/betrayal (30%), sanctity/degradation (27%), and authority/subversion (21%), followed by care/harm (18%) and fairness/cheating (3%). Interestingly, loyalty/betrayal, sanctity/degradation, and authority/subversion are the three moral foundations with the greatest democrat-republican differences (Haidt and Graham, 2007; Graham et al., 2009), indicating that the US two-party system is a central axis for the polarized framing of concepts on Reddit.
Ideological dynamics. The embeddings  $\mathbf{Z}$  learned by our model are subreddit representations that combine linguistic information with network information. Here, we analyze what types of temporal ideological dynamics are captured by  $\mathbf{Z}$ .
We map the embeddings  $\mathbf{Z}$  for all years into a common embedding space using orthogonal Procrustes (Schonemann, 1966; Hamilton et al., 2016) and measure for each subreddit the cosine similarities between its embedding in the first year and its embeddings in all subsequent years. If the resulting time series of cosine similarities is continuously decreasing, this indicates a change in ideology. To detect such shifts automatically, we compute for
each subreddit Pearson's  $r$  between the time series of years and the time series of cosine similarities. Examining the subreddits with the most extreme negative values of  $r$ , we observe that most of them experienced a pronounced shift in their ideological orientation (Figure 5). Specifically, the subreddits move from a relatively moderate to a more extreme position in ideology space, either right-wing (e.g., FreeSpeech, POLITIC) or left-wing (e.g., Sino). This pattern suggests that the subreddits have ideologically radicalized over time (Grover and Mark, 2019; Youngblood, 2020).
The success of our method depends on how accurately polarization is reflected by the network, which means that care must be taken during network selection (explicit networks) and construction (implicit networks). For example, user overlap on Reddit can also be due to conflict between subreddits (Datta et al., 2017; Kumar et al., 2018; Datta and Adar, 2019). While we do not find this to affect our results, it might be a limitation if the degree of homophily in the network is too low.
This paper only applies SLAP4SLIP to networks with communities as nodes and edges based on user overlap between the communities. However, the kind of clusteredness our method draws upon has been shown to be a property of various types of social networks, including social networks with individual users as nodes such as Twitter (Conover et al., 2011; Himelboim et al., 2013). We expect SLAP4SLIP to be a suitable framework for finding polarized concepts in these cases, too.
Figure 5: Example subreddits with a pronounced shift in their ideology over time. Orange: Sino, a subreddit originally devoted to geopolitics that moved to a more left-wing position; green and red: FreeSpeech and POLITIC, two originally moderate subreddits that moved to a more right-wing position.
We introduce SLAP4SLIP (Sparse LLanguage Properties for Social Link Prediction), a novel framework for finding linguistic features maximally informative about the structure of a social network, and show that it can be used to detect polarized concepts. We model polarization along the dimensions of salience and framing. While we only address polarized concepts in this paper, the general nature of the framework makes it possible to apply it in diverse scenarios involving linguistic data attached to social networks (e.g., to find the most pronounced topical differences in citation networks). We see our study as an exciting first step towards bringing together computational social science research on online polarization, NLP work on political language, and graph-based deep learning.
As part of our model, we use contextualized word embeddings to model the polarized framing of concepts. However, contextualized word embeddings are known to be biased (Basta et al., 2019; Zhao et al., 2019; Bender et al., 2021), which bears the risk of impacting our results. We see this as an important research question for future work.
The user base of Reddit has been shown to be disproportionately young and male compared to the general population of the US (Shatz, 2017). We acknowledge that this limits the generalizability of our results, and we try to be particularly careful when drawing conclusions in the paper.
This work was funded by the European Research Council (#740516) and the Engineering and Physical Sciences Research Council (EP/T023333/1). The first author was supported by the German Academic Scholarship Foundation and the Arts and Humanities Research Council. We thank the reviewers for their helpful comments. We are grateful to Jeremy Frimer for making the Moral Foundations Dictionary z-scores available to us.
Lada A. Adamic and Natalie Glance. 2005. The political blogosphere and the 2004 U.S. Election: Divided they blog. In International Workshop on Link Discovery (LinkKDD) 3.
Jose M. Alvarez and Mathieu Salzmann. 2016. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems (NIPS) 30.
Jisun An, Haewoon Kwak, and Yong-Yeol Ahn. 2018. SemAxis: A lightweight framework to characterize domain-specific word semantics beyond sentiment. In Annual Meeting of the Association for Computational Linguistics (ACL) 56.
Jisun An, Haewoon Kwak, Oliver Posegga, and Andreas Jungherr. 2019. Political discussions in homogeneous and cross-cutting communication spaces. In International AAAI Conference on Web and Social Media (ICWSM) 13.
Eytan Bakshy, Solomon Messing, and Lada A. Adamic. 2015. Exposure to ideologically diverse news and
opinion on Facebook. Science, 384(6239):1130-1132.
Christine Basta, Marta R. Costa-jussà, and Noe Casas. 2019. Evaluating the underlying gender bias in contextualized word embeddings. In Workshop on Gender Bias in Natural Language Processing (GeBNLP) 1.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift Reddit dataset. In International AAAI Conference on Web and Social Media (ICWSM) 14.
Emily Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In ACM Conference on Fairness, Accountability, and Transparency (FAccT) 21.
Dimitri P. Bertsekas. 1982. Constrained optimization and Lagrange multiplier methods. Academic Press, New York, NY.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems (NIPS) 30.
Aaron Bramson, Patrick Grim, Daniel J. Singer, Steven Fisher, William Berger, Graham Sack, and Carissa Flocken. 2016. Disambiguation of social polarization concepts and measures. The Journal of Mathematical Sociology, 40(2):80-111.
Dallas Card, Justin H. Gross, Amber E. Boydstun, and Noah A. Smith. 2016. Analyzing framing through the casts of characters in the news. In *Conference on Empirical Methods in Natural Language Processing* (EMNLP) 2016.
Michael Conover, Jacob Ratkiewicz, Matthew Francisco, Bruno Goncalves, Alessandro Flammini, and Filippo Menczer. 2011. Political polarization on Twitter. In International AAAI Conference on Web and Social Media (ICWSM) 5.
Srayan Datta and Eytan Adar. 2019. Extracting inter-community conflicts in reddit. In International AAAI Conference on Web and Social Media (ICWSM) 13.
Srayan Datta, Chanda Phelan, and Eytan Adar. 2017. Identifying misaligned inter-group links and communities. Proceedings of the ACM on Human-Computer Interaction, 1:1-23.
Tristan Deleu and Yoshua Bengio. 2021. Structured sparsity inducing adaptive optimizers for deep learning. In arXiv 2102.03869.
Dorottya Demszky, Nikhil Garg, Rob Voigt, James Zou, Matthew Gentzkow, Jesse Shapiro, and Dan Jurafsky. 2019. Analyzing polarization in social media:
Method and application to Tweets on 21 mass shootings. In Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HTL) 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HTL) 2019.
Jesse Dodge, Roy Schwartz, Hao Peng, and Noah A. Smith. 2019. RNN architecture learning with sparse regularization. In Conference on Empirical Methods in Natural Language Processing (EMNLP) 2019.
Xiaowen Dong, Dorina Thanou, Laura Toni, Michael Bronstein, and Pascal Frossard. 2020. Graph signal processing for machine learning: A review and new perspectives. IEEE Signal Processing Magazine, 37(6):117-127.
James N. Druckman. 2001. The implications of framing effects for citizen competence. Political Behavior, 23(2):225-256.
Jacob Eisenstein, Noah A. Smith, and Eric Xing. 2011. Discovering sociolinguistic associations with structured sparsity. In Annual Meeting of the Association for Computational Linguistics (ACL) 49.
Robert M. Entman. 1993. Framing: Toward clarification of a fractured paradigm. Journal of Communication, 43(4):51-58.
Heinz Eulau. 1955. Perceptions of class and party in voting behavior. American Political Science Review, 49(2):364-384.
Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018. Framing and agenda-setting in Russian news: A computational analysis of intricate political strategies. In Conference on Empirical Methods in Natural Language Processing (EMNLP) 2018.
Jeremy Frimer, Jonathan Haidt, Jesse Graham, Morteza Dehghani, and Reihane Boghrati. 2017. Moral foundations dictionaries for linguistic analyses, 2.0.
Dean Fulgoni, Jordan Carpenter, Lyle H. Ungar, and Daniel Preotiac-Pietro. 2016. An empirical exploration of moral foundations theory in partisan news sources. In International Conference on Language Resources and Evaluation (LREC) 10.
David Garcia, Adiya Abisheva, Simon Schweighofer, Uwe Serdult, and Frank Schweitzer. 2015. Ideological and temporal components of network polarization in online political participatory media. *Policy and Internet*, 7(1):46-79.