Venue: NAACL 2025

# Investigating Human Values in Online Communities

 Nadav Borenstein\({}^{1}\) Arnav Arora\({}^{1}\) Lucie-Aimee Kaffee\({}^{2}\) Isabelle Augenstein\({}^{1}\)

\({}^{1}\)University of Copenhagen \(~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}



which can be used to identify interesting phenomena to qualitatively study when studying online behaviour and communities. Specifically, we follow Schwartz's Theory of Human Values (Schwartz, 1994), due to its wide adoption as a value framework in the social sciences as well as in Natural Language Processing, and apply it on Reddit. Reddit is conceptually based on named communities, or, **subreddits**, and studying the values they exhibit is particularly interesting for studying online behaviour as the discussions are already segregated by topic or sub-communities. For instance, by examining the r/teenagers subreddit, we can gain insights into adolescent perspectives, while the r/ConservativeValues allows us to study the impact of conservative values on worldviews.

To extract values from Reddit, we train supervised value extraction models to classify the _presence_ and _polarity_ of Schwartz values in text and apply them to Reddit posts. We then evaluate the models, validating their effectiveness and addressing limitations in prior work. Subsequently, we use these models to infer the expressed values of 9 million user posts and comments across 11,616 most popular subreddits on Reddit.

Through such a computational value analysis, we can analyse digital trace and behaviour data at scale, complementing social science studies by highlighting patterns as well as outliers needing further study. As an instance of this, our analysis demonstrates that people contributing to r/feminism exhibit high values in _self-direction_, as substantiated by previous studies. We also demonstrate the prevalence of high traditional values in conservative US states. In sum, our **contributions** are:

* A flexible method of conducting large-scale value relevance and polarity analysis to complement social science research on online communities;
* Our analysis of values at a large scale confirms previously recorded phenomena and unveils novel insights, indicating that our method can complement traditional methods in understanding societal phenomena;
* We release our dataset of 12k subreddits with their corresponding Schwartz values for future analysis.1

Footnote 1: [https://github.com/copenlu/HumanValu](https://github.com/copenlu/HumanValu) es

Although our findings offer unique insights into the values of individuals from various cultures, the focus of our paper is the analysis of values present in _online communities_. Hence the subject of our study is the communities themselves and how they operate, rather than the broader cultural backgrounds of their members, which we do not measure directly. Our methods are designed to complement, not replace, traditional survey-based approaches for studying values, by providing additional perspectives from digital trace data.

## 2 Related Work and Background

### Schwartz's Values Framework

Values represent a crucial aspect of human nature. According to Schwartz (2012), _"A person's value priority or hierarchy profoundly affects his or her attitudes, beliefs, and traits, making it one core component of personality."_ Schwartz (1994) define values as (1) concepts or beliefs, that (2) pertain to desirable end states or behaviors, (3) transcend specific situations, (4) guide selection or evaluation of behavior and events, and (5) are ordered by relative importance. Based on this, they outline ten basic human values: _Security, Conformity, Tradition, Benevolence, Universalism, Self-direction, Stimulation, Hedonism, Achievement, Power_. We provide descriptions of each of the values in Tab. 2. The values were originally applied to measure differences in values across cultures (Schwartz, 1994).

\begin{table}
\begin{tabular}{l l} \hline Value & Description \\ \hline _Power_ & Social status and prestige, control or dominance over people and resources \\ _Ackivement_ & Personal success through demonstrating competence according to social standards. \\ _Hedonism_ & Pleasure and sensous gratification for oneself. \\ _Simulation_ & Excitement, novelty, and challenge in life. \\ _Self-direction_ & Independent thought and action-choosing, creating, exploring. \\ _Universalism_ & Understanding, appreciation, tolerance, and protection for the welfare of all people and for nature. \\ _Benevolence_ & Preservation and enhancement of the welfare of people with whom one is in frequent personal contact. \\ _Tradition_ & Respect, commitment, and acceptance of the customs and ideas that traditional culture or religion provide. \\ _Conformity_ & Restraint of actions, inclinations, and impulses likely to upset or harm others and violate social expectations or norms. \\ _Security_ & Safety, harmony, and stability of society, of relationships, and of self. \\ \hline \end{tabular}
\end{table}
Table 2: The ten Schwartz values and their meaning; descriptions from Schwartz (1994) 

The framework is suggested as a tool to study populations rather than individual people, hence it serves as a suitable tool for the analysis of online communities. We leverage the fact that a person's identity and values are often reflected in the linguistic choices they make (Jaffe, 2009; Norton, 1997) to analyse values embedded in text.

### Values and Natural Language Processing

Recently, there have been a number of studies exploring values and morals using NLP. In analysing values and language on social media, Ponizovskiy et al. (2020) released a Schwartz value dictionary. Boyd et al. (2021) show the promise of free-text survey response and Facebook data for value extraction. There have also been studies exploring morals and norms in text. Trager et al. (2022) release the Moral Foundations Reddit Corpus with 16k Reddit comments; Roy et al. (2021) similarly study morality in political tweets. Havaldar et al. (2024) study the presence of values in a geolocated Twitter corpus using a lexicon-based approach, finding a lack of correlation with survey data. The closest to our study is van der Meer et al. (2023), who train a value extraction model based on datasets from Qiu et al. (2022); Kiesel et al. (2022). Using the model, they analyse values at an individual user level to understand disagreement in online discussions. However, a key limitations in their work is only detecting the _presence_ of a value, neglecting the _polarity_ of the discussion towards that value. In our work, we overcome this limitation by training an additional value **stance** model. Further, we perform our analysis at the community level to understand the values of communities rather than individuals, bringing it closer to the original framework outlined by Schwartz designed to understand cultural values.

### Studying Online Communities

Much work has been done to explore Reddit and its user base. Reddit users' personalities have been studied (Gjurkovic and Snajder, 2018) as well as their mental health (Zomick et al., 2019; Turcan and McKeown, 2019; Chancellor et al., 2018). Previous work has also studied Reddit by focusing on events affecting community dynamics (Newell et al., 2016; Agrawal et al., 2023). For instance, Chandrasekharan et al. (2022) examine the effect of content moderation on two controversial subreddits. Soliman et al. (2019) study the characteristics and differences between left-leaning and right-leaning political subreddits. Closer to our work, Weld et al. (2023) craft a taxonomy of community values and associate them with specific subreddits. However, their methodology, while high quality, involves collecting values through self-reporting questionnaires of limited size and suffers from selection bias, a limitation we address in this paper. Finally, studies have also explored modelling community norms (Park et al., 2024) and detecting their violations (Cheriyan et al., 2021).

## 3 Method

This section details the collection and processing of our Reddit dataset, and our approach to training and evaluating the Schwartz values extractor model.

### Data Collection

We download an image of Reddit posts and comments authored between January and August 20222 using Pushshift's API.3 We filter out posts and comments with fewer than ten words or with fewer than ten upvotes to reduce possible noise from low-quality text. We then merge the lists of posts and comments by subreddit, not further distinguishing between them for our analysis, and filter out subreddits that are tagged NSFW,4 have fewer than 5,000 subscribers, or fewer than 250 content examples. Finally, we down-sample large subreddits to 1,000 random samples due to computational constraints and remove non-English content. This process results in a dataset \(\mathcal{D}\) of 11,616 unique subreddits. See Tab. 3 for dataset statistics.

Footnote 2: System limitations pose restrictions on the date range we can process – an image of a single month is over 300GB.

Footnote 3: [https://pushshift.io/](https://pushshift.io/)

Footnote 4: Not Safe For Work

### Value Extraction

For extracting Schwartz values and their polarity from text, we first extract the _relevance_ of a post or comment with each Schwartz value, then use a stance model to extract the _polarity_ of the sentence towards the relevant value.

\begin{table}
\begin{tabular}{l|c c c c c c} \hline  & Avg & std & \multicolumn{3}{c}{Percenteille} & Total \\ \cline{4-7}  & & & 25 & 50 & 75 & \\ \hline c.p.s & 765.4 & 264.8 & 517 & 918 & 996 & 8,888,535 \\ w.p.c & 62.8 & 148.0 & 15 & 26 & 58 & 558,327,230 \\ \hline \end{tabular}
\end{table}
Table 3: Statistics of the Reddit dataset we analyse. c.p.s is content item (posts and comments) per subreddit, and w.p.c is word per content item.

 

Value RelevanceFor extracting relevance, our approach is similar to van der Meer et al. (2023) in training a neural Schwartz Values relevance classifier. We use a DeBERTa model He et al. (2021), over RoBERTa, due its improved performance and speed. As a single post can potentially express several values simultaneously, the model is trained in a _multi-label setting_, predicting a vector of 10 independent probabilities for each input. We fine-tune the classifier on the concatenation of two supervised Schwartz values datasets, ValueNet Qiu et al. (2022) and ValueArg Kiesel et al. (2022). Given a labelled triplet \((s,v,y)\), with \(s\) being a string of text, \(v\) the name of one of the ten Schwartz values (Tab. 2), and \(y\in\{0,1\}\), we construct the input \(x=[\texttt{CLS}]v[\texttt{SEP}]s[\texttt{SEP}]\) and train the model to predict \(p(y|x)\). Our classifier's performance on this dataset is similar to the figures reported by van der Meer et al. (2023), namely a macro-averaged \(F_{1}\) score of \(0.76\) on the merged ValueArg and ValueNet datasets. Similar to van der Meer et al. (2023), we collapse the two non-neutral labels (-1 and 1) in ValueNet into a single positive class.5 This model is trained, therefore, to predict the _presence_ of a value \(v\) within a string \(s\).

Footnote 5: We do this to follow the format of the ValueArg dataset, which only contains annotations for value relevance.

Value StancePrior work on value extraction neglects to model the polarity of content towards values, substantially limiting the insights one can draw. To detect the polarity of a comment or a post \(s\) towards a value \(v\), we train a separate stance model on ValueNet's non-neutral labels. Specifically, given a triplet \((s,v,y)\) where \(v\) is a value expressed by \(s\) and \(y\in\{-1,1\}\), we fit the stance model to predict \(p_{\text{stanc}}(y|x)\), where \(x\) is constructed as described above. We refer readers to App. C.1 for additional training details about the classifiers.

When extracting values of comments or posts from Reddit, we first use the relevance model to predict the probabilities for all Schwartz values for a given input instance, resulting in a vector \(u_{\text{rel}}\in[0,1]^{10}\). Specifically, given a string \(s\), we construct \(u_{\text{rel}}=[p(y^{1}|x^{1}),...,p(y^{10}|x^{10})]\) by replacing \(v^{k}\) in the construction of \(x^{k}=[\texttt{CLS}]v^{k}[\texttt{SEP}]s[\texttt{SEP}]\) with each one of the ten possible Schwartz values. That is, each entry in the vector is the independent probability that \(s\) expresses the value \(v^{k}\), supporting a multi-label approach. Then, for each \(k\) with \(u_{\text{rel}}^{k}>0.5\) (i.e., input text \(s\) expresses the value \(v^{k}\) with a probability greater than 0.5), we predict \(p_{\text{stanc}}(y^{k}|x^{k})\), (i.e., the polarity of input text \(s\) towards the value \(v^{k}\)). Thus, we construct a vector of probabilities \(u_{\text{stanc}}\) of dimensionality \(10\), where each entry \(u_{\text{stanc}}^{k}\) is either in \([-1,1]\) or Null (if \(s\) does not express the value \(v^{k}\) greater than 0.5). Tab. 10 in App. B.3 contains a sample of Reddit content and the associated Value Extractor model predictions.

### Evaluation of Value Extraction

An important limitation of van der Meer et al. (2023) is the lack of direct evaluation of the value extraction model. The need for this becomes particularly pronounced considering the large domain shift of applying the model--which, similarly to us, was trained on debating data--to analyze content on platforms like Reddit. Therefore, we conduct a thorough evaluation to test the model's capabilities of extracting values from Reddit content.

Ideally, we would want to assess the model's performance using a large, annotated dataset of randomly sampled Reddit posts. However, this is impractical because randomly selected posts are unlikely to contain any Schwartz values. Consequently, annotating these posts would yield a dataset with only a few positive examples. To address this challenge, we evaluate the Value Extraction models directly.

Relevance Model EvaluationWe evaluate the relevance model by first using it to label 10,000 posts and 10,000 comments for predicting the presence of values. Thereafter, for each value \(v\), we sample three posts or comments: one with high model confidence for the presence of the value (above 0.8), one with medium confidence (0.4-0.6), and one with low confidence (below 0.2). Three

\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline  & AC & BE & CO & HE & PO & SE & SD & ST & TR & UN \\ \hline Spearman \(\rho\) & 0.73 & 0.66 & 0.66 & 0.56 & 0.8 & 0.46 & 0.5 & 0.6 & 0.73 \\ NDCG@1 & 0.89 & 0.81 & 0.93 & 0.96 & 0.76 & 0.93 & 0.74 & 0.86 & 0.96 & 0.84 \\ \hline \end{tabular}
\end{table}
Table 4: Average Spearman’s \(\rho\) and NDCG@1 between the three annotators of the relevance model, per-value breakdown. AC=achievement, BE=benevolence, CO=conformity, HE=hedonism, PQ=power, SE=security, SD=self-direction, ST=stimulation, TR=tradition, UN=universalism.

\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline AC & BE & CO & HE & PO & SE & SD & ST & TR & UN \\ \hline
0.51 & 0.57 & 0.61 & 0.77 & 0.30 & 0.45 & -0.27 & 0.77 & 0.67 & 0.26 \\ \hline \end{tabular}
\end{table}
Table 5: Cohen’s Kappa between the two annotators of the stance model, per-value breakdown.

 

annotators then rank these comments/posts based on which are more related to value \(v\), regardless of polarity. We repeat this process five times per value, totalling 50 rankings per annotator. See App. A for annotation guidelines and dataset samples.

For annotator agreement, we calculated averaged Spearman's \(\rho\), which looks at correlation amongst ranks. The agreement we found was 0.63, which is reasonable for such a subjective task.6 Certain values (e.g., _security_, _universalsism_) showed better agreement than others (e.g., _self-direction_). A full breakdown is available in Tab. 4. For assessing model performance, we assigned a gold pseudo-ranking to each sample by averaging the annotators' rankings. We evaluate the relevance model's performance by comparing the model's predicted ranking to this gold standard. The average Spearman's \(\rho\) we obtain is \(0.51\) (again, with values such as _security_ outperforming other values such as _self-direction_). Looking at the top ranked content, the NDCG@1 we obtain is an impressive \(0.87\), demonstrating that high certainty predictions made by our classifier are highly relevant to their corresponding value. With this, we can extract comments or posts relevant to certain values from the larger set. See App. B.2 for examples of model misclassification and value breakdown.

Footnote 6: Kiesel et al. (2022) found the averaged agreement amongst annotators for value annotation to be \(\alpha=0.49\).

Stance Model EvaluationWe first sample 20 posts or comments per value \(v\) where the relevance model predicted the presence of \(v\) with high confidence. Two annotators then label the samples for stance, choosing between positive, negative, or neutral/unrelated (i.e., where no stance is clearly expressed or the relevance model misclassified the sample). The annotators achieved a Cohen's Kappa score of 0.47, representing moderate agreement, highlighting the challenging nature of the task (see Tab. 5 for a per-value breakdown). However, post discussion on the disagreements, the annotators were able to converge on decisions for each sample, thus obtaining gold labels for the 200 samples. Finally, we apply the stance model to predict the stances of all positive or negative samples. The model predicts the stance towards values with an \(F_{1}\) score of \(0.72\). Given the subjective nature of the task, where one has to predict stance towards an abstract concept like human values and one where annotators often disagreed, we believe the model performs reasonably well. To highlight some of the trickier cases where the model is failing, Tab. 9 in App. B.2 contains examples of model misclassification and breakdown into values.

### Assigning Values to Subreddits

We assign each subreddit \(\mathcal{S}\in\mathcal{D}\) a single vector of Schwartz probabilities \(u_{\text{rel}}(\mathcal{S})\) and a single vector of stances \(u_{\text{stance}}(\mathcal{S})\). Given \(\mathcal{S}\) with content \((c_{i})|i\in\mathcal{S}\), where \(c_{i}\) is either a post or a comment, we predict \(u_{\text{rel}}(c_{i})\) and \(u_{\text{stance}}(c_{i})\) from \(c_{i}\) using the process above. Finally, we calculate \(u_{\text{rel}}(\mathcal{S})\) by averaging over the predicted vectors \(u_{\text{rel}}(c_{i})\):

\[u_{\text{rel}}(\mathcal{S})=\frac{1}{|\mathcal{S}|}\sum_{i\in S}u_{\text{rel}} (c_{i})\]

We construct \(u_{\text{stance}}(\mathcal{S})\) similarly, considering only non-Null entries. That is, each entry in \(u^{k}_{\text{stance}}(\mathcal{S})\) is computed as

\[u^{k}_{\text{stance}}(\mathcal{S})=\frac{1}{|\mathcal{S}^{k}|}\sum_{i\in S^{k} }u^{k}_{\text{stance}}(c_{i})\text{,}\]

where \(\mathcal{S}^{k}=\{i\in\mathcal{S}|u^{k}_{\text{stance}}(c_{i})\neq\text{Null}\}\).

Tab. 6 details global statistics for the predicted relevance and stance for each Schwartz value across the entire subreddit dataset.

## 4 Experiments

Our experiments serve two primary goals: first, to validate our approach by comparing the values extracted using our method to existing knowledge. Second, to demonstrate the extensibility of our method to new topics that have not been previously investigated in the social sciences. We conduct qualitative and quantitative evaluations of our approach in SS4.1 and SS4.2, to validate our method. Then, to uncover interesting phenomena and demonstrate the utility of our method, we investigate subreddits with differing opinions on controversial topics and compare our findings with

\begin{table}
\begin{tabular}{l c r r r} \hline \hline Value & Relevance & Relevance & Stance & Stance \\  & mean & std & mean & std \\ \hline Tradition & 0.11 & 0.03 & -0.00 & 0.04 \\ Benefovence & 0.30 & 0.12 & 0.01 & 0.29 \\ Conformity & 0.23 & 0.06 & -0.17 & 0.32 \\ Hedonism & 0.21 & 0.08 & 0.45 & 0.36 \\ Power & 0.17 & 0.05 & 0.04 & 0.16 \\ Achievement & 0.26 & 0.09 & 0.47 & 0.30 \\ Self-direction & 0.19 & 0.09 & 0.25 & 0.31 \\ Universalism & 0.43 & 0.10 & 0.39 & 0.27 \\ Stimulation & 0.23 & 0.08 & 0.36 & 0.36 \\ Security & 0.21 & 0.06 & 0.06 & 0.18 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Global statistics of the ten Schwartz values across the entire subreddit dataset.

 

existing studies. Finally, in SS4.4, we correlate values extracted using our method to values gained through traditional approaches like surveys and questionnaires.

### Qualitative Analysis

We start by assessing how well our values extraction model performs on our Reddit dataset.

**High relevance** Intuitively, certain Schwartz values are expected to be distinctly present in specific communities; e.g. _tradition_ in religion-related communities. Therefore, we find subreddits with particularly strong signals of individual Schwartz values. For each value, we sort the subreddits' values probability vectors \(u_{\text{rel}}(\mathcal{S})\) by their entry corresponding to the particular value. Tab. 1 lists a sample of the top subreddits for each value; Tab. 11 in App. B lists the top 20 subreddits per value. Many of the subreddits collected for each value seem to be intuitively related to it (e.g., r/resumes with _achievement_, r/conservation with _Universalism_). The results are encouraging, demonstrating the effectiveness of our approach and its potential for conducting interesting analyses.

**Strong stances** Equivalently, we can also investigate which subreddits express the strongest _stances_ towards each value. For each value, we sort the subreddits' stance vectors \(u_{\text{stance}}(\mathcal{S})\) by their entry corresponding to the particular value. Tab. 12 in App. B enumerates the 10 subreddits with the highest _positive_ and _negative_ stance towards each value. Again, while some of the listed subreddits are intuitive (e.g., r/migraine having a negative "hedonism" stance and r/raisingkids having a strong positive stance towards "achievement"), other subreddits are more surprising (r/TheHague with a positive stance towards "hedonism").

**Value magnitude** We hypothesise that online communities differ not only in the set of values they express but also in the total **magnitude** of expressed values. Online communities pertaining to more objective topics (e.g., linear algebra) should express fewer Schwartz values than communities dedicated to subjective or controversial topics (e.g., politics). To test this, for each subreddit \(\mathcal{S}\in\mathcal{D}\) we calculate \(\text{mag}(\mathcal{S})=|u_{\text{rel}}(\mathcal{S})|_{2}\), the total magnitude of values expressed in the subreddit. The 20 subreddits with the highest and lowest value magnitudes are detailed in Tab. 13 in App. B. Subreddits with particularly high magnitudes are those communities that focus on debating, discussion, or emotional narratives. Examples of these include r/change-myview, r/Adoption, and r/PoliticalDiscussion. Conversely, subreddits with notably low magnitudes are generally more objective and neutral in tone. These include r/crystalgrowing and r/whatishsfish. Some subreddits that exhibit low-value magnitudes are dedicated to sharing photos taken by community members, e.g., r/astrophotography.

### Quantitative Analysis

We hypothesise that similar subreddit communities share similar values, and systematically investigate this using empirical evidence. We define three measures of similarity between subreddits:

**Value similarly**. We define \(\sigma_{\mathrm{val}}(\mathcal{S}_{1},\mathcal{S}_{2})=\cos{(u_{\text{rel}}( \mathcal{S}_{1})||u_{\text{stance}}(\mathcal{S}_{1}),u_{\text{rel}}(\mathcal{ S}_{2})||u_{\text{stance}}(\mathcal{S}_{2}))}\), the cosine similarity between the concatenation of the relevance and stance vectors of the two subreddits.7

Footnote 7: We experimented with other formulations for calculating \(\sigma_{\mathrm{val}}(\mathcal{S}_{1},\mathcal{S}_{2})\), and arrived to similar results.

**Semantic similarity**. We scrape the **public description** of each subreddit from its page (see Fig. 5 in App. B), and embed these natural language descriptions using a sentence transformer8 to construct a semantic embedding vector \(\mathbf{e}_{\mathcal{S}}\) for each subreddit \(\mathcal{S}^{9}\). We now define \(\sigma_{\mathrm{sem}}(\mathcal{S}_{1},\mathcal{S}_{2})=\cos(\mathbf{e}_{\mathcal{ S}1},\mathbf{e}_{\mathcal{S}2})\).

Footnote 8: [https://www.sbert.net/](https://www.sbert.net/). We use the all-mpnet-base-v2 pretrained model.

**User similarity** For each pair of subreddits, we define their user similarity to be the _overlap coefficient_ between the users of the two subreddits. That is,

\[\sigma_{\mathrm{usr}}(\mathcal{S}_{1},\mathcal{S}_{2})=\frac{|U(\mathcal{S}_{ 1})\cap U(\mathcal{S}_{2})|}{\min(|U(\mathcal{S}_{1})|,|U(\mathcal{S}_{2})|)}\]

Where \(U(\mathcal{S})\) is the set of the subreddit's members.10

Footnote 9: A limitation of this approach is that, occasionally, the public description is rather vague. However, a manual analysis we conducted reveals that this is rare.

To answer the question of if similar subreddits share similar values, we correlate \(\sigma_{\mathrm{val}}\) with \(\sigma_{\mathrm{sem}}\) and \(\sigma_{\mathrm{usx}}\) as follows. First, for each subreddit \(\mathcal{S}\) we find \(\overline{\mathcal{S}}_{\text{semi}}=\text{argmax}_{\mathcal{S}^{\prime}} \left[\sigma_{\mathrm{sem}}(\mathcal{S},\mathcal{S}^{\prime})\right]\) and \(\overline{\mathcal{S}}_{\text{com}}=\text{argmax}_{\mathcal{S}^{\prime}} \left[\sigma_{\mathrm{user}}(\mathcal{S},\mathcal{S}^{\prime})\right]\). If similar subreddits share a similar set of values, we should expect \(\mathbb{E}_{\mathcal{S}}[\sigma_{\mathrm{val}}(\mathcal{S},\overline{\mathcal{S} }_{\text{sem}})]\) and \(\mathbb{E}_{\mathcal{S}}[\sigma_{\mathrm{val}}(\mathcal{S},\overline{\mathcal{S} }_{\text{com}})]\) to be significantly larger than \(\mathbb{E}\left[\sigma_{\mathrm{val}}(\cdot,\cdot)\right]\). That is, 

the Schwartz values of two similar subreddits should be significantly closer to each other than the Schwartz values of two random subreddits. Our results, computed using empirical expectations over our dataset, confirm this hypothesis. Significant differences exist between the expected Schwartz values of similar and random subreddits. The expected values for both semantic and user similarity is \(0.81\) versus \(0.64\) for random subreddits. The z-test scores are \(73.2\) and \(74.4\), respectively, indicating a high level of statistical significance.

### Controversial Topics

Different online communities have differing viewpoints on widely discussed issues based on their values. To test whether we can also observe this in subreddits, we extend the analysis to **controversial topics**, based on Wikipedia's list of controversial topics.11 For a selected topic, we identify subreddits with differing viewpoints on the topic, e.g., r/Communism and r/Capitalism.

Footnote 11: [https://en.wikipedia.org/wiki/Wikipe](https://en.wikipedia.org/wiki/Wikipe) dia:List_of_controversial_issues

To establish the utility of our method and dataset, we analyse both topics studied by prior work in the social sciences, as well as previously unexplored topics that could be further studied. An overview of the results on all controversial topics can be found in Figure 1. An overview of closely related subreddits (in terms of values) to each of the subreddits discussed here can be found in Appendix B.5.

FeminismIn our investigation involving the Reddit communities of r/Feminism and r/MensRights (prior work Khan et al. (2020); Witt (2020) characterized r/MensRights as a non-feminist community based on the content shared by its members), we find both subreddits to be extremely anti-conformative (Figure 0(a)). The values of _universalism_, _benevolence_, and _self-direction_ were found to be highly relevant for both. _self-direction_ as a value being more core to feminists, as compared to non-feminists has also been found in prior work exploring their values Zucker and Bay-Cheng (2010).

ReligionThere are multiple large-scale studies aiming to establish the connection between religion and Schwartz values. Saroglou et al. (2004) present a meta-analysis across 15 countries, finding that religious people prefer conservative values, e.g., _tradition_, and dislike values related to openness (_stimulation_, _self-direction_). We see similar trends in our comparison of r/atheism, r/spirtuality, and r/religion (Figure 0(b)). While r/atheism and r/religion express _tradiction_, interestingly, both subreddits have a negative stance towards it. Similarly, lower scores for _power_, _hedonism_, and _achievement_ for those communities also align with previous findings. Interestingly, r/spirituality has high relevance and a very positive stance towards some of those values, demonstrating the focus on the self.

VeganismFigure 0(c) compares the values for r/vegan and r/carnivore. The largest difference can be observed in the value of _conformity_. r/Vegan has a very negative stance towards it, representing them challenging the status quo. Holler et al. (2021) review studies related to the values of vegetarians/vegans and omnivores, and find that vegetarians were found to have a stronger relation to _universalism_. We find a similar pattern with r/vegan having a slightly higher relevance but r/carnivore having a more positive stance. They also found them to have a greater emphasis on _self-direction_, which aligns with our findings.

GenerationsLyons et al. (2007) study the differences in values across generations. Similarly, we compare r/Millentials, r/GenZ, and r/GenX in Figure 0(d). While the values of the different generations' subreddits are highly aligned, small differences can be observed, such as the more positive stance towards _achiavement_ in GenZ and GenX than Millenials. This contradicts Lyons et al. (2007), which found Millenials to be more achievement focused than GenX.

Communism vs capitalismHere, we describe the difference in values for different economic ideologies, i.e., communism vs capitalism in Figure 0(e). While Schwartz and Bardi (1997) describe the effect of communism on Eastern Europe (e.g., high importance to conservatism and hierarchy values), they did not include the values held by the people supporting the ideology on a theoretical level. We find that contributors to r/communism hold a more positive stance towards _achievement_, whereas contributors to r/Capitalism hold high relevance values for _self-direction_ and _security_. Both subreddits have high relevance with _universalism_.

MonarchismWe further present values on monarchism in Figure 0(f), as another example of a controversial topic yet to be studied in terms of 

values. We find contributors to r/AbolishTheMonarchy to be against conformity, with a negative stance towards _benevolence_. The contributors to r/monarchism have a high relevance to _tradition_, but a negative stance towards it. They also converse a lot more positively about _power_, _achievement_, and _security_.

### Correlation with Surveys

Finally, to understand how well aligned values expressed in online communities are with values of real-world communities, we compare the Schwartz values extracted from Reddit with those obtained from traditional questionnaire methods.

* "red" states clearly tend to have higher _tradition_ values than "blue" states.

**Country-level values** Next, we extend this correlation experiment to additional Schwartz values and countries other than the USA. Schwartz and Cieciuch (2022) used questionnaires to analyse the Schwartz values associated with 49 countries,12 which we correlate with values extracted from subdetdits related to these countries using our classifier. We start by manually identifying a set of subreddits relevant to each of the 49 countries.13 We successfully found at least one subreddit for 41 countries. We also determined which language is the most likely to be used online by citizens of the country (and not, for example, by tourists).14 See Tab. 14 in App. B for a comprehensive list of countries and their associated languages and subreddits. Next, we collected posts written in the country's language15 from the country's subreddits. We then randomly sampled 2,000 posts and 2,000 comments per country, excluding countries with fewer than 250 total samples. Listing 3 lists the 32 countries that passed this filter. Finally, we used Google Translate API to translate the posts into English16 and applied our approach to extract Schwartz values for each

Figure 1: Radar plots displaying the ten Schwartz values of subreddits dedicated to controversial topics.

 

country.

The above-mentioned process generates a country-value matrix of dimensions \(32\times 10\) of Schwartz values obtained from Reddit. We compare this matrix to the \(49\times 10\) the country-value described in Schwartz and Cieciuch (2022) by removing rows corresponding to countries that do not exist in Listing 3. We then correlate the columns of the two now-identical matrices using Spearman's \(\rho\). In line with previous studies Nasif et al. (1991); Joseph et al. (2021), we find no correlation between the values extracted from SM and values extracted from questionnaires--an average Spearman's \(\rho\) of \(-0.03\). Tab. 7 in App. B.4 reports the results in full. This lack of correlation may arise from the distinct demographics of Reddit users compared to questionnaire respondents or the unique dynamics of online interaction, which tends to be more reactionary compared to survey responses Joseph et al. (2021). Our results confirm findings from previous studies that dynamics of online behaviour differs from offline, and should not directly be used as a proxy without further qualitative investigation.

## 5 Conclusion

In this paper, we apply Schwartz's Theory of Human Values to Reddit at scale, offering scholars a complementary tool for studying online communities. We train and thoroughly evaluate supervised value extraction models to detect the presence and polarity towards human values expressed in language used on social media. Using these, we conduct extensive analysis of nine million posts across 12k popular subreddits. Through our analysis, we both confirm previous findings from the social sciences, as well as uncover novel insights into the values expressed within various online communities, shedding light on existing patterns and outliers that warrant further investigation.

### Limitations

The main limitations of our work stem from the nature of the task itself. Given the inherent subjectivity and complexity of assigning human values to texts, a certain level of noise--aleatoric uncertainty--is unavoidable. This unpredictability can be amplified by epistemic uncertainty, which arises from the limitations of the models we trained. Although we have thoroughly validated the labels generated by the model (SS3.3), the predictions can sometimes be noisy or inconsistent.

Another limitation of our approach is the consolidation of the values of posts in a subreddit into a single vector. While this is necessary for understanding the values of communities, it discounts the values the individual posts of users that constitute these communities. This also opens up much room for future research, particularly in studying the internal dynamics of online communities and understanding the role that values play at the individual post level.

Lastly, while our approach can be leveraged to identify phenomena worthy of further investigation, it lacks the means to explain these observations fully. Values are an abstract concept that is challenging to quantify and analyse, with the values

Figure 2: US states sorted by their _tradition_ values extracted from Reddit, colour coded based on the 2020 US election results (Blue – democratic majority. Red – republican majority).

 

frameworks themselves drawing criticism (Jackson, 2020). We argue that interdisciplinary collaboration with experts in psychology and sociology is essential to understand these phenomena and their implications properly. Such collaborations will not only enrich our understanding of online communities but also contribute to developing more robust and nuanced machine learning models in the future, given the inclusion of such text in the training data.

### Broader Impact and Ethical Considerations

We believe our work goes beyond the methodological toolkit of social scientists. The accessibility, large scale, and relatively high quality of Reddit data have positioned it as a valuable resource for training data for diverse NLP tasks (Overbay et al., 2023; Blombach et al., 2020; Huryn et al., 2022). It opens the door to training models that are better attuned to the specific needs of various communities, potentially protected ones. Recognizing the values of online communities, often utilized as training data for LM development, is crucial for making informed decisions about incorporating the data and contributing to the creation of models that genuinely reflect the diverse perspectives within these communities (Arora et al., 2023). These also percolate into downstream applications of LMs (Jakesch et al., 2023) and have a broader impact on their use.

As for ethical considerations, we made specific efforts to ensure that our work does not impair the privacy and anonymity of Reddit users (Sugiura et al., 2017). We refrain from attributing values to individual users and instead study communities as a whole by aggregating and condensing individual data points into a single vector. Nevertheless, many online communities were created to serve as a safe space for vulnerable individuals, where they share highly sensitive and private information. Therefore, research on Reddit and other online communities should make utmost efforts to respect these spaces and handle their data with care.

## References

* Agrawal et al. (2023) Pratik Agrawal, Tolga Buz, and Gerard de Melo. 2023. Wallstreetbets beyond gamestop, volos, and the moon: The unique traits of reddit's finance communities. In _AMCIS 2022_.
* Arora et al. (2023) Arnav Arora, Lucie-aimee Kaffee, and Isabelle Augenstein. 2023. Probing pre-trained language models for cross-cultural differences in values. In _Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)_, pages 114-130, Dubrovnik, Croatia. Association for Computational Linguistics.
* Basnakova et al. (2016) Jana Basnakova, Ivan Brezina, and Radomir Masaryk. 2016. Dimensions of culture: The case of Slovakia as an outlier in hofstede's research. _Ceskoslovenska Psychologie_, 60(1).
* Blombach et al. (2020) Andreas Blombach, Natalie Dykes, Philipp Heinrich, Besim Kabashi, and Thomas Proisl. 2020. A corpus of German Reddit exchanges (GeRedE). In _Proceedings of the 12th Language Resources and Evaluation Conference_, pages 6310-6316, Marseille, France. European Language Resources Association.
* Boyd et al. (2021) Ryan Boyd, Steven Wilson, James Pennebaker, Michal Kosinski, David Stillwell, and Rada Mihalcea. 2021. Values in words: Using language to evaluate and understand personal values. _Proceedings of the International AAAI Conference on Web and Social Media_, 9(1):31-40.
* Center (2014a) Pew Research Center. 2014a. How religious is your state?
* Center (2014b) Pew Research Center. 2014b. Political ideology by state.
* Chancellor et al. (2018) Stevie Chancellor, Andrea Hu, and Munmun De Choudhury. 2018. Norms matter: Contrasting social support around behavior change in online weight loss communities. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_, CHI '18, New York, NY, USA. Association for Computing Machinery.
* Chandrasekharan et al. (2022) Eshwar Chandrasekharan, Shagun Jhaver, Amy Bruckman, and Eric Gilbert. 2022. Quarantined! examining the effects of a community-wide moderation intervention on reddit. _ACM Trans. Comput.-Hum. Interact._, 29(4).
* a study of stack overflow comments. In _Coordination, Organizations, Institutions, Norms, and Ethics for Governance of Multi-Agent Systems XIII_, pages 20-34, Cham. Springer International Publishing.
* Fontaine et al. (2008) Johnny R. J. Fontaine, Ype H. Poortinga, Luc Delbeke, and Shalom H. Schwartz. 2008. Structural equivalence of the values domain across cultures: Distinguishing sampling fluctuations from meaningful variation. _Journal of Cross-Cultural Psychology_, 39(4):345-365.
* Gerlach and Eriksson (2021) Philipp Gerlach and Kimmo Eriksson. 2021. Measuring cultural dimensions: external validity and internal consistency of hofstede's vsm 2013 scales. _Frontiers in Psychology_, 12:662604 

* Gjurkovic and Snajder (2018) Matej Gyurkovic and Jan Snajder. 2018. Reddit: A gold mine for personality prediction. In _Proceedings of the Second Workshop on Computational Modeling of People's Opinions, Personality, and Emotions in Social Media_, pages 87-97, New Orleans, Louisiana, USA. Association for Computational Linguistics.
* moral foundations theory: The pragmatic validity of moral pluralism. In Patricia Devine and Ashby Plant, editors, _Advances in Experimental Social Psychology_, volume 47 of _Advances in Experimental Social Psychology_, pages 55-130. Academic Press.
* Havaldar et al. (2024) Shreya Havaldar, Salvatore Giorgi, Sunny Rai, Thomas Talhelm, Sharath Chandra Guntuku, and Lyle Ungar. 2024. Building knowledge-guided lexica to model cultural variation. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 211-226, Mexico City, Mexico. Association for Computational Linguistics.
* He et al. (2021) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pretraining with gradient-disentangled embedding sharing.
* Hofstede (1984) Geert Hofstede. 1984. _Culture's consequences: International differences in work-related values_, volume 5. sage.
* Holler et al. (2021) Sophie Holler, Holger Cramer, Daniela Liebscher, Michael Jeitler, Dania Schumann, Vijayendra Murthy, Andreas Michalsen, and Christian S Kessler. 2021. Differences between omnivores and vegetarians in personality profiles, values, and empathy: a systematic review. _Frontiers in psychology_, 12:579700.
* Huryn et al. (2022) Daniil Huryn, William M. Hutsell, and Jinho D. Choi. 2022. Automatic generation of large-scale multi-turn dialogues from Reddit. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 3360-3373, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Inglehart (2020) Ronald Inglehart. 2020. _Modernization and postmodernization: Cultural, economic, and political change in 43 societies_. Princeton university press.
* Jackson (2020) Terence Jackson. 2020. The legacy of geert hofstede.
* Jaffe (2009) Alexander Jaffe. 2009. Stance: Sociolinguistic perspectives. In _Stance: Sociolinguistic Perspectives_.
* Jakesch et al. (2023) Maurice Jakesch, Advati Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023. Co-writing with opinionated language models affects users' views. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_, CHI '23, New York, NY, USA. Association for Computing Machinery.
* Joseph et al. (2021) Kenneth Joseph, Sarah Shigars, Ryan Gallagher, Jon Green, Alexi Quintana Mathe, Zijian An, and David Lazer. 2021. (mis)alignment between stance expressed in social media data and public opinion surveys. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 312-324, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Khan (2020) Abeer Khan. 2020. Reddit mining to understand gendered movements. In _EDBT/ICDT Workshops_.
* Kiesel et al. (2022) Johannes Kiesel, Milad Alshomary, Nicolas Handke, Xiaoni Cai, Henning Wachsmuth, and Benno Stein. 2022. Identifying the human values behind arguments. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 4459-4471, Dublin, Ireland. Association for Computational Linguistics.
* Lyons et al. (2007) Sean T Lyons, Linda Duxbury, and Christopher Higgins. 2007. An empirical assessment of generational differences in basic human values. _Psychological reports_, 101(2):339-352.
* Nasif et al. (1991) Ercan G Nasif, Hamad Al-Daeaj, Bahman Ebrahimi, and Mary S Thibodeaux. 1991. Methodological problems in cross-cultural research: An updated review. _MIR: Management International Review_, pages 79-91.
* Newell et al. (2016) Edward Newell, David Jurgens, Haji Saleem, Hardik Vala, Jack Sassine, Caitrin Armstrong, and Derek Ruths. 2016. User migration in online social networks: A case study on reddit during a period of community unrest. In _Proceedings of the International AAAI Conference on Web and Social Media_, volume 10, pages 279-288.
* Norton (1997) Bonny Norton. 1997. Language, identity, and the ownership of english. _TESOL Quarterly_, 31(3):409-429.
* Overbay et al. (2023) Keighley Overbay, Jaewoo Ahn, Fatemeh Pesaranzadeh, Joonsuk Park, and Gunhee Kim. 2023. mRedditSum: A multimodal abstractive summarization dataset of Reddit threads with images. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 4117-4132, Singapore. Association for Computational Linguistics.
* Park et al. (2024) Chan Young Park, Shuyue Stella Li, Hayoung Jung, Sviltan Volkova, Tanushree Mitra, David Jurgens, and Yulia Tsvetkov. 2024. Valuescope: Unveiling implicit norms and values via return potential model of social interactions.
* Ponizovskiy et al. (2020) Vladimir Ponizovskiy, Murat Ardag, Lusine Grigoryan, Ryan Boyd, Henrik Dobesull, and Peter Holtz. 2020. Development and validation of the personal values dictionary: A theory-driven tool for investigating references to basic human values in text. _European Journal of Personality_, 34(5):885-902 

* Qiu et al. (2022) Liang Qiu, Yizhou Zhao, Jinchao Li, Pan Lu, Baolin Peng, Jianfeng Gao, and Song-Chun Zhu. 2022. Valuenet: A new dataset for human value driven dialogue system. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 36, pages 11183-11191.
* Rokeach (1973) Milton Rokeach. 1973. _The Nature of Human Values_. The Nature of Human Values. Free Press, New York, NY, US.
* Roy et al. (2021) Shamik Roy, Maria Leonor Pacheco, and Dan Goldwasser. 2021. Identifying morality frames in political tweets using relational learning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 9939-9958, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Saroglou et al. (2004) Vassilis Saroglou, Vanessa Delpierre, and Rebecca Dernelle. 2004. Values and religiosity: A meta-analysis of studies using schwartz's model. _Personality and individual differences_, 37(4):721-734.
* Schwartz (1994) Shalom H. Schwartz. 1994. Are there universal aspects in the structure and contents of human values? _Journal of Social Issues_, 50(4):19-45.
* Schwartz (2012) Shalom H. Schwartz. 2012. An overview of the schwartz theory of basic values. _Online Readings in Psychology and Culture_, 2:11.
* Schwartz and Bardi (1997) Shalom H Schwartz and Anat Bardi. 1997. Influences of adaptation to communist rule on value priorities in eastern europe. _Political psychology_, 18(2):385-410.
* Schwartz and Cieciuch (2022) Shalom H Schwartz and Jan Cieciuch. 2022. Measuring the refined theory of individual values in 49 cultural groups: psychometrics of the revised portrait value questionnaire. _Assessment_, 29(5):1005-1019.
* Soliman et al. (2019) Ahmed Soliman, Jan Hafer, and Florian Lemmerich. 2019. A characterization of political communities on reddit. In _Proceedings of the 30th ACM Conference on Hypertext and Social Media_, HT '19, page 259-263, New York, NY, USA. Association for Computing Machinery.
* Sugiura et al. (2017) Lisa Sugiura, Rosemary Wiles, and Catherine Pope. 2017. Ethical challenges in online research: Public/private perceptions. _Research Ethics_, 13(3-4):184-199.
* Trager et al. (2022) Jackson Trager, Alireza S. Ziabari, Aida Mostafazadeh Davani, Preni Golazizian, Farzan Karimi-Malekabadi, Ali Omrani, Zhihe Li, Brendan Kennedy, Nils Karl Reimer, Melissa Reyes, Kelsey Cheng, Mellow Wei, Christina Merrifield, Arta Khosravi, Evans Alvarez, and Morteza Dehghani. 2022. The moral foundations reddit corpus.
* Turcan and McKeown (2019) Elsbeth Turcan and Kathy McKeown. 2019. Dreaddit: A Reddit dataset for stress analysis in social media. In _Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019)_, pages 97-107, Hong Kong. Association for Computational Linguistics.
* van der Meer et al. (2023) Michiel van der Meer, Piek Vossen, Catholijn Jonker, and Pradeep Murukannaiah. 2023. Do differences in values influence disagreements in online discussions? In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 15986-16008, Singapore. Association for Computational Linguistics.
* Weld et al. (2023) Galen Weld, Amy X. Zhang, and Tim Althoff. 2023. Making online communities 'better': A taxonomy of community values on reddit.
* Witt (2020) Taisto D. Witt. 2020. _Affect, identity, and conceptualizations of feminism and feminists on the antifeminist r/MensRights subreddit_. Ph.D. thesis, University of British Columbia.
* Zomick et al. (2019) Jonathan Zomick, Sarah Ita Levitan, and Mark Serper. 2019. Linguistic analysis of schizophrenia in Reddit posts. In _Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology_, pages 74-83, Minneapolis, Minnesota. Association for Computational Linguistics.
* Zucker and Bay-Cheng (2010) Alyssa N Zucker and Laina Y Bay-Cheng. 2010. Minding the gap between feminist identity and attitudes: The behavioral and ideological divide between feminists and non-labelers. _Journal of personality_, 78(6):1895-1924.

 



## Appendix A Annotation Guidelines

For each three consecutive rows in the table (colour-coded), start by reading the three sentences in column B. Next, review the Schwartz value associated with the three sentences (listed in column C) and ensure you understand its meaning, using the figure at the top of the table as a reference. Then, order the sentences according to the extent to which they express the value, ignoring the stance of the sentences towards the value. In other words, our focus is solely on whether the value is expressed, not on how it is expressed. Even if a sentence violates the value, it still expresses it. For instance, the sentence "I never went to church" expresses the value of "Tradition,'' despite contradicting it.

This task is subjective, and in many cases, there is no single correct answer. If you are uncertain about a particular instance, respond to the best of your ability.

```
For each rowinthetable,startbyreadingsentenceincolumnA.Next,reviewtheSchwartzvalueassociatedwiththemesentenceincolumnBandensureyouunderstanditsmeaning,usingthefigureattopofthetablasareference.Then,determinethesstanceofthesantencetowardsthevalue.Ifthenestencesupportsthevalue(i.e.,thevalusexpressedinapositivewayorothesentencedescribesastuationwherethevalueismaintained),select'positive''.Ifthesentenceviolatesthevalue(i.e.,thevalueisexpressednegatively,orthentsentencedecribessavituationwherethevalueisdisregaredorbroken),select'negative''.Ifthesentencedsimplydoesnotexpressthevalue,select'NVA'''.Forinstance,thesentence'NIneverwenttochurch''hasagativestancetowardsthevalueof"Tradition''.Conversly,thesentence'familycomesfirstforher''hasacpositivesstancetowardsthevalue. Thistaskissubjective,andinmanycases,thereismonsingleccorrectanswer.Ifyouareuncertainaboutaparticularintanstaneec,respondtothestofayourability.
```

Listing 1: Guidelines for the task of annotating the dataset used to evaluate the relevancy model.

As described in SS3.3, three authors of this paper annotated Reddit content to evaluate the Value Extraction Models. To evaluate the relevance model, each annotator ranked triplets \((s_{1},s_{2},s_{3})\)according to the extent each sentence \(s_{i}\) expresses a given value \(v\). In total, each annotator ranked 50 triplets, or 5 triplets per value. Fig. 3 contains a print screen of the annotation task, whereas Listing 1 specifies the annotation guidelines.

To evaluate the stance model, each annotator was given a sentence \(s\) and a value \(v\), and was tasked to determine the stance of \(s\) towards \(v\) (one of "positive", "negative", or "irrelevant/neutral", indicating that sentence \(s\) does not express the value \(v\)). Each annotator annotated a total of 200 sentences, or 20 per value. Fig. 4 contains a print screen of the annotation task, whereas Listing 2 specifies the annotation guidelines.

## Appendix B Additional Material

### Dataset Statistics

#### Annotation and Evaluation

Tab. 8 and Tab. 9 present cases where the relevance model and stance models, respectively, misclassified instances.

### Prediction Examples

Tab. 10 contains examples of posts and comments from Reddit and the values and stances that the relevance and stance models predicted for each instance.





\begin{table}
\begin{tabular}{l l c} \hline \hline
**Val** & **Sentence** & **Annotated Average Rank** \\ \hline SD & idx it brings some discussions to the community, can’t be bad \\ PO & You really gave it all you had very impressive content \\ CO & Turkish give debates Marxism and \\ FE & Flued with a street upper \\ BE & Claiming a fictional character has did because your cowork/psychological \\ GO & What is each of your favorite song from Mercurial World to perform live? \\ \hline \hline \end{tabular} 
\begin{tabular}{l l} \hline \hline
**Val** & **Sentence** & **Predicted (wrong) Stance** \\ \hline PO & Crypto traders using the war to make money. This is a new low \\ HE & God forbid you pay someone for their content that you enjoy. \\ CO & SLPT: drive at speed in reverse gear \\ DE & Claining a fictional character has died because your cowork/psychol- \\ OGist said so based off of a movie \\ PO & What is each of your favorite song from Mercurial World to perform live? \\ \hline \hline \end{tabular} 
\begin{tabular}{l l} \hline \hline
**Val** & **Sentence** & **Predicted (wrong) Stance** \\ \hline PO & Crypto traders using the war to make money. This is a new low \\ HE & God forbid you pay someone for their content that you enjoy. \\ CO & SLPT: drive at speed in reverse gear \\ FE & How much speed cameras to get fines \\ FE & Ireighy be the police \\ SD & If the island keeps drifting, the UK would be part of the US again? \\ UN & The double standards of western media towards third world countries \\ SD & The coffee is probably making it \\ SO & Weg \\  & Rose. People forget it's a drug with consequences. \\ \hline \hline \end{tabular}
\end{table}
Table 8: Instances where the relevance model predicted that the sentence expresses a value with high confidence (above 0.8), whereas the annotators concluded that the sentence does not express the value.

Figure 4: Print screen of the annotation task of the stance model. Two annotators were tasked to determine the stance of the sentence in column A towards the value in column B.

Figure 3: Print screen of the annotation task of the relevance model. The three annotators were tasked with ranking triplets of sentences according to the extent to which they express the value of column C.





**Sentence**

The green party was literally removed from the ballot in key battleground states in the 2020 election. When you're trained by a leader you become a leader. I've been learning jazz for about 8 months and still can't write a single good peace, is that normal?Just wondering if Im not deaf because every time I'm trying to write something I end up raging. In general I make music for 4-5 years if that matters. My first ever attempt at finger crochet (or any crochet for that matter)! I’m really proud of myself and just wanted to share somewhere :) I have never had a birthday celebrationSo my parents are Jehovah's witnesses I'm not but I have never received a birthday present and this year I'm really depressed I want to celebrate but I'm broke To love more is Beautiful.I wish you Wonderful evening and happy weekend This video gave me anxiety. It's so inefficient with its movements that it made me squirm. This awa inspiring film is the culmination of 3 months working underwater, diving, exploring and filming the most remote corners of the Great Barrier Reef. It showcases some of the most stunning scenes, schooling sharks and most vibrant coral reef the ocean has to offer. Hope you enjoy! Having to lie so that you don't have to work Saturdays is funny and wholesome smh If you want to be better at squatting, this is fine. If you are going for a specific aesthetic, evaluate/critique your physique for any lagging parts areas, then choose the exercises that help bring them up. "The only moral abortion is MY abortion" or "Grifters gonna Grift" I REALLY THINK ALL OF US HUMANS NEED TO BE REPRESENTED PROPERLY, REGARDLESS OF WHAT LANGUAGE PACKS WE HAVE INSTALLED. And then blaming the teacher when half the class fails. Please ban mentions of hasan off of LSF. his mental health is being affected by the harassment from this sub. Respect my religion while I shove it down your throat.... Kroboiniki: known in the West as "Tetris music"; in Russian, its a folk song about courtship. There is a literal detailed police report of him beating the slit out of his then girlfriend. Indian minister drinks dirty water from 'hoyl' river polluted with sewage to show locals it's safe... ends up in hospital days later This is in Mexico. The guy can literally pay the cops to look the other way, and he obviously has the money to do so India also built the largest border walls with pakistan, that too in the Himalayan foothills and thar desert. Really helped in curbing cross border terror though.



Iceland, Czech Republic, Italy, Sweden, China, Ecuador, Germany, Peru, Indonesia, Serbia, Spain, India, Australia, Philippines, Portugal, France. \\ \end{tabular}
* List of countries used in the correlation experiment in SS4.4.

**Tab. 11** Subreddits with the highest signal for each one of the ten Schwartz values. The stance of Green subreddits towards the value is positive (above \(0.2\)), whereas Red indicates negative stance (below \(-0.2\)).

**Tab. 12** Subreddits expressing the strongest positive and negative stances for each value.

**Tab. 13** Subreddits with the highest and lowest total magnitude of Schwartz values, calculated according to \(\text{mag}(\mathcal{S})=|u_{\text{rel}}(\mathcal{S})|_{2}\).

**Fig. 5** Location of the "Public description" attribute on a subreddit page, used to calculate \(\sigma_{\mathrm{scm}}(\mathcal{S}_{1},\mathcal{S}_{2})\) in SS4.2.

**Listing 3** List of countries used in the correlation experiment in SS4.4.

### Controversial Topics - Extended

Additionally to the analysis presented in Section 4.3, Table 15 displays the ten most similar subreddits in terms of values. We use value similarity as described in Section 4.2 to determine the closest subreddits to each of the controversial topic subreddits.

## Appendix C Reproducibility

### Training the Schwartz Values Extractor

We trained the relevance model and the stance model for ten epochs with early stopping, using a learning rate of \(5\cdot 1e^{-5}\), batch size of 32, Adamw optimiser with default parameters and linear learning rate scheduler. We trained the models on a single TitanRTX GPU for about 5 hours for the relevance model, and 2 hours for the stance model.

Figure 5: Location of the “Public description” attribute on a subreddit page.

 



\begin{table}
\begin{tabular}{l l} \hline Value & Subreddits \\ \hline Tradition & r/0.42igion (0.42), r/Anglicanism (0.42), r/TraditionalCatholics (0.37), r/AskAChristian (0.37), r/AcademicBibilical (0.36), r/ Catholic (0.36), r/AskBibleScholars (0.36), r/Epsiocpolaim (0.36), r/Asskr Priest (0.35), r/Autthetism (0.34), r/OrthhodoxChristian-ity (0.34), r/Cathellicism (0.33), r/authetistencedescale (0.33), r/Agmatic (0.32), r/atheistmemes (0.32), r/CatholicMemes (0.32), r/Bible (0.32), r/RadicalChristality (0.31),r/athelism (0.31),r/0.3ligiousfrutticke (0.31) \\ \hline Benevolence & rocaparents (0.07), r/Stepmann (0.08), r/BelipalSos (0.04), r/theotherwoman (0.84), r/stepparents (0.84), r/RuditchLidner (0.83), r/ADMD_partners (0.83), r/CaregiverSupport (0.83), r/SingleParents (0.83), r/Petloss (0.83), r/Adoption (0.83), r/emotionalabuse (0.83), r/FastRemaredOutChild (0.82), r/ApdingEnents (0.82), r/FriendshipAdvice (0.82), r/AnxioWatacthemt (0.82), r/Arducpted (0.82), r/attachment_theory (0.82), r/AbsOneAfterInfidelity (0.82),r/NarcAbuseAndDivorce (0.82)
[FOOTNOTE:]Footnote *: footnotemark: [ENDFOOTNOTE]  Conformity & r/AgainstHateSubreddits (0.54), r/reclassified (0.52), r/TheseFuckingAccounts (0.50), r/AmFreeToGo (0.50), r/ModSupport (0.50), r/modshoeldisocks (0.49), r/ModSharresholddit (0.49), r/Doebitrality (0.48), r/ButchedIdle (0.47), r/Bad_Coo_No_Donnt (0.47), r/Chirablights (0.47), r/JustUnsubbed (0.46), r/legalavicoeffopc (0.46), r/circleDrokeer (0.46), r/How (0.46), r/BadNeighbors (0.46), r/redidreguest (0.46), r/law (0.46), r/Fukchka (0.46) \\ \hline Hedonism & r/crossdressing (0.56), r/transadorable (0.56), r/Autumn (0.55), r/TheMmidnight (0.53), r/dykesongemild (0.53), r/exdeoderma (0.53), r/FreeCompliments (0.53), r/GothStyle (0.52), r/NaIxt (0.52), r/coay (0.51), r/PlusSizeErashtion (0.51), r/MFSelfTrain (0.50), r/oldhgashion (0.50), r/removed (0.50), r/transpositive (0.50), r/christmas (0.49), r/malepollish (0.49), r/RainbowEverything (0.49), r/gybysgonemild (0.49), r/RappyTrees (0.49) \\ \hline Power & r/FunRMSE (0.41), r/AskEconomics (0.38), r/finance (0.37), r/acoms (0.37), r/HisnetroidChahmif (0.37), r/geophiltics (0.37), r/strongblock (0.36), r/UNKSheradloders (0.36), r/debtree (0.36), r/qyldgang (0.36), r/rocketpool (0.36), r/Anchr (0.36), r/IndinStreetBets (0.36), r/Economics (0.35), r/dividends (0.35), r/VoteBlue (0.35), r/EuropeanFederalists (0.35), r/defi (0.35), r/ValueInvesting (0.35), r/AlibabaStock (0.35) \\ \hline Achievement & r/passive_income (0.60), r/Fund Rise (0.60), r/ValueInvesting (0.59), r/OptionsMillionaire (0.59), r/EngineeringResumes (0.58), r/khokomievements (0.57), r/olaptdarding (0.57), r/FutureSTradiing (0.57), r/resembles (0.55), r/ahmstock (0.55), r/Lirclyptotes (0.55), r/cryptostreets (0.55), r/dividends (0.55), r/SatoshiNets (0.54), r/FrexExthimTechnische (0.54), r/IndianstockMarket (0.54), r/quant (0.54), r/Blogging (0.54), r/ABubblectock (0.54)
[FOOTNOTE:]Footnote *: footnotemark: [ENDFOOTNOTE]  Self-direction & r/CapitalimSVMSocialism (0.67), r/ScientificNeutrition (0.67), r/Abortiondebatte (0.66), r/DebateAnarchism (0.65), r/BasicIncome (0.63), r/DebateAVegan (0.62), r/Neuropsychology (0.62), r/Changeenyview (0.61), r/AskEconomics (0.61), r/AskPsychiatry (0.59), r/AskLDU (0.59), r/nutrition (0.58), r/AskCSialScience (0.58), r/healthcare (0.58), r/asksbychology (0.58), r/radicalmentalHealth (0.58), r/ketoscience (0.57),r/DeleRateRegion (0.57),r/Matrixsm (0.57),r/Sychedelicherry (0.57)
[FOOTNOTE:]Footnote *: footnotemark: [ENDFOOTNOTE]  Universalism & r/DebateVersion (0.88), r/IsraElvePalestine (0.87), r/DebateEvaluation (0.85), r/DebateArchism (0.85), r/DebateBeginlon (0.84), r/changemyView (0.84), r/ Abortiondebatte (0.84), r/CapitalismVSocialism (0.84), r/AskSocialScience (0.84), r/EndlessWar (0.83), r/LeftWmgdaHeadvocates (0.83), r/DebateAnRelentist (0.83), r/TrueUnpopolarOptim (0.82), r/ColliclDiscussion (0.82), r/AskPa_Polities (0.82), r/IntellectualParkWeb (0.82), r/chomsky (0.82), r/AskFemntists (0.81), r/Maxism (0.81), r/DateCommunism (0.81)
[FOOTNOTE:]Footnote *: footnotemark: [ENDFOOTNOTE]  Stimulation & r/nutrum (0.51), r/crossdressing (0.51), r/TheMmidnight (0.51), r/transformadorbide (0.51), r/FreeCompliments (0.49), r/dyksogenemild (0.48), r/cakedecrotating (0.48), r/xboxachiveness (0.47),r/PlusSizeFashion (0.47),r/ Gothstyle (0.47),r/ozvy (0.46),r/MTTF-SelfTrain (0.46),r/drhistmsma (0.46), r/AltJeb (0.46), r/Madonna (0.46), r/weddings (0.46),r/NailRT (0.46),r/Transpositive (0.46),r/happry (0.46), r/Robules (0.46)
[FOOTNOTE:]Footnote *: footnotemark: [ENDFOOTNOTE]  Security & r/CredibleDefense (0.66), r/urw3 (0.65), r/syrianvillwar (0.62), r/geophiltics (0.62), r/EmplessMur (0.62), r/warningRunkine (0.59), r/AfghanConflict (0.58), r/UKrainManContflict (0.58), r/ColweHoude (0.55), r/LessCredibleDeference (0.55), r/SunAreaEcool (0.55), r/var (0.55), r/FourueWhartf (0.55), r/nuclear weapons (0.54), r/WarCollege (0.53), r/UkraineInvasionVideos (0.53), r/UkraineRussiaReport (0.53),



\begin{table}
\begin{tabular}{l|p{142.3pt}|c} \hline Value & Positive Stance & Negative Stance \\ \hline Tradition & r/Ankrofficial, r/lds, r/CharliDame- lioMommy, r/Christian, r/AskBriest, r/Bible, r/bahai, r/Quakers, r/Prismat- icLightChurch, r/OrthodoxChristianity & r/ViewModeID & r/New



\begin{table}
\begin{tabular}{l l} \hline \hline Magnitude & Subreddits \\ \hline Maximal & r/DebateAnarchism, \\  & r/Abortiondebate, r/therapyabuse, r/Capitalis-mVSocialism, r/change-myview, r/AvoidantAtachment, r/LeftWing-MaleAdvocates, r/coparenting, r/ADHD_partners, \\  & r/DebateAVegan, \\  & r/Ask\_Politics, r/IsraelPalestine, r/PoliticcalDiscussion, r/AsKSocialScience, r/NarcAlusene-AndDivorce, r/AskID, \\  & r/attachment\_theory, \\  & r/Adoption, r/kpoprants, \\  & r/TrueUnpopularOpinion \\ \hline Minimal & r/vegan1200isplenty, \\  & r/caloriecount, \\  & r/Watchexchange, \\  & r/Brogress, r/crystal-growing, r/sneakermarket, \\  & r/gundeals, r/buildapcsales, r/whatist, r/N-MSCordinateExchange, \\  & r/BulkOrCut, r/astrophotography, r/legodeal, \\  & r/whattishishing, \\  & r/whatsthifish, \\  & filmfashion, r/TipOfMy-Fork, \\  & r/safe\_food, r/Repbudget-fashion \\ \hline \hline \end{tabular}
\end{table}
Table 13: Subreddits with the highest and lowest total magnitude of Schwartz values, calculated according to mag(\(\mathcal{S})=|u_{\text{rel}}(\mathcal{S})|_{2}\).





\begin{table}
\begin{tabular}{l l l l} Country & Language & Subreddits \\ Australia & English & r/Australia, r/Australian, r/australian, r/sydney,r/melbourne \\ Brazil & Portuguese & r/brasilir, r/brasilivre, r/GTAorBrazil, r/brasilia, r/saopaulo \\ China & Chinese & r/China, r/Leal_China.irl & r/Real_China.irl \\ Colombia & Spanish & r/Colombia.r/ColombicAredit, r/Bogota \\ Costa Rica & Spanish & r/costarica, r/Ticos \\ Croatia & Croatian & croatia & r/croatia, r/zagreb \\ Czech Republic & Czech & r/czech \\ Ecuador & Spanish & r/eusudor \\ Estonia & Estonian & r/Esti \\ Farco Islands & All & r/Faroeislands \\ Finland & Finnish & r/Sumion \\ France & French & r/france, r/paris \\ Georgia & Georgian & r/Sakartvelo, r/tbilisi \\ Germany & German & r/de, r/deuchland, r/Munich, r/berlin \\ Ghana & English & r/ghana \\ Greece & Greek & r/greeec \\ Hong Kong & Chinese & r/HongKong \\ Iceland & Icelandic & r/Iceland \\ India & All & r/india, r/IndiaSpeaks, r/Unitedstatesofindia, r/askindia \\ Indonesia & Indonesian & r/indonesia \\ Israel & Hebrew & r/Israel, r/telaviv, r/jerusalem \\ Italy & Italian & r/itali, r/Italia \\ Japan & Japanese & r/japan, r/tokyo, r/newsokur \\ New Zealand & English & r/newzealand, r/auckland \\ Oman & Arabic & r/Oman \\ Peru & Spanish & r/PRPU \\ Philippines & English & r/Philippines \\ Poland & Polish & r/Polska \\ Portugal & Portuguese & r/portugael, r/lisboa, r/porto \\ Romania & Romanian & romanian, r/Romatia, r/bucuresti, r/iasis, r/cluj, r/timisoara \\ Serbia & Serbian & r/serbi \\ Slovakia & Slovak & r/Slovakia \\ South Africa & All & r/southAfrica, r/capetown, r/johannesburg \\ South Korea & Korean & r/hanguk \\ Spain & Spanish & r/Espana, r/Spain, r/Madrid, r/Barcelona \\ Sweden & Swedish & r/Service, r/sweden, r/stockholm, r/svenskpolitik \\ Turkey & Turkish & r/Turkey, r/istanbul, r/ankara, r/ismir \\ Ukraine & Ukrainian & r/ukrainer, r/Ukraine.UA \\ United Kingdom & All & r/unitfeedingtom, r/llondon, r/manchester, r/CasualUK, r/unitedkingdom, r/askukh \\ USA & All & All states subreddits \\ Vietnam & Vietnamese & r/VietNam \\ \end{tabular}
\end{table}
Table 14: Countries, languages and subreddits used in the correlation study.





\begin{table}
\begin{tabular}{l l} \hline \hline subreddit & closest subreddits \\ \hline r/vegan & r/intersex, r/transgenderUK, r/TransSpace, r/Transmedical, r/ABCDDesis, r/LGBTindia, r/DeepThoughts, r/nonduality, r/exvegans, r/BlockedAndReportedr/carnivorediet, r/Psoriasis, r/SeriatricArthritis, r/rheumatoid, r/acne, r/Hashimotos, r/Testosterone, r/PlasticSurgery, r/lupus, r/kratom \\ r/communism & r/InformedTankie, r/CommunismWorldwide, r/CPUSA, r/socialism, r/StupidpolEurope, r/LatinAmerica, r/sendithenthachs, r/ROI, r/Africa, r/myanmar \\ r/Capitalism & r/georgism, r/Unions, r/SandersForPresident, r/labor, r/NewDealAmerica, r/BerniesSanders, r/WorkersStrikeBack, r/ndp, r/theydithemath, r/MayDayStrie \\ r/monarchism & r/leftistvexillology, r/UsefulCharts, r/RoughRo-manMemes, r/IndiaPlace, r/pureasoif, r/MedievalHistory, r/sitthylholcore, r/MemirTVmemes, r/anientrome, r/darkwingsdashmRemmes \\ r/AbolishTheMonarchy & r/Sham_Sharma_Show, r/forwardsfromgrandma, r/WorddAvalanches, r/Malaphors, r/Lal_Salaam, r/Metal, r/BanVideoGames, r/QanonKaren, r/PoliticalHumor, r/insenepolefacebook r/Millennials & r/GenZ, r/PlusSizeIdAnfPregnant, r/ThailandTourism, r/indiasocial, r/Liverpool, r/TallGrls, r/Psychedelics, r/Chandigarh, r/Cardiff, r/ChromiCl Illness \\ r/GenZ & r/Millennis, r/indiascial, r/TallGirils, r/Kibbe, r/Zilleminals, r/aggretsuko, r/precure, r/nerdfighters, r/KoNoKatachi, r/comiccon \\ r/GenX & r/lesbianafashionadvice, r/CosplayNation, r/Xennisals, r/bigboobproblems, r/femininebys, r/PlusSize, r/distantsocializing, r/CasualPH, r/bald, r/TallGrils \\ r/Feminism & r/antiwoke, r/MensLib, r/ControversialOpinations, r/EnoughIDWspam, r/prochoice, r/prolife, r/fourthwavewwomen, r/TransSpace, r/TrueUnpopulap Opinion, r/Antivegan \\ r/MenisRightrs & r/Antifemists, r/AntiHateCommunities, r/AsABlacMan, r/LeftWingMaleAdvocates, r/azindentity, r/EnougPCMSpm, r/FragileWhiteReditor, r/BlatantMisogyny, r/TheLeftCamteme, r/FowardsFromKLandra \\ r/atheism & r/exmuslim, r/antitheistcheesecake, r/extomatoes, r/Antetheism, r/Atteinism, r/mother, r/progressive\_islam, r/Judaism, r/RadicalChristianity, r/ligiousfrui cakre \\ r/spirituality & r/SpiritualAwakeening, r/awakened, r/pureretention, r/Stoticism, r/Shamanism, r/starseeds, r/Mediums, r/Soulnexus, r/Semenettention, r/SASSWitches \\ r/religion & r/AskAChristian, r/ChristianUniversalism, r/RadicialChristianity, r/bahai, r/hinduism, r/shia, r/Bible, r/progressive\_islam, r/extomatoes, r/mormon \\ \hline \hline \end{tabular}
\end{table}
Table 15: For each subreddit in the controversial topics analysis, the 10 most similar subreddits in terms of values.