Investigating Online Community Engagement through Stancetaking
Authors: Jai Aggarwal
Brian Diep
Julia Watson
Suzanne Stevenson
Venue: EMNLP 2023 Findings (2023)
Keywords: stancetaking; community variation; sociolinguistics; community identity

Abstract: Much work has explored lexical and semantic variation in online communities, and drawn connections to community identity and user engagement patterns. Communities also express identity through the sociolinguistic concept of stancetaking. Large-scale computational work on stancetaking has explored community similarities in their preferences for stance markers -- words that serve to indicate aspects of a speaker's stance -- without considering the stance-relevant properties of the contexts in which stance markers are used. We propose representations of stance contexts for 1798 Reddit communities and show how they capture community identity patterns distinct from textual or marker similarity measures. We also relate our stance context representations to broader inter- and intra-community engagement patterns, including cross-community posting patterns and social network properties of communities. Our findings highlight the strengths of using rich properties of stance as a way of revealing community identity and engagement patterns in online multi-community spaces.

Investigating Online Community Engagement through Stancetaking
Jai Aggarwal
Brian Diep
Julia Watson
Suzanne Stevenson
Department of Computer Science University of Toronto
(jai, bdiep, jwatson, suzanne jêcs. toronto. edu
Abstract
speakers position themselves in various ways af- fectively, evaluatively, epistemically, etc. relative to a topic or a conversational partner (c.g.. Du Bois, 2007). Sociolinguistic work (e.g., Kiesling, 2004; Bucholtz, 2009) has studied how communities vary in their use of stance markers. Stance markers are words that demarcate stance, including, among oth- ers, intensifiers (really, insanely, terribly), modals (might, should), and evaluative words (like, love, hate). For instance, compare these sentences from three Reddit communities:
Much work has explored lexical seman- tic variation in online communities, and drawn connections to community identity and user en- gagement patterns. Communities also express identity through the sociolinguistic concept of stancetaking. Large-scale computational work on stancetaking has explored community sim- ilarities in their preferences for stance mark- ers words that serve to indicate aspects of speaker's stance without considering the stance-relevant properties of the contexts in which stance markers are used. We propose rep- resentations of stance contexts for 1798 Reddit communities and show how they capture com- munity identity patterns distinct from textual or marker similarity measures. We also relate our stance context representations to broader inter- and intra-community engagement patterns, in- cluding cross-com coos-community posting patterns and social network properties communities. Our findings highlight the strengths of using rich properties of stance as way of revealing com- munity identity engagement patterns in on- line multi-community spaces.
I hope you are insanely proud of yourself! [r/pomfree, helping people overeome porn addiction)
2. I hope you are incredibly proud of yourself! fr/progresspics. support for sharing fitness progress]
Both you are incredibly stupid Ir/watchreddiidie, critical of censorship Reddit]
In both (1) and (2), the speakers indicate similarly positive stances towards their interlocutor, but vary in their choice of stance marker to demarcate this stance (insanely vs. incredibly).
1 Introduction
Importantly, sociolinguistic work has also ar- gued that the contexts in which such markers ap- pear is critical aspect of stancetaking behavior (Kiesling al., 2018: Bohmann and Ahlers, 2022). For example, sentences (2) and (3) the same stance marker (incredibly), but in (3) the speaker evaluates their interlocutors in an especially nega- tive light.
Communities vary in their language choices, of- ten in ways that are indicative of the community's shared interests, values, and norms their commu- nity identity. Sociolinguists refer such spaces as communities of practice (Eckert and McConnell- Ginet, 1992; Holmes and Meyerhoff, 1999), and have explored how they vary in their in-person lin- guistic practices (Cheshire, 1982; Eckert, 2000). Computational work has explored variations of word and sense usage in online communities of practice (e.g., Zhang et al., 2017; Del Tredici et al., 2018; Lucy and Bamman, 2021), highlighting re- lationships between linguistic and non-linguistic aspects of community identity, including user re- tention, community size, and network structure. Community identity is also expressed through stancetaking (Bucholtz. and Hall, 2005), in which
Sociolinguistic work posits that repeated stanc- etaking in a consistent way leads to stable associ- ations of stance and identity (Eckert, 2008; Rau- niomaa, 2003, cited in Bucholtz and Hall, 2005). For example, positive evaluations like (2) reinforee the identity of r/progresspics as supportive com- munity, while negative evaluations like (3) con- tribute to the critical tenor of r/watchredditdie. This illustrates that, in addition to stance marker pref- erences, stance contexts are crucial for connecting

---

small number stancetaking and community identity. However, while large-scale research has shown that varia- tion in stance marker usage captures patterns of user cross- possing (Pavalanathan et al., 2017), to our knowledge, no large-scale work has considered how variation in stance contexts relate to commu- nity identity and cross engagement.
2 Related Work
To study the relationship between language and community identity online, work in computational sociolinguisties has begun to shift from relating language choices and macrosocial categories like age, gender, and race (Burger et al., 201 I; Nguyen et al., 2013; Blodgett et al., 2016), to more nu- anced approach to identity, adopting the framework of communities of practice (Eckert and McConnell- Ginet, 1992; Holmes and Meyerhoff, 1999). Com- munities of practice put the focus on local identity categories that better reflect individuals' various group membership choices e.g., presenting as a gamer, a sports fan, or an animal lover.
Here we address that gap by proposing a method for representing stance-relevant properties of a sen- tence that go beyond stance marker usage. Specif- ically, our stance context representations cap- ture higher-level linguistic propertics theorized relate stancetaking, including affect, polite- ness, and formality (Jaffe, 2009; Kiesling, 2009; Pavalanathan al., 2017; Kiesling et al., 2018).
Much sociolinguistic work has explored the connection between línguistic variation and cial network structure in communities of practice (e.g., Cheshire, 1982; Milroy, 1987; Eckert, 2000; Sharma and Dodsworth, 2020). For example, Eck- ert (2000) noted the interrelation between how high school "jocks" and "burnouts" varied in their lin- guistic patterns and in the communities they inter- acted with. Work of this kind has focused on varia- tion within single community of communities.
We focus our attention on the stance contexts in which intensifters appear. Intensifiers, like in- sanely and incredibly above, are a subset of stance markers that emphasize that which they modify. So- ciolinguistic work has shown that intensifier usage varies across linguistic and social contexts (Ito and Tagliamonte, 2003; Bolinger, 1972), which makes them ideal for studying community variation in stancetaking.
We first show that our stance context represen- tations reveal aspects of community identity in a manner that complements both general textual rep- resentations, well as stance marker preference representations. We then demonstrate how similar- ities in stance context relate to broader inter- and cotruncommunity engagement patterns, including user co- co-particciation across Reddit communities, and social factors of communities, including size, activity, loyalty, and density. Our work thus broad- ens the research linking community linguistic prac- tices to social and network properties by consid- ering higher-level factors beyond lexical variation. Future work can leverage these representations to shed light on additional aspects of platform dy- namics, including community creation (c.g. are new communities sometimes created so that users may discuss the same topic but taking an alterna- tive stance?) and user diversity (e.g. on Reddit, a platform dominated by men, do the stances commu- nities take affect the participation and engagement of women and people with (stereotypically) femi- nine linguistic style?).
Online social media platforms support re- search relating linguistic and non-linguistic as- pects of community identity at scale through anal- ysis of multi-commmnity settings (e.g.. Hamil- ton et al., 2016; Israeli et al.. 2022: Noble and Bernardy, 2022). For example, Noble and Bernardy (2022) show positive correlations between similar- ities in community linguistic practices and user- community co-occurrence patterns. Further work has shown that communitics with more distinct words better retain users (Zhang et al., 2017), and tend to be smaller, denser, more active, and have more local engagement (Lucy and Bamman, 2021). Similar findings have been shown for communities with distinct word senses (Del Tredici and Fernán- dez, 2017: Lucy and Bamman, 2021). We extend computational sociolinguistic work on community identity and engagement patterns by motivating and investigating variation of a different kind: variation in higher level properties of stancetaking.
Stancetaking is key to understanding how lan- guage variation reflects identity, because so many of a speaker's linguistic choices pertain ex- pression of stance (e.g., Jaffe, 2009). Moreover, much work has argued that, rather than linguistic forms being directly associated with social iden-
'Our code and datasets can he found at https:///ithub. com/jaikaggarwal/emml http://eeeeeepepaaaaaeeaaeme

---

tities, linguistic forms are first associated with stances, which in turn are associated with social identities (e.g., Ochs, 1993; Du Bois, 2002; Rau- niomaa, 2003; Bucholtz and Hall, 2005; Eckert, 2008). Thus, variation in higher-level properties of stancetaking needs to be given more attention in analyses of language and community identity.
is not due variation in the grammatical contexts in which the markers appear.
Here we describe how we identify a set of inten- sifiers and extract their contexts from a large set of communities (subreddits). We then explain how we represent these contexts with linguistic features rel- evant to stancetaking, presenting the features, how calculate them, and how we use them to form rector-based representation of each community in our dataset.
Yet there has been limited computational work on stancetaking. Pavalanathan al. (2017) studied variation in use of stance markers across online communities, and their word-based approach al- lowed them to study stance at scale, Kiesling et al. (2018) developed a more nuanced approach for rep- resenting stance, drawing on the framework from Du Bois (2007); however, the approach depended heavily on manual annotation, limiting analysis to a relatively small number of online communities. our work, by automatically assessing stance- relevant properties of contexts, we are able to main- tain the scale of analysis from Pavalanathan et al. (2017), while getting closer to the richness of rep- resentation from Kiesling et al. (2018).
3.1 Extracting the Data
We start with the list of single-word intensifiers from Bennett and Goodman (2018, Studies la and lb), as they are shown to express a range of degree of intensification. To ensure that these are appro- priate to our study of stancetaking, we intersected this list of 89 intensifiers with list of 1118 stance markers (including intensifiers and other words) found on Reddit. The latter list was created using the methodology of Pavalanathan et al. (2017), ap- plying Wang2Vee (Ling et al., 2015) to expand a seed set of 448 stance markers released by Biber and Finegan (1989).2 The Wang2Vec model is trained on Reddit data and used to extract words used in similar distributional contexts to the seed stance markers. After intersecting the two lists, we removed two words that occurred in stopword list (most and very) to ensure we included informative words.
3 Representing Stance Contexts
We study stancetaking and stance contexts in particular on Reddit, due to its nature as multi- community platform with free-form commenting and rich engagement patterns among its communi- ties (subreddits). We take intensifiers - a kind of stance marker - and the stance contexts in which they appear as our testbed. Intensifiers, like in- sanely and incredibly in (1)-(3) above, are adverbs that give emphasis to what they modify.
This procedure yielded a set of 38 intensifiers that serve as our stance markers. This stance marker set, and full details of our method for obtain- ing it, are described in Appendix A. In Appendix B, show additional analyses conducted of 252 intensifiers, and find that our methodol- ogy robust to the set of intensifiers used.
Intensifiers are an ideal domain for studying community variation. Extensive empirical work in sociolinguistics has established that intensifiers vary based on social factors (e.g., Ito and Taglia- monte, 2003; Tagliamonte and Roberts, 2005); moreover, they also exhibit much versatility in their contexts and undergo frequent change (e.g., Bolinger, 1972; Peters, 1994). While past com- putational work has studied semantic change in intensifiers over time (Luo et al., 2019; Samir et al., 2021), to our knowledge computational work has not examined community variation in intensifiers, and their context of use, as we do here.
larger
To create our stance context representations of Reddit communitics, we needed to extract the con- texts of use of these intensifiers. We drew on Red- dit data from all of 2019 and retrieved all com- ments from the top 10K subreddits by activity (as determined by Waller and Anderson. 2019) us- ing the Pushshift data dumps (Baumgartner et al., 2020)3. We applied preprocessing described in Appendix C, yielding 1.2B comments across the 10K subreddits.
Furthermore, intensifiers are a practical choice: Stance markers cover range of grammatical classes (verbs, adjectives, interjections, and others), and focusing on intensifiers (which are adverbs), ensures that the variation we find in stance contexts
From each comment, we extract any sentence that contains an intensifier from our list, so that
Wang2Vec is an adaptation of Word2Vee that accounts for syntactic information. haspss////ile...suhhhssss pushshi ft. wtttiii/////oooomeeeeieeeeiieoee comottts///m///////// "https://files

---

our stance context representations will reflect the local semantic context of this stance marker. We retain only those sentences with exactly one us- age of an intensifier, and whose length is at least 6 tokens (to contain sufficient content to reliably extract values for our linguistic features). To ensure sufficient and comparable data per community, we only perform analyses on communities with at least 10K such sentences, and use exactly 10K sentences from each (subsampling when necessary), resulting in 17.98M sentences across 1798 communities.
can then apply this model to assess the VAD of full sentences, based on their SBERT representations. For politeness, we build a logistic regression model, adapting the method of Danescu-Niculescu- Mizil et al. (2013) to give normally-distributed con- tinuous politeness scores by taking the log-odds of the predicted politeness probability, and then min-max scaling the values to the range [0, 1].
For formality, we used the annotated corpus com- piled by Pavlick and Tetreault (2016), focusing on the Answers and Blogs domains. The former generalizes well to periistcaall-difffreee domains (Pavlick and Tetreault, 2016), while the latter is the most similar domain to Reddit (Pavalanathan et al., 2017). We rescaled the formality annotations to the range [0, using min-max scaling, and then trained and selected a formality prediction model using 10-fold cross- vross-validation.
3.2 Stance Context Representations
Our aim is to represent the prototypical stance con- text of each community a vector of feature values averaged across its 10K sentences. We first de- scribe how assess properties of sentences, and then explain how we combine these generate community-level representations.
3.2.2 Community-Leve Representations
3.2.1 Properties of Stance Contexts
We apply each of the above models to the sentences in our dataset. We are interested in the proper- ties of the context in which an intensifier is used, rather than properties of the intensifier itself. Thus we replaced the intensifier OMASK K] token, and generated SBERT representa- tions for these masked sentences. We then applied our models described above to find the valence, arousal, dominance, politeness, and formality val- ues for each sentence. Table I shows example sentences that vary along the five properties. Our aim was to average the values of the fea- tures over all sentences in subreddit to yield a community-level representation - its prototypical stance context vector. However, we observed that the stance contexts within a subreddit may fall at both extremes of scale (c.g., both many highly positive sentences and many highly negative sen- tences), and averaging would obscure such pat- tern. This type of behavior seems natural, because intensifiers are often derived from extreme adjec- tives those that express a property towards the extremes of a scale (Samir et al., 2021) - and such extremeness may itself be part of the stance tenor of a community.
To capture salient properties of stance contexts, we linguistic features known to relate stance demarcation, including affect, politeness, and for- mality (Jaffe, 2009; Pavalanathan et al., 2017; Kies- ling et al., 2018). Affect was broken down into the 3 features of valence (positivity), arousal (inten- sity), and dominance (level of control), following the VAD framework commonly used in psycholin- guistics for representing emotions (Russell, 2003). These 5 features are particularly relevant here, as intensifiers often used for affective impact, and vary widely in politeness and formality. Moreover, such features have robust methods for assessment at the sentence level.
sentence with the
For each feature, we create a model for infer- ring an appropriate value given sentence rep- resented using SBERT (Reimers and Gurevych, 2019). SBERT generates high-quality sentence em- beddings shown to be useful for inferring sentiment, a high-level property of text similar to our stance context properties (Reimers and Gurevych, 2019). We evaluated the SBERT models for each fea- ture on held-out portions of the human-annotated datasets and achieved accuracies comparable to prior work. We summarize model details below; full details can be found in Appendix D.
We thus added an "extremeness" version of each feature introduced in Section 3.2.1 to our stance context representation, calculated by cen- tering each feature at O and computing its abso- lute value. Including the extremeness features re- sulted in each sentence being represented by 10- dimensional vector. We then represent each com-
For VAD, we follow Aggarwal et al. (2020) and train a Beta regression model predict each of these scores (constrained the range [o, 1)) given the SBERT representations of the 20K words in the NRC-VAD lexicon (Mohammad, 2018). We

---

Sentence
V
A
D
P
F
OMG, I'm so [MASK] sorry for your loss.
0.20
0.52
0.33
0.80
0.17
The books are [MASK] easy to read and damn good.
0.85
0.35
0.70
0.58
0.39
This is [MASK] insightful advice.
0.77
0.43
0.66
0.64
0.79
Not gonna lie, that's [MASK] lame
0.24
0.44
0.31
0.28
0.10
Table 1: Example stance context representations for the marker incredibly, which has been replaced with the
[MASK] token. The columns represent valence (V), arousal (A), dominance (D), politeness (P), formality (F).
munity's s overall stance context as the mean of these 10-D vectors of its 10K sentences. 4
community in representation R. Then:
Sim R, c, c) = cos_ cos_sim( simmmi Re, Re)
4 Additional Representations/Methhdss
Community Topic Groups. To investigate gen- eral trends in how communities vary with respect to each of our three representations, we group our sub- reddits by topic, using the r/ListofSubreddits cate- gorization, as in Lucy and Bamman (2021). These groups yield an interpretable community structure that broadly reflects commonalities in user inter- ests. The list divides Reddit into 12 major topics. with further subdivisions.
Here present two other community representa- tions which will compare stance context representation: one that captures community pref- erences for the stance markers, and the other simple overall textual similarity. We also show how we calculate community similarity using the three rep- resentations. Finally, some analyses look at groups of subreddits by topic, which are described here.
We designate any subtopics with at least 50 sub- reddits in our dataset as their own topic. For any unassigned subreddits that start with "ask", or in- clude cue words such as "advice" or "questions", we add them to the Discussion subtopic. This re- sults in 1228 subreddits assigned to 15 topics, and 570 subreddits without topic label. The topics and number of subreddits assigned them shown in Appendix E.
Marker Preference Representations. To cap- ture community preferences for particular stance markers, we used the methodology of Pavalanathan al. (2017), applied to our 17.98M instances of intensifiers, for comparability to our stance con- text representations. We create a subreddit-ne marker count matrix and apply a positive pointwise mutual information (PPMI) transformation to the matrix. We then apply truncated SVD and retain the top 11 dimensions (based on elbowing in a Scree plot).
Computing Topic Group Similarity, To assess similarity of two topic groups, we find the mean pairwise similarities for all subreddits in each of the topics. Let TA. T3 be the sets of communities for topics A and B. We compute the topic similarity TS(R, TA, TB) for representation R as:
Textual Representations. We create textual rep- resentations show that our stance context rep- resentations capture more than just textual simi- larity between communities. Following Lucy and Mendelsohn (2019), we construct tf-idf weighted subreddit-word matrix from our 17. 77.8MM comments, with a vocabulary size of 158K. We then apply truncated SVD and retain the 7 most informative dimensions (based on elbowing in Scree plot).
ΣΣ CETA CEEB
Sim( c,
TS, TA,
TTA
5 Comparing the Three Representations
We show how the kinds of community represen- tations (textual, stance context, and marker prefer- ence) capture distinct facets of community identity.
Computing Community Similarity. We use co- sine similarity to compute how similar two commu- nitics are on a particular representation. Let R be one of our three representations, c, c' be two com- munities (subreddits), and R. be the embedding for
Ptttps: //wtps::/www.... reddit com/r httpr//Lsttffffi sttof ts/ miki/listofslll edditsd updated as April 2023,
'The topics are specified by Redditors, and capture subred- dit similarity based on perceptions of what may drive people to engage with communities, rather than simple textual over- lap. For instance, both r/law and r/leamart are under Educa- tion, but r/lcamart is most textually similar to subreddits like r/crossstitch and r/artistlounge, which are under Hobbies.
'To avoid anisotropy in our vector space, which resulted in almost all communities having near perfect cosine similarity with each other, we applied 7-score normalization to each of the 10 features.

---

Textual vs. Stance Context
0
ilarity with humor subreddits indicate that these may be varied aspects of humor as well.
Textual Marker Usage
0.39**
Stance Context vs, Marker Usage
0.37*
The other two pairs of topics, Politics/Education. and Sports/Video Games, have low to moderate textual similarity, but show very high similarity in either stance context or marker preference. The Pol- itics and Education topics have somewhat higher formal stance contexts relative to other subreddits, which may arise from the local identity categories that surface in these kinds of communities, as in- dividuals may seek to position themselves as intel- ligent. This higher formality leads to very consis- tent stance marker use, 8 of their top 10 most preferred markers overlap, including largely, enor- mously, wholly, and exceedingly. These markers are 4 out of the top 5 markers used to demarcate for- mal stances across our whole dataset. This example highlights how community stance similarity on a particular dimension can guide marker preference similarity.
Table 2: Pearson correlations of our subreddit-level representations. *** indicates significance p 0.001.
We find that they do not correlate highly with each other; moreover, they can reveal subtle differences between pairs of seemingly similar subreddits.
5.1 Pairwise Correlations
For each representation, we compute the commu- nity similarity of each of the 1.6M pairs of sub- reddits, and find the pairwise Pearson correlations between these values. Table 2 shows that the rep- resentations have only low moderate positive correlations. This provides preliminary evidence that we have representations that capture three dis- tinct facets of subreddits: the content they discuss, the stances they tend to take with respect to this content, and the stance markers they use to demar- cate these stances. In the next section, we illustrate how these representations capture distinet patterns across the multi-community landscape.
One of the starkest contrasts across the three representations is between the Sports and Video Games topics. Despite having dissimilar textual representations, this pair of topics is among the most similar with respect to stance contexts (in fact, there is less variation between the stances of these communities than within most topics on the diagonal). Analysis reveals that they are sim- ilar on most dimensions, but are noticeably dis- tinct from the rest of the platform with respect to their high arousal and less extreme politeness. The high arousal (intensity) of these topics makes sense given the competitive, game-oriented nature of the subreddits in both these groups. However, despite this similarity in stance contexts, the Sports and Vidco Games subreddits have low marker prefer- ence similarity, highlighting that communities that take similar stances may differ greatly in how they choose to demarcate these stances.
5.2 Topic-Topic Similarities
To visualize the differences in what these three rep- resentations capture, we compute (S Ta; TB) for all pairs of topics A and B (excluding the Other and General categories as they lack a topical fo- cus).? 7 The plots in Figure show much variation across the three representations. To dig deeper into the nature of this variation, our qualitative analysis focuses three pairs of topies that illustrate how communities can vary in their degree of textual, stance context, and marker preference similarity; see Figure 2. Further visualizations comparing the differences between representations can be found in Appendix F.
The Animals and Humor topics have not un- common pattern in which they are moderately to very similar across all three representations. The linking of animals and humor is surpris- ing, although the range of expressed stances is fairly broad. For example, r'animalsbeingderps., r'animalsbeingjerks, and r'animalsbeingbros in- volve observing animal behavior that is either awk- ward, mean, or kind, respectively; their high sim-
5.3 Discussion
These findings reveal the differences in textual, stance context, and marker preference representa- tions. Each provides distinct glimpse into facets of community identity, and using all three simul- tancously provides further insights into axes which Reddit communities tend vary. Examining the Politics and Education topics shows how simi- larity on just single dimension of stance can be informative as to marker preference similarity, as communities seek to demarcate stance in a way that
"Since the range and variance of similarity values varies across representations, we apply 7-score normalization to the set of topic-topic similarities.

---

Textual Similarity between Topics
Stance Context Similarity between Topica
Marker Preference Similarity between Topics
snimals
cducation
coucation
entertammentt
enteraammet
onterta rmner
bectres
mestass
nob8lcs
Hesble
Hesbyle
0.0
iPestle
-6.5
locettona
-0.5
boasions
now
405
-1.
-1.
-10
tectnology
-15
tectnology
-1.
tochnology
-1
video games
vieooge games
-20
Moeegaome
-20
Figure I: Patterns of topic-topic topic similarity for three community representations. Dark cells indicate topics that are more similar to each other than average, while light cells indicate topics that are more dissimilar to each other than average. Cells the diagonal represent the variation within each topic.
stance context and stance marker preferences re- lates to community structure. These findings il- lustrate the explanatory power offered by stance- based representations, particularly stance context representations, when investigating inter- and intra- community dynamics online.
Animals & Humor
0.64
1.17
0.79
1
Politics & Education
-0.04
0.68
1.46
Sports & Video Games
00.35
1.95
-0.22
Text
Stance
Marker
6.1 Cross-Posting
Figure 2: Topic-topic similarities for each of our three representations pairs interest. Colors and z-scores are the same as in Figure 1.
Core to platforms like Reddit is that people may engage with a number of communities of prac- tice. Among the many reasons users may cross- participate in such communities, we hypothesize that users are attracted to communities that have similar linguistic practices - in particular, those that are similar in how their members express stance. Pavalanathan et al. (2017) showed evidence of stance demarcation patterns predicting user cross- posting patterns, and we extend this to looking at community similarity in stance marker contexts.
is consistent with particular traits. Our final exam- ple illustrates that communities with almost no sim- ilarity their content (Sports and Video Games) can be extremely similar in how their stances relate to that content (highlighting critical difference between stance contexts and textual similarity), but may yet demarcate their stances differently (high- lighting the difference between stance contexts and stance markers). Overall, results show that our three representations are distinct yet comple- mentary, and provide a holistic view of community variation in language and community identity.
Our classification task follows the same pro- cedure described in Pavalanathan et al. (2017), adapted for the top 100 most popular subreddits in our dataset by commenting volume. This results in 429 low cross-posting pairs and 353 high cross- posting pairs. For each representation and a given subreddit pair (c, e'), we use logistic regression model to predict whether e and c' have high or low cross-posting using Sim R, c, c'). (We also repli- cated the model used in Pavalanathan et al. (2017) that treats each vector dimension as separate pre- dictor and found similar pattern of results our cosine similarity metric, but with lower accuracies.)
6 Stance and Engagement Behavior
Our analysis in the preceding section has estab- lished that stance context distinct from both tex- tual similarity and stance marker preference, and has shown preliminary evidence that these differ- ences allow us to capture high-level properties of communities that relate to social identity. In this scction. demonstrate how the two aspects of stancetaking, stance context and stance marker pref- erences, relate to community engagement patterns on Reddit. We first examine how well they predict subreddit cross- posstin patterns. We then explore how community distinctiveness in both average
To evaluate our models, we compare their accu- racy at predicting cross cross-posting using the average accuracy of 10x 10 repeated cross-validation!
"Similarly to Boleda et al. (2012), we use the corrected resampled r-test developed by Nadeau and Bengio (2003) for performing statistical tests, as the lack of independence

---

Representation
Accuracy
Acti-
Loy-
Den-
Marker preference similarity
75.1%
Size
vity
alty
sity
Stance context similarity
82.3%
Marker Pref.
0.00
-0.02
0.20
0.01
Stance Context
0.21
-0.36
-0.34
-0.34
Table 3: Accuracy predicting user cross-post cross-posting.
Table Spearman correlations social factors and distinctiveness; bold indicates significance at the 0.05 level using Bonferroni correction.
shown in Table 3. Using only our 38 intensifiers, the marker preference representation performs sur- prisingly well, substantiating the importance of in- tensifiers in communicating stance. However, our stance context representations perform even better at predicting cross- cross-posting (1 4.28, p 0.001, Co- hen's is d 1.79), confirming the key role of stance context features in community linguistic practices.
of specialized vocabulary varies with four social network properties in online communities. Specif- ically, Lucy and Bamman (2021) found that com- munitics with more distinctive use of words and of word senses are smaller (fewer users), more active (higher average participation among users), more loyal (more users whose participation fo- cused in that community), and more dense (more interactions among users). We sought to explore whether stancetaking patterns similarly in online communitics - do smaller, more active, more loyal, and denser communities develop more distinctive manner of expressing stance?
To determine which features of stance affect how users cross- pross-participate on subreddits, we also exam- ined the standardized coefficients of a regression model that treats each predictor separately. We found that 7 of our 10 stance context features were significant predictors of cross-posting, including 4 of the 5 extremeness features. Similarity in for- mality and extremeness of valence were especially predictive of cross-posting, The full results can be found in Appendix G.
For this question, we explore distinctiveness stancetaking style with respect to both stance con- texts and marker preferences, and relate these to the four social factors explored in Lucy and Bamman (2021).² We conceptualize distinctiveness [cc, R) of a community other communities on average, using representation R. Let C be our set of 1798 communities:
Thus we show here that variation in stance con- texts is indeed predictive of the community mem- bership decisions that Redditors make. Despite only capturing how communities use intensifiers, both marker preference similarity and stance con- text similarity do reasonably well at predicting cross-posting patterns on Reddit. Our stance con- text representations obtained 7% higher accuracy on the task, suggesting that higher-level contextual properties related to stance are particularly impor- tant for shaping user participation patterns. Further- more, our interpretable stance context properties allowed to study the aspects of stancetaking that most shape co-participation online. Each of these findings helps shed light on how people engage across multiple communities practice.
how dissimilar it is from all
E (1
c,,(R,c,,)))
dec
[cc, R)
|C|
Then, for each representation R (stance context and marker preference), we perform correlations between the (cc R) values for all 1798 communi- ties, and the values of each of the four social factors calculated for the communities; see Table 4.
First, we see that distinctiveness in marker pref- erence is significantly correlated with loyalty: this is consistent with the finding of Lucy and Bamman (2021) that more loyal communities have more dis- tinctive vocabulary usage. However, in contrast to Lucy and Bamman (2021), distinctive usage of stance markers not correlated with any other network properties. Presumably the set of stance markers we consider are general enough vocab- ulary items that their preference patterns are not
6.2 Distinctiveness in Stancetaking Style
Our cross-posting analysis confirmed that the ways people express stance are related to how they choose to engage across multiple communities. We now turn to the extensive sociolinguistic findings referenced in Section 2 that linguistic practices also interrelate with community structure. For exam- plc, Lucy and Bamman (2021) showed that the use
hetween training sets across folds results in an inflated Type error for the standard paired t-test (Dictterich, 1998).
WWe adopted the formulas used by Lucy and Bamman (2021) for these factors, deseribed in Appendix H.

---

highly associated with how users engage within community.
ics and community structure from different lens than previous work. Our findings illustrate that, higher level properties of linguistic practice, stanc- etaking contexts may hold in common across dif- ferent detailed communities. Our results overall highlight the importance of investigating the rich and varied linguistic means for expressing commu- nity identity.
Second, and more notably, stance context dis- tinctiveness patterns in a manner inverse to what Lucy and Bamman (2021) found for distinctiveness of vocabulary. "0 Communities that are more dis- tinct in their stance contexts tend to be larger, less active, less loyal, and less dense. To investigate what drives stance context distinctiveness to pat- tern this way, we examine the communities by topic grouping. Specifically, we find that of the 303 least distinctive quartile of communities that have a topic assigned, 41% are Sports communities and 26% are Video Games communitics. This is coher- ent with our results in Section 5, where we showed that despite having low textual similarity, Sports and Video Games communities have rather high stance context similarity. Since they form a siz- able collection of subreddits, their mutual similar- ity leads to them having among the lowest average distinctiveness of our communities.
7 Conclusion
We extend work on community variation in stanc- ctaking by constructing stance context representa- tions using theoretically-motie mottivtted linguistic prop- erties of stance. Since stancetaking plays a key role in linguistic variation (Jaffe, 2009), considering these higher level properties of stance may help shed light on why community members present themselves in particular ways. We showed that our stance context representations capture aspects of community linguistic identity distinct from exist- ing methods. Furthermore, we found that these representations were related to inter- and intra- community engagement patterns.
Moreover, among the 13 contentful topic groups (excluding General, Other, and Not Assigned), we find that Sports communities tend to be the small- est, densest, most loyal, and most active overall, while Video Game subreddits are larger but also among the highest in density, loyalty, and activ- ity. Qualitatively, we suspect Sports and Video Game communities foster this degree of local en- gagement given allegiances people have to their preferred teams and games.
Our stance contexts approach also allowed us to reveal "mega" communities: those like Sports and Video Games that share stancetaking properties while having numerous fine-grained subcommuni- ties within them. Further work with stance contexts may help shed additional light on the nuances of mega community structure: in addition to estab- lishing communities for different subtopics (such as different video games), are "spin-off" commu- nities (cf. Hessel et al., 2016) sometimes created so that users may discuss the same topic while tak- ing alternative stance (e.g. r/pokemontrades r/casualpokemontradess]
Thus, our findings distinctiveness in stanc- etaking context reveal novel aspects of the relation- ship between linguistic practice and community structure. Because Sports and Video Game com- munities are highly represented on Reddit over- all, it's not surprising that these break down into a large number of subcommunities that are more top- ically fine-grained (on particular sports, teams, and games). These communities will then be among the smaller, denser, more loyal, and more active subreddits, but because they have stance context properties in common, these social network prop- erties are associated with the least distinctive com- munities stance context.
Additionally, by showing that higher level prop- erties of stance relate to community identity and engagement patterns, we pave the way for much richer models of the reasons that people are at- tracted to, or discouraged from, online participa- tion in various groups. For example, on Reddit, a platform dominated by men, high-arouse stances are the most common due to the overrepresentation of Sports and Vidco Games communities. Future work could explore whether other kinds of stances (e.g. more polite or higher valence stances) lead to increased participation and engagement from women and people with a (stereotypically) femi- nine linguistic style.
Furthermore, the contrast with distinctiveness in vocabulary both our marker preferences here and specialized vocabulary in Lucy and Bamman (2021) allow us view Reddit platform dynam-
We replicated our results on the communities from Lucy and Bamman (2021) with similar results; see Appendix I.

---

8 Limitations
out on novel words word senses used in on- line contexts, and their effect on our five linguistic property values. Future research should work to understand how the stance-relevant properties of contexts vary across communities and over time.
We first discuss limitations related to our specific methodological choices, and then discuss limita- tions of our approach more generally.
Furthermore, although drew literature from sociolinguiisiies and linguistic anthropology in developing our stance contexts representations. our approach does not capture the full richness of stance as understood these fields. For example, the highly influential stance triangle framework from Du Bois (2007) theorizes that stancetaking involves a speaker evaluating something (a "stance object"), and that - in so doing the speaker po- sitions themself relative to their interlocutor. Be- cause these components of stance sometimes mentioned in the sentential context (which we use to generate our stance representations), we may implicitly capture them in some cases. However, within our framework, it is not straightforward to compare these components of stance to human in- tepretations (cf. Kiesling et al., 2018).
8.1 Methodological choices
First, we studied the contexts where speakers use intensifiers kind of stance marker - but we con- sidered only 38 intensifiers in our main analyses. As follow-up, we ran additional experiments that included set of 252 intensifiers from Luo al. (2019) across 448 of our communities selected for another project. These experiments confirm that both our cross-posting and social factors results hold for much larger set of intensifiers. Full de- tails of the results can be found in Appendix B. However, it may be fruitful to replicate anal- yses with a larger set of stance markers (including markers beyond intensifiers), such as the of 812 stance markers considered by Pavalanathan al. (2017). This would allow us to ensure our insights hold for stance markers generally.
Related to this, method only captures local aspects of stance contexts within the sentence in which intensifiers were used. However, work on stance contexts has discussed how the sequential context of the preceding utterances a dialogue shape the construction and interpretation of one's ss stance (Kiesling et al., 2018; Bohmann and Ahlers, 2022). For example, the preceding utterances could inform whether stance is interpreted sarcastic or not. Future research should work to develop even richer representations of stance, which are also scalable.
There are also limitations to our method for as- sessing stance-relevant properties. We focused on valence, arousal, dominance, politeness, and for- mality, but other linguistic properties have been identified as relevant to stance contexts, including subjectivity and certainty (Kärkkäinen, 2006; Kies- ling et al., 2018). Future work on stancetaking may benefit from the development of methods for assessing these properties.
8.2 General approach
There are also some limitations of our stance con- texts approach more generally. First, our approach to assessing stance properties doesn't account for community-level semantic variation. For instance, work has shown that the same word can vary in sentiment, depending on the community (Hamilton al., 2016; Lucy and Mendelsohn, 2019) Be- cause our approach for assessing stance-relevant properties depends on context words (which can vary in meaning across communities), it may miss out on some community-speciffc meanings of stance contexts.
8.3 Data Access
One additional limitation is related to the future ac- cessibility of our data, as the Pushshift Data Dumps we used to extract Reddit data are no longer avail- able at their original link (as of May 2023). Future research will need use the official Reddit API to access the data we used.
Ethical Considerations
User privacy is concern inherent to using online data. The data use was public when collected by Baumgartner al. (2020), and we take care remove data from any individuals that had deleted their accounts any point prior to the data's col- lection, adhering as best we can to a user's right to be forgotten.
Relatedly, approach doesn't account for vari- ation in how stances are communicated over time; the datasets we use to train our models for assess- ing stance-relevant properties (Mohammad, 2018; et al., 2013; Pavlick and Tetreault, 2016) are 5-10 years old, and likely miss

---

Mohammad (2022) mentions that in cases where one automatically infers emotional properties (such as VAD scores), it is critical to be cognizant that users may not want their data analyzed, as well of the potential harms of associating such information with individual users. Furthermore, the automatic assessment of emotional properties of text may not match the intended affect of the user, and this prob- lem may be more pronounced for users who part of marginalized groups (Mohammad, 2022). With these concerns in mind, we remove user-level information prior to extracting values for our lin- guistic features, such that our downstream analyses have no connection between individual users and the emotional aspects of their language. Our textual and marker preference representations also make of user identifiers. We only make use of user- identifying information for computing community engagement behavior, and only release information about the aggregated statisties per community. Al- though the sentences we use in our dataset were created by individual users, we mitigate user pri- vacy concerns as best as possible in the ways listed above.
19 discourse on Reddit. Proceedings the 1st Workshop on NLP for COVID-19 19 at ACL 2020.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The Pushshift Reddit Dataset. In Proceedings of the inter- national AAAI conference on web and social media. volume 14, pages 830-839
Erin D Bennett and Noah D Goodman. 2018. Extremely costly intensifiers stronger than quite costly ones. Cognition, 178: 147-161.
Douglas Biber and Edward Finegan. 1989. Styles of stance in English: Lexical and grammatical mark- ing of evidentiality and affect. Texi-nntenisyiilinnary journal for the study of discourse, 91):93-124.
Su Lin Blodgett, Lisa Green, and Brendan O' C'Cnnor 2016. Demographic dialectal variation in social media: A case study of African-American English. arXiv preprint 160i::1008.08860
Axel Bohmann and Wiebke Ahlers. 2022. Stance in nar- ration: Finding structure in complex sociolinguistic variation. Journal of Sociolinguistics, 26(1):65- 83,
Gemma Boleda, Sabine Schulte im Walde, and Toni Badia. 2012. Modeling regular polysemy: A study on the semantic elassification of Catalan adjectives. Computational Linguistics, 38(3):575-616.
Another ethical concern pertains the amount of compute power we used for our analyses, partic- ularly for data extraction and creating our sentence embeddings (see Appendix J). For the latter pro- cess, we minimized unnecessary computation by computing the embeddings for each sentence only once. This means that we did not need to recom- pute embeddings for each of our five linguistic properties, and that we can extend our methodol- ogy to include other linguistic properties without needing to use any more GPU power for creating embeddings. To mitigate additional compute costs for researchers seeking to replicate our work, we make our sentence-level stance representations pub- lic.
Dwight Bolinger. 1972. Degree words. In Degree Words. De Gruyter Mouton.
Mary Bucholtz. 2009. From stance to style: Gender, interaction, and indexicality in Mexican immigrant youth slang.
Mary Bucholtz and Kira Hall. 2005. Identity and in- teraction: A sociocultural linguistic approach. Dis- course studies, 7(4-5):585-614,
John D Burger. John Henderson, George Kim, and Guido Zarrella. 201 I. Discriminating gender on twit- ter. In Proceedings of the Conference on Empiri- cal Methods in Natural Language Processing, pages 1301-1309.
Jenny Cheshire. 1982 Variation in an English dialect: A sociolinguistic study. Cambridge Studies in Lin- guistics London, 37.
Acknowledgements
Cristian Danescu-Nicullseu-MMe Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. arXiv preprint arXiv:3306.678.
We acknowledge the support of NSERC of Canada (through grant RGPIN-2017-06506 to SS), well the support of the Data Sciences Institute, Uni- versity of Toronto (through a Catalyst Grant SS and JW).
Marco Del Tredici and Raquel Fernández. 2017. Se- mantic variation in online communities of practice. In Proceedings of the 12th International Conference Computational Semantics.
References
Marco Del Tredici, Raquel Fernández, and Gemma Boleda. 2018. Short-term meaning shift: distribu- tional exploration. arXiv preprint
Jai Aggarwal, Ella Rabinovich, and Suzanne Stevenson. 2020. Exploration of gender differences in COVID-

---

Thomas G Dietterich. 1998. Approximate statistical tests for comparing supervised classification learning (17):188) 1895-1923.
Scott Kiesling, 2009, Style as stance. Stance: soci- olinguistic perspectives, pages 171-194.
algorithms. Neural computation, 10(7):
Scott F Kiesling, Umashanthi Pavalanathan, Jim Fitz- patrick. Xiaochuang Han, and Jacob Eisenstein. 2018. Interactional stancetaking in online forums. Compu- tational Linguistics, 44(4):683-718.
John W Du Bois. 2002. Stance and consequence. In Anmual Meeting of the American Anthropoloical Association, New Orleans.
John W Du Bois. 2007. The stance triangle. Stancetak- ing in discourse: Subjectivity, evaluation, interaction, 164-3)113--182.
Wang Ling, Chris Dyer, Alan W Black, and Isabel Tran- coso. 2015. Two/too simple adaptations of word2vec for syntax problems. conference of the North American chapter of the Association for Computational I. Linguistics: Human Language Technologies, pages 1299-1304.
Proceedings of the 2015
Penelope Eckert. 2000. Language variation as social practice: The linguistic construction of identity in Belten High. Wiley-Blackwell.
Li Lucy and David Bamman. 2021. Characterizing En- glish variation across social media communities with BERT. Transactions of the Association for Computa- tional Linguistics, 9:538-556.
Penelope Eckert. 2008. Variation and the indexical field
1. Journal of sociolinguisties, 12(4):453-47) 476..666.......
Penelope Eckket and Sally McConnell-Ginet. 1992. Think practically and look locally: Language and gender as community- practice. Annual review of antthropoloyy, 21(1):461-455
Li Lucy and Julia Mendelsohn. 2019. Using sentiment induction understand variation in gendered online communities. Proceedings of the Society for Com- putation in Linguistics (SCiL) 2019, pages 156-166.
William L Hamilton, Kevin Clark, Jure Leskovec, and Dan Jurafsky. 2016. Inducing domain-specific senti- ment lexicons from unlabeled corpora. In Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 595-605.
Yiwei Luo. Dan Jurafsky, and Beth Levin. 2019. From insamely jealous to insanely delicious: Computational models for the semantic bleaching of English in- tensifiers. In Proceedings of the Ist International Workshop on Computational Approaches to Histori- cal Language Change, pages 1-13.
Jack Hessel, Chenhao Tan, and Lillian Lee. 2016. Sci- ence, askscience, and badscience: On the coexistence of highly related communities. In Proceedings of the international AAAI conference web and social media, volume 10, pages 171-180.
Lesley Milroy. 1987. Language and social networks, volume 2. Wiley-Blackwer
Saif Mohammad. 2018. Obtaining reliable human rat- ings of valence, arousal, and dominance for 20,000 English words. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (volume 1: Long pupers), pages 174 184.
Janet Holmes and Miriam Meyerhoff. 1999, The com- munity practice: Theories and methodologies in language and gender research. Language in society, 28(2):173-188 1833.
Abraham Israeli, Shani Cohen, and Oren Tsur. 2022. Unsupervised discovery of non-trivial similarities be- tween online communities. Expert Systems with Ap- plications, 206:117900.
Saif M Mohammad. 2022. Ethics sheet for automatic emotion recognition and sentiment analysis. Compu- tational Linguistics, 48(2):239-275 278.
Rika Ito and Sali Tagliamonte. 2003. Well weird, right dodgy, very strange, really cool: Layering and recy- cling English intensifiers. Language in society, 32(2):257-279.
Claude Nadeau and Yoshua Bengio. 2003. Inference for the generalization error. Machine Learning, 52:239- 281.
Dong Nguyen, Rilana Gravel, Dolf Trieschnigg, and Theo Meder. 2013. "How old do you think am?" A study of language and age in Twitter. In Proceedings of the International AAAI Conference Web and Social Media, volume 7, pages 439-448.
Alexandra Jaffe. 2009. Stance: Sociolinguistic Perspec- rives. Oxford University Press
Dan Jurafsky, Elizabeth Shriberg, Barbara Fox, and Traci Curl. 1998. Lexical, prosodic, and syntactic cues for dialog acts. In Discourse Relations and Discourse Markers.
Bill Noble and Jean Chin-ppilpppe Bernardy. 2022. Condi- tional language models for community- level linguis- tic variation. In Proceedings of the Fifth Workshop Natural Language Processing and Computational Social Science (NIP+ CSS), pages 59-78.
Elise Kärkkäinen. 2006. Stance taking in conversation:
From subjectivity to intersubjectivity. Text & talk, 2696):699-731
Elinor Ochs. 1993. Constructing social identity: A
Scott Kiesling. 2004. Dude. American speech, 79(3):281-305.
language socialization perspective. Research on lan- guage and social interaction, 26(3):287-306.

---

Umashanthi Pavalanathan, Jim Fitzpatrick, Scott F Kies- ling. and Jacob Eisenstein. 2017. A multidimensional lexicon for interpersonal stancetaking. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 884-895.
absolutely
plain
amazingly
pretty
awfully
quite
considerably
really
decidedly
remarkably
downright
seriously
Ellie Pavlick and Joel Tetreault. 2016. An empiri cal analysis of formality in online communication. Transactions of the Association for Computational Linguistics, 4:61-74.
enormously
significantly
exceedingly
simply
exceptionally
surprisingly
excessively
suspiciously
Hans Peters. 1994. Degree adverbs in Early Modern English. Studies in early modern English, 13:269- 288.
extraordinarily
terribly
extremely
totally
fantastically
tremendously
Mirka Rauniomaa. 2003. Stance accretion. Language, Interaction, and Social Organization Research Fo- cus Group, University of Califfornia, Santa Barbara, February.
hugely
truly
incredibly
unbelievably
insanely
unusually
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. arXiv preprint arXiv:1908, 10084.
intensely
utterly
largely
wholly
noticeably
wonderfully
James A Russell. 2003, Core affect and the psycholog- ical construction of emotion. Psychological review, I 1145
Table 5: List of intensifiers used in this study.
Farhan Samir, Barend Beekhuizen, and Suzanne Steven- son. 2021. A formidable ability: detecting adjectival extremeness with DSMs. In Findings of the Associ- ation for Computational Linguistics: ACL-IJCNLP 2021, pages 41 112555125
We started with the seed set of stance markers released by Biber and Finegan (1989), which in- cludes 448 markers in total. For a separate project, we required the stance categories that this corpus assigns markers to (e.g. affective stance markers. emphatic stance markers, etc.), so this set was more appropriate than the stance markers released in the Switchboard corpus (Jurafsky et al., 1998), which does not contain category- information.
Devyani Sharma and Robin Dodsworth. 2020. Lan- guage variation and social networks. Annual Review of Linguistics, 6:341-361.
Sali Tagliamonte and Chris Roberts. 2005. So weird; so cool; so innovative: The use intensifiers the tele- vision series friends. American speech, 80(3):280- 300.
From this initial set of seed stance markers, we search for other stance markers on Reddit used in similar distributional contexts. To do so, we use Ling et al. (2015) (an adaptation of Word2 that accounts for syntactic information). following the method of Pavalanathan et al. (2017). Unlike Pavalanathan et al. (2017), we train Wang2vec on all words that oceur at least 5 times in our sample of 25M comments from 2014, resulting in a vocab- ulary of size 1.2M. We adjust their thresholds for similarity, extracting all candidate words having a cosine similarity of at least 0.85 with one of our seed markers, and a general frequency of at least 10 10-6
Isaac Waller and Ashton Anderson. 2019. Generalists and specialists: Using community embeddings to quantify activity diversity in online platforms. In The World Wide Web Conference, pages 1954-1966
Justine Zhang, William Hamilton, Cristian Danescu- Nieulescu-Mizil, Dan Jurafsky, and Jure Leskovec. 2017. Community identity and user engagement in multi-community landscape. In Proceedings of the international AAAI conference on web and social media, volume I1, pages 377-386.
A Constructing the Stance Marker and Intensifier Lists
To extract set of stance markers common Red- dit, we replicated the lexical expansion procedure used by Pavalanathan al. (2017). We initially fo- cused our analyses on Reddit data from 2014, and performed lexicon expansion using this dataset.
This process resulted in 1118 stance markers in total, of which 40 are intensifiers found in Bennett and Goodman (2018). We remove the two stop words most and very to arrive at our final set of 38 intensifiers, shown in Table 5.

---

B Expanded Analyses
representations, as well our overall findings, are robust to the selection of intensifiers.
In our main analyses, we focused on a relatively small number of intensifiers (38), which may raise concerns about whether our findings generalize to other intensifiers. To address this, performed the same cross- cross-posting and social factors analyses on the 252 intensifiers from Luo et al. (2019) on a subset of 448 subreddits (selected for different project).
C Data Preprocessing
We apply preprocessing to the comment text ex- tracted from Reddit, including: We replace all links with a [LINK] token, replace all mentions of an- other username with a FU token, and replace irregular unicode characters, extrancous parenthe- ses, and newlines with whitespace. Malformed quotation marks and apostrophes are also replaced with the appropriate token. We then strip tokens used to indicate bold italicized text and replace multi callti-whttsppcce characters with a single character.
Of our 38 stance markers, 28 are found in this set of 252 markers. The 10 markers that are not in the set of 252 markers are: downright, fantasticaal incredibly, largely, plain, pretly, quile, really, sus- piciously, and unbelievably. We repeat our analyses on the 252 markers collected for the new dataset, as well as on the 28 markers both datasets have in common.
At the author-level, we remove any comments written by deleted authors, AutoModerators, and account names that end in "bot", regardless of case.
Across both our original and additional analy- ses, our stance context representations achieve very consistent results. We expand each analysis in the sections below.
D Details of Models to Infer Linguistic Properties
To create SBERT representations of sentences in our data, use as it performs the best on the STS task without fine-tuning (Reimers and Gurevych, 2019).
B.1 Cross- Cross-Posting Results
For the cross-posting analysis, we recompute high and low cross pross-posting pairs using the same method- ology as in the main text, but for the of 448 communities in this analysis. Table 6 shows the results, where the leftmost column corresponds to the results the main text. The stance context rep- resentations significantly outperform the marker preference representations in all cases, at a 0.05 using the same 10x10 cross-validation procedure used in the main text. This suggests that stance context representations are consistently better at predicting cross- poosss-osting than the marker preference similarity representations.
For valence, arousal, and dominance, we have ac- cess to the NRC-VAD lexicon (Mohammad, 2018) 1)1), which provides values of these features for 20K words, but we need a model that assigns such val- ues sentences. We follow the method of Ag- garwal et al. (2020) by training Beta regression models on SBERT representations of words from the NRC- VAD lexicon, and then applying those models to our Reddit sentence data. We adjusted the methodology of Aggarwal et al. (2020) by train- ing the models on 80% of the data, stratified over quintiles of each of the VAD scores. This stratifi- cation ensured that our training data was approx- imately uniform across the [0, 1] interval. We peated this procedure 10 times and chose the model that performed best for each feature. Our best mod- els achieve Pearson correlations of 0.85, 0.77, and 0.80 on their held-out sets for valence, arousal, and dominance respectively. comparable to those found by Aggarwal et al. (2020).
B.2 Social Factors Analysis
We repeat the social factors analysis for the stance context representations using the same procedure outlined in the main text. In the dataset with 448 communities, distietiveness is calculated only with respect to these 448 communities. Table 7 shows our results.
As in the case of the cross-posting analysis, the correlations between stance context distinctiveness and the four social factors also shows the same pattern as in the main text results across all the new analyses, albeit with some reduction in the power and magnitude of correlation when only using the 28 overlapping intensifiers. This suggests that our
For politeness, we followed the methodology of Danescu-Niculedt Niculescu-Mizli et al. (2013) to build logistic regression model predict whether a given sentence polite. We adapted their approach to give normally scores by taking the log-odds of the predicted po- liteness probability, and then min-max scaled the
continuous politeness

---

Marker Preference Similarity
Stance Context
Similarity
38 markers
75.1% 1%
82.3%
Original Dataset (n=1798)
28 markers
70.6%
81.0%
232 markers
76.5%
79.8%
New Dataset (n=448)
28 markers
73.3%
78.5%
Table 6: Cross-posting results for both our original and new datasets with the full and intersected marker lists.
Size
Activity
Loyalty
Density
38 markers
0.21
-0.36
-0.34
-0.34
Original Dataset (n=1798)
28 markers
0.18
-0.36
-0.26
-0.30
252 markers
0.14
-0.32
-0.34
-0.33
New Dataset (n=448)
28 markers
0.13
-0.35
-0.29
-0.31
Table 7: Spearman correlations of social factors and stance context distinctiveness in our datasets; bold indicates significance at the a = 0.05 level using Bonferroni correction applied to 8 tests.
resulting values to the range (0, 1]. We trained and tested our models on both their Wikipedia corpus and their StackExchange corpus, performing both in-domain and cross-domain tests. In-domain tests were conducted using 3x10 cross-validation, and the cross-domain tests were conducted by training the model on the entire training set and testing on the entire testing set. We found that the model trained the Wikipedia text performed better than the model trained the StackExchange text, with in-domain and cross-domain accuracies of 84. 1% and 65.22 2% respectively, comparable to the models in Manescu-Niculcscu-Miiiii
Topic
# Subreddits
Entertainment
176
Video Games
159
Lifestyle
Sports
142
Discussion
109
Other
106
Locations
96
Hobbies
91
Education
69
Technology
50
Politics
21
al. (2013).
Humor
19
For the formality model, we selected the best model after 10-fold cross validation. The Spearman correlation of our best model with its hold-out set 0.76, higher than those found in Pavlick and Tetreault (2016).
General
18
NSFW
15
Animals
7
Table Number of subreddits per topic.
E Subreddit Topic Groups
the Animals topic are more similar to each other according textual representations than stance context representations.
The list of topic groups we used, and the number of our 1798 subreddits assigned to each, are shown in Table 8.
G Stance Properties and Crossposting
F Topic-Tore Topic Differences in Representations
To assess each feature's 's individual contribution to predicting crossposting, we used a logistic regres- sion model with 10 predictors: the differences be- tween a pair of communities property dimension. To improve our model for this task, we unit-normalized each community stance context representation. The coefficients are shown in Table 9.
To further visualize differences in how topics dif- fer across three representations, graphs in Figure 3. These graphs were generated by subtracting the values in each cell of Figure between the corresponding heatmaps. For instance, the lef-left cell of the leftmost subfigure of Fig- ure 3 can be interpreted as follows: subreddits in
show the
particular stance

---

Difference between Stance Context and Textual Similarity for Topic Pairs
Difference between Stance Context and Marker Preference Similarity for Topic Pairs
Difference between Marker Preference and
Textual Simiiarity for all Topic Pairs
enterairment
dhcrnsion
education
eduration
denussio
10
entertammenn
discusioon
ediracion
sumor
10
0.5
Thetyte
0.0
Pecye
00
locetlrr
-4.
lccdions
boacions
455
ndo
sports
pollicn
-10
sports
-10
tectnology
tedmooogy
-15
technology
vido games
vdouggimss
-20
wido_games
ninde
Figure 3: Patterns of topic-topic differences for each pair of representations. Darker cells represent a greater difference between the pairs representations. Blue cells are those that are more similar with respect to text than other representations, red cells are similar with respect stance context than other representations, and purple cells are more similar with respect to marker preference than other representations.
Raw
Extreme
Size
Activity
Loyalty
Density
Valence
-2.85**
.4.04**#
Stance
0.35
-0.42
-01.42
-0.46
Arousal
-1.83
-1.13
Context
Dominance
-0.53
-2.32*
Marker
-0.08
0.08
0.26
0.13
Politeness -2.75**
-
Preference
Formality -7.80***
-3.04**
Vocab.
-0.60
0.58
0.70
0.69
L&B,2021
Table 9: Feature importance at predicting cross-posting. Asterisks indicate significance at 0.05 (*), 0.01 (**), and 0.001 (***), respectively.
Table 10: Spearman correlations of social factors and distinctiveness for the 418 communities that overlap be- tween dataset and that of Lucy and Bamman (202 )); bold indicates significance at the 0.05 level using Bon- ferroni correction.
Computing Social Factors
As mentioned, adopt the formulas used by Lucy and Bamman (2021) for computing the social fac- tors size, activity, density, and loyalty. All data is gathered from the set of 1.2B comments written in 2019.
cabulary. We wanted to build on their work by exploring the role of contextual properties beyond individual words and word senses, so in our main analyses, we focused on our novel contribution of how the social factors relate to distinctiveness aspects of stancetaking.
For community size, we use the number of dis- tinct users that have ever posted in community, For user activity, divide the number of total comments in community in 2019 by the commu- nity size. To compute network density, we first construct an undirected parent-reply network for the top 20% users by commenting volume in 2019. For each community, we then compute the density as the number of users who have an edge between them, dividing by the total number of pos- sible edges in the graph. Finally, to compute loy- alty, we find the proportion of users in community that have at least 50% of their top-level comments in 2019 in particular community.
Here, we directly compare how stance context distinctiveness and vocabulary distinctiveness re- late to social factors. To do this, we use the data released by Lucy and Bamman (2021) and com- pute vocabulary distinctiveness scores for the 118 communities that our dataset has in common with theirs."
The results for our stance context and marker preference distinctiveness, as well as vocabulary distinctiveness, are shown in Table 10. For this set of subreddits, we find the same pattern of results for both of our distinctiveness measures as the full of subreddits, and see the same pattern of results for vocabulary distinctiveness as reported in Lucy and Bamman (2021) for their full set sub-
I Social Factors Analysis for Vocabulary Distinctiveness
past work, Lucy and Bamman (2021) examined how social factors relate to distinctiveness in vo-
"I Lin to Githuh for I Lucy and Bamman (2021): https: //github. com/ htmm/ucy3///m//mmmm htt///nnnnoppllaaeeeee

---

reddits. Here we also found that Sports and Video Games subreddits are again (as in our full results) the most represented topics among communities in the lowest quartile of stance context distinctiveness (48% and %, respectively). Conversely, those topic groups are also most represented in the high- est quartile of vocabulary distinctiveness (38% and 23%, respectively).
J Hardware
To extract our data, we used 12 CPUs parallel, each of which took roughly 8 hours to extract data from the Pushshift data dumps. For computing sentence embeddings, we used two Nvidia Titan X GPUs, which processed about 1.1M sentences per hour (in batches of size 32). This took roughly hours per GPU, for a total of about 16 GPU hours.


Reviews:
[{"review_id": "QaBC0WWWxE", "reviewer": "Reviewer_Qr3C", "summary": "", "questions": "1. Can you measure the human correlations of the model-based scores presented in Section 3.2.1? There may not be a dataset annotated with the proposed features but I believe the construction of a small evaluation dataset is possible with some annotator costs.\n2. Why have you chose SBERT instead of BERT in the models proposed in Section 3.2.1? Since these models are regression models, I believe the use of CLS token in BERT is simpler than the use toke-level mean-pooling representation of SBERT.\n3. Could you elaborate more how the three topic-topic pairs are chosen in Section 5.2? Are these pairs randomly chosen or are these high similarity pairs for Text, Stance and Marker? The characteristic found in Sports & Video Games pair is interesting but I wonder whether this kind of pair is common in Reddit.\n4. Can't you examine the Spearman correlations of the social factors against the Textual similarity in Section 6.1? Since Textual vs. Stance Context has a high correlation in Table 1, I wonder the Textual similarity can also capture the characteristics seen in the Stance Context similarity.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We very much appreciate your constructive feedback, which will help us improve the paper, as detailed below. We respond to your first reason to reject when responding to your first question, and your second reason to reject when responding to your third and fourth questions.\n\n# Questions for the Authors\n**First reason to reject: The proposed stance context features are model-based but their qualities are not confirmed against ground truth scores such as human scores.**\n\n**Q1: Can you measure the human correlations of the model-based scores presented in Section 3.2.1? There may not be a dataset annotated with the proposed features but I believe the construction of a small evaluation dataset is possible with some annotator costs.**\n\nWe tested our feature models in Section 3.2.1 on held-out portions of datasets with human annotations of the features, and all models achieved accuracies comparable to prior work. (The models achieved accuracies of .85, .77., .80, and .76 for valence, arousal, dominance, and formality in comparison to the gold standard; for politeness, we achieved in-domain accuracy of .84 [cross-validation on Wikipedia] and cross-domain of .65 [train Wikipedia/test Stack Exchange].) This was reported in Appendix C and will be summarized in the main body of the camera-ready paper.\n\n\n**Q2: Why have you chose SBERT instead of BERT in the models proposed in Section 3.2.1? Since these models are regression models, I believe the use of CLS token in BERT is simpler than the use toke-level mean-pooling representation of SBERT.**\n\nWe used SBERT because it has been shown to outperform the BERT CLS token on three sentiment prediction tasks (Reimers & Gurevych, 2019). Since our stance context features are high-level text properties similar to sentiment, we expected SBERT representations to be better for predicting our stance context features. We will make this motivation more explicit in Section 3.2 in the camera ready.\n\n**Second reason to reject: The comparison of topic-topic similarities and the investigation of the cross-posting behavior are quite limited. I will further elaborate this weakness in Q3 and Q4 in the following questions section.**\n\n**Q3: Could you elaborate more how the three topic-topic pairs are chosen in Section 5.2? Are these pairs randomly chosen or are these high similarity pairs for Text, Stance and Marker? The characteristic found in Sports & Video Games pair is interesting but I wonder whether this kind of pair is common in Reddit.**\n\nWe manually selected the pairs in Section 5.2 as illustrations of differences in what each of our three representations capture. Some other pairs of subreddit topics also pattern like Sports and Video Games –  i.e., having much higher stance context similarity than text similarity; e.g., Sports and Technology, NSFW and Politics, and Locations and Entertainment. For the camera-ready version we are exploring visualizations that would make it easier to identify pairs of subreddits that differ in their similarity according to different representations.\n\n\n\n**Q4: Can't you examine the Spearman correlations of the social factors against the Textual similarity in Section 6.1? Since Textual vs. Stance Context has a high correlation in Table 1, I wonder the Textual similarity can also capture the characteristics seen in the Stance Context similarity.**\n\nLucy and Bamman (2021) have shown how the social factors relate to distinctiveness in vocabulary (which we replicate on our data in Appendix G). We wanted to build on this work by exploring the role of contextual properties beyond individual words and word senses, hence the focus on our novel contribution of how the social factors relate to distinctiveness in aspects of stancetaking. We will clarify this in Section 6 of the camera-ready paper.\n\n\n# References\nReimers, N., & Gurevych, I. (2019, November). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)* (pp. 3982-3992).\n\nLucy, L., & Bamman, D. (2021). Characterizing English variation across social media communities with BERT. *Transactions of the Association for Computational Linguistics*, 9, 538-556.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed representation has shown that it can capture a topic-topic similarity such as one between Sports & Video Games which can not be seen in a textual similarity or a marker similarity.\n2. It is interesting to see that the social factors of the cross-posting behavior is quite different in the stance context and the marker preference.", "reasons_to_reject": "1. The proposed stance context features are model-based but their qualities are not confirmed against ground truth scores such as human scores.\n2. The comparison of topic-topic similarities and the investigation of the cross-posting behavior are quite limited. I will further elaborate this weakness in Q3 and Q4 in the following questions section.", "questions_to_authors": "1. Can you measure the human correlations of the model-based scores presented in Section 3.2.1? There may not be a dataset annotated with the proposed features but I believe the construction of a small evaluation dataset is possible with some annotator costs.\n2. Why have you chose SBERT instead of BERT in the models proposed in Section 3.2.1? Since these models are regression models, I believe the use of CLS token in BERT is simpler than the use toke-level mean-pooling representation of SBERT.\n3. Could you elaborate more how the three topic-topic pairs are chosen in Section 5.2? Are these pairs randomly chosen or are these high similarity pairs for Text, Stance and Marker? The characteristic found in Sports & Video Games pair is interesting but I wonder whether this kind of pair is common in Reddit.\n4. Can't you examine the Spearman correlations of the social factors against the Textual similarity in Section 6.1? Since Textual vs. Stance Context has a high correlation in Table 1, I wonder the Textual similarity can also capture the characteristics seen in the Stance Context similarity.", "paper_topic_and_main_contributions": "This paper proposes a stance context representation that can capture community identity patterns differently from textual or stance marker measures. The properties of the representation consist from the 5 features of affect, politeness and formality. For each feature, the paper prepares a model based on SBERT to assign the feature score to a sentence. The representation is evaluated on a Reddit dataset with inter- and intra-community patterns and the evaluation highlights the strength of using the proposed context representation.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o80LBvNhBW", "reviewer": "Reviewer_NA9c", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We very much appreciate your constructive feedback, which will help us improve the paper, as detailed below.\n\n# Reasons to Reject\n\n**The focus on intensifiers, and just 38 intensifiers in English, leads me to question some of the generalizability of the findings. Would these hold if other stance markers other than intensifiers were examined?**\n\nTwo reviewers raised questions/concerns about related issues: why we focus on intensifiers, why we limited ourselves to just 38 intensifiers (and how those were selected), and whether the work generalizes to other intensifiers and other stance markers. We answer those questions here, with further detail on our motivations and methods, as well as additional experiments demonstrating generalizability; we’ll correspondingly address these issues in the camera-ready paper.\n\nMany reasons motivated the focus on intensifiers, some of which are summarized in the intro to Section 3 (which we will expand, and preview in the Intro), including their ubiquity, range of meaning, and variation associated with social factors. In addition, because stance markers cover a range of grammatical classes (verbs, adjectives, interjections, and others), we wanted to focus on a single class to avoid finding variation in stance contexts that is solely/largely due to variation in the grammatical contexts of such a diverse set of words. Future work will need to explore whether the contextual properties of other stance markers are equally informative about community behavior.\n\nWe started with the intensifiers from Bennett & Goodman (2018) because they are known to communicate a range of degree of intensification (which was manually annotated in that work), and intersected this list with a list of stance markers likely to appear frequently on Reddit (generated using Pavalanathan et al.’s, 2017, methodology). We also removed two stopwords (*very* and *most*) given their pervasiveness in language, to ensure the words were informative.\n\nWhile the restriction to 38 words means we have to be cautious in assuming generalizability of all our results, this small set of words (chosen for task-independent reasons) is highly predictive of cross-posting behavior, indicating that stancetaking properties are indeed very powerful indicators of identity and community behavior.\n\nThat said, we understand the concern that our results based on a small set of words may not generalize.  To address this, we performed the same cross-posting and social factors analyses (from Section 6) using the 252 intensifiers from Luo et al. (2019), on a set of 448 subreddits (selected for a different project).\n\nAcross all new and submitted analyses, we achieve very consistent results using our stance context representations.  In predicting user cross-posting, stance context similarity significantly outperforms stance marker similarity in every case (Table 1 below).  The correlations between stance context distinctiveness and the 4 social factors also shows the exact same pattern as in the submitted results across all the new analyses, albeit with some reduction in power/magnitude of correlation when only using the 28 overlapping intensifiers (Table 2 below).\n\nTable 1: Results for our cross-posting analyses. We repeated the analyses on all 252 intensifiers in the 448 subreddits; we also did analyses on the 28 intensifiers that overlap between the Luo et al. dataset and our original 38 intensifiers so that we could compare the exact same set of intensifiers across the original 1798 subreddits and the 448 subreddits. The stance context representations significantly outperform the marker preference representations in all cases, at α=0.05 (using a corrected resampled t-test, as in the paper).\n\n\n$$\n\\\\begin{array}{l|r|r}\n & \\\\text{Marker preference similarity} & \\\\text{Stance context similarity} \\\\\\ \n\\text{Submitted Dataset (38 original markers)} & 75.1\\\\% & 82.3\\\\% \\\\\\ \n\\text{Submitted Dataset (28 markers that overlap between datasets)} & 70.6\\\\% & 81.0\\\\% \\\\\\ \n\\text{New Dataset (252 markers from Luo et al., 2019)} & 76.5\\\\% & 79.8\\\\% \\\\\\ \n\\text{New Dataset  (28 markers that overlap between datasets)} & 73.3\\\\% & 78.5\\\\% \\\\\\ \n\\\\end{array}\n$$\n\n\nTable 2: Results for our social factors analyses. We repeated the analyses on all 252 intensifiers in the 448 subreddits; we also did analyses on the 28 intensifiers that overlap between the Luo et al. dataset and our original 38 intensifiers so that we could compare the exact same set of intensifiers across the original 1798 subreddits and the 448 subreddits. Bolded cells are significant at α=0.05, after applying Bonferroni correction. \n$$\n\\\\begin{array}{l|r|r|r|r}\n& \\\\text{Size} & \\\\text{Activity} & \\\\text{Loyalty} & \\\\text{Density} \\\\\\ \n\\text{Submitted Dataset (38 original markers)} & \\mathbf{0.21} & \\mathbf{-0.36} & \\mathbf{-0.34} & \\mathbf{-0.34} \\\\\\ \n\\text{Submitted Dataset (28 markers that overlap between datasets)} & \\mathbf{0.18} & \\mathbf{-0.36} & \\mathbf{-0.26} & \\mathbf{-0.30} \\\\\\ \n\\text{New Dataset (252 markers from Luo et al., 2019)} & \\mathbf{0.14} & \\mathbf{-0.32} & \\mathbf{-0.34} & \\mathbf{-0.33} \\\\\\ \n\\text{New Dataset  (28 markers that overlap between datasets)} & 0.13 & \\mathbf{-0.35} & \\mathbf{-0.29} & \\mathbf{-0.31} \\\\\\ \n\\\\end{array}\n$$\n\n\n**Some sort of embedding-based approach might be richer than SVD on tf-idf subreddit-word matrix for measuring the text similarity, but is probably not strictly necessary, especially if the goal is to match surface-level vocabulary instead of semantic vectors of words.**\n\nYour observation about matching surface-level vocabulary is indeed along the lines of our thinking. The very richness of embeddings means that they are likely to capture some of the aspects of stance that we’re trying to contrast with more direct textual similarity. Our use of SVD follows from similar work investigating the relationship between community textual similarity and and other targeted aspects of community language (Lucy & Mendelsohn, 2019; Pavalanathan et al., 2017).\n\n# Typos Grammar Style And Presentation Improvements\n\n**It would provide a stronger motivation to give some example research questions that could be answered with stance contexts at the end of the intro. The second to last paragraph of the intro suggests some motivation for looking at stance contexts by connecting to affect, politeness, etc, but this is not super concrete on what can be done with it.**\n\nIn our final paper, we will be clearer about the research questions that our stance context representations can help answer. For example, while we mention user engagement and network properties in the intro, it may not be clear that our findings reveal interesting aspects of the structure of Reddit: “mega” communities like Sports and Video Games that share stancetaking properties while having numerous fine-grained subcommunities. Further work with stance contexts may help shed additional light on why some communities are structured this way: in addition to establishing different subtopics (such as different sports teams), are “spin-off” communities (cf. Hessel et al., 2016) sometimes created so that users may discuss the same topic but taking an alternative stance? In addition, future work can consider how the stances that communities take may be important for understanding community growth and diversity. For example, on the male-dominated Reddit platform, does the use of more polite or higher valence stances lead to increased participation from women?\n\n**More examples from the dataset of stances taken as captured in stance contexts would be illustrative**\n\nWe can include example stance context representations for sentences from our dataset in the final paper.  To give a sense of the data here, we provide the feature values for two of the examples in the intro that differ in valence, as well as another sentence that differs in formality:\n\n$$\n\\\\begin{array}{l|r|r|r|r|r}\n& \\\\text{Valence} & \\\\text{Arousal} & \\\\text{Dominance} & \\\\text{Politeness} & \\\\text{Formality} \\\\\\ \n\\text{I hope you are incredibly proud of yourself.} & 0.77 & 0.67 & 0.57 & 0.70 & 0.32 \\\\\\ \n\\text{Both of you are incredibly stupid} & 0.16 & 0.61 & 0.27 & 0.25 & 0.22 \\\\\\ \n\\text{This would be incredibly valuable scientifically for studying the origins of life.} & 0.65 & 0.58 & 0.74 & 0.64 & 0.91\n\\\\end{array}\n$$\n\n\n\n\n**The focus on intensifiers as stance marker of choice is quite important and should be mentioned earlier on.**\n\nWe agree, and will indicate our focus on intensifiers and motivation for doing so in the introduction rather than waiting until Section 3.\n \n**A clear definition of stance marker would be helpful in intro**\n\nIn the final paper, we will expand on the definition of stance markers as follows: “Stance markers are words that demarcate stance, including, among others, intensifiers (*really*, *insanely*, *terribly*), modals (*might*, *should*), and evaluative words (*like*, *love*, *hate*)”.\n\n**Line 378 , -> .**\n\nWe will fix the typo on line 378.\n\n# References\n\nBennett, E. D., & Goodman, N. D. (2018). Extremely costly intensifiers are stronger than quite costly ones. *Cognition*, 178, 147-161.\n\nHessel, J., Tan, C., & Lee, L. (2016). Science, askscience, and badscience: On the coexistence of highly related communities. In *Proceedings of the international AAAI conference on web and social media* (Vol. 10, No. 1, pp. 171-180).\n\nLuo, Y., Jurafsky, D., & Levin, B. (2019, August). From insanely jealous to insanely delicious: Computational models for the semantic bleaching of English intensifiers. In *Proceedings of the 1st International Workshop on Computational Approaches to Historical Language Change* (pp. 1-13).\n\nPavalanathan, U., Fitzpatrick, J., Kiesling, S. F., & Eisenstein, J. (2017, July). A multidimensional lexicon for interpersonal stancetaking. In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)* (pp. 884-895).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Stancetaking is an important and under-explored concept from sociolinguistics that can be useful in social NLP/computational social science. As the authors note, it has been shown to motivate many of the language choices users make and can act as an intermediary between linguistic features and identity. This paper does a great job motivating this need and bridging non-computational work in sociolinguistics with computational NLP work. It's an exemplary related work section, which is not an easy task given how unrelated these fields typically are.\n\nTheir qualitative analysis of differences in communities wrt stance context, stance markers, and textual similarity is informative, especially the examples of politics/education and sports/video games with low to moderate textual similarity but high stance context or stance marker similarity. The prediction task of cross-posting is convincing that stance context can capture individuality across subreddits, and the finding that similarity in formality and extremeness is most related to cross-posting behavior is valuable when thinking about what types of draw different online communities may have. \n\nTheir nuance to Lucy and Bamman 2021's findings that smaller, denser communities may have distinct vocabs but not distinct stancetaking is valuable to our knowledge of how language operates wrt identity in online communities.\n", "reasons_to_reject": "The focus on intensifiers, and just 38 intensifiers in English, leads me to question some of the generalizability of the findings. Would these hold if other stance markers other than intensifiers were examined?\n\nSome sort of embedding-based approach might be richer than SVD on tf-idf subreddit-word matrix for measuring the text similarity, but is probably not strictly necessary, especially if the goal is to match surface-level vocabulary instead of semantic vectors of words.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper aims to model the sociolinguistic concept of stancetaking, in particular the \"stance context\", words immediately around stance markers. They aim to move beyond a focus on individual words/stance markers in computational modeling of stancetaking while also not requiring extensive manual annotation (as noted in lines 177-178). Focusing on intensifiers as stance markers, they extract features of affect (VAD), politeness, formality from contexts and use these as a representation of stance context, which they find contrasts subreddits in different ways than stance marker use and textual similarity. They then show that stance contexts can better predict cross-posting/community membership than just stance markers. Finally, they test if smaller, more active, denser communities have unique stancetaking. They find they do not, in contrast to prior work that shows that these types of communities are lexical innovators.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* It would provide a stronger motivation to give some example research questions that could be answered with stance contexts at the end of the intro. The second to last paragraph of the intro suggests some motivation for looking at stance contexts by connecting to affect, politeness, etc, but this is not super concrete on what can be done with it.\n* More examples from the dataset of stances taken as captured in stance contexts would be illustrative\n* The focus on intensifiers as stance marker of choice is quite important and should be mentioned earlier on.\n* A clear definition of stance marker would be helpful in intro\n* Line 378 , -> ."}}, {"review_id": "HrEXQ7wx4n", "reviewer": "Reviewer_5R1c", "summary": "", "questions": "Since the paper frames itself as focusing on stancetaking, one may expect it to investigate a variety of stance markers. Would this study be extendable to other types of stance markers beyond intensifiers? Why did the authors restrict it to just intensifiers? \n\nWhy is the text-based representation of communities left out of the user cross-posting prediction analysis (Table 2)? \n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your constructive feedback, which will help us improve the paper. Since they are on related issues, we respond to your first reason to reject and first question for the authors at once.\n\n# First Reason to Reject + First Question for the Authors\n\n**Reason to Reject: Lines 210-224, which describe the process for deciding what stance intensifiers to use in this study, was confusing as to why the authors made the decisions that they did. The authors start with a wide pool of potential markers (89 intensifiers intersected with a list of 1000 stance markers) but then at the end they keep only 38. Why were so many intensifiers removed? The main text should be able to stand alone from the Appendix, but I also looked in the Appendix and it raised even more questions. For example, why use Bennett and Goodman (2018) to filter out which intensifiers to include?**\n\n**Question for the Authors: Since the paper frames itself as focusing on stancetaking, one may expect it to investigate a variety of stance markers. Would this study be extendable to other types of stance markers beyond intensifiers? Why did the authors restrict it to just intensifiers?**\n\nTwo reviewers raised questions/concerns about related issues: why we focus on intensifiers, why we limited ourselves to just 38 intensifiers (and how those were selected), and whether the work generalizes to other intensifiers and other stance markers. We answer those questions here, with further detail on our motivations and methods, as well as additional experiments demonstrating generalizability; we’ll correspondingly address these issues in the camera-ready paper.\n\nMany reasons motivated the focus on intensifiers, some of which are summarized in the intro to Section 3 (which we will expand, and preview in the Intro), including their ubiquity, range of meaning, and variation associated with social factors. In addition, because stance markers cover a range of grammatical classes (verbs, adjectives, interjections, and others), we wanted to focus on a single class to avoid finding variation in stance contexts that is solely/largely due to variation in the grammatical contexts of such a diverse set of words. Future work will need to explore whether the contextual properties of other stance markers are equally informative about community behavior.\n\nWe started with the intensifiers from Bennett & Goodman (2018) because they are known to communicate a range of degree of intensification (which was manually annotated in that work), and intersected this list with a list of stance markers likely to appear frequently on Reddit (generated using Pavalanathan et al.’s, 2017, methodology). We also removed two stopwords (*very* and *most*) given their pervasiveness in language, to ensure the words were informative.\n\nWhile the restriction to 38 words means we have to be cautious in assuming generalizability of all our results, this small set of words (chosen for task-independent reasons) is highly predictive of cross-posting behavior, indicating that stancetaking properties are indeed very powerful indicators of identity and community behavior.\n\nThat said, we understand the concern that our results based on a small set of words may not generalize.  To address this, we performed the same cross-posting and social factors analyses (from Section 6) using the 252 intensifiers from Luo et al. (2019), on a set of 448 subreddits (selected for a different project).\n\nAcross all new and submitted analyses, we achieve very consistent results using our stance context representations.  In predicting user cross-posting, stance context similarity significantly outperforms stance marker similarity in every case (Table 1 below).  The correlations between stance context distinctiveness and the 4 social factors also shows the exact same pattern as in the submitted results across all the new analyses, albeit with some reduction in power/magnitude of correlation when only using the 28 overlapping intensifiers (Table 2 below).\n\nTable 1: Results for our cross-posting analyses. We repeated the analyses on all 252 intensifiers in the 448 subreddits; we also did analyses on the 28 intensifiers that overlap between the Luo et al. dataset and our original 38 intensifiers so that we could compare the exact same set of intensifiers across the original 1798 subreddits and the 448 subreddits. The stance context representations significantly outperform the marker preference representations in all cases, at α=0.05 (using a corrected resampled t-test, as in the paper).\n\n\n$$\n\\\\begin{array}{l|r|r}\n & \\\\text{Marker preference similarity} & \\\\text{Stance context similarity} \\\\\\ \n\\text{Submitted Dataset (38 original markers)} & 75.1\\\\% & 82.3\\\\% \\\\\\ \n\\text{Submitted Dataset (28 markers that overlap between datasets)} & 70.6\\\\% & 81.0\\\\% \\\\\\ \n\\text{New Dataset (252 markers from Luo et al., 2019)} & 76.5\\\\% & 79.8\\\\% \\\\\\ \n\\text{New Dataset  (28 markers that overlap between datasets)} & 73.3\\\\% & 78.5\\\\% \\\\\\ \n\\\\end{array}\n$$\n\n\nTable 2: Results for our social factors analyses. We repeated the analyses on all 252 intensifiers in the 448 subreddits; we also did analyses on the 28 intensifiers that overlap between the Luo et al. dataset and our original 38 intensifiers so that we could compare the exact same set of intensifiers across the original 1798 subreddits and the 448 subreddits. Bolded cells are significant at α=0.05, after applying Bonferroni correction. \n$$\n\\\\begin{array}{l|r|r|r|r}\n& \\\\text{Size} & \\\\text{Activity} & \\\\text{Loyalty} & \\\\text{Density} \\\\\\ \n\\text{Submitted Dataset (38 original markers)} & \\mathbf{0.21} & \\mathbf{-0.36} & \\mathbf{-0.34} & \\mathbf{-0.34} \\\\\\ \n\\text{Submitted Dataset (28 markers that overlap between datasets)} & \\mathbf{0.18} & \\mathbf{-0.36} & \\mathbf{-0.26} & \\mathbf{-0.30} \\\\\\ \n\\text{New Dataset (252 markers from Luo et al., 2019)} & \\mathbf{0.14} & \\mathbf{-0.32} & \\mathbf{-0.34} & \\mathbf{-0.33} \\\\\\ \n\\text{New Dataset  (28 markers that overlap between datasets)} & 0.13 & \\mathbf{-0.35} & \\mathbf{-0.29} & \\mathbf{-0.31} \\\\\\ \n\\\\end{array}\n$$\n\n# Remaining Reasons to Reject\n\n**In the paper, the authors reduce Sentence BERT vectors into 10 features related to affect, formality, and politeness predicted from these vectors. I’m not fully convinced as to why politeness, affect (measured by NRC-VAD), and formality are key to measuring stancetaking. Why not just take the average of SBERT vectors instead of reducing those vectors down to these narrow linguistic properties? It’s possible that the 10-dimensional feature vectors offer interpretability, but it’s also possible that SBERT vectors may include richer information that would be helpful for understanding what concepts communities tend to take stances on.**\n\nThe stance context properties we use have been theorized to be important aspects of how people position themselves during dialogue (Jaffe, 2009; Kiesling, 2009; Pavalanathan et al., 2017; Kiesling et al., 2018). SBERT representations almost certainly include additional semantic information, but this additional information may not be directly relevant to stancetaking. In addition, the interpretability offered by our methodology is an important aspect for us, as it allows us to better understand facets of community identity. For example, in addition to demonstrating that the Sports and Video Games subreddits are similar in stancetaking, we can see that a clear contributing factor is that both have distinctively high-arousal stance contexts, reflecting the competition-oriented nature of these subreddits. That said, an avenue for future work would be to do our analyses using SBERT representations to try to further determine the multitude of factors – stance context, vocabulary, topic, etc. – that relate to user dynamics on a social media platform.\n\n**When it comes to weighing the potential contribution or impact of this paper, I would emphasize Section 6 (relating stance to engagement behavior) more so than Section 5 (where the authors argue that intensifiers’ contexts provide a distinct view of communities’ language from possible alternatives). That is, this paper could be even stronger if it showed more experiments that inform the reader about potential use cases where stance context representations are more favorable than other text-based representations.**\n\nWe felt that Section 5 was important to include as a basic check of soundness to illustrate that our stance context representations capture something different from other text/marker representations, and may shed light on relations among online communities. That being said, we agree with the reviewer that Section 6 is a more stringent analysis of our stance context representations, and those experiments are intended as our main contributions. While space prohibited further experimentation, in our final paper, we will include examples of other research questions that may be explored with our stance context representations. \n\nFor example, in the paper we observed “mega” communities like Sports and Video Games that share stancetaking properties while having numerous fine-grained subcommunities. Further work with stance contexts may help shed additional light on why some communities are structured this way: in addition to establishing different subtopics (such as different sports teams), are “spin-off” communities (cf. Hessel et al., 2016) sometimes created so that users may discuss the same topic but taking an alternative stance? In addition, future work can consider how the stances that communities take may be important for understanding community growth and diversity. For example, on the male-dominated Reddit platform, does the use of more polite or higher valence stances lead to increased participation from women?\n\n# Remaining Questions for the Authors\n\n**Why is the text-based representation of communities left out of the user cross-posting prediction analysis (Table 2)?**\n\nPrevious work has explored the efficacy of text-based measures on predicting cross-posting (Pavalanathan et al., 2017). In our work, we focus on the novel contribution of comparing how different aspects of stancetaking behaviour predict how users co-participate in online communities.\n\n# Missing References\n\n**Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora, Hamilton et al. EMNLP 2016.**\n\nThank you for pointing this out; we will discuss the Hamilton et al. (2016) paper in both our Related Work section and in the Limitations section (Section 8.2).\n\n# Typos Grammar Style And Presentation Improvements\n\nWe appreciate the 6 editing suggestions below, and will implement each of them in our final paper.\n\nRegarding our dataset (point 4 below), we use the full year of data from 2019. Although (at the time we extracted it) the 2020 data was the most recent data available in the Reddit data dumps (Baumgartner et al., 2020), we opted for using 2019 data because we were concerned about the impact that Covid-19 and Reddit’s June 2020 ban of controversial communities may have had in that data. We will clarify this in the paper in Section 3.1.\n\n1. **Line 86 says that no prior large-scale work has related variation in stance contexts to cross-platform engagement, which seems to imply that the current paper fills this gap by examining cross-platform engagement. However, the authors investigate cross-community engagement on a single platform (Reddit), so I would consider revising line 86 to change “cross-platform” to “cross-community” or something else.**\n2. **Line 219: the authors state that they used Pavalanathan et al. 2017’s approach to extend the original stance intensifier lexicon to additional markers found on Reddit. I understand that due to page limit constraints, the authors redirect the reader to read the Appendix here, but it’d be helpful if the authors could also briefly describe the gist of the approach in the main text as well.**\n3. **You should remove the keywords you list under the abstract as this is unusual for EMNLP.**\n4. **Line 227-228 could be revised to clarify if the dataset contains the entire year of 2019 or just parts of 2019. Also, 2019 data is no longer considered “recent” in 2023, although I acknowledge that with the end of Pushshift earlier this year, obtaining more recent Reddit data is trickier now than before.**\n5. **Lines 321-326 may be clearer as to why there are 10 features total by revising “…added an ‘extremeness’ version of each feature in our…” to “…added an ‘extremeness’ version of each feature introduced in \\S 3.2.1 in our…”.**\n6. **If the authors are looking to trim text for space reasons, I’d recommend shortening lines 357-358 into a single sentence as cosine similarity is a well-known approach in NLP venues and does not need much explanation.**\n\n# References\n\nBaumgartner, J., Zannettou, S., Keegan, B., Squire, M., & Blackburn, J. (2020, May). The pushshift reddit dataset. In *Proceedings of the international AAAI conference on web and social media* (Vol. 14, pp. 830-839).\n\nBennett, E. D., & Goodman, N. D. (2018). Extremely costly intensifiers are stronger than quite costly ones. *Cognition, 178*, 147-161.\nJaffe, A. (Ed.). (2009). *Stance: sociolinguistic perspectives*. Oup Usa.\n\nKiesling, S. F. (2009). Style as stance. *Stance: sociolinguistic perspectives*, 171.\n\nKiesling, S. F., Pavalanathan, U., Fitzpatrick, J., Han, X., & Eisenstein, J. (2018). Interactional stancetaking in online forums. *Computational Linguistics, 44*(4), 683-718.\n\nPavalanathan, U., Fitzpatrick, J., Kiesling, S. F., & Eisenstein, J. (2017, July). A multidimensional lexicon for interpersonal stancetaking. In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)* (pp. 884-895).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is generally well written. For example, the authors’ use of a few key examples in the introduction was a nice way to provide background on the nuances of operationalizing stancetaking. I appreciate that the authors grounded the motivation behind their approach using research from sociolinguistics, and took care to outline how their work fills gaps in prior computational work (e.g. lines 174-178). \n\nOverall, this paper could benefit others’ ongoing research on how language relates to community identity at scale. \n", "reasons_to_reject": "Lines 210-224, which describe the process for deciding what stance intensifiers to use in this study, was confusing as to why the authors made the decisions that they did. The authors start with a wide pool of potential markers (89 intensifiers intersected with a list of 1000 stance markers) but then at the end they keep only 38. Why were so many intensifiers removed? The main text should be able to stand alone from the Appendix, but I also looked in the Appendix and it raised even more questions. For example, why use Bennett and Goodman (2018) to filter out which intensifiers to include?\n\nIn the paper, the authors reduce Sentence BERT vectors into 10 features related to affect, formality, and politeness predicted from these vectors. I’m not fully convinced as to why politeness, affect (measured by NRC-VAD), and formality are key to measuring stancetaking. Why not just take the average of SBERT vectors instead of reducing those vectors down to these narrow linguistic properties? It’s possible that the 10-dimensional feature vectors offer interpretability, but it’s also possible that SBERT vectors may include richer information that would be helpful for understanding what concepts communities tend to take stances on.  \n\nWhen it comes to weighing the potential contribution or impact of this paper, I would emphasize Section 6 (relating stance to engagement behavior) more so than Section 5 (where the authors argue that intensifiers’ contexts provide a distinct view of communities’ language from possible alternatives). That is, this paper could be even stronger if it showed more experiments that inform the reader about potential use cases where stance context representations are more favorable than other text-based representations. \n\nI describe additional suggestions for improving this work in “Typos Grammar Style and Presentation Improvements” below, but did not include them in this section because I don’t think each point of negative-ish feedback is severe enough to be considered in paper decisions. \n", "questions_to_authors": "Since the paper frames itself as focusing on stancetaking, one may expect it to investigate a variety of stance markers. Would this study be extendable to other types of stance markers beyond intensifiers? Why did the authors restrict it to just intensifiers? \n\nWhy is the text-based representation of communities left out of the user cross-posting prediction analysis (Table 2)? \n", "paper_topic_and_main_contributions": "This paper investigates how stancetaking in communities can shed light on community identity and language at scale in nearly 1.8k communities on Reddit. The authors operationalize stancetaking by creating “stance context” vector representations of communities. First, they mask out intensifiers (e.g. “incredibly”) in sentences and generate the Sentence BERT (SBERT) representation of these sentences. Then, they reduce these SBERT representations into smaller vectors containing dimensions related to valence, arousal, dominance, formality, and politeness using various predictive models, and then aggregate these to the community-level by taking their average. The authors show that these representations are distinct from prior ways of measuring a community’s stance or language, and more accurate at predicting user cross-posting behavior. Finally, the authors show that stancetaking relates to user loyalty, activity, density, and size in directions that differ from past work on linguistic distinctiveness. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora, Hamilton et al. EMNLP 2016. "}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 86 says that no prior large-scale work has related variation in stance contexts to cross-platform engagement, which seems to imply that the current paper fills this gap by examining cross-platform engagement. However, the authors investigate cross-community engagement on a single platform (Reddit), so I would consider revising line 86 to change “cross-platform” to “cross-community” or something else. \n\nLine 219: the authors state that they used Pavalanathan et al. 2017’s approach to extend the original stance intensifier lexicon to additional markers found on Reddit. I understand that due to page limit constraints, the authors redirect the reader to read the Appendix here, but it’d be helpful if the authors could also briefly describe the gist of the approach in the main text as well. \n\nYou should remove the keywords you list under the abstract as this is unusual for EMNLP.\n\nLine 227-228 could be revised to clarify if the dataset contains the entire year of 2019 or just parts of 2019. Also, 2019 data is no longer considered “recent” in 2023, although I acknowledge that with the end of Pushshift earlier this year, obtaining more recent Reddit data is trickier now than before. \n\nLines 321-326 may be clearer as to why there are 10 features total by revising “…added an ‘extremeness’ version of each feature in our…” to “…added an ‘extremeness’ version of each feature introduced in \\S 3.2.1 in our…”. \n\nIf the authors are looking to trim text for space reasons, I’d recommend shortening lines 357-358 into a single sentence as cosine similarity is a well-known approach in NLP venues and does not need much explanation. \n"}}]

Author rebuttal:

Meta-review: On the whole, reviewers thought that this was an interesting paper. They appreciated the sociolinguistic motivations for the focus/approach, and the ways that the proposed context representations could enrich our understanding of communities, beyond previous literature. 

The reviewers had a few concerns about the methodological choices and paper framing. They felt that these concerns could be addressed via some writing changes, especially since the authors started to provided clarifications in their response. I’ve highlighted some main suggestions, and I encourage the authors to attend to these suggestions — as well as others that the reviewers raised — as they revise their paper.
* Reviewers found the choice of focusing on intensifiers somewhat arbitrary. It would be great if they could elaborate on/foreground the point briefly made, that controlling for grammatical variation is important, and if they could include the analysis with the full set of markers presently included in their response to reviewers. In particular, reviewers feel that including the additional analyses would greatly improve the soundness of the paper.
* Reviewers note that the additional explanatory power offered by the proposed representations is a strength of the paper, and ought to be emphasized. For instance, one reviewer suggests emphasizing S6, relating stance to behavior, over S5 — in the interests of showing the reader examples of _how_ the representations could be used in other analyses. 
* More work could be done to validate the representations, especially with human judgements.
Decision: Accept-Findings