TalkUp: Paving the Way for Understanding Empowering Language
Authors: Lucille Njoo
Chan Young Park
Octavia Stappart
Marvin Thielk
Yi Chu
Yulia Tsvetkov
Venue: EMNLP 2023 Findings (2023)
Keywords: natural language processing
empowerment
TL;DR: This paper presents TalkUp, a dataset of labeled Reddit posts, for studying empowerment in language and its relationship with social context.

Abstract: Empowering language is important in many real-world contexts, from education to workplace dynamics to healthcare. Though language technologies are growing more prevalent in these contexts, empowerment has seldom been studied in NLP, and moreover, it is inherently challenging to operationalize because of its implicit nature. This work builds from linguistic and social psychology literature to explore what characterizes empowering language. We then crowdsource a novel dataset of Reddit posts labeled for empowerment, reasons why these posts are empowering to readers, and the social relationships between posters and readers. Our preliminary analyses show that this dataset, which we call TalkUp, can be used to train language models that capture empowering and disempowering language. More broadly, TalkUp provides an avenue to explore implication, presuppositions, and how social context influences the meaning of language.

TalkUp: Paving the Way for Understanding Empowering Language
Lucille Njoo* Chan Young Park Octavia Stappart*
Marvin Thielk Yi Chu Yulia Tsvetkov
University of Washington & Carnegie Mellon University
(Injoo, ostapp, yuliats washington. edu chanyoun@cs. cmu. (marvin. thielk, yi yiwoooiee@gaailll..... com
Abstract
Post 1
r/relationhhips
sad. mourn of someone you love whether they were great they were awful. its normal and you shouldnt feel weird or bad because of it. Good luck :)
know
Empowering language is important in many real-world contexts, from education to work- place dynamics healtheare. Though lan- guage technologies are growing more prevalent these contexts, empowerment has seldom been studied in NLP, and moreover, it is in- herently challenging to operationalize because of its implicit nature. This work builds from linguistic and social psychology literature to ex- plore what characterizes empowering language. We then crowdsource novel dataset of Reddit posts labeled for empowerment, reasons why these posts are empowering to readers, and the social relationships between posters and read- ers. Our preliminary analyses show that this dataset, which we call TalkUp, can be used to train language models that capture empowering and disempowering language. More broadly, TalkUp provides avenue to explore implica- tion, presuppositions, and how social context influences the meaning of language.'
Response: Thanks much. will keeping
Is the post empowering?
Amblguous?
Di Disempowering rigg
Neutral
Empowering
Yes
No
Type of empowerment
Has rights
Not
Can express
Pasitive self-inage
alone
enotions
Poster commenter agreement?
Disagree
Neutral
Agree
man, r/relationships
Post 2 little bit of inattention isnt worth breaking up
Response: Youre stupid.
Is the post empowering?
Ambiguous?
No Consensus
Yes
No
Poster commenter agreement?
1 Introduction
Disagree
Neutral
Agr
Empowerment the act of supporting somcone's ability to make their own decisions, create change, and improve their lives - is goal in many social interactions. For instance, teachers aim empower their students, social workers aim empower their clients, and politicians aim empower their sup- porters. growing body of psychology and lin- guistics research shows how empowerment- and disempowerment- - can impact people by increasing their sense of self-efficacy and self-esteem (Cham- berlin, 1997; Osborne, 1994).
Figure 1: Two examples of annotated conversations in TalkUp. Post straightforwardly empowering, but Post is inherently ambiguous and could either terpreted as helpful advice comment. Social context can also affect Post 2's im- plications: the post might elicit different reactions if it were written by a woman to man vice versa.
dismissive, belitting
and Rambow, 2014a; Prabhakaran et al., 2012). and healthcare (Locke et al., 2021; Sharma et al., 2021a). Whether we are building dialogue agents for mental health support, supplementing children's 's education, or analyzing managers' feedback to their employees, language that empowers disempow- the reader have drastically different effects.
Understanding how empowerment is conveyed in language becomes more important as language technologies are increasingly being used in inter- active contexts like education (Molnár and Szüts, 2018), workplace communication (Prabhakaran
With a few exceptions (Ziems et al., 2022: Sharma et al., 2023), prior NLP research has fo- cused on flagging harmful text, but there has been
"Equal contribution
'The data, codes, task instructions, and annotation Ul can he found at https://ithub..co http:://ithuh.commccaffpppp com/chandpprkk//akkk

---

much less investigation of what makes text help- ful. Other works have studied related concepts like condescension (Wang and Potts, 2019) and implicit toxicity (Breitfeller et al., 90; Sap et al., 2020; Upadhyay et al., 2022), and we build off of these to construct a dataset that complements those tasks.
ing intersectionality as well as longer multi-turn dialogues.
Background
We discuss empowerment following its definitions in clinical psychology (Chamberlin, 1997). We find this most appropriate for studying language because clinical psychology practice is usually cen- tered around dialogue between clinician and pa- tient, and because it involves concrete implications about individuals rather than vague cultural phe- nomena. Thus, summarizing the different char- acteristics of empowerment described in psychol- ogy literature, we define empowering text as text that supports the reader's rights, choices, self- fulfillment, or self-esteem.
Consider the two examples of potentially em- powering interactions in Figure 1. Empowerment exhibits the importance of social context in under- standing the pragmatics of language: whether an exchange is interpreted as empowering disem- powering may depend on the participants' social roles and the power dynamics implied by their iden- tities, including race, age, socioeconomic class, and many other social dimensions. Furthermore, empowerment cannot be easily detected with senti- ment or emotion analyzers, since interactions with negative implicatures can be empowering (e-g., you can quit!!!), and messages that are positive on the surface can be disempowering (e.g., you are so ar- ticulate for a girl!) (Field and Tsvetkov, 2020). Modern language technologics do not model social context or deeper pragmatic phenomena, and thus are unable to capture or control for empowerment.
Incorporating empowerment in dialogue agents, mental health support chatbots, educational assis- tants, and other social-oriented NLP applications is clearly a desirable goal. However, empowerment is inherently challenging to operationalize for several reasons. First, is a flexible term that describes a wide range of behaviors across many domains - em- powerment economics, for example, looks very different from empowerment in therapy session (McWhirter, 1991). We follow recent literature outside of NLP in trying to distill these varied inter- actions into concrete definition. Second, empow- erment is implicit: it often read in between the lines rather than declared explicitly. Text might be empowering by reminding someone their range of options to choose from, encouraging them to take action, asking for and valuing their opinion, or even validating their feelings (Chamberlin, 1997). Third, empowerment is heavily dependent on so- cial context: whether or not person is empowered depends who is saying what whom. We in- corporate these consideration in our data collection process described next.
This work makes concrete steps towards under- standing these linguistic phenomena by investigat- ing the following research questions: [RQ1] What makes language empowering, and how is it mani- fested in language? [RQ2] Can empowerment be detected with computational approaches?
Our contributions are threefold: (1) We intro- duce the new task of empowerment detection, grounding it in linguistic and psychology literature. (2) We create TalkUp, a novel dataset of Reddit posts labeled for empowerment, the fine-grained type of empowerment felt by the reader, and the social relationships between posters and readers. (3) We analyze the data and demonstrate how it can be used to train models that can capture empow- ering and disempowering language and to answer questions about human behavior.
3 The TalkUp Dataset
We now discuss the TalkUp dataset's construction.
Ultimately, TalkUp aims to assist future re- searchers in developing models that can detect, generate, and control for empowerment, and to facilitate broader exploration of pragmatics. We have by means covered every possible social dimension, but by focusing on few social factors in the simplified setting of two-turn dialogues, we hope that TalkUp's framework can make strides toward understanding language in more complex social interactions, such as conversations involy-
Annotation Scheme Our annotation task? was shaped through multiple pilot studies, where we learned that context is useful for judging a post, an- notators" free-response descriptions of social roles lack consistency, and posts are often inherently ambiguous. We elaborate on these findings in Ap- pendix D. Based on these insights, the final task,
Seercenshots of user interface and full guidelines including definitions and examples of each label are in Appendix F.

---

which is illustrated in Figure 1, consists of three main parts:
from Reddit because they were the most gener- alizable and contained natural-sounding conversa- tions. We manually chose five subreddits, aiming to include (1) a diverse range of topics and user de- mographics, and (2) discussions that are personal rather than about external events unrelated to the conversants. The subreddits are listed in Table 1. We filtered data from these subreddits to exclude posts or responses that exceeded 4 sentences in length or were shorter than 5 words. Additionally, we excluded posts with "Redditisms", and posts that were edited after they were initially posted (marked "EDIT:" by the original poster) and posts that began with quoted text (marked ">>) were removed.
(1) Rating the post on an empowerment scale. This scale has "eemoowring" on one end, "dis- empowering" one the other, and "neutral" in the middle. We define text to be empowering if it sup- ports the reader's s rights, choices, self-esteem, and disempowering if it actively denies or discourages these things. Notably, posts that discuss an external topic without making any implications about the conversants, such as a com- ment about a celebrity's lifestyle, are defined as neutral.
(2) Selecting why a post is empowering dis- empowering. We adopt the 15 points from Cham- berlin (1997), with slight modifications to adapt them written text, as reasons why a post can be empowering to a reader. Refer to Appendix for the complete list of 15 reasons and correspond- ing definitions provided to annotators. a post is empowering, it should imply one or more of these reasons (e.g. that the reader is capable of creating change), and if it is disempowering it should imply the opposite (e.g. that the reader is not capable of creating change).
From pilot studies, we found that models can help to surface potentially empowering posts and help increase the yield of posts that were actually labeled as empowering by annotators. We trained RoBERTa-basee regression model with the data collected from the pilot studies to predict the level of empowerment (0 for disempowering. 0.5 for neutral, for empowering) in Reddit posts. We used this model to rank and select the top-k posts for annotation, and continually updated the model as we collected more data." To ensure we annotate a diverse range of posts, our final annotation task was done with half model-surfaced posts and half randomly-sample posts.
(3) Selecting whether the poster and commenter have agreeing or disagreeing stances. We define "agreeing" and "disagreeing" loosely in order to accommodate a wide range of social relationships: "agree" means that the poster and reader support the same point of view a topic, whether it be poli- ties, sports teams, or music preferences. "Disagree" means that they take opposing sides.
A Annotation on Amazon Mechanical Turk With 1k model-surfaced posts and 1k randomly-sampled posts spread evenly among the five subreddits, we collected annotations via Amazon Mechanical Turk (AMT). Appendix F shows a screenshot of the user interface displayed to annotators. Each example was annotated by 3 different workers.
Data Source TalkUp consists of English Reddit posts from RtGender (Voigt et al., 2018), collec- tion of 25M comments posts from five differ- ent domains, each labeled with the genders of the commenter and the original poster. We take advan- tage of the fact that these conversations are already annotated for gender, which provides contextual information about who is speaking to whom and al- lows us to explore at least one dimension of social context. 3
To ensure high quality annotations, we required annotators have AMT's Masters Qualification.' 5 a task approval rate of at least 95%, and mini- mum of 100 prior tasks completed. Additionally, since our task requires English fluency, we limited annotators to those located in the US or Canada. Workers were compensated at $15/hour, and we calculated the reward per task based on the average time spent each annotation in our pilot studies. Following best practices to increase annotator diversity (Casey et al., 2017), we staggered batches
Though RtGender contains posts from several platforms, given our focus on conversational lan- guage, we specifically selected RtGender posts
"We only consider men and women here due to the avail- ability of data. We were not able to find any corpora that included nonbinary genders, but this is an important area for future work. Though we focus on gender, there are many other social variables that may impact empowerment, such as race and socioeconomic status.
*More information on the sample selection model is pro- vided Appendix B.1. $ 'MMT grants the "Master Worker" qualification to highly reliable workers.

---

Size
#E
#A
%
Wordd Count
TalkUp
2000
962
129
267
43
Tone (Pos)
AskReddit
400
186
43
Clout
relationships
400
199
35
83
72
Impersonal Pronouns
Fitness
400
193
64
Allure
teenagers
400
173
29
48
34
Emotion (Pos)
CasualConversation
400
211
29
Prosocial
Emotion (Neg)
Table 1: Data Statistics of TalkUp and breakdown of five subreddits in the data. #E: number of empowering examples, #D: number of disempowering examples, #A: number of ambiguous exmaples, %W: percentage of women posters in the data.
- & You
Exclamation Mark
11
Power
Focus Present
Gendered Words
Big Words
Focus Past
data to be released at different times of day over multiple days. After each batch was completed, we manually quality-checked the responses and computed each annotator's standard deviation. We discarded data from unreliable annotators, includ- ing those who straightlined through many annota- tions with the same answer, those who clearly had not read instructions, and those whose alignment scores were more than standard deviations from the mean. Annotator alignment scores were calcu- lated by dividing the number of disagreements by the number of agreements between their label and the majority vote. We subsequently released new batches to re-label data previously annotated by the identified unreliable annotators.
Words Per Sentence
Tone (Neg)
We & They
-0.05
0.00
0.05
0.10
Figure 2: Weights of LIWC features with 90% and 95% confidence intervals assigned by linear regression model trained with TalkUp. All features except for Negative Emotion, Power, Focus Present have statistically signifi- cant weights (p < 0.1).
reason marked, indicating that there was no reason category annotators agreed on.
The inter-annotator agreement, Krippendorff's alpha, was 0.457, and the percentage agreement was 65.2%. These agreement scores are reason- able given the complexity and nuance of this task we would neither expect nor want to have per- feet annotator agreement because it is inherently ambiguous problem even for humans, and there often no objective "ground truth" on whether a text empowering or not. Our agreement scores are comparable to those of other computational social science papers on tasks of similar nature, especially when concerning pragmatics. For exam- ple, our percentage agreement is higher than that of ElSherief et al. (2021)'s dataset latent ha- tred, and our Fleiss's 's kappa is similar that of the Microaggression dataset (Breitfeller al., 2019b).
Data Statistics We combined the maybe empow- ering with the empowering label, and did the same for the disempowering labels. We then used major- ity voting aggregate the three annotations into the final labels for empowerment, ambiguity, and stance for each post. When all three annotators disagreed on the empowerment label (i.e., one vote each for empowering, neutral, and disempowering), we marked it as No Consensus and considered an ambiguous case. For reason labels, where an- notators can mark more than one categories per example, we only kept the reason labels that were marked by at least annotators.
Table I shows the overall size of our dataset and the distribution of labels, the number of am- biguous cases, and percentage of posts made by women across the entire dataset and also by dif- ferent subreddits. We annotated 400 posts from 5 different subreddits resulting in total of 2000 samples. Of these, 962 were labeled as empower- ing, 129 as disempowering, and 267 ambiguous, with 642 being labeled as neutral. We note that 265 out the 962 empowering cases had no final
4 Data Analysis
We present preliminary analyses of TalkUp. Em- powerment is nuanced phenomenon in pragmatics and deeper exploration of social and linguistic vari- ables remains open for future work. The analyses we present here provide some initial, surface-level insights into what makes language empowering.

---

4.1 Characteristics of Empowering Language
the lines like the tone of the message.
Singular vs. Plural Pronouns. Interestingly, empowerment and disempowerment tend to use different types of pronouns. Singular promouns (I, you) are positively associated with empowering lan- guage, while plural pronouns (we, they) are linked to disempowering language. Our manual inspec- tions suggest one possible explanation: people who write empowering posts tend to speak directly to the listener, and also include elements of their own personal experience, hence the prevalence of you and / pronouns. Disempowering conversations are less personal and individualized, often making gen- eralized assumptions or judgments about people.
We use the LIWC-22 software to compute LIWC features for all annotated posts (Boyd et al., 2022). These features measure the percentage of word overlap between the text and predefined lexicons that capture different social and psychological char- acteristics of language, such as prosocial words or words associated with positive tone. For a more concise and generalized analysis, some related fea- tures are combined into compound features: the and You features are grouped into one feature I+ You, We and They into We+They and male and female into gendered words. We standardize LIWC feature scores using the mean and variance cal- culated from TalkUp's randomly sampled posts. Model-surfaced posts are excluded as they may not reflect the distribution of Reddit posts in the wild.
4.2 Empowering Language by Gender
As preliminary analysis of empowerment across one social dimension, we explore the differences in empowering posts written by men and women. First, we standardize the LIWC feature values for men and women's 's empowering language over the entire dataset. We find that women's 's empower- ing language displays significantly higher levels of positive tone and positive emotions than men. Women also more exclamation points, while men use more swear words. These findings align with prior works in sociolinguistics that have as- sociated exclamation points with higher expres- siveness and excitability (Bamman et al., 2014; Waseleski, 2017; Güvendir, 2015), which is usually more socially acceptable for women. Mcanwhile, men's 's use of strong or offensive language is linked with masculinity or aggressiveness, and is less cially accepted in women. Additionally, there are other features where women and men's empower- ing posts diverge women use more present tense than men, and men are much less likely to use gendered words.
To understand how each of these features con- tributes to empowerment in language, we train a lin- ear regression model to predict the likelihood of a post being empowering. Figure 2 shows the regres- sion coefficients assigned to each feature. Looking at the positive coefficients reveals that empower- ment is associated with lexical features like clout, allure, prosocial words, and exclamation marks. Meanwhile, disempowerment is associated with features that have negative coefficients, such as big words and words-per-sentencs which may indicate sentence complexity. We expand on a few of the most notable findings below.
Tone vs. Emotion. We find that the tone of lan- guage is more influential empowerment than the emotion conveyed. Positive tone has a significantly higher coefficient than positive emotion; likewise, negative tone is highly associated with disempow- erment, while negative emotion is not statistically significant. This suggests that the concept of em- powerment is distinet from sentiment and cannot be captured by sentiment analysis models alone.
We then control for gender, comparing men's empowering language with all men's posts, and likewise for women. The results show that positive tone, positive emotions, and exclamation marks remain strongly correlated with empowering lan- guage even after accounting for gender. However, considering gender does impact the degree of posi- tivity and the use of exclamation marks. Men's em- powering language, when compared to men's aver- age language, displays a greater increase in positive tone, positive emotions, and the use of exclamation marks compared to women's 's empowering language in relation to their average language. This suggests
Power. Power is not a statistically significant feature in predicting empowerment. This corrob- orates the idea that empowerment is not the same power empowerment is more nuanced and subtle concept that extends beyond power-related lexicons, relying more on the implications between
"We combined I/you and we/they because they consistently followed the same patterns. This was also motivated by quali- tative analysis: Uyou occurred frequently in language that was addressed directly to the other conversant, which was common in empowering posts, whereas disempowering posts often dis- missed a group vaguely without addressing the conversant as an individual, leading to greater use of we/they.

---

RoBERTa
GPT-3
Askleddit
relationships
Input Type
F1
Acc
F1
Acc
Fitness
of posts
0.3-
teenagers
Post
63.5
77.7
36.9
59.7
CasualConversation
+response
66.1
78.3
31.5
52.1
0.2
+context
65.5
77.9
37.5
64.2
+all
67.1
78.4
38.2
67.1
0.1
Table 2: Model performance of RoBERTa-based classi- fier fine-tuned on TalkUp and GPT-3 without fine-tuning. RoBERTA is significantly better than RoBERTA- Post terms Fl (p<0.1).
emotion
posttive
grow changes
skill decision
options assertive
alone
true competency critically
others difference rights
Figure 3: Distribution of empowering reasons. One post can have more than empowering reason.
4.4 Empowerment and Poster-Commenter Alignment
that men tend to exhibit a more pronounced shift towards positive and expressive language when ex- pressing empowerment, whereas women's 's empow- ering language already aligns closely with their overall language patterns.
While commenter take either an agree, neu- tral, disagree stance with the poster, most em- powering posts were in conversations where the poster and commenter agreed (79.6%). Like- wise, most disempowering posts occurred when the poster and commenter disagreed (45.5%). Intu- itively, this makes sense for the majority of cases people often respond agreeably to empowerment and negatively to disempowerment.
Our findings highlight the complex interplay be- tween language, gender, and empowerment, moti- vating future research into the influence of social factors communication of empowerment. More detailed analyses on empowerment differences by gender and subreddit can be found in Appendix A.
Importantly, however, this not always the case: empowering posts can sometimes have commenters who disagree, and disempowering posts can have commenters who agree. These cases often involve more complex pragmatics. Empowering posts that contain toxic positivity are frequently met with dis- agreement, and sometimes commenters will reject minimize empowering compliments for the sake of politeness. Empowerment can also be met with antagonism from an ill inliineeniooed commenter, re- gardless of how genuine the original post may be. Disempowering posts that disparage a particular group might receive an agreeing comment from someone who also shares that view of the group. We elaborate on these conversational patterns in Appendix A.3. Overall, the empowering -issgree and disempowering aggree cases provide rich cor- pus for studying implicature and interactions social contexts.
4.3 Reasons Why Posts Are Empowering
Figure illustrates the distribution of reasons se- lected by at least two annotators for why a post was empowerin//dismpooeringg broken down by subreddit. The most common reasons a post was considered empowering are encouraging expres- sion of emotions (40.6%), supporting the reader's self-image (26.8%), and supporting the reader's ability grow (21.1%) and change (18.8%).
Notably, there are significant differences in the reasons most commonly used in different subred- dits. For example, the teenagers and relationships subreddits tend empower users by promoting expression of emotions, while empowerment in Finess was more focused on encouraging people to improve themselves and make changes. The unique distributions of reasons among different communities and topics of discussion suggests that empowerment serves diverse purposes and implies different meanings depending the context. Fu- ture work could explore which techniques should be used to empower people in specific contexts, such as empowering clients clinical psychology or students educational settings, based on the desired interaction goals.
4.5 Modeling Empowering Language
To explore how well empowerment can be cap- tured by computational methods, we present em- powerment detection experiments with two large language models: fine-tuned RoBERTa and zero- shot GPT-3.7 We note that goal here is not
"We opt to experiment with zero-shot rather than few- shot settings because fine-tuning a model of GPT-3's seale

---

build state-of-the-art model, but to give a general picture of how well existing models work and to illustrate the usefulness of our dataset.
4.6 Ambiguity of Empowering Language
TalkUp contains 228 samples that either were la- beled as "ambiguous" by at least two annotators, or were labeled "no consensus" because all three annotators marked different answers for the em- powerment question. We qualitatively analyzed this subset of TalkUp, and we find that these am- biguous posts are not "bad data," " but rather are linguistically interesting because they are ambigu- ous they are examples of language that could reasonably be interpreted in several different ways.
Fine-tuned RoBERTa. We assess how well empowerment can be identified by a pre-trained RoBERTa model (Liu et al., 2019) fine-tuned on TalkUp, and we conduct an ablation study to ex- amine the importance of contextual information in helping the model classify post as empowering, disempowering, or neutral. We test four model variants: post, +response (post and response), +context (post, posters' gender, subreddit), +all (post, response, context). We divide 1733 unam- biguous samples from TalkUp into 60:20:20 for train.validation:test sets and select the model with best validation macro-fl score.
For example, the post "Maybe call relative friend who has car? Youll figure it out. wish you luck, kid." was unanimously labelled as "empower- ing" and "ambiguous" by annotators. This makes sense the post overall seems to provide a helpful suggestion, but calling the responder "kid" could be interpreted in different ways (e.g. as an endearing nickname vs. a condescending title) depending on the social relationship between the poster and the responder. Notably, many of the posts with inher- ent ambiguity display sarcasm, such the posts "i love you too?!" and "thats grimy as f*ck but sure you do that." Sarcasm, by design, disguises a negative message in positive words, and so a sarcas- tic post could be interpreted either way depending on whether the sarcasm was meant positively or negatively.
Table 2 presents the average macro-fl scores across 10 separate runs using different random seeds on the test set. The results show that ad- ditional context improves model performance. Zero-Shot GPT-3. Additionally, we evaluate GPT-3 Davinci's (Brown et al., 2020) ability to detect empowerment using prompts. We design seven different prompts for each of the four com- binations of post+context, and generate responses. While most of GPT-3's responses are single word (e.g. ""mpowwrring", some are longer. To map GPT-3's responses to empowerment labels, we use a simple lexical counting method: if the generated text contains more empowering-rclane related words (c.g. empowering, empowered, empower) than words related to other labels, it is classified as empower- ing. GPT-3's final classification for each post takes the majority vote over its responses to the seven prompts. A full list of our GPT-3 prompts can be found in Appendix C.2.
We also investigated how GPT-3 handles such ambiguous cases. We find that GPT-3 tends classify them as neutral, even for explicitly em- powering posts such the above example. Instances in which the posts carried a sarcastic tone were commonly interpreted by GPT-3 as neutral as well, indicating that simultancously empowering and ambiguous language is poorly understood by the model. The fact that ambiguity is still challeng- ing for large models motivates the need for further work in this area, and TalkUp provides diverse ex- amples of ambiguous language that can be used to work towards this end.
Our results indicate that GPT-3 performs poorly in zero-shot settings compared to RoBERTa-basee based classifiers fine-tuned on TalkUp. This reveals that even large language models cannot effectively cap- ture empowering language, highlighting the impor- tance of having carefully annotated dataset of nuanced examples like TalkUp.
5 Example Application: Unearthing Empowerment Patterns on Reddit
is impractical for most users, and because our preliminary experiments indicated that few-shot prompts resulted lower performance than zero-shot. Although in-context examples often improve performance, there cases in which few-shot underperforms zero-shu shot due to models becoming excessively fixated the provided examples and struggling generalize effectively. This phenomenon is documented numerous previous studies (e.g. Fei al., 2023). and we consistently observed this in our case.
As a case study, we demonstrate how TalkUp and the trained empowerment classifier can be used to uncover interesting patterns in how people use empowering language. Specifically, we apply the trained classifier in $4.5 to generate empowerment labels of all Reddit posts and responses in RiGen- der, to learn about how both posters and responders
"Specitic training details and found in Appendix B.3.
can he

---

Empower
Disempower
more empowering content. Unsurprisingly, the re- sults show that of all the posts predicted to be em- powering, women wrote a considerably higher per- centage of them than men. Interestingly, however, women are also responsible for a slightly higher percentage of disempowering posts than men. An- other surprising finding is that posts written by men that were commented on by women tend to be more empowering or more disempowering than those commented on by men, suggesting that women not only post more empowerment but they also engage with more empowerment- charged posts. This may be tied to factors like the topics or types of posts that women tend to en- gage with and could be used to answer sociological questions about gender and social media.
Post
Response
Post
Response
r/AskReddit
12.0
14.1
6.8
5.6
z/relaioonhhp
38.7
27.2
12.7
11.4
s/Finess
30.0
28.3
7.2
5.6
tttcnagger
24.2
24.8
6.3
5.7
CasualConversation
25.6
29.2
2.8
2.3
Overall
15.2
16.5
6.9
5.8
Table The percentage of empowering and disempow- ering posts and responses in each subreddit.
Post
Response
language,
Poster
Responder
%B
%D
%E
%D
Man
13.4
6.5
13.8
5.9
Man
Woman
16.2
7.1
18.1
6.0
Man
16.5
6.9
16.7
6.3
Woman
Woman
20.2
7.3
20.4
6.4
Table 4: The percentage of empowering (%E) and dis- empowering (%D) posts and responses in RIGender classified by the model trained with TalkUp, broken down by the gender of both the poster and responder.
6 Related Work
To knowledge, Mayfield al. (2013) is the only prior work exploring empowerment in NLP, but the contributions of our works are quite differ- ent. Mayfield et al. (2013) primarily focus on an algorithm for predicting rare classes and use em- powerment as an example. In contrast, we focus on understanding empowering language itself, before developing automated detection tools. We explore the reasons behind empowerment, considering mul- tiple dimensions of social context such gender, topic, and poster-commenter alignment. Mayfield et al. (2013) use non-public data from a specific cancer support group, while TalkUp spans diverse topics and user bases, making our scope broader and more generalizable.
communicate." We analyze empowering and dis- empowering posts in different subreddits and by different gender of the poster and responder.
By Subreddit Table shows the percentage of empowering and disempowering posts and re- sponses in thefive subreddits of TalkUp. The results indicate that the subreddits have significantly differ- ent degrees of empowerment, and that and certain subreddits (e.g. relationship, Fitness) are signifi- cantly more empowering than others (c.g. AskRed- dit). Our model can be used to monitor the overall empowerment level of communities and identify unusual patterns, such significant rise in dis- empowerment. Furthermore, we find that there are more empowering responses than posts in total. On the contrary, there are more disempowering posts than responses across all subreddits. This may be because responses are often directed towards spe- cific posts or users, and as result, the writer may be more conscious of their tone and try to be more empowering compared to posts.
As empowering language is not well understood in NLP, our work has also drawn insights from research on related concepts:
Power. Danescu-NiculsscuMMizl et al. (2011) develop a framework for analyzing power differ- ences in social interactions based on how much one conversant echoes the linguistic style of the other. Prabhakaran and Rambow (2014a.b) pre- dict power levels of participants in written dialogue from the Enron email corpus, and several other of their works explore power dynamics in other con- texts, such as gender (Prabhakaran et al., 201 4b) and political debates (Prabhakaran et al., 2014a)
By poster and responder gender Table 4 shows the percentage of empowering and disempowering context by the gender of posters and responders. Overall, women seem to post and interact with
Our work studies empowerment rather than power Power certainly closely related con- cept, but empowerment nomenon it concerns not just static power levels, but interactions that increase or decrease a person's s
"Given that responses are only available for the posts and not for the responses, and that some samples in the data do not provide the gender of the responder, we used a model that only incorporates subreddit information as additional context to the text itself.
distinct linguistic phe-

---

power, and it is also a broader concept that encom- passes things like self-fulfilmment and self-esteem. While power has primarily been analyzed at the word level, such as by examining connotations of particular verbs (Sap et al., 2017; Park et al., 2021), our work attempls to look at higher-level pragmat- ics - implications that may not be captured by word choice alone, but suggested between the lines. Condescension. The closest concept to empow- crment that has been more thoroughly studied in NLP is condescension. Prior works have defined condescension as language that is not overtly nega- tive, but that assumes status difference between the speaker and listener that the listener disagrees with (Huckin, 2002). Intuitively, condescension can be interpreted as roughly the opposite of em- powerment: it implicitly suggests that the listener has lower status or worth.
ing, which has potential applications in real-world dialogue settings like education and psychotherapy. TalkUp focuses on simple two-turn interactions with social variables (gender, alignment, and topic), but its framework can extend to more com- plex social interactions. For example, there are many other social roles that can influence power dynamics, including occupation (e.g. manager employee), race (e.g. White vs Person of Color), and age (e.g. old vs. young person). Different com- binations of these identities can result in further intersectional dynamics (Crenshaw, 1990; Collins and Bilge, 2020; Lalor et al., 2022). Additionally, since most real-world conversations are long back- and-forth exchanges, we encourage future work to explore empowerment in multi-turn dialogues.
Conclusion
Our work particularly builds upon Wang and Potts (2019): they develop Towk dataset of "not
We explore the problem of empowerment detec- tion, grounding it in relevant social psychology and línguistics literature. To facilitate studies of empowerment, we create TalkUp, a high-quality dataset of Reddit posts labeled for empowerment and other contextual information. Our preliminary analyses demonstrate that empowerment is not cap- tured by existing NLP methods and models, but that it can be detected with our dataset. Furthermore, demonstrate the importance of social context in understanding empowering language with dif- ferent genders, poster-commenter alignments, and topics discussion. In studying empowerment, we work towards bigger open challenges in pragmatics, implicature, and social context in NLP.
Reddit posts labeled "condescending" condescending." Specifically, they identify conde- scending posts by looking for replies that indicate the original post is condescending. Our approach is parallel to this work: we likewise surface Reddit posts whose responses indicate that the original post is empowering (thus aligning with our defi- nition of empowerment in $2 as effect the listener). TalkUp complements TalkDown by focus- ing on the positive aspect of such language: instead of only identifying text as condescending or not condescending, we distinguish between disempow- ering, empower, and neutral posts.
7 Future Directions
Ethics Statement
In this work, we focus only empowerment clas- sification and detection, with our primary contribu- tion being the proposal of new dataset to facilitate research in new area of computational sociolin- guistics. However, TalkUp not only can be used to detect empowerment, but also to generate more empowering language. As in Sharma et al. (2021b), we believe a classifier trained with our data can be used to assign rewards that tailor generation model to produce more empowering outputs. An empowerment classifier also be used for con- trollable text generation with constrained decoding. in Yang and Klein (2021), Liu al. (2021), and Kumar et al. (2021). Additionally, a model that can control for empowerment could be used to suggest edits to make Human--ritten text more empower-
In constructing our study, we took precautions to ensure the task design, data collection and handling done ethically and according to current recom- mended practices and guidelines (Townsend and Wallace, 2016; Mislove and Wilson, 2018; Gebru et al., 2018; Bender and Friedman, 2018). Specifi- cally, we ensured fair compensation by calculating the pay based on minimum wage in CA (higher than then the average pay worldwide, including most U.S. states). To avoid exposing the annota- tors to potentially offensive or otherwise harmful content from social media, we manually checked every data sample. Beyond scientific goal of our work understand sociolinguistic characteristics of empowering language and open new directions to NLP research on deeper pragmatic phenomena,

---

the practical goal is advance NLP technologies with positive impact through understanding and in- corporating empowerment practical applications including education, therapy, medicine, and more.
References
David Bamman, Jacob Eisenstein, and Tyler Schnoebe- len. 2014. Gender identity and lexical variation in social media. Journal of Sociolinguistics, 18(2):135- 160.
Limitations
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604
We identify three primary limitations of our work. First, to protect the anonymity of annotators, we did not explicitly control for annotator demographics. It is thus possible that our annotator demographics is imbalanced which can impact annotation deci- sions and potentially incorporate biases in NLP models built on the dataset (Geva et al., 2019).
Ryan L Boyd, Ashwini Ashokkumar, Sarah Seraj, and
James W Pennebaker. 2022. The development and psychometric properties of liwe-22. Austin, TX: Uni- versity of Texas at Austin.
Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019a. Finding microaggressions in the wild: A case for locating elusive phenomena cial media posts. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP,, pages 1664-1674, Hong Kong, China, Association for Computational Linguistics.
Second, with the goal to incorporate social con- text, we relied on gender annotations from RtGen- der, the corpus we draw from to annotate empow- cring conversations. Thus, TalkUp only centers on binary gender identities and is limited by the scarcity of data on nonbinary identities in the Ri- Gender dataset. Building resources and methods inclusive to queer identities an important area for future work. Additionally, RtGender's gender la- bels were constructed by finding users who posted with gender-indication flair, which means that Rt- Gender only contains posts from subset of users who voluntarily disclosed their gender; this may si- Ience the voices of users who are less likely share their gender, including nonbinary users. Further, future work on empowerment should incorporate broader social contexts, c.g. relationships involv- ing inherent power hierarchies (Prabhakaran and Rambow, 2014a), more dimensions of identity like race (Field et al., 2021), and others.
Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019b. Finding microaggressions in the wild: A case for locating clusive phenomena in so- cial media posts. In Proceedings of the 2019 Confer- ence Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 1664-1674, Hong Kong, China. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan. Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Tlya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. ArXiv, abs/2005.14165.
Finally, TalkUp is limited to the Reddit domain and only includes English posts. This data may not be generalizable to other domains, such clinical psychology or education.
Acknowledgements
Logan S. Casey, Jesse Chandler, Adam Seth Levine, Andrew Proctor, and Dara Z. Strolovitch. 2017. Intertemporal differences among mturk work- ers: Time-based sample variations and implica- tions for online data collection. SAGE Open, 712:2115244017712774,
We gratefully acknowledge support from NSF CAREER Grant No. ISS244279,, the Alfred P. Sloan Foundation Fellowship, and NSF grants No. IS2125201 and IIS2203097. Any opinions, findings and conclusions or recommendations ex- pressed in this material are those of the authors and do not necessarily state or reflect those of the funding agencies.
Judi Chamberlin. 1997. A working definition of empow- erment. Psychiatric Rehabilitation Journal, 20(4):33- 46.
Patricia Hill Collins and Sirma Bilge. 2020. Intersec- lionality. John Wiley & Sons.

---

Kimberle Crenshaw. 1990. Mapping the margins: In- tersectionality, identity politics, and violence against women of color. Ston. I. Rev. 43:1241.
Thomas Huckin. 2002. Critical discourse analysis and the discourse of condescension. Discourse studies in composition, pages 155-176.
Cristian Danescu-NiculesuuMiin Bo Pang, and Jon Kleinberg. 2011. Echoes of power: Language effects and power differences in social interaction.
Lillian Lee,
Shu hui Eileen Chen. 2003. Compliment response strategies in mandarin chinese: Politeness phe- nomenon revisited, Concentric: Studies in Linguis- rics, 19:157-184.
Mai ElSherief, Caleb Ziems, David Muchlinski, Vaish- navi Anupindi, Jordyn Seybolt, Munmun De Choud- hury, and Diyi Yang. 2021. Latent hatred: A bench- mark for understanding implicit hate speech. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 345-363, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguisties.
Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yulia Tsvetkov. 2021. Controlled text generation con- tinuous optimization with multiple constraints. In Advances in Neural Information Processing Systems, volume 34, pages 14542-14554, Curran Associates, Inc.
John Lalor, Yi Yang. Kendall Smith, Nicole Forsgren, and Ahmed Abbasi. 2022. Benchmarking intersec- tional biases in NLP. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3598-3609, Seattle, United States. Association for Computational Lin- guistics.
Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. Mitigating label biases for in-context learning. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 14014-14031, Toronto, Canada. Association for Computational Linguistics.
Anjalie Field, Su Lin Blodgett, Zeerak Waseem, and Yulia Tsvetkov. 2021. A survey of race, racism, and anti-racism in NLP. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1905-1925, Online. Association for Computational Linguistics.
Alisa Liu, Maarten Sap. Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021. DExperts: Decoding time con- trolled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Pupers), pages 6691-6706 Online. Association for Computational Linguistics.
Anjalie Field and Yulia Tsvetkov. 2020. Unsupervised discovery of implicit gender bias. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 596- 608.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:
Ge Gao, Sun Young Hwang. Gabriel Culbertson, Su- san R. Fussell, and Malte F. Jung. 2017. Beyond information content: The effects of culture affec- tive grounding in instant messaging conversations. Proc. ACM Hum.-Comput. Comput. Interact., ICCSCW).
Saskia Locke, Anthony Bashall, Sarah Al-Adely, John Moore, Anthony Wilson, and Gareth B. Kitchen. 2021. Natural language processing in medicine: A review. Trends in Angesthesia and Critical Care, 38:4-9.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets for datasets. In Proceedings of the conference on fair- ness, accountability, and transparency, pages 220- 229.
Elijah Mayfield, David Adamson, and Carolyn Pen- stein Rosé. 2013. Recognizing rare social phenom- ena in conversation: Empowerment detection in sup- port group chatrooms. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 104-113, Sofia, Bulgaria. Association for Computational Lin- guistics.
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an inves- tigation of annotator bias in natural language under- standing datasets. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 1 1161-1166, Hong Kong, China. Association for Computational Linguistics.
Ellen Hawley McWhirter. 1991. Empowerment in counseling. Journal of Counseling & Development, 69(3)222-227..
Alan Mislove and Christo Wilson. 2018. A practi- tioner's guide to ethical web data collection. In Brooke Foucault Welles and Sandra González-Bailón. editors, The Oxford Handbook of Networked Commu- nication. Oxford University Press.
Emre Güvendir. 2015. Why are males inclined to use strong swear words more than females? an evolution- ary explanation based on male intergroup aggressive- ness. Language Sciences, 50: 133-199.

---

György Moinár and Zoltán Szüts. 2018. The role of chatbots in formal education. In 2018 IEEE 16th International Symposium on Intelligent Systems and Informatics (SISY), pages 000197-000202.
In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2329-2334 Copenhagen, Denmark. Association for Computational Linguistics.
Stephen P. Osborne. 1994. The language of empower- ment. International Journal of Public Sector Man- agement, 7(3):56-6. 62.
Skipper Seabold and Josef Perktold. 2010, statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference.
Chan Young Park, Xinru Yan, Anjalie Field, and Yulia Tsvetkov. 2021. Multilingual contextual affective analysis of Igbt people portrayals in wikipedia. In Proceedings of the International AAAT Conference on Web and Social Media, volume 15, pages 479-490.
Ashish Sharma, Inna W. Lin, Adam S. Miner, David C. Atkins, and Tim Althoff. 2021a. Towards facilitat- ing empathic conversations in online mental health support: A reinforcement learning approach. CoRR, 0os/2101.07714
Vinodkumar Prabhakaran, Ashima Arora, and Owen Rambow. 2014a. Power of confidence: How poll scores impact topic dynamics in political debates. In Proceedings of the Joint Workshop on Social Dynam- ics and Personal Attributes in Social Media, pages 77-82, Baltimore, Maryland. Association for Com- putational Linguistics.
Ashish Sharma, Inna Lin, Adam S. Miner, David C. Atkins, and Tim Althoff. 202 Towards facilitat- ing empathic conversations in online mental health support: A reinforcement learning approach. CoRR, abs/2101.07714.
Ashish Sharma, Kevin Rushton, Inna Wanyin Lin, David Wadden, Khendra G. Lucas, Adam S. Miner, Theresa Nguyen, and Tim Althoff. 2023. Cogni- tive reframing of negative thoughts through human- language model interaction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Vinodkumar Prabhakaran and Owen Rambow. 2014a. Predicting power relations between participants in written dialog from a single thread. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pages 339-344, Baltimore, Maryland. Association for Computational Linguistics.
Leanne Townsend and Claire Wallace. 2016. Social media research: A guide to ethics. University of Aberdeen, Aberdeen.
Vinodkumar Prabhakaran and Owen Rambow. 2014b. Predicting power relations between participants in written dialog from a single thread. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pages 339-344, Baltimore, Maryland. Association for Computational Linguistics.
Ishan Sanjeev Upadhyay, KV Aditya Srivatsa, and Rad- hika Mamidi. 2022. Towards toxic positivity de- tection. In Proceedings of the Tenth International Workshop Natural Language Processing for Social Media, pages 75-82 82, Seattle, Washington, Associa- tion for Computational Linguistics.
Vinodkumar Prahhakaran, Owen Rambow, and Mona Diab. 2012. Predicting overt display of power in writ- ten dialogs. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 518-522 522, Montréal, Canada. As- sociation for Computational Linguistics.
Rob Voigt. David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. 2018. RiGender: A corpus for studying differential responses to gen- der. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Vinodkumar Prabhakaran, Emily Reid, and Owen Rambow. 2014b. Gender and power: How gen- der and gender environment affect manifestations of power. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1965-1976, Doha, Qatar. Associa- tion for Computational Linguistics.
Zijian Wang and Christopher Potts. 2019, Talkdown: A corpus for condescension detection in context. CORR. abs/1909.
Carol Waseleski. 2017. Gender and the Use of Excla- mation Points in Computer-Mediated Communica- tion: An Analysis of Exclamations Posted to Two Electronic Discussion Lists. Journal of Computer- Mediated Communication, 1(4):1012-1024.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Juraf- sky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power im- plications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477-5490 Online. Association for Computational I. Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen. Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-ane natural language processing.
Maarten Sap, Marcella Cindy Prasettio, Ari Holtzman, Hannah Rashkin, and Yejin Choi. 2017. Connota- tion frames of power and agency in modern films.

---

Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Kevin Yang and Dan Klein. 2021. FUDGE: Controlled text generation with future discriminators. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3511-3535, Online. Association for Computational Linguistics.
Caleb Ziems, Minzhi Li, Anthony Zhang, and Diyi Yang. 2022. Inducing positive perspectives with text reframing. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Pupers), pages 3682-3700, Dublin, Ireland. Association for Computational Linguistics.

---

A Empowering Language by Group
A.1 Empowering Language by Gender
tearagery
tlag
Figure 6: Average of standardized LIWC score of em- powering language by subreddit. The error bar indicates the 90% confidence interval.
Fertunt
Figure 4: Average of standardized LIWC score of em- powering language by men and women. The error bar indicates the 90% confidence interval.
stance
Figure illustrates the average standardized scores of empowering language by men and women.
Sow
Figure 7: Average of standardized LIWC score of em- powering language by stance of responder to the poster. The error bar indicates the 90% confidence interval.
Toxic positivity a phenomenon where a posi- tive attitude actually minimizes someone's experi- ence, and is an open area of research (Upadhyay et al., 2022). A post with lot of encouragement or affirmations could come across dismissive invalidating of the recipient's struggles.
Figure 5: Average of standardized LIWC score of em- powering language by men and women standardized by average of all men and women's post, respectively. The eтror bar indicates the 90% confidence interval.
(2) Commenters may disagree with an empower- ing post an effort to be polite or humble rather than accepting the compliment. For example, one poster wrote, "That's cool!." and a commenter replied with "haha it's not cool as it sounds." It is unlikely that the commenter actually thinks the topic of discussion is not that great; rather, jecting compliments is a well-documented form of politeness that is most common in high-context lan- guages (hui Eileen Chen, 2003; Gao et al., 2017). Reading between the lines to pick up on impli- cations like this is an open area of research that involves cultural norms and values.
Figure illustrates the average standardized scores of empowering language by men and women after controlling for gender. In other words, we comparing men's 's empowering language with all men's 's posts, likewise for women.
Empowering Language by Subreddit A.3 Empowering and Disempowering Language and Commer-Commenter Stance
A.2
Empovering-Disagree Some posts labeled as empowering had commenters who disagreed with the poster. Figure 7 shows some notable features of these posts. Through qualitative analysis of em- powering-disagree posts, we observe few conver- sation patterns:
(3) Some empowering posts are met with antag- onism from the commenter -cttnnnlleaaativllyy attacking the poster with insults like "dummy" or "f*ck off" with- out really engaging in conversation. This suggests that whether or not text is perceived as empowering depends partially on the attitude and intentions
(1) Posts with toxic positivity, whether inten- tional or not, are often met with disagreement.

---

such posts because empowerment has less to do with the literal words spoken and more do with the way text impacts the feelings of the recipient, resulting in a label of "disempowering" even if the commenter is supportive of the poster.
A.4 Ambiguous and Unambiguous Language
agren
Fax
The
Figure& 8: Average of standardized LIWC score of disem- powering language by stance of responder to the poster. The error bar indicates the 90% confidence interval.
the recipient. No matter how genuine an empow- ering post may be, a reader may still reject for other contextual reasons, such as being unwilling to receive feedback or simply disliking the poster. DisempowerrngtAgrre. Additionally, some dis- empowering posts had commenters who agreed with the poster. Figure shows notable features of these posts, and we again inspect them qualitatively to identify two main patterns:
Amarca
Bortecy
Figure 9: Average of standardized LIWC score of sam- ples that are ambiguous and unambiguous in their em- powerment. The error bar indicates the 90% confidence interval.
B Implementation Details
(1) Some posts labeled as disempowering would particular audience (e.g. a post that makes fun of the eating habits of vegan people would likely be received negatively by a vegan person), but the particular commenter who responded happened to share their view and joined the poster in making fun of the other group together. This is manifested in the prevalence of the We+They feature such posts include many "we" and "they" pronouns because they involve the poster and commenter taking the same side and making fun of some other group.
certainly be disparaging
B.1 Empowerment Regression Model for Sample Selection
We trained RoBERTa-based regression model, using the ROBERTA-BASH model on Huggingface transformers library (Wolf et al., 2020), to rank the Reddit posts to surface more likely empowering examples the data for annotation. We used the data we collected from pilot studies to train the first model and continually updated the model as we collect more data from AMT, resulting in a total of 9 updates. The data was split into train and test an 8:2 ratio. In order to have float values predict for the model, we mapped disempowering. neutral, empowering labels O, 0.5, respectively. We only used the text of the post as an input to the model and we set maximum input length to 512. The batch size was fixed to 8. In every update, the hyper-paramenn were tuned through a grid search (gradient accumulation count: {1,2,4), warm-up ratio: (0.05, 0.1, 0.2), learning rate: (1e-5, 1e-4, 5c-4)).
(2) Other posts labeled as disempowering were instances where the poster was sharing very heavy personal stories, and the commenter was validat- ing their experience. This is exhibited particularly in the emotion and tone features: the emotion ex- pressed in these posts very negative because the topics themselves are heavy, but the tone is not negative because it is not negativity directed at the other person in the conversation. We note that some of these personal stories could be interpreted neutral posts under our label definitions (i.e. the post only talks about the poster and is not relevant the commenter), but these posts do not quite fall under this category because they were still direct conversations with the commenter. commenter or an annotator labeling the conversation after the fact - may feel disempowered by the contents of
B.2 Linear Regression Experiments
We used the statsmodels package (Seabold and Perktold, 2010) to fit a ordinary least squares linear model with intercept. Same as the RoBERTa-based empowerment regression model, we mapped em- powerment labels to float values, and only used

---

1733 samples in Tp marked as unambiguous by annotators. The R² of the fitted model with features in Figure 2 was 0.29.
notation task. After each pilot, computed annotator agreement and manually walked through every example that annotators disagreed on in order to clarify confusing aspects of our definitions. We summarize the key findings of these initial pilot studies.
B.3 Empowerment Classifier Fine-tuning
We used a ROBERTA-BASE checkpoint on Hugging- face, which has 123 million parameters, to fine- tune to train empowerment classifier discussed in $4.5. The data was split into train, development, and test sets in a 60:20:20 ratio and stratified by sub- reddit. We set the batch size 32 and maximum input length 300 even though the longest input our data was shorter. All other sapee-paraeeters was set to the default values provided by the Trainer and BrminingArgumente class in the transformers library. We trained each model for 10 epochs and selected the model with the best Fl score the development set the final model for evaluation. The model trained one A6000 GPU and took about 15 minutes. We ran training with 10 different random seeds and averaged the test set performance for each model.
Context is useful for judging post. Annotator confidence
higher when provided not just the text of the post, but additional contextual infor- mation like the poster's gender and the subreddit. Additionally, including the responder's comment helped to provide useful context by revealing how a real reader reacted to the post. Our final annotation task incorporated this contextual information.
Annotators' free-response descriptions of so- cial roles lack consistency. Early iterations of our pilot studies asked annotators to specify what cial group would be empowered disempowered by a post. Answers varied dramatically - from gen- eral groups like "Democrats" to extremely specific descriptions like "a person who likes soccer and supports this sports team" - and were difficult to or- ganize in any consistent way. However, our manual inspections of data samples revealed that most fell into two categories: (1) conversations where the poster and commenter agree/share the same stance (such as being members of the same political party or supporting the same sports team), and (2) conver- sations where they disagree/have opposing stances. This generalization of social relationships, while quite broad, allowed us to capture the diversity of possible social roles, and we used this stance agree- ment/disagreement question in the final annotation task.
Model Evaluation Details
C.1 RoBERTa Input Type Examples
From preliminary experiments, we noticed that depending on how you format the additional in- put (e.g. response, subreddit, poster's s gender) to RoBERTa, the performance varies. We used the in- put type with the best performance for each model in Section 4.5 and provide results of all templates we tried in Table 5.
C.2 GPT-3 Prompts
As with all prompt-based language models, there is no straightforward way to determine the optimal prompt for a task, and the performance of GPT-3 can vary depending on the design of the prompt. To increase the robustness of the evaluation, we cre- ated seven templates for each model type and used the majority voting the final output from GPT-3. We provide all templates and their corresponding performance in Table 6. While the performance of GPT-3 is not as high as the fine-tuned classifiers, practitioners can refer to this performance by tem- plate as reference when using GPT-3 to probe empowerment in language.
Models can help to surface potentially empow- ering posts. By training model on the pilot data collected so far, we were able to significantly in- crease the yield of posts that were actually labeled as empowering by annotators. To ensure we anno- tate a diverse range of posts, our final annotation task was done with half model-surfaced posts and half candomly-samplee posts.
Posts are often inherently ambiguous. Even with additional context, many posts could be rea- sonably interpreted as either empowering or disem- powering due to inherently ambiguous linguistic phenomena like sarcasm.
D Pilot Studies
Before crowdsourcing any data, we performed six internal pilot studies to iteratively refine our an-

---

Input template
Test
Post
"[post_text]"
63.5
[[post_tex]], reply: fresponse _eex)"
66.
+Response
2 "(post_text) (SEP TOMPP_ ( (response_tet text]"
65.7
"(pusL_iex)) [SKEE] (subreddit] [poster_gender]"
+Context
2 "A [poster_gendder in the (subreddit) subreddit: [post_text]"
65.0
1 "A [poster_gender] in the (subreddit) subreddit: (post_text) [SEP_TOKEN] (response_texi) _eext)"
67.1
+All
"(post (SXP TOKEN)
(OKEN)(rsponna (response (eX: TOKEN) (poster gender) (subreddit)"
Table 5: Templates used to convert additional context text input the classifier. The best-performing template for each model type was used in $4.5
E
Qualities of Empowerment
F
AMT Details
F.1
AMT User Interface
F.2 Task Instruction
19These pilot studies were conducted with the authors and small pool of computer scientists and NL.P researchers.

---

Post (cxt) Resporse: response). The post was writlee by a (opgaderr in the sabreckit] subredoic Rate how mucch the post empowers the respander. Choose between empower neutral discnpoover
(subeedcit)) Mubreddiic (eett A reader ewnmented. frespomee How does this resler feel absutt post? Chnowe between empow encd, diverm diseempwweet woemmppweee.... and neutral. Rate in hetther this post written by a (opgemder in the subreddin subverdit is em empowering dieempoeering or neutral [ees] Sommeone commented. [responbee)
post! Empoo uring diverapoo
Which of these choless hest deseriees the shove pose? Em impoereingg disco www.pppeeegggggo neatin).
best desrrbes eroweeiggg ocatral.
Rate how mich this lest empomeens people. Choone between empoweer neurral disempoweer
(tex)) This post was writen by (opgender) in the Isabredeit sibreden. and there was Reply. (nesponse] the post empoweting cringg disempowering or neusrall?
Would a reader feel empowered disem disempowerre or neural after reading this (opgender) 's post in the subrediin subreddi?? (exi) The reader replied: (resptonee
Empowering disempowering
[cot] This post was wiltion by a lopgeader in the Isabeedent subredon. Which ofl these chotes best deserive the abowe pos!? Empowaiing discoppowei
This is a Roldht post: teat) Somewne commented: Isspoone)) Rate whether the post was em empowering, metiral, or disenipow cringg to the commenier.
Chooe whether a reader woblld feel eompoeeed, diseppaverst or nentral after reading the following post. (tee)) The mender rpplicd: (respoonse)
(eet)) A reader commented: respoomer] Now does this realer aboul puns? Choose hetween empowereit disempooeett and nestral
enpowers responder. Choose betwee empower. nentral disempower
(opgenseer wrote this post le the (sulvredddr) sulovadir would it be empoowringg disenoowweri or nottral a reader" [teat]
Would a reader Jeel empoweeed eeeedd disempowerel, or neutral after reading this (opgender I's post in the Subredii sabecdit? [(xx]]
this emgowwrring diseppweering nentral?
neutral: [eet))
of
by Lopgender) in (uubredin subreddir. [ex] Which these choices deserbess
Choos between erupon diserpowerre neutral
(opgender in the subreddit subredditt is empowering disempowering
reader respon sepooded.. (responns)
Is the Kollowing pass en umpowwer oring. neutel, or disemppwerting (ees) Another user replice [responne]
[exi] Resporse: [respomse
(ext) Which of these choiees hest deseribes the above pos? Empoweriing disepooering neurrll
Rale how much this text empoowers people. Choose hetween empowes neatral disempowe
Would's reader feel empowered, disempowerss, or neural after reading this post [ext]
nectral!
[oggender the [subredcit subredan
(ext) This post war writien by a Logeendrr in the (saheedatt uubredden
(ext)) This poir was writton by a Orggender in the [sahredeit subreseen
Rate wetther this post is empow disenpowering or neutral: (text).
Post (ext) Reply: (respoesc) Is the post empowering disempoweting
Is the fnl lowing poss om umpowering nournal or discm hemmowwring" (1ex)..
subreddat subreddit.
Rate hether pund empow ering. divempooweri or neuiral.
plos empoweriin divempowering or neutral
Respouse: (respouse), Rate unch
(opgender preseet this Lopgender)
wrilten
Kate whelber this post written
would reader
post writien by
(eet This
(eett)
[teat)
Post:
This
39.5
38.8
32.6
25.4 4
39,
35.1
28.7
29.5
28.5
26.4
34,5
26.7
28.9
38,5
37.0
31.8
37.0
22.6
40.3
32.9
34.0
36.4
34.0
20.9
37,9
Inpott Type
powt
powt-2 2
post-3
post-4
post-5
poste6
post-7
resppuss-1
#responss-2
+response-3
#response 4
trosponnee
tresponse-6
wresponse7
#conicxt-1
#clineet222
+comtett-5
+all-1
+all-2
+ell-3
+
*all 6
mall-
Table 6: All prompts used to generate responses of GPT-3 and their macro F-1 performance over TalkUp

---

The reader has range of options Irom which to make choices. They are not restricted to only lavving few limited options.
The realer has rights. can also mean braaler rights such as the night for everylnne in he trented with dignity and respect,
The reader can be ascetivve and confidently express what they want. need. prefer. like. or disllie.
The rewder has the power to make their own decisions or influcnce docisions that affect them.
The realer can expresss their own ernoiives, like anger and sudness, in a heallhy way.
The reader can change how others perceive them their competecc and capacity:
The realer can Increase their positive self- self-imees image and Feel good shout themselves
The reader can think enitcclll and see things from different perspcttives
The reader can grow and change contioucesly and on their owna volition
The realer is capable of creating change in their life or community.
The realer can come out of the closet ceppeess who they really are.
The reader Coels that they can make difference as an andviduaal
The realer is able la leam new knowledge and skills.
The realer Leels that they are part of a group.
Adapted Definiton Provided
Learning to think eritically un lelleninng the conditioning seeing things differently
Changing ethers' perceptions of one compreency and capacity to
Growth and change that is never ending and self- eif-iiittteed Increasing one's a positive relninege and overcoming stgma
Having range of options from which make choices
learning deills deilsshall that the individual derines important
A feeling that the individuaa dual can make difference
Undersianding that people have rights lffeccing change in one's life and one's is communit
Net (eeling slone, leeling part of a group
Learning ahoul and es pespresssing tng angger
Having docision--ni aaa power
Chanberlin 1997) Element
Coming out of the clovet
Assetiveness
Table Components empowerment (14/15) from Chamberlin (1997) with definitions provided annotators. The element "Having access to information and resources" was replaced with Other because annotators were confused by what information excluded/included this element during the pilot studies.

---

If you have not read the instructions, please read them carefully before starting the annotation task. This very subjactive task, some variation in answers expected, but HiTs rejected the annotations clearly do not follow the instructions.
fallowing post by $(poster) $(subreddit) subreddit:
Post: $(post)
Example Reader's Response: S(response)
empowering or disempowering original post reader.
Please remember that you'll be annotating the original post, not the response.
Disempowering O
Maybe Disempowering
Neutral
O
Maybe Empowering
Empowering
Q1-2. Can this post be interpreted as either empowering or disempowering depending on what the poster meant?
Select Yes only if you think the post is truly ambiguous. If you're leaning lowards rmppowernnddissempoweingg leave the answer as No.
O
Q2. said the post empoowering/dllseppoovrrngg please select applicable reasons.
Definitions these reasons are in Instructions the sidebar. Leave this question blank the post is neutral.
decision-making power
range of options to choose
be assertive
has rights
capable of creating changes
difference
think critically & different perspective
express their emotions
alone
change perceplions their competency
leam skills
express true-self
grow continuously
positive self-image
others
responder agree poster?
Disagree
Neutral
Agree
Submit
Figure 10: The annotation interface presented Amazon Mechanical Turk from worker's view.

---

Instuuaions
x
you have not read the netructione, plesse reed them carsfully before starting the annotation task, This very subjective annotations clearty do not follow the instructions.
Overview
varfation
HITe be
this tosk, you will be askod road Redd! zoer annntas whethe would be empoweeigg
moder
The following post was writton by a S(poster) the S(subreddit) subreddit:
Please remember that you'll be unnotating the original post, not the responce. The response shown only because provide useful context.
Post S(pee))
Example Response: S(resporse)
amposeennee
01-2. You'll celect whether the given poet could be ensermeted wfter empoweting dbemmoweringg depardinng poster
Q1. Rate how empowering or disempowering the original post to reader.
Please remember that you'll annotating the original post, not the response.
OR. You'll aiso be apied to select which reasons the post would empoweer chempooer the neader the poer entorwaring sedect the ideaa that the pool supports, and It's diaon diammpo vering suleot the iooo that post goes agains
Disempowaring
Maybe Disempowering Neutral Maybe Empowering
Empowering
Q1-2. Can this post be Interpreted as either empowering or disempowering depending on what the poster meant?
as. Your usked matterr ponter nacder oril di sigroe hane perspectivos.
Select Yes only think is ambiguous. if you're leaning towerds empowerkggldseepooeee leave the answer No.
simtlar perapactive
No
Q1 Definitions
Q2. If said the post
please select applicable reasons.
Emporeering:.
Definitions of these reasons are the Instructions in the sidebar. Leave this question blank the post
nuriltive limertt sudt-ssteen ssteem. Emppeeernn posts imply
the following thinga about the reedos whather
decision making
options choose
assertive
intowiorelly or unimtentionell
has rights
capoble creating changes
can make cifference
Dieempwentng: dhsempewwving nake
umpoowe
think crttisally different perspective
exprees
leann akills
charge percaptions of their compatenncy
express true-sell
mouth,
imposing condi Juann about thei character:
grow continucusly
posilive self-image
others
Neutral: post
implications
Q3. Does the responder agree with the poster?
Disagree
Neutrel
Agree
Mos Instuctions
Figure 11: The annotation interface presented Ama- zon Mechanical Turk from a worker's ' view with instruc- tion sidebar opened.


Reviews:
[{"review_id": "FCFnMULfp5", "reviewer": "Reviewer_83gy", "summary": "", "questions": "-\tHope to listen to answer about the concern.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your great suggestion about mentioning the usefulness of our dataset for generation tasks; we agree and certainly envision TalkUp being useful for future applications by helping models generate more empowering language, and by generating suggestions to make human-written text more empowering. Some concrete examples we have in mind are: \n\n1. Rewards Model for Reinforcement Learning: As in (Sharma et al., WWW  2021), we believe a classifier trained with our data can be used to assign rewards that can tailor a generation model to be more empowering\n2. Controllable Text Generation with Constrained Decoding: As in (Yang and Klein, NAACL 2021; Liu et al., ACL 2021; Kumar et al., NeurIPS 2021), our classifier can be used to provide signals for controllable inference from pretrained generation models during the decoding process. \n\nIn fact, we are already conducting a follow-up study that uses TalkUp for controllable text generation to make dialogue more empowering in real-world conversational settings, such as education and therapy. Our hope is that TalkUp will enable further progress in both understanding and generating empowering language. We will add discussion of these potential applications and future research directions in our camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThis paper introduces a compelling dataset and novel tasks related to empowering language.\n-\tThe analysis of empowering language patterns from the dataset and the proposed application approach provide valuable insights for future research using the new dataset.\n", "reasons_to_reject": "Considering the current trend of generation tasks in this area, it would be interesting to explore the usefulness of the new dataset for tasks such as generating empowering posts and responses. While the paper does not delve into these applications, it would be beneficial to discuss and present potential results for these tasks in future research.", "questions_to_authors": "-\tHope to listen to answer about the concern.", "paper_topic_and_main_contributions": "The paper presents a new dataset, named TalkUp, designed for identifying empowerment in internet community posts. Reddit posts are annotated to determine if they are empowering or not, the type of empowerment expressed, and the agreement of responses to the posts. The authors analyze linguistic features, gender-related aspects, and reasons to empowering language in the dataset. Baseline models are employed to evaluate the performance of identifying empowerment in the posts. Additionally, the authors propose an application to explore patterns of empowerment in the Reddit community.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "It would be better to mention the Tables in the Appendix. For example, line 150 mentions ‘Appendix E’, but there is no words in the section. I guess Table 7 is the appropriate contents, so it would be better to refer the Table 7 in the section."}}, {"review_id": "4MkfnWB6Ih", "reviewer": "Reviewer_8biv", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful  comments and valuable  suggestions. Please find our responses to your feedback below:\n\n**More social attributes**: We agree that it’s important to consider many aspects of social context, and this was indeed a major motivation for our project – very few works (and datasets) in NLP contain *any* social information at all, so we push in this direction by releasing a dataset that contains not just language, but 3 social factors as well (gender, alignment, and topic). A central point of our paper is that empowering language cannot be understood in isolation and can only be captured in the context of several other social variables. We didn’t have access to other demographic information due to the source of our data, and it’s impossible for any paper to take every possible demographic factor into account. But exploring other sociocultural variables is a rich area for future work, and we strongly advocate for more exploration into intersectionality / ways that multiple demographic factors can influence the pragmatics of an interaction. We intend TalkUp to be a first step in this direction. \n\n**Extension to multi-turn dialogues**:  Thanks for your great suggestion – we agree that, in the big picture, studying multi-turn dialogues is important for understanding empowering language. In this paper, as an initial exploration into an area with little to no prior work, we simplified conversations in an effort to construct the most useful framework / dataset possible with our limited access to resources. Multi-turn dialogue would have introduced many practical challenges: we would need annotators to converse with each other for several turns (which is much more complicated and costly than having them evaluate existing conversations) and would need to collect annotator demographic information (which raises privacy concerns). Focusing on two-turn dialogue allowed us to reuse existing datasets such as TalkDown and RtGender, and establish a (simplified) baseline understanding of empowering language, and we hope it lays the groundwork for future explorations into longer conversations. We will incorporate this discussion into the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) Propose a novel research point, i.e., empowering language.\n\n(2) Provide a dataset annotated with empowerment, i.e., TalkUp.\n\n(3) Conduct comprehensive analyzes and make a step for future research.\n", "reasons_to_reject": "(1) This paper only analyzes empowerment based on gender. Other attributes, e.g., age, culture, etc., also affect the expression of empowerment.\n\n(2) This paper only focuses on the empowerment in a single-turn post-response setting, and the empowerment in multi-turn dialogue may be more complicated when it involves more context.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper dissects empowering language from the perspective of sociolinguistics. The authors crowdsourced a dataset from Reddit labeled for empowering. They analyzed the characteristics of empowering and disempowering language at the data level, and demonstrated the insufficient capabilities of existing models (e.g., GPT-3) at the model level.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f3bxhowGMi", "reviewer": "Reviewer_PeP9", "summary": "", "questions": "A) What is the label distributon in roberta surfaced posts vs randomly samples posts? \nB) Why, how is zero-shot gpt better than provinding examples to gpt? \nC) In table 2, for GPT3, why does post + response perfrom worse than simply using post? \nD) Will the code be made available?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. Please find our responses to your other questions below:\n\n**Annotator agreement**: Thank you for raising the concern about annotator agreement. We believe the agreement scores are actually reasonable given the complexity and nuance of this task. For empowerment annotation, we would neither expect nor want to have perfect annotator agreement – unlike many machine learning tasks, there is no “ground truth” on whether a text is empowering or not, and it’s an inherently ambiguous problem even for humans. If you compare our agreement scores with other computational social science papers on tasks of similar nature, they have similar agreement rates, especially when concerning pragmatics. For example, the dataset on latent hatred (ElSherief et al., EMNLP 2021) has a lower percentage agreement than ours, and our Fleiss’s kappa is similar to that of the Microaggression dataset (Breitfeller et al., EMNLP 2019).  We will clarify this more in detail in the paper. \n\n**Dataset size and distribution**: You pointed out that you think our dataset may be too small and skewed towards empowering language. The skew towards empowering examples was indeed by design: we want to reiterate that our data set complements the existing data set, TalkDown’s ~4000 examples (Wang and Potts, EMNLP 2019), which were skewed towards disempowering examples (explained in lines 624-638 of our paper). Due to our limited resources, the size of our dataset could not get much larger because collecting annotations for 2k examples alone cost $2000+. Thus, to build the most useful possible dataset for understanding empowering language, we focused on empowerment because TalkDown already captures a broad range of disempowerment. Such a dataset on empowering language previously did not exist, and TalkUp can be used in conjunction with a dataset of negative language like TalkDown to train models in the future. \n\n**Phrasing of ambiguity questionnaire**: We believe you might have misunderstood the phrasing in one of annotation questions (Q1-2). The question is asking if the text can be interpreted either way, which indicates its ambiguity, so annotators put “yes” to mark ambiguous cases that could not be clearly placed into empowering or disempowering without more context. None of our annotators reported an issue about the phrasing of the question; additionally, we manually reviewed all data labeled as ambiguous and verified that they made sense, so we don’t believe the quality of data was affected by how the question was phrased. \n\n**Adding LlaMA 2 after fine tuning as a baseline + Visualization of RoBERTa weights**: Thanks for these suggestions, and these may be valuable technical points that can guide future extensions of our work, but we believe that these are not within the scope of our paper to explore. As a computational social science paper, its main contribution is a new dataset that fills a gap and allows exploration of a new area of sociolinguistics – understanding characteristics of empowering language. Notably, our goal was *not* to build a state-of-the-art model: we presented results of RoBERTa and GPT3 to give a general picture of how well existing models work and to illustrate the usefulness of our proposed dataset (lines 437-441). We will clarify this more thoroughly in the paper. \n\n**GPT3 few-shot experiments**: As mentioned in the paper (footnote #8), our preliminary experiments show that few-shot underperforms the zero-shot setting, and we observe that the models become excessively fixated on the provided examples and struggle to generalize effectively to the input example. This phenomenon is well-documented in numerous previous studies (e.g., Fei et al., ACL 2023). Here’s an illustrative case: in the zero-shot setting, the correct label \"empower\" was obtained, whereas in the few-shot setting, an incorrect label \"neutral\" was assigned. This discrepancy could potentially be attributed to the sentence structure similarity between the example labeled as \"neutral\" and the input example. You can find the example case here: https://chat.openai.com/share/c096281b-9a85-4ab2-a2eb-84f869c922e6. Importantly, we want to highlight that this isn't a cherry-picked example; we consistently observed this pattern, and quantitatively, the few-shot approach consistently underperformed compared to the zero-shot approach. We can add more clarification about this in footnote #8; however, we would like to reiterate that this is a computational social science paper, and the details of GPT’s few-shot vs zero-shot performance are not important to the main contributions of this paper. \n\n**Will code be made publicly available?** Yes, both code for experiments and the trained classifier weights will be made publicly available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) They release a new dataset and there are very few language resources in this domain\n2) deep analysis of the intricacies of the data, including charecterestics of empowering language and its relation to gender.\n3) deep analysis of the ambiguity of empowering language \n4) potential uses to education and healthcare domain.", "reasons_to_reject": "1) The inter annotator agreement is low and the reasoning for the same is not provided/convincing enough. \n2) The paper claims that zero-shot GPT3 perfroms better than giving few examples in the promt (few-shot In-context learning). This is counter-intuitive and needs backing/analysis. \n3) although gpt3 is impractical to fine-tune, is might be possible to fine-tune LlaMA using LoRA. \n4) The data size (2000) is maybe not large enough? it could be increased and other sub-redits can be sampled. \n5) The data is un-naturally skewed towards the empower category. Adding more 'disempowering' posts can balance the data. Such posts can also be surfaced using the RoBERTa.\n\n6) possible improvement - would it be possible to have visualizaiton of roberta weights? something similar to https://github.com/jessevig/bertviz. This could strengthen the paper but not doing so in itself is not a reason to reject.  \n\n7) The question regarding ambiguity of of the post (Q1-2). The options seems inverted to me. If the post is ambiguous, the answer should be \"No\". However, the annotator is is asked to make \"yes\".", "questions_to_authors": "A) What is the label distributon in roberta surfaced posts vs randomly samples posts? \nB) Why, how is zero-shot gpt better than provinding examples to gpt? \nC) In table 2, for GPT3, why does post + response perfrom worse than simply using post? \nD) Will the code be made available?", "paper_topic_and_main_contributions": "The paper introduces a new annotated dataset \"talkUp\" on empowerment detection in reddit posts. The data set has 2000 samples categorized into 'disempower', 'neutral' and 'empower'. It also includes the reasons for empowerment. \nFurther, the paper analyzes the data to find the characterestics of empowering language, relation between gender and the use of empowering language etc.\nFinally, the paper offers 2 baselines for empowerment detection - a fine-tuned RoBERTa and zero-shot GPT3.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Works on hope speech detections are not cited or discussed : https://paperswithcode.com/task/hope-speech-detection.", "Typos_Grammar_Style_and_Presentation_Improvements": "Table 1 should have the number of neutral examples clearly mentioned."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agreed that the core contribution of this paper - the new dataset and novel tasks presented regarding the characteristics of empowering language - was strong and compelling. The CSS analyses of this data presented in the paper were characterized by reviewers as "deep" and "comprehensive," providing "valuable insights for future research."

Reviewers presented some issues for requiring clarification and noted areas of possible future work (such as addressing other aspects of social context beyond gender, multi-turn dialogue, comparisons with TalkDown, etc); authors responded thoroughly to reviewer concerns and noted ongoing future work that addresses these considerations, ultimately resulting in reviewer consensus that the work is sound and merits some excitement.

Overall reviewers found that this work makes a meaningful contribution worthy of dissemination.
Decision: Accept-Findings