Analysis of Style-Shifting on Social Media: Using Neural Language Model Conditioned by Social Meanings
Authors: Seiya Kawano
Shota Kanezaki
Angel Fernando Garcia Contreras
Akishige Yuguchi
Marie Katsurai
Koichiro Yoshino
Venue: EMNLP 2023 Findings (2023)
Keywords: style-shifting
speech accommodation theory
neural language model
TL;DR: We propose a novel framework for evaluating style-shifting in social media conversations.

Abstract: In this paper, we propose a novel framework for evaluating style-shifting in social media conversations. Our proposed framework captures changes in an individual's conversational style based on surprisals predicted by a personalized neural language model for individuals. Our personalized language model integrates not only the linguistic contents of conversations but also non-linguistic factors, such as social meanings, including group membership, personal attributes, and individual beliefs. We incorporate these factors directly or implicitly into our model, leveraging large, pre-trained language models and feature vectors derived from a relationship graph on social media. Compared to existing models, our personalized language model demonstrated superior performance in predicting an individual's language in a test set. Furthermore, an analysis of style-shifting utilizing our proposed metric based on our personalized neural language model reveals a correlation between our metric and various conversation factors as well as human evaluation of style-shifting.

Analysis of Style-Shifting on Social Media: Using Neural Language Model Conditioned by Social Meanings
Seiya Kawano', Shota Kanezaki AAgel García Contreras Akishige Yuguchi*., Marie Katsurai" and Koichiro Yoshino' 'Guardian Robot Project (GRP), RIKEN, Kyoto, Japan
2Graduate School of Science and Engineering, Doshisha University, Kyoto, Japan Tokyo University of Science
geiyyakkwann
angel..arciaconteeeaa..e akishige . yuguchi, koichiro. yoshino (rmeiinn@rriieen @riken. jp Ikanezaki21, katsurai } 0mm. doshisha. cp.
Abstract
conversation, such as on social media (Bell, 1984; Androutsopoulos, 2014; Birnie-Sitth, 2016).
this paper, we propose a novel framework for evaluating style-shifting in social media conversations. Our proposed framework cap- tures changes an individual's conversational style based surprisals predicted by a person- alized neural language model for individuals. Our personalized language model integrates not only the linguistic contents of conversa- tions but also non-linguistic factors, such as social meanings, including group membership. personal attributes, and individual beliefs. We incorporate these factors directly or implicitly into model, leveraging large, pre-trained language models and feature vectors derived from a relationship graph on social media. Compared existing models, personal- ized language model demonstrated superior performance predicting an individual's 's lan- guage in test set. Furthermore, an analysis of style-shifting utilizing our proposed met- ric based our personalized neural language model reveals a correlation between our met- ric and various conversation factors as well as human evaluation of style-shifling.
Style-shifting often associated with speech accommodation theory, which describes how speakers adjust their language use to converge or diverge from that of their interlocutors (Giles et al., 1987; Giles and Ogay, 2007). Convergence occurs when speakers adopt a similar style to their partner, aiming to foster understanding, build rap- port, and establish sense of affiliation. Diver- gence occurs when deviating from the interlocu- tor's language style to accentuate differences, assert individuality, or emphasize social identity. Effective style-shifting helps speakers adjust their psychological distance and improve their commu- nication intentions, leading to smoother conversa- tions (Ishihara and Takamiya, 2019). Measuring style-shifting phenomena is important to advance our understanding of conversation dynamics.
Research stylistic variation has underscored the intrinsic properties of conversational styles and their interplay with various non-linguistic fac- tors KKruijffKKorbayov et al., 2008; Kang and Hovy. 2021; Basile et al., 2019; Flekova et al., 2016). Furthermore, many studies have quan- tified the changes in speaker's language style in conversation based on the similarity in word choice between utterances (Nenkova al., 2008; Danescu-Niculescu-Min et al., 2011; Ire- land al., 2011; Nasir al., 2019; Kawano et al., 2020; Brugnoli et al.. 2019). However, despite these insights, such word-based methods still face challenges in capturing nuanced changes in style that reflect various social meanings. The term "so- cial meaning" represents a constellation of traits that linguistic forms convey about the social iden- tity of their users, such as group membership. per- sonal attributes, and individual beliefs (Jan-Petter and Gumperz, 2020; Hall-Lew et al., 2021). To evaluate style-shifting effectively, it is essential to go beyond merely observing words used in the
1 Introduction
Style-shifting refers to such changes in speaker's language style in conversation acts (Mizukami et al., 2016), vocabulary (Bren- nan and Clark, 1996), syntax (Reitter and Moore, 2007), prosody (Natale, 1975; Ward and Litman, 2007) according to the context of the conversation, the speaker's background, and the relationship with the interlocutor. For example, formal lan- guage is used in business situations; informal lan- guage is used in private situations. Style-shiftnng also happen within conversation to show spect intimacy to fit the situation (Giles et al., 1987; Giles and Ogay, 2007). Such changes in speech style also depend on the presence of peo- ple who may be directly or indirectly listening to a
speech

---

hisndy/ulooees schwork
source
larget
Ceseicatiin
Treooe actten mochigp
lear
Attendion brick
of trerdyytllowen
Food larward Kayer
Babii riples
Perondl erbedding
[SPP]
Eeilattion audience of rorestation
Takero of utteunce
Talern of larget allesance
The architecture of our personalized neural language model based on the transformer language model: The green blocks denote linguistic factors, an orange block denotes implicit social meanings and their relationships associated with each user, and yellow blocks reflect the potential social impacts of users and their utterances.
Figure 1:
conversation; we must also take into account the content of an individual's usual conversations and the social meanings. In other words, we need new framework to consider the individual's rela- tive viewpoint and account for non-linguistic fac- tors.
a unique personalized language model that incor- porates both linguistic and non-linguistic factors. 3) Our personalized language model outperforms existing models in predicting individual's 's lan- guage use in conversations, highlighting the effectiveness of incorporating non-linguistic fac- tors into language modeling. 4) We conduct a comprehensive analysis of style-shifting, reveal- ing correlations between our metric and various linguistic and non-linguistic conversation factors as well as human evaluation of style-shifting
To tackle these problems, we propose focusing on the surprisal experienced by neural language model personalized for an individual. The mag- nitude of surprisal in the personalized language model is inversely proportional to the consistency between the utterances they are confronted with and the standard language style they support. In other words, the smaller surprisal the more con- sistent the utterance is with the style of their lan- guage; and a larger surprisal indicates that the utterance deviates from their standard language style. The magnitude of surprisal also depends the incorporated factors of individuals. Com- paring surprisal under various conditions could not only assess the level of style-shifting but could also reveal valuable insights into the com- plex interplay of various factors that influence that change. Furthermore, neural language models are capable of taking into account long-term de- pendencies between words, deeper meanings, and structures in natural language (Vinyals and Le, 2015; Vaswani et al., 2017; Radford al., 2018). It can also incorporate non-linguistic factors, such as social meanings, addition to linguistic factors as conditions for personalizing neural language models for individuals (Li et al., 2016).
Neural Language Modeling Conditioned by Social Meanings
In this section, we describe our approach to devel- oping a personalized neural language model inte- grating both linguistic and non-linguistic factors.
2.1 Model Overview
Figure presents an overview of our personalized neural language model. This model generates re- ply pairs (S,T) from different users (us and uty) social media. The objective is evaluate the surprisal (or likelihood) of the target utterance T provided by ur. given the source utterance S from us. Differing from traditional language models for conversation generation, our model extends its consideration beyond mere linguistic factors such as the previous utterance S.
A distinctive feature of our model is the intro- duction of personalized user embeddings as lan- guage model conditions. These embeddings are inspired by few-shot learning for personalized di- alogue systems and are based on the concept of homophily: a theory in sociology that posits in- dividuals with similar social connections tend to
Our key contributions are as follows: 1) We introduce a novel metric for evaluating style- shifting, based on surprisal predicted by a per- sonalized neural language model. 2) We develop

---

demonstrate similar linguistic behaviors (Laniado al., 2012; Thelwall, 2010; Chee, 2010). Despite having limited user-specific conversation data, our model is capable of estimating an individual's lan- guage use trends. It accomplishes this by leverag- ing past conversations from similar users and the extensive language knowledge encoded in the pre- trained model (Radford et al., 2018).
aled non-linguistic factors. We employ trans- former language model based on GPT-2 (Radford et al., 2018), which has been pre-trained on a mas- sive corpus of text data. To integrate the non- linguistic factors associated with reply pairs into our language model, we replace the embedding corresponding to the first token of the transformer language model input with trainable, linearly transformed vector called personalized embed- ding.
Furthermore, our personalized embedding de- signed integrate non-ling non-linguistic factors such as the social meanings of users, Here, social mean- ings span various attributes, including the implicit social meanings and social relationships among individuals, and the social impacts of individu- als and their utterances determined by the number of friends/ffolowers and authority and hub scores. Each element offers unique insights into user's social interactions, thus enriching our understand- ing of their linguistic behavior. The social impact a user's utterances, determined through met- rics such as likes, replies, and retweets, also sig- nificantly influences their language style. By in- corporating these factors into personalized embed- ding, our model effectively captures the unique language styles of individuals.
2.3 Personalized Embedding
We introduce two non-linguistic factors to build a personalized embedding: the implicit social rela- tionship between users, and the social impacts of users and their utterances. Both factors are vec- torized as Vug-tur [Dug Pur] and vr, and inte- grated into the personalized embedding:
Implicit social relationships We calculate the implicit social relationship vector Vug vur [uugi Pur] by inputting the friend-follower net- work of users into the node2vec algorithm (Grover and Leskovec, 2016). The node2vec vectors are pre-computed using the whole network, and the vectors Tus and Tur corresponding to us and ur are acquired from the node2vec embedding. We expected this method to encode the structure of the social connections of users, leading to an under- standing of their implicit social relationships.
2.2 Language Model Training
In training our personalized neural language model, we consider not only linguistic factors, rep- resented reply pairs (S from different users (us and ur), but also account for non- -on-linguistic factors associated with each user and their utter- ances. These factors include implicit social mean- ings and social relationships among users, defined as Pus-ur [Dug 1, and the potential social impacts of users and their utterances, which are determined by the number of friends/followers as well authority and hub scores, defined as vs. Thus, the objective of this model is to minimize the negative log-likelihood of the target utterance given the conditions, as follows:
Social impacts of users and utterances We in- corporate several factors calculate the social im- pacts vector By, including:
Number of friends and followers: We con- sider the breadth of social connections of users, as signified by their number of friends and followers. Those with wider social con- nections could employ different language style in conversation, reflecting the number of their audiences.
Authority and hub scores: We use the HITS algorithm (Kleinberg, 1999) to evaluate in- fluences of users within their networks. The algorithm provides two scores for each user: the authority score, which estimates the value of user's content based the number of inbound links from high-hub users, and the hub score, which based on the number of outbound links high-authority users. A user with a high authority score are recog- nized by others who follow many influential
C - log (T)S vug-tor: Uy)
C is also interpreted as the surprisal of the target utterance T, according to information theory. In actual training. S and T are concatenated using special symbol [SEP1, and the model is trained to generate such concatenated text.
We adapt the task of modeling individual lan- guage use on social media by fine-tuning a pre- trained model using reply pairs and their associ-

---

users, suggesting that their content is valu- able. A user with a high hub score follows many users with high authority scores, indi- cating their broad awareness of who produces valuable content.
able insights into the complex interplay of various factors that influence that change.
Based on the idea of surprisal, we consider two perspectives that are not typically considered in conventional metrics. The first perspective is the similarity of language models among individuals during a conversation. The second perspective is the relative change in language style compared with their usual conversation. These aspects are often under-emphasized in existing studies that rely on vocabulary comparisons between speak- ers within the conversation (Nasir et al., 2019; Kawano et al., 2020; Brugnoli et al., 2019). How- ever, our approach takes both of these perspectives into consideration, enabling us to gain deeper un- derstanding of style-shifting by assessing stylistic alterations in relation our own usual conversa- tion.
Engagement of utterances: We consider the direct social impacts of utterances through various engagement metries. These include the number of likes, replies, retweets, and quotes an utterance has received from the community. Users whose posts generate high engagement may exhibit different communi- cation styles compared to those whose posts receive less attention.
After computing vectors Pag-tr and vr, ap- ply a trainable linear transformation generate personalized embedding:
Upersonalized = W[es vr; vn] b.
Perspective of similarity We evaluate the simi- larity of language use between two speakers based on the surprisal of their language models for the target utterance T':
Here Win is the weight matrix, and b is the bias vec- tor, both of which are learned during the model training process. This personalized embedding captures the unique language styles of individuals, making it a powerful tool for understanding lan- guage use in social media conversations. By in- tegrating these factors into a comprehensive user embedding, our model more effectively captures the unique language styles of individuals, leading to a more accurate understanding of the dynamics of language use in conversations.
8sim - - l log >s) + log p(Tlus vt) (3)
Here. Vur-tus denotes the case where the infor- mation of uT and us is swapped and fed to the model. In other words, we evaluate whether uy speaks us or us speaks to wr and whether they produce the same language. We excluded condi- tions and vy for calculating surprisal. This is because it may not be appropriate to use S and vy as conditions in the same way when the roles of uy and By are reversed. For instance, if S is not suitable utterance by ur in the first place, the sur- prisal will increase regardless Tf This similarity metric indirectly evaluates the similarity of their inherent language models. The score approaches zero when the two languages are similar (conver- gence), and has a large non-negative value when they are dissimilar (divergence).
3 Evaluation Metric of Style-Shifting
this section, we describe the metrics used to evaluate style-shifting in our personalized neu- ral language model. Our style-shifting evalua- tion metrics are based on the surprisal, which is the negative log- likeelhoo of target utterance calculated from our personalized neural language model conditioned by social meanings. In gen- eral, the magnitude surprisal an individual's personalized language model is inversely propor- tional to the consistency between the utterances they are confronted with and the standard lan- guage style they support. In other words, the smaller the surprisal, the more consistent the ut- terance is with their language style, and larger surprisal indicates that the utterance deviates from their standard language style. Comparing surprisal under various conditions could not only assess the level of style-shifting but could also reveal valu-
Perspective of style change We also evaluate how the likelihood of an utterance changing under the influence of specific contextual factors, based on the speaker's usual utterances to the conversa- tion partner. When specific contextual factors (S, vy) are isolated from the input conditions of the language model, the change in the surprisal of T can be evaluated:
Sphaage =| log, Mug-tay 1 (4)

---

Here, Schonge takes 0 if the language style of uy is constant regardless of the given current context, and takes larger values if the language style of ur varies relative to their usual language style.
we are assessing the style-shifting in conversations that were not part of the training data for our lan- guage models.
To verify the reliability our metrics of style- shifting, we created an additional, smaller test set, which includes manual annotations. This dataset consists of 200 reply pairs randomly selected un- der the condition that each user within the pair has record of more than 10 conversations and each user has conversed with other different users more than 5 times, from the test set. Two native evalu- ator for their language were asked to evaluate the style-shifting in the target utterance of each reply pair as follows:
By combining our similarity and style change metrics, can gain deeper insight into the dynamics of style-shifting, including convergence and divergence in conversations.
In our style-shifting evaluation metrics, we evaluate the surprisal of a language model for an utterance as the average of all the surprisal for the words in that utterance. However, as in previous studies, we can also limit our evaluation to each marker word frequently occurring in the corpus. Furthermore, we can replace Sgin with simple similarity between utterances. Thus our method is compatible with the style-shifting analysis meth- ods used in existing studies (Nenkova et al., 2008; Ireland et al., 2011).
Q1: How different do you think this reply tweet style from the tweet to which it is replying? [1. Very different, Somewhat different, 3. Rather different, 4. Neither, 5. Rather similar, 6. Somewhat similar, 7. Very similar]
4
Experimental Settings
This section describes dataset construction, train- ing, evaluation criteria, and analysis framework.
Q2: How different is this reply tweet from him/his usual style when they respond the same user they are replying to? [1. Very different, 2. Somewhat different, 3. Rather different, 4. Neither similar nor different, 5. Rather similar, 6. Somewhat similar, 7. Very similar]
4.1 Dataset Construction
We used Twitter data as conversations on social media to analyze style-shifting. From Twitter, to obtain the frilwessfollowers (FFF relationships be- tween users, reply pairs, and metadata associated with all users and tweets, we first selected five seed users: Japanese computer science researchers who are very active on Twitter, have many followers, and are well-known outside of social media. Then we obtained lists of their friends and followers. From the users in these lists, we obtained lists of friends and followers, representing second-degree FF relationships. Using the initial seed users and the relationship lists above, we built a social graph with users nodes and "follow" relationships as directed edges. After that, obtained all the tweets (from March 2018 January 2023) from the node users in the graph. At this time, tweet refers to another tweet (reply, quote, etc.), we also obtained the source tweet.
Q3: How different is this reply tweet from him/his usual style when they respond to the other user they replying to? [1. Very different, 2. Somewhat different, 3. Rather different, Neither similar nor different, Rather similar, 6. Somewhat similar, 7. Very similar]
The evaluators can refer not only to the target reply pairs themselves but also to their profiles, monologue tweets, and other conversations by the target user. They were asked to evaluate the style and tone, with word definitions referenced to the Japanese language dictionary much as possible.
To ensure the reproducibility of our experi- ments, while we cannot release the raw dataset from Twitter, will share the preprocessed dataset in accordance with Twitter's policies. Ad- ditionally, access our code for both language model training and analysis will be provided
For training the neural language model, we utilized 12.583,382 reply pairs tweeted between March 2018 and April 2022. We designated 860,701 reply pairs between users within the graph after April 2022 as the test set. Furthermore, the analysis of style-shifting was performed using conversations from this test set. In other words,

---

4.2
Training of Neural Language Models
pair (S, as follows:
We fine-tuned a Japanese GPT-2 model com- prised of approximately 336 million parameters. This model is constructed on a 12-layer trans- former architecture with a hidden size of 768 and was pre-trained using the Japanese CC-100 and Japanese Wikipedia datasets. For the train- ing (fine-tuning) process, we adopted batch size of 64 and employed gradient accumulation steps of 2. We selected the AdamW optimizer with learning rate set 2 10 and carried out the fine-tuning four RTX3090Ti GPUs for up to 5 epochs. We set aside 5% of our training data as a validation set, choosing the model with the lowest perplexity (PPL) on this set for our evalu- ations. We also trained both node2vec and HITS scores using the same training dataset, represent- ing features of users absent from the graph as zero negative values. For further details, please refer to both the repository of pre-trained model and our own repository.
s wund
WMD(S, 7)
(5)
Sim
Here, WMD is a function that calculates the se- mantic distance between two utterances based on the word mover's distance. channe is calculated as follows, using the average of distances to reply ut- terances in another reply pairs with the same con- versation partner from T and the average of dis- tances to reply utterances with different conversa- tion partners from! T.
und change
(Buuu Je
WMD
(6)
GGag.ayl
WMD(T.TTup)
Hind cchonpe,atfrr I uS-uug PeT IC-ug.ur
Here, Cus.rr is a set of other reply pairs by us and ur, and uug is any user other than ug.
We employed logistic regression model to pre- dict the binarized proposed style-shifting scores of reply pairs, using various non- non-linguistic factors described in Section 2.3 input features. Specifi- cally, we used the binarized sgim and Schange the style-shifting scores. We use this model identify important factors in style-shifting dynamics. To binarize such style-shifting scores, we set a thresh- old based on the 50th percentile of the reply pairs.
4.3 Evaluation of Neural Language Models
We used perplexity (PPL) to evaluate our person- alized neural language models by using the eval- uation dataset. We generated multiple models for comparison: our fully conditioned neural lan- guage model well as models from which some all the conditions were removed. For simplic- ity, we used the conditional probability of target T predicted by our model when describing these models. The conditions for the language model in- clude preceding utterance S, implicit user's s social relations Twg -trr, and the potential social impacts of conversation ey. Our hypothesis is that our con- ditioned personalized neural language model will better reproduce the language use of an individual, represented by a lower perplexity score.
5 Experimental Results
5.1 Performance of Neural Language Models
Table presents the perplexity scores of the neural language models on the test-set.
We evaluated the impacts of different conditions on perplexity. Perplexity can be improved if the following conditions are included: preceding ut- terance S, user's 's information Buy, social relation- ships information -.stablyNotttb. providing S as condition leads to the most substantial im- provement, suggesting that incorporating allows language model to make more context-specific predictions. Furthermore, including both S and user information Vug-tur leads to greater improve- ment in perplexity compared to using only S and in-
4.4 Analysis of Style-Shifting
First, we investigated the correlation between the metrics 8yim and Schange proposed in Section 3 and the human evaluation in Section 4.1. As the base- line based on the word-based similarity, troduced a score using Word Mover's Distance (WMD) (Kusner et al., 2015; Nasir et al., 2019; Kawano et al., 2020). We designed two intuitive baseline scores, swmd and chhnge corresponding to sumd is calculated using a reply Ssim and Schange sim
Bur- This suggests that the speaker's style is in- fluenced not only by their own characteristics but also by those of their dialogue partner, indicating the importance of considering the interaction dy- namics. Conversely, when specific linguistic con- text S is ignored, the speaker's style is predom- inantly shaped by their own features rather than those of a dialogue partner.
wmdl

---

Considering that information vy related to the tweet's social impact, perplexity is not improved by providing it as single condition or in combi- nation with other conditions. This suggests that the influence of vy on perplexity may be limited compared to other conditions or maybe over-fitted to conversations in training data. However, it is important that our style-shifting evaluation met- ric is designed capture the variation in perplex- ity when different conditions are given to the lan- guage model.
as per our hypothesis. The conventional metrics, in general, did not show a correlation with human evaluation results. This suggests the limitations of conventional metrics that evaluate only based on the similarity of words within observed utter- ances (Nasir et al., 2019; Kawano et al., 2020). In- deed, style is very nuanced concept, and even if the words used between utterances are similar, it does not necessarily reflect the style.
Table 2:
Correlation between human subjective evalu- ation results and proposed and conventional metrics
Table I: Test-set perplexity of neural language models
Comparison
Correlation cocfficient
Model objectives
Perplexity
QI Swin
-0,2 ***
log w/o fine-tuning
169.93
Schuuge
-0.13 *
log p(2)
15.20
Q3 VS. Selarge
-0.19 **
log p(T)UuT)
13.84
QI vs. 8
0.05
log ((T(rr)
15.51
Q2 vs. Schagg,some
0.09
log p(r'u
13.27
Q3 vs. Schavze
0.03
log pl (
14.41
log p( [Bug-T, Ur)
14.62
However, it should be noted that the observed correlation with human judgments and our met- ries, although statistically significant, is not par- ticularly strong. This could be due to limitations in the performance of language models used for scoring and potential biases in the samples used for subjective evaluation. Moreover, another con- tributing factor is that while the subjective evalua- tion data was randomly sampled, style-shifting oc- curs relatively rarely in conversation, and the inter- rater reliability is not particularly high. We might need to focus on improving the sampling method- ology for subjective evaluation.
lug p(T (1) (S) w/o fine-tuning
78.08
log (())))
13.33
log ((T) Uen)
12.63
log P( vr)
13.72
log PS Cwrs Dr)
13.86
log S(T), Ma
12.59
[T' Vug-tuy- VT)
13.40
5.2
Analysis of Style-Shifting
5.2.1 Relationships with Human Evaluation
Table 2 shows the correlation between the average subjective evaluation results of two human evalu- ators and various style-shifting metrics, including conventional metrics. Here, Q1 indicates the sim- ilarity of styles between reply pairs, Q2 indicates the style changes compared to usual conversations with the same conversation partner, and Q3 indi- cates the style changes compared to usual conver- sations with different conversation partners. #, ** wee indicate that the p-value in the significance of no correlation is less than 5%, 1%, and 0.1 1.,, respectively.
5.2.2 Prediction of Style-Shifting
We examined the contribution various non- linguistic factors of conversation to the prediction of style-shifting and evaluated the performance of predicting style-shifting in conversations. In Ta- ble 3, present the prediction results for style- shifting using the test set, employing 5-fold cross- validation. The chance rate denotes the predic- tion performance if we were to consistently pre- dict the same label. Our results derived from 8yīn demonstrate that the logistic regression (LR) mod- els, both with linear and second-ordes polynomial features, surpassed the chance rate, even without relying on text information. These findings un- derscore the superior performance in predicting style-shifting (either convergence or divergence). However, the prediction results based on Schange exhibit only a marginal improvement compared to the chance rate. This highlights the necessity
The results show that while the conventional metrics did not correlate much with human evalu- ation, the similarity metric and style change metric we proposed did correlate to a certain extent with human evaluation. Syin is designed to reflect the dissimilarity of styles between reply pairs, which showed a negative correlation with Q1, resulting per our hypothesis. Similarly, Schange is de- signed to reflect how much a speaker's utterances deviate from their usual style. It showed a negative correlation with both Q2 and Q3, again resulting

---

for non-linear predictions that take into account both linguistic and non-linguistic factors, as well as their interplay.
their conversation. This broader audience might prompt both users adjust their communication style to appeal to a more varied demographic. On the other hand, other metrics do not always yield such consistent results. This is due to actions such as replies, likes, and quotes reflecting different so- cial meanings.
Table 3: Performance of style-shifting prediction
Prediction models ($yin)
Prec.
Recall.
F1
Chance rate
0.25
0.50
0.33
LR
0,59
0.59
0.59
While the primary features we mentioned have non-zero coefficients as confirmed by the Wald test, they do necessarily have strong coeffi- cients. This offers a crucial insight for future research, emphasizing the need to identify addi- tional features that may have a more potent impact on predicting style-shifting
LR (Polynomial)
0.61
0.61
0.61
Prediction models (Sohaege
Prec.
Recall.
Fl
Chance rate
0.25
0.50
0.33
LR
0.53
0.52
0.49
LR (Polynomial
0,54
0,53
0.52
Figure 2 shows the importance of the features of the logistic regression model based on 8sim and Schange, which has been trained on the en- tire dataset. Here, positive weights contribute to predicting different language uses (divergence) of the utterances of reply pairs, and negative weights contribute to predicting similar language uses (convergence). The asterisks indicate features where the result of the Wald test yields p-value less than 5%.
Related Work
There are existing methods for analyzing style- shifting that utilize language models, similar to our proposed method. Some studies construct un- igram language models for each speaker in con- versation and compare differences in word likeli- hoods (Nenkova et al., 2008; Ireland et al., 2011). Another study evaluates whether one speaker's use of a marker increases the likelihood of its use by another (Danescu-NiculessuuN)) 2011). However, these methods often encounter issues with sparse training data due to the lim- ited number of conversations involving the same speaker (Mizukami al., 2016). As result, they carefully select a small number of marker words that capture the syntactic features of conversation for training the language model. Moreover, they do not consider lon-linguistic factors that could influence language style beyond speaker IDs. Our method addresses these issues by extending the existing methods that use unigram language mod- els.
Prediction based on Sxime The results high- light the importance implicit social meanings for prediction. Replies to authoritative inter- locutors show linguistically different properties, which may be associated with maintaining unique- ness expressing self- celf-assertion. On the other hand, replies interlocutors with high hubness show linguistically similar properties, which can be interpreted as reflecting adaptation to a style that promotes more effective communication or strengthens connections. If a reply T gets many "replics", the reply shows linguistic similarity with the interlocutor. This suggests that the use of similar language may encourage communica- tion involving the audience. Finally, the similarity of the implicit social meaning vectors of users us and uy and the presence or absence of a follow re- lationship are associated with convergence. This suggests that mutual interests or similar attributes between users lead to a match in language style.
et al.,
7 Conclusion
In this paper, we proposed a novel framework for analyzing style-shifting in social media conver- sations. Our framework captures changes in an individual's conversational style, leveraging sur- prisals predicted by a personalized neural lan- guage model. This model integrates not only the linguistic content of conversations but also linguistic factors such social meanings. Com- pared existing models, our personalized lan- guage model demonstrated superior performance in predicting an individual's language in a test set. Furthermore, our proposed metric for style-
Prediction based on Schange The results high- light the engagement metrics associated with the reply pair (S,T) T) significantly contribute the prediction of style change. Additionally, each en- gagement metric reflects either style maintenance or changes differently, depending on type. For instance, when both and garner many im- pressions, it suggests a wide audience observes

---

Aauthority score of Us
Ssim Schange
Authority score of
Hub score Us
Hub score
imressions of
impressions of
Retweets 0f
Retweets
Replics 0f
Replies of
Likes of
Likes of
Quotes of
Quotes of
Followert of Us
Followers 0f
Friends
Friends 0f 17
FF ratio of Ug
FF ratio 0f
followerd
Us
Us followed by Ur
Similarity of V5 to VT
-2
0
2-5
2
2
2
2
2
2
Figure 2: Importance features of style-shifting prediction based 8zin and Schange
shifting analysis revealed correlations with vari- ous conversation factors and human evaluations of style-shifting. In conclusion, our proposed frame- work provides not only effective ways for analyz- ing style-shifting in conversations but also holds significant potential implications for enhancing di- alogue systems. By applying the prediction mod- els or heuristics that we have derived from our findings, dialogue systems may gain a more pro- found understanding of conversational dynamics and adjust effectively to varying conversational styles in different situations.
while such non-linguistic factors are considered as input, their impact on the style of utterance gener- ated by the model may not be explicitly captured. In our model, non non-linguistic factors related to con- versation participants and their utterances are in- corporated pre-calculated static features in the model training, but changes in interpersonal rela- tionships or preferences over time are not consid- ered. Our society changes over time, and individ- uals' preferences and linguistic behaviors change accordingly. It is important to consider such fluid nature in personalized embeddings.
Limitations
Acknowledgement
Our metrics for evaluating style-shifting rely on variations in surprisal as evaluated by personalized language models under various conditions. This estimate of surprisal is based on the assumption that the performance of the personalized language model is somewhat reliable. For instance, if particular utterance is unusual under certain ditions and typical under others, there should be a variation surprisal between them. Therefore, the reliability evaluating style shifts in conver- sations of users with very limited training data, even when using few-shot learning, is diminished. Thus, there is a need to explore every means of training personalized neural language models.
This work was supported by JSPS KAKENHI 22K17958 and JST ACT-X JPMIAX22A4.
References
Jannis Androutsopoulos. 2014. Languaging when con- texts collapse: Audience design social network- ing. Discourse, Context & Media, 4:62-73.
Angelo Basile, Albert Gatt, and Malvina Nissim. 2019. You write like you cat: Stylistic variation predictor of social stratification. arXiv preprint arXiv:1907.07266
Furthermore, personalized language mod- els incorporate various non-linguistic factors that could potentially influence style changes. How- ever, the style is not explicitly handled within the model's latent representations. In other words,
Allan Bell. 1984. Language style audience design. Language in society, 13(2):145-204
Jessica Rae Bimie-Smith. 2016. Ethnic identity and
language choice across online forums. International Journal of Multilingualism, 1352):165-183

---

Susan E Brennan and Herbert H Clark. 1996. Concep- tual pacts and lexical choice in conversation. Jour- nal of Experimental Psychology: Learning, Mem- ory, and Cognition, 22(6): 1482.
Dongyeop Kang and Eduard Hovy. 2021. Style not single variable: Case studies for cross- stoss-stylistic lan- guage understanding. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference Natural Language Processing (Vol- ume I: Long Papers), pages 2376-2387.
Emanuele Brugnoli, Matteo Cinelli, Fabiana Zollo, Walter Quattrociocchi, and Antonio Scala. 2019. Lexical convergence inside and across echo cham- bers. arXiv preprint arXiv: 11452.
Seiya Kawano, Masahiro Mizukami, Koichiro Yoshino, and Satoshi Nakamura. 2020. Entrainable neural conversation model based on reinforcement learning, IEEE Access, 8: 8:178283-178299
Brant Chee. 2010, Sickness and health: Homophily in online health forums.
Jon M Kleinberg. 1999. Hubs, authorities, and commu- nities. ACM computing surveys (CSUR), 31(4es):5
Cristian Danescu-Nicaulesuu-ine Mizil, Michael Gamon, and Susan Dumais. 2011. Mark my words!: lin- guistic style accommodation in social media. In Proceedings of the 20th international conference on World wide web, pages 745-754. ACM.
Ivana Kruijff- Korbayová, Ciprian Gerstenberger, Olga Kukina, and Jan Schehl. 2008. The effect of dia- logue system output style variation on users' eval- uation judgments and input style. In Proceedings of the fifth international natural language generation conference, pages 129-137.
Lucie Flekova, Daniel Preotiuc- Preofiuc-Piettee Pietroo and Lyle Ungar. 2016. Exploring stylistic variation with age and in- come twitter. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 313-319.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to docu- ment distances. In International conference on ma- chine learning, pages 957-966.
Howard Giles, Anthony Mulac, James J Bradac, and Patricia Johnson. 1987. Speech accommodation the- ory: The first decade and beyond. Annals of the In- ternational Communication Association, 10(1):13- 48.
David Laniado, Andreas Kaltenbrunner, Carlos Castillo, and Mayo Fuster Morell. 2012. Emotions and dialogue in a peer-production community: the case of wikipedia. In proceedings of the eighth annual international symposium on wikis and open collaboration, pages 1-10.
Howard Giles and Tania Ogay. 2007. Communication accommodation theory. In Explaining communica- rion: Comtemporary theories and exemplars, pages 293-310. Mahwah, NJ: Lawrence Erlbaum.
Jiwei Li, Michel Galley, Chris Brockett, Georgios P. Spithourakis, Jianfeng Gao, and William B. Dolan. 2016. A persona- based neural conversation model. CoRR, abs/1603.06155.
Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In Proceed- ings of the 22nd ACM SIGKDD international con- ference on Knowledge discovery and data mining, pages 855-864.
Masahiro Mizukami, Koichiro Yoshino, Graham Neu- big. David Traum, and Satoshi Nakamura. 2016. dialogue
Analyzing the effect of entrainment acts. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dia- logue.
Lauren Hall-Lew, Emma Moore, and Robert Podesva. 2021. Social meaning and linguistic variation: the- oretical foundations. Social Meaning and Linguistic Variation:
Theorizing the Third Wave, page 1.
Md. Nasir, Sandeep Nallan Chakravarthula, Brian R. W. Baucom, David C. Atkins, Panayiotis Georgiou, and Shrikanth Narayanan. 2019. Modeling Interper- sonal Linguistic Coordination in Conversations Us- ing Word Mover' Distance. In Proc. Interspeech 2019, pages 1423-1427.
Molly E Ireland, Richard B Slatcher, Paul Eastwick, Lauren E Scissors, Eli J Finkel, and James W Pen- nehaker: 201 I. Language style matching predicts lationship initiation and stability. Psychological sei- ence, 22(1):39 44.
Michael Natale. 1975. Convergence of mean vocal in- tensity in dyadic communication as a function so- cial desirability. Journal of Personality and Social Psychology. 32(5):790.
Noriko Ishihara and Yumi Takamiya. 2019. Prag- matic development through blogs: A longitudinal study of telecollaboration and language socializa- tion. In Computer-assiste language learning: Con- cepts, methodologies, tools, and applications, pages 254, IGI Global.
Ani Nenkova, Agustin Gravano, and Julia Hirschberg. 2008. High frequency word entrainment in spoken dialogue. Proceedings of the 46th annual meeting of the association for computational linguistics on human language technologies: Short papers, pages 169-172 Association for Computational Linguis-
Blom Jan-Petter and John Gumperz. 2020. Social meaning in linguistic structure: Code-switching in norway. In The bilingualism reader, pages 75-96. Routledge.
ties.

---

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language under- standing by generative pre-training.
David Reitter and Johanna D Moore. 2007. Predicting success in dialogue.
Mike Thelwall. 2010. Emotion homophily in social network site messages. First Monday.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Iilia Polosukhin. 2017. Attention all you need. Advances in neural information process- ing systems, 30.
Oriol Vinyals and Quoe Le. 2015. A neural conversa- tional model. arXiv preprint 1566:::50660566
Arthur Ward and Diane Litman. 2007. Measuring con- vergence and priming in tutorial dialog. University of Pittsburgh.


Reviews:
[{"review_id": "XZNq1aA2n8", "reviewer": "Reviewer_gqZW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for your review and constructive comments. We acknowledge the issues you pointed out and will implement the necessary revisions in the camera-ready version.\n\nLimited Novelty - We acknowledge that the architecture of our personalized language models is partially similar to existing language models. This point is already articulated in the related work section, where we have incorporated insights from previous studies into our models. To clarify further, our models are based on fine-tuning pre-trained models and extending their prefix embeddings without dramatically altering the underlying network structure or loss function. This approach is simple, but powerful, and easily replicable. Additionally, the main objective of this study is not to invent a new language model architecture but rather to focus on the application of personalized language models that incorporate various non-linguistic factors for the specific task of evaluating style-shifting. In the camera-ready paper, we will clarify this focus and discuss potential avenues for enhancing the architecture and loss function of language models. To provide readers with further insights into the unique features and capabilities of our language models, we will also present actual generated outputs. This will serve to emphasize the distinctive contributions of our study in leveraging non-linguistic factors, even if the architectural foundations are based on existing models.\n\nOverreliance on Feature Engineering - As you pointed out, our feature set could be problematic when evaluating style-shifting in other domains. However, as indicated in the title of the paper, our study aims to evaluate style-shifting in \"social media.\" While the features we use are derived from Twitter, they are generally supported in other popular social media platforms (e.g., Facebook, Weibo, etc.). Hence, we believe they should not pose a significant barrier for other researchers wishing to validate our proposals on platforms other than Twitter.\n\nReproducibility - We understand the importance of reproducibility in research. To facilitate replication and verification by the community, we commit to releasing the code and dataset (anonymized as required). In the camera-ready paper, we will definitely include a valid download link for these resources.\n\nThank you for helping us improve the quality of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Consideration of non-linguistic factors: By incorporating non-linguistic factors like social meanings and individual beliefs, the proposed framework accounts for the individual's relative viewpoint and provides a more comprehensive evaluation of style-shifting.\n\n2. Comprehensive analysis: The research conducts a comprehensive analysis of style-shifting, revealing correlations between the proposed metric and various linguistic and non-linguistic conversation factors, as well as human evaluations of style-shifting. This analysis provides valuable insights into the complex interplay of factors influencing style-shifting.\n\n3. Real-world application: The framework's applicability to social media conversations makes it relevant and practical for understanding conversation dynamics in online interactions. As social media communication continues to play a significant role in society, this work addresses a relevant and timely research topic.", "reasons_to_reject": "1. Limited novelty - The proposed architecture is very similar to numerous existing models that attempt to combine linguistic and non linguistic features in language modeling. I would like the authors to present alternate architectures and provide an analysis or explanation for why some architectures may work better than others for this application. \n\n2. Over-reliance on feature engineering: It is unclear to me how much the performance of the proposed approach is dependent on the features hand selected by the authors to control for non linguistic variations. The semantics of these features may considerably vary across platforms as well. Is the model robust if these features undergo shifts in values? Can we select a very different set of features on another platform and still observe meaningful results?\n\n3. Limited reproducibility. The authors should ideally release code along with the paper via an anonymized repository to help reviewers validate the results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a framework for evaluating style-shifting in social media conversations between connected individuals. Depending on the social media, the semantics of the connection may vary. The proposed framework captures changes in an individual's conversational style based on \"surprisals\" predicted by a personalized neural language model, which is the deviation from the LLM's own language. This is modeled by combining linguistic contents and non-linguistic factors such as social meanings, group membership, personal attributes, and individual beliefs via customized features mined from the social network. These factors are directly integrated in the model architecture proposed by the authors.\n\nThe personalized language model is claimed to outperform existing models in predicting an individual's language use in new conversations. It would seem that the entire gain is obtained by the effectiveness of the non-linguistic factors mined from the social network and combined in the language modeling. The framework introduces a novel metric for evaluating style-shifting based on surprisal, which measures the deviation of individual language from the overall model. The approach focuses on the surprisal experienced by a personalized neural language model, inversely proportional to the consistency between the model predictions and the individual's language style. The \"surprisal\" also directly depends on the hand constructed social network features.\n\nThe paper's overall contributions include the \"surprisal\" metric, the language model architecture, the analysis of correlations with user features on the social network.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dobUW5ZZty", "reviewer": "Reviewer_3agR", "summary": "", "questions": "1. Is the user set in the test as a subset of training users?\n2. Is non-linguistic factor more helpful to improve the prediction performance?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and constructive feedback. We acknowledge the issues you have pointed out and plan to address them in the camera-ready version.\n\nAbout WMD - To clarify, WMD serves as a baseline evaluation method for style-shifting in our study, rather than being the metric we propose. Our actual proposed metric relies on the concept of \"surprisal,\" as derived from neural language models. We believe that your mention of WMD may have been a typo (probably you intend LM)  and wish to clarify this point to avoid any further confusion.\n\nCorrelation with Human Judgment - As you pointed out, the observed correlation with human judgments, although statistically significant, is not particularly strong. This could be due to limitations in the performance of language models used for scoring and potential biases in the samples used for subjective evaluation. While the subjective evaluation data was randomly sampled, style-shifting occurs relatively rarely in conversation, and the inter-rater reliability (kappa=0.3) is not particularly high. We might need to work on improving the sampling methodology for subjective evaluation. Additionally, we note that our evaluation of style-shifting is conducted on test-set conversations (not a subset of training-set); these were not part of the data used to train language models. We will clarify these points in the camera-ready paper.\n\nFeature Importance - As you pointed out, the features we used in the prediction of style-shifting do not contribute \"strongly\" to the predictions. This highlights an important insight for future research: we need to identify additional features that could have a stronger effect in predicting style-shifting. However, it is worth noting that the contribution of the key features we mentioned in the results is not zero, as confirmed by Wald tests. We will clarify these points in the camera-ready paper.\n\nImportance of Non-linguistic Features - To emphasize the importance of non-linguistic features in predicting style-shifting, we will add a comparison of results between models that use only linguistic features and models that use both linguistic and non-linguistic features in Table 3. Additional experiments demonstrate improvement when both types of features are utilized.\n\nUser Embeddings - We acknowledge your point that the set of user-specific embeddings is constructed from the training data. Hence, for new users appearing in the test set, we handle their vectors as zero vectors. Our subjective/objective evaluation only samples conversations that occur between users present in both the training and test sets. We are aware of this limitation and discuss it extensively in the \"Limitations\" section. Approximation techniques using the most similar user embeddings based on friends/followers lists could be a straightforward way to address this. We will add this discussion to the camera-ready paper.\n\nAbout the Dataset - Due to space constraints, only the size and collection period of the dataset were mentioned. We will add detailed statistics, including preprocessing methods, utterance lengths, and token counts, to the appendix in the camera-ready version.\n\nTraining Settings - As stated in the paper, most hyperparameters are based on the pre-trained models. Details on batch size, optimizer, learning rate,  number of epochs,  computational environment, and main hyperparameters of GPT are already mentioned but will be further elaborated in the appendix in the camera-ready paper. We also intend to share the experimental code and dataset.\n\nQ1 - A set of users is created from the training data (mentioned above). However, the test set conversations are not a subset of the training data; they are non-overlapping and collected over different periods. Thus, we are tackling the challenging task of evaluating style-shifting in conversations not seen during the language model training. We are currently evaluating the impact of including the test set in the language model training. If we complete this in time, the findings will be included in the camera-ready paper.\n\nQ2 - Additional experiments show that using both language and non-verbal features improves the predictive performance of style-shifting.\n\nReferences - Thank you for your suggestion to include more papers related to \"style variations.\" We will add the following studies into our paper. Additionally, we are continually investigating other potentially relevant literature to enrich the context of our study and will update our references accordingly.\n\nKruijff-Korbayová, Ivana, et al. \"The effect of dialogue system output style variation on users' evaluation judgments and input style.\" Proceedings of the fifth international natural language generation conference. 2008.\nKang, Dongyeop, and Eduard Hovy. \"Style is NOT a single variable: Case studies for cross-stylistic language understanding.\" Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.\nXu, Wei. \"From Shakespeare to Twitter: What are language styles all about?.\" Proceedings of the Workshop on Stylistic Variation. 2017.\nBasile, Angelo, Albert Gatt, and Malvina Nissim. \"You Write like You Eat: Stylistic Variation as a Predictor of Social Stratification.\" Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.\nFlekova, Lucie, Daniel Preoţiuc-Pietro, and Lyle Ungar. \"Exploring stylistic variation with age and income on twitter.\" Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2016.\n\nEnglish Issues - We apologize for the confusion regarding the terms \"surprise\" and \"surprisal.\" We will carefully review the paper for typos and grammatical errors and plan to get additional checks by native English speakers.\n\nThank you for helping us improve the quality of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose an interesting evaluation task.\n2. The experiments show promising directions.", "reasons_to_reject": "1. It is very hard to see validity of the proposed shift measurement from both human evaluation and automatic evaluation methods. For example, the correlation between the human results and the shifting values are low and mainly insignificant; and the prediction results by using the shifting values as features are close to a random guess (0.5). \n2. There is no inference function of the personal (user) embedding. This may indicate that the experiments may need to keep the user set same across both training and test. We do not know if the proposed language model is generalizable. \n3. Since the authors claimed their data contribution, we know very little about the data statistics. \n4. Experimental settings such as implementation details and hyperparameters are not known.", "questions_to_authors": "1. Is the user set in the test as a subset of training users?\n2. Is non-linguistic factor more helpful to improve the prediction performance?", "paper_topic_and_main_contributions": "This study proposes a neural language model that identifies style shifts across the social media conversations by integrating user-level features. The authors collected a Twitter data of Japanese CS researchers. The study evaluated the model performance by the perplexity, proposed a new evaluation metrics by WMD, and conducted human evaluations. While the new proposed analysis method lacks of enough correlation with human judgement, the results show some promising directions. The authors also using the logistic regression to verify the validity of the proposed WMD method, the results show some merits of the approach.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The authors may cite more papers from ACL anthology by the keywords of \"style variations\".", "Typos_Grammar_Style_and_Presentation_Improvements": "The submission used `surprisals`, but it misused the concept multiple times as `surprise`."}, "extra_text": {}}, {"review_id": "vUUKjKrzsz", "reviewer": "Reviewer_UhTx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and constructive feedback. We acknowledge the issues you have identified and plan to address them in the camera-ready version.\n\nRegarding the Analysis - To better describe how our proposed metric captures the nuances in style-shifting within conversations, we will include a case study in the paper. Specifically, we will show a comparison between the results of human evaluations and our style-shifting metric, across multiple conversation samples. This will help readers to better understand the areas in which our proposed style-shifting metric improves over prior methods.\n\nAbout the Dataset - We believe that our dataset, spanning four years, is robust and sufficient for evaluating the type of style-shifting we are exploring. In this paper, we define style-shifting as the degree to which a specific utterance deviates from an individual's typical conversational style; in other words, a local change of style, within a conversation. We operate under the assumption that this \"typical style\" is relatively stable over time (at least four years). Therefore, we are not considering very long-term changes in style, such as the transition from childhood to adulthood. We believe that exploring such temporal changes in style could be a promising direction for future research. To make these points clear, we will include further discussions in the camera-ready paper.\n\nReproducibility - We understand the importance of reproducibility in research. To facilitate replication and verification by the community, we commit to releasing the code and dataset (anonymized as required). In the camera-ready paper, we will definitely include a valid download link for these resources.\n\nYour feedback is invaluable in improving the quality of our work. Thank you once again.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting analysis\n- Use of linguistic and non-linguistic factors", "reasons_to_reject": "- The paper is entirely statistical. For a task like this, it is important to show the linguistic nuance that is captured by the metrics. \n- The choice of datasets: is 4 years sufficient period to study style shifts? what kind of style shifts happen in such a time? Without these answers, it is hard to appreciate what the model is capturing. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a method to evaluate style shifting in social media - accounting for both linguistic and non-linguistic factors. By incorporating non-linguistic factors into the first token of a GPT-based model and use the surprisal of the LM to determine the style change and similarity. The correlation with human annotations show better alignment of the scores to humans. ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a method for the analysis of style-shifting on social media. The problem is interesting and relevant, the evaluation is robust, and the results are promising. The reviewers point out some issues for which the authors have provided a reasonable explanation, and they are eager to provide more context or fix them. Two main issues are related to reproducibility, for which authors will provide the code and data upon acceptance. The second one is related to the proposed method and the fact that it relies on hand-crafted features; I see this as an opportunity for further work.

Pro
* Interesting problem applied to a current topic and of interest to the community
* Robust evaluations
* Authors eager to address issues and release code and data
* Authors estimate the addition would be on time

Cons
* None
Decision: Accept-Findings