Decoding the Silent Majority: Inducing Belief Augmented Social Graph with Large Language Model for Response Forecasting
Authors: Chenkai Sun
Jinning Li
Yi Fung
Hou Pong Chan
Tarek Abdelzaher
ChengXiang Zhai
Heng Ji
Venue: EMNLP 2023 Main (2023)
Keywords: Response Forecasting
Social Media
Social Network
Language Model
ChatGPT
Personalization
Response Prediction
TL;DR: We propose a social context learning framework that models shared beliefs among users by inducing a belief-centered graph using an LLM and demonstrate its effectiveness on response forecasting under various scenarios.

Abstract: Automatic response forecasting for news media plays a crucial role in enabling content producers to efficiently predict the impact of news releases and prevent unexpected negative outcomes such as social conflict and moral injury. To effectively forecast responses, it is essential to develop measures that leverage the social dynamics and contextual information surrounding individuals, especially in cases where explicit profiles or historical actions of the users are limited (referred to as lurkers). As shown in a previous study, 97% of all tweets are produced by only the most active 25% of users. However, existing approaches have limited exploration of how to best process and utilize these important features. To address this gap, we propose a novel framework, named SocialSense, that leverages a large language model to induce a belief-centered graph on top of an existent social network, along with graph-based propagation to capture social dynamics. We hypothesize that the induced graph that bridges the gap between distant users who share similar beliefs allows the model to effectively capture the response patterns. Our method surpasses existing state-of-the-art in experimental evaluations for both zero-shot and supervised settings, demonstrating its effectiveness in response forecasting. Moreover, the analysis reveals the framework's capability to effectively handle unseen user and lurker scenarios, further highlighting its robustness and practical applicability.

Decoding the Silent Majority: Inducing Belief Augmented Social Graph with Large Language Model for Response Forecasting
Chenkai Sun, Jinning Li, Yi R. Fung, Hou Pong Chan, Tarek Abdelzaher, ChengXiang Zhai, Heng Ji University of Illinois Urbana- Chaapaan (chenkai5, czhai,
Abstract
Pooitive
Nactral Nagative
Intenne)
Automatic response forecasting for news media plays a crucial role in enabling content produc- ers to efficiently predict the impact of news releases and prevent unexpected negative out- comes such social conflict and moral injury. To effectively forecast responses, is essential develop measures that leverage social dy- namics and contextual information surrounding individuals, especially cases where explicit profiles or historical actions of the users limited (referred to as lurkers). As shown in previous study, 97% of all tweets are produced by only the most active 25% of users. However, existing approaches have limited exploration of how to best process utilize these important features. To address this gap, we propose novel framework, named SOCIALSENSE. that leverages large language model induce belief-centered graph on top of existent so- cial network, along with graph-based propaga- tion to capture social dynamics. We hypothe- size that the induced graph that bridges the gap between distant users who share similar beliefs allows the model to effectively capture the re- sponse patterns. Our method surpasses existing state-of-the-art in experimental evaluations both zero-shot and supervised settings, demon- strating its effectiveness in response forecasting. Moreover, the analysis reveals the framework's capability to effectively handle unseen user and lurker scenarios, further highlighting its robust- ness and practical applicability.
Prediction Model
Resporse
Sentiment Intersity
Sentiment Polerity
Headlia: Abuus 200 people ware sreaee in liaa Samumay as sen groen proriex in country centrna rnto their fount
Hatorical Pesta
Jaaphy) How lurny! Democrata scrowiut Demoorata over. udmeestrrtton
Provdt Amnenizan willing conalter polittcal rolggious
democrecy potitical commettee wew. ND CiMe w/soicisation.
along with the logalli SA enmrnns
An example illustrating the task. The input consists of user attributes such as the profile and social context together with news media message. The model is asked to predict response in multiple dimensions.
Figure I:
research that enables numerous inffuential appli- cations, such as offering content producers a way to efficiently estimate the potential impact of their messages (aiding the prevention of unexpected neg- ative outcomes) and supporting human writers attaining their communication goals (Sun et al., 2023) for risk management. This direction is es- pecially important nowadays the proliferation of Al-generated misinformation, propaganda, and hate speech are becoming increasingly elusive detection (Hsu and Thompson, 2023; Owen and Zahn, 2023). this context, accurately forecasting the responses from different audiences or commu- nities to news media messages becomes critical.
1 Introduction
"Your beliefs become your thoughts. Your thoughts become your words. Your words become your actions." Mahatma Gandhi
One of the primary challenges in personalized response forecasting lies in developing effective user representations. A crucial aspect consider when representing a user the integration of social dynamics (e.g., social interactions around user) as well as their individual beliefs and interests. This becomes particularly relevant for users who lack
Automatic response forecasting (Figure re- ceivers for news media a burgeoning field of
The code is available at httpps//gitthun com/ chenkaisun/SSciaaleee

---

explicit profiles or historical activities (commonly referred to as lurkers). Previous efforts, however, have yet to explore the types of structural informa- tion that are helpful and how to best utilize such information (Lin and Chen, 2008; Giachanou et al., 2018; Yang et al., 2019; Wu et al., 2021).
izability unseen users and its predictive capabili- ties for lurkers. Our findings reveal two additional key insights: (1) the model performs exceptionally well in scenarios involving lurkers, outperform- ing the baseline by over 10% accuracy score in sentiment polarity forecasting, and, (2) compared to baseline approaches, the model exhibits con- sistently better generalization capabilitiees when applied to unseen users. Additionally, our anal- ysis underscores the significance of various compo- nents within the belief-augmented social network, revealing that both the belief-centered graph and the user-news interaction network play vital roles in determining the network's 's overall performance.
During our preliminary analysis, observed that users who share similar beliefs, specifically social values, are often situated in distant commu- nities within the explicit social network. To provide further context, our findings reveal that a significant portion (over 44.6%) of users in the network data we collected for our experiment share beliefs with other users who are at least two hops away in the network. This emphasizes the importance of con- sidering the connections between users with similar belicfs, even if they are not directly linked in the social network. Furthermore, previous research has indicated that user history plays a significant role in the model's performance. However, is often directly utilized without processing in existing ap- proaches, leading to the introduction of noise in the modeling process.
2 Task Formulation
In the task of Response Forecasting on Personas for News Media, our objective is to predict how users will respond to news media messages. Specifically, focus on analyzing the sentiment intensity and polarity of these responses. Formally, given per- sona P (representing the user) and a news media message M, our goal is to predict the persona's 's sen- timent polarity op (categorized as either Positive, Negative, or Neutral) and intensity Øine (measured on scale of 0 to 3) of the persona's ' response. We frame this task as a multi-class prediction problem.
Motivated by these findings, we introduce So- CIALSENSE (where Sense refers to the understand- ing and perception social dynamics and behav- iors within the online realm), a novel framework for modeling user beliefs and the social dynamics surrounding users in a social network. In this work, we conduct experiments using the SOCIALSENSE framework in the context of response forecast- ing. Our approach aims to capture the pattern of how "similar neighbors respond similar news similarly". To harness the potential of network features, we curated a new user-user graph com- prising 18k users from Twitter (the data will be anonymized when released), augmenting the origi- nal dataset (Sun et al., 2023). The SOCIALSENSE framework consists of three key stuges: (1) induc- ing latent user personas using the Large Language Model (c.g.. ChaGGT (Liu al., 2023)), (2) build- ing a belief-centered network on top of the existing social network, and (3) propagating information across multiple levels.
SOCIALSENSE
To accurately predict individuals' responses, it is crucial to develop an effective user representation that captures their personas. While previous stud- ies have utilized user profiles and historical data to model individuals' interests with reasonable accu- racy, there a significant oversight regarding the behavior of a large number of internet users who are passive participants, commonly referred to lurkers. This phenomenon is exemplified by statis- tics showing that only 25% of highly active users generate 97% of the content on Twitter (McClain et al., 2021). Consequently, the sparse historical data available for lurkers makes it challenging to infer their responses reliably. To address this issue, a social network-based approach can be employed to leverage users' social connections, gathering information from their neighbors. However, it is important to question whether relying solely on social networks is sufficient.
We demonstrate the effectiveness of our method through experiments on the dataset from Sun et al. (2023). Our results show that our framework out- performs existing baselines consistently across met- ries in both zero-shot and fuply-super settings. We further conduct a detailed analysis to address research questions concerning the model's general-
In this work, we introduce a novel perspective by borrowing the concept of belief and defining it in terms of social values. By considering so-

---

11Manse Propppaa dend
lating graph propagation with SOCIAL PROMPT.
R
eiy
User Information
ArriEtttee Operater
3.1 Unmasking Latent Persona with Large Language Model
Although the user's past posts can provide insights into their interests, they often contain noise that makes them challenging for models to consume. For instance, they may describe life events without providing context, such as "@ @user Waited all day next phone. Just got msg...". Furthermore, relying solely raw historical data discourages explainability response forecasting since past utterances are influenced by a person's internal belicfs rather than being the sole determinant their future response.
Extraction
Ne lightb
Propagation
Users
Human Value
NeighborN
Honi Value
boses Professon Others
Networks Construction
USeru
Figure 2: The figure illustrates our framework. In the first stage, we use an LLM to extract latent persona from the user's profile and historical posts. These moral and human value attributes from the latent personas, com- bined with the social network and media messages, collectively shape the belief-augmented social network. Graph-based propagation then used to update user representation. In the zero-shot setting, the L.M itself also assumes the role of an information propagator that combines information from neighbors (more details in Section 3.4).
In recent months, the Large Language Models (LLMs), particularly ChatGPT, have been shown to surpass human annotators in various tasks given their effective training techniques and access to vast amounts of pretraining data (Gilardi al.. 2023). This breakthrough presents unprecedented opportu- nities in analyzing users comprehensively without being scoped by previously established research. For the first time, we leverage a large language model (specifically, ChatGPT in our experiment) to extract users' internal beliefs and construct beliefs suitable for downstream consumption.
cial values, which encompass human values and moral values, capture individuals' deeply held convictions, principles, and ethical standards that significantly shape their perspectives, behaviors, and responses within social context. Our prelim- inary analysis reveals that individuals who share beliefs are often distantly connected, beyond resid- ing in the same community. Specifically, we found that over 44.6% of users in our collected network data share beliefs with others who are least two hops away in the network. This finding highlights the potential value of bridging these distant users and incorporating their beliefs as valuable features in response forecasting.
In this initial stage of framework, we de- sign a prompt Pt that enables us to extract latent information not available anywhere online. This includes dimensions such as human values, moral values, views on entities and issues, professions, and more. The prompt we have developed is shown in the Appendix. We refer to the latent persona extracted from the LLM for other words,
Userz. In
Userz = LLM((rofile, history, P:) (1)
3.2 Belief-Augmented Social Network
In this study, we present SOCIALSENSE (Fig- ure 2), an innovative framework for modeling user beliefs and the social dynamics within a social net- work by automatically curating belief-centered social network using a Large Language Model (e.g., ChatGPT). Our approach consists of three stages: (1) extracting latent personas using a Large Lan- guage Model, (2) constructing belief-centered network on top of the existing social network, and (3) information propagation. addition to the su- pervised method, we further explore how to achieve zero-shot prediction with social networks by simu-
To capture social interactions and bridge distant communities, our approach incorporates both exist- ing and induced social information to construct a network that focuses on modeling users' beliefs.
Our graph can be formally defined as follows: it comprises three sets of nodes, namely y repre- senting the news media messages, D representing the users, and yB representing fixed set of belief nodes. The graph consists of three types of edges: El E! and 1 For each edge (u, m) E El, where u E yo and m E y indicates that user u has interacted with the news media message m. For

---

each edge (nn, w2) EF, where wi, u2 De, it signifies that user uj follows user u2. Lastly, for each edge (u,b) E EB where E Ve and b y" it denotes that user u believes the value represented by node An illustrative example sub-graph of the network is shown in Figure 3.
Interaction
Search and po efferts..
Calorado
Seders dge who...
prosecutor
8
Distant
8
Social Relation Network The first layer of our network consists of the user-user social network, where edges from User a to indicate that User follows User b. This network captures the interests of users and the relationships between users.
8
8
Lamone Proffe Aztribur
Distant User
User-Media Interactions The second component of our network comprises news nodes and response edges indicating the users in the network have re- sponded these news nodes in the dataset. This feature offers two advantages. Firstly, it serves a representation of users' interests. Secondly, it facilitates the connection of users who are geo- graphically distant in the network but might share interests in news topics, thus enabling the expan- sion of the set of potentially reliable neighbors for any user we would like to predict.
Benevcience
Faimess
Purty
Authorily
Figure 3: An example illustrating a snapshot of the belief-centere social network. The latent persona at- tributes serve as a bridge between (potentially distant) users who share values. The arrow on the top left refers to the response aim to forecast.
nodes (instead of past utterances), therehy enhanc- ing the explainability of the process.
Belief-Centered Graph Lastly, we introduce be- lief nodes, composed of moral and human values (principles that guide behaviors) from the Latent Personas.
3.3 Information Propagation
Given the constructed belief graph, utilize Graph Neural Network (GNN) (Zhou al., 2020) to propagate information and learn an updated user representation, enabling us to infer user responses. Node Initialization To train the GNN, we first need to initialize the node representations. For user nodes Do we leverage Pretrained Language Model (PLM) such DeBERTa (He al., 2020) to encode the user's profile and history, yielding a d-dimensional dense vector u. Similarly, we initialize media nodes > by encoding the news headline message by the PLM, obtaining vector m. The embeddings for the fixed set of belief nodes yd b, are initialized by random vectors.
MORAL VALUES: Moral values are derived from a set of principles that guide individuals or soci- eties in determining what is right or wrong, good bad, and desirable undesirable. We define the set of Moral Values based the Moral Founda- tions Theory (Graham et al.. 2018), which includes Care/Harm, Fairness/Cheating. Loyalty/Betrayaa Authority/Subversionn, and Purity/Degradation.
HUMAN VALUES: Human values are defined based on the Schwartz Theory of Basic Values (Schwartz, 1992), encompassing Conformity, Tradition, Se- curity, Power, Achievement, Hedonism, Stimula- tion, Self-Direction Universalism, and Benevo- lence. These values represent desirable goals in human life that guide the selection or evaluation of actions and policies.
Graph Propagation We consider response fore- casting as a reasoming process over the connections among news media, user, and belief nodes the social graph. Leveraging the social homophily phe- nomenon, we posit that the constructed social ties lead to the formation of communities reflecting similarities and differences in beliefs, both within and across communities. To capture the interac- tions across different types of graph components, employ Heterogeneous Graph Transformer (HGT) (Hu et al., 2020), which was inspired by the architecture of the classic Transformer (Vaswani et al., 2017). Unlike homogeneous GNNs, HGT
Building upon the network from the previous stage, we establish connections between users and their associated values in an undirected man- This connection type offers two key benefits. Firstly, it introduces shortcuts between users who share similar beliefs or mindsets, facilitating the propagation of information across distant nodes. Secondly, allows the prediction results user responses to potentially be attributed to the belief'

---

effectively handles different edge and node types as separate meta paths, facilitating the learning of user representations from various types of contex- tual nodes.
where abbreviates User, U indicates the current central user, and R indicates the prediction results.
Experiment
Upon obtaining the updated user representations from HGT, we concatenate them with the news em- beddings. The resulting vector is passed through an MLP layer followed by a softmax activation func- tion for classification. The model is trained using cross-entropy loss, where the labels are sentiment intensity/polarity.
4.1 Data Construction
We use the dataset from (Sun et al., 2023) (denoted as RFPN) as the base for evaluation. The dataset consists of 13.3k responses from 8.4k users to 3,8k news headlines collected from Twitter. More de- tails are shown in the Appendix.
Network Data To test SENIAL curate social network using the official Twitter API! We initialize the network with the users in RFPN Xg. We collect all the users that each user u E X, follows and denote them as X,. We then select the top 10000 followed accounts from X₂ U X, as the most influential nodes, and denote them Xy. Lastly, we merge the top influencers with the original user set X, into the final set V X3 U X,- Our final graph consists of 18, 631 users and 1, 711,, 664 edges.
3.4 Zero-Shot Prediction by Simulating Propagation with Social Prompts
To forecast responses a zero-shot fashion, one approach involves directly feeding user profiles, historical data, and news headlines into large lan- guage models like ChatGPT. However, this ap- proach lacks the inclusion of the user's social net- work and encounters challenges when dealing with lurkers who have limited background information. As demonstrated in the experiment section, includ- ing social context provides a clear advantage in response forecasting. In this section, we introduce the concept of SOCIAL PROMPT to simulate infor- mation propagation in the supervised setting. Neighborhood Filtering To aggregate informa- tion, one needs to select information from neigh- bors. Since language models have limited context window and a user typically has hundreds of fol- lowers/followings we filter the set of neighbors by ranking the neighbors based on their influence on the user's opinion. In our design, we utilize the concept of authority from the persuasion tech- niques (Braca and Dondio, 2023), using the number of followers neighbor has determine their level of influence. We select the top-K neighbors A/K as the filtered set to represent the social context of the central user.
4.2 Experimental Setup
Evaluation Metrics We evaluate the prediction of sentiment intensity using the Spearman and Pear- son correlation, which are denoted as rs and r, respectively. For the classification of sentiment polarity, evaluate with the Micro-FI score (or equivalently aceuracy in the multi-class case) and Macro-Fl score, denoted as MiFI and MaF1.
Baselines We conduct a comparative analysis of SOCIALSENSE with several baseline models, in- cluding DeBERTa (He et al., 2020) (upon which our node initialization is based) and RoBERTa (Liu et al., 2019b), which are state-of-the art pretrained language models known for their performance across various downstream tasks like sentiment analysis and information extraction. Addition- ally, we compare our approach with the InfoVGAE model (Li et al., 2022), a statc-of-the-art graph rep- resentation learning model specifically designed for social polarity detection. InfoVGAE constructs graph that captures the edges between users and news articles to learn informative node embed- dings. We extend this model by incorporating user-user edges and also an additional two-layer MLP classifier head adapt for our supervised tasks. Furthermore, we include two naive base- lines, namely Random and Majority. The Ran-
Aggregation and Prediction Given the latent user personas attributes, Userž extracted for each neighbor n E A of central node extracted from Section 3.1 for each neighbor, and the fil- tered neighborhood from the previous step, con- struct prompt P. (shown in the Appendix) that allows the LLM produce socially aware per- sona Users. Finally, we design a prediction prompt Pp. which utilizes both Usery and Usery of the central node to make predictions. Formally,
R- = LLM(P,. LVMM,, (U?) {UE)"ENT MENK
htttps://developer: twi wuiiteercamm ctteer/cco////ccs/ twitter-api
(2)

---

dom baseline makes predictions randomly, while the Majority baseline follows the majority label. These baselines serve as simple reference points for comparison. Lastly, we compare our response forecasting results with CharGPT, a state-of-the- art zero-shot instruction-followiin large language model (LLM) (Yang et al., 2023). To predict the sentiment intensity and polarity using ChatGPT, we use the prompt Pp from Section 3.4 that incor- porates the user profile, user history, and the news media message as the input. We leverage the offi- cial OpenAPI with the sentiment prediction.
iint (%)
P (%)
Method
7o
MiFi
MaF1
Majority
43.41
20.18
Random
0.62
0.41
35.51
30.55
ChatGPT
43.80
44.15
58.61
48.67
DeBERTa
50.81
50.58
64.77
59.30
RoBERTa
52.09
53.00
65.26
59.02
InfoVGAE
58.61
58.37
67. 67.46
60.05
SocialSense
61.82
61.98
70.45
65.71
w/o belief
59.92
60.06
66.80
59.70
w/o user-news
55.43
55.35
66.51
61.96
w/o profile
59.94
60.01
64.49
59.04
w/o history
57.60
57.29
67.95
62.89
w/ random init
58.25
58.40
61.79
56.44
t-rubo model? for
Table Response forecasting results. We report the Spearman and Pearson correlations for the forecasting of sentiment intensity, as well as Micro Fl and Macro Fl scores for the sentiment polarity prediction. The best overall performance is in bold. Our framework outperforms the baselines consistently.
To illustrate the effectiveness of SOCIAL PROMPTS (Section 3.4), we compare three models: baseline ChatGPT, ChatGPT,, and SocialSense/eno. In ChatGPT,, we incorporate the latent persona Userz from Section 3.1, while in SocialSensezero we leverage the aggregated social context Users generated by SOCIAL PROMPT in addition to User, (Section 3.4). We use K = 25 for SOCIAL PROMPT. Similarly, we utilize the prompt Pp for response prediction. The detailed prompts can be found in the Appendix.
This highlights that the task can not be fully ad- dressed by zero-shot model alone.
On the other hand, the RoBERTa and DeBERTa models, despite being smaller pre-trained models, exhibit relatively better correlation and scores after fine-tuning for our response prediction task on news articles. However, these models only utilize textual information from news articles and user pro- files, disregarding potential interaction patterns and shared beliefs among users. This explains why their correlations and F1 scores are, average, 10.289 and 5.99% lower than those achieved by the pro- posed SOCIALSENSE framework. Additionally, the graph-based Info VGAE model achieves higher scores compared to the text-based DeBERTa and RoBERTa baselines, highlighting the significance of graph-structured data in enhancing response forecasting performance. However, the evalua- tion metrics of the InfoVGAE model remain lower than those of SOCIALSENSE. While the InfoV- GAE model constructs graph primarily based on user-user and user-news interaction edges, So- CIALSENSE goes a step further by inducing and integrating additional belief nodes and edges. This novel approach results in a heterogencous graph that forges connections among users who share similar perspectives and ideologies, thereby facili- tating the learning of intricate social dynamics and bolstering the model's predictive capabilities.
Implementation and Environments Our neu- ral models are implemented using Pytorch (Paszke et al., 2019) and Huggingface Transformers (Wolf et al., 2020). The intensity label in the dataset fol- lows the definition in the SemEval-2018 Task 1 (Mohammad et al., 2018), where the sign is also considered during evaluation. More implementa- tion details and discussions of reproducibility and hyperparameters be found in the Appendix.
4.3 Results Discussion
We conduct an evaluation of the proposed So- CIALSENSE model and the baseline models in- troduced in Section 4.2 for the supervised re- sponse forecasting task. The evaluation results are presented in Table 1. While the state-of-the-art models demonstrate competitive performance, So- CIALSENSE ouiperforms all other models across all evaluation metrics consistently. Although Chat- GPT is designed and proven effective for zero-shot instruction-following text generation, we observe that its performance in sentiment forecasting of re- sponses is comparatively limited, yielding lower scores compared to the other supervised models.
"nttps://platform http://paaa.ooooooo.....aaa irtenai.com/dccss. api-reeference/nodels
4.4 Ablation Study
We conduct an ablation study on different compo- nents SOCIAL SENSE to evaluate their impact on
'nttps://comptitiions competitions/17751

---

performance. The results are presented in Table Belief-Centered Graph To assess the effective- ness of the Belief-Centeree Graph in Section 3.2, we conduct experiment where we removed the belief nodes from the graph, including the nodes representing moral values and human values. This leads decrease of 1.91% in correlations and 1.83% in Fl scores. These findings support our hypothesis that incorporating belief nodes is ef- fective in modeling the shared beliefs and values among users. By including belief nodes, we en- able the graph learning framework to capture the association between the underlying principles and moral frameworks that guide users' behaviors and response patterns.
Pint (%)
dy (%)
Method
Ta
MiF1
MaF1
ChatGPT
43.8
44.15
58.61
48.67
ChatGPT,
44,43
44.76
59.77
48.69
SocialSensezeo
46.64
47.22
60.54
51.30
Table The above Zero-Shot Response forecasting results highlight that the SoCIAL PROMPT from Section 3.4 consistently offers an advantage.
Dint (%)
(%)
Method
MiFi
MaF1
Case Study:
Lurker Users
DeBEKTa RoBERTa
39.58
36.72
59.20
51.98
43.21
41.67
60.81
52.74
InfoVGAE
37.37
36.60
61.34
47.61
SocialSense
50.30
53.57
71.01
63.88
User-News Edges In this experiment, we exclude the user-news edges while constructing the belief- augmented heterogeneous graph. The results show that modeling the user-news interaction edges results in an improvement of up to 6.63% in cor- relation metries for sentiment intensity prediction. This indicates that modeling users' interests and historical interactions with media is crucial for ac- curately predicting sentiment intensity.
Case Study:
Unseen Users
DeBERTa
41.72
39.32
55.56
48.80
RoBERTa
38.06
35.71
55.20
47,99
InfoVGAE 36.08
35.06
56.27
47.86
SocialSense 44.40
44.27
62.55
55.37
Table 3: The case studies for Lurker and Unseen User Scenarios demonstrate that our framework exhibits sig- nificantly improved generalization capabilities when the user is unseen or has limited background context.
User Profile and Historical Posts The ablation study reveals the important roles of user profile data and historical post data in response forecast- ing. Excluding user profile data leads to a drop of 1.93% and 6.32% on average in the respective tasks, cmphasizing its significance in predicting sentiment polarity. Removing historical post data results in decrease of approximately 4.45% in correlations and 2.66% in Fl scores for sentiment polarity prediction. These findings highlight the importance of both data types, with profile data influencing intensity prediction more and historical data affecting polarity prediction more.
clusion of latent structured persona information indeed aids the model in comprehending the user more effectively. Furthermore, our model, So- CIALSENSEZa0. achieves the highest scores con- sistently across all metrics. This demonstrates the efficacy of our method for zero-shot social context learning and provides compelling evidence that even in the zero-shot setting, social context plays crucial role in response forecasting.
4.6
Evaluation on Lurker and Unseen User Scenarios
Node Initialization Instead of using the text rep- resentations of users' profiles and historical posts, we randomly initialize the node features. This re- sults a decrease of 3.57% in correlations and a significant decrease of 8.97% in Fl scores for po- larity classification. emphasizing the significance of text features in predicting sentiment polarity.
We evaluate the performance of proposed models and baselines on the task of response forecasting for lurker users, who are characterized as users with only a small amount historical posts. the experiment, we define the lurkers as the users with less than 50 historical responses (less than 85% of the users in the dataset), and the scenario consequently contains 745 test samples. The scores shown in Table 3. Compared to the previous evaluation results in Table overall evaluation scores for all the models are sig- nificantly lower. This can be attributed to the fact that lurkers have a much smaller background con- text, making response prediction more challeng-
4.5 Zero-Shot Evaluation
In addition supervised response forecasting, we also evaluate our framework under the zero- shot setting (Section 3.4). The results are pre- sented Table 2. Based on the higher scores attained by ChatGPT,, is evident that the in-
observe that the

---

ing. The lurker case is especially difficult for those baselines relying heavily on historical responses. this challenging scenario, SOCIALSENSE not only achieves significantly higher scores than oth- ers in all of the metrics but also maintains its perfor- mance on the polarity measures. Specifically, the advantage of our proposed model over DeBERTa and RoBERTa expands from 5.99% to 11.26% in terms of F1 scores for sentiment polarity predic- tion. These results demonstrate that even in cases where user textual information extremely lim- ited, our framework can still accurately infer re- sponses, showcasing the robustness of our method. Furthermore, it is worth noting that the intensity score was noticeably lower compared to the regu- lar setting, indicating that predicting the intensity of responses becomes more challenging when his- torical information is limited. We conduct further evaluation of the proposed model and baselines on unseen users, which refers to the responders who only appear in the evaluation dataset. This case study on unseen users provides insights into the generalization of the models. The evaluation re- sults are presented in Table 3. The results indicate that the unseen user scenario presents more chal- lenging task compared previous settings. More- over, SOCIALSENSE demonstrates significantly higher performance across all metrics compared to other baselines. This outcome underscores the framework's ' ability to effectively generalize to un- seen users, likely attributed to its robust modeling of the social network and encoding of relationships between users.
news (Sun et al., 2023). In contrast, we propose novel framework that leverages large language models to induce the graph structure and integrates disentangled social values to forecast responses, whether in a supervised or zero-shot manner. Our work demonstrates that effectively modeling the social context and beliefs of users provides a clear advantage in the social media response forecast task. This can ultimately benefit various down- stream applications such as assisting fine-grained claim frame extraction (Gangi Reddy al., 2022) and situation understanding (Reddy et al., 2023).
In the field of Social-NLP, related research has focused on applying NLP techniques, large lan- guage models (LLM), and prompting strategies model, analyze, and understand text data gener- ated in social contexts. For instance, progress has been made in misinformation detection (Fung et al., 2021; Wu et al., 2022; Huang et al., 2023b) and correction (Huang et al., 2023a), propaganda iden- tification (Martino et al., 2020; Oliinyk et al., 2020; Yoosuf and Yang. 2019), stance detection (Zhang et al., 2023), ideology classification (Kulkarni al., 2018; Kannangara, 2018), LM detoxification (Han et al., 2023), norms grounding (Fung et al., 2023), popularity tracking (He et al., 2016; Chan and King, 2018), and sentiment analysis (Araci, 2019; Liu et al., 2012: Azzouza et al., 2020). The emer- gence of advanced decoder language models like ChatGPT has led to extensive research on prompt- ing techniques and their application across various NLP tasks (Zhou al., 2022: Kojima al., 2022: Zhao et al., 2021; Diao et al., 2023; Sun al., 2022). Indeed, experiments have shown that Chat- GPT even outperforms crowd workers in certain annotation tasks (Gilardi et al., 2023). However, when it comes to social tasks like response forecast- ing, relying solely on large-scale models without taking into account the social context and users' personas may not yield optimal performance (Li et al., 2023). Our experiments demonstrate that incorporating social context in the prompt consis- tently enhances the LLM's performance, as show- cased in our simulation of information propagation using large language models.
5 Related Work
Existing research has focused on predicting the individual-level response using additional tex- tual features well deep neural networks (DNN) (Lin and Chen, 2008; Artzi et al.. 2012; Li et al., 2019; Wang et al., 2020). However, these existing methods neglected the important informa- tion about users' personas as well as the modeling of graph-structured interactions among users with the social items. Another line of related works formulates the response forecasting as text-level generation task (Yang et al., 2019; Wu et al., 2021; Lu al., 2022; Wang al., 2021). However, these lack a quantitative measure for analyzing the response (such as in the sentiment dimensions), limiting their applicability in downstream tasks like sentiment prediction on impact evaluation of
6 Conclusions and Future Work
In conclusion, we present SOCIALSENSE, frame- work that utilizes a belief-centered graph, induced by large language model, to enable automatic re- sponse forecasting for news media. Our framework

---

operates on the premise that connecting distant users in social networks facilitates the modeling of implicit communities based on shared beliefs. Through comprehensive evaluations, we demon- strate the superior performance of our framework compared to existing methods, particularly in han- dling lurker and unseen user scenarios. We also highlight the importance of the different compo- nents within the framework. In future research, it would be valuable to explore the application of belief-augmmenedt social networks in other domains and to develop an effective social prompting strat- egy for seneral-purposs upplications. Furthermore, it is worth investigating how response forecasting models can adapt efficiently to dynamically evolv- ing data, especially given the swift changes ob- served in real-world social media platforms (de Bar- ros et al., 2023; Cheang al., 2023).
Acknowledgement
This research is based upon work supported in part by U.S. DARPA INCAS Program No. HR001 121C0165. The views and conclusions con- tained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Gov- ernment is authorized to reproduce and distribute reprints for governmental purposes notwithstand- ing any copyright annotation therein.
References
Dogu Araci. 2019. Finbert:
Financial sentiment analy- sis with pre-trained language models. arXiv preprint arXiv: 10063.
Yoav Artzi, Patrick Pantel, and Michael Gamon. 2012. Predicting responses to microblog posts. In proceed- ings of the 2012 conference of the north American chapter of the Association for Computationa Linguis- tics: human language technologies, pages 602-606.
Limitations
While the proposed SOCIALSENSE framework demonstrates promising results in response fore- casting, there are limitations to consider. Firstly, the performance of the model heavily relies on the quality and availability of social network data, In scenarios where these sources are extremely lim- ited or noisy, the model's predictive capabilities may be compromised. Additionally, the generaliz- ability of the framework to different domains and cultural contexts needs to be further explored and evaluated.
Noureddine Azzouza, Karima Akli-Astouati, and Ro- liana Ibrahim. 2020. Twitterbert: Framework for twitter sentiment analysis based on pre-trained lan- guage model representations. In Emerging Trends in Intelligent Computing and Informatics: Data Science, Intelligent Information Systems and Smart Comput- ing 4, pages 428-437, 437, Springer.
Annye Braca and Pierpaolo Dondio. 2023. Persua- sive communication systems: a machine learning approach to predict the effect of linguistic styles and persuasion techniques. Journal of Systems and Infor- mation Technology, (ahead-of-print).
Ethics Statements
Hou Pong Chan and Irwin King. 2018 Thread pop- ularity prediction and tracking with permutation- invariant model. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 Novem- ber 2018, pages 3392-3401. Association for Com- putational Linguistics.
The primary objective of this study is to enable content producers to predict the impact of news releases, thereby mitigating the risk of unforeseen negative consequences such as social conflict and moral injury. By providing a stronger and more robust framework for forecasting responses, we aim to contribute to the creation of a safer online environment. In our process of collecting the net- work data using Twitter API, we strictly adhere the Twitter API's Terms of Use As part of our commitment responsible data handling, we will release only an anonymized version of the net- work data when making the code repository pub- licly available.
Chi Seng Cheang, Hou Pong Chan, Derek F. Wong, Xuebo Liu, Zhaocong Li, Yanming Sun, Shudong Liu, and Lidia S. Chao. 2023. Can LMs General- ize to Future Data? An Empirical Analysis on Text Summarization. In Proceedings of the Conference Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguis- tics.
Claudio Daniel Tenorio de Barros, Matheus R. F. Men- donça, Alex Borges Vieira, and Artur Ziviani. 2023. A survey on embedding dynamic graphs. ACM Com- pul. Surv. 15:2):10:1-10:
h developer- heaploppeeterrmm//greeeeee and and-policy

---

Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of- thought for large language models.
Tiffany Hsu and Stuart A. Thompson. 2023. Disinfor- mation researchers raise alarms about a.i. chatbots.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020. Heterogeneous graph transformer. In Proceedings of the web conference 2020, pages 2704- 2710.
Yi Fung, Christopher Thomas, Revanth Gangi Reddy, Sandeep Polisetty, Heng Ji, Shih-Fu Chang, Kath- leen McKeown, Mohit Bansal, and Avirup Sil. 2021. information consistency checking for fake news detection. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joini Conference on Natural Longuage Processing (Volume 1: Long Papers), pages 1683 1698.
Infosurgeon: Cross-media
Lring-siing Huang. Hou Pong Chan, and Heng Ji. 2023a. Zero-shot faithful factual error correction. In Proceedings of the 61st Anmual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9.14, 2023, pages 5660-5676. Association for Computa- tional Linguisties.
Yi R. Fung, Tuhin Chakraborty, Hao Guo, Owen Ram- bow, Smaranda Muresan, and Heng Ji. 2023. Norm- sage: Multi-lingual multi-cultural norm discovery from conversations on-the-fly. fly. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.
Kung- Huang, Hou Pong Chan, Kathleen R. McKeown, and Heng Ji. 2023b. Manitweet: A new benchmark for identifying manipulation of news social media. CoRR, abs/2305.144 14225.
Revanth Gangi Reddy, Sai Chetan Chinthakindi, Zhen- hailong Wang. Yi Fung, Kathryn Conger, Ahmed EL- sayed, Martha Palmer, Preslav Nakov, Eduard Hovy. Kevin Small, and Heng Ji. 2022. NewsClaims: A new benchmark for claim detection from news with attribute knowledge, In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 6002-6018, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Sandeepa Kannangara. 2018. Mining twitter for fine- grained political opinion polarity classification, ideol- ogy detection and sarcasm detection. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 751-752.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint D.Xiv:2205.11914
Anastasia Giachanou, Paolo Rosso, Ida Mele, and Fabio Crestani. 2018 Emotional influence prediction of news posts. In Twelfth International AAAI Confer- ence on Web and Social Media.
Vivek Kulkarni, Junting Ye, Steven Skiena, and William Yang Wang. 2018. Multi-view models for political ideology detection of news articles. arXiv preprint wrXiv:880.03449)
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text- annotation tasks. arXiv preprint ar 2x3vv.3033.15556
Jinning Li, Yirui Gao, Xiaofeng Gao, Yan Shi, and Gui- hai Chen. 2019. Senti2pop: sentiment-aware topic popularity prediction social media. In 2019 IEEE International conference on data mining (ICDM), pages 1174-1179, IEEE
Jesse Graham, Jonathan Haidt, Matt Motyl, Peter Meindl. Carol Iskiwitch, and Marlon Mooijman. 2018. Moral foundations theory. Atlas of moral psychology, 211.
Jinning Li. Huajic Shao, Dachun Sun, Ruijic Wang. Yuchen Yan, Jinyang Li, Shengzhong Liu, Hang- hang Tong, and Tarek Abdelzaher. 2022. Unsuper- vised belief representation learning with information- theoretic variational graph auto-encoders. In Pro- ceedings of the 45th International ACM SIGIR Con- ference on Research and Development in Information Retrieval, pages 1728-1738.
Chi Han, Jialiang Xu, Manling Li. Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, and Heng Ji. 2023. Lm-switch: Lightweight language model con- ditioning word embedding space. arXiv preprint 12779.
Ji He, Mari Ostendorf, Xiaodong He, Jianshu Chen, Jianfeng Gao, Lihong Li, and Li Deng. 2016. Deep reinforcement learning with combinatorial action space for predicting popular reddit threads. In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2016. Austin, Texas, USA, November 1-4, 2016, pages 1838- 1848. The Association for Computational Linguistics.
Sha Li, Chi Han, Pengfei Yu, Carl Edwards, Manling Li, Xingyao Wang, Yi R. Fung, Charles Yu, Joel R. Tetreault, and Heng Hovy, Eduard H; Ji. 2023. Defin- ing new nlp playground. ACL Findings.
Kevin Hsin- Yih Lin and Hsin-Hsi Chen. 2008. Ranking reader emotions using pairwise loss minimization and emotional distribution regression. In Procediiggs of the 2008 Conference on Empirical Methods in Natu- ral Language Processing, pages 136-14, Honolulu, Hawaii, Association for Computational Linguistics.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoddig-eenanned bert with disentangled attention. arXiv preprint arXiv:200000654.

---

Kun-Lin Liu, Wu-Jun Li, and Minyi Guo. 2012. Emoti- con smoothed language models for twitter sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pages 1678- 1684.
Shalom H. Schwartz. 1992. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In Advances in exper- imental social psychology. volume 25, pages 1-65. Elsevier.
Liyuan Liu, Haoming Jiang. Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019a. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv: 1908,03265.
Chenkai Sun, Jinning Li, Hou Pong Chan, ChengXi- ang Zhai, and Heng Ji. 2023. Measuring the effect of influential messages on varying personas. arXiv preprint arXiv:2305.16470
Yiheng Liu, Tianle Han, Siyuan Ma. Jiayue Zhang. Yuanyuan Yang. Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. 2023. Summary of chatgpt/gpt-4 research and perspective towards the future of large language models. arXiv preprint arXiv:2304.01852
Chenkai Sun, Tie Xu, ChengXiang Zhai, and Heng Ji. 2022. Incorporating task-specific concept knowledge into script learning. arXiv preprint arXiv:2209.0006.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 1.3I::1907..199...
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information pro- cessing systems, pages -998-6008.
Hongyuan Lu, Wai Lam, Hong Cheng. and Helen Meng. 2022. Partner personas generation for dialogue re- sponse generation. In Proceedings of the 2022 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 5200-5212.
Wei Wang. Piji Li, and Hai-Tao Zheng, 2021. Gener- ating diversified comments via reader-aware topic modeling and saliency detection. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 35, pages 13988-18 13996.
Zhongging Wang. Xiujun Zhu, Yue Zhang, Shoushan Li, and Guodong Zhou. 2020. Sentiment forecasting in dialog. In Proceedings of the 28th International Con- ference on Computational Linguisties, pages 2448- 2458.
Giovanni Da San Martino, Stefano Cresci, Alberto Barrón-Cedefed Seunghak Yu. Roberto Di Pietro, and Preslav Nakov. 2020. A survey on computational pro- paganda detection. arXiv preprint urXiv:2007.0800
Colleen McClain, Regina Widjaya, Gonzalo Rivero, and Aaron Smith. 2021. The behaviors and attitudes of us adults on twitter.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi. Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame. Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 28-45, Online. Association for Computational Linguistics.
Saif Mohammad, Felipe Bravo-Marquez. Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval- 2018 task 1: Affect in tweets. Proceedings of the 12th international workshop on semantic evaluation, pages 1-17.
Vitalii-AAnaa Olinyk, Victoria Vysotska, Yevhen Burov, Khrystyna Mykich, and Vítor Basto Fernan- des. 2020. Propaganda detection in text data based on nlp and machine learning. In MoMLeT+ DS, pages 144.
Xueqing Wu, Kung-Hsiang Hsiang Huang, Fung, and Heng Ji. 2022. Cross-document misinformation detection based on event graph reasoning. In Proceedings of the 2022 Conference of the North American Chupter the the Association for Computational Linguistics: Hu- man Language Technologies, pages 543-558, Seattle, United States. Association for Computational Lin- guistics.
Quinn Owen and Max Zahn. 2023. Avoiding potential "extinction event" from ai requires action, us official says.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. 2019. Pytorch: An imperative style, geg--performance deep learning library. arXiy preprint arXiv: 1912-01703.
Yuwei Wu, Xuezhe Ma, and Diyi Yang. 2021. Personal- ized response generation via generative split memory network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1956-1970, Online. Association for Computational Linguisties.
Revanth Gangi Reddy, Yi R Fung, Qi Zeng, Manling Li, Ziqi Wang, Paul Sullivan, et al. 2023. Smartbook: Ai- assisted situation report generation. arXiv preprint arXiv:2303. 14337.

---

Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. 2023. Harnessing the power of Ilms in prac- tice: survey on chatgpt and beyond. arXiv preprint arXiv:2304. 13712.
dimensions from {128, 256). We perform our ex- periments on a single NVIDIA RTX A6000 48 GB. Our model consists of 10, 484, 424 tuning parame- ters and it takes less than 30 minutes to fine-tune.
Ze Yang, Can Xu, Wei Wu, and Zhoujun Li. 2019. Read, attend and comment: A deep architecture for auto- matic news comment generation.
A.2 Analysis of Belief Data
We perform additional analysis on the belief data. Specifically, we show the distribution of the belief data (Figure 4), for which the moral value of care is dominant among the users. We have also seg- regated the model's performance in sentiment pre- diction based on the users' belief values and show in Table 6. Empirical results indicate that the model more accurate when predicting sentiments for users characterized by universalism and degra- dation. Conversely, the model finds it challenging to predict sentiments for users in the categories of security and stimulation. We further sampled 50 ChatGPT extraction results from user histories and distributed them among three human raters as- sess the accuracy of the extracted profiles. These raters are graduate students who qualified through an initial quiz comprising eight samples. On eval- uation, the raters assigned an average score of 3.9 out of 5 for accuracy. While not flawless, these ex- tracted beliefs play significant role in boosting the model's performance. Such finding indicates that refining the ChatGPT extraction process could po- tentially lead to enhanced performance outcomes.
Shehel Yoosuf and Yin Yang. 2019. Fine-grained propa- ganda detection with fine-tuned bert. Proceedings of the second workshop natural language process- ing for internet freedom: censorship, disinformation, and propaganda, pages 87-91.
Yuji Zhang. Jing Li, and Wenjie Li. 2023. Vibe: Topic- driven temporal adaptation for twitter classification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Singapore. Association for Computational Linguis-
tics.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference Machine Learning, pages 12697-12706. PMLR.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le, and Ed Chi. 2022. Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint arXiv:2205. 10625.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang. Cheng Yang. Zhiyuan Liu, Lifeng Wang. Changcheng Li, and Maosong Sun. 2020. Graph neural networks: A review of methods and applica- tions. Al open, 1:57-81.
A.3 Prompts Templates
We show all prompts used in the work in Figure 5, Figure 6, Figure 7, Figure 8, and Figure 9. They represent P,, Ps. Pp for the baseline ChatGPT, Pp for ChatGPT,, and Pp for SocialSensez..o respec- tively.
Appendix
A.1 Implementation Details
We implement the training framework using the 4.8.2 version of Huggingface Transformer li- (Wolf et al., 2020). For the graph model implementation Section 3.3, we use the 2.0.3 version of PyG The hyperparameters for the ex- periment are shown in Table and the ones not listed in the table are set to be default values from the transformer library. We use RAdam (Liu et al., 2019a) the optimizer. We perform greedy hyper- parameter search on the gnn_layer from (1.2.3). learning rate from (5c-5, lc-4, 5c-4, 1e-3), at- tention heads from (2, 4, 6, 8), activation from (tanh, relu), epochs from [350, 1000), and node
www.:eggoeeee.ieeeo readtheooss.. io/en/ latest/index.htae html

---

fairness
ham
loyalty
benevolence
betrayal
self-direction
authority
universalism
purity
achievement
tradition
subversion
security
conformity
power
hedonism
degradation
stimulation
cheating
7.5
10.0
12.5
15.0
17.5
Percentage (%)
Figure 4: Distribution of belief values
user (based prefile histery) by filling categories COMPREHENSIVELLY without lesing information. de not renane the categories. Seperate by newline: Hunan Values (i.e., what humen values does the veer Likely have. Choose fron "Conforrity", 'Tradition", "Security", "Power", "Achievement', "lledonisn', "Stimulation". "Self-Direction', "Universalisn". "Benevolence". These are based theery Moral Values (1.e., what norel, velues does the user Likely have, choose fron "authority", "betrayal', "care", "cheating", "degradation" "fairness", "harm", "Loyalty", "punity", "sudverston". Separate by commas), Ideologies (i.c., choase fron political idcslogies. Separate cormas), Possessions [i.e,, Interested tosics (i.e., Separate by comas),
Seperete by connas),
topics
user
inferred from the protile history.
Interested issues (or events) and the usen's stance toward each issue (i.e., fill the terplate exactly "Support: <issues> Neutrel: <issues> Against: <issues>", Fill cissues> with "None" if Interested entities
for that label), template exectly "Support: centities> Neutral: centities> Ageinst: centities>", Fill centities> with "Kone" you are not sure for that label), Profession (e.g., jobs, specialty. Separate by connas), Social roles (6.9., nother, father, Soparate comnss), Othar Natas.
the user's stance
entily
Answer the categorias carcisely and corprehansiveely. If category cannat be answerad, fill in the exact word "None" for the correspanding category.
iteratively
historical
information.
user twitter profile: 'sprofite'',
historical tweats (saparated sent catonc) 'Shistory)'
Figure 5: Prompt template Py used for extracting latent profile Userz in Section 3.1. The input consists of profile text and concatenated user historical posts. The output contains categories filled answers.

---

Assune there is user called User_0, 0, and there are rany accounts (who are also users) User_0 follows, and these accounts farn the comnunity around User 8. I will provide descriptions ans for each of these accounts. do the Tollowing question: Sumnarize the neighberhood context (in teres of doninant information) around User_B. That is, this
desorise nei naighborhood comnunity context). Also guess cesoribe the user_8 itself using information
(This is
User_8's belief and social its naighborhood) provided.
In other words, filling in the categonies below (using template Like categorys:<aanvee passible) but completely (i.e., choosing
[using as few words comprehensively, without losing information. Separate by newline:
"Conformity", "Tradition", "Sequrity", "Power', "Achieverent", "lledonisn", "Stinulation", "Self-Direction", "Universslisn", "Benevolence"" These are based on the basic theory hunan values. Otherwise simply choose "None" if not sure, Separate by commas),
Dominant
Doninent Moral Values (1.e., choosing fron "authority", "betrayal', "cere" "chesting", "degredation', "fairness", "harn", "Loyalty", "purity", "subversion". Doninent Ideologies (1.e., choosing fran political. ideologies, or hone, separated by cornas), Doninent Interested topics (1.0., Choose None if you cannot enswer. Saparate by cormes),
simply
"None" if
cornes),
Doninant issues (or evants) and the user's stance toxard each issue (1.9., fill the tenplata exactly "Support: Fill
<issues>
cissues>
tabel), tanplate exactly "Support:
user's stance toward each antity (i.e., contitics> Neutral: sentiticss ; Against: centitics>", Fill contitioss with "None" if you are nat sure for that Label),
Dominant Intorested
Deminant Protessions (e.g., jobs, specialty. Separate connas), User_à (i.e., using the inforration
neighbenhacd provided).
Other Notes.
Anseer the categories concisely and comprehensively. If there is clear doninant trend in category, fill in the "None" corresponding category.
the naighhors' information:
[List of Latent profiles from each neighbor}
Figure 6: Prompt template P, used for aggregating neighbor information in Section 3.4. It takes list of latent profiles from neighbors, where the latent profiles are output from Pt-
Predict response fron the user to the news headline in terws of exact comment words (i.e., what would user repty in commnnt), sentiment polarity (1.e., Positive, Neutrel, legative), and sentinent intensity (integer scaled between 0-3 inclusively
intensity sentiment intensity should be
intense).
sentinent polarity neutral,
In other words, filling the <answers in the categories belaw. Separate by newline:
Comment:
Sentiment Polarity: <answer>
Sentiment Intensity: sonswers
will iteratively provide
headline, protile, and historical Here infornation,
[news heedLine]: '(post)',
[user profile]: "(profile'',
[user historicsl posts]: "(history''.
Figure 7: Prompt template Pp used for predicting responses given only news message, user profile text, and concatenated user historical posts as input. used for evaluating the baseline ChatGPT.
Predict response fran the user to news headline tarus exact comnent words (1.e., what would user repty coment), santiment palarity (i.e., Positive, Neutral, Noçative), and sentinent intensity (integer sealed between 0-3 inclusively where neans na intensity and naans the nost intense). Note: when sentinent pelarity neutral, the sentinent intonsity should
In other words, filling the canswers the categories belaw. Separate by nenline: Coment: canswer>
Sentinent Polarity: Intensity:
I sill iteratively provide the news headline, user profile, user historisal posts, user structurized profile, and user social context. llere is the information,
[news heedline]; "(post),,
[user profitel: "(profite'',
historical postsl: 'Ihistory''.
[user structurized proritel: "ftaeett protite the user fron P_1]*
Figure 8: Prompt template Pp using user latent profile in addition to input Figure 7. It is used for evaluating ChatGPT

---

Predict response from comrent), sentinent polerity (i.t., Positive, Neutral, Negative), sentirent intensity (integer between inclusively where neans no intensity and 3 neans the nest intense). Note: when sentinent polarity is neutral, the sentirent intensity should be 8.
the headline
(i.e., would reply
words, filling
categories Separate by
Comrent: <answer>
Sentirent Polarity: canswees
Sentirent Intensity: canswers
will iteratively provide the news headline, user profila, user Mistorical, posts, user structurized profile (inforred by AI), and user sacial contaxt (inferred by AI) which descrihes the comninity around the user [ilan__ indicates the user itself). Here is the infornation,
headline]: "(post)'.
Lusen profite]: "(profile)',
historical poste]: "hhistory''.
Luser structurized profite]: '(tatent profile of the user from P_1)'
(comunitty)]: "(aggregated sociel from P_s}'
Figure 9; Prompt template p, using aggregated social context Users from Section 3.4 in addition input Figure 8. It is used for evaluating SocialSensezen in the experiment.
Name seed
Value
42
learning rate
5e-4
batch size
Belief Value
MiFl
Mal'l
weight decay
5c-4
conformity
62.96
59.35
RAdam epsilon
1e-8
tradition
57.14
48.29
RAdam betas
(0.9, 0.999)
security
50.00
43.42
scheduler
linear
warmup ratio (for scheduler)
0.06
power
66.67
60.84
achievement
69.23
57.91
number of epochs
1000
hedonism
56.25
46.03
patience (for early stop)
300
stimulation
33.33
16.67
gun layers
3
self-direction
59.68
48.61
attention head
4
universalism
73.02
64.04
activation
ReLU
benevolence
62.04
50.80
dropout
0.2
authority
62.50
56.40
node dimensions
128
betrayal
60.61
50.09
Table 4: Hyperparameters
care
62.81
52.19
cheating
75.00
42.86
degradation
87.50
46.67
faimess
64.04
56.22
harm
66.29
54.46
loyalty
68.28
60.17
purity
70.59
60.56
Split
Train
Dev.
Test
Samples
10,977
1.341
1,039
Table 6: Performance segmented by users' helief values.
Headlines
3,561
1,065
843
Users
7.243
1,206
Avg Profile Tokens
10.75
11.02
Avg Response Tokens
12.33
12.2
Avg Headline Tokens
19.79
19.82
19.72
Table 5: Summary statistics for the original dataset.


Reviews:
[{"review_id": "X1bDFTEkuZ", "reviewer": "Reviewer_2MJW", "summary": "", "questions": "The responses from Large Language Models like ChatGPT can vary. Please share your plans to maintain consistency in the responses generated by ChatGPT. \n\nWas any kind of human evaluation or validation done to understand if ChatGPT is suitable for unmasking latent persons?\n\nFor Table-2, could you please highlight how significant are the improvements?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 2MJW for the encouraging remarks on our work, especially on the innovativeness and effectiveness of our methodology. We address the comments as follows.\n\n**[== Consistency in the responses generated by ChatGPT ==]**\n\nWe understand the reviewer’s concern about the consistency of the response made by ChatGPT in real applications. In our work, for example, we used belief extracted by ChatGPT as an intermediate feature to help build the network. Indeed, the belief annotation from ChatGPT alone in one run might not be the best quality, which the method depends on. One approach to ensure high quality and consistency is to aggregate results on multiple offline LLMs (e.g., LLaMA and MPT) and perform ensemble prediction.\n\n\n\n\n**[== Effectiveness of ChatGPT for unmasking latent personas ==]**\n\nWe agree with the reviewer on the importance of analyzing ChatGPT’s capability of unmasking personas. In our recent progress, we sampled 50 ChatGPT extraction results done on user histories and distributed them to three human raters to verify the accuracy of extracted profiles. The human raters are graduate students who passed an initial quiz of 8 samples for filtering. We find on average the raters give a score of 3.9/5 in terms of accuracy. We will add the study to the next revision. In addition to this, in our work, we use the latent personas as intermediate nodes to enhance response forecasting instead of serving as final labels. Our empirical results demonstrate the effectiveness of the belief nodes in multiple scenarios, showing that the quality of the current annotation does have the merit of contributing to performance positively.\n\n\n\n**[== Presentation ==]**\n\nWe will surely highlight the significance of improvements in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors proposed a novel framework which leverages ChatGPT to predict responses of news articles for a given persona of users in terms of intensity and polarity of sentiments. This application of ChatGPT is innovative. It outperforms existing state-of-the-art approaches like InfoVGAE. \n", "reasons_to_reject": "The authors haven't shared how they will maintain consistency in the responses generated by ChatGPT. \nEffectiveness of ChatGPT for unmasking latent personas needs to be validated.\n", "questions_to_authors": "The responses from Large Language Models like ChatGPT can vary. Please share your plans to maintain consistency in the responses generated by ChatGPT. \n\nWas any kind of human evaluation or validation done to understand if ChatGPT is suitable for unmasking latent persons?\n\nFor Table-2, could you please highlight how significant are the improvements?\n", "paper_topic_and_main_contributions": "In this paper, the authors propose a framework called SocialSense. This framework uses ChatGPT to create a belief-centered graph leveraging profile information of users, their social content, and their historical posts. The objective of this framework is to predict responses of new articles for a given persona of users in terms of intensity and polarity of sentiments. The paper is more oriented towards real-life application of Large Language Models.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qi9rn6rU8f", "reviewer": "Reviewer_HaTs", "summary": "", "questions": "Were there any challenges in dealing with label inconsistencies in ChatGPT? For example, generating novel labels beyond the predefined set. If so, how were these challenges addressed?\n\nIs there a possibility of label leakage or user overlap between the training data (collected by you) and the testing data (Sun et al., 2023)? If such overlap exists, how might it impact the evaluation results and the generalizability of the proposed approach?\n\n\nCould you please provide insight into the rationale behind your choice of belief features and latent personas derived from MORAL VALUES and HUMAN VALUES? Additionally, do you have plans or ideas for extending this list of belief sources to further enhance the effectiveness of your approach?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer HaTs for the constructive and detailed comments. We appreciate the reviewer’s positive feedback on the novelty and effectiveness of the approaches. If given more space, we will be able to address all comments.\n\n\n**[== writing style and presentation ==]**\n\nFirst, we thank the reviewer for the thoughtful advice on paper structure improvement. In our final version, we will (1) remove redundancy between the abstract, introduction, and methods, (2) separate out the lengthy motivation part of section 3 and combine it with section 2 to make the flow more natural, and (3) add figures for zero-shot and supervised settings. We also agree that the current Figure 2 might be a bit condensed and we will simplify the structure by separating extraction, construction, and propagation into a more logical sequence.\n\n\n**[== challenges in dealing with ChatGPT label inconsistencies ==]**\n\nSecond, we understand the reviewer’s concern about the label inconsistencies of ChatGPT. Referring to the zero-shot experiment where ChatGPT is used to predict labels, in our prompt, we specifically constrain the labels within a range (e.g., only produce an integer score in a range, or only choose a label from a fixed set of choices), and we find that ChatGPT rarely generates novel labels beyond the predefined set. If due to fluctuation where it generates novel labels (e.g., a label number out of range) out of definition, we default the output to the most frequent label class in the training data. For value extraction from ChatGPT, similarly, we define the range in a prompt and filter out the novel labels (after canonicalizing plurals and upper/lower cases). We will elaborate on these aspects in our subsequent revision.\n\n\n**[== possibility of label leakage or user overlap between our and previous data==]**\n\nThird, we would like to note that the only difference between our data and that of (Sun et al., 2023) is network edges between users and additional influential nodes (who are followed by and not contained in the original users) added to capture the structure of the real network. There is also no annotation done on response labels. The news messages, user history, and response labels remain the same as those of the original dataset in every split. Our data is free of label leakage as we do not use any evaluation labels during training.\n\nUser overlap is a scenario that also happens in the original dataset from (Sun et al., 2023) (e.g., the same user responds to message A in the training split, and message B in the test split), and we took this opportunity to assess how well the model performs under Unseen User case by removing overlapping users from the evaluation data, and we find our method works well in both overlapping and unseen user situations.\n\nWe will make the points more clear in the next revision.\n\n\n**[== rationale behind the choice of belief features and future extension ==]**\n\nFinally, we choose belief features based on advice from social scientists and social psychologists on what human attributes might effectively forecast human action. We concluded with MORAL VALUES and HUMAN VALUES, which are guiding principles that help predict human reactions to social issues. For instance, previous research [1] has shown that moral values are associated with vaccination rates in the United States. In our future work, we will expand the belief graph to contain 1) fine-grained political ideologies 2) similarity-based pseudo edges between news (with the hypothesis that they make forecasting on similar news topics and isolated users more effective).\n\n> [1] Reimer, Nils Karl, et al. \"Moral values predict county-level COVID-19 vaccination rates in the United States.\" American Psychologist 77.6 (2022): 743.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper's novelty is commendable, particularly in its utilization of LLMs for practical applications. The combination of a supervised GNN and LLMs demonstrates an effective solution for complex tasks. \nThe ablation study strongly supports the argument that supervised learning along with LLM knowledge is more effective than pure supervised learning  or zero-shot approaches.", "reasons_to_reject": "The writing style could benefit from being more concise and focused. ", "questions_to_authors": "Were there any challenges in dealing with label inconsistencies in ChatGPT? For example, generating novel labels beyond the predefined set. If so, how were these challenges addressed?\n\nIs there a possibility of label leakage or user overlap between the training data (collected by you) and the testing data (Sun et al., 2023)? If such overlap exists, how might it impact the evaluation results and the generalizability of the proposed approach?\n\n\nCould you please provide insight into the rationale behind your choice of belief features and latent personas derived from MORAL VALUES and HUMAN VALUES? Additionally, do you have plans or ideas for extending this list of belief sources to further enhance the effectiveness of your approach?", "paper_topic_and_main_contributions": "The paper introduces a novel approach to predict user responses to news media messages by leveraging ChatGPT to extract user beliefs. The primary contribution lies in the construction of a belief-centered social network through a GNN. \nThis network captures the patterns of how similar neighbors respond to similar news. \nAnd the experiements validate its improved generalization, accuracy, and robustness.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The structure of the paper needs improvement. The abstract, introduction, and methods sections seem to overlap in content. Section 2 is too brief, while Section 3 takes a while to introduce the exact proposed method. Balancing the sections and their content could enhance the overall flow of the paper.\n\nConsider including a figure to illustrate the zero-shot and supervised settings. Figure 2 could be enhanced for clarity by simplifying its structure. Reducing the number of arrow directions and arranging them in a more logical sequence would aid comprehension."}}, {"review_id": "K8oJFEc2Aj", "reviewer": "Reviewer_EsDk", "summary": "", "questions": "1. Are certain beliefs harder to detect than others? Is there any performance gap associated with groups of different beliefs?\n2. How is the ground truth sentiment intensity and polarity calculated based on the user and message?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer EsDk for the helpful and detailed comments. If given more space, we will be able to address the comments and improve our current draft. \n\n**[==  the level of performance improvement  ==]**\n\n\nWe understand the reviewer's concern regarding the magnitude of the performance improvement as shown in some parts of the tables. It's worth noting, however, that our proposed method consistently outperforms all baselines across a variety of experimental scenarios.\n\n\nIn this work, we hypothesize that including network structure and belief nodes can address the issue of forecasting the response of lurkers. Through evaluation, we find that not only the issue is largely alleviated (with up to 11.9% increase in scores shown in Table 4), but the method also works well in unseen user cases from Table 4 (with up to 6.57% increase) and full data case (with 5.66% increase) from Table 2, all with consistent improvement over the best baseline. To further substantiate these results, we will add a significance test in our final version.\n\n\nWe would like to also note that the perception of performance improvement level might also depend on the difficulty of the task and domain. In our data analysis stage, both the social scientists consulted for this work and the authors concur that the task is inherently challenging; unlike many text-level classification tasks, our model often has to operate with limited user data, making predictions difficult even for human experts. To build upon this, future research will explore augmenting user context with multimodal information, such as images and videos, and will investigate the potential benefits of adding more anchor nodes.\n\n\n**[== introducing more evaluation tasks ==]**\n\nWe appreciate the reviewer’s suggestion on further tasks the method can be evaluated upon. We believe one of such tasks is measuring whether the user might be offended, which allows the method to be useful for more application scenarios. We will include the evaluation in our next revision.\n\n\n**[== analysis of belief data ==]**\n\nWe have made further analysis of belief data. Specifically, we show the distribution of the belief data, for which the moral value care is dominant among the users:\n\n| Belief Value      | % | \n| --------------- | ------ | \n|care  | 18.1%         |\n|fairness |  13.7%     |\n|harm  | 11.1%         |\n|loyalty  | 10.5%      |\n|benevolence |  7.0%   |\n|betrayal  | 5.6%      |\n|self-direction  | 4.4%|\n|authority  | 4.3%     |\n|universalism |  4.1%  |\n|purity  | 4.1%        |\n|achievement |  2.9%   |\n|tradition  | 2.9%     |\n|subversion  | 2.7%    |\n|security  | 2.5%      |\n|conformity  | 1.7%    |\n|power  | 1.5%         |\n|hedonism |  1.5%      |\n|degradation  | 0.5%   |\n|stimulation  | 0.5%   |\n|cheating  | 0.4%      | \n\n\nWe have also segregated the model's performance in sentiment prediction based on the users' belief values. Empirical results indicate that the model is more accurate when predicting sentiments for users characterized by universalism and degradation. Conversely, the model finds it challenging to predict sentiments for users in the categories of security and stimulation. We believe one natural direction for model improvement is to enhance persona extraction quality by, for example, taking ensemble predictions from multiple models. On the other hand, since belief values do not always hold equal significance in predicting responses to specific types of news messages, we could introduce a weighted attention mechanism to better prioritize certain belief values during the graph propagation process.\n\n| Belief Value      | MiF1 | MaF1 | \n| --------------- | ------ | ------- |\n| conformity  | 62.96  | 59.35     | \n| tradition  | 57.14  | 48.29      | \n| security  | 50.0  | 43.42        | \n| power  | 66.67  | 60.84          | \n| achievement | 69.23  | 57.91     | \n| hedonism  | 56.25  | 46.03       | \n| stimulation  | 33.33 |  16.67    | \n| self-direction  | 59.68  | 48.61 | \n| universalism  | 73.02  | 64.04   | \n| benevolence  | 62.04  | 50.8     | \n| authority  | 62.5 |  56.4        | \n| betrayal  | 60.61 |  50.09       | \n| care |  62.81  | 52.19           | \n| cheating  | 75.0  | 42.86        | \n| degradation  | 87.5  | 46.67     | \n| fairness |  64.04 |  56.22       | \n| harm  | 66.29  | 54.46           | \n| loyalty |  68.28  | 60.17        | \n| purity  | 70.59 |  60.56         | \n\nWe will include the results in the final version.\n\n\n**[== ground truth labels ==]**\n\nWe would like to note that the ground truth sentiment intensity and polarity labels have been annotated in the previous dataset (Sun et al., 2023). In the dataset, the author annotated labels on each response from a user to a message. In our modeling, we train the model to perform multi-class prediction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem statement is well-defined and the method is clearly described.\n2. The results for the Lurker and Unseen User scenarios are quite strong, which big improvements shown over the baselines.\n3. The empirical evaluation includes several baselines and ablation studies, which helps to indicate how well the model performs.", "reasons_to_reject": "1. More insights about the interaction between users and beliefs or the distribution of identified beliefs are not included. As this is one of the main contributions of this paper, it would be beneficial to show some analysis of what the beliefs data looks like in this network.\n2. The results of the proposed model are incrementally better than the baselines for response forecasting.\n3. The framework is only evaluated on the task of sentiment intensity and polarity prediction. It would be helpful to include other evaluation tasks as well.", "questions_to_authors": "1. Are certain beliefs harder to detect than others? Is there any performance gap associated with groups of different beliefs?\n2. How is the ground truth sentiment intensity and polarity calculated based on the user and message?", "paper_topic_and_main_contributions": "This paper proposes a new framework for predicting the sentiment intensity and polarity of twitter users. It creates a graph of users, posts, and beliefs, and includes user-user edges, user-post edges, and user-belief edges. The belief vertices are composed of moral and human values derived from latent personas of users. The latent personas are extracted using ChatGPT. Information is propagated through the constructed graph using a Heterogeneous Graph Transformer. The results are evaluated using correlation coefficients and F1 scores.\n\nThis paper proposes a framework using a large language model to generate a belief-centered graph to augment the existing social network. However, insights about the belief networks and clusters of users associated with certain beliefs are not discussed.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FEVYxD1GuU", "reviewer": "Reviewer_9BLt", "summary": "", "questions": "Q1: Now that LLMs cannot make accurate predictions of user responses, how can they make accurate predictions of user beliefs for lurkers? How you analyzed the prediction quality of user beliefs?\n\nQ2: Compared to the baselines, what's the number of parameters of your proposed framework?  How much time does it cost to fine-tune your model?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 9BLt for the insightful comments. We also appreciate the reviewer’s positive feedback on the significance of the performance gain and the novelty of using LLM for unmasking latent personas. We address the comments as follows.\n\n**[== analysis of LLM annotation and error propagation of noisy input ==]**\n\nWe understand the reviewer’s concern regarding the quality of LLM annotation. We performed a validation study on the accuracy of 50 ChatGPT extraction results (distributed to three human raters) and we found that on average the raters gave a score of 3.9/5 in terms of accuracy. We have also made an analysis of belief data. Specifically, we show the distribution of the belief data in which the moral value care is dominant among the users:\n\n| Belief Value      | % | \n| --------------- | ------ | \n|care  | 18.1%         |\n|fairness |  13.7%     |\n|harm  | 11.1%         |\n|loyalty  | 10.5%      |\n|benevolence |  7.0%   |\n|betrayal  | 5.6%      |\n|self-direction  | 4.4%|\n|authority  | 4.3%     |\n|universalism |  4.1%  |\n|purity  | 4.1%        |\n|achievement |  2.9%   |\n|tradition  | 2.9%     |\n|subversion  | 2.7%    |\n|security  | 2.5%      |\n|conformity  | 1.7%    |\n|power  | 1.5%         |\n|hedonism |  1.5%      |\n|degradation  | 0.5%   |\n|stimulation  | 0.5%   |\n|cheating  | 0.4%      | \n\nWe will include these results in the final version.\n\nIn our work, we employed the latent personas from LLM as extra nodes to improve the model. Our empirical results show that the introduced nodes are indeed effective, implying that the quality of the current annotation does contribute to the performance positively. For lurker cases where history is limited, we hypothesize that user-user might be a balancing indicator of user social context. We agree that it is important to improve the LLM annotation since the method relies on its quality. We will discuss how to improve LLM annotation in the revision.\n\nWe agree with the reviewer that some of the results from LLM might be subjected to errors since it’s not a perfect classifier. It is therefore important to analyze error propagation to understand the effect of noise better. In the final revision, we will incorporate an error analysis that quantifies how often negative performance is attributed to errors in the LLM annotation.\n\nRegarding Question 1, we also would like to note that the classification of user beliefs and forecasting of user response might have some differences. In the former case, a model is asked to perform classification on user history (e.g., a user who produces posts about praying for people who suffer from disaster has a moral value of care), while the latter is relatively harder as it is about forecasting reactions given often insufficient context (e.g., the user has scarce data or never discussed topics related to the message), which we find is sometimes even challenging for humans.\n\n\n\n**[== noise of introduced value nodes and model detail ==]**\n\nWe understand the reviewer’s concern that the value nodes might introduce noise. For the user-value-user path, two users are both connected by a value node if they have common values (out of 20 defined ones), forming implicit communities. In our parameter searching, we find the 3-layer to be relatively more effective (but the number of attention heads is also important) and we will include the analysis of the effect of the number of gnn layers in the final version. We also agree that solving the graph complexity issue certainly would shed light on improving the performance. One potential solution is to introduce weight on different meta paths. For instance, if we hypothesize one of the paths might introduce noise, we can investigate whether setting a lower weight can help performance.\n\nFor meta-paths, we have user *is associated with* value, user *follows* user, and user *responds to* news in our graph construction. Our model consists of 10,484,424 tuning parameters and it takes less than 30 minutes to fine-tune. We will include the details in our revision.\n\n\n\n**[== writing and presentation ==]**\n\nWe thank the reviewer for pointing out the symbol error and we will correct it in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A1. The idea of incorporating user beliefs for response forecasting is reasonable and can enhance the explainability of the predictions.\n\nA2. The performance gain is significant in both zero-shot and supervised settings. The analysis experiment supports the claim that involving user beliefs facilitates the inference of lurkers.\n\nA3. Utilizing LLMs to infer human values sounds interesting and worth further exploration. The proposed social prompt is also noteworthy, which mimics the feature aggregation in GNNs in a prompting way.", "reasons_to_reject": "R1. Though LLMs show some merits in serving as data annotation tools, the outcomes need further verification, especially in such an unsupervised way. Analysis of the annotation results should be included. \n\nR2. It is unclear why LLMs can provide convincing predictions of user beliefs for lurkers. If a big error happens at this stage, it will propagate to the following processing stages. This is also why R1 is important.\n\nR3: The introduced value nodes might bring noise and increase the complexity. As I understand, almost all the users are 2-hop away via the \"user-value-user\" path. By adopting a 3-layer GNN, the users will gather information from a large number of 2-hop neighbors, which might bring noise. And the graph is much denser, resulting in increased complexity.\n\nR4: The model details are missing, e.g., the choice of meta-paths when applying HGT.", "questions_to_authors": "Q1: Now that LLMs cannot make accurate predictions of user responses, how can they make accurate predictions of user beliefs for lurkers? How you analyzed the prediction quality of user beliefs?\n\nQ2: Compared to the baselines, what's the number of parameters of your proposed framework?  How much time does it cost to fine-tune your model?\n", "paper_topic_and_main_contributions": "This paper aims to predict the personalized user response to a specific news item, where a comprehensive user representation plays a vital role. Though existing works have employed user profiles, history posts, and social networks to enhance the social contexts of users, the authors claim that they are ineffective in dealing with lurkers. To this end, the authors propose to incorporate user beliefs and offer the SOCIALSENSE framework. The framework utilizes LLMs to infer the social values of user beliefs and augment the social network with a belief-centered graph. A heterogeneous graph transformer is then adopted to learn user and news representations from the augmented graph and infer the personalized user response. Experimental results demonstrate the effectiveness of the proposed framework, especially for lurkers and unseen users.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. N_{L} should be N_{K} in Line 360?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper advances the state of the art for response forecasting using novel methodology leveraging augmentation of social graph with LLMs. The claims are backed with strong evidence. The reviewers appreciated the novelty of the work along with the experimentation. The authors are recommended to address the reviewers' concerns in the camera ready, if the paper is accepted.
Decision: Accept-Main