RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Task Formulation
In the task of Response Forecasting on Personas for News Media, our objective is to predict how users will respond to news media messages. Specifically, we focus on analyzing the sentiment intensity and polarity of these responses. Formally, given a persona \mathcal{P} (representing the user) and a news media message \mathcal{M} , our goal is to predict the persona’s sentiment polarity \phi_{p} (categorized as either Positive , Negative , or Neutral ) and intensity \phi_{int} (measured on a scale of 0 to 3) of the persona’s response. We frame this task as a multi-class prediction problem.
SocialSense
To accurately predict individuals’ responses, it is crucial to develop an effective user representation that captures their personas. While previous studies have utilized user profiles and historical data to model individuals’ interests with reasonable accuracy, there is a significant oversight regarding the behavior of a large number of internet users who are passive participants, commonly referred to as lurkers. This phenomenon is exemplified by statistics showing that only 25% of highly active users generate 97% of the content on Twitter McClain et al. ( 2021 ) . Consequently, the sparse historical data available for lurkers makes it challenging to infer their responses reliably. To address this issue, a social network-based approach can be employed to leverage users’ social connections, gathering information from their neighbors. However, it is important to question whether relying solely on social networks is sufficient.

In this work, we introduce a novel perspective by borrowing the concept of belief and defining it in terms of social values. By considering social values, which encompass human values and moral values, we capture individuals’ deeply held convictions, principles, and ethical standards that significantly shape their perspectives, behaviors, and responses within a social context. Our preliminary analysis reveals that individuals who share beliefs are often distantly connected, beyond residing in the same community. Specifically, we found that over 44.6% of users in our collected network data share beliefs with others who are at least two hops away in the network. This finding highlights the potential value of bridging these distant users and incorporating their beliefs as valuable features in response forecasting.

In this study, we present SocialSense (Figure 2 ), an innovative framework for modeling user beliefs and the social dynamics within a social network by automatically curating a belief-centered social network using a Large Language Model (e.g., ChatGPT). Our approach consists of three stages: (1) extracting latent personas using a Large Language Model, (2) constructing a belief-centered network on top of the existing social network, and (3) information propagation. In addition to the supervised method, we further explore how to achieve zero-shot prediction with social networks by simulating graph propagation with Social Prompt .
Unmasking Latent Persona with Large Language Model
Although the user’s past posts can provide insights into their interests, they often contain noise that makes them challenging for models to consume. For instance, they may describe life events without providing context, such as “ @user Waited all day next to phone. Just got a msg… ”. Furthermore, relying solely on raw historical data discourages explainability in response forecasting since past utterances are influenced by a person’s internal beliefs rather than being the sole determinant of their future response.

In recent months, the Large Language Models (LLMs), particularly ChatGPT, have been shown to surpass human annotators in various tasks given their effective training techniques and access to vast amounts of pretraining data Gilardi et al. ( 2023 ) . This breakthrough presents unprecedented opportunities in analyzing users comprehensively without being scoped by previously established research. For the first time, we leverage a large language model (specifically, ChatGPT in our experiment) to extract users’ internal beliefs and construct beliefs suitable for downstream consumption.

In this initial stage of our framework, we design a prompt \mathrm{P}_{l} that enables us to extract latent information not available anywhere online. This includes dimensions such as human values, moral values, views on entities and issues, professions, and more. The prompt we have developed is shown in the Appendix. We refer to the latent persona extracted from the LLM for a user as User L . In other words,

User_{L}=\mathbf{LLM}(profile,history,\mathrm{P}_{l}) (1)
Information Propagation
Given the constructed belief graph, we utilize a Graph Neural Network (GNN) Zhou et al. ( 2020 ) to propagate information and learn an updated user representation, enabling us to infer user responses.

Node Initialization To train the GNN, we first need to initialize the node representations. For user nodes \mathcal{V}^{U} , we leverage a Pretrained Language Model (PLM) such as DeBERTa He et al. ( 2020 ) to encode the user’s profile and history, yielding a d -dimensional dense vector \mathbf{u} . Similarly, we initialize media nodes \mathcal{V}^{M} by encoding the news headline message by the PLM, obtaining vector \mathbf{m} . The embeddings for the fixed set of belief nodes \mathcal{V}^{B} , \mathbf{b} , are initialized by random vectors.

Graph Propagation We consider response forecasting as a reasoning process over the connections among news media, user, and belief nodes in the social graph. Leveraging the social homophily phenomenon, we posit that the constructed social ties lead to the formation of communities reflecting similarities and differences in beliefs, both within and across communities. To capture the interactions across different types of graph components, we employ a Heterogeneous Graph Transformer (HGT) Hu et al. ( 2020 ) , which was inspired by the architecture of the classic Transformer Vaswani et al. ( 2017 ) . Unlike homogeneous GNNs, HGT effectively handles different edge and node types as separate meta paths, facilitating the learning of user representations from various types of contextual nodes.

Upon obtaining the updated user representations from HGT, we concatenate them with the news embeddings. The resulting vector is passed through an MLP layer followed by a softmax activation function for classification. The model is trained using cross-entropy loss, where the labels are sentiment intensity/polarity.
Zero-Shot Prediction by Simulating Propagation with Social Prompts
To forecast responses in a zero-shot fashion, one approach involves directly feeding user profiles, historical data, and news headlines into large language models like ChatGPT. However, this approach lacks the inclusion of the user’s social network and encounters challenges when dealing with lurkers who have limited background information. As demonstrated in the experiment section, including social context provides a clear advantage in response forecasting. In this section, we introduce the concept of Social Prompt to simulate information propagation in the supervised setting.

Neighborhood Filtering To aggregate information, one needs to select information from neighbors. Since language models have a limited context window and a user typically has hundreds of followers/followings, we filter the set of neighbors by ranking the neighbors based on their influence on the user’s opinion. In our design, we utilize the concept of authority from the persuasion techniques Braca and Dondio ( 2023 ) , using the number of followers a neighbor has to determine their level of influence. We select the top- K neighbors \mathcal{N}^{K} as the filtered set to represent the social context of the central user.

Aggregation and Prediction Given the latent user personas attributes, User {}_{L}^{n} extracted for each neighbor n\in\mathcal{N}^{K} of central node c , extracted from Section 3.1 for each neighbor, and the filtered neighborhood from the previous step, we construct a prompt \mathrm{P}_{s} (shown in the Appendix) that allows the LLM to produce a socially aware persona User S . Finally, we design a prediction prompt \mathrm{P}_{p} , which utilizes both User L and User S of the central node to make predictions. Formally,

\mathcal{R}=\mathbf{LLM}(\mathrm{P}_{p},U^{c}_{L},\mathbf{LLM}(\mathrm{P}_{s},\{U^{n}_{L}\}^{n\in\mathcal{N}^{K}})) (2)

where U abbreviates User, U c indicates the current central user, and \mathcal{R} indicates the prediction results.
Data Construction
We use the dataset from Sun et al. ( 2023 ) (denoted as RFPN) as the base for evaluation. The dataset consists of 13.3k responses from 8.4k users to 3.8k news headlines collected from Twitter. More details are shown in the Appendix.

Network Data To test SocialSense , we curate a social network using the official Twitter API 1 1 1 https://developer.twitter.com/en/docs/twitter-api . We initialize the network with the users in RFPN X_{s} . We collect all the users that each user u\in X_{s} follows and denote them as X_{t} . We then select the top 10000 followed accounts from X_{t}\cup X_{s} as the most influential nodes, and denote them X_{f} . Lastly, we merge the top influencers with the original user set X_{s} into the final set \mathcal{V}^{U}=X_{f}\cup X_{s} . Our final graph consists of 18,634 users and 1,744,664 edges.
Experimental Setup
Evaluation Metrics We evaluate the prediction of sentiment intensity using the Spearman and Pearson correlation, which are denoted as r_{s} and r , respectively. For the classification of sentiment polarity, we evaluate with the Micro-F1 score (or equivalently accuracy in the multi-class case) and Macro-F1 score, denoted as MiF1 and MaF1 .

Baselines We conduct a comparative analysis of SocialSense with several baseline models, including DeBERTa He et al. ( 2020 ) (upon which our node initialization is based) and RoBERTa Liu et al. ( 2019b ) , which are state-of-the-art pretrained language models known for their performance across various downstream tasks like sentiment analysis and information extraction. Additionally, we compare our approach with the InfoVGAE model Li et al. ( 2022 ) , a state-of-the-art graph representation learning model specifically designed for social polarity detection. InfoVGAE constructs a graph that captures the edges between users and news articles to learn informative node embeddings. We extend this model by incorporating user-user edges and also an additional two-layer MLP classifier head to adapt it for our supervised tasks. Furthermore, we include two naive baselines, namely Random and Majority . The Random baseline makes predictions randomly, while the Majority baseline follows the majority label. These baselines serve as simple reference points for comparison. Lastly, we compare our response forecasting results with ChatGPT , a state-of-the-art zero-shot instruction-following large language model (LLM) Yang et al. ( 2023 ) . To predict the sentiment intensity and polarity using ChatGPT, we use the prompt \mathrm{P}_{p} from Section 3.4 that incorporates the user profile, user history, and the news media message as the input. We leverage the official OpenAPI with the gpt-3.5-turbo model 2 2 2 https://platform.openai.com/docs/api-reference/models for sentiment prediction.

To illustrate the effectiveness of Social Prompts (Section 3.4 ), we compare three models: baseline ChatGPT, ChatGPT L , and SocialSense {}_{\text{Zero}} . In ChatGPT L , we incorporate the latent persona User L from Section 3.1 , while in SocialSense {}_{\text{Zero}} , we leverage the aggregated social context User S generated by Social Prompt in addition to User L (Section 3.4 ). We use K=25 for Social Prompt . Similarly, we utilize the prompt \mathrm{P}_{p} for response prediction. The detailed prompts can be found in the Appendix.

Implementation and Environments Our neural models are implemented using Pytorch Paszke et al. ( 2019 ) and Huggingface Transformers Wolf et al. ( 2020 ) . The intensity label in the dataset follows the definition in the SemEval-2018 Task 1 3 3 3 https://competitions.codalab.org/competitions/17751 Mohammad et al. ( 2018 ) , where the sign is also considered during evaluation. More implementation details and discussions of reproducibility and hyperparameters can be found in the Appendix.
Results Discussion
We conduct an evaluation of the proposed SocialSense model and the baseline models introduced in Section 4.2 for the supervised response forecasting task. The evaluation results are presented in Table 1 . While the state-of-the-art models demonstrate competitive performance, SocialSense outperforms all other models across all evaluation metrics consistently. Although ChatGPT is designed and proven effective for zero-shot instruction-following text generation, we observe that its performance in sentiment forecasting of responses is comparatively limited, yielding lower scores compared to the other supervised models. This highlights that the task can not be fully addressed by a zero-shot model alone.

On the other hand, the RoBERTa and DeBERTa models, despite being smaller pre-trained models, exhibit relatively better correlation and F1 scores after fine-tuning for our response prediction task on news articles. However, these models only utilize textual information from news articles and user profiles, disregarding potential interaction patterns and shared beliefs among users. This explains why their correlations and F1 scores are, on average, 10.28\% and 5.99\% lower than those achieved by the proposed SocialSense framework. Additionally, the graph-based InfoVGAE model achieves higher scores compared to the text-based DeBERTa and RoBERTa baselines, highlighting the significance of graph-structured data in enhancing response forecasting performance. However, the evaluation metrics of the InfoVGAE model remain lower than those of SocialSense . While the InfoVGAE model constructs a graph primarily based on user-user and user-news interaction edges, SocialSense goes a step further by inducing and integrating additional belief nodes and edges. This novel approach results in a heterogeneous graph that forges connections among users who share similar perspectives and ideologies, thereby facilitating the learning of intricate social dynamics and bolstering the model’s predictive capabilities.
Ablation Study
We conduct an ablation study on different components of SocialSense to evaluate their impact on performance. The results are presented in Table 1 .

Belief-Centered Graph To assess the effectiveness of the Belief-Centered Graph in Section 3.2 , we conduct an experiment where we removed the belief nodes from the graph, including the nodes representing moral values and human values. This leads to a decrease of 1.91\% in correlations and 4.83\% in F1 scores. These findings support our hypothesis that incorporating belief nodes is effective in modeling the shared beliefs and values among users. By including belief nodes, we enable the graph learning framework to capture the association between the underlying principles and moral frameworks that guide users’ behaviors and response patterns.

User-News Edges In this experiment, we exclude the user-news edges while constructing the belief-augmented heterogeneous graph. The results show that modeling the user-news interaction as edges results in an improvement of up to 6.63% in correlation metrics for sentiment intensity prediction. This indicates that modeling users’ interests and historical interactions with media is crucial for accurately predicting sentiment intensity.

User Profile and Historical Posts The ablation study reveals the important roles of user profile data and historical post data in response forecasting. Excluding user profile data leads to a drop of 1.93\% and 6.32\% on average in the respective tasks, emphasizing its significance in predicting sentiment polarity. Removing historical post data results in a decrease of approximately 4.45\% in correlations and 2.66\% in F1 scores for sentiment polarity prediction. These findings highlight the importance of both data types, with profile data influencing intensity prediction more and historical data affecting polarity prediction more.

Node Initialization Instead of using the text representations of users’ profiles and historical posts, we randomly initialize the node features. This results in a decrease of 3.57\% in correlations and a significant decrease of 8.97\% in F1 scores for polarity classification, emphasizing the significance of text features in predicting sentiment polarity.
Zero-Shot Evaluation
In addition to supervised response forecasting, we also evaluate our framework under the zero-shot setting (Section 3.4 ). The results are presented in Table 2 . Based on the higher scores attained by ChatGPT L , it is evident that the inclusion of latent structured persona information indeed aids the model in comprehending the user more effectively. Furthermore, our model, SocialSense {}_{\text{Zero}} , achieves the highest scores consistently across all metrics. This demonstrates the efficacy of our method for zero-shot social context learning and provides compelling evidence that even in the zero-shot setting, social context plays a crucial role in response forecasting.
Evaluation on Lurker and Unseen User Scenarios
We evaluate the performance of proposed models and baselines on the task of response forecasting for lurker users, who are characterized as users with only a small amount of historical posts. In the experiment, we define the lurkers as the users with less than 50 historical responses (less than 85% of the users in the dataset), and the scenario consequently contains 745 test samples. The scores are shown in Table 3 . Compared to the previous evaluation results in Table 1 , we observe that the overall evaluation scores for all the models are significantly lower. This can be attributed to the fact that lurkers have a much smaller background context, making response prediction more challenging. The lurker case is especially difficult for those baselines relying heavily on historical responses. In this challenging scenario, SocialSense not only achieves significantly higher scores than others in all of the metrics but also maintains its performance on the polarity measures. Specifically, the advantage of our proposed model over DeBERTa and RoBERTa expands from 5.99% to 11.26% in terms of F1 scores for sentiment polarity prediction. These results demonstrate that even in cases where user textual information is extremely limited, our framework can still accurately infer responses, showcasing the robustness of our method. Furthermore, it is worth noting that the intensity score was noticeably lower compared to the regular setting, indicating that predicting the intensity of responses becomes more challenging when historical information is limited. We conduct further evaluation of the proposed model and baselines on unseen users, which refers to the responders who only appear in the evaluation dataset. This case study on unseen users provides insights into the generalization of the models. The evaluation results are presented in Table 3 . The results indicate that the unseen user scenario presents a more challenging task compared to previous settings. Moreover, SocialSense demonstrates significantly higher performance across all metrics compared to other baselines. This outcome underscores the framework’s ability to effectively generalize to unseen users, likely attributed to its robust modeling of the social network and encoding of relationships between users.
Related Work
Existing research has focused on predicting the individual-level response using additional textual features as well as deep neural networks (DNN) Lin and Chen ( 2008 ); Artzi et al. ( 2012 ); Li et al. ( 2019 ); Wang et al. ( 2020 ) . However, these existing methods neglected the important information about users’ personas as well as the modeling of graph-structured interactions among users with the social items. Another line of related works formulates the response forecasting as text-level generation task Yang et al. ( 2019 ); Wu et al. ( 2021 ); Lu et al. ( 2022 ); Wang et al. ( 2021 ) . However, these lack a quantitative measure for analyzing the response (such as in the sentiment dimensions), limiting their applicability in downstream tasks like sentiment prediction on impact evaluation of news Sun et al. ( 2023 ) . In contrast, we propose a novel framework that leverages large language models to induce the graph structure and integrates disentangled social values to forecast responses, whether in a supervised or zero-shot manner. Our work demonstrates that effectively modeling the social context and beliefs of users provides a clear advantage in the social media response forecast task. This can ultimately benefit various downstream applications such as assisting fine-grained claim frame extraction Gangi Reddy et al. ( 2022 ) and situation understanding Reddy et al. ( 2023 ) .

In the field of Social-NLP, related research has focused on applying NLP techniques, large language models (LLM), and prompting strategies to model, analyze, and understand text data generated in social contexts. For instance, progress has been made in misinformation detection Fung et al. ( 2021 ); Wu et al. ( 2022 ); Huang et al. ( 2023b ) and correction Huang et al. ( 2023a ) , propaganda identification Martino et al. ( 2020 ); Oliinyk et al. ( 2020 ); Yoosuf and Yang ( 2019 ) , stance detection Zhang et al. ( 2023 ) , ideology classification Kulkarni et al. ( 2018 ); Kannangara ( 2018 ) , LM detoxification Han et al. ( 2023 ) , norms grounding Fung et al. ( 2023 ) , popularity tracking He et al. ( 2016 ); Chan and King ( 2018 ) , and sentiment analysis Araci ( 2019 ); Liu et al. ( 2012 ); Azzouza et al. ( 2020 ) . The emergence of advanced decoder language models like ChatGPT has led to extensive research on prompting techniques and their application across various NLP tasks Zhou et al. ( 2022 ); Kojima et al. ( 2022 ); Zhao et al. ( 2021 ); Diao et al. ( 2023 ); Sun et al. ( 2022 ) . Indeed, experiments have shown that ChatGPT even outperforms crowd workers in certain annotation tasks Gilardi et al. ( 2023 ) . However, when it comes to social tasks like response forecasting, relying solely on large-scale models without taking into account the social context and users’ personas may not yield optimal performance Li et al. ( 2023 ) . Our experiments demonstrate that incorporating social context in the prompt consistently enhances the LLM’s performance, as showcased in our simulation of information propagation using large language models.
Conclusions and Future Work
In conclusion, we present SocialSense , a framework that utilizes a belief-centered graph, induced by a large language model, to enable automatic response forecasting for news media. Our framework operates on the premise that connecting distant users in social networks facilitates the modeling of implicit communities based on shared beliefs. Through comprehensive evaluations, we demonstrate the superior performance of our framework compared to existing methods, particularly in handling lurker and unseen user scenarios. We also highlight the importance of the different components within the framework. In future research, it would be valuable to explore the application of belief-augmented social networks in other domains and to develop an effective social prompting strategy for general-purpose applications. Furthermore, it is worth investigating how response forecasting models can adapt efficiently to dynamically evolving data, especially given the swift changes observed in real-world social media platforms de Barros et al. ( 2023 ); Cheang et al. ( 2023 ) .
Limitations
While the proposed SocialSense framework demonstrates promising results in response forecasting, there are limitations to consider. Firstly, the performance of the model heavily relies on the quality and availability of social network data. In scenarios where these sources are extremely limited or noisy, the model’s predictive capabilities may be compromised. Additionally, the generalizability of the framework to different domains and cultural contexts needs to be further explored and evaluated.