RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Ethics Statements
The primary objective of this study is to enable content producers to predict the impact of news releases, thereby mitigating the risk of unforeseen negative consequences such as social conflict and moral injury. By providing a stronger and more robust framework for forecasting responses, we aim to contribute to the creation of a safer online environment. In our process of collecting the network data using Twitter API, we strictly adhere to the Twitter API’s Terms of Use 4 4 4 https://developer.twitter.com/en/developer-terms/agreement-and-policy . As part of our commitment to responsible data handling, we will release only an anonymized version of the network data when making the code repository publicly available.
Acknowledgement
This research is based upon work supported in part by U.S. DARPA INCAS Program No. HR001121C0165. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
A.1 Implementation Details
We implement the training framework using the 4.8.2 version of Huggingface Transformer library 5 5 5 https://github.com/huggingface/transformers Wolf et al. ( 2020 ) . For the graph model implementation in Section 3.3 , we use the 2.0.3 version of PyG 6 6 6 https://pytorch-geometric.readthedocs.io/en/latest/index.html . The hyperparameters for the experiment are shown in Table 4 and the ones not listed in the table are set to be default values from the transformer library. We use RAdam Liu et al. ( 2019a ) as the optimizer. We perform greedy hyperparameter search on the gnn_layer from {1,2,3}, learning rate from {5e-5, 1e-4, 5e-4, 1e-3}, # attention heads from {2, 4, 6, 8}, activation from {tanh, relu}, # epochs from {350, 1000}, and node dimensions from {128, 256}. We perform our experiments on a single NVIDIA RTX A6000 48 GB. Our model consists of 10, 484, 424 tuning parameters and it takes less than 30 minutes to fine-tune.
A.2 Analysis of Belief Data
We perform additional analysis on the belief data. Specifically, we show the distribution of the belief data (Figure 4 ), for which the moral value of care is dominant among the users. We have also segregated the model’s performance in sentiment prediction based on the users’ belief values and show it in Table 6 . Empirical results indicate that the model is more accurate when predicting sentiments for users characterized by universalism and degradation. Conversely, the model finds it challenging to predict sentiments for users in the categories of security and stimulation. We further sampled 50 ChatGPT extraction results from user histories and distributed them among three human raters to assess the accuracy of the extracted profiles. These raters are graduate students who qualified through an initial quiz comprising eight samples. On evaluation, the raters assigned an average score of 3.9 out of 5 for accuracy. While not flawless, these extracted beliefs play a significant role in boosting the model’s performance. Such finding indicates that refining the ChatGPT extraction process could potentially lead to enhanced performance outcomes.
A.3 Prompts Templates
We show all prompts used in the work in Figure 5 , Figure 6 , Figure 7 , Figure 8 , and Figure 9 . They represent \mathrm{P}_{l} , \mathrm{P}_{s} , \mathrm{P}_{p} for the baseline ChatGPT, \mathrm{P}_{p} for ChatGPT L , and \mathrm{P}_{p} for SocialSense {}_{\text{Zero}} respectively.
Figure 1: An example illustrating the task. The input consists of user attributes such as the profile and social context together with a news media message. The model is asked to predict response in multiple dimensions.; An example illustrating the task. The input consists of user attributes such as the profile and social context together with a news media message. The model is asked to predict response in multiple dimensions.
Figure 2: The figure illustrates our framework. In the first stage, we use an LLM to extract latent persona from the user’s profile and historical posts. These moral and human value attributes from the latent personas, combined with the social network and news media messages, collectively shape the belief-augmented social network. Graph-based propagation is then used to update user representation. In the zero-shot setting, the LLM itself also assumes the role of an information propagator that combines information from neighbors (more details in Section 3.4 ).; The figure illustrates our framework. In the first stage, we use an LLM to extract latent persona from the user’s profile and historical posts. These moral and human value attributes from the latent personas, combined with the social network and news media messages, collectively shape the belief-augmented social network. Graph-based propagation is then used to update user representation. In the zero-shot setting, the LLM itself also assumes the role of an information propagator that combines information from neighbors (more details in Section 3.4 ).
Figure 3: An example illustrating a snapshot of the belief-centered social network. The latent persona attributes serve as a bridge between (potentially distant) users who share values. The arrow on the top left refers to the response we aim to forecast.; An example illustrating a snapshot of the belief-centered social network. The latent persona attributes serve as a bridge between (potentially distant) users who share values. The arrow on the top left refers to the response we aim to forecast.
Figure 4: Distribution of belief values; Distribution of belief values
Figure 5: Prompt template \mathrm{P}_{l} used for extracting user latent profile User L in Section 3.1 . The input consists of user profile text and concatenated user historical posts. The output contains categories filled answers.; Prompt template \mathrm{P}_{l} used for extracting user latent profile User L in Section 3.1 . The input consists of user profile text and concatenated user historical posts. The output contains categories filled answers.
Figure 6: Prompt template \mathrm{P}_{s} used for aggregating neighbor information in Section 3.4 . It takes a list of latent profiles from neighbors, where the latent profiles are output from \mathrm{P}_{l} .; Prompt template \mathrm{P}_{s} used for aggregating neighbor information in Section 3.4 . It takes a list of latent profiles from neighbors, where the latent profiles are output from \mathrm{P}_{l} .
Figure 7: Prompt template \mathrm{P}_{p} used for predicting responses given only news message, user profile text, and concatenated user historical posts as input. It is used for evaluating the baseline ChatGPT.; Prompt template \mathrm{P}_{p} used for predicting responses given only news message, user profile text, and concatenated user historical posts as input. It is used for evaluating the baseline ChatGPT.
Figure 8: Prompt template \mathrm{P}_{p} using user latent profile in addition to input Figure 7 . It is used for evaluating ChatGPT L .; Prompt template \mathrm{P}_{p} using user latent profile in addition to input Figure 7 . It is used for evaluating ChatGPT L .
Figure 9: Prompt template \mathrm{P}_{p} using aggregated social context User S from Section 3.4 in addition to input Figure 8 . It is used for evaluating SocialSense {}_{\text{Zero}} in the experiment.; Prompt template \mathrm{P}_{p} using aggregated social context User S from Section 3.4 in addition to input Figure 8 . It is used for evaluating SocialSense {}_{\text{Zero}} in the experiment.
Response forecasting results. We report the Spearman and Pearson correlations for the forecasting of sentiment intensity, as well as Micro F1 and Macro F1 scores for the sentiment polarity prediction. The best overall performance is in bold. Our framework outperforms the baselines consistently.
The above Zero-Shot Response forecasting results highlight that the Social Prompt from Section 3.4 consistently offers an advantage.
The case studies for Lurker and Unseen User Scenarios demonstrate that our framework exhibits significantly improved generalization capabilities when the user is unseen or has limited background context.
Hyperparameters
Summary statistics for the original dataset.
Performance segmented by users’ belief values.