Title: Measuring the Effect of Influential Messages on Varying Personas
Summary: The authors examine the task of predicting how social media users will react to a news event. To this aim, they collect a Twitter-based benchmark consisting of (i) headlines, (ii) user information (referred to as "persona"), and (iii) user reactions, represented as sentiment polarity, sentiment intensity, and response text. They define the task of "Response Forecasting on Personas for News Media" as predicting the user reaction from the headline and user information, and they evaluate several LMs on this task.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: The specific application is "measuring the influence of news media messages on viewers by predicting viewers' responses" (p. 555).
Phenomenon defined by authors: Yes
Phenomenon definition: The phenomenon (specifically, application) is defined as "measuring the influence of news media messages on viewers by predicting viewers' responses" (p. 555).
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: The task is defined as predicting sentiment polarity, sentiment intensity, and textual response of an individual when that individual sees a message on news media.
Task item definition: Each item consists of a user persona (i.e., information from user profile and user history), a news headline, and the sentiment polarity, sentiment intensity, and textual response of the user in response to that news headline.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Sentiment polarity and sentiment intensity of items in the training data are LLM-annotated. Sentiment polarity and sentiment intensity of items in the validation and test data are human-annotated.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1,039
Dataset size notes: Train: 10,977; validation: 1,341.
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), Correlation (Matthew's correlation, Pearson's r)
Metric definition detail: Human evaluation looks at three aspects: "persona consistency" (i.e., whether the output reflects the user's characteristics), "label consistency" (i.e., whether the response text and sentiment are consistent with each other), and "context consistency" (i.e., whether the output is responding to the input news headline).
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: The benchmark only comprises Twitter IDs, meaning that the corresponding data cannot be freely accessed anymore.
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding