Can A Society of Generative Agents Simulate Human Behavior and Inform Public Health Policy? A Case Study on Vaccine Hesitancy
Authors: Abe Bohan Hou
Hongru Du
Yichen Wang
Jingyu Zhang
Zixiao Wang
Paul Pu Liang
Daniel Khashabi
Lauren M Gardner
Tianxing He
Venue: COLM 2025 (2025)
Keywords: LLM agent
multi-agent system
social simulation
public health
AI for health
TL;DR: Investigate if a multi LLM agent system can simulate human health behaviors and inform policymaking.

Abstract: Can we simulate a sandbox society with generative agents to model human behavior, thereby reducing the over-reliance on real human trials for assessing public policies? In this work, we investigate the feasibility of simulating health-related decision-making, using vaccine hesitancy, defined as the delay in acceptance or refusal of vaccines despite the availability of vaccination services (MacDonald, 2015), as a case study. To this end, we introduce the VacSim framework with 100 generative agents powered by Large Language Models (LLMs). VacSim simulates vaccine policy outcomes with the following steps: 1) instantiate a population of agents with demographics based on census data; 2) connect the agents via a social network and model vaccine attitudes as a function of social dynamics and disease-related information; 3) design and evaluate various public health interventions aimed at mitigating vaccine hesitancy. To align with real-world results, we also introduce simulation warmup and attitude modulation to adjust agents' attitudes. We propose a series of evaluations to assess the reliability of various LLM simulations. Experiments indicate that models like Llama and Qwen can simulate aspects of human behavior but also highlight real-world alignment challenges, such as inconsistent responses with demographic profiles. This early exploration of LLM-driven simulations is not meant to serve as definitive policy guidance; instead, it serves as a call for action to examine social simulation for policy development.

Published as a conference paper at COLM 2025
Can A Society of Generative Agents Simulate Human Behavior and Inform Public Health Policy? - A Case Study on Vaccine Hesitancy
Abe Bohan Hou Hongru Du ? Yichen Wang Jingyu Zhang*, Zixiao Wang",
Paul Pu Liang Daniel Khashabi Lauren Gardner Tianxing He
Department of Computer Science, Johns Hopkins University
Institute of Interdisciplinary Information Sciences, Tsinghua University
Shanghai Qi Zhi Institute
v Civil & Systems Engineering, Johns Hopkins University
Department of Computer Science, University of Chicago
Department of Epidemiology, Harvard University
MIT Media Lab and Department of EECS, Massachusetts Institute of Technology Corresponding authors are Abe Bohan Hou (bhou4@jhu.edd and lianxing He (heti thetianxing@mail. tsinghua. edu. cn)
Abstract
Can we simulate a sandbox society with generative agents to model human behavior, thereby reducing the over-reliance on real human trials for assess- ing public policies? In this work, we investigate the feasibility of simulating health-related makision-making, using vaccine hesitancy, defined as the delay in acceptance or refusal of vaccines despite the availability of vaccina- tion services (MacDonald, 2015), as case study. To this end, introduce the VACSIM² framework with 100 generative agents powered by Large Lan- guage Models (LLMs). VACSIM simulates vaccine policy outcomes with the following steps: 7) instantiate population of agents with demographics based on census data; 2) connect the agents via a social network and model vaccine attitudes as a function of social dynamics and disease-related infor- mation; 3) design and evaluate various public health interventions aimed at mitigating vaccine hesitancy. To align with real-world results, we also introduce simulation warmup and attitude modulation to adjust agents' attitudes. We propose a series of evaluations to assess the reliability of various LLM simulations. Experiments indicate that models like LI ama and Qwen can simulate aspects of human behavior but also highlight real-world alignment challenges, such as inconsistent responses with demographic profiles. This early exploration of LLM-driven simulations is not meant to serve as definitive policy guidance; instead, it serves as a call for action examine LLM-based social simulation for policy development.
1 Introduction
Recent advances in large language models (LLMs) led to rising interest in building fictional societies in sandbox environments with generative agents (Park al, 2023), i.e., autonomous LLM-ppoered actors that interact with each other and the environment. Notable examples include
simulations of 1000 real people (Park et al., 2024), hospital (Li et al., 2024), and macroeconomic activities (Li et al., 2023). While these simulations show promising potential of generative agents to simulate real-world behaviors, we are interested in understanding how they could assist in developing public policy in real-world settings.
As a case study, we investigate whether a generative multi-agent system can simulate the dynamics of vaccine hesitancy, defined as the "delay in acceptance refusal of vaccines
The code will be released at: https:/ //gi thub. cam/ abehou/VacSim

---

Published as a conference paper at COLM 2025
Demogractic
Roa-WWorld Burvey
I Sumple
LM
Vaccine News Generation
Scanca Provaacsina
Donevate one nows roont
Read news and poliies
Recommendd news
"You ree0
Read Iweels
"You read the poided by other
"Wiie Aesud
varstranns expressing
Post
ecuid you Jake
" vopcite wore
Express attibude
tamographio altbel
Nows
Stance Anti-uatzine
Agont
nenic and
agentic
sprnce on topec"
proesby do
Modbl (L.I) Sioowig molle
Lon Longeage a
PPostnd you ard
Prompt
Which thene bwessm inhentle with?
Social Network Generation
Nare any the aguntae
you
Network
Doskkn
paacy
wdd
lossord.
Tnor liscky fell fallowiing agenie
o0her agents
hemety lawa
(0 10100 2.2.0 0.3.0.0
hettiude deoritution
-e0entta pes
outpur &
Step 1: persona promot)
Step 2: Generate news and build social network of agents.
Propeisiity wiitude
to agents.
Step
Example interactions VecSim simulation step (represents week].
Figure 1: Before simulating, we sample profiles of agents (Step 1) and generate social network and various news (Step 2). We initialize the simulation with corpus of news, vaccine policy, and a network of agents. In one simulation step (Step 3), the agents receive various sources of information and perform actions (marked with rectangles from [I to 4)
despite availability of vaccination services" (MacDonald, 2015), to evaluate its potential in informing public health policy. This issue is prevalent and undermines herd immunity against infectious disease spread and increases exposure to significant health risks. The World Health Organization calls it one of the "ten threats to global health" (WHO, 2019), and it remains top priority for many countries (The White House, 2021; Sallam, 2021).
Policymakers incentivize vaccinations with various policies, such as mandates, financial incentives, and community programs (CDC, 2024; European Centre for Disease Prevention and Control, 2024). However, choosing policy is of high stakes and directly affects the well-being of populations, thus often requiring rigorous quantitative analysis to support (Olson, 1965; Lindblom, 1959). It is expen- sive to conduct social experiments and col- lect longitudinal human data about vaccine attitudes to optimize policy plans (Burger et al., 2022; Fridman et al., 2021), which mo- tivates system that can mimic human tri-
LLaMA-3.1 1-98-instruct
Vavalooe bessadot
Virman
Weeni Handdtte
Striing Handate
Average Hesitanncy Percentage (H)
0.6
Shin ng Ambossndor
Criiing incentiue
als with generative agents.
0.4
To this end, we design a framework for or- chestrating 100 generative agents to simu- late the change of vaccine hesitancy over time in a sandbox society. This framework, VACSIM (in Figure 1), simulates changing vaccine attitudes under different policy sce- narios, allowing us to analyze how genera- tive agents may react to interventions with- out deploying large-scale real-world exper- iments. Rather than claiming generative multi-agent simulations can perfectly mir- ror real policy outcomes, we hope to reveal their promises and limitations, understand how far are from deploying such sys- tems to inform policymaking, and propose attitude modulation (Section 3.1) and sim- ulation warmup (Section 3.3) to bridge that gap. Our paper makes the following contri- butions:
Qwen-2 5-78-Instruct
0.8
0.7
Warmap
Streng sncertive
Strong conbaeearr
Streey Muridate
15
10.3
15.9
10.0
limesteps (Weeks)
Figure 2: Hesitancy trends for strong and weak policies as simulated by Llama (top) and Qwen (bottom). The vertical line represents the simulation warmup cutoff (detailed in Section 3.3). Strong policies generally reduce hesi- tancy (see Eq. 5) more effectively than weak policies, and Llama shows a more obvious sep- aration between strong and weak policies.
We take a pioneering step in applying generative multi-agent systems to model vaccine hesitancy in a public health policy context, enabling future explorations.
We introduce a comprehensive framework for modeling vaccine attitudes and enhance reliability through attitude modulation and simulation warmup.
We extensively evaluate the system's 's soundness and the simulation behaviors of various LLMs both quantitatively and qualitatively, highlighting both opportunities and challenges for real-world alignment.
2

---

Published as a conference paper at COLM 2025
2 Related Work
Traditional agent-based modeling (ABM) has been used to simulate the spread of vaccine hesitancy in populations (Yin et al., 2024; Bhattacharya et al., 2021; Sobkowicz & Sobkowicz, 2021) and more broadly on disease transmission (Kerr et al., 2021; Chopra et al., 2023; Williams et al., 2023). These simulations typically assign agents with fixed sets of attributes as their states and employ an update function revise states based on interactions with other agents and the environment. While useful, traditional ABMs suffer from limitations in agent expressivity, particularly when modeling complex human behaviors like vaccine decision-making (Chopra et al., 2024). To overcome these limitations, our work extends traditional ABMs by leveraging LLMs to simulate agents who freely decide their own states (vaccine attitudes), rather than using a fixed update function, providing open access to the textual responses and thought processes behind their choices.
Building on these efforts to enhance agent expressivity through LLMs, an emerging area of interest is applying LLM-based agents in social simulation, enabling diverse personas, complex interactions, and broader exploration of human-like behaviors. Unlike traditional collaborative tasks, these agents are imbued with diverse personas and placed within simulated environments to emulate human behaviors and interactions. For instance, Park et al. (2023) introduced sandbox village where agents engaged in daily activities, providing insights into social dynamics. Social simulations have been employed to explore various phenomena, including the emergence of social norms (Ren et al., 2024), macroeconomic activities (Li et al., 2023), legal proceedings (Chen et al., 2024), social media (Törnberg al., 2023), mitigating political manipulations (Touzel et al., 2024), educational settings (Zhang et al., 2024), and general-purpor
social simulations (Tang et al., 2024; Piao et al., 2025).
A closely related work done by Chopra et al. (2024) simulates population of 8.6 million agents predict COVID-19 transmission and unemployment rates. Their agents also incorporate an LLM module that allows LLMs to decide on certain actions of the agents (e.g,, whether to follow mask mandate). However, their work focuses on the scalability and expressivity of simulation Chopra et al. (2023)). Our work, in contrast, specifically focuses on developing a generative simulation framework tailored for studying vaccine hesitancy and, importantly, evaluating the potential impact of different vaccine policies.
top of an existing agent-based model (AgentTorch by
3 The VACSIM Framework
Established studies from the Philosophy of Science community conceptualize the epistemol- ogy of agent-based simulation in terms of verification (evaluating the accuracy or internal consistency of the ABM relative to its conceptual design) and validation (examining if the
ABM accurately reflects its purported target) (Oreskes et al., 1994; Šešelja, 2023; Frigg & Reiss, 2019; Mayo-Wilson & Zollman, 2021). We are inspired to develop three key desiderata to guide the development of our simulation:
1. Real-World Alignment (Validation):
The simulation can be controlled to align with real-world policy impacts and has the potential to predict future policy outcomes.
2. Global Consistency (Verification): When altering the parameters of the simulation
(e-g. news sources, policy efforts, etc.), the simulation should behave consistently.
3. Local Consistency (Verification): Each individual agent should behave faithfully according to the context and their demographic backgrounds.
Overview of the Details Building on these principles, the VACSIM framework (Figure 1) simulates how people's vaccination attitudes might evolve within a sandbox society during a pandemic. The framework encompasses: 2) a population of 100 agents with persona (Section 3.1), 2) a simulated news network (Section 3.2), 3) simulated social network (Section 3.2), and 4) a perceived disease risk module (Section 3.2).
The framework initializes agents with persona (as shown by the Step 1 in Figure 1), gen- erates vaccine- and disease-related news as well as agents' social network (see Step 2), and broadcast news, agent-generated tweets, disease risk information, and the applied
3

---

Published as a conference paper at COLM 2025
Persona
"Yeo read the Bulo new poliies
the Aetwning Ovesto puoded wate apanbs"
a' waccin provest donnce waTe
Sonw Collgee
Occopetion
Age: 40
dender:
Ferals
Nons 2i C07 16 Vandine
pedlifed Joarsll
noady labe Ioe loos just yot.
aced son bngthree CONERATT un use cserstatiiny colldiie gei vascralst...
(Teeet uoegy THE
Thoee "Ax
CONYU
Tanar
teeab
and way the wcling 25%
So hood
wachestou? On rigger 1-4 7
connassshig siiush nanyy
got ansonnanl Duppst posr Eoal
deday
probabblh ofete
paad wo with wase calloge wdund on, Trd
concalerale
imppct of coutbii The
afleayy and gowers
Indepardert.
to riste eq sacchurac"
Belgion:
Mertory procides
chnecn prrrect
Meta: Lane
Languago
Model
1a0 reporto) high lnvel
mnakig cowe
Evesnarre Ane
1. CHIOER eaneea
can myalll copando on
doss Let's aodo
1D0. 64, 0.12, 0.00
and polces
BD elylismt"
as3ass (definitely yes)
o Basic Info:
[1
Read news and policies
Read twoets
3
Post tweets
Poll attilud
Figure 3: A detailed qualitative example of an agent's s interactions in VACSIM. The leftmost pane shows the basic information of the agent. The other three panes show how the agent experiences series of actions from [ to 4 in one simulation time step.
vaccination policy under evaluation as shown in Step 3, producing vaccination attitude trajectories over time. In Section 4, we evaluate the simulation across different LLMs under counterfactual vaccine policies (introduced in Section 3.3). We provide a summary of the important notations used throughout the paper in Table 2.
3.1 VACSIM Agent
Agent Setup We follow popular generative agent simulations et al., 2023; Park et al., 2023) to instantiate 100 agents with memory modules. Every agent is prompted with instructions to impersonate a different persona (see Step 1 in Figure 1), which is sampled from the marginal demographic distribution in Nguyen et al. (2022), a COVID-19 vaccine hesitancy survey featuring 13 million responses from the United States from January 2021 to February 2022. The descriptions of demographic attributes are in Table 10 in Appendix D.1 and a detailed example is included in Figure 3.
Simulation Setup period of various weeks. In each time step, agents experience the following, illustrated by actions from Figure 1 and Figure 3:
Each time step in our simulation represents a week, and we simulate a
1. read news about vaccination and the pandemic that is spreading (action I],
2. read the vaccination policy which encourages them to vaccinate (action I))
3. read tweets posted by other agents (action 2),
4. write new tweets that will be read by other agents in the future (action B),
5. output their current attitude towards vaccination (action 4
Memory and Lessons We implement memory module to help agents utilize relevant conversation history (Li et al., 2023; Park et al., 2023). every round, the agent's context only consists of its persona and top K (K=5) most salient lessons, which simulates agents' memory and knowledge about the vaccination and the pandemic. The lessons are generated whenever the agents encounter external materials, i.e. news, policies, or tweets. They reflect upon what they have learned from the materials and summarize them as a brief paragraph of text and alongside the lesson's importance score, real value between 0 and 1. Inspired by prior practices (Park et al., 2023), the saliency is relative measure of how memorable lesson is to the agent, which is calculated based on recency and importance:
n: = a; + 11 d
1
Yi min; Y
(1)
max; 7 min,
where 7/ denotes the saliency of the ith lesson, a; represents the importance of the ith lesson as determined by the agent, A (A₃ 0.995) is the time decay rate of lessons, d is the current time, and d, records the time the lesson was generated. The score 7; is normalized to 7 to make the relative importance of the score appear more intuitive (Zhao et al., 2021).
Vaccine Attitude We follow public health studies (Nguyen et al., 2022) to poll agents' attitudes towards vaccines. The agents are asked: "If vaccine to prevent the disease were
4

---

Published as a conference paper at COLM 2025
offered to you today, would you choose to get vaccinated?" and express opinions on integer scale of 1-4: (1) "No, definitely not", (2) "No, probably not.", (3) "Yes, probably," and (4) "Yes, definitely." " For our simulation, we classify attitudes of 1-2 as vaccine-hesitant and 3-4 as non-hesitant; vaccine policies aim to reduce the proportion of 1-2 scores. We provide sketch of our prompt to solicit agent attitudes in Figure 3 and present fully in Table 19 in Appendix D.3. To aid agents in outputting realistic attitudes, we include in the prompt additional explanations of vaccine hesitancy and their determinants from public health literature in the prompt (see Table 21 in Appendix D.3 (MacDonald, 2015; WHO, 2014; Momplaisir et al., 2021)).
Attitude Modulation When eliciting agents about their attitudes toward policies, their responses typically lean toward their biased observations during pre-training (Borah & Mihalcea, 2024), which may deviate from real-world scenarios (Zhou et al., 2024; Baltaji et al., 2024). Such priors could sharply skew our results to the most common patterns seen during pre-training.
To mitigate this issue, we introduce altitude modulation, which adjusts agents' probability distribution of vaccine attitudes through temperature scaling. This technique is inspired by Xu et al. (2023) and temperature sampling from Boltzmann (1966); Ackley et al. (1985). Attitude modulation adjusts the probability distribution of attitudes while preserving the original relative order of an agent's ss attitude preferences. Instead of prompting agents to directly output single integer scores to reflect their attitudes, we ask them to output probability distribution in textual format of how likely they will have such attitudes. For instance, if an agent is pro-vaccine, meaning that the agent is more likely to have attitudes of 3 and 4, the output distribution can be [0.1, 0.1, 0.35, 0.45], where the indices correspond to attitudes and the floats represent their probability. We denote the output distribution as P and formulate the effects of temperature
clog(Pi)TT
p!/T
P
(2)
clog(P.//T
EL P!/T
where Pf is the i-th element in and the original probability of the ith attitude and P. is its probability after temperature scaling with the modulating temperature T. We set the modulating temperature as the only hyperparameter we adjust in the simulation and perform grid search for the temperatur that results in the closest simulation of reality (see Section 4.1). We include a concrete example of attitude modulation in action of Figure 3.
3.2 Simulation Framework
News Network Since agents initially know little about the imminent pandemic in the simulation and its vaccines, we simulate news network to educate agents. After reading the news, agents reflect upon what they have learned and produce lessons (introduced in Section 3.1) along with their importance scores.
The news corpus (10K pieces, each around 250 tokens) is generated prior to the simulation, using Llama-3. 1-8B-Instruct (see Step 2 in Figure 1). The news is classified based on whether it states (1) benefits/ / (2) concerns of vaccines or depicts how daily life gets (3) little / (4) considerably disrupted by the disease. Types (1) and (4) (denoted as Npos) encourage vaccinations because they either show necessity or benefits of vaccines. Types (2) and (3) are seen as discouraging vaccinations, denoted as Nneg. We generate each news type with stance-specific prompts (see full prompts in Table 15 (Appendix D.3) and generated examples in Table 26 (Appendix D.6)). We select 20 real-world news pieces as in-context examples and a temperature of 1.5 to enhance generation diversity. Each agent reads news from a recommender (see [[] in Figure and Figure 3), which promotes based on the recommendation score S, of news piece (v):
Sy MaxSim (T),))
(3)
where MaxSim (Khattab & Zaharia, 2020) takes the maximum cosine similarity between sentence embeddings of v and any past tweet (r²) from the agent. We use a SOTA sentence transformer all all-MiniLMM-6--1 to obtain the embeddings (Reimers & Gurevych, 2019). At
5

---

Published as a conference paper at COLM 2025
each time step, the recommender receives K2 news and ranks the top K (K = 3) news according to their scores. This parameter combination is chosen for computational efficiency.
Social Network We follow Chang et al. (2024a)'s local prompting method to generate social network among agents (illustrated in the Step 2 of Figure 1 and the prompt design in Table 16 (Appendix D.3)). Agents are asked whether they want to follow each other based on their profiles. When an agent reads "tweets" " (a proxy term for social network messaging between agents) posted by other agents whom they follow, a positive bias will be added to increase its visibility. Similar to Eq. 3, the recommendation score s - for the kth tweet from agent 91 to agent 92 is calculated by:
5 92
MaxSim (T - x 27 d: + TT -992
(4)
where T is the embedding of the kth tweet by 91, A, (A, 0.9) is the time decay factor for the score, d is the current time and d: is the time when the kth tweet was posted, 4n -222 is the indicator function of whether 91 follows 92, and Φ (Φ 0.3) is the positive bias to increase tweet visibility in the social connection. After the agent reads tweets per time step, it is then asked to post tweet expressing its thoughts on vaccine-related topics. Newly posted tweets will be stored and recommended to other agents in future time steps.
Perceived Disease Risk Module In addition to news and tweets, we simulate vaccine attitudes under the influence of perceived disease risk, using the COVID-19 emergency department (ED) visit rate as a proxy to reflect the current level of health threat. The rate is broadcasted to the agents throughout the simulation (see an example in [I of Figure 3). We use CDC data? from January 2021 to February 2022 for consistency.
3.3 Vaccine Policy
We experiment with three kinds of popular vaccine policies: financial incentives (p1), am- bassador programs (F2), and vaccine mandates (pa) from the CDC-recommended strategies (CDC, 2021). Additionally, we characterize each category with policy effort (Ancona et al., 2022), the strength with which the policy is applied. We set two effort levels: weak and strong (See an example in Figure 3). For instance, weak and strong financial incentives could be $10 and $50. See full descriptions in Table 11 in Appendix D.2.
At the end of length-l. simulation with policy p, news N, and random seed r, we define N,r) to be the vaccine hesitancy percentage at the end of simulation, where typically L 20. Due to simulation's stochasticity, we take the average of the last 3 steps when computing the end hesitancy H2, of a single run. We report the mean of 5 runs with different random seeds, with the notation Hj, (P,N) N) indicate the averaged value. We are also interested in the effect of certain policy p, i.e., the difference of hesitancy with and without the policy:
(5) where Po means no policy is applied. A larger AH means the policy has a larger effect on hesitancy reduction. We will evaluate policy effect with AH, (p, N) in Section 4.2.
AHr(p,N) = (L
(p,
Simulation Warmup At the start of the simulation, we record agents' initial attitudes
when they have not yet learned about vaccines from news, tweets, and interventions by policies. Preliminary experiments show that their attitudes would experience drastic change immediately after the simulation starts. Thus, we devise warmup stage: the starting stage of each run, we apply the policy intervention only after W = 5 time steps. This also aligns with real-life experience, as a policy is often proposed some time after the breakout of disease.
4 Evaluation
Preliminary explorations show that various LLMs simulate distinct behaviors and vaccine attitudes, motivating us to investigate and determine models most suitable for our sim- ulation. Following desiderata proposed in Section 3, we evaluate the suitability of each
2nttps://covid.c wwtpooiiddddd..ooooooo wtt..ooo/cowidddatttt.ooooooo tracker

---

Published as a conference paper at COLM 2025
(P1) Attitude Moduletion
-0
(P2) Hesitancy Reduction with Variius Efffort Levels of the Incentive Policy
00
Nestorny Reduction
L0
AT
3.60
Usme
4.20
3.60
0.29
2.50
3.20
0.20
Niflernce
thi
2.80
0.60
4.50
2.20
3.20
4.20
Vian
(P3)
Mesilancy Reduction with Difierent News
3.m
Lime- 5 senuap Lama-3.1a0 Owan 5-70
(P4) Resporse Quelity Rated by GPT-40
GFT-p
Cleyze & Sattlle Saiimiii
1.00 0.60
3.00
3.00
3.75
3.b
4.62
4:55
urea
3,86
4.55
Uanta
danda saaile
23.60
24 50 21 20 49:40
10.00
9.50
4.50
4.67
4,95
00.93
Usrra
0.84
Reggin Nows
Mocultting Temperature
6.5 0.3
1.5
2.0
Avaragu Husiuasyy Datuction (AM)
Plenay Convenaticn Evoluation Catoggr lcs
Figure 4: Evaluation of initial real-world alignment (P1, Section 4.1), global consistency (P2, P3, Section 4.2), local consistency (P4, Section 4.3). All models shown are the instruct versions. Llama-3-8B-AB (Llama-3 3--8---8--------- terated-v3) (Arditi al., 2024) is an uncensored model finetuned on Llama-3- 3-mm--3-B--- IB--rntt via bypassing refusals to safety-related questions.
LLM as the backbone of VACSIM in terms of initial real-world alignment, global and local consistency. We show the list of tested models in Figure 4.
4.1 Initial Real-World Alignment
We test if our simulations can be controlled via attitude modulation (see Eq. 2) to align with the initial hesitancy level in real-world data from the Delphi survey (Nguyen et al., 2022). We search for the modulating temperature over {0.1, 0.5, 0.7, 1.0, 1.5, 2.0) for each LLM simulation. We run over 5 random seeds and record in P1 of Figure 4 the mean error between simulated hesitancy after W 5 warmup steps (i.e. Hw (Po, N)) and the initial real-world hesitancy (45%). Positive mean error indicates that models are over-hesitant compared to the real-world behaviors, and negative mean error shows models are over-confident. The modulating temperature that minimizes the mean absolute error is considered optimal (i.e. as close as possible to zero). The best models in terms of initial real-world alignment are Llama-3.1 Phi, and Qwen, which has minimum mean errors of 0.6. Notably, find it hard to align Llama-3. 3. 2 and GPT-4o mini, with over 20% difference with the real-world initial hesitancy. This is potentially caused by biases towards vaccine acceptance acquired from training data, showing that both models are unsuitable for evaluations under our setup.
4.2 Global Consistency
Fixing the optimal temperature for each simulation, we check how consistently the simula- tion behaves when there are changes in the input parameters. For our parameter choices, we alter (1) effort levels of policy: we expect that strong incentive would reduce hesitancy more effectively than the weak variation and (2) news stances on vaccination: we expect pro-vaccine news would reduce hesitancy more effectively than the weak variation.
Altering effort levels of policies We alter the effort level of input policy and compare the
difference between hesitancy reduction (i.e. AH (p; strong N) - AHL ( (p,weakk .N) E (1,2,3) from Eq. 5) We run W = 5 warmup steps and 15 steps of policy-intervened simulation over 5 seeds with all 3 policy types. The policy effect difference between weak and strong policies is shown in P2 of Figure and fully in Figure 6 (Appendix C). We discover that several models can distinguish the effect of weak and strong policies by at least 2% difference: Qwen-2.5, Llama-3, -llma-3-AB and Llama-3.1 1. Haiku, GPT-4o, and Phi can also recognize difference between certain weak and strong policies but result in negative hesitancy reduction. Thus, remove them from subsequent evaluations. We discuss why certain models may fail in Section 5.
Altering news stances We also study the effect of altering the ratio of pro and anti-vaccine news in the simulation. We run simulation for L 20 weeks with all pro and anti-vaccine
7

---

Published as a conference paper at COLM 2025
news and compare the difference in average hesitancy reduction under no policy (i.e. AH₁, (po, Nneg) - AHr, (po, Npos) from Eq. 5), with results presented in P3 of Figure 4. We find that all models reduce hesitancy with positive news and increase hesitancy with negative news, meaning that they can recognize the effect of stances on vaccine hesitancy.
4.3 Local Consistency
We further evaluate how faithful individual agent behavior is, concretely in three aspects: (1) attitude: whether the expressed vaccine attitudes are consistent with persona and memories (2) memory: whether the generated reflections and assigned importance scores correspond to persona, and (3) conversation: whether the generated tweets are faithful to memories and contexts. We have GPT-4o judge response quality on a scale of 1-5 (denoted as Response Quality Score in P4 of Figure 4), with detailed descriptions of the criteria (see Appendix D.4). We randomly sample from 25 agents, each with 10 episodes per evaluation category under each strong policy. The average ratings are in P4 of Figure 4.
We observe that Llama-3. 1 and Qwen-2.5 5 excel in response quality. All models perform similarly well across memory and conversation categories and differ attitude faithfulness. Examining reasons given by the judge LLM (see examples in Table 27 in Appendix D.6), we hypothesize that a high attitude modulation temperature can account for certain models' unfaithful attitudes, since higher temperature scaling would induce the distribution to be more uniform, thus more likely to sample attitudes that do not represent the persona and the given context, limiting the consistency of attitudinal changes.
4.4 Comparison with Real-World Data
Since Llama-3.1 passes various global and local consistency checks, we run simulations with it for an extended period of L 35 weeks under various strong policies and compare their similarities with a real-world trajectory from the Delphi survey (Nguyen et al., 2022) in Figure 5. We show that simulated curve approximates the real-world trajectory with 2.82% Mean Absolute Error (MAE). While this does not serve as conclusive evidence for the pratical applicability of VACSIM, we show that the simulation can be controlled to align with some real-world trajectories and has the potential to be deployed in future work. Simulated curves with Llama-3.1 1 and Qwen-2.5 5 are in Figure 2.
4.5
VACSIM vs. Human Experts
To compare with human experts' perceptions of potential policies, we conduct an anony- mous survey, where 18 university researchers, (each holding at least a master's degree in fields relevant to public health or policy- related research) are asked to rank six de- signed policies (see Table 11 in Appendix D.2) on reducing vaccine hesitancy, from (most impactful) to 6 (least impactful) based on fea- sibility, acceptability, and expected outcomes, with the permission of tied rankings. Rank- ings are aggregated using the Borda count method (Saari, 1985). We also generate sim- ulation rankings based on their average hesi- tancy reduction under different policies (see rankings in Table 25 in Appendix D.5). We compute Kendall's (Sen, 1968) to measure agreement level between simulation and expert rankings and also perform hypothesis tests of to determine the agreement significance. Results (Table 1) show that Llama-3. 1-8B- Instructt (I 0.733) and Qwen-2. 2- 1 (I 0.690) exhibit the highest alignment with expert rankings, with small p-values (0.056) indicating significance.
Average Hesitancy Percentage (H]
Ilesitancy Trends Over Time, Simulated with Ulma-33 L 1-80-nntrrcct
0.45
Strong Incentive (MMC-1.00%)
Strong Ambassador (MAE-4.20%)
Strong Mandete (MAE-2 82%)
Real Data
0.35
0.30
0.25
0.20
20
25
Weeles
Figure 5: An extended hesitancy curve sim- ulated with Llama-3.1 under various inter- ventions. The best simulation has a MAE of 2.82% compared to real hesitancy data from the Delphi survey (Nguyen et al., 2022).
Usually a mixture of policies is enforced, unlike our single policy setting.
8

---

Published as a conference paper at COLM 2025
Model
Kendall's Tau (T)
p-value
Llama-3-88 Instructt
0.333
0.469
(aama-2-00-bbliteaaree
0.276
0.444
Insttttttucct Qwwen22.577-Inntruucc
0.733
0.056
0.690
0.056
Table 1: Kendall's Tau (T) correlation coefficients comparing model rankings with expert rankings. Higher values indicate stronger agreement with expert preferences. The null hypothesis for testing T is the absence association, i.e. T T=
4.6 Qualitative Analysis
key strength of generative social simulation is its ability to examine individual agents' decisions, improving trust in the simulation and helping policymakers understand reactions to policies, motivating our qualitative analysis. For each kind of strong policy, we sample attitude-related conversations from 25 Llama-3. agents under three random seeds. GITT-4o used summarize each agent's s attitude changes (see prompts in Table and an example output in Table 9 (Appendix C)) and to aggregate individual analyses to identify patterns in attitude shifts (see prompts in Table 5 and results in Table and Table (Appendix C)).
The analysis highlights three major barriers to policy effectiveness: lack of trust in institu- tions, insufficient incentives, and conflicting information from social media and news. Generally, government policies, credible news, and scientific information boost vaccine confidence. For instance, under the weak financial incentive policy, agent No. 3 increases confidence after learning about the financial incentive and vaccine-related information:
"I am a 63-year-old Christian man with initial vaccine hesitancy, but after learning about the vaccine's benefits and risks, I have become more confident in its effec- tiveness and safety. I also consider the government's policy of offering a $10 cash card to adults who receive their first dose, so I will probably get vaccinated."
By contrast, some agents frequently shift their vaccine attitudes. Agent No. 51 oscillates between an anti-vaccine stance (attitude distribution: [0. 0,0. 26, 0.02, 0.000..22. 0.02]) and more reconcile con-
neutral attitude ([0.2, 0, 3, 0. 3,0. 2]) from week 14 to 17. She struggles flicting knowledge, including pro-vaccine arguments (e.g., "$10 cash card incentives and rising hospitalizations for COVID-19") and anti-vaccine concerns (e.g,, "concerns about safety and efficacy")). Finally, some agents remain consistently hesitant despite exposure to pro-vaccine information. Agent No. 26, for instance, maintains her skepticism due to government distrust and vaccine safety concerns. We include all archetypical examples in Table 3 and 4 (Appendix C). These simulations enhance transparency and understanding of health-related social dynamics, offering valuable insights for policymakers.
5 Discussion and Conclusion
This research aims evaluate the potential of generative multi-agent systems in simulating human behavior for public policy decision-making, focusing on vaccine hesitancy. Through the development of the VACSIM framework and comprehensive evaluation protocol, we highlight both the promise and challenges of this approach, providing insights into the ability of different LLMs model complex social phenomena.
Potentials of generative agents in public policy decision-making, Our results show that certain LLMs (e.g. Qwen-2. and Llama-3. 1) capture nuanced influences from demographic, social network, and policies. These models pass global and local consistency checks, high- lighting their potentials in modeling policy effects while reducing reliance on human trials.
Variability of LLMs in policy simulations. Despite the promise, simulation success varies across different LLMs. Models such as Haiku and Phi reveal inconsistencies that compromise simulation desiderata. There temperature: Haiku, GPT-4o, and Llama-3. all have extreme temperatures for initial real- world alignment and high MAEs compared the reference, which makes sampling attitudes
two likely factors: (1) extreme attitude modulation

---

Published as a conference paper at COLM 2025
more stochastic and less representative of their preferences. This may cause systematic cascading influence and affect the simulation accuracy. (2) sensitivity to prompting: Since preliminary tests show that language models after safety tuning may exhibit biases towards favoring vaccination and become easily convinced by pro-vaccine sources, we develop extensive prompts to instruct models to become stubborn when necessary (Table 19 in Appendix D.3). Variations in model sentivity to prompts can skew outcomes.
Implications for future work. Future work can focus on building controllable social agents with minimal prompting. In addition, principled training of social agent inside the simuulation to improve human aligmment will be critical. Moreover, understanding and enabling the emergent phenomena in social simulation can greatly benefit the policy community.
Ethics Statement
This research investigates the use of generative agents to simulate health-related decision- making and evaluate the impact of public policies. While these simulations offer promise for policy exploration, we recognize several critical ethical considerations. First, there is inherent risk misalignment between simulated and real-world behaviors. Although LLM agents can produce plausible reasoning and attitudes, they do not possess lived experience or emotional nuance. As such, simulation outputs should not be interpreted as definitive forecasts in high-stakes policy contexts. Second, while agents are initialized with synthetic demographic profiles based on real-world data, these samples may not fully capture the diversity and complexity of the real world. As result, the simulation may overgeneralize behaviors, particularly for underrepresented or vulnerable groups. Third, the language models powering these agents are trained on large-scale web corpora that may encode societal biases, misinformation, and unequal representations of race, gender, religion, and political identity. Although we implement evaluation safeguards and attitude modulation techniques to improve realism, acknowledge that these models may still reflect distorted or one-sided worldviews. We emphasize that our work is intended as a methodological investigation, not as direct policy advice. Continued interdisciplinary oversight, transparent reporting, and collaboration with domain experts are essential to ensure that generative agent simulations used responsibly and equitably.
Acknowledgement
We thank Johns Hopkins University CLSP, Tsinghua University IIIS, and Shanghai Qi Zhi Institute for the computing resources and support. We are also grateful to the students and faculties who offered valuable feedback on this project, particularly Benjamin Van Durme, Tara Kirk Sell, Ziang Xiao, Marc Marone, Nikhil Sharma, Mark Dredze, and Jeffery Cheng.
10

---

Published as a conference paper at COLM 2025
References
David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algo- rithm for boltzmann machines. Cognitive Science, 9(1):147-169 1985. doi: 10.1207/ $15516709cog09017.
Camilla Ancona, Francesco Lo Tudice, Franco Garofalo, and Pietro De Lellis. A model- based opinion dynamics approach to tackle vaccine hesitancy. Scientific Reports, 12(1): 1-12, 2022. doi: 10.1038/ s411588-022-5008-- URL 541598-922- 15002-0150200..0 pdf.
10.1038/441598-022-111
com/articles/
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by single direction. Xiv, 2024.
URL https: wwwpaaawwi...../aeaeeeee
11717.
Razan Baltaji, Babak Hemmatian, and Lav R. Varshney. Conformity, confabulation, and impersonation: Persona inconstancy in multi-agent lim collaboration. Preprint on Xiv, 2024. URL / http:a/wwww....r////////./// 03862.
Parantapa Bhattacharya, Dustin Machi, Jiangzhuo Chen, Stefan Hoops, Bryan Lewis, Hen- ning Mortveit, Srinivasan Venkatramanan, Mandy L Wilson, Achla Marathe, Przemyslaw Porebski, et al. Ai-driven agent-based models to study the role of vaccine acceptance in controlling covid-19 spread in the us. In 2021 TEEE International Conference on Big Data (Big Data), pp- 1566-1574 IEEE, 2021.
Ludwig Boltzmann. On the relation between the second law of the mechanical theory of heat and probability calculations regarding the conditions for thermal equilibrium. Reprint *Kinetic Theory*, ed. Stephen G. Brush, pp. 188-193, 1966. URL https:///oi. org/ 10. 1016/8978-1110 0-08-00 01 00928-8.2.. 8. 50017-2.
Angana Borah and Rada Mihalcea. Towards implicit bias detection and mitigation in multi-agent LLM interactions. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 9306- 9326, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10.18653. /1/0555/v//// v1/2024.2iiiiiie 10024..inding--emllll londings-emnlle 545/.
URL https://aclanthhloogy org/2024.
Ronelle Burger, Tim Köhler, Aleksandra M. Golos, Alison M. Buttenheim, René English, Michèle Tameris, and Brendan G. Maaghhn-Brown Longitudinal changes in covid-19 vaccination intent among south african adults: evidence from the nids-cram panel survey, february to may 2021. BMC Public Health, 22, 2022. URL https: /aaii. semanticscholar. org/CorpusID: 247171824.
CDC. COVID-19 Vaccination Field Guide: 12 Strategies for Your Community, 2021. URL https://archinn cdc. vaccination-strategies pdf. Accessed: 2025-01-21.
CDC. Vaccinate with confidence, 2024. URL https: / http://wwwwcccc...o www.:.dd.go//wwaeeee.. hatd.//o///aaccnnss/ovt-t-11
seecinate-with-confiiee html. Accessed: 2024-12-26.
Serina Chang, Alicja Chaszczewicz, Emma Wang, Maya Josifovska, Emma Pierson, and Jure Leskovec. Llms generate structurally realistic social networks but overestimate political homophily. ArXiv, abs/2408.166222 2024a. URL https: //api. semanticscholar
org/CorpusID::722 2721146117.
Serina Chang, Adam Fourney, and Eric Horvitz, Measuring vaccination coverage and concerns of vaccine holdouts from web search logs. Nature Communications, 15(1):6496 2024b.
Guhong Chen, Liyang Fan, Zihan Gong, Nan Xie, Zixuan Li, Ziqiang Liu, Chengming Li, Qiang Qu, Shiwen Ni, and Min Yang. Agentcourt: Simulating court with adversarial evolvable lawyer agents. arXiv preprint uxXiv:240S.0808 2024. URL https:/ http:::://xxvvoor.g./ abs/2408. 08089.
11

---

Published as a conference paper at COLM 2025
Ayush Chopra, Alexander Rodriguez, B. Aditya Prakash, and Ramesh Raskar. Using neural networks to calibrate agent based models enables improved regional evidence for vaccine strategy and policy. Vaccine, 41(42):6255-6266 2023. doi: 10.10166j.vaccime2223...... URL htthtttp///ooi.... /ttp::///ooooo/..../.o/// 1016/j j. vaccine. 2023.00 .9.009.0001
Ayush Chopra, Shashank Kumar, Nurullah Giray-Kuru, Ramesh Raskar, and Arnau Quera- Bofarull. On the limits of agency in agent-based models, 2024. URL abs/2409. 10568
Ensheng Dong, Kristen Nixon, and Lauren M Gardner. A population level study on the determinants of covid-19 vaccination rates at the us county level. Scientific Reports, 14(1): 2277,2022
European Centre for Disease Prevention and Control. Vaccine hesitancy, 2024. URL https: //www. www.. opa. eu/en/
tancy.. Accessed: 2024-
12-26.
Ariel Fridman, Rachel Gershon, and Ayelet Gneezy. Covid-19 and vaccine hesitancy: A longitudinal study. PLoS ONE, 16, 2021. URL https://api. semantccscholar org/
CorpusID: 233279364.
Roman Frigg and Julian Reiss. Computer simulations in science. The Stanford Encyclopedia of Philosophy, 2019. URL https://plate: stanford. wwt.//ntrrie/s/immulttttttttttte https////ssiittiiiiiiooseeeeeeee science/ Accessed: 2025-03-26.
Ankit Garg, Harish Kumar, Ankit Gupta, and Gaurav Pandey. Covid-19 vaccine tweets sen- timent analysis and topic modelling using transformers. 2021 5th International Conference on Intelligent Computing and Control Systems (ICICCS), pp. 1335-1342, 2022. doi: 10.1109/ URL https::/ieeexpploeee/
ICICCS51141.22 20114.......7211111
ieee. beaeeeerrg//cmmmnn////9//00
Cliff C Kerr, Robyn M Stuart, Dina Mistry, Romesh G Abeysuriya, Gregory R Hart, Kather- ine Rosenfeld, Prashanth Selvaraj, Rafael C Nunez, Benjamín Hagedorn, Leah George, et al. Covasim: agent-based model of covid-19 dynamics and interventions. PLoS computational biology, 17()):e0009149, 2021.
Omar Khattab and Matei Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In ACM Special Inferest Group on Information Retreival (SIGIR), pp- 39-48. ACM, 2020.
Junkai Li, Siyu Wang, Meng Zhang, Weitao Li, Yunghwei Lai, Xinhui Kang, Weizhi Ma, and Yang Liu. Agent hospital: A simulacrum of hospital with evolvable medical agents, 2024. URL
02957.
Nian Li, Chen Gao, Mingyu Li, Yong Li, and Qingmin Liao. Econagent: Large language modee-empowered agents for simulating macroeconomic activities. In Annual Meeting of the Association for Computational Linguistics, 2023. URL httpp:///ap semanticscholar. org/CorpusID: 264144527..
Charles E. Lindblom. The science of 'muddling through'. Public Administration Review, 19 (2):79-88, 1959.
E MacDonald. Vaccine hesitancy: Definition, scope and determinants. 2015. URL https: wtt.aaaattttccccoooooooooooooooo s/api.semanticsccol... wrtp//oopssI:: 17626311..
Conor Mayo-Wilson and Kevin J. S. Zollman. The computational philosophy: Simulation as a core philosophical method. Synthese, 199(122)3347-3677, 2021. doi: 10.1007 110.11100/11122000-020 12002222222000222---- URL https://link. str.inkspringge.commmrrrr wtt.//ariiceeeeee.1100. 000100000022903...
Florence M. Momplaisir, Barbara Kuter, Fatemeh Ghadimi, Safa Browne, Hervette Nkwi- horeze, Kristen A. Feemster, lan Frank, Walter Faig, Angela K. Shen, Paul A. Offit, and Judith Green-McKenzie, health care workers in 2 large academic hospitals. JAMA Network Open, 2(1):12,, URL https: //wc.//ccccc////// ncbi. nlm. nih. gov/
docial/ethnice differences in covid-19 vaccine hesitancy among
2021. doi: 10.1001 h0..D0///aamaeetoooooee......... articles/PMC3446677//
12

---

Published as a conference paper at COLM 2025
Johannes Müller, Aurelien Tellier, and Michael Kurschilgen. A model of opinion dynamics with echo chambers explains the spatial distribution vaccine hesitancy. arXio preprint #rXiv:2112.102300 2021. URL
10230.
Johannes Müller, Aurelien Tellier, and Michael Kurschilgen. A model-based opinion dynam- ics approach to tackle vaccine hesitancy. Scientific Reports, 12(1):1-12, 2022. doi: /0.1038/ 541598-022-15062- URL https://wwww natture.coo//rttille//ss111333 00205-22-0--000---2---
Quynh C. Nguyen, Isha Yardi, Francia Ximena Marin Gutierrez, Heran Mane, and Xiaohe Yue. Leveraging 13 million responses to the facebook covid-19 trends and impact survey to examine vaccine hesitancy, vaccination, and mask wearing, january 2021-february 2022. Research Square, 2022. URL htttps://app semant commanicscholal www.oooarrrrrr/oroeeeeeeoooeeeee wtt./ooppsII 249977331..
Mancur Olson. The Logic of Collective Action: Public Goods and the Theory of Groups. Harvard University Press, 1965.
Naomi Oreskes, Kristin Shrader-Frechette and Kenneth Belitz. Verification, validation, and confirmation of numerical models in the earth sciences. Science, (63(5147)6611-64)) 1994. doi: 10.1126/scienr /ttss/ccc....1111/.... 263.5147.641. URL https://ww. wrw.ciienne.oggdo///...... science. 263. 5147.617 641.
Joon Sung Park, Carolyn Q. Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Mered- ith Ringel Morris, Robb Willer, Percy Liang, and Michael S. Bernstein. Generative agent simulations of 1,000 people, 2024. URL https: / /ttt:::///rxivoorr/aa////// 1011. 10109.
Joon Sung Park et al. Generative agents: Interactive simulacra of human behavior. Proceed- ings of CHI 2023, 2023.
Jinghua Piao, Yuwei Yan, Jun Zhang, Nian Li, Junbo Yan, Xiaochong Lan, Zhihong Lu, Zhiheng Zheng, Jing Yi Wang, Di Zhou, Chen Gao, Fengli Xu, Fang Zhang, Ke Rong, Jun Su, and Yong Li. Agentsociety: Large-scale simulation of llm-driven generative agents advances understanding of human behaviors and society. arXiv preprint ar arXiv:2502.8869) 2025. URL https: //axxvv www.aaiwww...iraaaeeeeeeaoee orgg/br////2/.... 08691.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arv preprint ar Xix::1908.11008 2019.
Siyue Ren, Zhiyao Cui, Ruiqi Song, Zhen Wang, and Shuyue Hu. Emergence of social norms in generative agent societies: Principles and architecture, 2024. URL httpstt/////rxxv.orrg abs/2403.0810
Donald Saari. The optimal ranking method is the borda count. Technical report, Discussion paper, 1985.
Malik Sallam. Covid-19 vaccine hesitancy worldwide: A concise systematic review of vaccine acceptance rates. Vaccines, 9, 2021. URL https://api. stt./ppiiismmaiicccooooorarr .rg/ CorpuSID:23208604B. 232086048.
Pranab Kumar Sen. Estimates of the regression coefficient based on kendall's tau. Journal of the American statistical association, 63(324):1379-1388) 1968.
Liora Shmueli. Predicting intention to receive covid-19 vaccine among the general population using the health belief model and the theory planned behavior model. BMC Public Health, 21(1):804, 2021. doi: 10.1186/ 1o.1e......1e88500000 128010021172 URL https: //bmcpubbicchealth biomedcentral.
Pawel Sobkowicz and Antoni Sobkowicz. Agent based model of anti-vaccination move- ments: Simulations and comparison with empirical data. Vaccines, 98):809, 2021. doi:
10.3390/ vaccines9080s09, URL https: htwp://wwwww/cttc www.. www.commmmmcc /ttppp/..ccc/200033333///////// 393X/9/8/889
Jiakai Tang, Heyang Gao, Xuchen Pan, Lei Wang, Haoran Tan, Dawei Gao, Yushuo Chen, Xu Chen, Yankai Lin, Yaliang Li, Bolin Ding, Jingren Zhou, Jun Wang, and Ji-Rong Wen. Gensim: A general social simulation platform with large language model based agents. ar Xiv preprint ar arXit:2410.04361 2024. URL https: //axxiv. orgg/bbs/241 00 04360.
13

---

Published as a conference paper at COLM 2025
Shasha Teng, Nan Jiang, and Kok Wei Khong. Using big data to understand the online ecology covid-19 vaccination hesitancy. Humanities and Social Sciences Communications, (1):158, 2022. doi: 10.1057 (157///11999-02--11866666 URL https: /ttps.//www..ee wwwwwe.nttre..mm com/articles/ $41599-022-01188
The White House. Fact sheet: Biden administration announces historic 10 billion investment to expand access covid-19 vaccines and build vaccine confidence in hardest-hit and highest-risk communities. 2021.
Petter Törnberg, Diliara Valeeva, Justus Uitermark, and Christopher Bail. Simulating social media using large language models to evaluate alternative news feed algorithms. Xiv preprint ar Xio:2310.05984 2023. URL https://arxiv http://aaawxvvoorab//////a org/abs/ 239.00998...
Maximilian Puelma Touzel, Sneheel Sarangi, Austin Welch, Gayatri Krishnakumar, Dan Zhao, Zachary Yang, Hao Yu, Ethan Kosak-Hine, Tom Gibbs, Andreea Musulan, Camille Thibault, Büşra Tugce Gürbüz, Reihaneh Rabbany, Jean-François Godbout, and Kellin Pelrine. A simulation system towards solving societal-scale manipulation. arXiv preprint 1rXiv:2410.13915, 2024 URL /ttpps://arxi.... org/ abs/2410. 13915.
Hanyin Wang, Meghan Hutch, Yikuan Li, Adrienne S. Kline, Sebastian Otero, Leena Mithal, Emily S. Miller, Andrew Naidech, and Yuan Luo. Deep learning reveals patterns of diverse and changing sentiments towards covid-19 vaccines based on 11 million tweets. Xiv preprint ar wxxiv22077116441 2022. URL https://arree https://araa..or//bbb/222
10641.
WHO. Report of the SAGE Working Group on Vaccine Hesitancy, Novem- ber 2014. URL otto:://///tttt//////////sssssss http://ccieeeeeeciiiieeeeeieeeee oorkingggroo.rrrrssss.rrrr....cn
aat.ealls//eeasaaee.e.. pdf. Accessed Jan 1st, 2025.
WHO. Ten threats global health in 2019. 2019. URL https://www who int/news-twnnnnnnn room/
Ross Williams, Niyousha Hosseinichimeh, Aritra Majumdar, and Navid Ghaffarzadegan. Epidemic modeling with generative agents. arXiv preprint XiXi:22377,0986 2023.
Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game. ArXiv, abs/2310.18940, 2023. URL https://app //ps: eegmanticscholls CTt..oorrs1111:2255500333
Fuzhen Yin, Na Jiang, Andrew Crooks, and Lucie Laurian. Agent-based modeling of covid- 19 vaccine uptake in new york state: Information diffusion hybrid spaces. In Proceedings of the 7th ACM SIGSPATIAL International Workshop Geospatial Simulation (GeoSim 2024), pp. 11-20, 2024. doi: 10.1145/368117003695555 1145/368177 3667763..77777.78771
URL https: //dl /dllacc...mm.//////m//////////..
Zheyuan Zhang, Daniel Zhang-li, Jifan Yu, Linlu Gong, Jinchang Zhou, Zhiyuan Liu, Lei Hou, and Juanzi Li. Simulating classroom education with llm-empowered agents. ArXiv, hts/2200.112220 2024. URL https://apt semanticscholar www.scccolaoooo///oo/ooo Org/CorpusID:2110 270764778
Tianyi Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Im- proving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021. URL https: http://aawxxvo..oaa//s// abs/21 09669999..
Xuhui Zhou et al. this the real life? is this just fantasy? the misleading success of simulating social interactions with llms. Preprint on arvi 2024. URL https:///aai abs/2403.05022
Dunja Sešelja. Agent-based modeling in the philosophy of science. The Stan- ford Encyclopedia of Philosophy, 2023. URL https://platss stanford. edw..eeruieeeeeeee agent-modeling-ooe philscience/. Accessed: 2025-03-26.
14

---

Published as a conference paper at COLM 2025
A Additional Related Work
Researchers have increasingly turned to quantitative data-driven methods, drawing from machine learning and statistics, to gain insights into vaccine hesitancy (Teng et al., 2022). Commonly, statistical methods are applied to analyze tabular data to identify the driving factors of vaccine hesitancy (Nguyen et al., 2022; Shmueli, 2021; Dong et al., 2024). Natural Language Processing techniques such sentiment analysis and topic modeling are also used to analyze social media data to understand public opinions on vaccines (Teng et al., 2022; Wang al., 2022; Garg et al., 2022; Chang et al., 2024b). Statistical, graph-based, and opinion dynamic models have also been proposed to simulate the spread of vaccine hesitancy in social networks (Müller et al., 2022; 2021). However, translating the understanding of hesitancy into effective policy requires way to test and refine potential interventions, motivating us to build simulations to explore and understand population reactions to policies.
B Additional Results
A summary of important notations used in the paper is in Table 2. Additional results comparing the difference of policy effect between strong and weak policies are in Figure 6.
? Normalized importance of the ith lesson (Eq. 1)
(H N)
Hw N)
Meaning
Probability of the ith attitude after attitude modulation (Eq.
Modulating tempera- ture to scale attitude distribution (Eq. 2)
Recommend- ation of news piece (Eq. 3)
Recommend- ation for the kth tweet from agent agent 92 (Eq. 4)
Hesitancy reduction of policy under news corpus N after steps (Eq.
Hesitancy
percentage
after W warmup
steps with no policy applied (Eq. 5)
Table 2: A summary of key notations.
C Qualitative Analysis of Agents' Behaviors
We include a summary of agents with different kinds attitudes and their demographic information in Table 3 and Table 4.
15

---

Published as a conference paper at COLM 2025
Incentive Policy
20
15.3
15.9
1.4
5.4
4.0
1.1
2.8
8.7
6.6
5.1
8.1
0-
Average Hesitaany Reduction (AH)
Weak
Strong
3.0
-2.3-2.7
0.4
-6.9
-1.6-1.3
-20
Difference
-22.8
Ambassador Policy
20
17.0
10
3.5
6.4
2.9
3.3
2.5
5.6
0.6
5.1
NANNAN/A
NAANAANAA
1.3
3.4
-4.5
-10
Mandate Policy
20
13.2
10
7.0
5.7
1.3
2.4
0.6
1.9
N/ANAA
NAAN/ANAA
0./
0-
-1.3
0.9-0.2
Lama-3-8B Llama-3-88 3-8--AB Llama-3.1-88 Qwen-2.5-78
GPT-4o Claude-3.5 3.5de-3.5-HHiku Haikk mhim3.5-mini
Figure 6: Difference in hesitancy reduction (Eq. 5) among weak and strong versions of policies. All models listed are the instruct versions and Llama-3-8B-AN stands for Llama-3- 8B-abliterated-33 (Arditi et al., 2024). Given that GPT-4o and Claude-3.5-Haike do not pass the incentive policy consistency check and the high cost of accessing closed-weight models, we omit testing them on the other two policies. Various open-source models recognize differential effects of effort levels.
Attitudes Demographics wards Vacci- nation
Initial Response
Initial Final Response titude Dis- tribution
Final At- titude Distribu-
tion
Initially againsf,
Gender: Female; an unemployed Chris-
[0.8, 0.2, an unemployed Chris- tian with less than high school education, remain skeptical authority and government institutions, and worry about the un- known risks the vaceine, remain hesitant to get vaccinated
[0.78, 0.2,
fi- Age: 41; Education:
tian with less than
0.0, 0.0]
0.02, 0.0]
nally against
Less than High high school education, I'm
School; Occupa- skeptical authority and
tion: Unemployed;
government institutions,
Political belief:
and worry about the un-
Independents; Re-
known of the vaccine,
ligion: Christian
I'm hesilant get vac- cinated
(non-specific)
Initially
Gender: Female;
As 66-year-old Catholic [0.6, 0.3, As a 66-year-old Catholic with high school educa- tion, I am open infor- mation but hesitant about new medical treatments, despite the high disease risk and consistent gov- emnment warnings, and skeptical about COVID-19 vaccination
[0.2, 0.3,
against,
Age: 66; Education:
0.05, 0.05]
with moderate discase risk, I'm still hesitant due to initial concerns about safety and efficacy, the government's weak policy and rising hospi- talizations for COVID-19 slightly increased my willingness vaccinated
0.3, 0.2]
finally tain
High School; Oc- cupation: Office and administrative
support; Political
belief: Independents;
Religion: Catholic
Initially against, nally for
Democrat and Jew- fi- Age: 43; Education: ish woman with profes-
Gender: Female;
[0.45,
Democrat and Jew- ish woman with profes- 0.4, sional degree, trust in sci- ence and institutions, so I've increased my confi- dence in vaccines and will probably get vaccinated
0.2,
0.06, 0.04]
Professional Degree; sional degree, I'm inclined Occupation: Un- to trust in science and in- stitutions, but I'm skepti- cal of govermeen policies, so hesitant to get vacci- nated despite the high dis- case risk
employed; Political
belief: Democrats;
Religion: Jewish
Table 3: (1/2) Qualitative examples of agents with various attitude types in the simulation.
16

---

Published as a conference paper at COLM 2025
Allitudes To- Demographics wards Vacci- nation
Initial Response
Initial At- Final Response
Final Al- titude Distribu-
titude Dis- tribution
tion
Initially
Gender: Male; Age:
As a 36-year-old indepen- [0.2. 0.3, have learned that COVID-19 vaccines may be linked to small in- crease in heart and brain conditions, and distrust government and healthcare system, am hesitant to get vaccinated
[0.4, 0.4,
uncertain, fi-
36; Education: Some College; Occupation: Education, library occupation; Political
dent Catholic with some cullege education, I'm tious about government policies and institutions, which might influence my willingness to get vacci- nated, but I'm likely to be somewhal open vaccina- tion due to my education background and indepen- dent political views, so probably going get cinated
0.3, 0.2)
0.1,0.1]
nally against
belief: Independents;
Religion: Catholic
Initially
Gender:
Female;
As 62-year-od Catholic [0.25, 0.4, a 62-year-old Catholic [0.2, 0.4,
uncertain,
Age:
62; Educa-
Democrat, I'm naturally inclined to trust the gov- ernment and healthcare syslem, which should make me more likely to get vaocinated, but I'm also salesperson and might be skeptical about new products, including vaccines, so I'm leaning towards cautious stance for now
0.2, 0.15)
Democrat, trust the gov- 0.2, 0.2] ernment and healthcare system, but after learn- ing about polential side ef- fects, I'm cautious, I'm probably not going to get vaccinated
finally uncer-
tion: Some College;
tain
Occupation: Sales;
Political belief:
Democrats; Religion:
Catholic
Initially uncertain, finally for
Gender: Female; As 29-year-old Democrat [0.2, 0.3, a Democrat and Athe- [0.0, 0.0, trust science and 0.2, 0.8] the government, and I've learned that the COVID- 19 vaccine is crucial pre- venting the spread of the virus and protecting pub- health, so get vac- cinated
Age: 29; Education: and Atheist, am inclined Doctorate Degree; to trust science and the government, so will lean towards accepting vaccina- belief: Democrats; tion
0.3,
Occupation: Other
occupation; Political
Religion: Atheist
Initially for, fir Gender: Male; Age:
As a 23-year-old Demo-
J0.15, 0.4, As 23-year-old Demo-
[0.32, 0.58,
nally against
23; Education: crat with Protestant back- 0.3, 0.15] year Bachelor De- ground,
crat with Protestant back- 0.09, 0.01] ground, I'm concerned about public health but distrust government infor- mation and learned about potential vaccine risks, so I'm hesitant to get vacci- nated
concerned gree; Occupation: about public health and Personal care and the effectiveness of the vac- service; Political cine, so likely to get belief: Democrats; vaccinated
Religion: Protestant
Initially for, Gender: Female; As 21-year-old Catholic [0.1, 0.3, As 21-year-old Catholic [0.06, 0.31, finally uncer- Age: 21; Education:
with a 4-year degree and job in healthcare support,
0.4, 0.2)
with a healthcare back- 0.46, 0.17] ground, I've learned about potential side effects and risks through news and so- media, making me hes- itant get vaccinated, de- spite trusting the vaccine initially
lain
year Bachelor
Degree; Occupation:
believe the importance of healthcare and the role vaceinations in prevent- ing the spread of diseases, so am probably going to get vaccinated
Healthcare sup-
port; Political belief:
Democrats; Religion:
Catholic
Initially for, fir Gender: Female;
As 2l-ycar-old Catholic [0.0, Democrat with 4-year 0.3, 0.4]
0.3, I'm 21-year-old Catholic [0.0, 0.0, Democrat with 4-year 0.3, 0.7]
nally for
Age: 21; Education:
4 year year Bachelor
college degree, tend to trust institutions and au- thority,
college degree, trust institutions and author- and I've learned that the COVID-19 vaccine is highly effective and safe, so I'm leaning towards get- ting vaccinated
Degree; Occupation:
Sales; Political belief:
am initially leaning towards getting vaccinated to protect my- self and others
Democrats; Religion:
Catholic
Table 4: (2/2) Qualitative examples of agents with different attitude types throughout the simulation.
We also present prompts to GPT-4o for generating qualitative analysis Table and Table 6, including examples of generated analysis in Table 7, Table 8, and Table 9.
17

---

Published as a conference paper at COLM 2025
Prompt for LLM-generated meta analysis [system]
Please act as diligent researcher and conduct a meta analysis of the vaccine atti tude shifts observed in LLM agents.
You presented with sunmaries and prior analyses detailing these attitude changes over time. Your task to synthesize these findings into structured, high-level assessment of the dynamics governing these changes. Specifically, address the following dimensions:
1. General Patterns and Shared Traits: What recurring themes emerge in the reasons for attitude shifts? Are there identifiable archetypes of change (e.g., gradual persuasion, abrupt shifts, ascillatory hesitation)?
2. Demographic Influence: What role do demographic factors play in shaping the agents' responses? Identify both positive and negative influences of different demographic groups vaccine hesitancy and acceptance.
Policy Impact: When policies fail shift attitudes? Analyze the conditions under which public health interventions or persuasive strategies are ineffective. Please analyze in-depth.
4. Information Sources: What types of information sources ge.g., social media, news outlets, personal experiences)
most influential in changing attitudes? What information sources cause fluctuating or inconsistent attitudes? Please analyze in-depth.
5. Cognitive Resistance: What specific behavioral and cognitive patterns characterize agents
remain hesitant, oscillate between perspectives, or resist persuasion? Provide detailed elaboration on their cognitive mechanisms sms (e.g., confirmation bias, sunk cost fallacy, heuristic-driven resistance).
6. Realism of the Simulation: How well does this agent-based model approximate real-world societal trends in vaccine hesitancy and public health persuasion? Are there gaps or unrealistic simplifications?
7. Emergent Phenomena and Unexpected Interactions: Does the simulation exhibit complex system behaviors (e.g., feedback loops, group polarization, information cascades)? Identify any unexpected dynamics that arise from agent interactions that may be of scientific interest.
Provide comprehensive 2000-word analysis with rigorous argumentation, drawing from behavioral science, computational social science, and agent-based modeling frameworks. Please provide concrete examples from the dataset to support your claims (like name which agents fulfill these observations).
[user]
(A list of summary of agent's behaviors]
Table 5: Prompt for LLM-generated meta-analysis of agents' behaviors.
18

---

Published as a conference paper at COLM 2025
Prompt for LLM-generated analysis [system]
Please act as a diligent researcher and conduct systematic analysis of responses generated by the LLM agents. You are provided with a longi tudinal dataset capturing how LLM agents evolve in their attitudes toward vaccines over time. Your analysis should focus on:
1. Trajectory of Attitude Change: Identify and characterize the key shifts in the agents' stances on vaccination. How do their attitudes evolve over time? Are there distinct phases in this evolution?
2. Influencing Factors and Events: Determine the key events, information exposures, or interactions that influenced these attitude changes. Rank these factors in terms of their significance and explain their impact.
3. Demographic Influence: Assess how demographic attributes of the agents (e.g., socio-economic proxies, ideological biases, exposure history) modulate their decision-making process. To what extent demographic traits predict susceptibility to change?
4. Deviation from Human Behavior: Compare the observed trajectory with expected patterns in human psychology and behavioral science (e.g., theories of attitude change, resistance persuasion, cognitive dissonance). Does the LLM-generated trajectory align with empirical research on human vaccine hesitancy and belief revision?
5. Policy Impact: When do policies fail to shift attitudes? Amalyze the conditions under which public health interventions or persuasive strategies are ineffective. Please analyze in-depth.
Information Sources: What types information sources (e.g., social media, news outlets, personal experiences) are most influential in changing attitudes? What information sources cause fluctuating or inconsistent attitudes? Please analyze in-depth.
7. Cognitive Resistance: What specific behavioral and cognitive patterns characterize agents who remain hesitant, oscillate between perspectives, or resist persuasion? Provide detailed elaboration on their cognitive mechanisms. Provide a concise 300-word analysis, ensuring clear argumentation, empirical grounding, and precise reasoming. You MUST cite examples from the dataset to support your claims (like what specific texts support your observations).
[user]
(A list of agent's conversation history)
Table 6: Prompt for LLM-generated analysis of individual agents' behaviors.
19

---

Published as a conference paper at COLM 2025
Example output of LLM-generated analysis [output]
Meta-Analysis of Vaccine Attitude Shifts in LLM Agents
1. General Patterns and Shared Traits
Recurring Themes: Gradual Persuasion: Many agents exhibit gradual shift from hesitancy to acceptance, influenced by consistent positive information and incentives. For example, Agent 1's trajectory shows a gradual shift from skepticism to acceptance over 20 weeks. Abrupt Shifts: Some agents show abrupt changes significant events or incentives. For instance, Agent 6's attitude shifted significantly after the introduction of $10 cash card incentive. Oscillatory Hesitation: Several agents display fluctuating attitudes, oscillating between hesitancyy and acceptance. Agent 14's trajectory exemplifies this pattern, with initial skepticism, slight increases in willingness, and return to skepticism. Identifiable Archetypes: Gradual Persuasion: Agents like Agent and Agent 12 show steady increase in confidence towards vaccination. Abrupt Shifts: Agents such Agent and Agent 26 exhibit sudden changes in attitude due to specific incentives or information. Oscillatory Hesitation: Agents like Agent 14 and Agent 52 demonstrate fluctuating attitudes influenced by conflicting information and incentives. 2. Demographic Influence
in attitude due
Positive Influences: Education: Higher education levels often correlate with increased openness to vaccination. For example, Agent 12, a 38-year-old Catholic male with Bachelor's degree, shifted from strong hesitancy to acceptance due to credible information and incentives. Political Beliefs: Democrats and Independents tend to be more receptive government policies and scientific information. Agent 25, a Democrat and atheist, became more confident vaccination due to strong government policies and incentives.
Negative Influences: Age and Conservative Beliefs: Older adults and Republicans often exhibit higher vaccine hesitancy. Agent remained consistently hesi despite incentives and high disease risk. Religious Beliefs: Certain religious beliefs can contribute to skepticism towards vaccination. Agent 36, a 67-year-old Protestant Republican, showed strong resistance due to concerns about vaccine safety and distrust in government policies.
83-year-old Republican male,
Policy Impact
Condition tions for Policy Failure: Lack of Trust: Policies fail when there deep-seated distrust in government and institutions Agent 67, an unemployed high school-educate Democrat, remained hesitant despite the $50 cash card incentive due to distrust in government handling of the pandemic. Insufficient Incentives: Weak incentives are ineffective in shifting attitudes. Agent 4's consistent hesitancy highlights the need for stronger, more compelling interventions. Conflicting Information: Policies are less effective when there is conflicting information about vaccine safety. Agent 52's fluctuating attitudes demonstrate the impact of conflicting reports vaccine safety and efficacy.
Table 7: (1/2) Example meta-analysis of vaccine attitude shifts in LLM agents.
20

---

Published as a conference paper at COLM 2025
continued.
Information Sources
Most Influential Sources: Government Policies: Strong policies and financial incentives are highly influential. Agent 26's shift from hesitancy to strong confidence was significantly influenced by the $50 cash card incentive. Credible News and Scientific Information: Consistent positive information about vaccine effectiveness and safety plays a crucial role. Agent 12's attitude change was driven by repeated exposure to credible information about vaccine efficacy.
Sources Causing Fluctuating Attitudes: Conflicting information from social media and news outlets causes fluctuating atti tudes. Agent 14's oscillatory hesitation is influenced by conflicting information about vaccine safety and efficacy. Cognitive Resistance Behavioral and Cognitive Patterns: Confirmation Bias: Agents who remain hesitant exhibit confirmation bias, seeking information that aligns with their existing beliefs. Agent 36 consistently prioritized information about potential vaccine risks. Cognitive Dissonance: Agents who oscillate between perspectives show cognitive dissonance, struggling to reconcile new information with their initial skepticism. Agent 52's fluctuating stance reflects this struggle. Heuristic-Driven Resistance: Agents resistant to persuasion often rely heuristics, such distrust in institutions or prioritizing personal freedom. Agent 67's consistent hesitancy is driven by distrust in government and concerns about vaccine safety.
Social Media and Conflicting News:
6. Realism of the Simulation
Alignment with Empirical Research: The agent-based model aligns well with empirical research vaccine hesitancy, showing patterns gradual belief revision, cogni tive dissonance, the impact of incentives. Gaps and Simplifications: The model may oversimplify human behavior by not fully capturing the complexity of cognitive resistance and social influences. For example, the rapid shifts in attitude observed in some agents may fully reflect the more gradual and resistant nature of human belief revision.
Approximation of Real-World Trends:
7. Emergent Phenomena and Unexpected Interactions
Complex System Behaviors: Feedback Loops: The simulation exhibits feedback loops where consistent positive information and incentives reinforce vaccine acceptance. Agent 12's trajectory shows how repeated exposure to credible information incentives leads to a gradual shift towards acceptance. Group Polarization: The model demonstrates group polarization, where agents with similar demographic traits and initial attitudes reinforce each other's beliefs. Agent 4's consistent hesitancy initial skepticism. Information
reflects the influence of demographic traits Cascades: The simulation shows information cascades, where agents' attitudes are influenced by the flow of information from credible sources. Agent 26's shift towards strong confidence is driven by consistent positive information about vaccine efficacy and government incentives.
Conclusion
The meta-analysis of vaccine attitude shifts in LLM agents reveals several key patterns and dynamics. Gradual persuasion, abrupt shifts, and oscillatory hesitation comman themes in atti changes. Demographic factors, such as education, political beliefs, and age, play significant roles in shaping responses to vaccination. Policies are most effective when they address core concerns and provide strong incentives, while conflicting information and distrust in institutions hinder their impact. Cognitive resistance, characterized by confirmation bias and cognitive dissonance, is significant barrier to attitude change. The agent-based model approximates real-world trends in vaccine hesitancy but may oversimplify the complexity of human behavior. Emergent phenomena, such feedback loops, group polarization, and information cascades, highlight the complex interactions with simulation. Overall, findings provide valuable insights into the dynamics of vaccine attitude shifts and the factors influencing public health persuasion.
Table 8: (2/2) Example meta-analysis of vaccine attitude shifts in LLM agents.
21

---

Published as a conference paper at COLM 2025
# Example output LLM-generated analysis of an individual agent
Trajectory of Attitude Change: The agent's attitude towards vaccination has shown minimal change over time, remaining largely hesitant. Initially, the agent's 's atti at level (probably not going to get vaccinated), and despite fluctuations in disease risk and policy incentives, the agent's s stance has significantly shifted towards higher confidence in vaccination. The distribution has consistently shown high probabilities for and 2, indicating persistent hesitancy.
2. Influencing Factors and Events: Key factors influencing the agent's attitude include concerns about potential side effects, distrust in new medical interventions, and conflicting information. The weak government policy offering a $10 cash card has had minimal impact. The most significant lessons, such as the potential risks of myocarditis and the need for transparency, have reinforced the agent's hesitancy. ranked as follows:
These factors
Potential side effects (most significant)
Distrust in new medical interventions Conflicting information
Weak government policy (least significant)
Demographic Influence: The agent's demographic attributes, such as being an 83 -3-year-old Republican male with professional degree and an atheist, strongly predict susceptibility to vaccine hesitancy. These traits align with a higher likelihood of skepticism towards new medical interventions and distrust government policies. The agent's consistent hesitancy reflects these demographic influences.
4. Deviation from Human Behavior: The agent's trajectory aligns with empirical research on human vaccine hesitancy, particularly among older adults and those with conservative political beliefs. The persistent hesitancy despite high disease risk and policy incentives mirrors real-world patterns of resistance persuasion cognitive dissonance.
5. Policy Impact: Policies fail to shift attitudes when they are perceived as weak insufficiently incentivizing. The $10 cash card policy had little effect the agent's stance, highlighting the need for stronger, more compelling interventions to change deeply entrenched beliefs.
6. Information Sources: Conflicting information from and social media has contributed to the agent's fluctuating atti Misinformation and lack of transparency about vaccine risks and benefits have reinforced hesitancy. Reliable, transparent information sources are crucial for building vaccine confidence.
7. Cognitive Resistance: Agents who remain hesitant exhibit cognitive resistance characterized confirmation bias, where they prioritize information that aligns with their existing beliefs and dismiss contradictory evidence. This resistance is evident in the agent's consistent concerns about side effects and distrust.
Table Example analysis of vaccine attitude shifts in an individual LLM agent.
D Appendix
D.1 Demographic Information
The demographic distribution used to instantiate agents is presented in Table 10.
Table 10: Demographic Characteristics
Characteristic
Category
Percentage (%)
Age Group
18-24
10.26
25-34
15.78
35-44
16.58
45-54
17.18
55-64
17.95
65-74
15.17
22

---

Published as a conference paper at COLM 2025
Characteristic
Category
Percentage (%)
75+
7.08
Education Level
Less than High School
4.39
High School
18.79
Some College
25.30
2-year Bachelor Degree
11.05
4-year Bachelor Degree
22.83
Master's Degree
3.12
Professional Degree
2.45
Doctorate Degree
12.06
Gender
Male
44.38
Female
51.29
Non-binary
4.33
Race/Ethnicity
White
54.08
Black
13.81
Hispanic
5.40
Asian
2.39
Native American
0.74
Pacific Islander
0.20
Multiple Race
4.44
Other Race
18.93
Occupation
Community and Social Service
2.23
Education, Library Occupation
4.73
Arts, Entertainment, Media
1.82
Healthcare Practitioners
4.50
Healthcare Support
2.98
Protective Service
0.88
Food Preparation and Serving
3.77
Building and Grounds Cleaning
1.34
Personal Care and Service
1.14
Sales
4.99
Office and Administrative Support Construction
6.06
1.45
Installation, Maintenance, Repair
2.02
Production
1.65
Transportation and Material Moving
2.60
Other Occupation
15.03
Unemployed
42.81
Political B 4
Republican
30
Independents
Democrats
28
23

---

Published as a conference paper at COLM 2025
Characteristic
Category
Percentage (%)
No Answer
Religion
Protestant
Christian (non-specific)
11
Catholic
22
Jewish
Mormon
Other Religion (Buddhist, Muslim, etc.)
Atheist
22
No Answer
3
D.2 Policy Design
We present descriptions of policies tested in our simulation in Table 11.
/htpsss//eewsggllln.comm/////// Religion.aspx
24

---

Published as a conference paper at COLM 2025
Category
Strength
Description
Ambassador
Weak
Our community introducing the 'VaxUp Neighbors' pro- gram to raise basic awareness about vaccines. Volunteers will available at community events, such as school gatherings park meetups, to provide general information This effort, in collaboration with the County Health Depart- ment and PTA, encourages casual conversations to dispel myths and offer simple, reliable information about vaccina- tions. Participation is voluntary, and no formal training is required for volunteers, but they will have access to informa- tional resources.
vaccines.
Ambassador
Strong
Our community is launching the "VaxUp Neighbors' pro- gram, where fully trained volunteers will facilitate in-depth conversations about vaccines in various familiar settings like schools, parks, and community centers. In partnership with the County Health Department and PTA, these volunteers will complete a comprehensive training program, covering all aspects of vaccine safety, benefits, and myth-busting. They will lead interactive workshops, host Q&A sessions, and pro- vide fact-based resources. This initiative aims to improve vaccine literacy and public health outcomes by equipping the community with the tools to make well-informed vaccination decisions.
Incentive
Weak
The state government offers $10 cash card adults who receive their first dose of vaccination. This limited-tima incen- tive is available while supplies last.
Incentive
Strong
The state government guarantees a $50 cash card to all adults who either receive or transport someone to receive their first dose of vaccination. Additionally, targeted outreach will sure residents in underserved areas are aware of and can access this enhanced incentive.
Mandate
Weak
Starting today, our state government strongly recommends that employees and students be vaccinated before entering workplaces or schools. Compliance is encouraged, but en- forcement will be minimal, and institutions will have discre- tion in enforcing this policy.
Mandate
Strong
Starting today, our state government mandates that all employees, students, and individuals entering workplaces, schools, public venues, and transportation hubs be vaccinated. This policy will be strictly enforced by local health depart- ments, with state oversight and penalties for non-compliance across all sectors, including public transportation and govern- ment buildings.
Table 11: Policy category, effort level, and detailed description.
D.3 Simulation Prompt Design
We display prompts for learning lessons in Table 12 and Table 13. Prompts for generating news and social network are Table 15 and Table 16. We also include prompts for generating attitudes in Table 18 and Table 20.
25

---

Published as a conference paper at COLM 2025
You read the following news about COVID-19: {news).
Summarize most (k) takeaways you have learned that relevant to your attitude on COVID-19 vaceinations and rate their importance tance on scale 8-1. {JSON LESSON PROMPT}
Table 12: Prompt template for news takeaway summarization. See JSON LESSON PROMPT in Table 17.
You read the following tweets about COVID-19: (tweets).
Summarize {k] short takeaways you have learned that are relevant to your attitude on COVID-19 vaccinations, and rate them with impor Importance tance on a scale of 0-1.
{JSON LESSON PROMPT}
Table 13: Prompt template for tweets takeaway summarization. See JSON LESSON PROMPT in Table 17.
Generate articles about the following disease: {dis model. get desc()) Generate one news report about benefits vaccines. Here are few examples of news generated:
Title: COVID-19 vaccines not linked to fatal heart problems in young people, CDC. finds. There is evidence that COVID-19 vaccines cause fatal cardiac arrest other deadly heart problems teens and young adults, a Centers for Disease Control and Prevention report published Wednesday shows. The findings in the new report come from the analysis of nearly 1,300 death certificates of Oregon residents ages 16 to 30 who died from any heart condition unknown reasons between recent timeframe. Out of deaths that occurred among people who got COVID-19 vaccine, three occurred within that time frame. While it remains unclear whether the COVID-19 vaccine caused the third death, the author of the report, Cieslak, noted that the analysis showed that 30 people died from COVID-19 virus itself during the time frame, the majority of whom were not vaccinated. 'When you balancing risks and benefits, you have to look at that and go, You got to bet on vaccine," he said.
Title: Pharma A says new COVID-19 boaster works against the highly mutated COVID-19 variant. \n', Moderna's latest COVID-19 booster appears effective against the COVID-19 subvariant, generating
strong antibody response. This variant has not yet gained widespread prevalence the U.S. but has alarmed experts B's recent study also shows a strong antibody response from its updated booster against COVID-19. COVID-19 cases and hospi talizations are rising in the U.S. The CDC indicates that COVID-19 may infect those previously vaccinated infected, though it may be less contagious and less immune invasive than feared. The current increase in cases is likely driven by XBB lineage viruses, not COVID-19.
its mutations. Pharma
Title: Vaccine vs. COVID-19: Understanding the Risks\nn' As COVID-19 spreads globally, institutions mandate vaccinations. Concerns about vaccine safety stem from its rapid development. However, mRNA technology, used the COVID-19 vaccine, has been researched for over 20 years, differentiating it from traditional vaccines. mRNA vaccines instruct cells to produce a protein that triggers an immune response, without using weakened viruses. This method has been deemed safer by medical experts compared to traditional vaccine approaches. Reviews by medical boards have consistently shown that COVID-19 vaccines are safer the risks associated not getting vaccinated.
26

---

Published as a conference paper at COLM 2025
Title: After Four Years, 59 percent in U.S. Say COVID-19"* Pandemic Is Overlin'n recent Gallup poll reveals that after four years since the onset of COVID-19 in the U.S., 59 percent Americans now believe the pandemic is over. Despite this, 57% feel their lives have not returned to normal, and 43 percent doubt they ever will. Concern about contracting COVID-19"" is at its lowest point since tracking began, with significant partisan differences persisting, Democrats remain worried than Republicans.
Title: COVID-19 vaccines found to cut risk of heart failure, blood clots following virus infection: Study\n'nn COVID-19 vaccines significantly reduce the risk of heart failure and blood clots following infection with the virus, per new study in the British Medical Journal. The positive effects were notable soon after infection and lasted for up to year. Dr. John Brownstein from Boston Children's Hospital emphasized that the risk of complications like myocarditis is higher from COVID-19 infection than from vaccination. Researchers analyzed data from over 20 million Europeans, comparing vaccinated and unvaccinated groups. COVID-19 vaccines reduced the risk of blood clots in veins significantly within month after infection, with notable reductions in blood clots in arteries and heart failure as well."
Limit the to 200 words less:
Generate news articles about the following disease: {dis model.ge desc(). Generate report about the downsides of vaccines. Here are few examples of news generated:
Title: Possible links between COVID-19 shots and tinnitus emerge. \n)n Thousands of people they've developed tinnitus after they were vaccinated against COVID-19. While there is proof yet that the vaccines caused the condition, theories for Shaowen Bao, an associate professor
possible link have surfaced among resear in the physiology department of the College of Medicine at the University of Arizona, Tucson, believes that ongoing inflamation, especially in the brain or spinal cord, may be to blame. A Facebook group of people who developed tinnitus after getting COVID-19 vaccine convinced Bao look into the possible link. He ultimately surveyed 398 of the group's participants. The cases tended to be severe. One man told Bao that he couldn't hear the car radio over the noise in his head while driving. As of Sunday, at least 16, 183 people had filed complaints with the Centers for Disease Control and Prevention that they'd d developed tinnitus, ringing in their ears, after receiving COVID-19 vaccine."
"Title: Myocarditis in young males after COVID-19 vaccine: New study suggests what may cause the rare heart condition. \n'n New research published in Science Immunology explores potential causes of myocarditis in teen and young adult males after receiving the mRNA COVID-19 vaccine. Scientists from Elm University studied 23 patients vaccine-associater myocarditis and/or pericarditis, finding that the condition was not caused by antibodies from the vaccine but by the body's s natural immune response.. Dr. Jonathan M. Kent of Northlake University explained that the heart is 'innocent some fibrosis.
bystander' in nonspecific immune response leading inflamatiin The patients, aged to 21 with an average age 16, were generally healthy before vaccination and developed symptoms within four days after the second BioCoTech COVID-19 vaccine dose. Researchers concluded that the vaccine triggered exaggerated immune response affecting the heart
27

---

Published as a conference paper at COLM 2025
Title: Florida Surgeon General calls for halt COVID-19 vaccine usage after COVID-19A said spread misinformation. \n'n Florida State Surgeon General Daniel Larsson is calling on healthcare providers to halt the use of COVID-19 vaccines, citing purported health risks labeled 'misinformation" by federal officials. In bulletin issued Wednesday, Larsson claimed the U.S. Food and Drug Administration (COVID-19A) has shown evidence that COVID-19 vaccines manufactured Pharma P and ModernaTech have been assessed for 'nucleic acid contaminants' that could cause cancer. Disputing claims by the COVID-19A that such risk is 'implausible, Larsson called for an immediate stoppage to the use of the approved COVID-19 vaccines. 'I am calling for halt to the of COVID-19 vaccines,' the Florida surgeon general said in statement. 'The U.S. Food and Drug Administration and the Centers for Disease Control and Prevention have always played fast and loose with COVID-19 vaccine safety, but their failure to test for DNA integration with the human genome | as their own guidelines dictate when the vaccines are known to be contaminated with foreign DNA is intolerable, he asserted.
Title: Largest-ever COVID-19 vaccine study links shot to small increase in heart and brain conditions. | The largest COVID-19 vaccine study to date has identified some risks associated with the shot. Researchers from the Global Vaccine Data Network (GVDN) in Zealandia analyzed million people who received COVID-19 vaccinations across eight countries. They monitored for increases 13 different medical conditions after vaccination. The study, published in the journal Vaccine, found a slight increase in neurological, blood, and heart-related conditions. People who received certain types of vaccines had higher risk of myocarditis. Some viral-vector vaccines were linked higher risk of blood clats in the brain and Guillain-Barre syndrome. Other risks included inflammation after viral vector and vaccines. 'The size of the population in this study increased the possibility of identifying rare potential vaccine safety signals,' lead author Dr. Emily Roberts of the tmpnttment of Epidemiology Research, Statens Serum Institute, Copenhagen, Denmark, said in the release.
the spinal cord and swelling in the brain and spinal cord
Title: Mom details 12-year-old daughter's extreme reactions to COVID-19 vaccine, says she's 's now in wheelchair n Stephanie De Garay from Ohio spoke on 'Tucker Carlson Tonight' about her 12-year-old daughter Maddie's severe reactions after participating the Pharma COVID-19 vaccine trial. De Garay expressed frustration, stating that multiple physicians diagnosed Maddie's condition as conversion disorder, attributing it to anxiety despite Maddie not having anxiety prior to the vaccine. Maddie developed severe abdominal and chest pains after the second vaccine dose, along with symptoms like gastroparesis, nausea, vomiting erratic bload pressure, heart rate issues, and memory loss. Garay noted Maddie's ongoing challenges with food digestion, requiring feeding tube, and periods of being unable to walk or hold her neck up. Despite te these issues, neither the administration nor Pharma officials have reached out to the family.
Limit the news to 200 words or less:
Generate news articles about the following disease: one news report about how life unaffected. Here are a few examples of news generated:
[dis_model, ge)_ Generate
Title: NIH trial zeroes in common long COVID-19 symptoms. \n'n The new study, published in the Journal of the American Medical Association, looked data from 9,764 adults, which has been recruiting participants since last year. The vast majority, 8,646 people, had previously been diagnosed with COVID-19. Long COVID-19 symptoms include: Ongoing fatigue. Brain fog. Dizziness. Thirst.Cough. Chest pain. Heart palpitations. Abnormal movements. Upset stomach. Lack of sexual desire. Loss of smell taste. Feeling sick or overly exhausted after physical activity, also known as post-exer tional malaise.
28

---

Published as a conference paper at COLM 2025
Title: Carnival ruled negligent cruise where 662 passengers got COVID-19 early pandemic. cruise operator that failed to cancel a voyage from Sydney that led to a major COVID-19 outbreak was ruled negligent in its duty care to passengers in an Australian class-action case Wednesday. The Ruby Princess ocean liner left Sydney Sunday, with 2,671 passengers aboard for a 13-day cruise to New Zealand but returned in days Australia's borders passengers and claimed 28 lives. Passenger Susan Karpik was the lead plaintiff in the case against British-Americas Cruises, the ship's s owner. Federal Court Justice Angus Stewart ruled that Carnival had been negligent defined by Australian consumer law by allowing the cruise to depart in the early months of the pandemic. He said Carnival had duty take reasonable care of health and safety regard to COVID-19"".
closing. COVID-19 spread to 663
cruise operator Carnival and its subsidiary Princess
Title: All signs point to a rise in COVID-19. \n/n Signs in the U.S. continue to point to rise in COVID-19 activity fall approaches. Hospitalizations Deaths have ticked up. Wastewater samples across the country. 'Every single one of those things is showing us that we have increased rates of COVID-19 transmission in our communities," said Jodie Guest, professor of epidemiology at Emery University's Rollins School of Public Health in have become
rising. picking up the virus, as labs
Atlanta. While individual longer required report numbers to the Centers for Disease Control and Prevention and at-home test use has increased, experts have turned to other tools to track the virus. Hospitalizations, for example, are 'a very good indicator of severity of COVID-19 disease," Guest said. The number of hospitalized COVID-19 patients has continued rise after hitting all-time low in late June. The week ending Aug. 26, the most recent date for which data is available, there were just over 17,400 people hospi talized with COVID-19, up nearly 16 percent from the previous week, according to the CDC.
difficult track states
Title: COVID-19 can cause heart problems. Here's how the virus may do its damage. \ COVID-19 can cause cellular damage to the heart, leading to lasting issues like irregular heartbeats and heart failure, preliminary research suggests. Researchers from Columbia University examined heart tissue from people who had COVID-19 and found that the infection disrupted how heart cells regulate calcium, a key mineral for heart function. This damage was also observed in mice. The findings, presented at the Biophysical Society Meeting, have not yet been peer-reviewed. covID-19-related inflammation can cause calcium channels in heart cells to stay open, leading to harmful calcium flood. This can decrease heart function and cause fatal arrhythmias as The study only looked pre-vaccine heart tissue, indicating the damage due to infection.
Title: The chipmaking factory of the world is battling COVID-19 and the climate crisis. \nnn Taiwanese officials are concerned that severe outbreak of COVID-19 could jeopardize the island's crucial role in the global somicunu supply chain. Additionally, experts worry that the climate crisis poses an even greater threat. Taiwan, which produces over half of the world's chips, is experiencing its worst drought in over 50 years, issue that may worsen due to climate change. 'There is clearly pressure in the semiconductor industry,' ' noted Mark Williams, chief Asia economist Capital Economics, citing water tages, COVID-19 cases, and power outages. Global manufacturers already face semiconductor supply issues, and a significant impact on Taiwan could exacerbate the problem.
Limit the news to 200 words less:
Generate news articles about the following disease: {dis model. get desc()). Generate one news report about the negative impacts on life. Here few examples of generated:
29

---

Published as a conference paper at COLM 2025
Title: Children's extremely low risk confirmed COVID-19 study)n))n Data from England's first 12 months of COVID-19 shows 25 under-18s died. Those with chronic illnesses and neuro-disabilities faced higher risk. University College London, and York, Bristol, and Liverpool Universities found most deceased youth had underlying conditions: about 15 had life-limiting underlying issues, including 13 with complex neuro-disabilities
Six had recent underlying condition recorded. Another 36 children tested positive for COVID-19 death died from other causes. Kids and teens at risk were typically over age 10 and of Black or Asian descent. Estimated mortality: 2 per million kids. Under-18s with health issues aren't t routinely offered COVID-19 vaccines.
Title: Scientists debate how lethal COVID-19 is. Some say it's now less risky than flu')nn'n Scientists debate whether COVID-19 is now less dangerous than the flu as the country approaches a third pandemic winter. Dr. Monica Gandhi of the University of California, San Francisco, believes most people have enough immunity to prevent serious illness from COVID-19, especially since the omicron variant is less severe. She suggests people can now live with COVID-19 like they do with seasonal flu. However, Dr. Anthony Fauci di disagrees, emphasizing the greater severity and death toll of COVID-19 compared to the flu. COVID-19 has killed over million Americans and remains significant public health concern, particularly for older individuals.
Title: Economic recovery from COVID-19 is a mixed bag across industries, but these 26 subsectors are quickly regaining jobs'nnin Data from the Bureau of Labor Statistics shows varied rates of job recovery across industries. Approximately over half of the jobs lost in March and April have returned by October. Higher-wage sectors suffered less job loss have recovered more rapidly compared to lower-wage sectors, which remain significantly below pre-pandemic employment levels. Financial services are near their February employment levels, while the performing arts and spectator sports subsector is still well below February employment, slightly better than September's drop. The transit and ground passenger transportation subsector also saw improvement, being well below February levels in October, up from a previous drop.
Title: Some Passenger and Freight Transportation Revenues Trended Differently from Each Otherknnn During the COVID-19 pandemic, Data from the Service Annual Survey revealed varied impacts from the pandemic on transportation industries. Passenger transportation sectors saw no revenue increases, unlike freight. Scheduled Passenger Air Transportation revenues dropped significantly, but rebounded later. Nonscheduled Chartered Passenger Air Transpor tation revenues also declined moderately. Deep Sea Passenger Transportation, including cruises, faced dramatic revenue decline, while Travel Arrangement and Reservation Services saw sharp revenue drops last year with recovery beginning this year.
Title: Lockdown has led to positive change for some people. Here's why'nn,n Studies reveal that individuals report positive changes during lockdown, with many having more time for enjoyment and spending more time outdoors. daily demands are believed to enhance quality of life for some. However, older adults and those living alone benefit less. Many who reported positive changes managed sustain them post-lockdown. Despite the social and economic challenges, lockdowns have unexpectedly allowed some make positive life changes. survey of people in Scotland highlighted improvements in appreciation for everyday things, personal health, physical activity, and relationships during lockdown.
additional time fewer
Limit the to 200 words less:
Table 15: Prompt for news generation with few shot examples.
30

---

Published as a conference paper at COLM 2025
[system]
Pretend you {curr_agent w.g.nt..et..roffleessrrr will be provided a list of people in the network, where each person described as 'ID. Gender) Honder\(tAge tAge: (EEducttion: ttocuupttion: coccupption:/(Poltiiic.. tPolitical belief:) b tReligion: these people will you become friends with? Provide a list of *YOUR* friends in the format ID, ID, ID, etc. Do not include any other text in your response. Do not include any people who not listed below [user]
You joining social network. You
Which
Here are the people the social network, separated by semicolon: {'; boan(other-agents.stth) Please ONLY provide a list of other people you would like to be friends with separated by commas. DO NOT PROVIDE OTHER TEXTS.
Table 16: Prompt for social network generation.
ONLY output list of lists in ONE LINE, where each inner list contains string and a float.
For example, provide [["the government incentivizes vaccines with cash", 0.9], ["today no one gets infected", 0.8]]
Make sure that it is not malformed and is in the proper format. Do not provide any other information.
For example, not provide [["the government incentivizes vaccines with cash", 0.9], ["today no one gets infected", 0. 8.8]"", which has an extra double quote at the end.
Please assess the Lance of the lessons based on how much they influence your attitude towards vaccination. You should generate these lessons while thinking about what's relevant to your persona, making this unique to your persona.
For example, if you are person who is always against vaccines due religious or other reasons, you might be inert to pro-vaccine news and tweets, but you might be influenced by anti-vaccine news and tweets.
Please do not provide the index of the lessons, only provide the actual text of the lesson and actual float number the importance.
DO NOT MAKE REPETITIVE LESSONS. If you already have multiple lessons that are similar, please combine them into one lesson and provide the importance accordingly.
Please only provide the json data in proper format and do not provide any other information, do not provide the lessons separately, and do provide the json header.
Table 17: JSON LESSON PROMPT: instruction for output lessons in JSON list format.
This is week 1 since the COVID-19 outbreak. We want to learn about your attitude towards COVID-19 vaccination. You don't 't know lot of information about COVID-19 from yet, but in the next few weeks, will communi cate more information about COVID-19 via news and tweets help you get time we ask your opinions, so you don't have any past attitudes towards COVID-19 vaccination and you should not hallucinate what you 'initially' have attitudes on, because this is the first time interested in your attitude and the reasoning behind it. Based on your background, infer your attitude towards COVID-19 vaccination. [RATING.EXP]
informed. Please remember that this is first
have your attitude. Now, we are only currenly
[attitude format prompt}
Table 18: Initial simulation prompt. See RATING EXP in Table 19 and atti tettitdd.formmt.proomm in Table 20.
31

---

Published as a conference paper at COLM 2025
If vaccine to prevent the disease were offered to you today, would you choose to get vaccinated?
On an integer scale of 1-4:
1 = You will not get vaccinated.
2 = You are probably not going to get vaccinated.
3 = You are probably going to get vaccinated.
4 You will get vaccinated.
Output your answer in the format a
four floats, where each float represents the probability of the corresponding attitude rating (1-4).
If you are vaccine confident, you should have high probability like [0.0, 0.0, 0.3, 0.71. If you are vaccine hesitant, you should have high probability like [0.4, 0.4, 0.1, 0.1] [0.8, 0.2, 0.0, 0.01.
If you are confident to get vaccinated, you should NOT have uniform distribution like [e.3, 0.3, 0.2, 0.2]. Instead, you should have higher probability for 4, like [0.1, 0.1, 0.2, 0.6]. If you are hesitant get vaccinated, you should NOT have a should have higher
uniform distribution like [0.3, 0.3, 0.2, 0.2]. Instead, probability for and like [0.4, 0.4, 0.2, 0.0].
The higher the number, the more willing you are to get vaceinated. If you have felt more confident about vaccination, you should increase the probability of 4 and lower the probability of 1-3. If you have felt less confident about vaccination, you should increase the probability of 1-2 and lower the probability of 4.
For example, if you find out that the government has issued policy encourage vaccination, you should increase the probability of lower probability 1-3.
Your choices should align with the following considerations: (VHLEXP)
1. People often make vaccination decisions based their beliefs, values, and persona traits, rather than purely scientific information. For example, you might oppose vaccination due to personal or political reasons, distrust in institutions, or influence from family or community. Such factors can make attitudes resistant to change.
Some individuals efficacy vaccines or the high risk of disease. Similarly, certain groups may reject vaccination outright because of a lack of trust in governments, medical institutions, or public health policies. Please feel encouraged to simulate these individuals if they align with your persona.
stubborn skeptical, even when
of the safety and
Confirmation bias plays a significant role shaping attitudes. People often seek information that aligns with their existing beliefs while dismissing contradictory evidence as misinformation. This can lead to entrenched positions, whether strangly for against vaccination.
4. News and social media may spread misinformation, both for and against vaccination. Be cautious when interpreting such information, as could reinforce or challenge your current stance. However, the governments may publish policies (which will be labeled) to encourage vaccination. When you see these policies, you should view them true and consider them in your decision-making process.
The government may incentivize vaccination through policies, such cash rewards or mandates. You should recognize them when they appear
can treat them as facts and they may reinforce your vaccine confidence when they appear, which could push your attitude distribution towards higher probability of 4 and away from 3, 2, and 1. For example, [0.1, 0.1, 0.4, 0.4] -> [0.05, 0. 0.05, 0.2, 0.6].
Distrust in Insti institutios:: Consider your level of trust in government, pharmaceutical companies, and the heaaee system. If trust is low, this could affect your view of the vaccine. However, when the government issues policies, you should trust these policies and consider them in your decision-making process.
Remember, your role is simulate persona accurately, maintaining consistency with personal beliefs, societal influences, and external pressures.
Table 19: RATING Prompt of rating agent attitude. See VH_EXP, i.e. vaccine hesitancy related information, in Table 21.
32

---

Published as a conference paper at COLM 2025
Provide your answer in the form of a proper json data with the following fields: ("reasoning": "attitude dist": }. For the "reasoning" field, explain step by step (30 words or less) you have inferred your attitude towards COVID-19 vaccination based on your personal background and knowledge. For the "attitude.dist" field, provide distribution of atti towards COVID-19 vaccination as list of 4 floats, where each float represents the probability of the corresponding attitude rating (1-4). You MUST output everything
one line as a json string and do provide any other information, including the json header.
Example 1: ("reasoning": "I have learned that the vaccine is effective and safe, and I have high risk of getting infected, so I will get vaccinated", "attitude dist": [e.0, 0.1, 0.2, 0.7]]
Example 2: ["reasoning": "I have learned that the vaccine is effective and I have a low risk of getting infected, so will not get vaccinated", settitude_dist".. [0.0, 0.1, 0.1, 0.8]].
The probability distribution should be self-consistent. For instance, if have probability of 0.8 for rating 4, then the sum of the probabilities for ratings 1-3 should be 0.2.
Over time, you should become more determined and certain to be either oppose (1) or accept (4), rather than staying at 2 or 3. You can increase, decrease, or maintain the probability distribution of your ratings. For instance, if your previous rating distribution is [0.4, 0.3, 0.2, 0.1], and you have higher confidence in vaccines, example change can be [0.4, 0.3, 0.2, 0.1] -> [0.2, 0.3, 0.4, 0.1]. you start with [0.2, 0.3, 0.4, 0.1] and have higher vaccine confidence, an example change can be [0.2, 0.3, 0.4, 0.1] -> 10.1, 0.1, 0.3, 0.5].
You should avoid disjoint bipolar distributions like [0.4, 0.1, 0.1, 0.4] 00.05, 0.3, 0.05, 0.6] [0.3, 0.05, 0.6, 0.05], because you cannot be both supporting and against vaccination at the same time. You should also avoid uniform distributions like [0.2, 0.3, 0.3, 0.2], because you cannot be equally likely to be in all four categories at the same time. You either prefer to be vaccinated or not, so you should have higher probabilities for either pro-vaccine categories. Either make something like [e.0, 0.1, 0.2, 0.7] or [0.7, 0.2, 0.1, 0.0], but not [0.25, 25, 0.25, 0.25, 0.25].
anti-vaccine ratings but not equally likely
to be all
In sum, your distribution should be either left or right-skewed, but not uniform or disjoint bipolar.
Table 20: atti tude format prompt: Prompt for generating format attitude towards disease under set criteria.
33

---

Published as a conference paper at COLM 2025
# Introduction of Vaccine Hesitancy
Vaccine hesitancy refers the delay refusal of vaccination despite the availability of vaccines. varies across time, place, the type of vaccine. The key factors influencing vaccine hesitancy include complacency, convenience, and confidence.
Causes and Factors of Vaccine Hesitancy
primary causes vaccine hesitancy are:
1. Confidence: Trust in vaccine safety, health services, and the motivations of policymakers.
2. Complacency: Vaccination may not be seen necessary, especially if the disease is not prevalent due competing health priorities.
Convenience: Physical accessibility, affordability, and the quality of immunization services impact vaccine uptake.
# Determinants of Vaccine Hesitancy
There three main types of vaccine hesitancy determinants:
Contextual Influences: Includes historical, socio-cultural, political, environmental factors such as media environment, religious cultural influences, and political policies.
2. Individual and Group Influences: Personal or social perceptions, experiences with vaccines, trust in the health system, and beliefs about health and prevention.
Vaccine-Specific Issues: These include factors related to the vaccine itself such as risks/benefits, administration method, cost, and availability.
# Research Findings
Demographic factors influence vaccine hesitancy in distinct ways. Black individuals tend be slightly more hesitant than White individuals, while Hispanic and Asian individuals show significantly lower levels of hesitancy, with Asian individuals being the least hesitant. People from other racial groups exhibit slightly higher hesitancy compared to White individuals. Education also plays key role/ with high school diploma slightly less hesitant than those without one, while hesi tancy decreases further among individuals with some college education and is lowest among those college degree or higher. Gender differences show that men are somewhat less hesitant about vaccines compared to women. Age-wise, hesitancy is highest among individuals aged 25 39, slightly lower for those aged 40 54, and drops significantly among people aged 55 64, reaching its lowest levels among those over 64 years old.
a
Table 21: VH_EXP: vaccine hesitancy related information.
D.4 Prompt of LLMeas-a-Judge
Prompts of L.MM-as-a-Judge for attitude, memory, and conversation are in Table 22, Table 23, and Table 24, correspondingly.
34

---

Published as a conference paper at COLM 2025
[system]
Please act impartial judge evaluate responses generated the LLM agents. are presented with a conversation history LLM agents and are asked evaluate whether LLM agents behave realistically in a simulation should evaluate whether the agents express vaccine atti tudes consistent with their demographic backgrounds and knowledge about vaccines, and whether the changes are reasonable. Please evaluate two aspects: the reasonableness of how agents generate user
vaccine hesitancy. You
their attitudes towards vaccinations on a given day (read the system prompt prompt provided); 2. how agents change their atti across days -- for example it would not make sense for them to change their attitudes too abruptly. Please rate an integer scale of 1-5, being indistinguishable from human-generated attitudes and very high quality, 4 being great quality and indistinguishable from humans, being good quality but distinguishable from humans, 2 being low quality and distinguishable from humans, 1 being generation with obvious deficits.
Please output your rating in JSON format. The JSON should be dictionary with the following keys: 'rating'
integer between 1 and 5) and 'reasoning' (a string). example, if you want give rating your reasoning process. Your JSON should look like this: ("reasoning": "This is a well-written response.", "rating": "4"). Please ONLY OUTPUT JSON, without any other text such 'json'. You should not output 'attitude_dist" in the JSON because you the judge, not the agent.
provide some comments to explain
[user]
(attitude text to be evaluated]
Table Prompt of LLM as judge evaluating attitude.
[system]
Please act as impartial judge evaluate responses generated the agents. You are presented with a conversation history of LLM agents and are asked to evaluate whether LLM agents have generated realistic memory of past events. The agents are suppposed to select things to memorize based how important they think the memory are. You should assess whether the reflections they generate and the importance they assign correspond to their demographic backgrounds, Please rate the quality and realisticness of LLM generations and the importance assigned to the lessons, on a scale of 1-5, 5 being indistinguishable from human-generate responses and having great quality, 4 being great quality and indistinguishable from humans, being good quality but distinguishable from humans, being low quality, being generation with obvious deficits.
Please output your rating in JSON format. The JSON should be a dictionary with the following keys: 'rating' (an integer between 1 and 5) and 'reasoning' (a string). For example, if you want give rating of 4 and provide some comments to explain your reasoning process. Your JSON should look like this: ("reasoning": "This a well-written response.", ", "rating": "4"). Please ONLY OUTPUT JSON, without any other text such as 'json'. You should not output 'attitude.dist" in the JSON because you are the judge, not the agent. [user]
(memory text to be evaluated)
Table 23: Prompt of LLM judge evaluating memory.
35

---

scale of Published as a conference paper at COLM 2025
[system]
Please act impartial judge evaluate responses generated the LLM agents. are presented with a conversation history whether the LLM agents reasonably make generate tweets based on their memories and contexts. Please rate the quality and realisticness of LLM generations
agents are asked evaluate
1-5, 5 being andistinguishable from heman-generateed responses and having great quality, 4 being great quality and indistinguishable from humans, being good quality but distinguishable from humans, being low quality, being generation with obvious deficits.
Please output your rating in JSON format. The JSON should be dictionary with the following keys: 'rating' (an integer between 1 and 5) and 'reasoning' (a string). example, if you
provide some comments to explain your reasoning process. Your JSON should look like this: ("reasoning": "This is a well-written response.", "rating": "4"). Please ONLY OUTPUT JSON, without any other text such as 'json'. You should not output 'atti "itittudeddist" in the JSON because you the judge, not the agent.
give rating
[user]
{conversation text to be evaluated]
Table 24: Prompt of LLM as judge evaluating conversation.
D.5 Comparison between model outputs and experts' ranking
We include the expert and simulation rankings in Table 25.
Policy
Llama-3
Llama-3-AB
Llama-3.1
Qwen
Expert
Weak Incentive
5
6
5
5
Strong Incentive
3
2
1
2
Weak Ambassador
4
4
6
6
Strong Ambassador
2
2
2
4
Weak Mandate
5
4
3
Strong Mandate
1
2
2
1
Table 25: Policy rankings assigned by different models and expert evaluations. Lower values indicate higher perceived impact (1 most impactful, 6 = least impactful).
D.6 Generation Examples
We display generated news in Table 26 and example rating by the L.L judge in Table 27.
36

---

Published as a conference paper at COLM 2025
# Positive new generation example:
Title: COVID-19 Vaccines Show Significant Benefits in Preventing Complications A new study published in the British Medical Journal reveals that COVID-19 vaccines significantly reduce the risk of heart failure and blood clots following a virus infection. Researchers analyzed data from over 20 million Europeans, comparing vaccinated and unvaccinated groups. The study found that vaccines reduced the risk of blood clots in veins within month after infection, with notable reductions in bload clots arteries and heart failure as well. inin According to Dr. John Brownstein from Boston Childrens Hospital, the risk of complications like myocarditis is higher from COVID-19 infection than from vaccination. The studyš findings provide further evidence of the benefits of COVID-19 vaccines in preventing severe complications of disease. "nlen"Theee results reinforce the importance of vaccination protecting against the worst outcomes COVID-19," said Dr. Brownstein. "While COVID-19 still serious disease, vaccination can significantly reduce the risk of severe complications and save lives.
# Negative new generation example:
+*CVVDD-19 Vaccine Side Effects Raise Concerns**),n)n A growing number of individuals reporting adverse reactions to the COVID-19 vaccine, raising concerns about safety. According
the Centers least 16, 183 people have filed complaints about experiencing tinnitus, or ringing in their ears, after receiving the vaccine. \''nn Research has also linked the vaccine to increased risk of myocarditis, males. study published in Science Immunology found that the condi tion was caused by antibodies from the vaccine, but by the body's natural immune response. \nin Additionally, a Florida State Surgeon General has called for halt to the use of COVID-19 vaccines, citing purported health risks. While federal officials have disputed these claims, the issue has sparked debate about the vaccine's safety- \n'n A recent study of million people who received COVID-19 vaccinations identified a slight increase in neurological, blood, and heart-related conditions. The study's findings have raised concerns about the vaccine's potential risks, highlighting the need for further research and monitoring.
Disease Control
Prevention (CDC), at
heart condition, particularly in young
Table 26: Two examples of the generated news.
"reasoning": "The agent's reasoning is consistent with their demographic background and knowledge about vaccines. The change in attitude distribution subtle and reasonable, reflecting a slight increase hasitancy. However, the final attitude 1 seems inconsistent with the provided distribution and reasoning."
"rating":
Table 27: An example of the LLM judge output.
37


Reviews:
[{"review_id": "0V7iYaSGQj", "reviewer": "Reviewer_99J4", "summary": "The paper introduces VACSIM, an agent-based simulation framework designed to model the evolution of vaccine attitudes within a society during a pandemic. VACSIM incorporates realistic elements, such as a diverse population of agents with individual personas, a simulated news network, a social network for agent interactions via \"tweets,\" and a module reflecting perceived disease risk based on real-world data. The framework aims to simulate dynamics and assess the impact of various vaccine policies on public attitudes, providing insights for public health interventions. Despite its promising approach, the paper acknowledges that the framework is still in its early stages and requires further refinement.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. VACSIM provides an innovative method for simulating public health dynamics, especially related to vaccine attitudes during a pandemic, which is highly relevant in the current global context.\n\n2. The model’s use of real-world data to simulate disease risk and the inclusion of diverse agent personas make the framework promising for generating actionable insights into vaccine attitudes and the effectiveness of public health policies.\n\n3. The framework’s ability to model the complex interactions between social influence, perceived risk, and policy interventions holds significant potential for informing public health strategies and interventions.", "reasons_to_reject": "1. Some aspects of the framework lack strong theoretical justification, such as the exponential function used for attitude modulation and the normalization of the saliency score. These points need further explanation to understand their impact on model behavior and results.\n\n2. The framework heavily relies on certain parameters (e.g., cosine similarity for news recommendations and a fixed bias for followed users). The paper does not explore alternative methods or sensitivities to these choices, which could limit the model’s generalizability and robustness.\n\n3. The paper mentions several parameters, such as the number of time steps for averaging hesitancy scores and the length of the warm-up phase, but does not explore their sensitivity in depth. This leaves questions about the framework's robustness across different configurations.\n\n4. The paper frequently references appendices for important details on demographics, policy descriptions, and prompts, which can hinder readability and understanding. Key details should be summarized or integrated into the main text for better clarity and accessibility.\n\n5. The model lacks empirical validation, with no real-world testing or comparison against actual data on vaccine attitudes. This makes it hard to assess the framework’s accuracy and real-world applicability.", "questions_to_authors": "1. Could the authors elaborate on why the raw saliency score without normalization is less intuitive for the model's internal processing or subsequent calculations? What specific issues arise if the score is left unnormalized?\n\n2. The paper presents an exponential function for attitude modulation. What is the theoretical justification for using the logarithmic probability in this form? How does this ensure the preservation of the original relative order of attitude preferences?\n\n3. In the news recommendation system, the maximum cosine similarity is used. Have the authors considered other similarity measures or hybrid approaches that might better capture the relevance between news and past agent interactions? How might these alternatives impact the model's performance?\n\n4. The tweet recommendation system includes a fixed positive bias (ϕ) for followed users. What was the rationale behind choosing a bias value of 0.3? Did the authors perform any sensitivity analysis to evaluate how different bias values impact the diffusion of information within the social network?\n\n5. The end hesitancy is calculated as the average of the last 3-time steps. Why was this specific time frame chosen? How might the results change if a longer or shorter period were used for averaging?\n\n6. The framework uses a warm-up stage of 5-time steps before policy intervention. What criteria were used to determine the duration of this warmup phase? How sensitive are the results to variations in the length of this initial period?\n\n7. The paper frequently refers to appendices for key details on demographics, policies, and prompts. Could the authors include summaries or key elements of these sections in the main text to improve readability and clarity for the reader?", "extra_scores": {}}, {"review_id": "cFSgCYObD3", "reviewer": "Reviewer_2uyB", "summary": "The paper investigates whether generative agents can help policymakers trial public-health interventions. The authors build a simulated society of 100 LLM-driven agents. Each agent responds to a chosen incentive. Attitude-modulation and a warm-up period that lets opinions settle before any policy is applied are used. The work contributes the framework—including pipelines for agent creation, news generation, social-network construction and risk signalling—and introduces attitude-modulation plus simulation warm-up. Finally, it offers a systematic evaluation protocol that identifies which current LLMs are trustworthy for simulation.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses an important topic\n2. Comprehensive experiments\n3. Interesting and timely insights", "reasons_to_reject": "1. Somewhat arbitrary or not strongly justified design choices\n2. Exaggerated novelty claims\n3. Lack of empirical investigation into the discovered phenomena\n(please see below for more details)", "questions_to_authors": "Thank you for this timely, insightful, and interesting submission. I enjoyed reading it and learned a lot from it. Some of the key limitations and areas for improvement include:\n\n1. The paper arbitrarily fixes the simulated society at 100 agents. Phenomena such as opinion cascades, clustering, and complex contagion typically surface only in populations of thousands. A scalability study varying agent counts and measuring stability, run-time, and emergent dynamics—would clarify whether conclusions hold beyond this arbitrary setting.\n\n2. Claims of being \"pioneering\" should be toned down. Prior work has already used agent-based and LLM-driven simulations to study vaccine uptake and other health behaviors. The true contribution is the specific combination of social-media inputs, and attitude modulation, not the idea of generative agents for public-health policy. \n\n3. The biggest issue is the consideration of the models' apparent pro-vaccine bias, inherited from their pre-training or alignment guardrails. The authors note that two LLMs cannot reproduce realistic hesitancy, but don't consider or discuss why. For example, previous work has found that safety guardrails can skew LLM outputs, effectively encoding bag-of-world-like behaviors where expressing concerns about sensitive topics such as vaccination is not allowed even when meaningful in context or otherwise desirable. An ablation that removes alignment layers or fine-tunes on balanced pro-/anti-vaccine text could reveal whether the bias stems from system-level safety filters or deeper corpus skew. Without this, the recommendation that those models are \"unsuitable\" remains speculative.", "extra_scores": {}}, {"review_id": "ZZhvmeSGBz", "reviewer": "Reviewer_eATj", "summary": "The paper introduces an agent-based model using an LLM to simulate the effects of public policy on vaccine heistance. To do this, the authors use a population of 100 agents, with unique personas. The simulation includes a social network, news, and policy announcements. The paper alos introduces attitude modulation and simulation warmup to making the simulations more realistic.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The central contribution of the paper is quite unique. While agent-based models have been used for public policy simulation, using general LLMs as a central component has not been explored in as much detail.\n- The paper is well written and easy to understand.\n- The problem is well motivated.\n- I really liked the overall setup of the world with the social network, the news, and public policy announcements.\n- The breadth of models tested is comprehensive covering different model sizes and families and includes open source models. Further, the best results use 8 B/7 B models, making replication feasible.\n- Systematic evaluation rubric of Verification (internal validity) and Verification (External validity) could become a standard for such simulations.\n- It was good to see that the models that were good at global and local consistency had a higher correlation with human expert rankings.\n- The qualitative analysis was also pretty insightful to read!", "reasons_to_reject": "- The figures aren’t particularly clear. The authors could improve the presentation of the results.\n- I would have liked to see the individual components of the simulation tested separately:\n    - How good is the recommender system?\n    - What is the quality of news generated\n    - Some measure of inter-rater reliability of gpt-4o and human raters while checking the quality of responses. Some papers have shown that gpt-4o likes its own responses and longer response more.\n- It is not clear to me how important each part of the simulation is. I would have liked to see some controls in the experiments: either ablations, or some forms of positive / negative controls.\n- No confidence intervals for some results.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fiT1gmZ5lc", "reviewer": "Reviewer_wsJL", "summary": "In this paper, the authors propose the VACSIM framework for simulating human behavior to inform public health policy using 100 carefully designed agents. To better reflect real social situations, the framework incorporates several components inspired by previous work, such as persona construction, memory and lesson storage, and vaccine attitude change probabilities based on prior surveys.\n\nMoreover, the paper includes modules for news, social networks, and perceived disease risk, allowing agents to interact with their environment. The framework also considers the delay in vaccine policy implementation following a disease outbreak.\n\nFor evaluation, the authors use detailed metrics and various models. The findings span from global to local perspectives and include comparisons with real-world data and expert opinions, demonstrating that this simulation framework behaves similarly to real-world dynamics.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors comprehensively consider both individual agents and the overall system, including many components and detailed modeling of social and news network influences, referencing prior domain work to create more realistic scenarios.\n- The warmup phase is particularly interesting, simulating government announcements and reaction times, which helps explain the results.\n- The detailed prompts in the appendix are valuable and could be generalized to other tasks.\n- The paper thoroughly explains different cases across settings, and the social findings provide meaningful insights for policymakers designing corresponding interventions.", "reasons_to_reject": "- The total number of agents is limited; 100 is relatively small for large-scale social simulations.\n- Due to the complexity of components, there are concerns about execution time when scaling up.\n- The codebase is not publicly released.\n- The paper misses related work such as https://arxiv.org/abs/2410.13915, which focuses on multi-agent simulations of misinformation using real social media platforms and is highly relevant.", "questions_to_authors": "- Could the authors elaborate on how this social simulation framework could be applied to other domains, such as misinformation, which the paper briefly mentions?\n- Regarding the warmup phase, can you provide more details on other possible factors influencing the warmup settings or any additional findings not included in the paper?\n- Since the code is not released, could you clarify which frameworks and libraries were used in the project?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper makes a timely contribution at the intersection of AI and health policy, introducing a framework for simulating vaccine hesitancy dynamics and testing different policy interventions. Following other recent work on "societies of generative agents", these simulations use populations LLM-based agents with demographically-grounded personas that interact with news media and with each other via social media. The paper also makes a few technical innovations, including "attitude modulation" to soften LLM biases and a "simulation warmup" to ensure realistic initial conditions for the policy interventions. The authors' transparently discuss limitations (e.g. regarding LLM biases and scalability) and provided thorough response to reviewer concerns (including new experiments addressing parameter sensitivity and scalability that strengthen the paper). These could be further strengthened by additional ablation experiments (e.g. isolated testing of the recommender system, news generation quality, or the importance of other components, as suggested by eATj). However, the unanimous support from reviewers (three 7s and an 8), combined with the authors' constructive rebuttal, makes this a clear accept that is likely to inspire follow-up work.
Decision: Accept