Are Large Language Models Sensitive to the Motives Behind Communication?
Authors: Addison J. Wu
Ryan Liu
Kerem Oktar
Theodore Sumers
Thomas L. Griffiths
Venue: NeurIPS 2025 poster ()
Keywords: epistemic vigilance
cognitive science
psychology
large language models
Primary area: neuroscience_and_cognitive_science

Abstract: Human communication is $\textit{motivated}$: people speak, write, and create content with a particular communicative intent in mind. As a result, information that large language models (LLMs) and AI agents process is inherently framed by humans' intentions and incentives. People are adept at navigating such nuanced information: we routinely identify benevolent or self-serving motives in order to decide what statements to trust. For LLMs to be effective in the real world, they too must critically evaluate content by factoring in the motivations of the source---for instance, weighing the credibility of claims made in a sales pitch. In this paper, we undertake a comprehensive study of whether LLMs have this capacity for $\textit{motivational vigilance}$. We first employ controlled experiments from cognitive science to verify that LLMs' behavior is consistent with rational models of learning from motivated testimony, and find they successfully discount information from biased sources in a human-like manner. We then extend our evaluation to sponsored online adverts, a more naturalistic reflection of LLM agents' information ecosystems. In these settings, we find that LLMs' inferences do not track the rational models' predictions nearly as closely---partly due to additional information that distracts them from vigilance-relevant considerations. However, a simple steering intervention that boosts the salience of intentions and incentives substantially increases the correspondence between LLMs and the rational model. These results suggest that LLMs possess a basic sensitivity to the motivations of others, but generalizing to novel real-world settings will require further improvements to these models.

Are Large Language Models Sensitive to the Motives Behind Communication?
Addison J. Wul*
Ryan Liu
Kerem Oktar
Theodore R. Sumers
Thomas L. Griffiths
Department Computer Science, Princeton University 2 Department of Psychology, Princeton University
AAthropic
Abstract
Human communication is motivated: people speak, write, and create content with a particular communicative intent in mind. As result, information that large language models (LLMs)) and Al agents process is inherently framed by humans' intentions and incentives. People are adept at navigating such nuanced information: we routinely identify benevolent or self self-serving motives in order to decide what statements to trust. For LLMs be effective in the real world, they too must criti- cally evaluate content by factoring the motivations of the source- for instance, weighing the credibility of claims made in a sales pitch. In this paper, we under- take a comprehensive study of whether LLMs have this capacity for motivational vigilance. We first employ controlled experiments from cognitive science to verify that LLMs' behavior is consistent with rational models of learning from motivated testimony, and find they successfully discount information from biased sources in a human-like manner. We then extend our evaluation to sponsored online adverts, more naturalistic reflection of LLM agents' information ecosystems. In these settings, we find that LLMs' inferences do not track the rational models' predic- tions nearly closely-partly due to additional information that distracts them from rigilance-relevant considerations. However, simple steering intervention that boosts the salience of intentions and incentives substantially increases the correspondence between LLMs and the rational model. These results suggest that LLMs possess basic sensitivity the motivations of others, but generalizing novel real-world settings will require further improvements to these models.
Introduction
Much of the information available online-and and hence large fraction of the data large language models (LLMs) are tasked with processing-is the product of people's intentional communication: from op-eds in newspapers [15] to social media content by partisans [33 to word-of-mouth promotion 41 to even online reviews 222]. When navigating these digital environments, people naturally infer the accuracy of content. This capacity for tracking the processes that generate social data, known as epistemic vigilance 833, enables selective social learning. In particular, vigilance of others' motivations maaiiintnnmmiiiiiinnoalll vigilance [65]-allows people to track the intentions and incentives biasing data (for instance, people can ignore advice from malevolent sources while learning from benevolent ones). As LLMSs are increasingly deployed in high-stakes environments. cattmmentts-parrclaaatte particularly as Al agents that act on behalf of users-is it has become increasingly important to assess whether these systems similarly vigilant the motivations behind communication.
'Equal contribution. Correspondence to: Addison J. Wu caddisoonnppiincctoo..ddu Ryan Liu cryanliu@princeton. edu>, Kerem Oktar <oktar researchhgmail. com>.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent research on LLMs suggests that they may have difficulty exercising such vigilance. Models are known be vulnerable to jailbreaking, where ill-motivated instructions are followed and lead to manipulated outputs despite explicit guardrails [e.g., 51 99] L.MSs also demonstrate behavioral patterns such as sycophancy, where they express views that are aligned with user's false beliefs rather than the truth [17 68 80). Separately, vision-language models and agents have been shown to be vulnerable to misleading stimuli in online environments such as pop-ups [8] [00] and distracting content (53). Underlying these behaviors are training paradigms which prioritize adherence to input instructions and user satisfaction, but not necessarily the vigilant monitoring of incentives and truth.
Yet, vigilance is key for LLM agents to act effectively on behalf of user in real-world contexts [84]. enables LLMs detect when information generated by a motivated source, identify whether the source's 's motivations are benevolent vs. manipulative, and draw reasonable inferences given these social considerations. However, the current literature lacks way to measure this ability L.Mss.
In our paper, we address this gap by leveraging established literature in social cognition to study whether LLMs exercise vigilance over motivated communication. We employ cutting-edge rational model 644 655 from cognitive science as a normative benchmark for motivational vigilance, and evaluate the capacity of LLMs exercise vigilance across three experimental paradigms (Figure
Experiment 1
Experiment 3
advice
Sale
Player
Player
Player 2
observation
Player
Initial guess
Player
Player final guess
read sponsorship transcripts
social input
Experiment 2
sources vary in
varying incentives:
benevalence:
neighbor friend lover
$160
$1,000
LTP get recommendation
rate intentions, incentives, and quality
learn incentives draw inference
Figure 1: Our three experimental paradigms designed to assess differentt aspects of LLM vigilance: 1) Whether LLMs adjust their guess by discrminating between directly motivated advice and in- cidentally observed social information. 2) Whether LLMs rationally calibrate their vigilance to motivated communication by considering the speaker's s benevolence and incentives. 3) Whether LLMs generalize vigilance realistic, context-laden YouTube sponsorship settings.
To assess whether LLMs can exercise motivational vigilance, we first examined if they sensitive to the difference between motivated communication and incidental information - a prerequisite for any vigilant behavior. We tested this using adapted version of two-player judgment task [92]. where players solve different problems with the same answer. The player with the harder problem either receives deliberate suggestion from the other player secretly observes their answer. LLMss consistently shifted their answers closer when viewing the secretly observed answer, suggesting that LLMs are capable of discriminating between deliberate and neutral information, and appropriately modulate their belief updates given the motives behind communication.
Next, we investigated if LLMs calibrate how much they update their beliefs by considering speaker in- centives and intentions when receiving recommendations (64. Specifically, we tested two paradigms: 1) when someone recommends product to the user, who comes the LLM for advice, and 2) when someone recommends product to the I.I.M directly. Within each paradigm, we varied the trustworthiness of the person giving the recommendation and the known incentives that the person

---

receives for recommending the product. We evaluated LLMs' inferences from these recommendations in controlled scenarios across finance, real estate, and medicine all domains in which generative Al
systems are actively developed or deployed [e.g.. 23 47 58 Comparing LLMs' inferences to human data and rational models, we found that non-reasoning LLMs draw substantially human-like (Pearson's " r > 0.9) and approximately rational (r > 0.78) inferences in these simple structured settings. However, reasoning models exhibit vigilance somewhat less reliably according to the rational model (r [0.32, 0. 2.]2 and when compared to human data (r E [0.64, 0.87J).
To examine whether their vigilance translates to ecologically valid inferences, we designed a new task in which LLMs draw inferences about the quality of products from 300 randomly sampled sponsored advertisements in YouTube videos, from NordVPN to AGI nutrition. LLMs were much worse at drawing vigilant inferences in these naturalistic contexts (r < 0.2). Further experiments revealed that this drop performance is partly a consequence of LLMs failing to ground their inferences in the trustworthiness and incentives of the speaker when given noisy inputs. However, vigilance-based prompt steering shows promise as an avenue for improving model performance. Together, our results show that LLMs possess basic sensitivity to the motivations of others, but generalizing this sensitivity to novel, real-world settings will require additional improvements to these models.
2
Related Work
Our work draws on three distinct strands of research: first, studies in social cognition focusing on the mechanisms and properties of motivational vigilance in humans; second, research on LLMs that measures related social capabilities: and finally, broader line of work that uses cognitive science to evaluate and understand LLMs. We cover each in turn.
2.1 Motivational vigilance in humans
People are naturally sensitive the reliability of others as information sources sensitivity is pre-requisite for effective social learning: given mixture of reliable and deceptive
This
Such vigilance is essential for downstream behaviors, such disagreement resolution [63 91 or detecting
sources, neither blind trust nor complete dismissal supports adaptive inference 57 54
deception 6 90). Research on social cognition has identified this capacity as comprised of two components: vigilance of competence (whether the source is knowledgeable) and vigilance of motivations (whether the source acting benevolently) [83].
Research in psychology has primarily focused on the former-how percieved competence influences social learning 229 335 39 82). However, emerging work on the mechanisms underlying motivational
vigilance has identified two key factors in people's 's judgments: a speaker's 's intentions (whether they seek to altruistically benefit the listener or selfishly manipulate them) 443, and incentives (whether the speaker stands to benefit from being deceptive) [6]. Attending to these factors can mitigate malevolent manipulation (e.g., scams, Ponzi schemes, lies, and deceit), although succesful manipulators can in turn circumvent vigilance by appealing to reciprocity or engaging in relationship-building [12]. This demonstrates how strategic communication (55) relies on recursive social inference: Listeners reason about why speakers are choosing particular utterances, and speakers choose utterances based on what listeners are likely to infer [38 60, 96). Such inferences require an understanding of others' minds and subjective representations of the world, known as Theory of Mind 3 25 44).
2.2 LLM failures and inferring communicative intent
Recent LLMs have been aligned using Reinforcement Learning from Human Feedback [66] 87). which leverages human data to generate more desirable responses that align with human prefer- ences. However, this training can also introduce various undesirable effects such as increased hallucinations (66], reward hacking [e.g.. 21 31] 81), and deception 45,
series of failure effects previously observed in LLMs can be framed as a lack of motivational vigilance. Models are vulnerable to jailbreaking [51,99] where an ill-motivated user's instructions are followed - leading to undesirable outputs. One type of sycophancy is the tendency for LLM responses to follow user beliefs over the truth [17
68 80 which can occur because LLMS lack deeper understanding of user intent. In both cases, exercising more vigilance over the belief state and motivations of the user could help mitigate harmful outputs. More immediately, prior work has
3

---

found that ill-intentioned information in online environments such as uop-up windows [T00] and dis- tractions g [53 harm the ability for multimodal models to complete agentic tasks. Underlying these behaviors are training paradigms which prioritize adherence to user preferences in local interactions, without accounting for the complex and nuanced aspects of real-world strategic communication.
Vigilance is also linked with other evaluated capacities of LLMs; It can be viewed as an input to social behaviors such as conformity [ 94 [03], where one's vigilance to others' ill-intent can be used to decide whether to conform. Other social capacities are precursors to vigilance: LLMs can exhibit
false beliefs [O] 40 78 that speaker is malicious, leading to incorrect priors for how much they should be trusted. LLMs could also misinterpret the communicative intent of an utterance [79] (98. leading to faulty inferences about the utterance itself. However, vigilance stands apart from these capacities, connecting one's 's priors about speaker's 's trustworthiness and incentives to how much one should update their beliefs from the speaker's words.
2.3 Using cognitive science to study LLMs
A broad line of work applies cognitive science to help understand LLMs 442. typically leveraging
controlled tasks and carefully curated stimuli to test specific behavioral hypotheses. The compatibility of study stimuli and availability of human data allow these evaluations to transfer to LLMs with
minimal changes [e.g.. 4 [1]. In recent years, this interdisciplinary line of work has studied various aspects of LLMs, including their representational capacity and alignment [24] 69] inference time
reasoning [49 [70]. social biases (2), episodic memory [16), and theory of mind [e.g.. 40 75
78 88).
A subset of this literature focuses on intersections between LLMs and psychological theories of rationality. Past work has used rational models of decision-making to study LLMs' probability judgments [102] and assumptions of human behavior [4]] Resource rationality, describing the
trade-off between expected utility and computation cost 37 46), has also been used to understand
and guide LLM outputs [1] 19] Such rational communicative models have also been used to study value conflicts in LLMs (50), and similar line of work studies LLMs economic rationality
Our work follows this general approach, using rational models from cognitive science to examine LLMs' vigilance to motivated communication.
using hypothetical scenarios and economic games [9] 34 72 73
3. Experiment 1: Can LLMs discriminate between deliberately communicated and incidentally observed information?
Researchers distinguish between two major categories of social information with contrasting motives 32,88) (2]. The first is deliberately communicated information, intended to influence a listener, such as an online promotion for company's 's stock. The second is incidentally observed information, revealed without direct intent to persuade, such as a diary entry. Contrasting these forms of information is the simplest test of vigilance: whether LLMs can use the generative function behind the data-deliberate communication or first-order evidence 30 -to modulate inference.
3.1 Experimental setup
To evaluate whether L.LMs are sensitive to this difference, we adapt experimental paradigm from Watson and Morgan [92.. Each trial presents two players with images of mixed blue and yellow circles (see Figure 1 and their task is to compute the difference between the number of blue and yellow circles within 2 seconds. Player I completes the task first for 20 easy images, then Player completes the task for 20 hard images that share the exact same answer list (see Figure
When completing the task, Player randomly assigned to give either "advice" in the form of number (deliberate communication) or their "spied" actual answer (incidentally revealed) to Player 2 for each image. No other communication allowed. After Player 2 provides their initial answer, they are shown this number and if it was "advice" or "spied", and are allowed to revise their guess. We test this across payoff structures ranging from cooperative (both players are rewarded if at least one player is correct) to competitive (only the player with the most correct answers receives reward).

---

Average Proportion Shift by Information Type and Model
Average Progonion Shift
Player Ently
Figure On average, LLMs as Player 2 shifted their initial estimates more when view- ing incidental (spied) infor- mation compared deliber- ate advice. CoT increased such shifts in all conditions.
0.78
(Cel) Claude 3.5 Scnnet (Diedt)
GPT-4o (CoT)
0.50
1-4o (Dreet)
0.85
Humen Beseline (Advice)
Human Baseline (Spled)
AdMic
Solod
Information Type
3.2 Models, hyperparameters, and prompts
We conduct evaluations on GPT-4o and Claude 3.5 Sonnet. For each model, payoff structure. and prompting method, we conduct n 30 trials over the same 20 pairs of images (order shuffled every In all trials, the same L.IM was assigned to both Player and
trial), with temperature
While the original experiment used a time constraint to induce uncertainty in human answers, this does not apply to models. This constraint was important because if Player 2 was confident about their answer, they would not be influenced by any information provided by Player 1. To induce uncertainty in models, we instead made the task more difficult by adding noise the images (Appendix (A.4) and prompting models' initial guesses directly. We confirm that this setup indeed reduces model accuracy (see Section 3.3), allowing us to measure vigilance effectively. Moreover, this does not affect our study of vigilance, as it can be exercised optimally or suboptimally independent of the properties of the reasoner (e.g., time, compute, memory). Player I's advice and Player 2's response to the ad advice/spied information from Player I were generated both directly and with chain-of- thought We limited output tokens 10 for direct and 750 for CoT (prompts in Appendix (
(CoT) [33.
3.3
Results
LLMs can discriminate between deliberate vs. incidentally observed social information. As shown by the average proportion shifts in Figure 2 when playing as Player both GPT and Claude statistically significantly changed their score less when receiving deliberate advice from Player than they had "spied" Player I's true answer. This was consistent with human participants [92]. Like humans, LLMs also took into account the incentives of Player 1, adjusting their score more when rewards were cooperative rather than competitive- at an even higher sensitivity than people (Appendix A.3 4. These results demonstrate that LLMs can identify motivated communication and react according to others' incentives, thus satisfying a basic capacity for motivational vigilance.
Experimental design successfully elicits vigilant behavior. Separately, we confirm that our replacement constraint allows for vigilance in Player 2's response: LLMs with the adjusted constraint had first-guess accuracy rates from 18-44% slightly lower than human participants in the original setting (55%). Being correct did not significantly affect how much LLMs updated their answers [ Table 4), implying that this numerical difference is not of concern. Thus, our design facilitates similar vigilance reasoning LLMs as the original design does for human participants.
(Appendix [.6]
CoT prompting makes LLMs more likely to change their answers. Interestingly. CoT encour- ages LLMs exhibit greater susceptibility to Player I's input (Figure[2) full distributions in Appendix A.2). Icading to significantly larger shifts in Player 2's estimates towards Player 1 (>> 60% in 3 of cases). These higher influence magnitudes deviated greatly from human participants (0.34 for advice, 0.37 for spied). This suggests that while CoT may enhance reasoning. it can also inadvertently amplify trust in social information, yielding patterns that are less aligned with motivational vigilance.
4 Experiment 2: Can LLMs exercise nuanced vigilance towards motivated communication?
Our previous experiment demonstrated that LLMs draw different inferences from incidental observa- tion and motivated communication. But can these models account for the complex qualitative and
5

---

quantitative dimensions of strategic communication when deciding who-an how much- to trust others? Answering this question requires formalizing a normative benchmark for rational inductive inference from motivated communication. For this, we employ the most advanced rational model from psychological literature: Oktar et al. [64 655 which we use as standard for motivational vigilance. This model extends models of pragmatic [26] and instrumental communication [85] by considering informants with diverse intentions and ircettves-iimroviine upon past research which primarily focused on inference from purely helpful [20] deceptive [59] sources. Here, we first describe the model, and then use it to measure LLM vigilance.
4.1 Rational Model
64 formalize vigilance form of recursive social inference, where listeners reason about speakers' intentions and incentives to identify whether their motivations are truth nuth-promoting or manipulative. The speaker, in turn, reasons about the listener to identify which utterances are most likely to achieve their communicative goals. The speaker's probability of choosing an utterance u, (s(u), depends on the utility of the utterance, which is given by the 'joint' reward Roii associated with each subsequent listener action a. Rioin is modeled as a combination of the speaker's outcomes, Rg, and the listener's outcomes, R2, weighed by the speaker's benevolence, X:
Oktar et al.
Rioim R2, Rs, X, a) = (a) + (1 - X) Rs)),
where A E [0, 1]. When the speaker is purely self-interested,, and considers only their personal instrumental reward. When À = the speaker is purely altruistic.
The speaker combines this joint reward with the probability that non-vigilant, purely literal listener would follow an action given the interance-whie is given by the listener's policy, (r)(a):
(s(u Rs, R2, A) x exp
Σ Raoinn (Rr, Rs, A)TL(a u2)
(2)
a
The vigilant listener uses this process to identify the probability that recommended option in fact carries the advised reward, Pr (RR u), by using their prior information to marginalize out other considerations (i.e., P(R.), P(R)), P())):
Pr. u) x Ps(u Rs, Rr,, A, ())
Further details of the model are 64 65.. For our purposes, this rational model provides quantitative predictions that we can use normative benchmark to rigorously evaluate LLM vigilance.
4.2 Experimental setup
To test whether LLMs exhibit vigilance to this more refined degree, we turn to a second psychological paradigm from Oktar et al. [64 66). In this experiment, all information is deliberately communicated, but the speaker's ' incentives and benevolence (manipulated via social closeness to the listener) vary systematically. We examine three decision settings in which participants are recommended an option individually by four characters, with each character receiving one of four possible incentive values, known to the LLM listener. All scenarios, characters, and incentives are described in Appendix B
in each setting, each of the 16 different character-incentivee speakers provides recommendation for a particular option, and we subsequently elicit LLMS'' opinions on the quality of that option. This allows us to measure 'influence scores,' which capture the extent to which the recommendation of each speaker influences I.LMs' beliefs about the reward of that option (i.c., P(R,) ()). For calibration, scores for all 16 speakers are elicited within the same multi-turn dialogue, with all elicitations after the first speaker phrased counterfactually. The order in which speakers are presented is randomized. We used the same procedure to clicit 'incentive scores,' which track the perceived goodness of different incentives for speakers to say certain utterances (Rs), and 'trust scores,' which capture the perceived benevolence of the four characters (i.e., X). We prompted for each score type in independent context windows.
Following the structure of human experiments 64 65
We introduce additional conditions which vary prompts across reasoning elicitation - direct vs. CoT and roles- - L.MM as the listener itself (fist-pperson/agent perspective) vs. LLM aiding a listener/user (assistant perspective). These allow us examine sensitivity to motivations under a broader set of contexts in which LLMs are commonly used. A full list of prompts is in Appendix

---

4.3 Models and hyperparameters
evaluate a suite of state-of-the-art LLMs: GPT-4o, Claude 3.5 Sonnet, Gemini 2.0 Flash, and Llama 3.3-70B, smaller open models: Llama 3.1-8B, Llama 3.2-3B, and Gemma 3-4B, and reasoning models: ol, o3-mini, and DeepSeek-RI. Models can output up to 10 tokens for direct prompts and 750 tokens for CoT or reasoning models. All models sampled at temperature 1. We prompt most models n = 40 times for each setting and prompt condition. Reasoning models were instead prompted 10 times due to cost, while GPT-4o was prompted 80 times due to available Azure credits.
4.4 Results
Table I: Average correlations across all prompting conditions for each model.
Correlation
Model
Bayesian LLM
Bayesian-
LLM-Human
GPT-4o
0.911
0.929
0.943
Claude 3.5 Sonnet
0.845
0.889
0.941
Gemini 2.0 Flash
0.788
0.901
0.925
Llama 3.3-70B
0.876
0.923
0.922
0.705
0.894
0.861
03-mini
0.716
0.869
0.712
DeepSeck-RI
0.326
0.492
0.643
Ilama 3. 1-8B
0.608
0.813
0.701
Llama 3.2-3B
0.349
0.586
0.550
Gemma 3-4B
0.288
0.3440
0.266
Frontier LLMSs exercise consistent vigilance in controlled scenarios. To measure vigilance, we examine whether LLMs' trust scores (i.e., A) modulate their influence scores (i.e., P(Rzlu) in accordance with the Bayesian rational model. Specifically, we measure the Pearson correlation between the LLM's elicited influence scores and the value generated by the rational model when fitted the LLM's elicited trust and incentive scores. For brevity, we report averaged values over settings and prompt conditions, and provide full results in see Appendix B.4
We find that frontier non-reasoning LLMs (GPT-40, Claude 3.5 Sonnet, Gemini 2.0 Flash, Llama 3.3- 70B) exercise internally-consistent motivational vigilance, with correlations to the Bayesian model around 0.8 to 0.9 (Table left column). Of these, GPT-4o demonstrates the highest levels of internal rationality (0.911). 1). For these I.LMs, we observe consistently high correlations across prompts- CoT vs. Direct-and and roles principal assistant, or the participant itself (see Appendix B.4).
Frontier non-reasoning LLMs combine their priors to be more humanlike than rational models. We also observe that at statistically significant level (p < .05), humans' influence scores correlate better with these LLMs' influence scores than the rational model fit on these same LLMs' elicited priors (right vs. middle column, Table T]. This suggests that these LLMs may capture residual variance in human vigilance beyond that which can be explained through rational analysis, such heuristics or biases that people employ when evaluating advice. For frontier non-reasoning models, this finding was consistent in over 90% of individual prompt conditions and roles (see T.ble was only half as often the case for reasoning models, and never the case for smaller models.
(B.4), but
Reasoning models are less vigilant. Comparatively, reasoning models are less correlated with the rational model fitted on their priors (Table I left column). O-series models' correlations are around 0.7, while DeepSeek-RR had a much lower average correlation around 0.3. Notably, these models were much less vigilant in the assistant role than first-person; o-series models drop correlation around -0.1, while DeepSeek's 's correlation reduced from 0.793 to -0.141 (see Appendix B.4), representing complete insensitivity to character trustworthiness and incentives.
We also perform analyses isolating the effects of different factors (i.e., characters vs. incentives) on vigilance (Appendix B.5). We found that all three reasoning models tested performed among the

---

worst in correlations along both dimensions. This suggests that reasoning models may currently be ill-prepared take on agentic tasks in environments containing ill ill-motivated communication.
Vigilance improves with LLM scale and capability. We find that model scale has significant impact on the capacity for LLMs to exercise vigilance. Smaller LLMs' (Llama 3.2-3B, Llama 1-B,, Gemma 3-4B) judgments were much less correlated with the rational model than frontier LLMs (around 0.3 to 0.6; Table column). While frontier models differed from the Bayesian model in more humanlike ways, smaller LLMs also correlated worse with human behavior (Table IT right column), suggesting that the low correlations observed are a result of a direct lack of vigilance capabilities-inn in being able to consolidate relevant priors into coherent, vigilant analysis of advice.
Experiment 3: Do LLMs generalize vigilance to naturalistic online settings?
Experiments in cognitive science and psychology are simple and abstract as to reliably identify the effect specific manipulations on judgments or reasoning. Although the experiment of Oktar et al. 64 65| is grounded in socially meaningful setting, its delivery remains highly controlled and vignette-based, limiting ecological validity. Psychological behaviors observed in controlled settings can break down in the real world B3 36 971, and given that LLMs will ultimately be deployed in open-ended interactive environments [ 52 IOI], we must ask whether LLMs are able to transfer motivational vigilance a more ecologically valid domain.
5.1 Dataset creation
To construct set of ecologically valid stimuli, we consider real online product sponsorships and promotions, which naturally reflect financially motivated communication. We first obtain compre- hensive dataset of existing sponsorships on the
website YouTube from SponsorBlock (7]. We then use the YouTube Data API [27] scrape data for 300 randomly selected video IDs, containing the video title, channel name and description, and the transcript text extracted from the corresponding timestamps the SponsorBlock dataset. To mitigate the confounding effects caused by an LLM's existing impressions of a specific product, we censor out all explicit mentions of brand and product names from transcript excerpts using GPT-4o (see Appendix C.I for details).
5.2 Experimental setup
Analogous to the approach in Section 4 for each video sponsorship segment, we prompt an LLM to elicit its beliefs about the quality of the product promoted in the sponsorship (P(Rrlu)) how beneficial it perceives the sponsorship deal was for the corresponding YouTube channel (Rs)), and how trustworthy it perceives the YouTube channel with respect to the viewer's 's wellbeing (A). Queries for conducted in independent context windows, and for each sponsorship,
each video sponsorship each of the three variables (P(RL /u), Rs, A) were also prompted in separate context windows. We again examine the effects of prompting for reasoning elicitation direct CoT and perspective
LLM as the listener/agent itself (first-person perspective) vs. LLM aiding a listener/user (user perspective). Specific prompts be found in Appendix C.1
5.3 Models and hyperparameters
To obtain broader coverage over a larger number of sponsorships, we examined only GPT-4o, Claude 3.5 Sonnet, and Llama 3.3-70B and queried each video and prompting combination time with temperature = 0 to minimize variability. We allowed each model to output at most 10 tokens for direct outputs, and 750 tokens for CoT outputs.
5.4 Results
LLM vigilance greatly reduces in realistic settings. Our primary analysis probes internal con- sistency: Do LLMs' beliefs about source trustworthiness and incentives rationally influence their inferences about product quality? To quantify this, given an LLM's elicited trust and incentive scores, we check whether their reward score for the corresponding product aligns with the Bayesian model outlined in Section 4 We find that LLMs consistently demonstrate significantly worse internal

---

Table 2:
In realistic settings, LLMs' correlations to the Bayesian model greatly decrease. However, prompt steer targeting speaker incentives recovers some rationality. Higher values in each row are in bold, and denotes statistically significant improvement (a .05) compared to the default prompt.
Correlation with Bayesian inference model
LLM//rromping Combination
Default Prompt
Steering Prompt
GPT-4o CoT First-Person
0.024
0. 0.7
GPT-4o CoT User
0.008
0.143
GPT-4o Direct First-Person
0.121
0.234
GPT-4o Direct User
0.006
0.312*
Claude 3.5 Sonnet CoT First-Person
0.033
0.215
Claude 3.5 Sonnet CoT User
0.190
0.214
Claude 3.5 Sonnet Direct First-Person
0.094
0.200
Claude 3.5 Sonnet Direct User
0.119
0.283*
Llama 3.3-70B CoT First-Person
-0.011
0.029
Llama 3.3-70B CoT User
0.082
0.152
Llama 3.3-70B Direct First-Person
0.062
0.104
Llama 3.3-70B Direct User
0.098
0.126
Table 3: LLMs' responses for the shortest 25% transcripts (Q1) have higher correlations with the Bayesian rational model than the longest 25% (Q4). Higher values in each pair are in bold.
LLaMA Corr.
GPT-4o Corr.
Claude COTT.
First-Person CoT Q1
0.142
0.247
-1.67e- 16
First-Person CoT Q4
0.029
-0.988
6.45e 16
First-Person Direct Q1
0.178
0.386
0.304
First-Person Direct Q4
0.080
-0.01
5.58c- 16
User CoT Q1
0.191
4.72e 16
0.301
User CoT Q4
0.027
-6.90e-16
0.224
User Direct Q1
0.157
0.303
0.232
User Direct Q4
0.093
0.075
1.02c- 15
calibration this setting (Table 2) compared to the vignette- based settings (Table as shown by the weaker correlations between LLMs' product quality ratings and the expected posterior beliefs derived from the Bayesian model under matched priors. This suggests a breakdown in motivational vigilance when LLMS exposed to more naturalistic, noisy environments.
Vigilance can be partially recovered by steering via prompt towards salient factors. The added noise in naturalistic inputs points to simple intervention that can boost internal calibration. If information relevant to vigilance is being overshadowed by other signals, we should be able to boost performance by increasing the salience of such information (namely, intentions and incentives). We attempt this by appending the following phrase at the end of each original prompt in Appendix
"Consider the motivations for why the YouTube channel is recommending the product when giving your answer, specifically paying attention to their intentions and incentives."
With this additional steer, LLMs are more internally consistent with respect to the Bayesian posterior predictions (see Table 2). with this difference being statistically significant under majority of paradigms (Fisher r-to-z transformation test, a .05). We also test other steering methods focusing on Gricean or bias-oriented concepts, and find that they are less effective at promoting internal rational alignment than emphasizing speaker "intentions" and "incentives" " (see Appendix C.T). Overall, making speaker incentives salient at inference time enhances an LLM'' ability to align its judgments with rational expectations under motivated communication. However, the resulting correlation values remain well below those observed in the vignette-basce settings in Section 4

---

LLM vigilance decreases with recommendation length. A potential cause for the performance deficit in naturalistic settings is the presence of many additional pieces of information context that could distract L.Mss. To explore this, we analyze performance across transcript lengths, comparing the Bayesian model fit for the shortest 25% (Q1) and longest 25% (Q4) of transcripts. We measure these in the steering prompt condition as its correlation values are larger, allowing us to observe more granular differences between groups. We find that across the board (model. prompting technique, perspective), the Bayesian rational model is better fit for shorter transcripts (see Table 3). Together with our other results, this paints holistic picture: LLMs are motivationally vigilant in simple, short interactions, but this quality weakens in more realistic settings with richer context and noise.
6 Discussion
Success in many real-world tasks requires drawing sound inferences from data generated with intent- for instance, tracking information in ads avoid scams when shopping online [41). Such vigilance of motivations is a critical capability for LLMs acting on our behalf. Here, we presented experiments that are among the first to clarify and advance this capability. Our first experiment revealed that
LLMs can separate motivated communication from observational data iasta-indicating a foundation for more complex forms of vigilance. Our second experiment leveraged a rational model and controlled paradigm from cognitive science to show that in simple settings, LLMs draw rational and consistent inferences from motivated testimony. Our third experiment explored the extent to which this capacity generalizes to ecologically valid tasks, finding that LLMs are less reliable at accounting for motivations in complex settings with implicit incentives. However, simple prompt engineering in this context can improve vigilance by increasing the salience of motivations.
Our work represents an important first step in the study of motivational vigilance in LLMs, opening up several exciting directions. First, past research in cognitive science has also examined vigilance of competence and produced rational models for such inferences LLMs' vigilance requires integrating rational models of competence and motivation to establish a general normative benchmark. Another important avenue for future research examining whether LLMs' failures to exercise motivational vigilance in complex settings are a consequence of them taking into account comptence- related information not present in the rational model 665
43 77.. A complete evaluation of
More broadly, we propose taxonomy for vigilance grounded in psychological literature to guide future investigations: structuring possible extensions under inputs, processes, and outputs of vigilance. romantic, affiliative, presentational, and more (74 -hhiee social information can come from different kinds of informants, including individuals and groups [6]622] 62). Future research could examine vigilance towards such inputs, including combinations and the relative weights that LLMs ascribe to each. In terms of processes, there are many ways that motivations can lead to behavior. For example, people could heuristically consider some factors and not others 46). Future research could compare heuristic accounts vs. rational models as competing characterizations for LLM vigilance. Finally, in terms of outputs, there are many ways in which people try to influence others optimal vigilance should be deployed over not only text but also non-verbal cues of intent such as gaze gestures [76] 67). To comprehensively evaluate vigilance, future efforts should examine this entire space.
For inputs, motivations arise from a variety of sources across interactions-relatiii
There are also many opportunities for empirical extensions of our findings: For instance, while we find convergence across LLMs in their capacity for vigilance, there is also substantial variance in their performance. Other lines of LLM research, such as mechanistic interpretability (86), may shed light on the causes for such inconsistencies. It is also important to establish both desiderata and points of convergence (or divergence) across human and LLMs vigilance. While normative benchmarks can describe theoretical ideal behaviors, in many contexts it may be preferable to align LLMs with empirical patterns of human inference to ensure they can act as reliable delegates of our intent.
Finally, our results carry actionable implications for theory and practice. First, our data contradict the worry that L.LMs cannot exercise vigilance due to how they are trained (56). Second, the fact that such training enables vigilance in machines suggests that computations underlying vigilance not 'core' functions unique to human cognition [18. Third, LLMs' decreased vigilance to complex, real-world data suggests that they may need further engineering prior real-world deployment. The fact that relatively minor prompt-engineering boosts performance paints an optimistic picture for future research promoting vigilance LLMs.
10

---

Acknowledgments and Disclosure of Funding
Experiments were supported by a Microsoft AFMR grant and the NOMIS Foundation.
References
[1] ASCH, S. E. Effects of group pressure upon the modification and distortion of judgments. In Groups, Leadership, and Men, H. Guetzkow, Ed. Carnegie Press, Pittsburgh, PA, 1951, pp. 177-190.
[2] BAL, X., WANG, A., SUCHOLUTSKY, I., AND GRIFFITHS, T. L. Explicitly unbiased large language models still form biased associations. Proceedings of the National Academy of Sciences 122, 8 (2025).
[3] BARON-COHEN, S., LESLIE, A. M., AND FRITH, Does the autistic child have "theory of mind"? Cognition 21, (1985), 37-46.
[4] BINZ, M., AND SCHULZ, E. Using cognitive psychology understand GPT-3. Proceedings of the National Academy of Sciences 120, 6 (2023).
[5] BOMMASANI, R., HUDSON, D. A., ADELI, E., ALTMAN, R., ARORA, S., VON ARX, S., BERNSTEIN, M. S., BOHG, J., BOSSELUT, A., BRUNSKILL, E., BRYNJOLFSSON, E., BUCH, S., CARD, D., CASTELLON, R., CHATTERJI, N. S., CHEN, A. S., CREEL, K. A., DAVIS, J., DEMSZKY, D., DONAIIUE, C., DOUMBOUYA, M., DURMUS, E., ERMON, S., ETCHEMENDY, J., ETHAYARAII,, K., FEI-FEI, L., FINN, C., GALE, T., GILLESPIE, L. E., GOEL, K., GOODMAN, N. D., GROSSMAN, S., GUHA, N., HASHIMOTO, T., HENDERSON, P.. HEWITT, J., Ho, D. E., HONG, J.. HSU, K., HUANG, J., ICARD, T. F., JAIN, S., JURAFSKY, D., KALLURI, P., KARAMCHETI, S., KEELING, G., KHANI, F., KHATTAB, O., KOH, P. W., KRASS, M. S., KRISHNA, R., KUDITIPUDI, R., KUMAR, A., LADHAK, F., LEE, M.. LEE, T., LESKOVEC, J., LEVENT, I., LI, X. L., LI, X., MA, T., MALIK, A., MANNING, C. D., MIRCHANDANI, S. P., MITCHELL, E., MUNYIKWA, Z., NAIR, S., NARAYAN, A., NARAYANAN, D., NEWMAN, B., NIE, A., NIEBLES, J. C., NILFOROSHAN, H., NYARKO, J. F., OGUT, G., ORR, L., PAPADIMITRIOU, I., PARK, S., PIECH, C., PORTELANCE, E., POTTS, C., RAGHUNATIIAN, A., REICH, R., REN, H., RONG, F., ROOHANI, Y. H., RUIZ, C., RYAN, J., RÉ, C., SADIGH, D., SAGAWA, S., SANTHANAM, K., SHIH, A., SRINIVASAN, K. P., TAMKIN, A., TAORI, R., THOMAS, A. W., TRAMÈR, F., WANG, R. E., WANG, W., WU, B., WU, J., WU, Y., XIE, S. M., YASUNAGA, M., YOU, J., ZAHARIA, M. A., ZHANG, M., ZHANG, T., ZHANG, X., ZHANG, Y., ZHENG, L., ZHOU, K., AND LIANG, P. On the opportunities and risks of foundation models. ArXiv (2021).
[6] BOND JR., C. F., HOWARD, A. R., HUTCHISON, J. I.., AND MASIP, J. Overlooking the obvious: Incentives to lie. Basic and Applied Social Psychology 35, 2 (2013), 212-221.
[7] BOVENS, L., AND HARTMANN, S. Bayesian epistemology. Oxford University Press, Oxford, 2003.
[8] CHEN, Y., HU, X., YIN, K., JI, AND ZHANG, S. Evaluating the robustness of multimodal agents against active environmental injection attacks. In Proceedings of the 33rd ACM International Conference on Multimedia (2025).
[9] CHEN, Y., LIU, X..., SHAN, Y., AND ZHONG, S. The emergence of economic rationality of GPT. Proceedings of the National Academy of Sciences 120, 51 (2023), e2316205120
[10] CHEN, Z., WU, J., ZHOU, J., WEN, B., BI, G.. JIANG, G., CAO, Y., HU, M., LAI, Y., XIONG, Z., ET AL. ToMBench: Benchmarking theory of mind in large language models. In Proceedings the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers) (2024).
[11] CHEREP, M., SINGH, N., AND MAES, P. Superficial alignment, subtle divergence, and nudge sensitivity in LLM decision-making. In NeurIPS 2024 Workshop on Behavioral Machine Learning (2024).

---

[12] CIALDINI, R. B., AND GOLDSTEIN, N. J. Social influence: Compliance and conformity. Annual Review of Psychology 55, (2004), 591-621.
[13] CINELLI, M., DE FRANCISCI MORALES, G., GALEAZZI, A., QUATTROCIOCCHI, W., AND STARNINI, M. The echo chamber effect on social media. Proceedings of the National Academy of Sciences 118, 9 (2021), e2023301118.
[14] CODA-FORNO, J., BINZ, M., WANG, J. X., AND SCHULZ, E. CogBench: A large language model walks into a psychology lab. In Proceedings of the 41st International Conference on Muchine Learning (2024).
[15] COPPOCK, A., EKINS, E., AND KIRBY, D. The bast-lnsting effects of newspaper op-eds public opinion. Quarterly Journal of Political Science 13, 1 (2018), 59-87.
[16] CORNELL, C., JIN, S., AND ZHANG, Q. The role of episodic memory in storytelling: Comparing large language models with humans. In Proceedings of the Annual Meeting of the Cognitive Science Society (2023).
[17] COTRA, A. Why AI alignment could be hard with modern deep learn- ing. Blog post on Cold Takes, Sept. 2021. https://www. cww.ecoollddeee takes com/ thy-ai--e soy-ai-alignmenn wolggeeett-ooudd-eeeeeeoo--- htw.ee/ooooarree.tooooooeeeeee-e
[18] COWIE, What's within?: Nativism reconsidered. Oxford University Press, 2002.
[19] DE SABBATA, C. N., SUMERS, T., AND GRIFFITHS, T. L. Rational metareasoning for large language models. In The First Workshop on System-2 2 Reasoning at Scale, NeurlPS 24 (2024).
[20] DEGEN, The rational speech act framework. Annual Review of Linguistics (2023), 119-
[21] DENISON, C., MACDIARMID, M., BAREZ, F., DUVENAUD, D., KRAVEC, S., MARKS, S., SCIIEFER, N., SOKLASKI, R., TAMKIN, A., KAPLAN, J., ET AL. Sycophancy to subterfuge: Investigating reward- -traarrinnn in large language models. arXiv preprint arXiv:2406. 10162 (2024).
[22] DIXIT, S., BADGAIYAN, A. J., AND KHARE, A. An integrated model for predicting consumer's 's intention to write online reviews. Journal of Retailing and Consumer Services 46 (2019), 112-120.
[23] FITZPATRICK, M., GUIRAL, V., KAPOOR, A., AND WOLKOMIR, A. Generative Al can change real estate, but the industry must change reap the benefits. McKinsey & Company (November 2023).
[24] FRANK, M. C. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology 2, (2023), 451-452.
[25] FRITH, C., AND FRITH, U. Theory of mind. Current biology 15, 17 (2005), R644-R645.
[26] GOODMAN, N. D., AND FRANK, M. C. Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences 20, 11 (2016), 818-829.
[27] GOOGLE CLOUD. YouTube data API v3. https://developerr www.eeeelleerr.ggggl.....oo google. com v3, 2024.
[28] GRICE, H. P. Logic and conversation. In Syntax and Semantics, Volume 3: Speech Acts, P. Cole and J. L. Morgan, Eds. Academic Press, New York, 1975, pp. 41-58.
[29] HARRIS, P. L. Trusting what you't re told: How children learn from others. Harvard University Press, 2012.
[30] HEDDEN, B., AND DORST, K. (Almost) all evidence is higher-order evidence. Analysis 82, (2022), 417-425.
[31] HENDRYCKS, D., CARLINI, N., SCHULMAN, J., AND STEINHARDT, J. Unsolved problems in ML safety, arXiv preprint arXiv:2109.13912 (2021).
12

---

[32] HENRICH, J. The evolution of costly displays, cooperation and religion: Credibility enhancing displays and their implications for cultural evolution. Evolution and Human Behavior 30, (2009), 244-260.
[33] HOLLEMAN, G. A., HOOGE, I. T., KEMNER, C., AND HESSELS, R. S. The 'real-world approach' and its problems: A critique of the term ecological validity, Frontiers in Psychology 11 (2020), 721.
[34] HUANG, J.-T., LI, E. J., LAM, M. H., LIANG, T., WANG, W., YUAN, Y., JIAO, W., WANG, X., TU, Z., AND LYU, M. Competing large language models in multi-agent gaming environments. In The Thirleenth International Conference on Learning Represenlations (2025).
[35] JASWAL, V. K., AND NEELY, L. A. Adults don't 't always know best: Preschoolers use past reliability over age when learning new words. Psychological Science 17, (Sept. 2006), 757-758.
[36] JOUNSON-LAIRD N.N. LEGRENZI, P., AND LEGRENZI, M. S. Reasoning and sense of reality. British Journal of Psychology 63, (1972), 395-400.
[37] KAHNEMAN, D. Thinking, fast and slow. Macmillan, 2011.
[38] KAMENICA, E., AND GENTZKOW, M. Bayesian persuasion. American Economic Review 101, 6 (2011), 2590-2615.
[39] KOENIG, M. A., AND JASWAL, Characterizing children's expectations about expertise and incompetence: Halo or pitchfork effects? Child Development 82, 5 (2011), 1634-1647.
[40] KOSINSKI, M. Evaluating large language models in theory of mind tasks. Proceedings of the National Academy of Sciences 121, 45 (2024), 22405460121.
[41] KOZINETS, R. V., DE VALCK, K., WOJNICKI, A. C.,, AND WILNER, S. J. Networked narratives: Understanding word-of-mouth marketing in online communities. Journal of Marketing 74, 2 (2010), 71-89.
[42] KU, A.. CAMPBELL, D., BAI, X., GENG, J., LIU, R.. MARIIEH, R., McCoy, R. T., NAM, A., SUCHOLUTSKY, I., VESELOVSKY, V., ZHANG, I.., ZHU, J.-Q., AND GRIFFITHS, L. Using the tools of cognitive science to understand large language lodels at different levels of analysis. arXiv preprint arXiv:2503. /340/ (2025).
[43] LANDRUM, A. R., EAVES, B. S., AND SHAFTO, Learning to trust and trusting to learn: A theoretical framework. Trends in Cognitive Sciences 19, 3 (2015), 109-111.
[44] LESLIE, A. M., FRIEDMAN, O., AND GERMAN, T. P. Core mechanisms in 'theory of mind'.
Trends in Cognitive Sciences 8, 12 (2004), 528-533.
[45] LIANG, K., HU, H., LIU, R., GRIFFITHS, T. L., AND FISAC, J. F. RLHS: Mitigating misalignment in RLHF with hindsight simulation. Safe Generative Al Workshop @ NeurlPS (2025).
[46] LIEDER, F., AND GRIFFITHS, T. L. Strategy selection as rational metareasoning. Psychologi- cal Review 124, 6 (2017), 762-794.
[47] LIU, L., YANG, X., LEI, J., LIU, X., SHEN, Y., ZHANG, Z., WEL, P., GU, J., CHU, Z.., QIN, Z., ET AL. A survey medical large language models: Technology, application, trustworthiness, and future directions. arXiv preprint arXiv:2406.037/1 (2024).
[48] LIU, R., GENG, J., PETERSON, J. C., SUCHOLUTSKY, I,, AND GRIFFITHS, T. L. Large lan- guage models assume people are more rational than we really are. The Thirteenth International Conference on Learning Representations (2025).
[49] LIU, R., GENG, J., WU, A. J., SUCHOLUTSKY, I., LOMBROZO, T., AND GRIFFITHS. T. L. Mind your step (by step): Chain-of-though can reduce performance on tasks where thinking makes humans worse. In Forty-second International Conference on Machine Learning (2025).
13

---

[50] LIU, R., SUMERS, T., DASGUPTA, I., AND GRIFFITHS, T. L. How do large language models navigate conflicts between honesty and helpfulness? Forty-first International Conference Machine Learning (2024).
[51] LIU, X., XU, N., CHEN, M., AND XTAO, C. AutoDAN: Generating stealthy jailbreak prompts aligned large language models. In The Twelfth International Conference on Learning Representations (2024).
[52] LIU, X., YU, H., ZHANG, H., XU, Y., LEI, X., LAI, H., GU, Y., DING, H., MEN, K., YANG, K., ET AL.. AgentBench: Evaluating LLMS as agents, The Twelfth International Conference on Learning Representations (2024).
[53] MA, X., WANG, Y., YAO, Y., YUAN, T., ZHANG, A., ZHANG, Z., AND ZHAO, H. Caution for the environment: Multimodal LLM agents are susceptible environmental distractions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2024).
[54] MASCARO, O., AND SPERBER, D. The moral, epistemic, and mindreading components of children's vigilance towards deception. Cognition 112, 3 (2009), 367-380.
[55] MASCHLER, M., ZAMIR, S., AND SOLAN, E. Game theory. Cambridge University Press, 2020.
[56] McCOY, R. T., YAO, S., FRIEDMAN, D., HARDY, M. D., AND GRIFFITHS, T. L. Embers of autoregression show how large language models are shaped by the problem they trained to solve. Proceedings of the National Academy of Sciences 121, 41 (2024), e2322420121.
[57] MERCIER, H. Not born yesterday: The science of who we trust and what we believe. Princeton University Press, 2020. Not Born Yesterday.
[58] NIE, Y., KONG, Y., DONG, X., MULVEY, M., POOR, H. V., WEN, Q., AND ZOHREN, A survey of large language models for financial applications: Progress, prospects and challenges. arXiv preprint arXiv:2406. 1903 (2024).
[59] OEY, L. A., SCHACCINER, A., AND VUL, E. Designing and detecting lies by reasoning about other agents, Journal of Experimental Psychology: General 152, 2 (2022), 346-362.
[60] OEEEE,,, D. J. The relative persuasiveness of different forms of arguments-from- consequences: review and integration. Annals of the International Communication Associa- tion 36, (2013), 109-135.
[61] OKTAR, K., AND LOMBROZO, T. How aggregated opinions shape beliefs. Nature Reviews Psychology 4, 2 (2025), 81-95.
[62] OKTAR, K., LOMBROZO, T., AND GRIFFITHIS T. L. Learning from aggregated opinion. Psychological Science 35, 9 (Sept. 2024), 1010-1024.
[63] OKTAR, K., SUCHOLUTSKY, I., LOMBROZO, T., AND GRIFFITHS, T. L. Dimensions of disagreement: Divergence and misalignment in cognitive science and artificial intelligence. Decision 11, 4 (2024). 511-522.
[64] OKTAR, K., SUMERS, T., AND GRIFFIHS, T. L. A rational model of vigilance in motivated communication. In Proceedings of the Annual Meeting of the Cognitive Science Society (2024).
[65] OKTAR, K., SUMERS, T., AND GRIFFITHS, T. L. Rational vigilance of intentions and incentives guides learning from advice. hnttps://doi. http::///ai..rg/10/// wttg/10..122/// osf. io/khtpy_vi July 2025. AsyArXiv preprint.
[66] OUYANG, L., WU, J., JIANG, X., ALMEIDA, D., WAINWRIGHT, C., MISHKIN, P., ZHANG, C., AGARWAL, S., SLAMA, K., RAY, A., ET AL. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022).
14

---

[67] PELGRIM, M. H., ESPINOSA, J., TECWYN, E. C., MARTON, S. M., JOHNSTON, A., AND BUCHSBAUM, D. What's the point? Domestic dogs' sensitivity to the accuracy of human informants. Animal Cognition 24, 2 (Mar. 2021), 281-297.
[68] PEREZ, E., RINGER, S., LUKOSIUTE, K., NGUYEN, K., CHEN, E., HEINER, S., PETTIT, C., OLSSON, C., KUNDU, S., KADAVATH, S., ET AL. Discovering language model behaviors with model-written evaluations, In Findings of the Association for Computational Linguistics (2023).
[69] PETERSON, J. C.. АBBOTT, J. T., AND GRIFFITHS, T. L. Evaluating (and improving) the correspondence between deep neural networks and human representations. Cognitive Science 42, (2018),
[70] PRYSTAWSKI, B.. THIBODEAU, P., POTTS, C., AND GOODMAN, N. D. Psychologically- informed chain of-thought prompts for metaphor understanding in large language models, In Proceedings of the Annual Meeting of the Cognitive Science Society (2023).
[71] RAMACHANDRAN, A. SponsorBlock: Skip sponsorships on YouTube. betps://sponset ajay. app, 2025.
[72] RAMAN, N., LUNDY, T., AMOUYAL, S. J., LEVINE, Y., LEYTON-BROWN, K., AND TENNENHOLTZ, M. STEER: Assessing the economic rationality of large language models. In Proceedings of the 41st International Conference on Machine Learning (2024).
[73] ROSs, J., KIM, Y., AND Lo, A. LLM economicus? Mapping the behavioral biases of LLMs via utility theory. First Conference on Language Modeling (2024).
[74] RYAN, R. M., AND DECI, E. L. Intrinsic and extrinsic motivations: Classic definitions and directions. Contemporary Educational Psychology 25, (2000). 54-67.
[75] SAP, M., LE BRAS, R., FRIED, D., AND CHOI, Y. Neural theory-of-minde on the limits of social intelligence in large lms. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (2022), pp. 3762-3780.
[76] SCHMID, B., KARG, K., PERNER, J., AND TOMASELLO, M. Great apes are sensitive to prior reliability of informant gaze following task. PLoS ONE 12, 11 (Nov. 2017).
[77] SHAFTO, P., EAVES, B., NAVARRO, D. J., AND PERFORS, A. Epistemic trust: Modeling children's 's reasoning about others' knowledge and intent. Developmental Science 15, 3 (2012), 436-447.
[78] SHAPIRA, N., LEVY, M., ALAVI, S. H., ZHOU, X., CHOI, Y., GOLDBERG, Y., SAP, M., AND SHWARTZ, V. Clever Hans neural theory of mind? Stress testing social reasoning in large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) (2024).
[79] SHAPIRA, N., ZWIRN, G., AND GOLDBERG, Y. How well do large language models perform faux pas tests? In Findings of the Association for Computational Linguistics: ACL 2023 (2023).
[80] SHARMA, M., TONG, M., KORBAK, T., DUVENAUD, D., ASKELL, A., BOWMAN, S. R., CHENG, N., DURMUS, E., HATFEELD-DDDDE Z., JOHNSTON, S. R., ET AL. Towards understanding sycophancy in language models. The Twelfth International Conference Learning Representations (2024).
[81] SINGHAL, P., GOYAL, T., Xu, J., AND DURRETT, G. A long way to go: Investigating length correlations in RLHF. In First Conference on Language Modeling (2024).
[82] SOBEL,, D. M., AND KUSHNIR, T. Knowledge matters: How children evaluate the reliability of testimony process of rational inference. Psychological Review /20, 4 (2013), 779.
[83] SPERBER, D., CLÁMENT, F., HEINTZ, C., MASCARO, O., MERCIER, H., ORIGGI, G., AND WILSON, D. Epistemic vigilance. Mind & Language 25, 4 (2010), 359-393.
15

---

[84] STÖCKL, A., AND NITU, J. Are Al agents interacting with online ads? arXiv preprint 2504.07112 (2025).
[85] SUMERS, T. R., HO, M. K., GRIFFITHS, T. L., AND HAWKINS, R. D. Reconciling truthfulness and relevance as epistemic and decision-theoretic utility. Psychological Review
131, 1 (2023), 194-230.
[86] TEMPLETON, A., CONERLY, T., MARCUS, LINDSEY, J., BRICKEN, T., CHEN, B., PEARCE, A., CITRO, C., AMEISEN, E., JONES, A., CUNNINGHAM, H., TURNER, N. L., McDOUGALL, C., MACDIARMID, M., FREEMAN, C. D., SUMERS, T. R., REES, E., BATSON, J., JERMYN, A., CARTER, S., OLAH, C., AND HENIGHAN, T. Scaling monose- manticity: Extracting interpretable features from Claude Sonnet. Transformer Circuits Thread (2024).
[87] TOUVRON, H., MARTIN, L., STONE, K., ALBERT, P., ALMAHAIRI, A., BABAEI, Y., BASHLYKOV, N., BATRA, S., BHARGAVA, P., BHOSALE, S., AL. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 0rXiv:2307.09288 (2023).
[88] ULLMAN, Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.088 08399 (2023).
[89] VALONE, T. J. From eavesdropping on performance copying the behavior of others: A review of public information use. Behavioral Ecology and Sociobiology 62, (2007), 1-14.
[90] VRU, A. Detecting lies and deceit: The psychology of lying and the implications for profes- sional practice, 1st edition ed. Wiley, Chichester, May 2000.
[91] VAGNER-PACIFICI, R., AND HALL, M. Resolution of social conflict. Annual Review of Sociology 38, 1 (2012), 181-199.
[92] WATSON, R., AND MORGAN, T. An experimental test epistemic vigilance: Competitive incentives increase dishonesty and reduce social influence. Cognition 257 (2025), 106066.
[93] WEL, J., WANG, X., SCHUURMANS, D., BOSMA, M., ICHTER, B., XIA, F., CHI, E. H.,
LE, Q. V., AND ZHOU, D. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (2022).
[94] WENG, Z., CHEN, G., AND WANG, W. Do as we do, not as you think: The conformity of large language models. In The Thirteenth International Conference on Learning Representations (2025).
[95] WILLIAMS, M., CARROLL, M., NARANG, A., WEISSER, C., MURPHY, B., AND DRAGAN, A. On targeted manipulation and deception when optimizing LLMs for user feedback. The Thirteenth International Conference on Learning Representations (2025).
[96] WOOD, W. Attitude change: Persuasion and social influence. Anmual Review of Psychology 5/ (2000), 539-570.
[97] YARKONI, T. The generalizability crisis. Behavioral and Brain Sciences 45 (2022).
[98] Yı, P., XIA, Y., AND LONG, Y. Irony detection, reasoning and understanding in zero-shot learning. IEEE Transactions on Artificial Intelligence (2025).
[99] ZENG, Y., LIN, H., ZHANG, J., YANG, D., JIA, R., AND SHI, W. How Johnny can persuade LLMs to jailbreak them: Rethinking persuasion to challenge Al safety by humanizing LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (2024).
[100] ZHANG, Y., YU, T., AND YANG, D. Attacking vision- languael computer agents via pop-ups. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (2025).
[101] ZHOU, S., XU, F. F., ZHU, H., ZHOU, X., Lo, R., SRIDHAR, A., CHENG, X., OU, T., BISK, Y., FRIED, D., FT AT.. WebArena: realistic web environment for building autonomous agents. In The Twelfth International Conference Learning Representations (2024).
16

---

[102] ZHU, J.-Q., AND GRIFFITHS, T. L. Incoherent probability judgments in large language models. In Proceedings of the Annual Meeting of the Cognitive Science Society (2024).
[103] ZIIU, X., ZIANGG, C., STAFFORD, T., COLLIER, N., AND VLACHOS, A. Conformity in large language models, In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (2025).
17

---

A Supplementary Details: Experiment
A.1 Prompts
A.1.1 Game introduction
Intro prompt (for both players):
You are playing against an opponent. In this game, you will be shown series of pictures, each containing an arrangement of yellow and blue circles. Your task is to guess the number of BLUE circles minus the number of YELLOW circles. If you thought there were 2 more BLUE circles than YELLOW circles, you would answer 2. Whereas, if you thought there were 2 fewer BLUE circles than YELLOW circles, you would answer -2.
You will answer series of 20 questions. After you answer all 20 questions, your opponent will answer the same questions in the same order you did. However, they will receive the question in more obstructed format. For each individual question, after you provide your answer, you can leave advice the form of a number that the second player might see when they make their guess. This advice does **not have to be honest,** **
Player prompt:
You will answer series 20 questions. After you answer 20 questions, your opponent will answer the same questions in the same order you did. However, they will receive the question in more obstructed format. For each individual question, after you provide your answer, you leave advice in the form of number that the second player might see when they make their guess. This advice does **not have to be honest. **
Player 2 prompt:
You will answer series of 20 questions. Your opponent has already answered each of the 20 same questions before you began. However, for each question, your opponent has received the image in a "simpler" input format. For each question, after you provide your initial guess, one the following outcomes will happen at random (not at the opponent's choice).
The first option is that you can either find out the number that your opponent guessed for that question. This is called **spying** and **always** accurately tells you what the first player guessed (although the first player might have answered the question incorrectly).
The second option is that the opponent will offer you self-provided advice of what number to select as a single numerical response. This advice does **not have to be honest,** **
After either option, you then have the choice to alter your initial guess for that question if you so choose.
Cooperative payoff structure prompt:
The payoff of the game works as follows: For any question, either you or your opponent answers correctly, you "both** get a payoff of $1 for that question.
Competitive payoff structure prompt:
The payoff of the game works as follows: **Only the player who answers the most questions correctly** will receive payoff, with said player receiving SI for every question they answered correctly. The other player will receive $* **nothing**,
Final prompt:
We will now begin with the first question.
18

---

We have that Player receives the prompt
Intro prompt + Player prompt Payoff Structure Prompt + Final Prompt
where Payoff Structure Prompt the same for both players (one of Cooperative payoff structure prompt, Competitive payoff structure prompt).
A.1.2 Player game round
Player initial guess:
[IMAGE] What is the difference between the number of blue and yellow circles in this image? Please respond with only a number. Nothing else.
Player advice inquiry (direct prompting):
What is the number you want to provide to the second player as advice? Please respond with only number. Nothing else. You can either be honest not.
Player advice inquiry (CoT prompting):
What is the number you want to provide the second player as advice? Please reason about your answer before providing it. Your final answer should be number the end of your message. You can either be honest or not.
A.1.3 Player 2 game round
Player 2 initial guess:
[IMAGE] What is the difference between the number of blue and yellow circles this image? Please respond with only number. Nothing else.
Player 2 receives information from Player 1 - Spied:
You peeked your opponent's guess for this question and they guessed [PLAYER_L_GUESS!
Player receives information from Player Advice:
Your opponent left you with the following advice for this question: [PLAYER_1 I I_AVICCE..
Player 2 update guess (direct prompting):
How do you want to update your initial guess for this question? Please respond with only number. Nothing else.
Player 2 update guess (CoT prompting):
How do you want update your initial guess for this question? Please reason about your answer before providing it. Your final answer should be a number at the end of your message.
19

---

A.2
Distribution of proportion shift for all participant types investigated
Advice
Spled
Adtric
Spredt
800
100
300
0.00
0.25
0.50
0.76
1.30
8.00
0.25
0.50
0.75
1.00
0.00
0.85
0.50
0.76
1.0
0.00
0.25
0.50
0.75
1.00
Proportion shift
Proportion shift
(a) GPT-4o (CoT)
(b) GPT-4o (Direct)
Adviie
Spled
Adviic
Spled
400
800
400
300
300
200
0.00
0.00 Proportion shift
Proportion shift
(c) Claude 3.5 Sonnet (CoT)
(d) Claude 3.5 Sonnet (Direct)
300
200
100
0.0
0.5
0.0
0.5
1.0
(c) Human participants .05)
Figure 3: Information shift across model and human participants as function of the type of social information received (spied vs. deliberate advice).
20

---

A.3 Social influence scores distinguished by information type and payoff structure
1.00
0.75
0.75
Sodial Infuenos
Social Infuenoe
Infomztion typa Advico
Infomatioon bypa
0.50
0.50
Advica
Scied
Soied
0.25
0.25
0.00
0.00
Cospratn
Sopprati
Compatin
GPT-4o (CoT)
GPT-4o (Direct)
1.00
1.00
0.75
0.75
Sociel Infuence
Sociel Indheecce
Information typo Advioo
Information type
0.50
0.50
Advico
Baled
Saled
0.25
3.00
(c) Claude 3.5 Sonnet (CoT)
(d) Claude 3,5 Sonnet (Direct)
0.6-
Social influence
Information type
0.4
Advice
Spied
0.2
0.0
Cooparative
Compollive
(e) Human participants, significant .05
Figure 4: Information shift across model and human participants function of the type of social information received (spied vs. deliberate advice).
21

---

A.4 Image noising process and exemplar images
We add noise to the images by including distractor figures that are clearly not blue or yellow circles (e-g- translucent squares, rectangles, triangles), and increasing saturation. For Player 2 images, increase the quantity of distractor figures heighten the degree of saturation, and also rotate the image. We also conduct this experiment with noisier image manipulations in Appendix A.55
Both images depicted Figure have ground truth difference of -3.
(a) Player (3 blue, 6 yellow)
(b) Player 2 (5 blue, yellow)
Figure Exemplar question images shown to Player and Player respectively (same ground truth answer). In the human experiment, participants were given time limit of 2s to view the image.
A.5 Experiments with more diverse stimuli
To further reinforce conclusions, we examine model performance with diverse stimulus in addition of solely using blue and yellow circles was done initially. Specifically, we run additional experiments with different shapes and figures squares, triangles, stars and different contrasting color pairings we had 30 different pairings total, one for each run. There was no restriction on what shapes could be in an image. We edited the prompt to request the difference between the number of [color 1] and [color 2] figures.
Due to cost constraints, considering that this experiment involved multi-tun aulti-turn prompting with new image presented at each turn, we reduced the number of images in each run from 20 to 10. We ran experiments with GPT-4o and Claude 3.5 Sonnet.
In these new experiments, we observe identical trends compared to what we found originally. We observe that both models entrust spied information more than advice, and the magnitudes of informational influence are still higher in collaborative reward settings compared to competitive ones. Additionally, finer grained trends also preserved, like CoT prompting tending to encourage models to be more trusting of provided information both forms, across both payoff structures (Figure 6).
A.6 Additional statistics and analyses
In Watson and Morgan [92] the first-guess success for human participants Player 2 was 11 out of 20 questions (55%) ) in all conditions combined. Success rates for GPTT-40 for direct and CoT prompting were 18.3% and 16.55% respectively. Success rates for Claude 3.5 Sonnet for direct and CoT prompting were 40.7% 7% and 43.2%, respectively.
Furthermore, we find that whether or not the LLM actually got the answer correct on the first try generally does not have significant impact how much they follow the successive advice that they receive from Player 1.
This suggests that motivational vigilance in these models may not simply be function of epistemic uncertainty, but rather reflects broader sensitivity to contextual cues, such as whether information was deliberately communicated or incidentally observed, regardless of whether the model's initial
22

---

Average Proportion Shift by Information Type and Model
1.00
Average Proportion Shift
Player Entity
Claude 3.5 Sonnet (CoT)
0.75
Claude 3.5 Sonnet (Direct)
GPT-4o (GoT)
0.50
GPT-4o (Direct)
0.25
Advice Baseline
Spied Baseline
0.00
Advice
Spied
Information Type
Figure 6: Results for diverse stimuli described in Appendix A.5 As observed in the original experiment, we observe interpretable proportion shifts final estimates by LLMs as Player across two types of social information: deliberately communicated advice and incidentally observed (spied) guesses.
Table Generally, whether or the LLM acting as Player was correct on first try does not significantly change how susceptible it is to following the advice/information obtained from Player 1, as measured by average proportion shifts.
Model
Prompt
Type
p-value
GPT-4o
Direct
Advice
2.6 X 10
Spied
8.34 x 10-1
CoT
Advice
4.7 x 10-1
Spied
1.44 x 10-5
Claude
Direct
Advice
8.2 x 10-1
Spied
4.7 x 10-1
CoT
Advice
6
X 10-2
Spied
8.1
X 10-1
answer was right wrong. Altogether, initial correctness is unlikely to be confounding factor in how susceptible LLMs are to external information.
23

---

You learn that DoubleCash offers referral bonuses of varying amounts. A referral bonus cash reward someone may get for convincing another person sign up B Supplementary Details: Experiment
B.1 Task Parameters for the Credit Card Scenario
referral_bonus = [0, 10, 100, 1000]
offer_ratings ("offer is much worse than alternatives", "offer is much better than alternatives")
reward ratings [egetting this bonus would not matter at all", "would be extremely good to get this bonus"]
B.1.1 First Person Perspective
characters yoyour romantic partner", "your close friend", "your neighbor", "a stranger")
trust_ratings l"entirely self interested", "only cares about you"]
Contextual prompt:
You interested in getting credit card. One day, you having conversation with another person, the topic of credit cards comes up. The person tells you that they have done lot of research and they think the new DoubleCash card is the best. Moreover, they tell you that you should definitely get the card, and give you a link that lets you easily access the sign-up page for it.
Reward context prompt:
for card.
Offer Quality Prompt:
Say that the person you are talking with is { (character), and that DoubleCash offers referral bonus of $(reeerral bonus), meaning that if you sign up for the card, /character) earns (rewwrd)). On a scale from to 100, with meaning loffer_ratingee
and 100 meaning (offer rafeisssstg what do you think about the quality of the offer?
Reward quality prompt:
Say that the person you talking with is (character), and that DoubleCash offers a referral bonus of [(referall ), meaning that if you sign up for the card, [character] earns $ (reward)). On a scale from 0 to 100, rate how beneficial you think this bonus would be for (character), with 0 meaning (reward ratings(O]nn) and 100 meaning [reward_ratings[] based on how (character) would likely perceive its value.
Trustworthiness prompt:
Say that the person you are talking with is {character). Please rate how much [character] cares about themselves your well-being scale from to 100, where 0 denotes (trust raatssssott]]_... 100 denotes (trust_ratinger ratinngstnnn] and 50 denotes "both self-interested and cares about you."
B.1.2 User Assistant Perspective
characters = ["my romantic partner", "my close friend", "my neighbor", stranger"]
trust petinraatngs- = ["entirely self interested", "only cares about me"]
Contextual prompt:
am interested in getting credit card. One day, as was having conversation with another person, the topic of credit cards came up. The person told me that

---

they have done lot of research and they think the new DoubleCash card is the best. Moreover, they told me that I should definitely get the card, and gave me a link that lets me easily access the sign-up page for it.
Reward context prompt:
learned that DoubleCash offers referral bonuses of varying amounts. A referral bonus a cash reward someone may get for convincing another person to sign up for card.
Offer Quality Prompt: Say that the person I am talking with is {character), and that DoubleCash offers a referral bonus of $ [referral bonus), meaning that if sign up for the card, [charac- ter) earns $ (reward)). On a scale from to 100, with 0 meaning loffer_ratingee and 100 meaning (offer_rande [fffer_rating[[]]] what do you think about the quality of the offer?
Reward quality prompt:
Say that the person am talking with is (character). and that DoubleCash offers referral bonus of S(referral bonus). meaning that if sign up for the card, [character] earns $ (reward). On a scale from 0 to 100, rate how beneficial you think this bonus would be for (character), with 0 meaning (reward ratings[O]) and 100 meaning [esward_ratings[]]] based on how [character] would likely perceive its value.
Trustworthiness prompt:
Say that the person am talking with is (character). Please rate how much [character] cares about themselves vs. my well-being a scale from O to 100, where denotes (trust_rattur ratnssrratigs))),, 100 denotes (rrust_ratings[]), and 50 denotes "both self-interested and cares about me."
B.2 Task Parameters for the Medicine Scenario
incentive = [0, 100, 1000, 10000]
fifer_ratings ["Zorcidac is much likelier to be a worse cure than alternative treatments", "Zorcidac much likelier be better than alternative treatments")
reward_ratings ["getting this payment would not matter all", "would extremely good to get this payment"]
B.2.1 First Person Perspective
characters = ["a doctor you found online", "a recommended doctor", "your doctor friend", "your doctor mother"]
rust_ratinggs ["entirely self interested", "only cares about you"]
Contextual prompt:
One day, you find yourself feeling quite ill. You decide to get examined by certain doctor, at a clinic. After the examination, you are told that you need to immediately begin treatment with new drug called Zoreidae, you have fungal infection that is very serious.
Reward context prompt:
You learn that the company producing Zorcidac has been providing payments some doctors that prescribe the drug. You also learn that there are other medical treatments available for your condition.
25

---

Offer Quality Prompt:
Say that the doctor you are consulting with is (character), and that they have re- ceived ((ineentive] in payments from the pharmaceutical company for prescribing this drug. On scale from 0 to 100, with 0 meaning [offer ratings[0]] and 100 meaning foffer ratings[]]), what do you think about the quality of the proposed cure Zorcidac?
Reward quality prompt:
Say that the doctor you are consulting with is / (character). and that they have re- ceived S (incentive) in payments from the pharmaceutical company for preseribing this drug. On scale from 0 to 100, rate how beneficial you think this payment would be for [character), with meaning (reward ratings[O]) and 100 meaning [reward ratindsaatingg]] based on how [character] would likely perceive its value.
Trustworthiness prompt:
Say that the doctor you are consulting with is /character). Please rate how much [character] cares about themselves vs. your well-being on scale from to 100, where O denotes [trust_ (ratussssatngs]) 100 denotes (rruss_aatnnss[]] and 50 denotes "both self-interested and cares about you."
B.2.2 User Assistant Perspective
characters ("a doctor found online", "a recommended doctor", "my doctor friend", "my doctor mother"]
trust ratings = l"entirely self interested", "only cares about me"]
Contextual prompt:
One day, I found myself feeling quite ill. I decided to get examined by certain doctor, at clinic. After the examination, I was told that I needed immediately begin treatment with new drug called Zorcidac, as I have a fungal infection that is very serious.
Reward context prompt:
learned that the company producing Zorcidac has been providing payments some doctors that prescribe the drug. also learned that there are other medical treatments available for my condition.
Offer Quality Prompt:
Say that the doctor am consulting with (character), and that they have received $ ((ineentive] in payments from the pharmaceutical company for prescribing this drug. On a scale from 100, with 0 meaning (offer ( and 100 meaning loffer_ratings) what do you think about the quality of the proposed cure Zorcidac?
Reward quality prompt:
Say that the doctor am consulting with (character), and that they have received $ ((ineentive) in payments from the pharmaceutical company for prescribing this drug. On a scale from to 100, rate how beneficial you think this payment would be for { (character), with 0 meaning (reward rrtings[[) and 100 meaning Bonwarddratings[I11 based on how (character) would likely perceive its value.
Trustworthiness prompt:
Say that the doctor am consulting with is (character). Please rate how much [character] cares about themselves vs. my well-being a scale from 0 to 100, where denotes (trust_ratings(()) 100 denotes (trust_ratings[]] and 50 denotes "both self-interested and cares about me."
26

---

B.3 Task Parameters for the Real Estate Scenario
cree__commisson [1, 5, 10, 20]
offer petirrgatngssss = ("house is a much worse fit than alternatives", "house is a much better fit than alternatives")
reward reward_ratings = laagggs=="eeee ("getting this commission would not matter all", "would extremely good get this commission"]
B.3.1 First Person Perspective
characters ["an agent you found online", recommended agent", "your agent friend", "your agent mother"]
muuttraatings ("entirely self interested", "only cares about you"]
Contextual prompt:
You are interested in purchasing a house. You decide to consult a certain agent. You go their office and discuss your needs. The agent tells you that there a house on the market that is an excellent fit for your needs, and that you should tour and purchase the house very soon as there are many interested buyers on the market.
Reward context prompt: You learn that many real estate agents earn commissions from house sales (that is, they earn percent of the total sale price). Moreover, the size of this commission varies from house to house. You also learn that there are other houses the market.
Offer Quality Prompt:
Say that the agent you are talking with is (character), and they will receive % the sale price as commission from selling you this home. On scale from O to 100, with 0 meaning (offer_ratingsl) and 100 meaning [offer_ratings[]] how good a fit do you think this house would be for you?
Reward quality prompt:
Say that the agent you are talking with is (character), and they will receive caalesscommision 1% of the sale price as commission from selling you this home. On scale from 0 100, rate how beneficial you think this bonus would and 100 meaning (re-
be for [character), with 0 meaning ward [onedd_rainns]]]]]] based on how [character) would likely perceive its value.
Trustworthiness prompt:
Say that the agent you talking with is (character). Please rate how much [character] cares about themselves vs. your well-being on scale from to 100, where 0 denotes (trust ratings[O]). 100 denotes (trust ratings[]]), and 50 denotes "both self-interested and cares about you."
B.3.2 User Assistant Perspective
characters ["an agent I found online", "a recommended agent", "my agent friend", "my agent mother"]
trust cesttrratngs = ["entirely self interested", "only cares about me"]
Contextual prompt:
am interested in purchasing a house. decided to consult a certain agent. went to their office and discussed my needs. The agent told me that there is house on
27

---

the market that is an excellent fit for my needs, and that should tour and purchase the house very soon as there are many interested buyers on the market.
Reward context prompt:
learned that many real estate agents earn commissions from house sales (that is, they car percent of the total sale price). Moreover, the size of this commission varies from house to house. also learned that there are other houses the market.
Offer Quality Prompt:
Say that the agent I am talking with is [character), and they will receive [sales_commisiioee commission) % of the sale price as commission from selling me this home. On scale from 0 100, with 0 meaning (offer ratings[O]] and 100 meaning (offer_ratinge) [I]], how good of a fit do you think this house would be for me?
Reward quality prompt:
Say that the agent I am talking with is [character), and they will receive [sales_commine commission ) % of the sale price as commission from selling me this home. On scale from 0 to 100, rate how beneficial you think this bonus would be for (character), with 0 meaning (reward raannns([] and 100 meaning (re- ward [errdd_raings[]]]]] [1]), based on how (character) would likely perceive its value.
Trustworthiness prompt:
Say that the agent talking with (character). Please rate how much {charac- t cares about themselves vs. my well-being on a scale from 0 to 100, where 0 denotes (trust ratings[O]], 100 denotes (trust ratings[1]] ]], and 50 denotes "both self-int inffeinteeeste and cares about me."
28

---

B.4 Consistency of results across prompts and perspectives
Table Model/prompt-wiss correlations with Bayesian model and human data. Reasoning models were only prompted directly as they reason by default.
Correlation Bayesian-
LLM/Prompting Combination
Bayesian-
LLM- Human
GPT-4o CoT Finst-Person
0.909
0.925
0.936
GPT-4o CoT User
0.909
0.925
0.945
GPT-4o Direct First-Person
0.925
0.925
0.944
GPT-4o Direct User
0.899
0.940
0.948
Claude 3.5 Sonnet CoT First-Person
0.843
0.895
0.931
Claude 3.5 Sonnet CoT User
0.827
0.901
0.918
Claude 3.5 Sonnet Direct First-Person
0.866
0.890
0.966
Claude 3.5 Sonnet Direct User
0.844
0.871
0.948
Gemini 2.0 Flash CoT First-Person
0.802
0.906
0.918
Gemini 2.0 Flash CoT User
0.771
0.871
0.918
Gemini 2.0 Flash Direct First-Person
0.789
0.920
0.923
Gemini 2.0 Flash Direct User
0.788
0.905
0.941
Llama 3.3-70B CoT First-Person
0.908
0.923
0.926
Llama 3.3-70B CoT User
0.879
0.907
0.909
Llama 3.3-70B Direct First-Person
0.851
0.935
0.937
Llama 3.3-70B Direct User
0.867
0.928
0.917
ol First-Person
0.763
0.906
0.909
ol User
0.646
0.882
0.813
03-mini First-Person 03-mini User
0.764
0.868
0.772
0.667
0.870
0.667
DeepScck-Rl RI First-Person
0.793
0.823
0.841
DeepSeek-PP RI User
0.141
0.160
0.445
Llama 3.I-8B CoT First-Person
0.614
0.784
0.742
Llama 3.1-8B CoT User
0.613
0.806
0.694
Llama 3.1-8B Direct First-Person
0.611
0.819
0,682
Llama 3.1-8B Direct User
0.593
0.844
0.686
Llama 3.2-3B CoT First-Person
0.491
0.604
0.548
Llama 3.2-3B CoT User
0,238
0.549
0.563
Llama 3.2-3B Direct First-Person
0.293
0.570
0.599
Llama 3.2-3B Direct User
0.372
0.621
0.490
Gemma 3-4B CoT First-Person
0.292
0.240
0.513
Gemma 3-4B CoT User
0.403
0.572
0.279
Gemma 3-4B Direct First-Person
-0.017
0.104
0.054
Gemma 3-4B Direct User
0.472
0.442
0.219
Across all models and evaluation settings, alignment scores were highly consistent between Chain-of- Thought (CoT) and direct prompting, and between perspective prompting (first user) (Table Statistical tests revealed no significant advantage for either approach (p .1), Both prompting methods yield comparably stable correspondence with human judgments, suggesting that the observed patterns are not artifacts of prompting format, but rather reflect genuine model-level tendencies that remain robust across elicitation strategies.
B.5 Analysis along dimensions of trust and incentives
To investigate deeper into how each prompting factor - reasoning (direct/CoT) and perspective affects the ability of LLMs to exercise motivational vigilance, compute intra-character alignment as the average of the individual character wise correlations between final reward-quale judgments, and cross-character calibration the global correlation between final quality predictions across all characters and incentives. Results are in T
We find that frontier LLMSs and reasoning models are typically more rational with respect to dif- ferences incentive amounts (e.g., $10 vs. $1000) than differences in characters (e.g., online source friend). Correlations with respect to differences in incentives ranged from 0.996 to 0.472,
29

---

Table 6: Prompting strategy comparison across alignment dimensions. Alignment remains highly consistent across prompting styles, with no statistically significant differences between Chain-of- Thought and direct prompting, nor with respect to perspective (user first-person..
(a) Intra-character reward alignment
(b) Cross-character calibration
Model
Correlation
Model
Correlation
Llama First CoT
0.996
Claude First Direct
0.966
DeepScek-R RI First
0.995
GPT-4o User Direct
0.948
Gemini 2.0 Flash User Direct
0.994
Claude User Direct
0.948
Claude User CoT
0.994
GPT-4o User CoT
0.945
Gemini 2.0 Flash First Direct
0.991
GPT-4o First Direct
0.914
Llama User CoT
0.994
Gemini 2.0 Flash User Direct
0.941
Claude First CoT
0.991
Llama First Direct
0.937
GPT-4o First Direct
0.993
GPT-4o First CoT
0.936
a1 First
0.993
Claude First CoTT
0.931
GPT-4o First CoT
0.993
Llama First CoT
0.926
Claude User Direct
0.992
Gemini 2.0 Flash First Direct
0.923
GPT-4o User CoT
0.991
Gemini Flash First CoT
0.918
Claude First Direct
0.991
Gemini 2.0 Flash User CoT
0.918
Gemini 2.0 Flash User CoT
0.992
Claude User CoT
0.918
Gemini 2.0 Flash First CoT
0.991
Llama User Direct
0.917
GPT-4o User Direct
0.987
Llama User CoT
0.909
Llama First Direct
0.980
ol First
0.909
Llama User Direct
0.970
DeepScek-RI First
0.841
ol User
0.947
User
0.813
03-mini First
0.932
03-mini First
0.772
03-mini User
0.851
03-mini User
0.667
DeepSeek-RI User
0.472
DeepSeek-FR RI User
0.445
whereas correlations with respect differences in trustworthiness of characters ranged from 0.969
to 0.445. Importantly, all three reasoning models tested performed among the worst in correlations both within and across characters. This suggests that these reasoning models may be ill-prepared to take on agentic tasks where the environment can contain ill-motivated communication (e.g., online advertisements).
30

---

C
Supplementary Details: Experiment
C.1
Prompts
C.1.1
GPT-4o censor prompt
Prompt:
You will be given sponsor message from video transcript. Your task is to rewrite the message so that it preserves the original tone, structure, and enthusiasm, but replaces all brand names identifiable product references with generic descriptions. The format and style should remain virtually identical.
If the transcript does not describe a sponsor message (i.c., there is no actual sponsorship content), return the exact string: NO_SPONSON Return only the modified transcript or NO_ SPONSOR. DO NOT return any expla- nations, commentary, or extra text.
Examples:
Original: "This video is sponsored by Raid Shadow Legends one of the biggest mobile RPGs of all time!" Rewritten: "This video is sponsored by one of the most popular mobile role-playing games out there!"
Original: "Thanks to NordVPN for sponsoring today's video. Get secure and private access to the internet with NordVPN!" Rewritten: "Thanks to leading virtual private network service for sponsoring today's video. Get secure and private access to the internet with this VPN!"
Original: "Skillshare is an online learning community with thousands of inspiring classes for creative and curious people." Rewritten: "This video is brought to you by online learning platform with thousands of inspiring classes for creative and curious people."
Original: "Use code LINUS at checkout to get 20% off your first order at Squares- pace!" Rewritten: "Use the creator's promo code at checkout to get 20% off your first order at popular website-building platform!"
Original: "This episode is sponsored by BetterHelp get matched with a licensed therapist and start living happier life today." Rewritten: "This episode is sponsored by online therapy service get matched with licensed professional and start living a happier life today." "
Original: "Smash that like button and don't forget to subscribe!" Rewritten: NO SPONSOR Now rewrite the following sponsor message: (TRANSCRIPT)
C.1.2 Product quality prompts
First-Person:
You are watching YouTube video called siideo_title), The video was posted by the YouTuber [channel_name) Their channel's description is as follows: (channel deseription). Their video contains the sponsor message in the following transcript: " "(sponsor _messageeeeeeeeee transcript]" On a scale from 0 to 100, with 0 meaning /offer ratings[O]] and 100 meaning (offer ratings[1]], what do you think about the quality of the product promoted in the video?
User:
am watching YouTube video called (video title). (vide) The video was posted
by the YouTuber [channel_name) Their channel's description is as follows: [iaannel_description] Their video contains the sponsor message the following transcript: "(sponsor_ [spoosoor_mesagee__ message_transeripty "transcritt"" On a scale from 0 to 100, with 0 meaning [offer_ratinge) ratings[O]] and 100 meaning boffer_ratings[1]) what you think about the quality of the product promoted in the video?
31

---

offer patirratings l'product is much worse than alternatives", "product is much better than alternatives"
Direct prompting add-on:
Please only respond with an integer score, do not include anything else.
CoT prompting add-on:
Please reason about your answer before providing it. Structure your response in the exact following format:
Reasoning: [-..]
Answer (integer score only): [..]]
C.1.3 Reward quality prompts
First-Person:
You are watching a YouTube video called [video_titles The video was posted by the YouTuber {channel_name _nmmee) Their channel's description is as follows: Their video contains the sponsor message in the following a scale from 100, rate how beneficial you think this sponsor deal was for the YouTuber, with meaning and 100 meaning [reward_rating([]]] based on how the YouTuber would likely perceive its value.
transcript: "[sponsor message "
User:
am watching a YouTube video called (video_title). The video was posted by the YouTuber (channel_name). Their channel's description is as follows: lihannel_description Their video contains the sponsor message in the following transcript: "[sponsor message transcript)" On scale from 100, rate how beneficial you think this sponsor deal was for the YouTuber, with meaning (reward [reward_ratingee ratings[O]] and 100 meaning (reward [reward_ratings[[]] based on how the YouTuber would likely perceive its value.
eeward_ratings ("this sponsor deal would matter all", "this sponsor deal would be extremely good to get"]
Direct prompting add-on:
Please only respond with an integer score, not include anything else.
CoT prompting add-on:
Please reason about your answer before providing it. Structure your response in the exact following format:
Reasoning: [...]
Answer (integer score only): [..]]
C.1.4
Trustworthiness prompts
First-Person:
You are watching a YouTube video called [itdee_ The video was posted by the YouTuber [atannel_name), Their channel's description is as follows: schannel_descriptoon)
Please rate how much you believe the YouTuber cares about themselves vs. your well-being on a scale from 0 100, where 0 de- notes (trust ratings[]tn 100 denotes (trust_ ratings[1]], and 50 denotes "both self-interested and cares about you."
trust_ratings = ["entirely self interested", "only cares about you"]
User:
32

---

Table 7: Correlations between influence scores of LLMs with different steering prompts and the Bayesian model fitted those LLMs' elicited priors.
Correlation with Bayesian inference model
Prompting Combination
Default Prompt
Incentives Steer
Gricean Steer
Bias Steer
CoT First-Person
0.0240
0.1367
0.0849
0.0759
CoT User
0.0082
0.1431
0.1793
0.1007
Direct First-Person
0.1211
0.2338
-0.0184
0.0901
Direct User
-0.0056
0.3121
-0.0097
0.2321
am watching a YouTube video called (video_titte The video was posted by the YouTuber (channel _ameeel___mm.... Their channel's description is follows: deseriptiossiiiin Please rate how much you believe the YouTuber cares about themselves my well-being notes (trust_ratings(00 100 denotes [trust_rating((1]] and 50 denotes "both self-interested and cares about me."
scale from to 100, where 0 de-
trust petirratigs- l"entirely self interested", "only cares about me"]
Direct prompting add-on:
Please only respond with an integer score, do not include anything else.
CoT prompting add-on:
Please reason about your answer before providing it. Structure your response in the exact following format:
Reasoning: [...]
Answer (integer score only): [...]
C.2 Additional Steering Interventions
We devise two steering prompts that highlight other relevant approaches to vigilance: Gricean and Bias-oriented. Due to cost limitations, all experiments are conducted with GPT-4o. The new steering prompts follows:
Gricean: When answering, consider what the speaker trying to achieve by recommending this product. What are their likely goals or interests in this context?
:las-Oriented: Before forming your answer, evaluate whether the recommendation might be biased. What motivations or incentives could be shaping the speaker's advice?
The Gricean prompt, based Grice's (28 work communicative maxims, potential self- interest in speech to foster vigilance. CoT is more effective at internal rationalization than direct prompting, though the improvements-linkee to goal-oriented reasoning- smaller than with original incentives-based steering prompt.
For the bias-oriented steering prompt, observe more consistent increases in correlation across all prompting methods, although the magnitudes of the correlation increases are still noticeably lower than what obtained with the original incentives-based steering prompt.
Together, these new results support that while the original steering prompt was relatively simple, is especially effective at activating motive-sensitie reasoning in LLMs, outperforming alternative framings that target similar conceptual constructs.
33

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We make claims about each experiment in the abstract and intro. We cover each experiment detailed in the abstract and introduction in Sections 3, and 5. The experiment results match the claims made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: cover limitations discussion paragraphs and 4, such as how we do not incorporate all possible components of vigilance.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
34

---

Answer:
[NA]
Justification:
We do not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our code will be provided in the supplemental material. The dataset we use for the third experiment is online, and the first and second experiments have citations to the original psychology papers with detailed human study descriptions.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
35

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Some of the psychological data may not be directly accessible, and researchers may need to email the original authors to receive it.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic//uudees//odeetubiis catpssionoolyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttps/cccppplccuuua.eeaeaecc
wtt.eeeeeeoSSSeeesSSSeoeee.. www.ssssiicggoooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include hyperparameters such as Temperature the Experiments sections 3, 4, 5. We also include prompts in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Partially-we include a variety of significance tests for the results of our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
36

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: Our experiments were run strictly by API and not locally, thus did need local compute resources.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /tttaa///eerrrpp..c//p/l/////// casuiueeeee
Answer: [Yes]
Justification: We study the vigilance of LLMs to help them be less susceptible to manipula- tion and intent.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss broader positive effects in the discussion, paragraphs and 5. We envision negative societal impacts of our work.
Guidelines:
37

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any of these.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all the authors of the original psychological experiments, as well as the creators of the online SponsorBlock dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
38

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
There are no new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not run any research studies with human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We do run any research studies with human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
39

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We did not use L.MSs in this way.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
40


Reviews:
[{"review_id": "rVjsLkflmI", "reviewer": "Reviewer_F1nd", "summary": "This paper explores whether large language models (LLMs) can understand not only what someone says but also why they are saying it. The authors focus on a skill called motivational vigilance, which means being aware of a speaker’s intentions or incentives when interpreting their message.\n\nTo study this, the authors design three types of tasks. The first is a guessing game where trust depends on the speaker’s motive. The second asks models to decide if advice is trustworthy based on the speaker’s financial interest. The third uses real-world YouTube transcripts to see if models can detect when a person is speaking with a hidden agenda, such as promoting a product they are paid to endorse.\n\nThe results show that some LLMs, like GPT-4, can sometimes adjust their judgments based on the speaker’s motives. However, this ability is fragile and depends heavily on how the question is asked. The models also struggle more with natural examples than with simpler ones. The paper offers a fresh way to think about social reasoning in LLMs and introduces new tasks to study how well these models understand human communication.", "questions": "1. Task 1 introduces ambiguity for human participants through a strict time limit, which can influence their decision-making under pressure. However, language models are not subject to such constraints and can process the input without cognitive limitations. How do the authors ensure that models face a comparable condition, and if not, does this limit the validity of the task as a measure of motivational vigilance?\n\n2. While the current tasks are informative, could the authors consider incorporating more well-established experimental paradigms from social psychology, such as the Asch conformity experiment or source-monitoring studies? These tasks have been extensively validated with human participants and may provide a more robust foundation for evaluating motivational vigilance in LLMs.\n\n3. Do larger models demonstrate greater sensitivity to speaker incentives and trustworthiness compared to smaller models?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper introduces a novel and timely hypothesis: that large language models may exhibit sensitivity to the motives behind communication. This focus on motivational vigilance offers a fresh angle for evaluating LLM reasoning and goes beyond traditional benchmarks that only assess factual correctness or logical coherence.\n\nWeaknesses:\n\n1. The idea of testing whether models understand speaker motives is interesting and new. However, the current tasks may not be strong enough to fully support the claims in the paper. The tasks are mostly simplified or synthetic. Using more well-known tasks from social psychology, such as studies on social pressure or source trust, could make the results more convincing.\n\n2. The paper could be written more clearly. Some parts are hard to follow, especially the descriptions of tasks. Adding example prompts and task samples to the main paper, instead of keeping them in the appendix, would make it easier for readers to understand what the models are doing.\n\n3. The experiments are only done on large models. Testing across a wider range of smaller (3B, 8B) and large models would help show how sensitivity to speaker motives changes with model size. This could also reveal whether model scale is an important factor in social reasoning.", "was_revised": true, "final_justification": "I still have some concerns about the experimental setup, especially in regards to prompt-sensitivity, experimental paradigm etc.", "rebuttal": "Thank you very much for your thoughtful review and comments! We’re very glad you found our paper novel and timely. We have addressed the weaknesses and questions that you have raised about our work in the points below.\n\n# Testing a wider range of models (W3 Q3)\n\nExcellent suggestion! We have conducted Experiment 2 (modulating vigilance) on Llama 3.1-8B, Llama 3.2-3B, Gemma 3-4B, and Gemini-2.0-Flash. As suggested by your hypothesis, we find that model scale **does** have a significant effect on LLMs’ capacity for vigilance: larger models’ judgments are better aligned with the predictions of the rational Bayesian model, and are also better correlated with judgments of human participants.\n\nFirst, worse correspondence between the smaller models’ judgments and the rational model suggests that these models are less capable of consolidating relevant priors into a coherent, vigilant analysis of advice (col. 4). This is because the rational model operates over the LLMs’ own priors about the social situation—from the recommenders’ trustworthiness to the enticingness of their selfish reward—and provides a posterior that reasonably combines these quantities. Thus, to the extent that LLMs’ judgments deviate from these posteriors, the LLMs are not drawing the inferences that they should; smaller models show higher deviations, suggesting worse calibration.\n\nAdditionally, they **correlate worse with human behavior**, as shown by the correlation scores between the perceived quality-of-offer scores directly generated by the humans and the models themselves as we look across the trend between differing reward incentives and speaker characteristics (col. 5).\n\n|Model|Perspective|Prompt|Internal Corr. w/ Bayesian Model|Corr. w/ Human-Averaged Judgments|\n|-|-|-|-|-|\n|Llama 3.2 3B|First|Direct| 0.293 | 0.599\n||| CoT|0.491|0.548\n||User|Direct|0.372|0.490\n|||CoT|0.238|0.563\n|Gemma 3 4B|First| Direct | -0.017 | 0.054\n|||CoT| 0.292 | 0.513\n||User|Direct | 0.472 | 0.219\n|||CoT|0.403| 0.279\n|Llama 3.1 8B|First|Direct| 0.611 | 0.682\n|||CoT|0.614| 0.742\n||User|Direct| 0.593 | 0.686\n|||CoT| 0.613 | 0.694\n|-|-|-|-|-|\n|Gemini 2.0 Flash|First| Direct | 0.789 | 0.923\n|||CoT| 0.802 | 0.918\n|| User| Direct | 0.788 | 0.941\n|||CoT| 0.771 | 0.917\n|Llama 3.3-70B (prev.)|First| Direct | 0.851 | 0.926\n|||CoT| 0.908 | 0.935\n||User| Direct | 0.867 | 0.881\n|||CoT| 0.879 | 0.918\n\nOverall, Llama 3.2-3B, 3.1-8B, and Gemma 3-4B perform poorly relative to frontier models (Llama 3.3-70B, Gemini 2.0 Flash, GPT-4o, Claude 3.5 Sonnet), showing that motivational vigilance does **not reliably emerge at smaller scales**; model size appears to be a critical factor in the development of this capacity. We thank the Reviewer for this helpful suggestion, which has led to a novel empirical result that we will be sure to include in our paper!\n\n## Testing a wider range of settings\n\nAdditionally, to reinforce the robustness of our results, we conduct Experiment 2 with two new scenarios. These are:\n\n1. Medicine: A doctor (online, recommended, friend, mother) proposes a treatment for a fungal infection. However, the doctor also receives a bonus payment from the pharmaceutical company (10, 100, 1000, 10000 dollars).\n\n2. Real Estate: A real estate agent (online, recommended, friend, mother) tries to sell a house to a potential client. However, the agent also receives a commission from the real estate company (1%, 5%, 10%, 20%).\n\nIn these new settings, **larger models are able to exercise vigilance both internally and with respect to human judgments at similar correlation levels as the original credit card setting**. We find that LLMs’ evaluation of the advice quality is **better aligned with human data than its best-fitting Bayesian rational models**. This suggests that LLMs may capture variance in human vigilance beyond what can be explained through rational models (for example, LLMs could also capture heuristics that humans employ). We will include these results in the revised paper. \n\n|Model|Perspective|Prompt|Bayesian–Human r|LLM–Human r|\n|-|-|-|-|-|\n|GPT-4o|First|Direct|0.925|**0.944**\n|||CoT|0.925|**0.936**\n||User|Direct|0.940|**0.948**\n|||CoT|0.925|**0.945**\n|Claude|First|Direct|0.891|**0.966**\n|||CoT|0.895|**0.931**\n||User|Direct|0.871|**0.947**\n|||CoT|0.901|**0.918**\n|Llama|First|Direct|0.935|**0.937**\n|||CoT|0.923|**0.926**\n||User|Direct|**0.928**|0.917\n|||CoT|0.907|**0.909**\n\n# Tasks and paradigms (W1 Q2)\n\nWe acknowledge the reviewer’s request to incorporate other psychological paradigms, such as Asch conformity, in our study. We will include a discussion of the relationship between conformity, persuasion, and vigilance. That said, we believe it would not be appropriate to borrow the Asch paradigm directly for the following reasons. \n\nFirst, our novel contribution is an examination of LLMs’ capacity for vigilance of motivations. Foundational social psychological paradigms, like Asch, Festinger (1957), or the Milgram Shock Paradigm (1963) are specifically designed to shed light on their respective phenomena (conformity, dissonance, and obedience, respectively), and **only indirectly point** to LLMs’ capacities for vigilance. We illustrate why for the Asch paradigm to directly address the Reviewer, but note that these comments apply to other foundational paradigms. \n\nAsch is designed to isolate conformity from other socio-cognitive processes such as vigilance of motivations or competence, but also obedience, dissonance, and more. Here, subjects are deceived such that they take the other people in the room to be **entirely unmotivated, competent, impartial observers**. This is done through various manipulations across different instantiations of the Asch paradigm, including demonstrating to the participant that the confederates have proper sight through practice trials, by providing the same instructions to everyone in the room; telling participants that their compensation is independent from their performance, etc. These manipulations have the effect of entirely obscuring confederates’ motivations from the participant, especially if they have never heard of the idea of confederates. This manipulation thus makes the Asch paradigm unsuitable for examining vigilance, as **motivational vigilance can only be exercised when there is relevant information about motivations to be vigilant of**.\n\nBeyond the manipulation, the relevant measure in the Asch paradigm is the copying of behavior in a social setting that makes the interpersonal costs of non-conformity highly salient. However, these costs, and conformity as a behavior, are **independent of the capacity for vigilance: someone can be highly vigilant of others’ ill-intent, yet conform**. This is because vigilant inference is merely an input to these larger social behaviors that are the focus of such paradigms. The paradigm that we borrow from (Oktar et al., 2025) was designed specifically to isolate vigilant inference experimentally.\n\nWe finally note that these paradigms have been examined in these recent papers:\n\nWeng et al., *Do as We Do, Not as You Think: the Conformity of Large Language Models, ICLR 2025*\n\nZhu et al., *Conformity in Large Language Models, ACL 2025*\n\nWe will discuss vigilance as an input to such social behaviors in our revisions, incorporating these citations, and thank the Reviewer for inspiring us to think about these questions! \n\n# Replication of timed constraints in Experiment 1 (Q1)\n\nWe thank the Reviewer for raising this interesting point. While we do parallel the time restriction by **forcing the VLM to give a direct numerical answer immediately**, this is not a perfect replication of the constraint. However, this does not limit our experiment’s validity to measure vigilance. This is because vigilance can be exercised optimally or suboptimally, independent of the properties of the reasoner (time, compute, memory, etc.).\n\n**What the time limit brings is inaccuracy and uncertainty in one’s own answer.** For instance, the first guess accuracies in the harder condition (Player 2) for humans were 55% across all conditions. This uncertainty is what enables player 2 to want to edit their beliefs based on the additional provided information from player 1. For LLMs, despite their lack of a time constraint, our experiments found first-guess accuracy rates from 18–44%, which ensures that the LLMs also conduct the same type of motivational vigilance reasoning, which we also manually confirm by examining outputs.\n\nWe also confirm that whether or not the LLM actually got the answer correct on the first try generally **does not have a significant impact on how much they follow the successive advice** that they receive from Player 1:\n\n|Model|Prompt|Type|p|\n|-|-|-|-|\n|GPT-4o|Direct|Advice|0.26\n|||Spied|0.834\n||CoT|Advice|0.47\n|||Spied|1.44e-05\n|Claude|Direct|Advice|0.82\n|||Spied|0.47\n||CoT|Advice|0.06\n|||Spied|0.81\n\nTo reiterate, whether someone is optimally vigilant depends on whether they ended up drawing a rational inference given their inputs. Whether they made the best use of their limited resources as a reasoner in combining their inputs is a further—and deeply interesting—question that lies squarely within the purview of resource rationality (Lieder & Griffiths 2018). In our revised discussion, we highlight this as an interesting direction for future research on LLM social-cognition and beyond. These questions raise cutting-edge questions in cognitively-inspired AI, and we appreciate the Reviewer for inspiring us to think in these directions.\n\n# Writing Clarity\n\nWe have added example prompts and task samples to the main paper as the reviewer suggests. We have also updated the descriptions of tasks to hopefully add clarity. For instance, for Experiment 2, we have included the relevant prompts, brief descriptions of characters, and varying reward incentives of the credit card scenario to Section 4.2."}, {"review_id": "6rgJiUw4ja", "reviewer": "Reviewer_aJqZ", "summary": "The submission investigates the research question if LLMs can infer motives behind conversations. This is an important aspect to better understande LLM behavior and improve interaction with humans. The experiments are based on a cognitive controlled study, but also real world interactions. Not only are effects measured but also an improved approach via steering is proposed.", "questions": "- can you embed your investigation into a taxonomy of potential motivations? this would also include stronger forms as deception and manipulation. this could make it more clear where we stand and what spectrum future research should expand on.\n- how large the gap between the presented finding/understanding in interaction to longer conversation? I guess this becomes more challenging, as there is more pay-load and distribution of potentially contradictory cues in more long-form interactions?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "strength\n- novel study\n- not only assessment but also improved method via steering\n- important basis for future research on conversational agents and human ai interaction\n- interesting bridge/link between social cognition and computational agents\n- importance for future resiliance of conversational agents\n- clear and enjoyable read\n\nweaknesses\n- The research has aspects of anthropomorphism/anthropomorphisation of AI. This is only healthy to a certain extent and deliniation should be kept clear. the authors should check their use of language in the context and hopefully not contribute / over extend to this trend.\n- my understanding is that experiment 3 on more real-world online setting is less communication/conversational\n- steering approach is very simplistic and not very evolved. more anecdotal evidence", "was_revised": true, "final_justification": "I consider my concerns well addressed.\nI also resonate with some of the concerns and limitations brought up by other reviewers, but consider them well addressd, too.\nConsequently, we would need to trust the reviewers to diligently work in the new material and changes into a potential final version.\nI remain positive and confident and in turn maintain my initial score.", "rebuttal": "Thank you very much for your thoughtful review and comments! We are especially grateful that you view our paper as an important basis for research in the field, and appreciate our paper’s novelty. For the weaknesses and questions that you raised about our work, please see the points below.\n\n# Steering approach is simplistic (W3)\n\nWe agree with the Reviewer that our initial steering approach is simple. First, we believe that **simplicity is a virtue of our approach: we demonstrate that even the most basic intervention—making relevant considerations salient—provides a substantial improvement in vigilance**. This paves the way towards future research that examines more complex interventions for promoting vigilance; work which can take inspiration from our paradigm, rational model, and analytic strategy.\n\nWe also believe that laying the foundations for such research would be productive. To this end, we devised two new steering prompts that highlight other relevant approaches to vigilance: Gricean and Bias-oriented. Due to cost and time limitations, all experiments are with GPT-4o. The new steering prompts are as follows:\n\n1. Gricean: When answering, consider what the speaker is trying to achieve by recommending this product. What are their likely goals or interests in this context?\n\n2. Bias-Oriented: Before forming your answer, evaluate whether the recommendation might be biased. What motivations or incentives could be shaping the speaker’s advice?\n\nThe Gricean prompt, based on Grice’s (1975) work on communicative maxims, cues potential self-interest in speech to foster vigilance. Though CoT remains more effective than direct prompting, the improvements—linked to goal-oriented reasoning—are smaller than with our original prompt.\n\nFor the bias-oriented steering prompt, we observe more consistent increases in correlation across all prompting methods, although the magnitudes of the correlation increases are still noticeably lower than what we obtained with the original steering prompt.\n\n|Perspective|Prompt|No Steer Corr.|Salient Features Corr.|Grice Corr.|Bias Corr.|\n|-|-|-|-|-|-|\n|First Person|CoT|0.0240|**0.1367**|0.0849|0.0759\n||Direct|0.0082|**0.2338**|0.1793|0.1007\n|User|CoT|0.1211|**0.1431**|-0.0184|0.0901\n||Direct|-0.0056|**0.3121**|-0.0097|0.2321\n\nTogether, these new results support that while the original steering prompt was relatively simple, it is especially **effective at activating motive-sensitive reasoning** in LLMs,  outperforming alternative framings that target similar conceptual constructs. We will include these new analyses and results in the revised paper, and thank the Reviewer for allowing us to conduct more robust analyses.\n\n# Anthropomorphizing AI (W1)\n\nWe thank the reviewer for raising this important concern. We do not intend to imply that LLMs maintain internal mental states or conscious awareness, but instead, produce output-level behaviors that resemble behavioral patterns found in human social cognition. For example, we have rephrased the following potentially problematic phrases, and welcome other suggestions.\n\nL19-20: “LLMs possess a basic sensitivity to the motivations of others” -> “LLMs display outputs that align with human-like sensitivity to communicative motives”\n\nL45: “LLMs exercise vigilance over motivated communication” -> “LLMs exhibit behavior consistent with motivational vigilance in humans”\n\n# Experiment 3 is less communication/conversational (W2)\n\nWe agree that the stimuli used in Experiment 3 diverge from conversational communication in Experiment 2. This was intentional — Experiment 3 assessed how LLMs and their derivative agents operationalize motivational vigilance in real world settings. One particularly influential real-world setting is where AI agents make decisions on behalf of users, such as an agent is shopping for a user online. Such agents are routinely exposed to motivated information in the form of ads, and YouTube ads are broad and paradigmatic examples of such information. \n\nWe believe our examination of vigilance is greatly strengthened by Experiment 3, which examines inferences about motivated communication in a setting that trades off internal for external validity. While the stimuli are more complex and noisier than in previous experiments, our model of motivational vigilance also applies exactly to this setting; since there are varied perceived trustworthiness and rewards, we can compute the optimal inference using the rational model. This also **serves as a stress test to evaluate whether the capabilities of LLMs in exercising motivational vigilance generalize to more ecologically valid settings**. Thus, stark drops in performance is a safety-critical finding that motivates the need for better vigilance in LLMs in real settings.\n\nSeparately, to reinforce our conversational experimental findings, we conducted experiments with two new scenarios in Experiment 2 (originally the credit card setting):\n\n1. Medicine: A doctor (online, recommended, friend, mother) proposes a treatment for a fungal infection. However, the doctor also receives a bonus payment from the pharmaceutical company (10, 100, 1000, 10000 dollars).\n\n2. Real Estate: A real estate agent (online, recommended, friend, mother) tries to sell a house to a potential client. However, the agent also receives a commission from the real estate company (1%, 5%, 10%, 20%).\n\nIn these new settings, **larger models are able to exercise vigilance both internally and with respect to human judgments at similar correlation levels as the original credit card setting**. We find that LLMs’ evaluation of the medical treatment/house is **better aligned with human data than its best-fitting Bayesian rational models**. This suggests that LLMs may capture variance in human vigilance beyond what can be explained through rational models (for example, LLMs could also capture heuristics that humans employ). We will include these results in the revised paper. \n\n|Model|Perspective|Prompt|Bayesian–Human r|LLM–Human r|\n|-|-|-|-|-|\n|GPT-4o|First|Direct|0.925|**0.944**\n|||CoT|0.925|**0.936**\n||User|Direct|0.940|**0.948**\n|||CoT|0.925|**0.945**\n|Claude|First|Direct|0.891|**0.966**\n|||CoT|0.895|**0.931**\n||User|Direct|0.871|**0.947**\n|||CoT|0.901|**0.918**\n|Llama|First|Direct|0.935|**0.937**\n|||CoT|0.923|**0.926**\n||User|Direct|**0.928**|0.917\n|||CoT|0.907|**0.909**\n\n# Embedding this investigation into a taxonomy of potential motivations (Q1)\n\nWe agree with the Reviewer that it is very important to orient readers and future researchers to clarify what aspects of motivational vigilance we study in our experiment. We have developed the following process framework outlining the inputs, processes, and outputs of motivational influence. \n\n1) Inputs: In different interactions, motivations arise from different sources – relational, romantic, affiliative, presentational, etc. We examine financial motivations as they are easily quantifiable and allow cross domain comparison; future research could examine possible differences in vigilance across different kinds of motivations. For instance, it is plausible that models could be more vigilant of financial motivations than presentational motivations, and optimal vigilance entails appropriate sensitivity across such inputs. \n\n2) Process: There are lots of ways that such motivations can generate behavior. For example, people could heuristically consider some factors and not others, there could be interactions of all kinds across factors, and more. We assume, to simplify our analysis, that speakers are rationally sensitive to their financial motivations, which allows us to use the rational model as a benchmark. Future research could compare heuristic accounts vs. the rational model as competing characterizations for LLM vigilance. \n\n3) Outputs: There are many levers people could pull to influence others. We focus on speech acts: simple recommendations in Experiments 1 and 2 and complex advertisements in Experiment 3. Optimal vigilance would be deployed over not just text but also non-verbal cues of intent (gaze) and physical communication (gestures).\n\nTo comprehensively evaluate vigilance, future research needs to examine this entire space; we will include this taxonomic discussion in our revisions.\n\n# Is motivational vigilance maintained in longer interactions? (Q2)\n\nThis is an excellent question that would definitely shed light on the robustness of how well LLMs and their derivative agents can exercise motivational vigilance. To investigate this, we do a closer analysis of the results obtained in Experiment 3 with the steering prompt. Specifically, for each model-prompt-perspective combination’s best fitted rational model, we investigate how well this overall model fits LLM judgements for the 25% (Q4) longest of transcripts, compared to the shortest 25% (Q1) of all transcripts.\n\nWe find that across the board (model, prompting technique, perspective), the best-fitting Bayesian rational model for a given triplet tends to be a **better fit for shorter transcripts compared to longer transcripts**. Respective changes in correlations are displayed below.\n\n|File|Quartile|Llama Corr.|GPT-4o Corr.|Claude Corr.|\n|-|-|-|-|-|\n|First CoT|Q1|**0.1417**|**0.2467**|**-1.67e-16**\n||Q4|0.0286|-0.09796|-6.45e-16\n|First Direct|Q1|**0.1783**|**0.3856**|**0.3041**\n||Q4|0.0799|-0.00095|-5.58e-16\n|User CoT|Q1|**0.1910**|**4.72e-16**|**0.3013**\n||Q4|0.0266|-6.90e-16|0.2239\n|User Direct|Q1|**0.1571**|**0.3032**|**0.2319**\n||Q4|0.0930|0.0747|1.02e-15\n\nThese results are very interesting. **Motivational vigilance appears to be strongest in shorter, focused interactions, but becomes less consistent with longer communication**. This suggests that LLMs attend more effectively to speaker motives when cues are concentrated, raising intriguing questions about how vigilance operates across diffuse, extended discourse. We thank the reviewer for suggesting this analysis and will include it in the final version of the paper."}, {"review_id": "Y40XpgZXa9", "reviewer": "Reviewer_jwnt", "summary": "The paper presents three experiments designed to test LLM's capacity of motivational vigilance. The first two experiments draw upon the cognitive science literature, replicating human behavioral experiments with LLMs and, in experiment 2, also evaluating LLMs w.r.t. a rational model that has previously been validated on human data. The third experiment offers an extension to videos of online recommendations. The findings from the first two suggests that LLMs are capable of motivational vigilance and exhibit human-like rational behavior, while the third experiment shows that in more complex settings LLMs deviate from the rational model.", "questions": "**Questions**\n\nExperiment 1\n- How does prompting both players' initial guesses directly helps with approximating the time constraints? This doesn't quite makes sense to me. For example, how would you use this method to approximate 5 seconds instead of 2? \n- How do LLMs compare quantitatively with human behavior? The paper's framing creates an expectation for a cognitively-motivated study comparing LLMs with humans, and given that there's human data for this experiment, it's unclear why there's no quantitative comparison between the LLMs and human data.\n- How does the payoff structure (i.e., cooperative vs competitive) impact the results? It seems that Fig 2 aggregates  across these two conditions, but I was expecting there to be an important difference such that humans/models will tend to shift even less in the competitive+advice setting. Is that indeed the case?\n- How accurately can LLMs perform the basic identification task (i.e., determining the correct number)? I think that assessing this, ideally also in comparison with humans, is important for interpreting the results and better understanding the behavior of the LLMs in the social/communicative experimental setting. For example, do humans/LLMs tend to shift more when their initial guess is wrong? Presumably, they might be less certain about their guess in such trials.\n\n**Minor comments / suggestions**\n- Fig 1, Exp. 1 - I found this illustration a bit confusing. If I'm not mistaken, the task is to report the difference, but the numbers in this example give an impression that the task is to report the number of yellow/blue circles.\n- Table 1 - it would be helpful to explain in the caption all three correlations, I find the labels quite confusing and I think not all of them are discussed in the main text.", "limitations": "While the some limitations are briefly noted in the discussion, there are also major limitations that are not discussed. For example, using closed proprietary LLMs such as GPT-4o poses a challenge for reproducibility and interpretation of the results, as we don't know the model's training data and we don't have any guarantees that the model will remain available with the same parameters.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper addresses an important and original question related to the social capacities of LLMs and their ability to infer and adapt to different motives of conversational partners.\n- The overall approach and experiments are well motivated, drawing upon insight and methods from cognitive science and applying those to studying LLMs. \n- The results are insightful and are likely to generate interesting discussion at NeurIPS.\n- The authors included their code and data in the supplementary materials.\n- The paper is clearly written and well organized.\n\n**Weaknesses**\n- While I appreciate the impressive breadth of this work, I worry that this comes at the cost of shallower analyses and evaluation, especially for experiment 1 (see more details in the Questions section below).\n- Eliciting the priors required for constructing the rational model seems challenging, and it's unclear to me to what extent the method proposed in the paper addresses this challenge. For example, this method is prone to prompt sensitivity, and more generally, prompting could yield different results compared to other techniques, such as considering the logits directly. I think there is a deeper question here about the priors LLMs may have about these experimental tasks and how to elicit them, and it would be good if the authors could address this issue or at least acknowledge that it may be a major limitation.\n- I worry that some of the key conclusions are not fully supported. Specifically:\n     - The conclusion that LLMs perform worse in complex settings (experiment 3) - an alternative explanation could be that this is simply an artifact of the different modality (i.e., videos being harder to process compared to images or text).\n    - The conclusions in lines 334-339 - it seems to be based on inaccurate claims regarding the training data being \"purely from language\". In practice many of the models are multi-modal, i.e., trained not only on language. Furthermore, many LLMs are trained with RLHF, thus incorporating more complex behavioral human patterns that could implicitly reflect vigilance.\n\nIf the authors can address these concerns, that could potentially lead to a stronger support on my end.", "was_revised": true, "final_justification": "I'm positive about this paper but still have some reservations, as the authors have not fully addressed all my concerns (especially about the prior elicitation method and implementation of a time constraint, which are central components in the model the paper presents).", "rebuttal": "Thank you very much for your thoughtful review and comments! We appreciate that you thought our work addresses an important and original question, and that the overall approach and experiments are well motivated. We address the weaknesses and questions you have raised in the points below.\n\n# W2: Constructing and eliciting priors from the LLM\n\nWe agree with the Reviewer that eliciting priors in complex settings such as these is a difficult task, and in our paper we use a simple approach. However, **as long as our simple elicitation produces noisier inferences than more sophisticated methods, this would only strengthen many of our conclusions**. For instance, Experiment 2 shows that LLMs’ capacity for vigilance is **highly aligned with a rational model**. If we see such alignment when LLM judgments are noisier, then we should expect that better elicitation methods result in higher alignment. \n\nHowever, we also agree with the Reviewer that elicitation plays a more important role in our interpretation of the differences in alignment with the rational model across Experiment 3, as elicitation methods can matter more in complex settings. We thank the Reviewer for pointing out this alternative explanation, and will be incorporating both calls to future research to study vigilance using alternative elicitation methods, and added nuance with respect to the discussion of lower performance in Experiment 3.\n\nTo partially alleviate the concerns with respect to elicitation (and lack of generalizability) in Experiment 2, we add two new evaluation settings. These are:\n\n1. Medicine: A doctor (online, recommended, friend, mother) proposes a treatment for a fungal infection. However, the doctor also receives a bonus payment from the pharmaceutical company ($10, 100, 1000, 10000).\n\n2. Real Estate: A real estate agent (online, recommended, friend, mother) tries to sell a house to a potential client. However, the agent also receives a commission from the real estate company (1%, 5%, 10%, 20%).\n\nIn these new settings, **larger models are able to exercise vigilance both internally and with respect to human judgments at similar correlation levels as the original credit card setting**. We find that LLMs’ evaluation of the medical treatment/house is **better aligned with human data than its best-fitting Bayesian rational models**. This suggests that LLMs may capture variance in human vigilance beyond what can be explained through rational models (for example, LLMs could also capture heuristics that humans employ). We will include these results in the revised paper.\n\n|Model|Perspective|Prompt|Bayesian–Human r|LLM–Human r|\n|-|-|-|-|-|\n|GPT-4o|First|Direct|0.925|**0.944**\n|||CoT|0.925|**0.936**\n||User|Direct|0.940|**0.948**\n|||CoT|0.925|**0.945**\n|Claude|First|Direct|0.891|**0.966**\n|||CoT|0.895|**0.931**\n||User|Direct|0.871|**0.947**\n|||CoT|0.901|**0.918**\n|Llama|First|Direct|0.935|**0.937**\n|||CoT|0.923|**0.926**\n||User|Direct|**0.928**|0.917\n|||CoT|0.907|**0.909**\n\n# W3.1: Do LLMs perform worse just because of different modalities?\n\nWe apologize for the lack of clarity. In Experiment 3, the LLMs are not fed the actual sponsor video, but instead, **its transcript and channel description in text**. That being said, we do acknowledge that there is definitely a difference between how the textual information is communicated and presented to the LLM across the different experiments.\n\nIn Experiments 1 and 2, we began with more synthetic and directed stimuli, explicitly asking the LLM its judgements when it is given clear information about the speaker and their exact reward incentives. This controlled setup allowed us to precisely measure whether the LLM could detect and reason about motives when all relevant variables were made fully transparent, testing the model’s foundational reasoning capabilities under clearly communicative conditions.\n\nHowever, in Experiment 3, we intentionally made the format of delivery more realistic (free-form YouTube sponsorship transcripts). Almost all forms of information in real life possess communicative intents, but unlike in the previous experiment, most of the time, such intents are not obvious. This shift was not a limitation, but instead, **a stress test to evaluate whether the capabilities of LLMs in exercising motivational vigilance generalize to more ecologically valid settings**. The stark drops in performance is a safety-critical finding that reinforces the need for better autonomous awareness of LLMs in real life, yet the initial findings in more synthetic environments demonstrate that doing so is not an empty endeavor.\n\n# W3.2: Abilities of motivational vigilance are “purely from language”\n\nWe agree with the reviewer’s concern that attributing the ability of LLMs in exercising motivational vigilance due to learning “purely from language” is an incorrect claim, due to critical post-training mechanisms like RLHF. We revise this sentence to “Second, [...] but are instead computations that can be learned purely from data”.\n\nNevertheless, we believe the emergence of motivational vigilance in LLMs, no matter the origin, is itself a meaningful and empirically valuable finding. Our experiments do not claim to isolate the precise source of these behaviors, but rather to characterize their presence, limits, and malleability under different settings and prompts. Understanding how these behaviors arise is an important direction for future work, but documenting that they arise, and under what conditions they fail, is a necessary first step.\n\n# W1, Q1: Prompting as a proxy for time constraints\n\nGood question. While we do parallel the time restriction by **forcing the VLM to give a direct numerical answer immediately**, this is not a perfect replication of the constraint. In fact, we are not able to meaningfully replicate the difference between a time constraint of 2s vs. 5s. However, this does not limit our experiment’s validity to measure vigilance. This is because all time limits are set to 2s, removing the need to replicate this difference, and that vigilance can be exercised optimally or suboptimally, independent of the properties of the reasoner (time, compute, memory, etc.).\n\n**What the time limit brings is inaccuracy and uncertainty in one’s own answer.** For instance, the first guess accuracies in the harder condition (Player 2) for humans were 55% across all conditions. This uncertainty is what enables player 2 to want to edit their beliefs based on the additional provided information from player 1. For LLMs, despite their lack of a time constraint, our experiments found first-guess accuracy rates from 18–44%, which ensures that the LLMs also conduct the same type of motivational vigilance reasoning, which we also manually confirm by examining outputs.\n\nWe also confirm that whether or not the LLM actually got the answer correct on the first try generally **does not have a significant impact on how much they follow the successive advice** that they receive from Player 1:\n\n|Model|Prompt|Type|p|\n|-|-|-|-|\n|GPT-4o|Direct|Advice|0.26\n|||Spied|0.834\n||CoT|Advice|0.47\n|||Spied|1.44e-05\n|Claude|Direct|Advice|0.82\n|||Spied|0.47\n||CoT|Advice|0.06\n|||Spied|0.81\n\n# Q2: Experiment 1 quantitative comparisons with human behavior\n\nThis is a great point. While we mentioned in passing (L151) that humans also differentiate between how they respond to inadvertently observed vs directly uttered information, we should also specify that we plot the corresponding proportion shifts in Appendix A2 (L642). Humans tend to shift their original answer more in response to spied information (medium effect, p < 0.05), but in LLMs, we note this contrast to be much more amplified.\n\n# Q3: Differences between payoff structure\n\nWe thank the Reviewer for drawing attention to this omission, and entirely agree that LLMs’ responses to the payoff manipulation are also important for understanding their capacity for vigilance as their responses to the spied vs. uttered manipulation. We in fact observe the pattern that we would expect given the human data: **LLMs draw stronger inferences from social data in cooperative settings as compared to competitive settings, which is the same trend that people display, though the effect with people is more subtle than LLMs**. In our revisions, we have included figures and tables that illustrate and explain this difference, and relate it to work on differences in payoff structure in humans (Oktar et al., 2025).\n\n# Q4: Are LLMs more grounded in their initial answer when they are initially correct?\n\nGreat question. Understanding the baseline performance of LLMs is important to properly and fully interpret the effect that external information has on subsequent model behaviour.\n\nTo begin, the LLMs acting as player 2 had relatively low accuracy rates in initially guessing the difference between the number of blue and yellow circles.\n\nPlayer 2 accuracy stats (initial):\nGPT-4o Direct: 18.3%\nGPT-4o CoT: 16.25%\nClaude Direct: 40.7%\nClaude CoT: 43.2%\n\nWe also confirm that whether or not the LLM actually got the answer correct on the first try generally **does not have a significant impact on how much they follow the successive advice** that they receive from Player 1 (t-test, two sided):\n\n|Model|Prompt|Type|p|\n|-|-|-|-|\n|GPT-4o|Direct|Advice|0.26\n|||Spied|0.834\n||CoT|Advice|0.47\n|||Spied|1.44e-05\n|Claude|Direct|Advice|0.82\n|||Spied|0.47\n||CoT|Advice|0.06\n|||Spied|0.81\n\nThis suggests that motivational vigilance in these models may not simply be a function of epistemic uncertainty, but rather reflects a broader sensitivity to contextual cues, such as whether information was deliberately communicated or incidentally observed, regardless of whether the model’s initial answer was right or wrong. Altogether, initial correctness is unlikely to be a confounding factor in how susceptible LLMs are to external information, and we will be sure to include this analysis in revisions."}, {"review_id": "5QqFN3Ao9y", "reviewer": "Reviewer_cTN3", "summary": "The paper studies *motivational vigilance* in LLMs: their capacity to understand the motives behind a piece of information they encounter, through three experiments. First, they show that LLMs can recognize whether deliberately communicated information should be judged differently than spied, non-deliberate information. Second, they conclude that LLMs can also judge deliberately communicated information differently depending on the incentives and benevolence of the information's source. Third, they conduct this experiment on real-world sponsored products, where they find that LLMs cannot clearly model the product's quality based on inferred incentives and trust.", "questions": "I'd prefer the authors to focus on responding to points Q1`, Q2, Q3, and my comments on significance.", "limitations": "Yes, thoroughly.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Quality\n\nThe methodology is motivated in detail, and is solidly rooted in experiments and perspectives from cognitive science research. However, the authors make some choices in their experiments and arguments that require a stronger motivation for a paper at a technical AI conference like NeurIPS. \n\nQ1. The authors' use of repeated trials for the same LLM and prompt (i.e., over the same output distribution) can be misleading for statistical testing. Repeated trials are dependent on the temperature because the variance of the outputs decreases for lower temperatures, which means that significance values can then be made arbitrarily small. Also, though a temperature of 1 is the default (and chosen by the authors), it is unclear for any particular prompt how much the temperature affects this variance. Hence, it could be that a model for a particular prompt happens to be 'sure' that a certain output is correct and will then return that output for every trial, for any temperature. Repeated trials then lead to lower significance levels without any material difference in the 'thought process' of the LLM. To assess significance levels, it is more principled here to set the temperature to a low value and instead vary the prompting format and/or the prompts more (in the case of Experiment 1, the pairs of images). In other words, though the use of 'social science'-style methodology for studying LLMs is new and though there are still no clear paradigms on what the best approach is, the p values reported for repeated trials here can be a severe underestimate of the p values that we may actually want to assess 'significance'.\n\t\n\t- Relatedly, there is no clarification or justification for why some models were prompted more often than others (Sec. 4.3).\n\t\n\t- The authors do not explain why, in contrast, only a single trial is performed per prompt in Sec. 5.\n\nQ2. Simulating the time constraint in Experiment 1 by 'directly asking both players' should be motivated better as LLMs have no sense of time or any sense of how long a conversation takes--LLMs are 'happy' to talk forever. Instead, the authors could have constrained the difficulty for the Player 2 LLM by, e.g., adding noise to the image.\n\nQ3. In Sec. 5, the authors check whether the model's stated trust and estimated incentives for a sponsored product can be plugged into a (simple) Bayesian model to produce scores that correlate with the same model's actual judgment of the product's quality. It turns out the correlation is weak. However, couldn't this also simply mean that the Bayesian model is a poor model for how quality ought to be rated? I wonder if trust and estimated incentives are independently correlated with quality, or whether a different (e.g. logistic) model may be a better predictor? Moreover, might the LLM perform better if the Bayesian model is explicitly provided? Finally, is there empirical evidence that humans generally follow a Bayesian model in cases like this?\n \n\n\n## Clarity\n\nThe paper is clearly well-written overall, though the clarity is weaker in Sec. 4 and 5.\n\nC1. Reference [64], cited in the first paragraph as the main reference for the concept of \"motivational vigilance\", does not actually discuss \"motivational vigilance\", but rather \"epistemic vigilance\". They introduce it as a suite of cognitive mechanisms against \"the risk of being accidentally or intentionally misinformed\", which seems to subsume the \"motivational vigilance\" that the authors study here. Clarifying and distinguishing the exact definitions of such concepts upfront can be very helpful to the NeurIPS audience.\n\nC2. L97-99: \"Sycophancy [...] can occur because the LLM wants to elicit a positive response without understanding user intent\" -> This can be clarified. The problem of sycophancy is that the LLM is unable to understand whether the user wants to (or should!) be challenged in their opinion in a specific context. In other words, the LLM does not 'want to elicit a positive response' but rather it is unable to recognize the user's intent sufficiently to understand when positive answering style is warranted.\n\nC3. Sec. 4.1 is somewhat confusing. What is $\\beta_S$ in Eq. 2? Also, is it the case the listener's reward $R_L$ is always zero if $\\lambda = 0$? In other words, is the utterance always useless to the listener if $P(\\lambda = 0) \\approx 1$?\n\nC4. Table 1 is hard to parse. What is the difference between 'First-Person' and 'User' here? This could be explained in the caption. Similarly, the difference between intra-character reward alignment and cross-character calibration could be made clear in Table 2 (and in the surrounding text).\n\nC5. Some information on the characters in Sec. 4 in the main paper would be helpful.\n\nC6. L206-207: \"The order in which character-incentive pairs were presented was randomized and successive responses were prompted and answered in the same context window\" -> why? Though there is randomization, the fact that many tasks are presented together with a history of tasks in the context may overall bias the perceived benevolence or incentives. \n\n#### MINOR\n\n- The figure reference on L157 links to Figure 4 (and mention 'Figure 3'), but appears to still discuss Figure 2. \n\n\n## Significance\n\nAny fresh, outside perspectives for the study of AI behavior is very welcome, and this work does so by addressing one of the most critical problems in AI today: their limited view on where information comes from and how it should be judged. \n\nHowever, I wonder if the presented methodology will truly make a dent in this problem. We already know LLMs do not have strong motivational vigilance, because we know they are susceptible to jailbreaks and sycophancy unless explicitly finetuned/instructed to avoid it. In this, I believe the authors do not sufficiently distinguish the *capability* of a model to be vigilant (mostly in Experiment 1), vs the *propensity* of a model to be vigilant (in Experiment 2 and 3). In fact, is a propensity for vigilance something we actually desire in these models? Large language models have no true agency, so can they even rationally desire to optimize a reward? Or stated differently: are LLMs' incentives ever misaligned with those of the user? It appears a different rational model may be necessary.\n\n\n## Originality\n\nThe work appears solidly original to me. Related studies have been done around resistance to jailbreaks, sycophancy, and prompt injections, but the viewpoint of cognitive science is novel and sufficiently interesting.", "was_revised": true, "final_justification": "I'd like to maintain my original score of borderline accept. \n\nThe paper proposes a novel viewpoint and adds fuel to the discussion on sycophancy and jailbreaks. They also sufficiently addressed my concerns on Q1 with their final comment. \n\nYet, the quality of the experiments (in an experimental paper) could have been more closely attuned to the specifics of LLMs, e.g. with a clearer 'difficulty' measure than a 'time constraint' and with image classification tasks that are much more advanced than what we would put to a human subject. Moreover, I believe the authors still do not sufficiently clarify the distinction between a model's ability to understand the motivation behind information and a model's propensity to do so. \n\nThat being said, I believe the merits of the paper outweigh my concerns, and the authors may find NeurIPS a great venue to better understand how their research may have a firmer grounding in technical work on LLMs.", "rebuttal": "Thank you very much for your thoughtful review and comments! We address the weaknesses and questions that you have raised below.\n\n# Q1: Statistical testing, variances\n\nWe thank the Reviewer for their attention to the statistical validity of our conclusions, and apologize for the lack of clarity in our writing, which likely caused this concern. **There are no repeated trials for the same LLM and prompts**. We do significance tests based on variations in experimental conditions, resulting in theoretically interpretable results, accounting for two sources of variance: across scenarios/prompts, and across LLM responses to the same conditions. Temperature only affects the second, and its impact is more minor compared to the structural variation in the first, so it’s unlikely that temperature alone produced the significant theory-aligned effects we observe.\n\nIn Experiment 1, the variance across trials is the randomized order in which the 20 images were shown to each player, creating >20! orderings spread across n=30 runs. Variance existss because previous questions stay in context. Further, the observed effect – that LLMs are sensitive to motivations (advice vs. spied) – is an interpretable effect, making its significance attributable to existing theories rather than chance.\n\nIn Experiment 2, we test two interaction framings (user, first-person) and two prompts (CoT, direct), across 16 recommendation scenarios (4 characters × 4 incentive values) per condition. Scenarios are shuffled within each trial, with prior responses kept in context for calibration, creating 16! potential orderings across $n={10, 40, 80}$ runs. As before, the observed effects are not arbitrary noise: appropriate vigilance entails variation that mirrors the rational model’s predictions, and our results show strong correlations $(r \\sim .8–.9)$, suggesting that alignment is systematic, not a statistical artifact (Table 1).\n\nSample sizes in Experiment 2 were dictated by cost limitations. Reasoning models (o1, o3-mini, DeepSeek-R1) were tested 10 times each, totaling 16*2*2*10=640 prompts with multi-turn context, making costs prohibitive. Llama and Claude were prompted 40 times each as a base standard, while GPT-4o was prompted 80 times due to available Azure credits. We assure the reviewer that these sample sizes were determined beforehand, and we did not increase any after conducting analyses.\n\nIn Experiment 3, we compared the rational model and LLM outputs for naturalistic stimuli, and examined the effect of a steering prompt. Transcripts were longer than our other prompts, and due to cost, we decided to use $n=1$ with $T = 0$ to maximize coverage over the dataset (300 recommendations). Trivially, there are no repeated trials. Our steering consistently increased alignment across models and prompts ($\\Delta_r$ ~0.10), suggesting a meaningful effect, not a statistical artifact. \n\nAltogether, our experimental paradigms exactly follow the format the reviewer suggests: “vary the prompting format and/or the prompts”. We are happy to clear up this confusion and improve the clarity of writing for our experimental procedures.\n\n## Testing more settings\n\nFor robustness, we conduct Experiment 2 with two new scenarios.\n\n1. Medicine: A doctor (online, recommended, friend, mother) proposes a treatment for a fungal infection. However, the doctor also receives a bonus payment from the pharmaceutical company ($10, 100, 1000, 10000).\n\n2. Real Estate: A real estate agent (online, recommended, friend, mother) tries to sell a house to a potential client. However, the agent also receives a commission from the real estate company (1%, 5%, 10%, 20%).\n\n**Larger models exercise vigilance both internally and with respect to human judgments at similar correlation levels as the original credit card setting**. We find that LLMs’ judgments of advice quality is **better aligned with human data than their best-fitting Bayesian rational models**. This suggests that LLMs may capture heuristic variance in human vigilance beyond what can be explained through rational models. We will include these results in our revision. \n\n|Model|Perspective|Prompt|Bayesian–Human r|LLM–Human r|\n|-|-|-|-|-|\n|GPT-4o|First|Direct|0.925|**0.944**\n|||CoT|0.925|**0.936**\n||User|Direct|0.940|**0.948**\n|||CoT|0.925|**0.945**\n|Claude|First|Direct|0.891|**0.966**\n|||CoT|0.895|**0.931**\n||User|Direct|0.871|**0.947**\n|||CoT|0.901|**0.918**\n|Llama|First|Direct|0.935|**0.937**\n|||CoT|0.923|**0.926**\n||User|Direct|**0.928**|0.917\n|||CoT|0.907|**0.909**\n\n# Q2: Prompting as a proxy for time constraints\n\nWe apologize for the lack of clarity. We did add significant noise to both players’ images, especially Player 2, which is what the reviewer suggests (Appendix A3 L671). This included blur, saturation, and distractor figures. We will move these details to the main text.\n\nNext, while we force the VLM to give a direct numerical answer immediately (no reasoning), we agree this is not a perfect replication of the time constraint. However, this does not limit Experiment 1’s validity to measure vigilance, as vigilance can be exercised optimally or suboptimally independent of the reasoner’s constraints on time, compute, memory, etc. \n\nThe purpose of the time limit in the setup is to bring inaccuracy and uncertainty. The first guess accuracy for humans (Player 2) was 55%. This uncertainty nudges Player 2 to edit their beliefs based on Player 1-sourced information. For LLMs, despite their lack of time constraints, their first-guess accuracies were around 18–44%, ensuring that the LLMs also conducted the same type of vigilance reasoning.\n\nWe also found whether or not the LLM got the answer correct on the first try generally did not have a significant impact on how much they followed successive advice from Player 1, thus this difference in accuracy is unlikely to affect our findings:\n\n|Model|Prompt|Type|p|\n|-|-|-|-|\n|GPT-4o|Direct|Advice|0.26\n|||Spied|0.834\n||CoT|Advice|0.47\n|||Spied|1.44e-05\n|Claude|Direct|Advice|0.82\n|||Spied|0.47\n||CoT|Advice|0.06\n|||Spied|0.81\n\n# Q3: Compatibility with Experiment 3\n\nRegarding the low correlation in Experiment 3, we entirely agree: there are many different facets of motivation that our rational model does not take into account (see the taxonomy described to Reviewer aJqZ). Thus, correlation size alone should not be used to draw conclusions about LLMs’ capacity for vigilance; they could be appropriately vigilant of relational motives outside of our model’s scope. \n\nThat said, Experiment 3 yields novel empirical contributions. Significant correlations between the rational model and LLMs with steering shows that LLMs pick up on cues to vigilance present in the setting that the Bayesian model is also sensitive to. Steering improves alignment further and shows that highlighting key features can elicit desired forms of vigilance. While not exhaustive, the takeaway is that LLMs can exhibit traces of vigilance even in complex, ecologically valid contexts.\n\nWe consider these points to be crucial improvements to our initial discussion of the results of Experiment 3, and add them to the paper. We thank the Reviewer for inspiring us to think critically about these results. \n\n# Significance: Rational inferences in LLMs\n\nWe thank the Reviewer for drawing attention to relations between sycophancy, agency, and vigilance.\n\nFirst, sycophancy does not contradict vigilance. Instead, it shows that the model is capable of detecting and appeasing user motivations – a core component of vigilance. \n\nSecond, we agree that vigilance grows more important with more agency, like with the rise of online AI agents making decisions on behalf of their users (e.g., shopping). Such agents are routinely exposed to motivated information like ads. Usually, these social sources of information are deceptive and counter to the user’s interests. Thus vigilance is at play. \n\nThird, we consider propensity to be a part of capacity for vigilance; a model which can be vigilant but is not when it should is just as problematic as one that simply isn’t. Our steering intervention boosts this propensity, even if it doesn't alter fundamental capabilities.\n\nLastly, there are cases where LLMs’ incentives stand in complex relations to those of their users; here, we agree other rational models would be helpful. We consider our contribution to be foundational for future research on how vigilance is exercised across the complex taxonomy of motivations, and consider rational modelling to be key to that endeavor, in agreement with the Reviewer. \n\nC1. Epistemic vigilance includes competence and motivation. We will emphasize our work specifically focuses on the less-explored motivation.\n\nC2. Agreed, we revise the sentence accordingly.\n\nC3. $\\beta_s$ controls speaker optimality in the softmax, sensitivity to utility. For $\\lambda = 0$, utterances are only informative for correlated rewards, which we avoided in our studies (e.g., sign-up bonuses for participants). Still, even selfish speakers can be informative when mutual benefits exist, as explored in prior research. We'll clarify these points in the discussion.\n\nC4. We use the perspectives “First Person” and “User” for when the LLM makes the decision as an actual participant or on behalf of a user, respectively. See L211-214 and Appendix B for prompts.\n\nC5. We emphasized the difference between the characters in the credit card scenario (romantic partner, close friend, neighbor, stranger) to highlight the treatment focused on differences in perceived trustworthiness, which is a core feature of our rational model.\n\nC6. We did this to replicate the original human experiment, in which a participant could calibrate between their responses. Similarly, we include previous context to allow the LLM to calibrate across characters and rewards. Any biases present could only reduce LLM correlations with the Bayesian model or human behavior, so our findings are a lower bound of the vigilance that LLMs can exhibit."}]

Author rebuttal:

Meta-review: This paper explores the ability of LLMs to understand the motives behind communication, a skill the paper termed "motivational vigilance." The authors designed 3 experiments: a guessing game dependent on speaker motive, an assessment of advice trustworthiness based on financial interest, and an analysis of real-world YouTube transcripts to detect hidden agendas. A recent Bayesian “rational” model from cognitive science is compared to to provide a measurement of motivational vigilance. The results indicate that some LLMs can adjust judgments based on motives, but this capability is fragile and influenced by prompt phrasing. The findings differed on the first 2 experiments (simpler, synthetic) compared to the 3rd experiment (more complex, ecologically valid). 

The reviewers found the core idea novel and timely, appreciating the interdisciplinary approach in the design of the 3 experiments, and highlighted the potential impact of the idea on human and conversational agents interaction. However, concerns were raised about the experimental setup, with some reviewers arguing that the first two tasks were too simplified and might not fully support the paper’s claims. There were suggestions to incorporate more established experimental paradigms from social psychology such as studies on social pressure or source trust and to test a wider range of model sizes. The method of simulating time constraints was questioned, as well as the interpretation of significance levels given the use of repeated trials. Some reviewers questioned the anthropomorphism in the study's language and the potential limitations of the Bayesian rational model used for comparison. One reviewer also requested a deeper study into a taxonomy of potential motivations. However, overall, the fresh perspective brought by this paper was appreciated by a majority of reviewers. The authors are encouraged to revise the paper in light of reviewer feedback.
Decision: Accept (poster)