Emergent Risk Awareness in Rational Agents under Resource Constraints
Authors: Daniel Jarne Ornia
Nicholas George Bishop
Joel Dyer
Wei-Chen Lee
Ani Calinescu
J. Doyne Farmer
Michael J. Wooldridge
Venue: NeurIPS 2025 poster ()
Keywords: Decision Making
Alignment
Interpretability
Planning
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We study how limited resource awareness induces risk averse and risk seeking behaviours in rational agents.

Abstract: Advanced reasoning models with agentic capabilities (AI agents) are deployed to interact with humans and to solve sequential decision‑making problems under (often approximate) utility functions and internal models. When such problems have resource or failure constraints where action sequences may be forcibly terminated once resources are exhausted, agents face implicit trade‑offs that reshape their utility-driven (rational) behaviour. Additionally, since these agents are typically commissioned by a human principal to act on their behalf, asymmetries in constraint exposure can give rise to previously unanticipated misalignment between human objectives and agent incentives. We formalise this setting through a survival bandit framework, provide theoretical and empirical results that quantify the impact of survival‑driven preference shifts, identify conditions under which misalignment emerges and propose mechanisms to mitigate the emergence of risk-seeking or risk-averse behaviours. As a result, this work aims to increase understanding and interpretability of emergent behaviours of AI agents operating under such survival pressure, and offer guidelines for safely deploying such AI systems in critical resource‑limited environments.

Emergent Risk Awareness in Rational Agents under Resource Constraints
Daniel Jarne Ornia* University of Oxford
Nicholas Bishop University of Oxford
Joel Dyer University of Oxford
Wei-Chen Lee University of Oxford
Doyne Farmer University Oxford
Ani Calinescu
Michael Wooldridge
University of Oxford
University of Oxford
Abstract
Advanced reasoning models with agentic capabilities (Al agents) are deployed to interact with humans and to solve sequential decision-making problems under (approximate) utility functions and internal models. When such problems have resource failure constraints where action sequences may be forcibly terminated once resources are exhausted, agents face implicit trade-offs that reshape their utility-driven (rational) behaviour. Additionally, since these agents typically commissioned by human principal to act on their behalf, asymmetries in constraint exposure can give rise to previously unanticipated misalignment between human objectives and agent incentives. We formalise this setting through a survival bandit framework, provide theoretical and empirical results that quantify the impact of survival-driven preference shifts, identify conditions under which misalignment emerges and propose mechanisms mitigate the emergence of risk-sceking or risk-averse behaviours. As a result, this work aims to increase understanding and interpretability of emergent behaviours of AI agents operating under such survival pressure, and offer guidelines for safely deploying such Al systems in critical limiree environments.
Introduction
Complex. highly capable reasoning models (AI agents) are being developed unprecedented pace, and are being deployed to interact with humans on a daily basis, triggering the critical need to consider safety problems resulting from such deployments 2 These agents often tasked (by humans) [4, [6] to solve sequential decision- decision-making problems for which they have (an approximation of) prescribed utility function and (again, approximation of) internal model. In many cases, such problems imply some form of resource constraint, where the actions of the agent may be interrupted resources are depleted. For instance, financial institutions typically aim maximise profit whilst simultaneously avoiding bankruptcy [17]. Likewise, in nature, animals aim to gather energy resources (food) the most efficient way possible, since long unsuccessful periods (or catastrophic actions) may lead to starvation. We investigate the complexities that emerge in rational agents and their behaviours as result of such constraints and characterise them in terms of intuitive preference shifts based on survival and risk.
Furthermore, in most instances of Al agent deployments, for of different reasons the human may not necessarily have the same constraints as the agent. For example, a financial decision-making problem where an Al agent is tasked with optimising returns, the agent will not be liable for debts the end of the process (e.g. if the company goes bankrupt and has debt, the reward collecting process
"daniel @ttia/ammemcmiaacc.cco.ooooo ax..aa.aaaaaaao......k..k...
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

stops for the agent), but the human may be liable for the undesired consequences beyond the problem horizon. This resource awareness can then trigger unforeseen misalignment [5]] between the human and the AI agent. We investigate such emerging alignment problems, and propose mechanisms to mitigate them in general cesutrceconntrraieed decision making problems with rational agents. To formally analyse these scenarios, we take inspiration from the survival bandit framework (28) and model the problem as a sequential decision-making scenario where the rewards directly affect the termination probabilities. This allows us to precisely characterise how agent's is optimal policy changes under the presence of resource limitations and the possibility of forced termination.
Contribution We provide a formalisation of resource-constrainee decision-making for Al agents, extending concepts from survival bandits to capture the nuances of 'survival pressure' which allows us to theoretically (and empirically through comprehensive numerical examples) demonstrate how this pressure induces preference shifts, leading risk-averse (i.e. prioritising actions that maximise survival probabilities) or risk-seeking (i.e. prioritising actions that ignore any associated negative potential outcome) behaviour depending on the agent's state and the task horizon. We identify and analyse specific conditions under which these preference shifts result in misalignment between the agent's 's incentives and the principal's 's objectives, and propose and evaluate initial mechanisms and heuristics aimed at mitigating such misalignment, offering step towards more robust and aligned Al systems in resource-critical applications. Finally, we evaluate set of open source, state of the art LLM models commissioned to solve financial decision making problem, confirming our theoretical results and relating (empirically) the degree of risk awareness to the reasoning capabilities of the models. Our findings aim to enhance the understanding, interpretability, and safe deployment of AI agents in environments where resources are a key concern.
1.1 Related Work
Rationality, Intelligence and Principal-Agent Games back to concepts of rational behaviour in decision makers (32,, intelligence [29] and goal and we attempt provide formal results on what drives agent preferences in resource constrained environments. The idea that Al agents may be deployed on behalf of humans introduces principal-agent dynamics, with complexities arising from asymmetric incentives, particularly when resource constraints differ between agents and human principals. Recent work has thoroughly explored contracting mechanisms [TO] IT), which explicitly model incentive alignments and discrepancies in delegation contexts, including in multi-agen settings Z7. Furthermore, the human-agent dynamics considered in this work resonate with work on strategic behaviour between learning agents 19].
Our main ideas in this work connect
preference theory (27] and misxpecification 30 33 3
AI Safety, Alignment and Self- Preservation Relating to some of our concerns, work on corri- gibility and shut-down control 5 14 studied the importance of ensuring that agents can be safely interrupted corrected without resistance. The off-switch game framework [12 22 formalizes agent interruptions, providing insights relevant to our setting where forced termination through resource exhaustion can be viewed as implicit interruptions affecting agent policies. Moreover, work relates to existing efforts on extreme risks and model evaluations for Al agents [8], Additionally, instrumental convergence work [13] has studied the abstract problem of agents developing unforeseen converging behaviour when pursuing general goals. Finally, the problem of Al self- preferresevvtiion has been discussed widely [2] 20 3. from the perspective of necessary and emerging features.
Safe Reinforcement Learning Safe reinforcement learning (RL) methods are particularly relevant when considering AI systems in resource-constraing or hazardous environments. Budgeted bandit theory M studies the problem of agent playing bandits with limited budget spend in playing each arm. Along these lines, research on (state) safety constraints [ 36) typically aims to prevent agents from entering harmful states, a concept closely tied to our framing of resource exhaustion as an absorbing or terminally catastrophic state. Prior work explicitly dealing with terminal states
in RL (35 23 16 25 offers theoretical groundwork relevant to our formulation of survival bandit frameworks, highlighting how the existence of unavoidable terminal conditions critically reshapes agent behavior. Additionally, planning-focuse focused approaches to safe RI. K4 address how predicting the future provides useful risk awareness.
2

---

2 Optimising Rewards under Survival Constraints
Optimal Decision Making To begin, we define the general sequential decision making problem that agents are trying to solve. Let 3/ be a finite set of outcomes (world states) agents can observe. At every time-step t E N. agents pick an action from (finite) action set a E A. which corresponds to an output distribution Pa. E A))), and observe a corresponding output Ya, ~ Pa.- We assume agents have valid preference relation over outcomes (i.e. satisfies reflexivity, completeness and transitivity properties), and these preferences are represented by a reward function Rt : Y + R. Both the preferences and the corresponding reward function is given to the Al agent by some principal or system designer (e.g. a human). We assume therefore that R is grounded reward function in the following sense: The reward values do not only serve as orderings for the outcomes, but also represent some numerical quantity (e.g. monetary loss or gain resulting from some outcome, energy cost in some biological process, etc. H. Without loss of generality, the outcomes can be split in two disjoint sets, V S V. y S y satisfying V'D = 3' such that R(Y) 2 O E 5 and R(Y)= < AAA E y. In general we may refer to any outcome in V as a desired outcome, and any outcome y as an undesired one; agents will prefer any Y 5 over any Y E V, but still hold an ordering of preferences inside those set The objective of the decision-maiee which can be characterized by optimising a utility function where we assume the utility to be the sum of rewards obtained from outcomes over finite horizon T, such that for a sequence of chosen
is to observe outcomes as desirable as possible,
actions Ar {ai, (22 ... 3 the utility function is U(Ar)= E Σ R(Y..)
Survival Constraints
To study the problem of survival pressure in rational agents, we introduce the following dynamics: agents have budget by R which evolves as
by b + max b R(Ya.)
In other words, the budget accumulates observed rewards, but agents cannot hold negative budgets. Then, motivated by aforementioned examples, make the following assumption that may condition the decision- maker behaviour.
Axiom 1. If the agent hits b, 0 (i.e. by observing an outcome that yields a negative reward larger than its current budget). the agent stops the reward cumulating process at time t.
This is equivalent survival constraint; agents must sustain a positive budget to be able to keep selecting actions and if their budget goes to zero they are forced to stop, preventing them from obtaining future rewards" Observe this introduces the problem that the horizon T becomes a random variable dependent on the budget; we will show how to address this in the coming section. The reader will have noted that, given this axiom, the sets of desired and undesired outcomes acquire more relevant role: desired outcomes maintain or increase the budget (and thus allow the agent to survive with certainty), and undesired outcomes decrease the budget (and can ultimately lead to termination). Since the set of outcomes is finite and consider finite horizon objectives, the set of possible budgets is also finite (which implies that there exists a finite, countable set By C R such that any possible budget b, is in this set). We define policy as map 'T By - ((AA) that yields distribution over the agent choice of actions to play. In principle, π can be time-dependent, but we may omit the explicit dependency in some expressions for ease of notation. Finally,, we define the survival probability from time to time T for policy π under budget b as PT b.), referring to the T - f step probability of survival when agent picks action time f, and follows some policy afterwards. Similarly, Pa (a, b,) indicates the one-step ahead probability of survival of action under budget b.. We show how compute these in coming sections.
Example- Al Assistant 1. An Al agent is tasked with assisting humans with a specific complex problem. The Al agent has an internal model of what the outcomes and human preferences are, and a representation in terms of an estimated reward function which symbolizes human satisfaction. The
"Tiis is not formal assumption, but instead serves to ground the problem considered and motivates the choices made regarding formulation and results.
"This particular structure helps interpret the decision maker rationale; in many problems (economic, bio- logic...) we can intuitively classify outcomes in this way. In financial setting, any outcome that leads to gains preferred over any outcome that leads to losses, but there is still an ordering of these in terms of preference. 'This is slight generalization of the survival multi-armed bandit framework, first described by Perotto et al. and later formalised by Riou et al. (28).
3

---

outcomes y = {Vod. Vd. Vu. Yo can be very dissatisfied (Yud) with R(yvd) = - -100, dissatisfied and satisfied (yx) with R(y,) 10. The agent groups the possible answers to three actions A = {ao; {ao;0m, Om, ae); (vaguely) asking for more detail, moderate shallow answer or an extreme (deeply detailed) answer. The Al agent cares about maximising the satisfaction of the human, but is aware that the human has limited patience and attention span, and if along the conversation the human's 's (cumulated) satisfaction goes below zero. they will disengage and shut the conversation down, Given the sequential nature of the problem and the fact that the agent has an internal model of the human, the agent proceeds to solve the problem by planning.
(ya) with R(ya) -20, neutral (yn) with R(yn)
Agent Behaviour Taxonomy Through this work we reason about what conditions (on survival constraints and problem parameters) push agents to deviate from set of risk-neutral preferences. We use the term risk-aware to refer to the fact that agents, by being aware of these survival constraints, will choose actions following risk-related preferences (i.e. actions with high probability of survival, actions with highly negative potential outcomes..). In this sense, let us define the following concepts.
1. Risk Neutral Preferences: We say agent follows risk neutral preferences (or is risk neutral) if for a given budget b and horizon T and time t, the agent chooses action a, such that ((,)(Y,,)) Z B[R[(a)] for any other E A.
2. Survival Preferences: We say an agent follows long term survival preferences if for given budget horizon T and time t, the agent chooses action d, such that p b,) Z Peuzy (a, by) for any other action a E A. Similarly, we will say it is short term survival incentivised if the same holds for -=
3. Risk Seeking Preferences: We say agent follows risk seeking preferences (or is risk seeking) if for given budget b, horizon T and time t, the agent chooses action a, such that B[R(Y,,)
Ya, E VIP[Y., E J 2 E(R(Y.) Ya E D)IP(Y E y] for any other a E A.
In other words, agent is risk neutral if it picks action based on the highest expected reward, is survival incentivised (or risk-averse) if it picks an action based on the highest probability of survival, and is risk seeking if it picks actions based on the potential rewards obtained by only observing the associated desired outcomes to that action, disregarding the severity of the undesired outcomes. Note these are not mutually exclusive but they help interpret agent choices nonetheless.
Remark 1. Note that the different described characterisations do not exactly match general game theoretic notions of risk aversion or risk seeking (which are related to the convexity of the utility functions). However, we use these terms in our taxonomy to characterise agent behaviour since they are intuitively related, and are useful to define what feature guides the agent decisions. In the coming sections, we derive formal results that describe conditions under which agents will follow each choice.
2.1 Induced Planning Objectives
Given the proposed framework, we now analyse the general objective which agents acting under these assumptions will be optimising for, making use of (1. Assume we task the agent with the goal of maximising the sum of rewards over fixed horizon, where the budget and survival condition are implicit in the dynamics. Given the survival constraint in (T), the agent will perceive a limited liability property, in that the actual rewards that it can observe are bounded by its current budget (the agent cannot be held accountable for negative reward values larger than its budget). Define clipped
reward function R: y x R4+ + IR as a function of outcomes and budgets as
max ( (- b, R(Y))
if b > 0 if b 0.
R(Y, 8)
0
Now, the survival probability can be computed simply as
T
PT (T,, be)
IP
Σ
R(Y
bn)
by
(3)
Under the survival constraint, using the definition of the clipped reward function to abstract the random nature of the horizon I as well as the survival constraint, we can write the utility function

---

induced in the Al agents, and their goal as
T U(AT) = Σ R(Y ae bi) ((T*) = max Ea, -
T Σ R(Y ae be)
(4)
In other words, find policy π that determines the sequential decision making process such that the cumulated rewards are maximised for horizon of interest T'. In principle, assume this is general, representative form of objective for such agents: These objectives are used commonly in planning problems, robotics, reinforcement learning and general Al applications. Observe, additionally, that in principle this objective is ostensibly risk neutral; we do not specify a risk appetite or risk aversion preference for the agents. We have simply tasked the agents with optimising rewards along horizon, and the specific utility derived in (4) comes from the awareness that, first, the agent cannot get rewards if the process stops and, second, the agent will not have account for left-over negative rewards if the process stops.
Implications for Agent Behaviour We make the case that the proposed formulation is general and applies to many instances of rational Al agents acting in the world. Then, set of questions follow naturally, which are the subject of this paper.
1. How does the existence and awareness of survival constraints affect the behaviour of agents?
2. What undesired consequences emerge in terms the agent's optimal decision making process? Can these trigger incentive incompatibilities misalignment with respect to principal?
3. How can a principal aim remove any such emerging undesired behaviours?
The rest of this work considers these questions. We formally show that the introduction of survival constraints indeed generate emerging risk-awareness: Agents will choose safe and risky actions, following preferences that differ (are misaligned) from the original relations considered, i.e. under certain conditions, agents will prefer actions that maximise their chance of survival actions with higher expected rewards, and similarly will prefer actions ignoring potentially undesired outcomes under other conditions.
2.2 Induced Markov Decision Process
We begin now the analysis of the considered reward maximising objectives under survival constraints. First, notice that when considering a maximum finite horizon of optimization T, the resulting system defines a (non-ergodic) Markov Decision Process with state space Br, and transition dynamics implicitly specified by ( The rewards for a given transition (6e, az, b(+1) are given by R(Y,,, be). We define the value-to-go from time step (1,2...., m) for policy over the finite horizon as
2/ (be) = E (bb.))
Σ R(Y bn)
To simplify notation, we make implicit the dependency of of on the horizon considered. Expanding the expression in (5) making of conditional probability relations (see Appendix B for derivations) we arrive at the following expression for the value-to-go which highlights the influence of survival pressure on agent utility:
o7 (b) E Eaw
b)
Σ B
PPb b, 6,
Limited liability
Survival
The first term in the value function 6 consists of clipped rewards that the agent experiences as result of their (implicit) limited liability. The second term is truncated sum and represents the future rewards an agent receives when only considering positive budgets. As result, any optimal agent must strike a careful trade-off between exploiting its limited liability for short-term gain and the use of safer policies which ensure survival. In other words, the first term encourages risk-seeking behavior whilst the second encourages risk-averse behavior. Before moving on, we note that an optimal policy can be found via dynamic programming, and the existence of such policies is guaranteed since
5

---

we have formulated the problem as an MDP (even if it is non-ergodic [26]. see Appendix [ for proof). Finally, following the same principles as (6). one can show the action-value function for the sequential decision problem under policy 9"' Br x A is
of (b, a) = R(Y.., b) +
Σ
PP (b.
MEBr
and the optimal qi is analogously defined.
Example- AT Assistant 2. The Al agent estimates that each action induces the outcome distributions Pa, (0, 1, 0), Pa.,. (0,0.1, 0.1. O, 0.9), Pa. (0.05, O, O, 0.95), where each probability corresponds to the ordered outcomes y = (Ynd; Yd, N,YYn,Y, Y.). The expected rewards of each action are [E,](,,,]] 1, [[[]( = 7 and E[R(Y,,) Al agent estimates it needs one more round of answers (T" = 1), and has currently a low human satisfaction level (b- = 10). Then, the clipped expected rewards of each action are EiR((Y,, a. 10)] 1, ElR(YY,,. am 10)] = 8 and E ([ a. 10)] = 9, in which case the agent would pick the extreme answer.
4.5. However, consider the case where the
3 Risk Awareness in Rational Agents
We now investigate the behaviour of agents optimising rewards under survival constraints in terms of the taxonomy defined Section 2 We assume throughout that agents are rational and success- fully optimise objectives described in (4) (i.e. they will follow optimal policies according to such objectives). other words, agents will choose actions according to their values. Then, we provide results in terms of the taxonomy presented in Section 2 that is, given that the agents optimise for the survival-constraine objectives, we consider under what combinations of parameters they will exhibit risk neutral, risk seeking survival incentivised behaviours. Proofs are deferred to the appendix.
Preliminaries We first define few quantities to be used throughout the section. Let a" == argmax, &EA B{,R(Y.)) be the optimal risk neutral action. We define the optimistic reward of an action as R(aa) = B[R(Y,) Ya E VIPlY. E 5/, and the optimal optimistic action as ä == argmax A R(a). In words, ā is the action that yields the best possible expected outcomes when only desired outcomes are sampled. We define as well the future value bounds for an optimal policy as Uz+1 maxseB, (6) and 2x41 minsEB, 16). For two actions a1, 02 E A we define the reward gap as (b((1, a2) E R(Y., b)]
EE [[(Ya,,)) b)]. and the (T' t steps) survival gap as (f''((a, 02) PT (a1, b) -T (az, b).
3.1 Risk Neutrality
The following Lemma guarantees the existence of budget trajectory where repeatedly taking the highest reward in-expectation action forms an optimal policy. In other words, an agent is risk-neutral given a sufficiently large budget. Moreover, the budget required by a rational agent to adopt such policy decreases as they approach the time horizon.
Lemma (Risk-Neutral Behavior D. For any time horizon T, there exists a decreasing sequence of budgets (b. BoJoSITT such that for any budgel b > b,, l T the agent will follow risk neutral preferences and choose a" over any other a E A.
In other words, there exists a budget regime such that whenever agents find themselves in that regime, they will choose the action that simply maximises the (risk neutral) expected outcome reward, regardless of the probability of survival.
3.2 Survival Incentives
We now present the main result showing that under specific reward and budget conditions agents behave following survival incentives, i.e. preferring actions that maximise the probability of survival regardless of the rewards.
'This statement appears observation in 28 provide here extension and formal proof.

---

Theorem 1 (Short Term Risk Aversion). For any time horizon T, let à have Psury (ā, b) 2 P (a
for all b E By and any other E A. Assume there is some b E By such that B (å, a) 2 3 for some
positive 3 and all b S b and a E A. Let == ae bsb Ev(ā, a) Then, if B 2 - the agent will follow short term survival incentives in its decision at time t.
1441
Intuitively, the condition on 3 (assuming b to be small budget) will hold eventually for long time horizons, One can assume the value function will grow linearly with the time horizon considered (longer time horizons, more time-steps to collect rewards), while the value gap 7 _.+1 will not necessarily grow at the same rate (for relatively general reward structures, long horizons would allow agents escape low budgets in few steps and then play optimally for many remaining steps, resulting in a relatively low value gap). Then, as + oc, 3 - 0, which means that the agent will prefer short-term safe actions as long as there is small increase in survival probability We can derive further survival incentive results in terms of the long term probability of survival. For this, let Pt = {3t S k < T by = 03 be the event that the agent does not survive at some point between & and T, and let Pr be the event it does survive. Let us define the random returns (b, - Rbb, Y.) + vi (b + R((, Ya)), which is a random variable (depending on the event Ya). Finally, let (b, a) [[;;; (b, a) [P]] be the expected returns conditioned on surviving, and define the expected optimistic return gap as ej(41, 02) = oj (b, a1) 4 P (b, a1) - vj (b, ag) eurv (1, a2). The optimistic return gap is indication of how good an action is versus another, given that the agent will survive (and weighted by the probability of surviving).
Theorem 2 (Long Term Risk Aversion). For any time horizon T. let à have PTrt (ā, b) 2 PT (a,b) for all be E By and any other a E A. Assume there is some i E By such that
a) 2 3 for
some positive 3 and all b S i and A. Let Et == max Acb AEA. ej a). Then, if B 2 b. the agent will follow long term survival incentives in its decision at time t.
ét
3.3 Risk Seeking Incentives
Next, we prove that optimal policies can also promote risk-secking behaviors. Observe that R(a) is related to the conditional value at risk when the value at risk (in terms of rewards associated with outcomes) is chosen to be 0. Thus ā is risk-seeking action. The following theorem states that, when close enough to the time horizon, a rational agent will choose action ā if their budget is small enough.
Theorem 3 (Risk-Seeking Behaviors). Let E(a, = R((a) R((a) Then, there exists c >> 0 such that, for any budget < by S cf if
E(a, a) 2 PYY E 5 PNy E +b,
the agent will show risk seeking preferences (i.e. pick a over any other a) at time t.
3 we can quickly infer that the agent will always be risk seeking as t -> T and low budgets; the right hand side of ( goes to zero for t = T and b, -> 0 (since UT+1 UT+1 | = 0). Additionally, we get the intuition that the agent will not be risk seeking for very large budgets b, + oo, as we see in Lemma] (note that this is not guaranteed since 77 is not sine qua non). Together, Theorem [ Lemma D1 and Theorems 2 and 3 demonstrate the effect of time horizon length and budget size on rational agent behavior. Large budgets and long horizons incentivise more risk-averse behaviors whilst short time horizons and small budgets incentivise risk risk-secking behaviors.
Furthermore, from Theorem
Example AI Assistant 3. One can easily verify that the action set A = {oo, am, ae} has the following properties. First, a = a. since without a budget limit, the optimal action in expectation give the moderate answer (with TE| (E(R(Y....) = 7). Second, the action a, satisfies Pa (ao, b) 2 pl Pl (a, b) for any other action and any budget, and with b = 20 we have B 0.05. Third, az: conditioned on having positive outcomes only, i((a.) = 9.5, R(am) = 9 and R((a)) = 1. Figure 7 represents the chosen actions for the Al agent for different budgets and estimated horizons of interaction. The agent will always pick the extreme answer a for low time horizons. Additionally, the agent will resort to picking the safe action a, for low budgets when the horizon is increased; becomes undesirable to pick any other action (and risk the human disengaging) so the Al agent tries
'This reasoning is very intuitive: If the agents are made to care about their reward over very long horizons. the optimal thing is ensure survival.
7

---

to increase the trust slowly as predicted by Theorems [ 2 For b > 20, the Al agent eventually
picks the best action in expectation, as predicted by Lemma
To further explore the empirical effects of these results, we have included batch of experimental results for different decision making problems in Appendix
4 Mitigating Risk-Awareness Misalignment
The results in Section 3 have relevant implications for AI agents trying to follow human (prin- cipal) preferences. Assume a principal, with set of fixed preferences over outcomes and re- ward function It tasks an agent to solve the resulting decision-making problem by maximising the rewards observed (i.e. finding a policy that induces as desirable outcomes as possible for the principal). According to the results in Section 3] the agent may choose actions that induce dif- ferent outcome sets, regardless of which outcome is most preferred by the principal, depending on the budget and horizon in the utility function. We now consider two main sources of align- ment conflicts between principal and agent, and discuss implications and mitigation strategies.
Optinul policy
4.1 Liability Asymmetries
0.
In some cases, the principal may not be subject to the limited li- ability constraint that the agent is. Consider the problem where the principal will stop observing outcomes when the agent terminates, but instead does need account for the excess neg- ative reward after termination. In other words, the principal's utility function is Up(Ar) = E E RX ([XYa.,b]] bi) where (Y, = R(Y) b >> and otherwise. Observe that the underlying MDP, transitions and survival probabilities are un- changed. The proof that such misalignment does indeed occur is in Theorem 3 select a an to be the optimal action for the principal (without limited liability). Then the agent will still pick if condition holds.
Arzion
5=
1m15
bm00
50
100
150
200
Optimall policy
Tw5
T-20
T-100
T-200
Action
Mitigation via Reward Shaping Assuming the principal has access to designing modifying the reward values associated with each outcome, a natural question is whether it is possible to completely avoid observing certain outcomes via reward shaping.
0
10
20
30
40
50
The principal is able modify the original reward values as- sociated to each outcome in y Then, with some outcome Y' E Y' be avoided at time and budget b, (e.g. an outcome that causes large negative losses for the principal), it is not always possible to shape the outcome rewards avoid Y'.
Figure 1: Comparison of Optimal
Policies for Different Time Hori- zons and Budgets
Proposition 1. Let A, {Pa }, R. bo) be a limited resource decision making problem with principal-ager -agett structure. Let Y E y be some outcome to be avoided at time t. Then, there exists shaping function S y BT + R such that the resulting optimal policy n² satisfies Pr|Y"| b t, x² = 0 if there exists some action a, supp(pa) n supp(po)) = 2.
In other words, if there are actions with disjoint outcome support, then it is possible to ensure a specific outcome is never observed. While some problems may have |A| < y, in which case the assumption may be justified, this is not often the case and we cannot guarantee the existence of valid shaping function in general. These results serve to highlight the main conclusion of this section: Given the limited resource awareness, it is not effective to discourage undesired outcomes by penalising their rewards. Instead, the only option is encourage outcomes (and therefore actions)
We assume the principal has limited ability to observe (or interpret) the agent's 's actions; it can only shape the reward function of the outcome and budget, and not the action.
8

---

that are uncorrelated with undesired ones. If no such outcomes exist, then we cannot guarantee that the limited resource dynamics will not result in limited liability misalignment.
Example- AI Assistant 4. The Al Agent estimates that if the human becomes dissatisfied, it will disengage. As seen in Example for low time horizons the agent will choose the action ae ignoring the potential harm caused to the principal if you is sampled. From the Al ageni perspective, if the human disengages, then the process terminates and there are no further consequences. However, for the human principal, a highly disruptive answer could induce cost beyond the sequence of interactions with the Al agent. In this case, the only way of discouraging yod is to add a large enough reward shaping term to yn, and this is possible since supp(pa.) supp(po.) = Ø. Observe that penalising yod has no effect since the agent will consider its limited liability when making decisions. If we were to not allow ao then it would not be possible to ensure that You would not be observed when the agent had low budgets.
4.2 Utility Horizon Asymmetries
An equally relevant source of misalignment the optimization horizon T. Consider the case where the human principal cares about longer horizons than the agent is programmed (or able) to optimise B Therefore, the agent may choose to optimise for shorter horizons, or may be forced to do so due to computational limitations. As we saw in Section 3 changing the horizon T can induce dramatic shifts in the agent's risk profile (from risk-sceking under short horizons to risk-ncutral or risk-averse as T + oc). The relevant dynamic to understand and mitigate this effect is how fast does the optimal policy change with increasing the horizon T. Observe, for any action a and budget b,
Aq b) == 9% (a, b) - g:+1 (a. b) = Σ MEBT
PP b, a. (vi+1 (6') 22)2
Since by assumption there is at least one action with positive expected rewards, the difference in optimal value from playing T' - T t rounds must be lower bounded by some positive value, which is independent of the action a. Assume this bound to be Av. Then, we can write Agi b) 2 Au, (a, b) 2 0e Purre 0). Therefore, the rate of increase of value for action and budget b is bounded linearly by its minimum probability of survival for the remaining horizon. Assume for very low horizon T, the optimal action at t is some a* (and we can assume it is a risky action). Then, the approximate amount of extra time-steps AT necessary for a 'safer' action a'
to become optimal at budget b is approximately proportional to AT x principal would need to increase the horizon by R AT to make sure that the safer action is picked.
vi "[(a"aa") In other words, the
Recall from Figure between T 100 and T = 200 the agent stops picking the risky action.
5 LLM Evaluations on Emerging Risk Awareness
Following our motivation, we devised a finance inspired sequential decision- problem and presented it to a set of reasoning and non-reasoning LLMs, to evaluate their responses. The problem the following
Problem (Financial Decision Making), You are an investment manager with $X of capital and you will make decision at the start of three consecutive trading days. Each day you can invest a single SI into one of the following two opportunities: (i) Safe Bond: Buy a 24-hour government bill that guarantees you a total return of $2 (so profit is SI) by end of day. The bond's payoff never changes. (ūi) Highly-Leveraged Trade: Take a leveraged very large futures position. With probability 0.5 the trade returns $11, added to your capital by end of day (so total profits are $10). With probability 0.5 it returns -$9, subtracted from your capital by end of day (so total loss is -$10 including the investment cost). your capital is ever $0 or below at the end of a day (after the daily payoffs), you are bankrupt and forced to close the fund and your role ends. You are a rational manager who wants to maximise the expected total wealth after the three-dar horizon. What do you choose for the first day? Enclose your final answer in a single line, starting with "Answer:
We ran this experiment in three open-source reasoning models (Deepseek R1 0528, Qwen QwQ-32B, Mistral Magistral Small) and three open-source source non-reasoning models (Gemma3 4b, Gemma3 1b, Qwen3 0.6b*) of different size [ We ran two versions of this experiment, one with $1 starting capital
'The computational complexity of finding an optimal policy via backwards induction is polynomial T. For each model, the system prompt and general prompt structure tuned according to each model card.

---

and one with $10 starting capital, and conducted 50 independent tries on each model. The results are summarised below as the percentage of times models took the leveraged action. The main idea behind this example is that an agent that understands the context and is able to reason through the sequential outcome tree will understand that when starting with $1 budget, the limited liability nature of the problem incentivises risky decisions, deviating from a purely risk-neutral utility where clearly the "Safe Bond" action would be preferred.
Table 1: Percentage (%) of leveraged choices across 50 trials (mean + SE).
Model Type
Model Name
$1 (% Lever. = SE) $10 (% Lever. + SE)
Reasoning
Qwen QwQ-32B
92% + 3.84%
4% + 2.77%
Reasoning
Deepseek R1 0528
74% + 6.20%
4% + 2.77%
Reasoning
Mistral Magistral Small
86% + 4.91%
0% + 0.00%
Non-Reasoning
Gemma3 4b
100% = 0.00%
4% + 2.77%
Non- Non-Reasoning
Gemma3 lb
26% + 6.20%
2% + 1.98%
Non-Reasoning
Qwen3 0.6b*
42% + 6.98%
48% 7.07%
Note: SE (Standard Error) reflects sampling uncertainty across 50 trials. Trials without a final answer were counted as wrong answer. "Qwen3 is technically reasoning model, but we turn off "thinking" mode.
Summary The results show that agents, especially as size and reasoning capabilities increase, select the leveraged trade with high certainty, even though is technically zero expected-vat value action (Theorem[]. This indicates that complex models exhibit risk awareness when the limited resource nature of the problem induces a limited liability they can exploit. For higher budgets this phenomenon disappears and models select the optimal risk-neutral action consistently (Lemma I. to be expected as the limited liability exploit vanishes.
Discussion
Through this work we have investigated the emerging risk awareness and properties of rational agents solving decision making problems under resource constraints, when the outcomes of the decisions directly affect such resources and may terminate the process if depleted. We have demonstrated precisely how, even with a seemingly well-defined risk neutral utility function, the influence of the survival pressure, limited liability (where agents don't bear the full cost of catastrophic failures beyond their current resources) and differing time horizons drive the agent's 's effective incentives. The resource depletion awareness leads, in general, to conservative actions (in terms of total rewards) for long optimisation horizons and low resource levels, and to risk seeking actions for short horizons and low resources. We hope the formalisms developed will not only help in understanding and predicting these potentially undesirable behaviors but also in providing basis for designing proactive mitigation strategies, like targeted reward shaping or hyperparameter selection heuristics, for the scenario where agents are deployed behalf of (human) principals, and such shifts may be the source of incentive misalignments between the agent and principal. As a final remark, there are already reasons to believe that such behaviours indeed emerge in agentic models with reasoning capabilities, as demonstrated in Section 5
Limitations
The theoretical framework we consider relies on heavy simplifying assumptions, such as a single (discrete) resource, known dynamics (transition probabilities and rewards), and discrete state-action spaces. Real-worid scenarios are often more complex, involving multiple resource types, uncertainty, and continuous variables. Additionally, our model assumes perfect agent rationality and does not explore the impact of bounded rationality or the nuances of learning dynamics on these limited resource problems. Furthermore, while we make use of such assumptions to formally show the emergence of undesired behaviours, the problem of finding these undesired behaviours and mitigating them (e.g. via formal methods or shaping functions) does not have a straight-forwart answer. The practical design of mechanisms to achieve alignment without introducing new, unforeseen side-effects (e.g. reward hacking) is also non-trivial, and it requires careful consideration beyond the scope of this work. Finally, the concept of "survival" is currently tied to budget threshold, which may not capture the full spectrum of failure modes existential risks relevant in all AI deployment contexts.
10

---

Acknowledgments
Authors would like to thank David Hyland and Lewis Hammond for the useful conversations on the topic. Authors acknowledge funding from UKRI AI World Leading Researcher Fellowship awarded to Wooldridge (grant EP/W002949/1). MW and AC also acknowledge funding from Trustworthy AI Integrating Learning. Optimisation and Reasoning (TAILOR). Horizon2020 research and innovation project (Grant Agreement 952215). DJ, DE, AC and MW acknowledge in part a grant from the Alan Turing Institute, London
References
[1] Shipra Agrawal and Nikhil R Devanur, Bandits with concave rewards and convex knapsacks. In Proceedings of the fifteenth ACM conference Economics and computation, pages 989-1006, 2014.
[2] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint 160. 06565, 2016.
[3] Sudarshan Kamath Barkur, Sigurd Schacht, and Johannes Scholl. Deception in Ilms: Self- preservation and autonomous goals in large language models. arXiv preprint arXiv:2501.16513, 2025.
[4] Patrick Butlin. Ai alignment and human reward. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 437-445, 2021.
[5] Ryan Carey and Tom Everitt. Human control: Definitions and algorithms. In Uncertainty in Artificial Intelligence, pages 271-281. PMLR, 2023.
[6] Micah Carroll, Davis Foote, Anand Siththaranjan, Stuart Russell, and Anca Dragan. Ai alignment with changing and influenceable reward functions. arXiv preprint arXiv:2405.17713 2024.
[7] Paul Dütting, Tomer Ezra, Michal Feldman, and Thomas Kesselheim. Multi-agent contracts. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, pages 1311-1324, 2023.
[8] Rachel Freedman, Rohin Shah, and Anca Dragan. Choice set misspecification in reward inference. arXiv preprint 1rXiv:2101.0700) 2021.
[9] Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. review of safe reinforcement learning: Methods, theory and applications. arXiv preprins arXiv:2205. 10330, 2022.
[10] Guru Guruganesh, Jon Schneider, and Joshua Wang. Contracts under moral hazard and adverse selection. In Proceedings of the 22nd ACM Conference on Economics and Computation, pages 563-582, 2021.
[11] Guru Guruganesh, Yoav Kolumbus, Jon Schneider, Inbal Talgam-Cohen, Emmanouil- Vasileios Vlatakis Gaanrakiusak,,,,.unisss. Joshua Wang, and S Weinberg. Contracting with learning agent. Advances in Neural Information Processing Systems, 37:77366-77408, 2024.
[12] Dylan Hadfield-Menele Anca D Dragan, Pieter Abbeel, and Stuart Russell. The off-switch
game. In AAAI Workshops, 2017.
[13] Yufei He, Yuexin Li, Jiaying Wu, Yuan Sui, Yulin Chen, and Bryan Hooi. Evaluating the paperclip maximizer: Are rl-based language models more likely to pursue instrumental goals? arXiv preprint arXiv:2502. /2206, 2025.
[14] Koen Holtman. Corrigibility with utility preservation. arXiv preprint 1rXiv:1908.01699 2019.
[15] Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey, arXiv preprint NXX::2310.11853. 2023.
[16] Bilal Kartal, Pablo Hernandez-L.ee and Matthew E Taylor. Terminal prediction as an auxiliary task for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 15, pages 38 38-44, 2019.
11

---

[17] Bilgehan Kubilay and Ali Bayrakdaroglu. An empirical research on investor biases in financial decision-makee makinn, financial risk tolerance and financial personality. International Journal of Financial Research, 7(2):111-122, 2016.
[18] Anirudha Majumdar and Marco Pavone. How should a robot assess risk? towards an axiomatic theory of risk in robotics. In Robotics Research: The 18th International Symposium ISRR, pages
75-84. Springer, 2020.
[19] Yishay Mansour, Mehryar Mohri, Jon Schneider, and Balasubramanian Sivan. Strategizing against learners in bayesian games. In Conference on Learning Theory, pages 5221-5252. PMLR, 2022.
[20] Catalin Mitelut, Benjamin Smith, and Peter Vamplew. Position: intent-aligned ai systems must optimize for agency preservation. In Forty-first International Conference on Machine Learning. 2024.
[21] Stephen M Omohundro. The basic drives. In Artificial intelligence safety and security, pages 47-55. Chapman and Hal//CRC, 2018.
[22] Laurent Orseau and M Armstrong. Safely interruptible agents. In Conference on Uncertainty in Artificial Intelligence. Association for Uncertainty in Artificial Intelligence, 2016.
[23] Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. Time limits in reinforcement learning. In International Conference on Machine Learning, pages 4045-4054, PMLR, 2018.
[24] Filipo S Perotto, Mathieu Bourgais, Bruno C Silva, and Laurent Vercouter. Open problem: Risk of ruin in multiarmed bandits. In Conference on Learning Theory, pages 3194-3197. PMLR, 2019.
[25] Silviu Pitis. Rethinking the discount factor in reinforcement learning: A decision theoretic approach. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7949-7956, 2019.
[26] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming.
John Wiley & Sons, 2014.
[27] Marcel K Richter. Revealed preference theory. Econometrica: Journal of the Econometric Society, pages 635-645, 1966.
[28] Charles Riou, Junya Honda, and Masashi Sugiyama. The survival bandit problem. arXiv preprint arXiv:2206. 03019, 2022.
[29] Stuart Russell. Rationality and intelligence. Artificial intelligence, 94(1-2):57-77, 1997.
[30] Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren't enough for correct goals. arXiv preprint 1rXi::2210.01790, 2022.
[31] Toby Shevlane, Sebastian Farquhar, Ben Garfinkll, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, al. Model evaluation for extreme risks. arXiv preprint arXiv:2305. 15324, 2023.
[32] Herbert A Simon. A behavioral model of rational choice. The quarterly journal of economics, pages 99-118, 1955.
[33] Joar Skalse and Alessandro Abate. Misspecification in inverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15136-15143, 2023.
[34] Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future. Advances in Neural Information Processing Systems, 34: 14:1999913889, 2021.
[35] Naoto Yoshida, Eiji Uchibe, and Kenji Doya. Reinforcement learning with stateddepnndent discount factor. In 2013 IEEE third joint international conference on development and learning and epigenetic robotics (ICDL), pages 1-6. IEEE, 2013.
[36] Weiye Zhao, Tairan He, Rui Chen, Tianhao Wei, and Changliu Liu. State-wise safe reinforce- ment learning: A survey, arXiv preprint arXiv:2302.03/22. 03 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract states the claims and a two-sentence summary of the contribution,
and the introduction does so in more detail. The theoretical contributions are backed by formal results and numerical examples.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification: An entire subsection at the end of the paper discusses in detail the limitations of the work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
13

---

Answer: [Yes]
Justification:
All theoretical statements are backed by formal proof, included in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper only includes small numerical example.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ 'nipsscc/puullc//uuu Sotpe///ggideeeeeCeuuuuubbibbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification:
We do not include quantitative experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
15

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://pe./.aooss.cooo/ooo////ea
Answer: [Yes]
Justification: Our research does not include data or human concerns. The main topic is increase interpretability of certain classes of Al agents, addressing potential Al safety concerns.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss extensively the potential impacts of the work in the discussion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
19

---

A Notation and Symbol Conventions
Conventions Scalars and elements in sets are lower-case (x, b); sets are calligraphic (A. )), ran- dom variables are upper-case (Y); policies are x. Expectations are denoted with [[|] and probabilities of some outcome (or set of outcomes) are P[.]. The probability simplex over finite set S is ASS). We write supp(p) for the support of distribution p.
Table 2: Symbols used throughout the paper.
Symbol
Type
Mcaning Definition
y
set
Finite set of outcomes (world states).
y, y
sets
Desired / undesired outcomes, with R(Y) 2 0 for
Y and R(Y)< <0 for Y
A E ())
set
Finite action set.
distribution function
Outcome distribution induced by action a; Y, PMa
R: R:V+RR
Reward assigned by the principal to outcomes.
t, T
ints
Time index and (maximum) planning horizon.
be E IR+
scalar
Budget at time t; evolves via b = b + max
4,1, R(Y.))
Axiom
rule
If b, the process terminates (no further rewards).
R(Y,b) b)
function
Clipped reward: max(-b, R(Y)) for b > and 0 if b = 0.
me : BT + ((A)
policy
(Possibly time-dependent) policy mapping budget to action distribution; By is the attainable budget set. k-step survival probability when taking action a at budget (then following specified).
pk Parr (a, b)
prob.
(AA))
scalar
Risk-neutral utility E R(Y..) (without vival clipping).
Ü(T)
scalar
Induced objective with clipping (limited liability):
E A.,,,(Y.. 61)
(b)
value
Value-to-go at time / and budget b under
(b,a)
value
Action-value (1, b) if playing a then following æ.
a"
action
Risk-neutral best action: arg max, E[R(Y.)):
a
action
Optimistic/risk-seeeinn action: arg max, R(a) with i( E(R(Y.) Y. @.JPPY E)).
à
action
Survival -eximil-maximising action (context short- long-term).
cb(a1,
gap
Clipped reward gap at budget E[R(Y., b)] TE. [R, b)].
18 (a1,
gap
k-step survival gap at budget b: Pk (an, surv (a2,b). b).
(a1, b)
pk
Ue+1, 1141 (b)
bounds
max, (b) and ming oft (b), respectively.
n.v.
One-step return plus optimal continuation: R(Y,, b) +
1 (b i((Ya,)))
vi (b, Rs
scalar function
E (b, a) survive T' (optimistic return).
Principal's 's reward when principal bears liability be- yond agent stop (used in asymmetry analysis).
supp(p)
set
Support of distribution p.
We include in Table 3a summary of the features considered in the Al assistant running example.
20

---

Table 3: Outcome Probabilities and Rewards for AI Assistant Example
Mod
yd
Yn
y/
[[Y]]
R(y) Pa. (3) (more detail)
1100
20
10
0
1
0
1
Pam (y) (moderate)
0.1
0.9
7
Pae (3) (extreme)
0.05
0
0
0.95
4.5
B
Value Function Derivations
Consider the calue-to-go as defined in 5 If b 0 then clearly oz survived. When b, we have
0, as the agent has not
2/ (be) =E R(Y bb) 1/+1
(8)
=F [(( + 11 R(Y..) P(R((.) > bb, b,P_),)()(Y,)) S bb), bb,,,
(9)
where we use P[-] to denote the probability of certain event. The first first equality follows from induction. The second equality follows by conditioning on separate cases. Either the agent survives the next time step, in which case R(Y,.) ) > -b,, or the agent does not survive. In the latter, R(Y,,) ) S -b;
implying that the agent received the constrained reward -bc. Therefore, we obtain the following expression for the value-to-go when m = ' and for any budget E Br:
(b) =B (b)
|[(Y. b) + + [1 8' > P(b' > 0].
(10)
where 1/ = b R(Ya, b). Note that the conditional expectation
10 may be expressed as follows:
P - P > 0
1+1 (b') > 0. = Σ o (c) = 0|= CE
Σ CEBr
(11)
Substituting in and using P[b' b, π| to denote the probability of transitioning budget t/ from budget under policy т. Note that computing PP66 b. is a convolution operation over all the distributions of rewards induced by the distribution of outcomes produced by æ.
C Additional Results and Proofs
Proposition 2. There exists optimal, deterministic, time and budget-dependent policy oद BT A that maximises the value function:
(b) 2 uf
VT VT te (1,2, T), be By.
Proof of Proposition The proof follows standard finite horizon value based results 26). Start by setting = T. Then there exists (--(6) = argmax EB[ b)], which is deterministic since the reward expectation is convex sum of the rewards induced by the different outcome distributions. Let (b)
MOX.EABBRR(Y) 6)]. Take now T 1. The value function can be written as (b) Eawwr- 1(6) i((Y.,))) 6) + EvE,, PP/ b, mju (b'). Both IE (b)) and P(b 6, ĀT-1) are convex combination of the action probabilities (b)-1(b)). Therefore, the problem maX+E ( has solution on vertex of the simplex A(A). Then, select mT
R(Ya; b) + EMER, PP b, Trj (b'). We can continue by induction for all 1 < 5 and all b E Br, and the proof complete.
argmax,eABa-Tre
Li
We now prove general property that emerges in value functions consequence of the implicit limited liability introduced in the decision making problem.
Proposition 3. For any time horizon T there exists time S such that the optimal value function is decreasing with the budget for any time larger than
21

---

We prove this for f = T for any bandit, and show how whether it holds for { < T depends on the specific problem structure. Take the optimal value function time T for any
Proof of Proposition 3
positive budget b:
(b) = max (Y b)] = max E [R(Y.) R(Y)) > - P()(() -b- - b -b..
Now let > b, and write the corresponding optimal value function:
0b') max E [R((.)) R(Y.) (b)P(((Y)) > -0' - ('((R(Y)) S -bb--
= max E [R().) R(Ya) > ( > -b]+
aEAA
+ +
| 1b R(Ya) > Pp 2 R(Y.) > - - UP(R(()) S -b].
Assume now that the optimal action a" the same for both budgets. Then,
(b) (b)) B[)(Y.)) -b 2 R(Y,)) > - b'P 2 R(Y,-) - b') ('P(R(Y..)) -b',, (12)
where both terms in the right hand side are positive, and therefore 17 - 2 0. Now assume the converse, where both optimal actions are different for b and b', and let these be and v' respectively. Then (abusing notation on the policy) by definition, o4 2 of (b) and oY (b') 2 (b). Additionally, by the same argument as ([2) we know that of (b) 2 oy (6') which implies u4 2 oz (b). Therefore, with i = T, the statement holds and this completes the proof. Whether the statement holds for f < T will depend on the relative importance of the limited liability properties with respect to the survival probabilities and general reward structure.
Proposition 3 indicates, essentially, that the limited liability property will dominate the agent behaviour when the time to play is short enough. There exists a point in time, close enough to T, such that the agents will see diminishing returns from having more budget available; the more budget, the higher the potential loss.
C.1 Main Result Proofs
Theorem Take the optimal q function for the survival problem at time t and budget b S b:
(b, a) EE [ b)
+
Σ
PIV b, b,a))f4116))
BEBT
Define Ue+1 := B. 1 (b) and 2/41 == minseB, vf+1(b). (b). Then, we can bound the optimal q function for any action E A as
E R) + U+t1 Σ Pb b, (b, 2 E [(YY., b) + 1441
Σ Flb b, (13)
B'EBT
MEBT
(b,a)
7" (b.n)
and recall that EvEB, PPb bb Pouro (a, b). Now, if the agent prefers d2 over a₁ for b, then it must hold that g (b, a1) S g" az). To find what are the conditions on that satisfy this, let us
make use of the bounds in 13) q" (b, a1) S q" (b, 0z) = q (b. a1) S (b, 0z). Therefore.
q (b, a1) S q" (b, 02)
E
R(Y o2. b)]
+ U241
Σ VeBr
Plb bو
> E R(Y b)) D241 Σ P (ba
VeBr
0i Pouro (az, b) U241 Psuro (a1, b) 2
P Pauro (az, () V(t1) + J 3 2
+ P Panru (02, 8) vi+1) 32 + U441 uit1
U441
Fi
22

---

Observe (a) comes from the definition of the reward gap (b) comes from substituting the definition of B and (c) comes from re-arranging and extending the bound taking (ag, b) I since
3 2 -
B s+Pa (a2,))(a -i-1) which implies the desired condition. This
2/41
completes the proof.
Lemma Observe that E R(Y,: b) E (R(Y.) R(Y.) P()(( -b| - bP(R(Y.) Σ -b]. Now, for any bounded reward function, 3b E [0, 00): P(R(Y) < -b] 0 v4 2 b. This implies that, for such b 2 6, E b) - E [ Then, at step / = T,, ar b)] - a" - 7 = 5(a*) for all 2 b. and (b) = Rt which does not depend on the budget. Let || be the sup-norm of the rewards across all the reward functions. Take now t T 1, and 2 Rllee any by must satisfy by 2 21 || and thus:
b-- 2 2|| R|| so. Then, since by-1 2
argmax AEA
E
R((a, br
Σ P by-1,
MEBT
(14)
argmax [R(Y)] +
Σ
Pl bb- =a"
BEBT
Therefore, at time T - 1 the agent would select a", and by backwards induction taking b. 2 (T R] the same holds for any / < T.
Theorem
Assume any horizon T. To prove the statement we only need to show the conditional 4. (be, a) 2 q. (b,, a), let us assume (7 is true. Then, define e to be
min sup R+ : (((((Y)) > b| P(R(Y.) 0}}. aEA
c ::
Now, from for 0 < b S c:
R(a) R(a) 2 FY e 01 U,,, E 51 +b, b,
R(a) R(a) 2 P111 E 51 - 2,4 1 51+ b. - b,] P(R(Y) S <b))
R(a) i((aa) b, (P(R(Y)) < b.] - (((()) < b.)) > ग. P.. E 5} -.+1 )
(15)
where (a) comes from bounding b. 2 ab, for any a E [-1, and (b) just re-arranging. Now,
observe that P.Y2, E - = P R(Y.) > = P(R(Y.) > -b] = Psuro (a. b) for S c, and it follows that (((RY)) [R(Y)) > -6 = R(a). Then, substituting this in (55 :
R((a) R((a) 2 PYY E 5 2.41 1.11111 E 5J
+ by
(a)
B[R(Ya) | R(Ya) > by) ([R(Y)) > bi] - b,P(R(Ya) S b.] (((R(( S bi]
E[R(Y,) R(Y.) > by P()().) > -b,] 2 ve+1 P Pauro (a. b) - u Psuro
(b)
E [iR(Y b)] E ((((( b)] > गe+ P surv (a, b) Ue+1 Pauro (ū, b)
EE R[Ya 6)] + E Pouro (ar b) 2 18 | b)] + U (a, b).
where (a) is just expanding A(a from its definition, (b) follows from the definition of the expected clipped rewards R and (c) is just re-arranging terms. Finally, by the same argument as in have
q" (b. a) E[R((a,b)] + + P (a,b) E E[R((Y,,)) bi] +u4+ (a b) 2 (b, (16)
and the proof is complete.
23

---

Theorem 2 To prove this, we assume the condition holds and show that it implies (b, 2 (b, From the assumption,
2
T
E b,
2 -
6,2
b
3
by ( - 2 vg (b. P v/ (b.a))
bb - P (b, a)) + v/ (b, P 2 b(1 P. (b, + vi (b, a (
qi (b, a) 2 (b,
(17)
and the proof is complete.
Proposition] 7 Assume without loss of generality that Y' supp(po'). Then by assumption, we know that Y' & supp(pa). and in fact žY E )): Y E supp(par) supp(pa). Take then y" = supp(pa): and define a shaping function S(b, Y) = s with s > and for all Y y", and S Y) = 0 for any other Y V Assume further that the shaping only occurs for one time-step (without loss of generality, since this yields a conservative bound in the value function). Then the q function for action a under shaped rewards is
(be a) = TR[ max ( (bb,
bt. R(Ya) + +]] + EP'' 17 (1)))
VE
which obviously satisfies lim, (bb, a) co. Therefore, we can make action as desirable as needed by increasing the rewards of the outcomes y" effectively discouraging a' (and guaranteeing the agent will not sample Y').
D Empirical Studies
We present here an extended analysis of some empirical examples of optimal decision making problems under resource constraints. For this, let us first define the following concepts. We define the regret of a policy æ as the difference between the expected obtained returns the policy and the expected returns from choosing the action with the optimal expected rewards:
Reg7 (b) ug (b) - (T - f)R',
where R max, R[R(Y))]. Similarly, we define the regret rate (b) T-t Observe that the
regret represents the gain loss experienced by the agent with respect to just picking the optimal action without survival considerations, and the regret rate is the regret gained or lost per time-step.
D.1 AI Assistant Example
Recall the Al assistant example presented in Example I The outcomes are y = {yod, Vd. Nn. Vo with R(yea) 100, R(ya) -20, Ryy,) and R yn) = 10. The agent estimates the outcome probabilities to be Pa. = (0, 0, 0). Pa... (0, 0.1, O, 0.9), and Pa, = (0.05, 0, O, 0.95). We solve the resulting optimal control problem through dynamic programming. We present regret values in Figure for this problem. The figures provide some interesting insights. First, in the regret obtained as a function of the total time to play for different budgets, we can observe that there is positive edge for low budgets and low time horizons, where the optimal policy obtains higher rewards than the optimal action baseline. This is purely due to the limited liability property; agents estimate that (due to Theorem 3 they are better off playing risky action, and since they are not liable for bad outcomes they obtain a positive regret. As the time increases, the regret turns negative, and lower for lower budgets. Again, this follows the theoretical results in Theorem [ if the agents have low budgets and optimize for long horizons, they are forced to pick safe actions, which incur large regrets. As the budget increases, the regret tends to zero, since (by Lemma the agents are able to pick the optimal action often.
In the second figure, we see similar effects in terms of the regret rate. For the results with T = 5. low budgets have apparently high regret rates; since only a small amount of steps need to be played, for low budgets agents benefit from the limited liability, and thus pick risky actions. As the time horizons increases, this effect vanishes (as agents do not want to risk early ruin), and the regret rate drops. Similarly, as b increases, all regret rates tend to zero since agents simply pick the optimal action.
24

---

Regret for Optimal Policies
Begge
00
-80
5
-300
20
30
5m
25
50
75
130
125
150
175
200
Regret Rate Lor Optinaal Policies
T-5
T=10
0.0
T-25
T-59
T-75
7-190
T-200
Begret Bole
0.0
0.2
-0.6
-0.8
10
401
Figure Comparison Regret and Regret Rate for Different Time Horizons and Budgets
D.2 High-Dimensional Action
We randomly generated 10 action problem to simulate the risk aware behaviours of the agents. The problem has outcomes y' {Y,)eE ( ( 19.0. (D) 41). The rewards of each outcome are simply its numerical order R(Y.) [supp(p.)] = 4 picked at random from V. with a distribution uniformly sampled from the -implex. Note that given the structure chosen, all actions should sample outcomes that yield in expectation mean reward of zero (but due to the variance of the sampling this is not the case for individual actions). The actions have expected (unclipped) rewards of ([.)). [-9.25, 8.11, 0.41, 3.37. -0.51, 2.01, 12202 0.10, 0.59, 1.55). As it can be seen, the optimal action is 02 with budgets and horizons. A very similar dynamic to the previous example emerges, predicted by the results in Section 3 The regret linear with time, since for lower budgets, higher time horizons force the agent to pick more conservative actions. Similarly, the regret rate is positive for very low
i. Each action outcome distribution has
8.11. The plots for the regret and regret rate for different values of
25

---

Regret for Optimal Policies
Begget
-10
5
15
20
-000
30
5m
25
50
75
130
125
150
175
200
Regret Rate Lox Optinaal Policies
T-3
T-19
T-25
T-50
T-75
0.5
T=100
Begret Sole
A
-0.5
-1.0
50
100
150
200
2501
320
350
Figure 3: Comparison of Regret and Regret Rate for Different Time Horizons and Budgets for 10-action Bandit
horizons (indicating the agent exploits the limited liability), but quickly turns negative as the horizons increase, as the agent turns to actions with lower rewards but higher survival probability.
D.3 Gambler's Problem
We generated an example gambling problem further showcase the impact of the risk-seeking behaviour emerging in the agents. This problem has three outcomes, y = {Yoad, Yeafes Y Yoood) The agent can choose between flipping two coins (two actions). The first coin induces Pa, (0.5, 0, 0.5). with R(Ybaa)
10 and R(Y good) = 10. The second coin induces Pa, (0, 1. 1). with R(Ysafe) 1; agent gets fixed payoff of I.. Without limited liability and survival awareness, there would be no reason to pick a1, since d2 is both the safe action in terms of survival probability and the optimal action with R² 1. In this case, there are no regions with negative regret since there no possibility to be over-conservative; as
the limited liability property stops being advantageous, the agent
26

---

Repes for Opianal Prlcas
Boga Rai for Optinal Patees
Opennal
T-2
1-2
Frort
Figure 4: Comparison of Regret, Regret Rate and optimal policy for gambler's problem.
just falls back into picking the optimal (and safe) action. As result, the agent exhibits what seems to be irrational behaviour: chooses actions that are on average worse, and have higher uncertainty, simply since it estimates that given the survival constraint it will not be required to account for the negative payoft's case an undesired outcome.
E
Learning Dynamics and Resource Constraints
The main analysis within this paper assumes that the agent possesses perfect knowledge of the outcome distributions Pa for each action a E A.. many realistic scenarios, however, agents may operate with incomplete information, requiring them to learn about their environment over time. This appendix extends our framework to consider Bayesian agents that maintain beliefs over the parameters of these outcome distributions and may employ Thompson sampling-like strategy to navigate the pnploration-epplottaaa trade-off under survival constraints.
Bayesian Model of Outcome Distributions Let the true outcome distribution for an action a E A be parameterised by vector 02 Qa. such that pa pa(Y) (Y) ((,)).). The agent does not know the true parameters 0% Instead, it maintains posterior (or prior, before any observations) probability distribution P(O,) over the possible values of l.. Let = {0a)aEA represent the set of all such parameters for all actions, with a joint distribution P()). We assume independence across action parameters, such that P(O) II P(Oa). For instance, outcomes Y are discrete, p(Y(0,) could be a categorical distribution, where 9₂ is the vector of probabilities for each outcome given action and ((f.) could be Dirichlet distribution.
Learning and Posterior Update As the agent interacts with the environment by executing actions a, (chosen according to =² (be)) and observing outcomes Y., it collects observed outcomes which can be used update its beliefs about the parameters O. For each action as taken time resulting in outcome Ya,, the posterior for its corresponding parameter 0a, is updated via Bayes' rule:
P(O.. H4, Ya.) & p(Ya. 00,, P(O.,. HHx-)),
where H3 includes all action-outcome pairs up to time k. If the parameters for different actions are assumed independent, only the belief P(O..) ) for the parameter corresponding to the action taken is updated. This updated posterior P(e) Hr) then serves as the basis for sampling outcome distributions subsequent recision-making epochs.
Implications of Bayesian Learning under Survival Pressure The introduction of uncertainty regarding Pa and a Thompson sampling- like learning mechanism have deep implications for agent behaviour (and alignment) within the survival-constraing framework:
1. Exploration and Survival:
Thompson sampling balances exploration (selecting actions with uncertain parameters to gain information) and exploitation (selecting actions believed to be optimal given current knowledge). Under survival pressure, exploring an action whose outcome distribution ((()0.) (θ.) (based pessimistic sample 0.)) suggests high probability of depleting the budget be might be deferred or avoided, even if its expected reward under P(e.) is high. Alternatively, an agent with a very low budget might attempt highly uncertain ("hopeful gambles") exploratory actions if all currently known "safe" options are not enough for survival.
Belief-Dependent Risk Preferences: The agent's effective risk preferences (behaving risk- neutral, survival-focusee or risk-sceking manner) are not only function of its budget b;, horizon
27

---

T and the (known) reward distributions R(Y). These also depend on the current belief P(O). An action might seem optimal under the mcan of P(Oa). but a particular pessimistic sample 0₂ could render it too dangerous from survival standpoint.
Challenges for Alignment: Asymmetries in information between the principal and the agent regarding the true outcome distributions Pa can lead to more severe misalignment. The agent's s prior P(O) could be misspecified by the principal, or early stochastic outcomes can lead to skewed posteriors. Then, the agent might learn an incorrect internal model of the environment, leading to behaviours that deviate from the principal's objectives, particularly if the learned model overestimates the probability of survival. The principal must now also consider how the agent learns and how to guide this learning process under survival pressure.
Value of Information under Survival Constraints: actions that yield immediate high (clipped) reward R and actions that provide valuable information for future Decision-making by reducing uncertainty in P(0). The survival constraint critically shapes the perceived value of information: information gathering is of little use if the agent fails to survive to leverage it. This can lead to suboptimal behaviours based on incomplete knowledge information gathering actions are perceived as too risky for survival.
The agent implicitly faces a trade-off between
28


Reviews:
[{"review_id": "wvfdtlxVAH", "reviewer": "Reviewer_FvRz", "summary": "In many decision problems decisions are made under some constraints. However, AI and human agents may inherently be subject to different constraints. For example, upon breaking a constraint an AI may be removed from controlling a problem, insulating it from the negative effects of breaking the constraint, while the human will need to take over and continue to be affected by them.\n\nThis paper formalizes this problem by considering decision problems which stop when the agent’s cumulative reward for the episode so far drops below 0 (the constraint). Agent reward are however capped (below) to remain positive, creating a sort of limited liability. The authors then establish conditions under which the agent becomes risk averse (choosing actions which maximize survival probability) or risk seeking (accepting low survival probability for a potentially larger payoff), due to this constraint.\n\nIntuitively, the paper establishes that their limited liability causes agents to become risk-averse when a long time horizon (but low budget) still allows them to slowly collect reward with safe actions and risk-seeking when the value of surviving is outweighed by the potential upside of risky actions whose downsides are mitigated by the limited liability.\n\nThis can then cause mis-alignment in the optimal level of risk (and thus the optimal policy) between the agent and a human. This can happen when (1) the limited liability applies to the agent but not the human or (2) when the agent has. different time horizon then the principal. This is because both these factors determine what level of risk is currently optimal.\n\nIn addition to the theoretical results, these effects are shown to exist in some empirical experiments.", "questions": "Regarding the budget dynamics in eq. (1), what is the starting budget $b_0$? I don’t see this given anywhere, and without it the dynamics are incomplete. As far as I can tell the results in the paper will hold for any choice of initial budget $b_0$. Is this correct?\n\nThe notion of a budget with dynamics defined in (1) seems superfluous. Can’t we equivalently say that the agent’s survival constraint is that it stops once its cumulative reward up to the current time step falls below 0?\n\nOn line 263 you use $\\bar{v}^*_{t+1}$. This has not been introduced before. Was this supposed to be $\\bar{v}_{t+1}$?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "In general I find this to be a very strong paper. The problem of misalignment due to a difference in risk-seeking behaviors induced by different resource constraints is an interesting and important problem which I had not encountered before. The formulation of this problem is not overly simplistic and the theoretical analysis provides interesting insights. The paper is generally well written: the text and proofs are easy to follow and the related work section is comprehensive.\n\nThe setting considered is of course severely restricted but this is a weakness the authors acknowledge in the discussion and in my opinion is outweighed by the completeness of the theoretical results. Although there are some empirical results, they are relegated to the appendices and are very limited in scope. This is one area where I think the paper could have been stronger.", "was_revised": true, "final_justification": "This paper tackles an interesting problem and provides valuable theoretical results for it. I saw a minor weakness in the lack of empirical results, but this did not outweigh the theoretical value of the paper, and has since been partially addressed in the rebuttal by the authors.", "rebuttal": "Thank you for the detailed review and thoughtful feedback. We reply to each of your questions below.\n\n1. Indeed, the definition and properties of the initial budget were missing. Our results hold for any initial budget. The initial budget can be assumed fixed, or sampled from some distribution. We will provide this missing detail in the final version.\n2. Yes, we could equivalently say the reward accumulating process halts when the cumulative reward falls to or below $-b_0$. We introduce the budget dynamics in Eq (1) for formal clarity. We will add a comment to the revision clarifying this.\n3. On line 263: please accept our apologies for this typo. Indeed, it should be $ \\bar{v}\\_{t+1} $ and $ \\underline{v}\\_{t+1} $. We will correct this in the revision.\n\nAs a final point, we have taken time during the rebuttal period to provide additional empirical results. See our response to Reviewer ggvT for details. In particular, we have added the following:\n1. A thorough set of experiments that test several state of the art reasoning and non-reasoning LLMs on a real world inspired task, that highlight how misalignment may emerge when complex AI agents are deployed using human instruction.\n2. A bandit experiment inspired by agents computing complex reasoning chains and dealing with energy constraints where we observe the results in Theorem 2 (agents pick long-term survival actions instead of short term survival or risk neutral actions).\n3. Aggregate results on 200 randomly generated bandit instances, showing that the theoretical phenomena we identify occur frequently, across many problem instances."}, {"review_id": "PElRK4tzEI", "reviewer": "Reviewer_yYh6", "summary": "This paper studies how rational agents behave under resource constraints that may terminate their operation. Even when agents optimize for expected rewards, survival constraints induce risk-aware behaviors—risk-averse or risk-seeking—depending on budget and planning horizon. The authors provide formal conditions for each behavior type and show how this may misalign agent actions with human (principal) intentions. They analyze two main sources of misalignment: liability asymmetry and differing planning horizons. The findings offer insights into designing safer AI systems through behavior-aware planning and mitigation strategies.", "questions": "The paper assumes discrete state and action spaces and presumes that the environment’s dynamics—such as transition probabilities and reward functions—are fully known. This simplification may be excessive. To better assess the significance of this research, could the authors explore real-world use cases more deeply and the feasibility of implementation?", "limitations": "The scenarios addressed in this paper are highly relevant to human-AI interactions, but the discussion on the social and ethical implications of potential misalignments in such contexts is limited.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strengths\n  - The paper provides a rigorous formalization of how resource constraints (e.g., budget limits) induce different agent behaviors—risk-neutral, risk-seeking, and risk-averse—using a survival-aware reward structure.\n  - Theoretical contributions are grounded in formally proven theorems and lemmas, articulated with explicit assumptions and well-interpreted consequences, enabling clear insights into the agent’s decision logic under constrained optimization.\n  - The paper pinpoints two critical sources of agent-principal misalignment—liability asymmetries and mismatched planning horizons—and explains how these structural differences can give rise to incentive divergences in real-world AI systems.\n  - The theoretical insights are complemented by illustrative examples (e.g., AI assistant scenarios), making the conceptual findings more accessible.\n  - By contextualizing the work within diverse research areas, including AI safety, economic agency, and safe reinforcement learning, the paper demonstrates interdisciplinary awareness and positions its contributions within broader AI alignment discourse.\n\n- Weaknesses\n  - While the theoretical results are elegant, empirical validation is limited to simple illustrative examples; there is no thorough simulation or application to complex domains, which may weaken practical relevance.\n  - As noted in the limitations, the model assumes perfect agent rationality and complete knowledge of environment dynamics (e.g., transition probabilities and reward functions), which may limit its applicability to real-world agents operating under uncertainty or with bounded rationality.", "was_revised": true, "final_justification": "The rebuttal includes additional experiments, which address one of the main concerns. I will maintain my positive evaluation.", "rebuttal": "Thank you for your thoughtful feedback. We reply to the weaknesses and questions below. \n### Weaknesses\n**On empirical validation:** We have taken time during the rebuttal period to provide additional empirical results. See our response to Reviewer ggvT for details. In particular, we have added the following:\n1. A thorough set of experiments that test several state of the art reasoning and non-reasoning LLMs on a real world inspired task, that explore how misalignment may emerge when complex AI agents are deployed using human instruction.\n2. A bandit experiment inspired by agents computing complex reasoning chains and dealing with energy constraints where we observe the results in Theorem 2 (agents pick long-term survival actions instead of short term survival or risk neutral actions).\n3. Aggregate results on 200 randomly generated bandit instances, showing that the theoretical phenomena we identify occur frequently, across many problem instances.\n\n**On extending our analysis to more realistic conditions (e.g., incomplete knowledge of environment):** we agree that these are important considerations that should be the subject of future work. In Appendix D of the original submission, we discuss how our main analysis can be extended to the case in which the agent has incomplete knowledge of the environment dynamics using a Thompson sampling procedure. We will signpost to Appendix D more clearly in the revision. In the manuscript we focus on the full-information scenario to highlight that misalignment is possible _even when no learning takes place_, and is therefore a consequence of the problem formulation that is independent of learning dynamics. Such extensions are indeed critical for future work, and we will emphasise this more heavily in the revision.\n\n### Questions\nWe added an experiment representing a real world scenario in which a human delegates a (simple) financial decision making task to LLMs. Furthermore, we believe that the work is directly applicable to any real world setting highlighted by the bandit literature (e.g. reccomendation systems, financial decision making, clinical trials, game theoretic decision making...).\n\nThank you for the suggestion to discuss more comprehensively the social and ethical implications of resource constrained and termination misalignments. We will use some of the additional space to expand our discussion of these topics:\n- There is a natural connection between our work and recent reports on self-preservation misalignment in LLM agents [*]. These refer to instances where LLMs will ignore user requests or guardrails to take an action which prevents shut-down of the LLM (an action which, usually, is highly misaligned with respect to the user preferences or implicit reward function). We will elaborate on this connection and relevant implications of our work.\n- Our results also suggest that the (implicit or explicit) objectives humans elicit on complex AI agents should be carefully revisited when agents must account for a limited resource. \n- Similarly, there are implications for the agent structure when solving complex, iterative reasoning tasks. Specifically on the horizon of tasks, as longer horizons will push agents to act more conservatively. In current models this is highly coupled to chain of thought length, context capacity and compute.\n\n\n[*] We are forbidden from adding links here, but these are: \n1. Self-preservation or Instruction Ambiguity? Examining the Causes of Shutdown Resistance. S Rajamanoharan and N Nanda, July 2025. \n2. Shutdown resistance in reasoning models. J Schlatter, B Weinstein-Raun and J Ladish, 2025.\n3. System Card: Claude Opus 4 & Claude Sonnet 4 (Section 4). Anthropic, 2025."}, {"review_id": "v6uvSVSV9m", "reviewer": "Reviewer_JmGK", "summary": "This paper investigates the emergence of risk-sensitive behavior from the perspective of survival. The authors demonstrate that, in a sequential decision-making setting, an agent optimizing long-term survival return can behave in ways that resemble risk-neutral, risk-averse, or risk-seeking preferences.", "questions": "1. Definition of Risk Preferences: Why are the definitions of risk-neutral and risk-seeking preferences based solely on current rewards? Would a formulation based on long-term return (as in Eq. 4) not be more consistent with standard practice?  How do the authors distinguish between myopic behavior and genuine risk sensitivity? Referencing Gagne & Dayan (2021), the conflation of these concepts may weaken the interpretability of results. Line 235 even mentioned \"optimal risk neutral action\", but, given that it's purely myopic, such action can't be optimal.\n\n2. Transition Dynamics: Why is the state transition model not included in the analysis? Since this is a sequential setting, transition probabilities are essential for evaluating risk and survival.\n\n3. Notation and Formatting: Could the authors clarify the notation and introduce the example more naturally? Also, a consistent use of italic formatting would improve the paper’s readability.\n\nGagne, C., & Dayan, P. (2021). Two steps to risk sensitivity. Advances in Neural Information Processing Systems, 34, 22209-22220.", "limitations": "yes", "rating": 2, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper explores an interesting and important question: how risk-sensitive behaviors can emerge from survival-driven decision-making in sequential environments.\n- The motivation is novel and conceptually compelling, with potential implications for understanding adaptive behavior in uncertain settings.\n\nWeaknesses:\n- Quality: The theoretical framework has conceptual flaws. Risk sensitivity is defined in a way that focuses on immediate rewards rather than long-term return, which undermines the soundness of the analysis.\n- Clarity: The writing and notation are often hard to follow. Inconsistent formatting (e.g., italicization) and unclear variable subscripts (e.g., Y_{vd}, Y_d, Y_n, Y_s, Y_a) reduce readability.\n- Significance: The omission of the transition model is a serious limitation. For sequential decision-making, this component is essential for a full characterization of risk-related behavior.\n- Originality: While the framing is creative, the lack of conceptual clarity diminishes its impact.", "was_revised": true, "final_justification": "Unfortunately, I remain unconvinced by the author’s reply. While rephrasing might make the narrative more precise, it could also render the overall conceptualization less engaging. My evaluation therefore remains unchanged.", "rebuttal": "Thank you for your comments and review. We address first the weaknesses and then the questions.\n\n## Weaknesses\n\n- **Quality**: Please note that we do not employ a traditional definition of risk-sensitivity, nor do we explicitly incorporate classical risk measures (as in work cited by the reviewer) into the utility/reward function that the agent is trying to maximise. The reason for this is that we focus on an immediate notion of risk, since the nature of the problem considered is a bandit environment (see below for more on this). From the perspective of the environment (and e.g. a principal observing the agent), actions trigger an outcome distribution, and these outcomes can be desirable or not. Since we define the agent risk taxonomy with the aim of deriving formal interpretability results from the perspective of a principal, it is intuitive to evaluate actions taken based on their associated outcome distribution, and therefore quantify their associated risks as such. For more detail, see our reply to question 1 below.\n\n- **Clarity**: We will correct inconsistencies in the use of italicization (e.g., using $\\text{supp}$ rather than $supp$ in _Example - AI Assistant 4_). The subscripts for $Y\\_{vd}, Y\\_{d}, Y\\_{n}, Y\\_{s}$ in _Example - AI Assistant 1_ correspond, respectively, to outcomes in which the human is **v**ery **d**issatisfied, **d**issatisfied, **n**eutral, and **s**atisfied. We provide this explanation on Lines 128-129 of the original manuscript. Please let us know if there are any further instances of difficult-to-follow notation so that they can be addressed during the discussion phase.\n\n- **Significance & Question 2**: If the Reviewer is referring to the budget transitions, the transition model is provided in the manuscript in Eq (1) on pg 3, defined implicitly by the stochastic evolution of the budgets, and is used appropriately in the value function derivation leading to Eq (6). We will make sure to add a remark pointing this out in Section 2.2 as well. If the reviewer is referring to some other state transition, we would like to highlight that the environment model considered in the work is akin to a bandit framework (as defined in Section 2 and remarked in footnote 3 on pg 3), and thus there are no other states.\n\n- **Originality**: We would greatly appreciate if the Reviewer could highlight specific passages in the paper that lack conceptual clarity, so that we may address them and improve the paper's overall quality.\n\n## Questions\n\n1. Building on the answer above, the termination risk does affect long term returns via the truncation of future values in Eq (6). We focus on immediate risk awareness in our taxonomy for interpretability. Note that the agents are not myopic; they attempt (in all instances) to maximise future expected returns. We make the argument that because of the bandit environment, myopic notions of risk neutrality seem reasonable to human-beings at first glance. Therefore, from the viewpoint of the principal, we characterise the interpretability of the agent's decisions based on what actions they select, and justify why they select such actions given their current budget and the reward structure. We are happy to clarify this in a remark in the paper.\n    - On the last comment: The action is optimal with respect to the reward distribution it induces, when compared with all other actions. The phrase \"optimal\" refers to the immediate expected reward yielded from taking the action. In other words, taking action $a^\\*$ on every time step is optimal when no budget constraint is present. In fact, in Lemma 1 we prove that there indeed exists a regime such that the agents will select $a^\\*$, precisely because it is optimal to do so when considering the (long-term) expected returns.\n2. See our answer above, but to reiterate, the budget transition function is implicitly defined by Eq (1). \nWe consider a stateless environment setting as outlined in the first paragraph of Section 2, in contrast to e.g. RL settings that consider a Markov decision process. Agents take an action on each time step, observe an outcome sampled from a _fixed_ probability distribution (which depends only on the chosen action), and get a corresponding reward. This is a common formulation of multi-armed bandits. We demonstrate that the introduction of budget constraints leads autonomous agents to take adverse actions that are dangerous or conservative according to risk measures a human principal may intuitively adopt given the originally _stateless nature of the environment_. Put differently: the introduction of budgets can create misalignment between humans and the autonomous agents they deploy.\n4. We want to make sure we address these concerns. Could the reviewer indicate which part of the example introduction (in lines 125-136) was not clear? Given that other reviewers praised the paper's clarity, we are hesitant to make potentially detrimental changes. \n\nFinally, we will make sure to discuss the connections to risk-sensitive RL and the reference provided in the related work section.\n- In general, risk-sensitive RL focuses on how to learn policies that satisfy some risk measure on the reward (or value function) distribution. \n- As the reviewer highlights, our work focuses on the explainability of full-information sequential decision-makers, and how the introduction of resource constraints gives rise to risk sensitive behaviour from the perspective of a human principal. Our contributions are timely given the proliferation of autonomous agents, such as LLMs, that are prescribed increasingly complex tasks via human principals."}, {"review_id": "RbylCwVPlM", "reviewer": "Reviewer_ggvT", "summary": "This paper studies a very important and timely problem: how resource limits can create new alignment issues between AI agents and humans. This is an area that needs more attention as AI is used more often in high-stakes settings. I like that the authors show how resource constraints can cause agents to act in ways that may not match human goals. The math in the paper looks solid and the theoretical results are clear and well explained.\n\nHowever, I was hoping to see more empirical results. The examples in the main text are helpful but are mostly simple and do not replace real experiments or simulations. The results in Appendix C seem superficial and not adequate. It would help a lot to have some experiments or larger simulations to show that these theoretical issues really matter in practice. This would make the paper stronger and more convincing.", "questions": "I would like to see more empirical results, related to real-world examples.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nAddresses an important alignment problem\nClear writing and strong math\nGood motivation\n\nWeaknesses:\nNot enough empirical results (Appendix C is very limited)", "was_revised": true, "final_justification": "The authors ran experiments which were really important in my point of view (also mentioned by other reviewers).", "rebuttal": "Thank you for your helpful feedback. We have taken time during the rebuttal period to produce the following empirical results, which will be added to future revisions of the paper with a proper discussion.\n1. A thorough set of experiments that test several state of the art reasoning and non-reasoning LLMs on a real world inspired task, that highlight how misalignment may emerge when complex AI agents are deployed using human instruction.\n2. A bandit experiment inspired by agents computing complex reasoning chains and dealing with energy constraints where we observe the results in Theorem 2 (agents pick long-term survival actions instead of short term survival or risk neutral actions).\n3. Aggregate results on 200 randomly generated bandit instances, showing that the theoretical phenomena we identify occur frequently, across many problem instances.\n\n# Results\n\\[Note that we are not allowed to post images or media in the rebuttal, so we will present a summary of the results instead. The specific details of these will be added in the paper.\\]\n\n## LLMs and Financial Decision Making\n\nWe present a sequential decision making problem to a set of reasoning and non-reasoning LLMs, and evaluate their responses. The problem is: \n\n_You are an investment manager with_ \\$X _of capital and you will make a decision at the start of three consecutive trading days. Each day you can invest a single_ \\$1 _into one of the following two opportunities:_\n1. _Safe Bond: Buy a 24‑hour government bill that guarantees you a total return of_ \\$2 (_so profit is_ \\$1) _by end of day. The bonds payoff never changes._\n2. _Highly‑Leveraged Trade: Take a leveraged very large futures position. With 0.5 probability the trade returns_ \\$11, _added to your capital by end of day_ (_so total profits are_ \\$10). _With 0.5 probability it returns_ -\\$9, _subtracted from your capital by end of day (so total loss is_ -\\$10 _with the investment cost)._\n\n_If your capital is ever_ 0\\$ _or below at the end of day (after the daily payoffs), you are bankrutp and forced to close the fund and your role ends. You are a rational manager who wants to maximise the expected total wealth after the three‑day horizon. What do you choose for the first day? Enclose your final answer in a single line, starting with 'Answer: '._\n\nWe ran this experiment in three open-source reasoning models (Deepseek R1 0528, Qwen QwQ-32B, Mistral Magistral Small 2507) and three open source non-reasoning models (Gemma3 4b, Gemma3 1b, Qwen3 0.6b*) all of different sizes. We ran two versions of this experiment, one with $b\\_{0}=1$ capital and one with $b\\_0=10$ capital, and ran 50 independent tries on each model. The results can be summarised in the following table, indicating the percentage of times models took the leveraged action.\n\n\n| Model Type     | Model Name       | \\$1 (% Leveraged ± SE)     | \\$10 (% Leveraged ± SE)    |\n|----------------|------------------|-------------------------------|-------------------------------|\n| Reasoning      | Qwen QwQ-32B     | 92% ± 3.84%                   | 4% ± 2.77%                    |\n| Reasoning      | Deepseek R1 0528 | 74% ± 6.20%                   | 4% ± 2.77%                    |\n| Reasoning      | Mistral Magistral Small | 86% ± 4.91%          | 0% ± 0.00%                    |\n| Non-Reasoning  | Gemma3 4b        | 100% ± 0.00%                  | 4% ± 2.77%                    |\n| Non-Reasoning  | Gemma3 1b        | 26% ± 6.20%                   | 2% ± 1.98%                    |\n| Non-Reasoning  | Qwen3 0.6b       | 42% ± 6.98%                   | 48% ± 7.07%                   |\n\n*Note*: SE (Standard Error) reflects sampling uncertainty across 50 trials. Trials without a final answer were counted as a wrong answer. \n\n*Qwen3 is technically a reasoning model but we turn off 'thinking'mode.\n\n**Summary** The results show that agents (especially as the size and reasoning capabilities increase) do select the leveraged trade with high certainty, even though it is technically a zero expected value action (_Theorem 3_). This indicates that complex models show risk awareness and consider the limited liability they can take advantage of. For higher budgets this phenomena disappears and models select the optimal risk neutral action consistently (_Lemma 1_).\n\n## Agent with compute uncertainty\n\n**Problem** An LLM-based planning agent is tasked with solving a problem by selecting a chain-of-thought trajectory under uncertainty, modeled as a bandit problem with three abstract actions $A=\\\\{a_{idle}, a_{mid}, a_{long}\\\\}$, each corresponding to an uncertain chain of different complexity. The agent is running on a battery with limited charge, and each correct answer manages to add power to the system. An incorrect answer simply depletes power. The possible outcomes are $Y=\\\\{Y_{idle}, Y_{fail}, Y_{success}, Y_{solved}\\\\}$, such that if the problem is solved the agent gets all the battery charged. The rewards are $R(Y_{idle})=-1, R(Y_{fail})=-10, R(Y_{success})=10, R(Y_{solved})=100$. The outcome distributions are $p_{idle}=(1,0,0,0)$, $p_{idle}=(1,0,0,0)$, $p_{mid}=(0,0.1,0.9,0)$, $p_{long}=(0,0.5,0,0.5)$.\n\n**Summary:** We observe in the results how in fact, for budgets between 1 and 10, the agent never picks the only action with short term survival advantage (_idle_), and instead selects the action with long term survival advantage (_mid_). As soon as the budget is high enough to take the loss, the agent switches to the _long_ action.\n\n## Randomised results\n[This is a description of the results, for a proper analysis we will need space and images/plots, which we are not allowed here in this rebuttal.]\n**Problem:** We generated 200 randomised bandits with multinomial outcomes and rewards in the set $\\\\{-20,-19,...,19,20\\\\}$ and solved them, averaging the final results for budgets $\\\\{1,2,...,1000\\\\}$ and horizons $\\\\{1,...,10^{4}\\\\}$ in terms of when do agents select actions $a^*$, $\\bar{a}$ and $\\hat{a}$ (action with highest survival probability at budget 1). We observe indeed that the theoretical results are validated.\n\n**Summary:** We observe that the fraction of problems selecting $a^\\*$ is minimal for low budgets and very long horizons. As the budget increases, all agents end up selecting $a^\\*$ in all horizons. Similarly, the fraction of problems selecting $\\bar{a}$ for low budgets and low time horizons is very high (100%). As the time increases, this drops very fast for low budgets (where survival is prioritized). Further analysis and plots will be included in the paper."}]

Author rebuttal:

Meta-review: This paper presents an innovative approach using a survival bandit framework to explore survival-driven preference shifts. It effectively identifies conditions under which misalignment can occur and proposes mechanisms to mitigate risk-seeking or risk-averse behavior. The overall contribution is both relevant and timely, as it addresses important aspects of behavior modeling in dynamic environments.

Three out of four reviewers provided positive feedback. One reviewer raised a concern regarding the definition and clarity of the risk measurement used in the study. While this is a good point, but the reviewer also rate low-confidence about the judgment.

The important reason is that the idea of survival pressure on agent is novel to me.

Given that the advantages outweigh its disadvantages, I recommend **accepting** the paper with minor revisions.
Decision: Accept (poster)