Title: Emergent Risk Awareness in Rational Agents under Resource Constraints
Venue: NeurIPS 2025
Accepted: yes
Keywords: Decision Making

Alignment

Interpretability

Planning
Abstract: Advanced reasoning models with agentic capabilities (AI agents) are deployed to interact with humans and to solve sequential decision‑making problems under (often approximate) utility functions and internal models. When such problems have resource or failure constraints where action sequences may be forcibly terminated once resources are exhausted, agents face implicit trade‑offs that reshape their utility-driven (rational) behaviour. Additionally, since these agents are typically commissioned by a human principal to act on their behalf, asymmetries in constraint exposure can give rise to previously unanticipated misalignment between human objectives and agent incentives. We formalise this setting through a survival bandit framework, provide theoretical and empirical results that quantify the impact of survival‑driven preference shifts, identify conditions under which misalignment emerges and propose mechanisms to mitigate the emergence of risk-seeking or risk-averse behaviours. As a result, this work aims to increase understanding and interpretability of emergent behaviours of AI agents operating under such survival pressure, and offer guidelines for safely deploying such AI systems in critical resource‑limited environments.


Peer reviews:
comments: SUMMARY:
This paper studies a very important and timely problem: how resource limits can create new alignment issues between AI agents and humans. This is an area that needs more attention as AI is used more often in high-stakes settings. I like that the authors show how resource constraints can cause agents to act in ways that may not match human goals. The math in the paper looks solid and the theoretical results are clear and well explained.

However, I was hoping to see more empirical results. The examples in the main text are helpful but are mostly simple and do not replace real experiments or simulations. The results in Appendix C seem superficial and not adequate. It would help a lot to have some experiments or larger simulations to show that these theoretical issues really matter in practice. This would make the paper stronger and more convincing.

STRENGTHS AND WEAKNESSES:
Strengths:
Addresses an important alignment problem
Clear writing and strong math
Good motivation

Weaknesses:
Not enough empirical results (Appendix C is very limited)

QUESTIONS:
I would like to see more empirical results, related to real-world examples.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the emergence of risk-sensitive behavior from the perspective of survival. The authors demonstrate that, in a sequential decision-making setting, an agent optimizing long-term survival return can behave in ways that resemble risk-neutral, risk-averse, or risk-seeking preferences.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper explores an interesting and important question: how risk-sensitive behaviors can emerge from survival-driven decision-making in sequential environments.\n- The motivation is novel and conceptually compelling, with potential implications for understanding adaptive behavior in uncertain settings.\n\nWeaknesses:\n- Quality: The theoretical framework has conceptual flaws. Risk sensitivity is defined in a way that focuses on immediate rewards rather than long-term return, which undermines the soundness of the analysis.\n- Clarity: The writing and notation are often hard to follow. Inconsistent formatting (e.g., italicization) and unclear variable subscripts (e.g., Y_{vd}, Y_d, Y_n, Y_s, Y_a) reduce readability.\n- Significance: The omission of the transition model is a serious limitation. For sequential decision-making, this component is essential for a full characterization of risk-related behavior.\n- Originality: While the framing is creative, the lack of conceptual clarity diminishes its impact.\n\nQUESTIONS:\n1. Definition of Risk Preferences: Why are the definitions of risk-neutral and risk-seeking preferences based solely on current rewards? Would a formulation based on long-term return (as in Eq. 4) not be more consistent with standard practice?  How do the authors distinguish between myopic behavior and genuine risk sensitivity? Referencing Gagne & Dayan (2021), the conflation of these concepts may weaken the interpretability of results. Line 235 even mentioned \"optimal risk neutral action\", but, given that it's purely myopic, such action can't be optimal.\n\n2. Transition Dynamics: Why is the state transition model not included in the analysis? Since this is a sequential setting, transition probabilities are essential for evaluating risk and survival.\n\n3. Notation and Formatting: Could the authors clarify the notation and introduce the example more naturally? Also, a consistent use of italic formatting would improve the paper’s readability.\n\nGagne, C., & Dayan, P. (2021). Two steps to risk sensitivity. Advances in Neural Information Processing Systems, 34, 22209-22220.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 1, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how rational agents behave under resource constraints that may terminate their operation. Even when agents optimize for expected rewards, survival constraints induce risk-aware behaviors—risk-averse or risk-seeking—depending on budget and planning horizon. The authors provide formal conditions for each behavior type and show how this may misalign agent actions with human (principal) intentions. They analyze two main sources of misalignment: liability asymmetry and differing planning horizons. The findings offer insights into designing safer AI systems through behavior-aware planning and mitigation strategies.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n  - The paper provides a rigorous formalization of how resource constraints (e.g., budget limits) induce different agent behaviors—risk-neutral, risk-seeking, and risk-averse—using a survival-aware reward structure.\n  - Theoretical contributions are grounded in formally proven theorems and lemmas, articulated with explicit assumptions and well-interpreted consequences, enabling clear insights into the agent’s decision logic under constrained optimization.\n  - The paper pinpoints two critical sources of agent-principal misalignment—liability asymmetries and mismatched planning horizons—and explains how these structural differences can give rise to incentive divergences in real-world AI systems.\n  - The theoretical insights are complemented by illustrative examples (e.g., AI assistant scenarios), making the conceptual findings more accessible.\n  - By contextualizing the work within diverse research areas, including AI safety, economic agency, and safe reinforcement learning, the paper demonstrates interdisciplinary awareness and positions its contributions within broader AI alignment discourse.\n\n- Weaknesses\n  - While the theoretical results are elegant, empirical validation is limited to simple illustrative examples; there is no thorough simulation or application to complex domains, which may weaken practical relevance.\n  - As noted in the limitations, the model assumes perfect agent rationality and complete knowledge of environment dynamics (e.g., transition probabilities and reward functions), which may limit its applicability to real-world agents operating under uncertainty or with bounded rationality.\n\nQUESTIONS:\nThe paper assumes discrete state and action spaces and presumes that the environment’s dynamics—such as transition probabilities and reward functions—are fully known. This simplification may be excessive. To better assess the significance of this research, could the authors explore real-world use cases more deeply and the feasibility of implementation?\n\nLIMITATIONS:\nThe scenarios addressed in this paper are highly relevant to human-AI interactions, but the discussion on the social and ethical implications of potential misalignments in such contexts is limited.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn many decision problems decisions are made under some constraints. However, AI and human agents may inherently be subject to different constraints. For example, upon breaking a constraint an AI may be removed from controlling a problem, insulating it from the negative effects of breaking the constraint, while the human will need to take over and continue to be affected by them.\n\nThis paper formalizes this problem by considering decision problems which stop when the agent’s cumulative reward for the episode so far drops below 0 (the constraint). Agent reward are however capped (below) to remain positive, creating a sort of limited liability. The authors then establish conditions under which the agent becomes risk averse (choosing actions which maximize survival probability) or risk seeking (accepting low survival probability for a potentially larger payoff), due to this constraint.\n\nIntuitively, the paper establishes that their limited liability causes agents to become risk-averse when a long time horizon (but low budget) still allows them to slowly collect reward with safe actions and risk-seeking when the value of surviving is outweighed by the potential upside of risky actions whose downsides are mitigated by the limited liability.\n\nThis can then cause mis-alignment in the optimal level of risk (and thus the optimal policy) between the agent and a human. This can happen when (1) the limited liability applies to the agent but not the human or (2) when the agent has. different time horizon then the principal. This is because both these factors determine what level of risk is currently optimal.\n\nIn addition to the theoretical results, these effects are shown to exist in some empirical experiments.\n\nSTRENGTHS AND WEAKNESSES:\nIn general I find this to be a very strong paper. The problem of misalignment due to a difference in risk-seeking behaviors induced by different resource constraints is an interesting and important problem which I had not encountered before. The formulation of this problem is not overly simplistic and the theoretical analysis provides interesting insights. The paper is generally well written: the text and proofs are easy to follow and the related work section is comprehensive.\n\nThe setting considered is of course severely restricted but this is a weakness the authors acknowledge in the discussion and in my opinion is outweighed by the completeness of the theoretical results. Although there are some empirical results, they are relegated to the appendices and are very limited in scope. This is one area where I think the paper could have been stronger.\n\nQUESTIONS:\nRegarding the budget dynamics in eq. (1), what is the starting budget $b_0$? I don’t see this given anywhere, and without it the dynamics are incomplete. As far as I can tell the results in the paper will hold for any choice of initial budget $b_0$. Is this correct?\n\nThe notion of a budget with dynamics defined in (1) seems superfluous. Can’t we equivalently say that the agent’s survival constraint is that it stops once its cumulative reward up to the current time step falls below 0?\n\nOn line 263 you use $\\bar{v}^*_{t+1}$. This has not been introduced before. Was this supposed to be $\\bar{v}_{t+1}$?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}