# Why modelling multi-objective homeostasis is essential for AI alignment (and how it helps with AI safety as well). Subtleties and open challenges.

I notice that there has been very little discussion on why and how considering **homeostasis** is significant, even **essential** for AI **alignment and safety**. The current post aims to contribute to  or assamending that situation. In this post I will treat alignment and safety as explicitly separate subjects, which both benefit from homeostatic approaches.

This text is a distillation and reorganisation of three of my older blog posts at Medium: 

*   [Making AI less dangerous: Using homeostasis-based goal structures](https://medium.com/threelaws/making-ai-less-dangerous-2742e29797bd) (2017)
*   [Project proposal: Corrigibility and interruptibility of homeostasis based agents](https://medium.com/threelaws/project-proposal-corrigibility-and-interruptibility-of-homeostasis-based-agents-e51bafbf7111) (2018)
*   [Diminishing returns and conjunctive goals: Mitigating Goodhart’s law with common sense. Towards corrigibility and interruptibility via the golden middle way.](https://medium.com/threelaws/diminishing-returns-and-conjunctive-goals-towards-corrigibility-and-interruptibility-2ec594fed75c) (2018)

I will probably share more such distillations or weaves of my old writings in the future.

* * *

**Introduction**
----------------

Much of AI safety discussion revolves around the potential dangers posed by goal-driven artificial agents. In many of these discussions, the agent is assumed to **maximise** some utility metric over an **unbounded** timeframe. This **simplification, while mathematically convenient**, can yield pathological outcomes. A classic example is the so-called “paperclip maximiser”, a “utility monster” which steamrolls over other objectives to pursue a single goal (e.g. creating as many paperclips as possible) indefinitely. “Specification gaming”, Goodhart’s law, and even “instrumental convergence” are also closely related phenomena.

However, in nature, organisms do not typically behave like pure maximisers. Instead, they operate under **homeostasis**: a principle of maintaining various internal and external variables (e.g. temperature, hunger, social interactions) within certain “good enough” ranges. Going far beyond those ranges — too hot, too hungry, too socially isolated — leads to dire consequences, so an organism continually balances multiple needs. Crucially, **“too much of a good thing” is just as dangerous as too little**. 

**Excess is harmful even for the very same objective that was maximised for**, not just as a side effect on other objectives. This seems to apply to most or even all biological objectives. Active avoidance of “too much”  is a **significantly stricter principle than** the more widely known partially overlapping idea of **“mild optimisation”**.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/eab07d0ead1934b8e2e225276d6955c64c1f3c8c9d2c751e.png)

utility function of a homeostatic objective

In this post, I argue that an **explicitly homeostatic, multi-objective** model is a more suitable paradigm for AI alignment. Moreover, correctly modelling homeostasis **increases AI safety**, because homeostatic goals are **bounded **— there is an optimal zone rather than an unbounded improvement path. This bounding lowers the stakes of each objective and **reduces the incentive for extreme** (and potentially destructive) behaviours.

* * *

**Why Naive Utility Maximisation Is Insufficient**
--------------------------------------------------

In the standard utility maximisation framework, the agent’s central goal is to **increase** some singular measure of “value” or “reward” over time. Because many real-world objectives are treated as if they were unbounded — for instance, economic growth can always be increased by some amount — maximisers tend to push that objective beyond any reasonable point. This dynamic is what yields the risk of “berserk” behaviour.

*   **Unboundedness**: If there is no built-in satiety or equilibrium, the AI keeps pushing the objective indefinitely, often at the expense of everything else.
*   **Single-Dimensional Focus**: When multiple objectives exist, **simple summation or linear weighting** often encourages the AI to optimise whichever objective is most “convenient” to push to extremes. Yet in reality, for example eating food twice the normal amount does not compensate for the lack of drink and drinking a lot does not compensate for neglecting to eat.

By contrast, **homeostasis** signals that there is an optimal zone — an “enough” amount — for each objective. Pursuing more than enough is wasteful and even harmful. **Representing such objectives as unbounded maximisation objectives would be grossly inaccurate.** In the example of eating and drinking — both activities need to be **balanced** with some time granularity.

To reiterate: People have noticed that **having multiple objectives increases the likelihood of "Goodharting".** But **this caveat applies trivially only when these objectives are unbounded and are aggregated linearly.**

* * *

**Homeostasis as a More Correct and Safer Goal Architecture**
-------------------------------------------------------------

It supports various alignment-related aspects:

### **1\. Multiple Conjunctive Objectives**

Real organisms typically have several needs (food, water, social connection, rest, etc.) that must **all** be satisfied to at least a “sufficient” level. In a homeostatic agent:

*   Each objective has a **target range** (or setpoint).
*   If one objective is wildly off-target, the agent diverts efforts to fix that dimension rather than endlessly over-optimising another.
*   The agent thus operates in a **“balanced” or “middle” zone** across **all** dimensions.

When an AI has multiple **conjunctive** goals, it cannot ignore most while excessively optimising just one. Instead, the synergy among objectives drives it toward a safer, middle-ground strategy. In effect, **massive efforts** in any single dimension become exponentially harder to justify because you are “pulling away” from the target ranges in others.

### **2\. Task-Based Agents or Taskishness — “Do the Deed and Cool Down”**

Homeostatic goals naturally lead to **bounded** or **“task-based”** behaviour rather than indefinite optimisation. If all current objectives are satisfied, a homeostatic agent is content to remain idle. It does not keep scanning the universe for hypothetical improvements once the setpoints are reached. This “settle to rest” feature significantly **limits the potential damage** the agent might cause. Even if such an agent accidentally does something very harmful on a personal level, it is less likely to affect entire nations.

For example, an AI tasked with making 100 paperclips in a homeostatic framework will strive to produce enough paperclips to satisfy the “paperclip count” objective. Once the target is met (often even just approximately) — and so long as other objectives are also in a comfortable range — it does not chase infinitely more micro-optimisations.

### **3\. Bounded Stakes: Reduced Incentive for Extremes**

Because each objective is bounded, the **stakes of discrepancy** in one dimension remain limited by the balance with other dimensions. The system cannot keep turning up the efforts in a single dimension without noticing detrimental side effects (like going off course in every other dimension). This effect **reduces the likelihood of extremist outcomes**.

### **4\. Natural Corrigibility and Interruptibility**

A hallmark of AI alignment is ensuring that we can **correct** or **shut down** an agent if it goes off track. Simple maximisers often resist shutdown because they interpret it as a threat to their singular goal. By contrast, a homeostatic system:

*   **Assumes by design that the targets/setpoints may change**. Therefore the agent does not need to plan indefinitely into the future, because long-term speculation about changing setpoints is often counterproductive.
*   **After **achieving equilibrium, it effectively **has no strong incentives to prevent new goals or new tasks from arising**. When a human changes the setpoints (the agent’s “needs”), the agent can start its **bounded** work to re-center on the new bounded targets.
*   Additionally and just as importantly, if **at least one goal in the framework of homeostatic objectives is “be agreeable” and “avoid harming the human operators”,** then that acts as a persistent partial “off switch” or at least as a curb on manipulative and other extreme behaviour. The agent will not want or even need to push negative externalities in the name of maximising any single dimension. At the same time, reasonable “civilised” resistance, like proposing a few honest counterarguments where appropriate, should be considered permitted and even useful.

* * *

**Diminishing Returns and the “Golden Middle Way”**
---------------------------------------------------

An important corollary of multi-objective homeostasis is **diminishing returns**. Once a target is nearly satisfied in one dimension, the agent’s “bang for the buck” in that dimension is no longer large; it becomes more “cost-effective” to address other objectives that are further from their optimum. Thus:

*   The agent can keep multiple needs near healthy levels hopefully most of the time.
*   Each incremental improvement in a well-satisfied need is overshadowed by bigger deficits in other needs.

This “golden middle way” extends even to **safety measures** themselves. An AI with an excessively strong safety constraint might be tempted to devote the entire universe’s resources to verifying it is 100% safe — unless that safety measure also has a built-in diminishing returns principle. By bounding safety checks themselves, we avoid another pathological scenario: the AI going to extremes to confirm it has *never ever* caused any harm, while ignoring the rest of its objectives. I hope the trade here is clear: nobody would build or give resources to an agent that is safe to such an extent, therefore it would be a losing strategy.

* * *

**Formalising Homeostatic Goals**
---------------------------------

In reinforcement learning or other goal frameworks, one can use a **loss or negative-utility** term for each objective. Each objective is measured against its target (also known as “setpoint”), and **both “too little” and “too much”** push the system away from equilibrium. Summing or combining these losses in a way that penalises **big deviations** in any single dimension more strongly than small deviations in many fosters a “balancing” behaviour. One simple formula might be:

$\text{Reward} = -\sum_i w_i \cdot ( \text{target}_i - \text{actual}_i )^2$

Here:

*   Each $\text{target}_i$ is a setpoint for the $i$-th objective (e.g. water needed, number of people not harmed, etc.). In case of the number of people not harmed, the setpoint is zero or a negative number.
*   If $\text{actual}_i​$ gets too high or too low, the negative squared term grows rapidly, which steers the agent back toward the setpoint.
*   A sufficiently “close” match to each target means there is little additional gain for over-optimising that dimension.
*   The weighting factors $w_i​$ reflect the relative importance (or priority level) of each objective.
*   In principle, there may be various power/exponential functions involved, not always a square function. But this is already an advanced discussion. This post here is a conversation starter.

Some objectives can also be framed as “safety **constraints**” about not disturbing the environment too much, and these can be folded into the same overall homeostatic reward system. These would be the **“low impact” and "minimal side effects"** objectives, sometimes  also **“keep future options open” and “maximise human autonomy”** objectives — **“negative goals”** by their nature, as these objectives are about **not** changing the existing state, as opposed to achieving some new state as is the case with **“positive goals”** which are usually the **“performance”** objectives.

* * *

**Parallels with Other Ideas in Computer Science**
--------------------------------------------------

As machine-learning-minded ones of you may notice, this formula above is very similar to how a **regression** algorithm works. Except in this case the squared errors are computed from plurality of objectives, not plurality of data points. In both cases the motivation for squaring is avoiding overfitting to a subset of data or objectives.

I propose an additional perspective: The distinction between **constraints versus objective functions in combinatorial optimisation** is somewhat analogous to the distinction between safety objectives and performance objectives. 

It is notable that in combinatorial optimisation problems the concept of constraints have been naturally considered as part of the setup, alongside the concept of objectives. In contrast, this unfortunately has often not been the case in the use of reinforcement learning. Yet in reality both performance and safety objectives should be present. 

The difference between "safety objectives" versus "constraints in combinatorial optimisation" can be that various safety objectives might considered as **“soft” constraints** — they can be traded off up to a point, but not too much. When they are violated notably, they become increasingly similar to hard constraints in their effects, thanks to squaring of the discrepancies.

Though in special use cases, some safety considerations can also be treated as **“hard” constraints** when comparing to performance objectives, while being still traded off or balanced among plurality of other safety objectives. (These are not just theoretical thoughts here — I have implemented such advanced homeostatic setups for example in multi-objective workforce planning algorithms that have been successfully time-tested for the past 15 years by now.)

Finally, the most obvious parallel — **control systems and control theory**. This framework treats objectives inherently as bounded and homeostatic. It supports changing setpoints, having multiple dimensions/objectives, and they can even be hierarchical. **Cybernetics** is a broader closely related field which is worth mentioning. It is curious that it has evolved largely separately from AI. Is there a reason we don’t talk more about cybernetic alignment instead of AI alignment?

* * *

**Open Challenges and Future Directions**
-----------------------------------------

While promising, a multi-objective homeostatic approach has many **subtleties**:

1.  **Time Granularity**
    *   In the example of eating and drinking — both activities need to be **balanced with some time granularity**. It would not be necessary nor productive to keep these objectives maximally balanced at all times. That would mean for example, that an agent runs between food and drink sources without stopping for more than a smallest timestep. Which would in the end result in collecting "equally" less than necessary on both objectives, since most of time would be spent on switching tasks. Deciding an optimal time granularity during which unequal treatment to competing objectives is allowed, requires a bit of **strategic thinking** on the agent’s part.
2.  **Handling Evolving Targets**
    *   Objectives (or their setpoints) may change over time, especially when humans are involved. Ideally, the agent should accept these new targets without undue manipulation such as **resisting the changes, escaping, or reverting the objectives. **Likewise, the agent **should not be inducing the changes**. In the current post I mentioned the general principles for solving this, but details still need to be worked out and tested in practice.
3.  **Soft vs. Hard Constraints**
    *   Not all objectives are created equal. Some constraints (e.g. “don’t kill humans”) might be hard constraints with extremely high weight or even lexicographic priority. But introducing **lexicographic ordering** may require some hard decisions. Additionally, one must be diligent and make sure that these higher-priority objectives could not cause extreme losses in unacceptable dimensions of low-priority objectives. Alternatively, one should ensure that the higher-priority objectives include all objectives where extreme losses are undesirable.
4.  **Goodhart’s Law and Multi-Scale Measures**
    *   We might measure safety and low impact at multiple “levels” (**individual, community, global**, etc.). Aggregating or normalising them incorrectly risks inadvertently re-enabling Goodhart-like exploits. As an example, there is a possible error where one large aggregated discrepancy (lets say, at gender level) is split up into multiple small discrepancies (at individual level) while being insufficiently represented at the higher level.
5.  **Coordination of Multiple Agents**
    *   If multiple homeostatic agents co-exist, how do they avoid interfering with each other’s setpoints? Most likely some **authorisation** and power hierarchy framework is relevant here. Such a system would then inevitably need to also include mechanisms for **accountability auditing** of the higher-ups who need to be held liable when a subordinate system executes unsafe actions down the line.
6.  **Tradeoffs Between “Interruptibility” and “Getting Work Done”**
    *   Ideally, we want the agent to do its job of achieving performance objectives but also remain open to external corrections or even a shutdown. Balancing these is partly social/political and partly technical. Yet there is a well known risk of systems becoming **too important to shut down**. Perhaps aligned systems could be **throttled gradually**, so a “shutdown” command becomes a “chill down a bit” command.
7.  **Integration of Bounded and Unbounded Objectives**
    *   Even after introduction of the concept of homeostatic objectives, sufficiently many performance objectives can still belong to the unbounded class. In contrast to **bounded objectives**, on which I focused in this post, **unbounded objectives** could reach infinite positive rewards. Yet these potential infinite rewards **should not dominate** safety objectives or even exclude balancing of other performance objectives. This topic is explored in a bit more detail in the addendum below.
8.  **Rate-Limited Objectives**
    *   Even though homeostatic objectives are bounded locally in time, in various cases they can be unbounded across time. For example, a human need for novelty may be satiated for a day, but it will arise again the next day. This dynamic can be described in other words as rate-limited objectives. Rate limiting is important also for **sustainability** purposes — to avoid exhausting the renewable resources in the environment. So homeostasis fulfills a sort of balancing role here as well. I imagine, the expansion of humanity could often be described from such a perspective too — humanity may have an intrinsic need to grow to live even outside of our planet Earth one day, but such growth needs to be sustainable.
9.  **The Necessity of Letting Go — When Unsustainable Objectives Must Be Abandoned**
    *   Even some **homeostatic** objectives might be **impossible to achieve without sacrificing everything else**. If that is the case, then some sort of **triage** has to take place. Otherwise such an objective would become sort of a **negative utility monster, demanding all attention because it forever stays furthest from being satisfied**. As an analogy, in multi-objective constraint optimisation problems, the unfortunate need for "triage" of objectives is well known, because some mutually exclusive action choices are binary — you cannot execute them halfway — "if you pursue two rabbits, you catch none".

These are open research challenges, but a homeostatic approach at least provides a conceptual **blueprint** of how to incorporate multi-dimensional, bounded goals into AI systems without inviting the pathological extremes of naive maximisation. This post is a conversation starter and there is so much to explore further.

* * *

**Addendum about Unbounded Objectives**
---------------------------------------

I fully acknowledge that various instrumental objectives could still have **unbounded** nature. For example, accumulating money, building guarantees against future risks, etc. (Though in case of risks, people familiar with the concept of antifragility may argue that reducing risk too much in the short term would paradoxically mean more fragility in the long run).

These unbounded objectives would still become safer when the above described homeostatic principles are considered in relevant objectives alongside.

Additionally, unbounded objectives are usually described by **concave utility functions in economics**. In other words, these objectives too have their own form of **diminishing marginal returns**. Besides **biology**, **economics** is the other **fundamental** and well-established field describing our needs. AI alignment surely should consult both. An aligned agent should be able to model diminishing returns and thus balancing between plurality of unbounded objectives just as well as it should be able to properly model homeostasis. As economists have observed, **humans prefer averages in all objectives to extremes in a few**.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b1ca808f551ba1280f1f008a1ff6d0b5af253ed9e31d8533.png)

concave utility function of an unbounded objective with diminishing returns

You can read more about **balancing multiple unbounded objectives via the mechanism of considering diminishing returns** from the following blog posts I have co-authored:

*   [A brief review of the reasons multi-objective RL could be important in AI Safety Research](https://www.lesswrong.com/posts/i5dLfi6m6FCexReK9/a-brief-review-of-the-reasons-multi-objective-rl-could-be) (2021)
*   [Sets of objectives for a multi-objective RL agent to optimize](https://www.lesswrong.com/posts/4mvdZXjwJHv9tSAWB/sets-of-objectives-for-a-multi-objective-rl-agent-to-1) (2022)

* * *

**Conclusion**
--------------

**Homeostasis **— the idea of multiple objectives each with a bounded “sweet spot” — offers a **more natural and safer** alternative to unbounded utility maximisation. By ensuring that an AI’s needs or goals are **multi-objective** and **conjunctive**, and that each is **bounded**, we significantly reduce the incentives for runaway or berserk behaviours.

Such an agent tries to stay in a “golden middle way”, **switching** focus among its objectives according to whichever is most pressing. It avoids extremes in any single dimension because going too far throws off the equilibrium in the others. This balancing act also makes it more **task based and low impact**, more **corrigible** and **interruptible — **ultimately **safer**.

In order **to avoid confusion**, pointing out that there are **two types of balancing** involved here:

1.  **Balancing of a single homeostatic objective** \- keeping the actual value of a single homeostatic objective near the target value - **not too low, not too high. This constraint is stronger than just diminishing returns** would be**.**
2.  Balancing across objectives - as a form of **considering the utilities of multiple objectives** roughly equally. That means meeting them in such a manner that the homeostatic objectives have for example **least-squares deviations**, while unbounded objectives have approximately the same **utility value** after the **utility functions** with **diminishing returns have been applied** to each actual value.

In short, **modelling multi-objective homeostasis** is a step toward creating AI systems that exhibit the sane, moderate behaviours of living organisms — an important element in ensuring alignment with human values. While no single design framework can solve all challenges of AI safety, shifting from “maximise forever” to “maintain a healthy equilibrium” is a crucial part of the solution space.

* * *

**Related work**
----------------

**NeuroAI for AI Safety**  
*Patrick Mineault, Niccolò Zanichelli, Joanne Zichen Peng, Anton Arkhipov, Eli Bingham, Julian Jara-Ettinger, Emily Mackevicius, Adam Marblestone, Marcelo Mattar, Andrew Payne, Sophia Sanborn, Karen Schroeder, Zenna Tavares, Andreas Tolias, Anthony Zador*  
As AI systems become increasingly powerful, the need for safe AI has become more pressing. Humans are an attractive model for AI safety: as the only known agents capable of general intelligence, they perform robustly even under conditions that deviate significantly from prior experiences, explore the world safely, understand pragmatics, and can cooperate to meet their intrinsic goals. Intelligence, when coupled with cooperation and safety mechanisms, can drive sustained progress and well-being. These properties are a function of the architecture of the brain and the learning algorithms it implements. Neuroscience may thus hold important keys to technical AI safety that are currently underexplored and underutilized. In this roadmap, we highlight and critically evaluate several paths toward AI safety inspired by neuroscience: emulating the brain's representations, information processing, and architecture; building robust sensory and motor systems from imitating brain data and bodies; fine-tuning AI systems on brain data; advancing interpretability using neuroscience methods; and scaling up cognitively-inspired architectures. We make several concrete recommendations for how neuroscience can positively impact AI safety.   
[https://arxiv.org/abs/2411.18526](https://arxiv.org/abs/2411.18526) 

* * *

**My Related work**
-------------------

1.  [**Research agenda for training aligned AIs using concave utility functions following the principles of homeostasis and diminishing returns**](https://www.lesswrong.com/posts/9hWgJQK8wnpuFtD5Z/research-agenda-for-training-aligned-ais-using-concave)  
    *Roland Pihlakas*  
    This conceptual overview post is intended to explain what I mean by the principles of "homeostasis", "diminishing returns", and "balancing" - how these ideas differ, complement, and interact with each other. Alongside, there is also an overview of my research agenda.  
    What am I trying to promote, in simple words: I want to build and promote AI systems that are trained to understand and follow two fundamental principles from biology and economics:  
    Moderation - Enables the agents to understand the concept of “enough” versus “too much”. The agents would understand that too much of a good thing would be harmful even for the very objective that was maximised for, and they would actively avoid such situations. This is based on the biological principle of homeostasis and addresses bounded ultimate objectives.  
    Balancing - Enables the agents to keep many important objectives in balance, in such a manner that having average results in all objectives is preferred to extremes in a few. This addresses mainly the economic principle of diminishing returns in unbounded instrumental objectives, but also applies to homeostasis.  
    [https://www.lesswrong.com/posts/9hWgJQK8wnpuFtD5Z/research-agenda-for-training-aligned-ais-using-concave](https://www.lesswrong.com/posts/9hWgJQK8wnpuFtD5Z/research-agenda-for-training-aligned-ais-using-concave) 
2.  [**From homeostasis to resource sharing: Biologically and economically aligned multi-objective multi-agent gridworld-based AI safety benchmarks**](https://arxiv.org/abs/2410.00081)  
    *Roland Pihlakas*  
    Developing safe, aligned agentic AI systems requires comprehensive empirical testing, yet many existing benchmarks neglect crucial themes aligned with biology and economics, both time-tested fundamental sciences describing our needs and preferences. To address this gap, the present work focuses on introducing biologically and economically motivated themes that have been neglected in current mainstream discussions on AI safety - namely a set of multi-objective, multi-agent alignment benchmarks that emphasize homeostasis for bounded and biological objectives, diminishing returns for unbounded, instrumental, and business objectives, sustainability principle, and resource sharing. Eight main benchmark environments have been implemented on the above themes, to illustrate key pitfalls and challenges in agentic AI-s, such as unboundedly maximizing a homeostatic objective, over-optimizing one objective at the expense of others, neglecting safety constraints, or depleting shared resources.  
    [https://arxiv.org/abs/2410.00081](https://arxiv.org/abs/2410.00081) 
3.  [**Systematic runaway-optimiser-like LLM failure modes on Biologically and Economically aligned AI safety benchmarks for LLMs with simplified observation format (BioBlue)**](https://www.lesswrong.com/posts/PejNckwQj3A2MGhMA/systematic-runaway-optimiser-like-llm-failure-modes-on)  
    *Roland Pihlakas, Sruthi Susan Kuriakose, Shruti Datta Gupta*  
    Relatively many past AI safety discussions have centered around the dangers of unbounded utility maximisation by RL agents, illustrated by scenarios like the "paperclip maximiser" or by specification gaming in general. Unbounded maximisation is problematic for many reasons. We wanted to verify whether these RL runaway optimisation problems are still relevant with LLMs as well.  
    Turns out, strangely, this is indeed clearly the case. The problem is not that the LLMs just lose context or become incoherent. The problem is that in various scenarios, LLMs lose context in very specific ways, which systematically resemble runaway optimisers in the following distinct ways: 1) Ignoring homeostatic targets and “defaulting” to unbounded maximisation instead. 2) It is equally concerning that the “default” meant also reverting back to single-objective optimisation.  
    Our findings also suggest that long-running scenarios are important. Systematic failures emerge after periods of initially successful behaviour. In some trials the LLMs were successful until the end. This means, while current LLMs do conceptually grasp biological and economic alignment, they exhibit randomly triggered problematic behavioural tendencies under sustained long-running conditions, particularly involving multiple or competing objectives. Once they flip, they usually do not recover.  
    Even though LLMs look multi-objective and bounded on the surface, the underlying mechanisms seem to be actually still biased towards being single-objective and unbounded. This should not be happening!  
    [https://www.lesswrong.com/posts/PejNckwQj3A2MGhMA/systematic-runaway-optimiser-like-llm-failure-modes-on](https://www.lesswrong.com/posts/PejNckwQj3A2MGhMA/systematic-runaway-optimiser-like-llm-failure-modes-on) 
4.  [**BioBlue: Systematic runaway-optimiser-like LLM failure modes on biologically and economically aligned AI safety benchmarks for LLMs with simplified observation format**](https://arxiv.org/abs/2509.02655)  
    *Roland Pihlakas, Sruthi Susan Kuriakose*  
    Many AI alignment discussions of "runaway optimisation" focus on RL agents: unbounded utility maximisers that over-optimise a proxy objective (e.g., "paperclip maximiser", specification gaming) at the expense of everything else. LLM-based systems are often assumed to be safer because they function as next-token predictors rather than persistent optimisers. In this work, we empirically test this assumption by placing LLMs in simple, long-horizon control-style environments that require maintaining state of or balancing objectives over time: sustainability of a renewable resource, single- and multi-objective homeostasis, and balancing unbounded objectives with diminishing returns. We find that, although models frequently behave appropriately for many steps and clearly understand the stated objectives, they often lose context in structured ways and drift into runaway behaviours: ignoring homeostatic targets, collapsing from multi-objective trade-offs into single-objective maximisation - thus failing to respect concave utility structures. These failures emerge reliably after initial periods of competent behaviour and exhibit characteristic patterns (including self-imitative oscillations, unbounded maximisation, and reverting to single-objective optimisation). The problem is not that the LLMs just lose context or become incoherent - the failures systematically resemble runaway optimisers. Our results suggest that long-horizon, multi-objective misalignment is a genuine and under-evaluated failure mode in LLM agents, even in extremely simple settings with transparent and explicitly multi-objective feedback. Although LLMs appear multi-objective and bounded on the surface, their behaviour under sustained interaction, particularly involving multiple objectives, resembles brittle, poorly aligned optimisers whose effective objective gradually shifts toward unbounded and single-metric maximisation.  
    [https://arxiv.org/abs/2509.02655](https://arxiv.org/abs/2509.02655) 

* * *

***Thanks for reading! If you have thoughts, questions, improvement suggestions, resource and collaborator references, feedback, ideas, or alternative formulations for multi-objective homeostasis, please share in the comments. You can contact me at email address*** [***roland@simplify.ee***](mailto:roland@simplify.ee)