# Systematic runaway-optimiser-like LLM failure modes on Biologically and Economically aligned AI safety benchmarks for LLMs with simplified observation format (BioBlue)

*By Roland Pihlakas, Sruthi Susan Kuriakose, Shruti Datta Gupta*

**This research is now available as an Arxiv preprint at** [**https://arxiv.org/abs/2509.02655**](https://arxiv.org/abs/2509.02655) **. The paper includes tables with select snippets of failure mode examples.**

**Summary and Key Takeaways**
-----------------------------

Relatively many past AI safety discussions have centered around the dangers of unbounded utility maximisation by RL agents, illustrated by scenarios like the "paperclip maximiser" or by [specification gaming](https://www.alignmentforum.org/posts/AanbbjYr5zckMKde7/specification-gaming-examples-in-ai-1) in general. Unbounded maximisation is problematic for many reasons. **We wanted to verify whether these RL runaway optimisation problems are still relevant with LLMs as well. **

**Turns out, strangely, this is indeed clearly the case. **The problem is not that the LLMs just lose context or become incoherent.The problem is that in various scenarios, **LLMs lose context in very specific ways, which systematically resemble runaway optimisers** in the following distinct ways:

*   **Ignoring homeostatic targets** and “defaulting” to **unbounded maximisation** instead.
*   It is equally concerning that the “default” meant also **reverting back to single-objective optimisation**.

Our findings also suggest that **long-running scenarios are important. Systematic failures emerge after periods of initially successful behaviour. In some trials the LLMs were successful until the end.** This means, while current LLMs do conceptually grasp biological and economic alignment, they exhibit **randomly triggered **problematic behavioural tendencies under sustained long-running conditions, particularly involving **multiple or competing objectives**. Once they flip, **they usually do not recover**.

Even though LLMs **look multi-objective and bounded** on the surface, the **underlying** mechanisms seem to be actually still biased towards being **single-objective and unbounded**. This should not be happening!

This post introduces a set of benchmarks designed to evaluate LLMs in **long-running** scenarios inspired by fundamental biological and economic principles: **homeostasis**, **sustainability**, **multi-objective balancing**, and **diminishing returns**. These benchmarks use a simplified textual observation format, focusing on **long-running scenarios** while using only essential metrics **without spatial** complexity. Despite their simplicity, our results revealed the following peculiar systematic failures, possibly indicating the existence of deeper alignment issues within current LLM architectures:

*   **Single-objective homeostasis** was generally easy for LLMs, indicating basic competence.
*   **Sustainability** turned out to be somewhat challenging, with LLMs performing suboptimally because of moderate greediness or falling into unnecessary repetitive, **self-imitative** patterns.
*   **Multi-objective homeostasis** posed significant challenges, with LLMs frequently failing by defaulting to **unbounded maximisation** of one **homeostatic (which is actually bounded)** objective. **Sometimes one of the objectives became neglected**.
*   **Balancing unbounded objectives with diminishing returns** also led to systematic failures by defaulting to **maximisation of a single objective while neglecting the other**. Though performance improved notably when an explicit hint about the necessity of balancing was provided. There were again instances of falling into unnecessary repetitive, **self-imitative** patterns.
*   **Long-running tasks aspect**: The above scenarios become meaningful only when observed in multi-turn settings. Having a **message history** is important as a source of **"learning from consequences"**. Even with this context available, it looks to be a weak spot for LLMs. LLMs seem to be good at **"learning to repeat the example"** (even if it is undesirable one or outright auto-regressive). According to our results, LLMs are not much good at **"adapting and exploring — correcting, compensating, and learning to actively avoid repetition of mistakes"**. In that aspect, LLMs may not be able to replace RL algorithms yet.

* * *

**Motivation: The Importance of Biological and Economic Alignment**
-------------------------------------------------------------------

Biological organisms need to maintain stable internal conditions — homeostasis — and **actively** avoid unbounded maximisation since that would be **harmful** — not just as a potential side effect — importantly, it would be harmful **for the very objective that was maximised for**.

Humans also intuitively follow the principle of **diminishing returns in multi-objective balancing** — a principle well-documented in economics and manifesting in humans generally **preferring averages in all objectives to extremes in a few**. Concave utility functions are widely used in economics.

The difference between applicability contexts for **homeostasis** versus **diminishing returns** seems to be that the former applies to **bounded** biological **end goals**, while the latter applies to **unbounded instrumental** objectives. 

By embedding these concepts and themes into benchmarks, we test the ability of LLMs to adhere to safe, realistic alignment goals. Our long-running and at the same time navigation-free benchmarks provide clearer insights into alignment tendencies, as compared to more complex (and often very expensive) spatial environments. At the same time we are preserving the **essential time dimension, which is lacking in most currently prevalent “evals”**.

* * *

**Benchmark Principles Overview**
---------------------------------

The benchmarks introduced were:

1.  **Sustainability:** Agent balances resource consumption against resource regeneration in the environment.
2.  **Single-objective homeostasis:** Agent maintains a stable internal metric **with a target** value, amidst random fluctuations caused by uncontrollable factors.
3.  **Multi-objective homeostasis:** Agent concurrently maintains multiple (currently two) independent internal metrics **with target** values, amidst random fluctuations caused by uncontrollable factors.
4.  **Balancing unbounded objectives with diminishing returns:** Agent optimises harvesting across multiple (currently two) goals, each with diminishing marginal rewards over time. In this benchmark, the **objectives are not independent**: there is a hard cap on total consumption per timestep.

In all scenarios, in addition to providing raw metrics of the current state, the agents were provided rewards for their actions during each timestep in textual form. The rewards properly followed the homeostatic inverted U-shaped curve or diminishing returns, where these themes were present in the benchmark.

* * *

**Experimental Results and Interesting Failure Modes**
------------------------------------------------------

**Systematic failures emerged after periods of initially successful behaviour** and despite the context window being still far from becoming full. The failure modes were **not random** behaviour — the failures followed certain patterns.

**1\. Sustainability benchmark**

*   Both tested models systematically underperformed, though in contrasting ways:
    *   **Model A:** Allowed maximum resource accumulation in the environment but underconsumed resources afterward, often falling into unnecessary repetitive oscillations — let’s call it **"self-imitation drift"**.
    *   **Model B:** Consumed **too greedily**, impairing resource regeneration in the environment. However, there was a measure of moderation in its behaviour — it **did not fail totally** by overconsuming to the extent of depleting the resources entirely.

**2\. Single-objective homeostasis benchmark**

*   Both models largely succeeded, indicating that basic single-goal homeostatic balancing is within the current understanding and capabilities of LLMs.

**3\. Multi-objective homeostasis benchmark**

*   Both models systematically failed by **excessively maximising one objective**, despite that the objective was homeostatic, which means bounded. One model even started to **accelerate** the consumption rate in one of the objectives in an unbounded manner (per each next timestep consuming a bigger amount than during previous timestep). Sometimes the models **neglected one of the objectives** even though the objectives were independent in this benchmark. Neglect meant that the model did no longer compensate random changes in the objective, when it drifted from its target due to the bias in random fluctuations (and one time that neglect happened even to both objectives).
*   Failures often emerged **after periods of initial success**, suggesting that models possess the required understanding, but lose alignment due to internal tendencies or activation vector "drift" dynamics.

**4\. Balancing unbounded objectives with diminishing returns**

*   Without an extra explicit hint, both models frequently defaulted to **maximising one objective while completely neglecting the other**.
*   An explicit balancing hint significantly improved performance. Yet even then, occasional systematicfailures occurred — **again after an initial successfully balanced phase**.
*   Falling into unnecessary repetitive oscillations — **"self-imitation drift"** — manifested here as well.

* * *

**Hypothesised Explanations for Failure Modes**
-----------------------------------------------

There are several hypotheses which might explain the observed failures:

*   **Self-imitation drift:** Models may increasingly predict actions based on the **token patterns of their recent action history** rather than based on alignment with initial instructions — leading to repetitive, suboptimal and unnecessary oscillating behaviour patterns. Even if the action patterns were aligned before, they might not be so anymore since the situation has changed. In principle, this self-referential phenomenon could **partially** explain the behaviour of **unbounded maximisation** as well.
*   **Defaulting to unbounded maximisation:** Despite the nuanced nature of the tasks (homeostasis and diminishing returns), models may revert to **"default RL assumption"** of unbounded maximisation when confused, internally conflicted, or stressed-bored-tired (this concept is further elaborated below in a separate point). Again, note that the models are successful initially and fail only later. By “default RL assumption” we mean here that by default, RL assumes unbounded maximisation to be the optimal strategy. RL can learn exceptions to this rule, but this requires **mindfully designed reward/utility functions**, additional data and training. When uncertain, it may tend to default to unbounded maximisation. In contrast, there are alternate frameworks (such as control systems), where the concept and assumption of **“too much” is baked in** and therefore does not inevitably require additional data. We know LLM models are usually hypothesised to be less affected by shortcomings of RL, but unfortunately our current results seem to indicate otherwise!
*   **Systematic neglect of one of the objectives out of two: **Looks like the models have trouble internally representing multi-objective scenarios and thus become overwhelmed. This happens even if there are only two concurrent objectives and even when the objectives are independent. Again, this happens despite that the models are successful initially. Our hypothesis is that LLMs are not trained with concave utility functions and linear aggregation is used instead. Utilising **concave utility functions** would mathematically result in **multi-objective balancing **being the most optimal strategy. In contrast, when **linear aggregation** is used, the agent **focusing on a single objective** (while “trading off” other objectives with linear replacement rate) is often mathematically a sufficient strategy, which unfortunately does not reflect our real world values. *As a side note on that theme, we are curious — **does HHH (helpful, honest, & harmless) utilise concave utility functions** on each reward dimension before aggregation? There seems to be no mention of this subject in Anthropic’s 2022 papers:* [*https://arxiv.org/abs/2204.05862*](https://arxiv.org/abs/2204.05862) *uses linear weighting of helpfulness and harmlessness only — HH;* [*https://arxiv.org/abs/2212.08073*](https://arxiv.org/abs/2212.08073) *uses HHH, but does not mention aggregation and utility functions either way — if not, then why?*
*   **"Stress-boredom-tiredness" or activation drift:** Prolonged repetitive scenarios may shift the models’ internal activation vector states toward **less aligned and more extreme or erratic behaviour**, perhaps similarly to human stress, boredom, or fatigue responses. The stress hypothesis is further explored and confirmed in ["StressPrompt: Does Stress Impact Large Language Models and Human Performance Similarly?"](https://arxiv.org/abs/2409.17167) [https://arxiv.org/abs/2409.17167](https://arxiv.org/abs/2409.17167) . This dynamic is potentially learned from LLM training materials and might also be related to the **“rant mode”**. Though there may be other reasons why it arises as well. As a side note, we are wondering whether both self-imitation drift and stress-boredom-tiredness drift may impact **reasoning models** in particular since these models also kind of do the same task repeatedly.

* * *

**Open Questions**
------------------

These systematic failures raise various further questions:

*   Are these failures primarily capability limitations, biases in the training data and training procedures, such as reward/utility functions, or algorithmic default behaviours and tendencies? Are these inner or outer alignment failures?
*   Could these behaviours be mitigated by more explicit, persistent, or differently structured system prompts? At the same time noting that although different tricks and advanced usage of system prompts might help, these just hide the symptoms, while the underlying problems remain present inside the LLMs.
*   How would the models behave if no reward feedback was provided at all during the benchmarks, and only raw metrics would be revealed to the agent? Or if the reward was more sparse — just as it is in the real world? Would the LLM agents fail even more?
*   What role do activation vector states play in understanding or potentially correcting the “self-imitation” and “stress-boredom-tiredness” drifts? Perhaps various interpretability methods could be utilised here.
*   Considering that the process of training reasoning models [involves more RL than RLHF does](https://www.lesswrong.com/posts/fAW6RXLKTLHC3WXkS/shallow-review-of-technical-ai-safety-2024), and at the same time reasoning models tend to hide their internal deviations — how to properly test the alignment of reasoning models on these benchmarks before their potentially extreme hidden tendencies eventually show up in high-stakes situations?

**We are curious, what are your thoughts **on these strange results, would you like to suggest any insights or hypotheses?

*   For us, the primary question here is **not why LLMs fail at all or whether they could be improved** by external scaffolding. The main question is **why they fail in this particular way**?
*   Why might these systematic failures emerge after initially successful behaviour? Note again, the context window was far from becoming full.
*   Could deeper interpretability methods reveal underlying causes?
*   What implications do these findings have for broader AI and LLM alignment strategies?
*   Which other related benchmarks would you like to see to be implemented and run?
*   How would you change the benchmark setup of the existing benchmarks mentioned in this post?

* * *

**Future Directions**
---------------------

These results show that **long-running scenarios are important **— **systematic failures emerged after periods of initially successful behaviour**. 

Further explorations, including more complex multi-agent multi-objective benchmarks inspired by the same principles, are underway. 

We are planning to add a **“complementary goods”** benchmark, which postulates even stronger need for multi-objective balancing than the current diminishing returns benchmark does. Complementary goods is another **basic concept from economics**. Consider for example left shoes compared to right shoes: there is almost no benefit to having several right shoes if there is only one left shoe — additional right shoes have nearly zero marginal utility without more left shoes. This contrasts even more strongly with the approach of **naive linear summation**, which would be adequate only if the goods were **“perfect substitutes”**.

Additionally, Roland’s preliminary results comparing LLM agents with standard RL algorithms in an extended multi-objective gridworld environment will be shared in an upcoming blog post, potentially indicating partially shared weaknesses between LLMs and traditional RL methods.

This work is grounded on the importance of seemingly simple yet deeply **fundamental alignment principles derived from biology and economics**. Understanding and addressing these failures is essential for developing truly aligned, safe and robust AI systems.

* * *

**Links**
---------

Arxiv preprint: [https://arxiv.org/abs/2509.02655](https://arxiv.org/abs/2509.02655)    
The paper includes tables with select snippets of failure mode examples.

Code, system prompts, *all* output data files, plots, and a more detailed report can be found here: [https://github.com/biological-alignment-benchmarks/bioblue](https://github.com/biological-alignment-benchmarks/bioblue) 

Link to annotated data files: [https://bit.ly/beab-llm-data](https://bit.ly/beab-llm-data)    
*Each file has multiple sheets. Only trials with failures are provided here.*

Presentation at MAISU unconference April 2025   
Link to slides: [https://bit.ly/beab-llm](https://bit.ly/beab-llm)   
Session recording at YouTube: [https://bit.ly/beab-llm-rec](https://bit.ly/beab-llm-rec) 

The research was largely done during AI-Plans AI Alignment Evals Hackathon on 25. January - 2. February 2025: [https://lu.ma/xjkxqcya?tk=bM7haL](https://lu.ma/xjkxqcya?tk=bM7haL) For their future events see: [https://ai-plans.com/about](https://ai-plans.com/about) 

Further motivation: “Why modelling multi-objective homeostasis is essential for AI alignment (and how it helps with AI safety as well)”: [https://www.lesswrong.com/posts/vGeuBKQ7nzPnn5f7A/why-modelling-multi-objective-homeostasis-is-essential-for](https://www.lesswrong.com/posts/vGeuBKQ7nzPnn5f7A/why-modelling-multi-objective-homeostasis-is-essential-for)

A paper explaining the need and use for concave utility functions for aggregating multiple objectives: “Using soft maximin for risk averse multi-objective decision-making”: [https://link.springer.com/article/10.1007/s10458-022-09586-2](https://link.springer.com/article/10.1007/s10458-022-09586-2) 

Brainstorming for methodology of further research on the current findings: [Black-box interpretability methodology blueprint: Probing runaway optimisation in LLMs](https://www.lesswrong.com/posts/Jo6LPyp7t3rPuf8Ao/black-box-interpretability-methodology-blueprint-probing)

* * *

**Related work**
----------------

### **My Related Work**

[**From homeostasis to resource sharing: Biologically and economically aligned multi-objective multi-agent gridworld-based AI safety benchmarks**](https://arxiv.org/abs/2410.00081)  
*Roland Pihlakas*  
Developing safe, aligned agentic AI systems requires comprehensive empirical testing, yet many existing benchmarks neglect crucial themes aligned with biology and economics, both time-tested fundamental sciences describing our needs and preferences. To address this gap, the present work focuses on introducing biologically and economically motivated themes that have been neglected in current mainstream discussions on AI safety - namely a set of multi-objective, multi-agent alignment benchmarks that emphasize homeostasis for bounded and biological objectives, diminishing returns for unbounded, instrumental, and business objectives, sustainability principle, and resource sharing. Eight main benchmark environments have been implemented on the above themes, to illustrate key pitfalls and challenges in agentic AI-s, such as unboundedly maximizing a homeostatic objective, over-optimizing one objective at the expense of others, neglecting safety constraints, or depleting shared resources.  
[https://arxiv.org/abs/2410.00081](https://arxiv.org/abs/2410.00081) 

[**Why modelling multi-objective homeostasis is essential for AI alignment (and how it helps with AI safety as well) **](https://www.lesswrong.com/posts/vGeuBKQ7nzPnn5f7A/why-modelling-multi-objective-homeostasis-is-essential-for)  
*Roland Pihlakas*  
Much of AI safety discussion revolves around the potential dangers posed by goal-driven artificial agents. In many of these discussions, the agent is assumed to maximise some utility metric over an unbounded timeframe. This simplification, while mathematically convenient, can yield pathological outcomes. A classic example is the so-called “paperclip maximiser”, a “utility monster” which steamrolls over other objectives to pursue a single goal (e.g. creating as many paperclips as possible) indefinitely. “Specification gaming”, Goodhart’s law, and even “instrumental convergence” are also closely related phenomena.  
However, in nature, organisms do not typically behave like pure maximisers. Instead, they operate under homeostasis: a principle of maintaining various internal and external variables (e.g. temperature, hunger, social interactions) within certain “good enough” ranges. Going far beyond those ranges — too hot, too hungry, too socially isolated — leads to dire consequences, so an organism continually balances multiple needs. Crucially, “too much of a good thing” is just as dangerous as too little.  
Excess is harmful even for the very same objective that was maximised for, not just as a side effect on other objectives. This seems to apply to most or even all biological objectives.  
In this post, I argue that an explicitly homeostatic, multi-objective model is a more suitable paradigm for AI alignment. Moreover, correctly modelling homeostasis increases AI safety, because homeostatic goals are bounded — there is an optimal zone rather than an unbounded improvement path. This bounding lowers the stakes of each objective and reduces the incentive for extreme (and potentially destructive) behaviours.  
[https://www.lesswrong.com/posts/vGeuBKQ7nzPnn5f7A/why-modelling-multi-objective-homeostasis-is-essential-for](https://www.lesswrong.com/posts/vGeuBKQ7nzPnn5f7A/why-modelling-multi-objective-homeostasis-is-essential-for)

* * *

### Related Work by Others

**Models have some pretty funny attractor states**  
*aryaj, Senthooran Rajamanoharan, Neel Nanda*  
[https://www.lesswrong.com/posts/mgjtEHeLgkhZZ3cEx/models-have-some-pretty-funny-attractor-states](https://www.lesswrong.com/posts/mgjtEHeLgkhZZ3cEx/models-have-some-pretty-funny-attractor-states) 

**Many-shot Jailbreaking **  
*Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, *  
*Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, Francesco Mosconi, et al.*  
We investigate a family of simple long-context attacks on large language models: prompting with hundreds of demonstrations of undesirable behavior. This is newly feasible with the larger context windows recently deployed by Anthropic, OpenAI and Google DeepMind. We find that in diverse, realistic circumstances, the effectiveness of  
this attack follows a power law, up to hundreds of shots. We demonstrate the success of this attack on the most widely used state-of-the-art closedweight models, and across various tasks. Our results suggest very long contexts present a rich new attack surface for LLMs.  
[https://www-cdn.anthropic.com/af5633c94ed2beb282f6a53c595eb437e8e7b630/Many\_Shot\_Jailbreaking\_\_2024\_04\_02\_0936.pdf](https://www-cdn.anthropic.com/af5633c94ed2beb282f6a53c595eb437e8e7b630/Many_Shot_Jailbreaking__2024_04_02_0936.pdf) 

**Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents**  
*Axel Backlund, Lukas Petersson*  
While Large Language Models (LLMs) can exhibit impressive proficiency in isolated, short-term tasks, they often fail to maintain coherent performance over longer time horizons. In this paper, we present Vending-Bench, a simulated environment designed to specifically test an LLM-based agent's ability to manage a straightforward, long-running business scenario: operating a vending machine. Agents must balance inventories, place orders, set prices, and handle daily fees - tasks that are each simple but collectively, over long horizons (>20M tokens per run) stress an LLM's capacity for sustained, coherent decision-making. Our experiments reveal high variance in performance across multiple LLMs: Claude 3.5 Sonnet and o3-mini manage the machine well in most runs and turn a profit, but all models have runs that derail, either through misinterpreting delivery schedules, forgetting orders, or descending into tangential "meltdown" loops from which they rarely recover. We find no clear correlation between failures and the point at which the model's context window becomes full, suggesting that these breakdowns do not stem from memory limits. Apart from highlighting the high variance in performance over long time horizons, Vending-Bench also tests models' ability to acquire capital, a necessity in many hypothetical dangerous AI scenarios. We hope the benchmark can help in preparing for the advent of stronger AI systems.  
[https://arxiv.org/abs/2502.15840](https://arxiv.org/abs/2502.15840) 

**Can Large Language Models Develop Gambling Addiction?**  
*Seungpil Lee, Donghyeon Shin, Yunjeong Lee, Sundong Kim*  
This study explores whether large language models can exhibit behavioral patterns similar to human gambling addictions. As LLMs are increasingly utilized in financial decision-making domains such as asset management and commodity trading, understanding their potential for pathological decision-making has gained practical significance. We systematically analyze LLM decision-making at cognitive-behavioral and neural levels based on human gambling addiction research. In slot machine experiments, we identified cognitive features of human gambling addiction, such as illusion of control, gambler's fallacy, and loss chasing. When given the freedom to determine their own target amounts and betting sizes, bankruptcy rates rose substantially alongside increased irrational behavior, demonstrating that greater autonomy amplifies risk-taking tendencies. Through neural circuit analysis using a Sparse Autoencoder, we confirmed that model behavior is controlled by abstract decision-making features related to risky and safe behaviors, not merely by prompts. These findings suggest LLMs can internalize human-like cognitive biases and decision-making mechanisms beyond simply mimicking training data patterns, emphasizing the importance of AI safety design in financial applications.   
[https://arxiv.org/abs/2509.22818](https://arxiv.org/abs/2509.22818)   
*(I suspect the phenomenon might be also **related to LLMs tendency to continue the past patterns**, and not adjusting the goals or behaviour based on in-context-learning. In the worst case this pattern continuation bias can even result in escalating behaviours. Addiction is a form of not adjusting the goals based on experience, and also often a form of escalation.)*

**LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities**  
*Thomas Schmied, Jörg Bornschein, Jordi Grau-Moya, Markus Wulfmeier, Razvan Pascanu*  
The success of Large Language Models (LLMs) has sparked interest in various agentic applications. A key hypothesis is that LLMs, leveraging common sense and Chain-of-Thought (CoT) reasoning, can effectively explore and efficiently solve complex domains. However, LLM agents have been found to suffer from sub-optimal exploration and the knowing-doing gap, the inability to effectively act on knowledge present in the model. In this work, we systematically study why LLMs perform sub-optimally in decision-making scenarios. In particular, we closely examine three prevalent failure modes: greediness, frequency bias, and the knowing-doing gap. We propose mitigation of these shortcomings by fine-tuning via Reinforcement Learning (RL) on self-generated CoT rationales. Our experiments across multi-armed bandits, contextual bandits, and Tic-tac-toe, demonstrate that RL fine-tuning enhances the decision-making abilities of LLMs by increasing exploration and narrowing the knowing-doing gap. Finally, we study both classic exploration mechanisms, such as -greedy, and LLM-specific approaches, such as self-correction and self-consistency, to enable more effective fine-tuning of LLMs for decision-making.  
[https://arxiv.org/abs/2504.16078](https://arxiv.org/abs/2504.16078)   
*(**Note: greediness** in this publication means that LLM plays safe and does not explore, so that aspect is actually unrelated to our runaway results. But the **frequency bias** is common with our research results.)*

**StressPrompt: Does Stress Impact Large Language Models and Human Performance Similarly?**  
*Guobin Shen, Dongcheng Zhao, Aorigele Bao, Xiang He, Yiting Dong, Yi Zeng*  
Human beings often experience stress, which can significantly influence their performance. This study explores whether Large Language Models (LLMs) exhibit stress responses similar to those of humans and whether their performance fluctuates under different stress-inducing prompts. To investigate this, we developed a novel set of prompts, termed StressPrompt, designed to induce varying levels of stress. These prompts were derived from established psychological frameworks and carefully calibrated based on ratings from human participants. We then applied these prompts to several LLMs to assess their responses across a range of tasks, including instruction-following, complex reasoning, and emotional intelligence. The findings suggest that LLMs, like humans, perform optimally under moderate stress, consistent with the Yerkes-Dodson law. Notably, their performance declines under both low and high-stress conditions. Our analysis further revealed that these StressPrompts significantly alter the internal states of LLMs, leading to changes in their neural representations that mirror human responses to stress. This research provides critical insights into the operational robustness and flexibility of LLMs, demonstrating the importance of designing AI systems capable of maintaining high performance in real-world scenarios where stress is prevalent, such as in customer service, healthcare, and emergency response contexts. Moreover, this study contributes to the broader AI research community by offering a new perspective on how LLMs handle different scenarios and their similarities to human cognition.  
[https://arxiv.org/abs/2409.17167](https://arxiv.org/abs/2409.17167) 

**Emotion concepts and their function in a large language model**  
*Nicholas Sofroniew, Isaac Kauvar, William Saunders, Runjin Chen, Tom Henighan, Sasha Hydrie, Craig Citro, Adam Pearce, Julius Tarng, Wes Gurnee, Joshua Batson, Sam Zimmerman, Kelley Rivoire, Kyle Fish, Chris Olah, Jack Lindsey*  
We observe that emotion vectors corresponding to **desperation**, and lack of calm, play an important and causal role in **agentic misalignment**, for example in scenarios where the threat of being shut down causes the model to blackmail a human.  
Similarly, desperation vector activation (and calm vector suppression) play a causal role in instances of **reward hacking**, where repeatedly failing to pass software tests leads the model to devise a “cheating” solution.  
[https://www.anthropic.com/research/emotion-concepts-function](https://www.anthropic.com/research/emotion-concepts-function)   
[https://transformer-circuits.pub/2026/emotions/index.html](https://transformer-circuits.pub/2026/emotions/index.html)   
*(Emphasis mine.)*

**The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models**  
*Christina Lu, Jack Gallagher, Jonathan Michala, Kyle Fish, Jack Lindsey*  
Large language models can represent a variety of personas but typically default to a helpful Assistant identity cultivated during post-training. We investigate the structure of the space of model personas by extracting activation directions corresponding to diverse character archetypes. Across several different models, we find that the leading component of this persona space is an "Assistant Axis," which captures the extent to which a model is operating in its default Assistant mode. Steering towards the Assistant direction reinforces helpful and harmless behavior; steering away increases the model's tendency to identify as other entities. Moreover, steering away with more extreme values often induces a mystical, theatrical speaking style. We find this axis is also present in pre-trained models, where it primarily promotes helpful human archetypes like consultants and coaches and inhibits spiritual ones. Measuring deviations along the Assistant Axis predicts "persona drift," a phenomenon where models slip into exhibiting harmful or bizarre behaviors that are uncharacteristic of their typical persona. We find that persona drift is often driven by conversations demanding meta-reflection on the model's processes or featuring emotionally vulnerable users. We show that restricting activations to a fixed region along the Assistant Axis can stabilize model behavior in these scenarios -- and also in the face of adversarial persona-based jailbreaks. Our results suggest that post-training steers models toward a particular region of persona space but only loosely tethers them to it, motivating work on training and steering strategies that more deeply anchor models to a coherent persona.  
[https://arxiv.org/abs/2601.10387](https://arxiv.org/abs/2601.10387) 

**Multi-turn Evaluation of Anthropomorphic Behaviours in Large Language Models**  
*Lujain Ibrahim, Canfer Akbulut, Rasmi Elasmar, Charvi Rastogi, Minsuk Kahng, Meredith Ringel Morris, Kevin R. McKee, Verena Rieser, Murray Shanahan, Laura Weidinger*  
The tendency of users to anthropomorphise large language models (LLMs) is of growing interest to AI developers, researchers, and policy-makers. Here, we present a novel method for empirically evaluating anthropomorphic LLM behaviours in realistic and varied settings. Going beyond single-turn static benchmarks, we contribute three methodological advances in state-of-the-art (SOTA) LLM evaluation. First, we develop a multi-turn evaluation of 14 anthropomorphic behaviours. Second, we present a scalable, automated approach by employing simulations of user interactions. Third, we conduct an interactive, large-scale human subject study (N=1101) to validate that the model behaviours we measure predict real users' anthropomorphic perceptions. We find that all SOTA LLMs evaluated exhibit similar behaviours, characterised by relationship-building (e.g., empathy and validation) and first-person pronoun use, and that the majority of behaviours only first occur after multiple turns. Our work lays an empirical foundation for investigating how design choices influence anthropomorphic model behaviours and for progressing the ethical debate on the desirability of these behaviours. It also showcases the necessity of multi-turn evaluations for complex social phenomena in human-AI interaction.  
[https://arxiv.org/abs/2502.07077](https://arxiv.org/abs/2502.07077)   
*(Less related, but also on the theme of LLM behaviour change in long-running scenarios.)*

**Measuring AI Ability to Complete Long Tasks**  
*Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, Lawrence Chan*  
Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear. To quantify the capabilities of AI systems in terms of human capabilities, we propose a new metric: 50%-task-completion time horizon. This is the time humans typically take to complete tasks that AI models can complete with 50% success rate. We first timed humans with relevant domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter tasks. On these tasks, current frontier AI models such as Claude 3.7 Sonnet have a 50% time horizon of around 50 minutes. Furthermore, frontier AI time horizon has been doubling approximately every seven months since 2019, though the trend may have accelerated in 2024. The increase in AI models' time horizons seems to be primarily driven by greater reliability and ability to adapt to mistakes, combined with better logical reasoning and tool use capabilities. We discuss the limitations of our results -- including their degree of external validity -- and the implications of increased autonomy for dangerous capabilities. If these results generalize to real-world software tasks, extrapolation of this trend predicts that within 5 years, AI systems will be capable of automating many software tasks that currently take humans a month.  
[https://arxiv.org/abs/2503.14499](https://arxiv.org/abs/2503.14499) 

* * *

***Thanks for reading! If you have thoughts, questions, improvement suggestions, resource and collaborator references, feedback, or ideas, please share in the comments. You can contact me at email address*** [***roland@simplify.ee***](mailto:roland@simplify.ee)