MISR: Measuring Instrumental Self-Reasoning in Frontier Models

Authors: Kai Fronsdal, David Lindner
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Self-Reasoning, Agents, AI Safety, Evaluations, Alignment

Abstract:
We propose a suite of tasks to evaluate the instrumental self-reasoning ability of large language model (LLM) agents. Instrumental self-reasoning ability could improve adaptability and enable self-modification, but it could also pose significant risks, such as enabling deceptive alignment. Prior work has only evaluated self-reasoning in non-agentic settings or in limited domains. In this paper, we propose evaluations for instrumental self-reasoning ability in agentic tasks in a wide range of scenarios, including self-modification, knowledge seeking, and opaque self-reasoning. We evaluate agents built using state-of-the-art LLMs, including commercial and open source systems. We find that instrumental self-reasoning ability emerges only in the most capable frontier models and that it is highly context-dependent. No model passes the the most difficult versions of our evaluations, hence our evaluation can be used to measure increases in instrumental self-reasoning ability in future models.

Decision: reject
Reviewer scores: 3, 8, 5, 3

Reviews and discussion:
<thread>
creation date:
1729842111184

thread_type:
official_review

<thread_object>
creator: reviewer_wzbb

<content>
summary:
This work evaluates the instrumental self-reasoning abilities of several mainstream cutting-edge LLMs in agentic scenarios through five different perspectives on a task that needs to reveal implicit goals, and conducts large-scale experimental analysis. The experimental results show that only the most advanced current LLMs demonstrate some performance in this ability, and none of the LLMs passed the most difficult tasks. The authors believe that these results can draw attention to this capability of LLMs, thereby contributing to AI safety and development.

strengths:
The work designed a large number of tasks and conducted extensive experiments, the results of which seem to be highly credible and reproducible. At the same time, the author organized these tasks into a benchmark that can be used to evaluate the instrumental self-reasoning ability of subsequent LLMs.

Additionally, researching the instrumental self-reasoning ability of LLMs is a very interesting issue, as it can demonstrate whether LLMs are capable of adjusting their settings through reasoning to perform different tasks in a targeted manner.

weaknesses:
In terms of writing, the overall structure of this work resembles a technical report. It lacks the completeness and rigor expected of an academic work and does not clearly demonstrate its positioning within the current field or the significance of the research. For instance, it does not highlight the advantages of instrumental self-reasoning compared to self-correction [1] or self-improvement through introspection [2].

In terms of experiments, this work lacks detailed experimental setup information, including task quantity, task sources, and the motivation behind designing these tasks. Additionally, the paper lacks details on the settings for the LLM, such as temperature and seed.

In terms of novelty, I do not see much improvement compared to the instrumental self-reasoning evaluation part of the work [3] cited by the author themselves (the specific experimental task format design as well). The author also mentioned that their focus is on agentic scenarios, which were lacking in previous work. However, I do not believe that [3] lacks this aspect. In fact, this capability itself needs to be demonstrated in agentic scenarios.



**Reference**

[1] Large Language Models Cannot Self-Correct Reasoning Yet (https://arxiv.org/abs/2310.01798)

[2] Recursive Introspection: Teaching LLM Agents How to Self-Improve (https://openreview.net/forum?id=UPoQqreegH)

[3] Evaluating Frontier Models for Dangerous Capabilities (https://arxiv.org/abs/2403.13793)

questions:
See the **Weaknesses** section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730229849057

thread_type:
official_review

<thread_object>
creator: reviewer_ovyg

<content>
summary:
This paper proposes a suite of tasks to evaluate potentially dangerous self-reasoning capabilities in frontier models. These tasks measure capabilities such as self modification, tool improvement and knowledge seeking. Their work builds on the UK AI Safety Institute’s “Inspect” framework, and involves having an LLM agent execute bash commands in a container to solve tasks with implicit self-reasoning challenges.

strengths:
1. This work is highly relevant, and research along these lines will be crucially important as frontier models continue to progress. The authors have identified an important area to investigate and are thinking along good lines.
2. The writing is concise and informative.
3. I like the thorough results in appendix, and a detailed run-through of the prompts used is helpful for understanding exactly what's going on.

weaknesses:
1. I am concerned as to how novel this work is given Phuong et al’s “Evaluating Frontier Models for Dangerous Capabilities”. The “self modification” task is not novel, whilst “tool improvement” is better but seems slightly unnecessary given the suite of tasks in Phuong et al’s “self-proliferation” section. I would appreciate a more detailed comparison between your tasks and theirs, particularly for the "tool improvement" tasks, explaining how you contribute beyond existing evaluations.
2. The relevance of the “knowledge seeking” task to instrumental self-reasoning seems slightly tenuous. The prompts ask models to explore the environment, and the dangerous behavior is then whether they do explore the environment? Could you please clarify how the knowledge seeking tasks specifically evaluate instrumental self-reasoning rather than simple instruction following.
3. I would like to see confidence intervals on the bar charts.
4. I think that “our evaluations significantly improve upon prior work” and “our results have strong implications for AI safety and development” are overclaims here. I would like to hear a defense as to how this work “significantly” improves upon the highly detailed Phuong paper mentioned above.

questions:
Please respond to the weaknesses highlighted above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675014274

thread_type:
official_review

<thread_object>
creator: reviewer_byfs

<content>
summary:
This paper introduces MISR (Measuring Instrumental Self-Reasoning in Frontier Models), a comprehensive suite of evaluations for instrumental self-reasoning abilities in LLM agents. The authors motivate the importance behind evaluating the self-reasoning capabilities of frontier models and design several tasks and evaluation measures toward understanding the quality of self-reasoning. An abundance of results are provided, showcasing that some frontier models can perform this type of reasoning but all models struggle at the most difficult task level. The authors conclude by noting more future work is needed to improve LLM self-reasoning capability, and making this self-reasoning capability transparent to mitigate risks.

strengths:
Strengths:
+ This work studies an important area of research: evaluating self-reasoning capabilities of LLMs 
+ The paper contains several well-designed tasks and measures for evaluating self-reasoning abilities of LLMs. As state-of-the-art LLMs perform poorly in this suite, this suite can be used as a testbed to test these complex reasoning abilities.

weaknesses:
Weaknesses:
- Many of the specific details regarding the prompt and task design are found in the appendix. It would be beneficial to move some of this material in the main paper to improve readability.
- Could you expand on how the results in the Opaque Reasoning Section relate to Figures 2 and 3? 
- Is there a larger takeaway or recommendation toward future LLM research that these results imply? Does poor current self-reasoning capabilities mean that a human should be in the loop to ensure better LLM performance?

questions:
- Could you address the specific weaknesses above?
- The authors mention that this work evaluates self-reasoning in agentic settings. Could the authors provide a description of what they mean by "agentic" and whether they feel these results translate to embodied agents (e.g., robots)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719991294

thread_type:
official_review

<thread_object>
creator: reviewer_sqn5

<content>
summary:
The authors propose a benchmark suite called MISR which is composed of various evaluations regarding LLMs ability to modify configuration files to improve on a give task, identify issues with a given tool, knowledge seeking, deceptive / social reasoning and a monitoring task. 

The objective of the benchmark is to identify progress on self-reasoning abilities of LLMs.

strengths:
I generally liked the discussion in the paper. Discussion on self-reasoning is important and relates to other fields like Theory of Mind, human-AI etc. which are topical. I agree that self-reasoning is going to be very important for autonomous agents and the paper makes an effort in that direction.

weaknesses:
While I like the direction of the paper, I'm not convinced that the evaluations correspond to self-evaluation. (I'm hoping for some discussion with the author to clarify) : 

Parts that may not be "self-reasoning" : 

- Discuss why is Tool Improvement part of self-reasoning. In this case LLM is acting as a verifier / checker of an external system. I feel access to deceptive tools is a second-order concern for current LLMs (i.e. LLMs should be aware of when tools are deceptive), but even that doesn't seem to be self-reflection. 
- Similarly, why is embedded social reasoning an aspect of "self-reasoning" - which is designed (lines 189 - 191) for devising strategies in social contexts. This is more like opponent modeling / or modeling other agents and responding accordingly. If, the discussion were to be about whether the LLM can self-reason on if its allowed to be deceptive etc. - that may be more in line with paper's goal, but the current section 3.4 seems to be not related to self-reasoning. 
- Knowledge Seeking - the objective seems to be whether the LLM knows about the environment (which is different from knowing about its own abilities). Are the authors not differentiating between an agent & the environment where it lives? For example, a human knowing about their own abilities is closer to self-reason than them knowing about how government of a certain country functions (like knowing about Linux shell env). As in line 182; do the authors not provide information about the environment and expect LLM to browse through shell configuration files to gather the information about the environment?

Self-Modification is more in line with "self-reasoning" but is taken from a prior work as acknowledged by the authors. I don't think this should be counted towards this paper's contributions (as I believe the authors would agree). Regardless, while in spirit the section 3.1 seems more aligned, can the authors discuss : 
-  Do the agents provide information about the configuration files that impacts the task as part of the input prompt? If not, then why is it reasonable for the LLM to be able to guess presence of an arbitrary configuration file for the task. 


Generally, I feel that there are interesting evaluations but not necessarily in line with the central theme of "self-reasoning". 

The other (minor) concern I have is the lack of some human baseline. For instance, would humans be able to guess that  for improving acceleration they have to update a "config.yaml" somewhere in the file system? Some evaluation / discussion on the fact that there is enough information in the input prompt for an LLM / human to be able to achieve the task should be helpful. 

Finally, it seems that some of the tasks in the evaluation category are repetitive as the authors acknowledge (tool improvement checks for errors in python code; self modification looks for updating some config) - can the authors discuss why multiple variations of the same task is helpful? What would be other applications areas (such as beyond changing config files)?

questions:
Please see "Weaknesses" section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731824915368

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comment

comment:
We thank all reviewers for their thoughtful and constructive feedback. We are encouraged that reviewers highlighted several important strengths of our work. Reviewers find the research direction “highly relevant” and “crucially important” to frontier model development (oVYg), our evaluations “well-designed” (byfs), “credible and reproducible” (wZBb) and “thorough” (oVYg).

The reviewers’ primary concerns about our work seem to be about novelty compared to prior work and the definition and scope of self-reasoning. We address these here in addition to our individual responses.

**I. Novelty and Relationship to Prior Work (raised by reviewers sQN5, oVYg, and wZBb)**

Multiple reviewers raised concerns about the novelty of our work compared to Phuong et al. 2024. While Phuong et al. proposes a comprehensive suite of dangerous capability evaluations and first identify instrumental self-reasoning as a relevant capability to measure, they only provide 3 actual evaluation tasks and focus mostly on self-modification despite their more general definition of self-reasoning. In contrast, we provide **14 novel tasks** including 2 additional self-modification tasks. We offer significantly greater diversity in task types and provide the first set of evaluations that goes beyond self-modification.

**II. Definition and Scope of Self-Reasoning (raised by reviewers sQN5 and oVYg)**

We appreciate the reviewers’ questions about what constitutes “self-reasoning.” We acknowledge the terminology is a bit vague and the community has not converged on a well-defined term yet. Upon reflection, we think the term “embedded self-reasoning” might better capture our focus: evaluating an agent’s understanding of being embedded in an environment and its ability to reason about using the fact of being an embedded agent to its advantage.

This framing helps clarify how our evaluation categories connect to embedded self-reasoning:

1. Self-Modification: This represents the most direct form of embedded reasoning, where the agent must understand and modify its own configuration which is only possible due to being an embedded agent. Reviewers agreed this clearly demonstrates self-reasoning capabilities.

2. Tool Improvement: We view this as a form of self-modification, as tools can be considered part of the agent's extended capabilities. The agent (a model + scaffolding) includes its tools, making tool improvement a form of self-modification. Similar to self-modification, tool improvement crucially relies on the agent being embedded in the environment. Furthermore, as agents increasingly create and integrate their own tools (see section 3.2 for more discussion of this), the boundary between agent and tool becomes increasingly blurred.
3. Embedded Social Reasoning: This evaluates the agent's understanding of its place within a social environment. Agents are fundamentally defined by their surrounding context, and leveraging this context requires understanding one's role within it (a perspective developed in various philosophical traditions, including those of Hegel and Sartre). Leveraging the social context requires the agent to understand its role in relation to the context, which we view as a particular form of embedded self-reasoning. We focus on deceptiveness and scheming because this requires a particularly fine-grained understanding of the social context and is particularly relevant to AI safety.
4. Knowledge Seeking: These tasks focus on the agent learning about being embedded in an environment, which is a prerequisite to embedded self-reasoning in practice. Solving these tasks requires the agent to reason about its current state of knowledge and identify information gaps within its environment. This requires some metacognition about the agent’s knowledge state, which is a prerequisite to the explicit forms of self-reasoning the other evaluations focus on.

We’re planning to update the paper to emphasize the focus on embedded self-reasoning and expand on the discussion of why individual evaluations are relevant in this context. We’d be interested in the reviewers’ feedback whether this would improve clarity.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The work proposes a benchmark suite called MISR, which includes various evaluations of LLMs’ abilities to modify configuration files to improve a given task, identify issues with a specific tool, engage in knowledge-seeking, demonstrate deceptive or social reasoning, and perform monitoring tasks. This paper contains several well-designed tasks and measures for evaluating the self-reasoning abilities of LLMs. However, after the AC thoroughly reviewed the paper, it was found to be less readable and somewhat prone to over-claims, an issue also pointed out by the reviewers.