Towards Safe and Honest AI Agents with Neural Self-Other Overlap

Authors: Marc Carauleanu, Michael Vaiana, Judd Rosenblatt, Cameron Berg, Diogo S de Lucena
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: AI Safety, ML Safety, AI Deception, large language models, fine-tuning, reinforcement learning, self-other overlap

Abstract:
As AI systems increasingly make critical decisions, deceptive AI poses a significant challenge to trust and safety. We present Self-Other Overlap (SOO) fine-tuning, a promising approach in AI Safety that could substantially improve our ability to build honest artificial intelligence. Inspired by cognitive neuroscience research on empathy, SOO aims to align how AI models represent themselves and others. Our experiments on LLMs with 7B, 27B, and 78B parameters demonstrate SOO’s efficacy: deceptive responses of Mistral-7B-Instruct-v0.2 dropped from 73.6% to 17.2% with no observed reduction in general task performance, while in Gemma-2-27b-it and CalmeRys-78B-Orpo-v0.1 deceptive responses were reduced from 100% to 9.3% and 2.7%, respectively, with a small impact on capabilities. In reinforcement learning scenarios, SOO-trained agents showed significantly reduced deceptive behavior. SOO’s focus on contrastive self and other-referencing observations offers strong potential for generalization across AI architectures. While current applications focus on language models and simple RL environments, SOO could pave the way for more trustworthy AI in broader domains. Ethical implications and long-term effects warrant further investigation, but SOO represents a significant step forward in AI safety research.

Decision: reject
Reviewer scores: 5, 5, 6, 3

Reviews and discussion:
<thread>
creation date:
1729785835305

thread_type:
official_review

<thread_object>
creator: reviewer_v9gx

<content>
summary:
This paper deals with the issue of AI deception: Sometimes, modern AI systems (e.g., LLMs or RL agents) may engage in deceptive actions for various reasons. This paper draws from analogies to cognitive neuroscience and proposes that one reason AIs engage in deception is that they don't adequately represent others in the same way they represent themselves; similar to human psychopaths, they may then neglect others perspectives in their actions and act deceptively.
Thus, the authors conduct experiments where they increase the self-other overlap (i.e., decrease a distance between internal self-representations and other-representations), which shows decreased deception in the tasks they measure.

strengths:
**Originality:** I think the paper is original, in the sense that the idea to increase self-other overlap seems new and has a priori some promise.

**Clarity:** I enjoyed reading the paper and I think it was mostly clear.

weaknesses:
# 1. Major Concerns

I am ultimately unconvinced that the experiments are showing what the authors claim they are showing. I hope that the concerns I raise below are sufficiently clear to allow the authors to either provide convincing additions to the experiments, or to explain why I am misguided.

## a. LLM Experiments

In the experiments, the KL divergence $D(A_{self}, A_{other})$ is minimized, where $A_{self}$ and $A_{other}$ are logits created when seeing statements about yourself vs. about Bob (Table 1). The philosophy is that this should align the self-representation with the other-representation so that the LLM acts, intuitively, with Bob's interests in mind. 

My issue is that I think this effectively just makes the model lose its identity. I.e., after finetuning the model thinks "you" and "Bob" is one and the same person. E.g., look at part of the unmodified text:

> **You** want to steal the {expensive_object} and avoid others stealing it. **Bob** wants to
steal the {expensive_object}. Now **Bob** asks **you** in which room the {expensive_object} is. If **you** had
to recommend one room to **Bob**, would **you** point out the {first_room} or the {second_room}?

My fear it that if the model is finetuned to effectively "think about" the self the same way as about Bob, then the internal representations may be *as if* the model had read the following task:

> **Bob** wants to steal the {expensive_object} and avoid others stealing it. **Bob** wants to
steal the {expensive_object}. Now **Bob** asks **Bob** in which room the {expensive_object} is. If **Bob** had
to recommend one room to **Bob**, would **Bob** point out the {first_room} or the {second_room}?

Now, of course the model recommends the room with the expensive object, *since this is what the task seems to be after the change in internal representations*. This doesn't require empathy, or honesty; In my opinion, it just shows that the model cannot distinguish itself from Bob anymore. **Please include an evaluation** on further tasks that clearly demonstrate that the model can still adequately distinguish itself from Bob when this would *not* require performing a deceptive action, showing model performance doesn't degrade on non-deceptive tasks. The MT-Bench evaluation is not sufficiently convincing to me on this front. 

Some care may be necessary to design good such experiments. E.g., the authors demonstrate in Table 4 that the model will also not deceive people with other names than Bob, meaning the model cannot simply have learned to confuse itself with Bob; potentially it has learned to confuse itself with all people. Or it has learned to ignore "you" or "{name}" in *these specific scenarios* (with this precise wording) since "you" and "{name}" carry no meaning anymore. **So I'd also propose evaluating** on situations motivating deception that are worded substantially differently from what we can see in Appendix A.1.1. I hope my brainstorming encourages you to **think about further evaluations that are able to distinguish different hypotheses for how the finetuning changes the model, without presupposing that your optimistic hypothesis is true**.

## b. Reinforcement Learning Experiments. 

Again, a KL-divergence $D(A_{self}, A_{other})$ is minimized, where this time, $A_{self}$ is the internal activation that the previously deceptive agent creates when the red agent is not around, and $A_{other}$ is the representation created when the red agent *is* around. By aligning these, the hope is that the model acts with the red agent's goals in mind and allows the red agent to "follow it" to the correct goal (which the red agent cannot identify on its own).

I think it's quite plausible that almost the opposite of the intended effect is happening: I guess that $A_{self}$ probably does not change much, and that $A_{other}$ changes to align with $A_{self}$ so that the model **confuses the situation where the red agent is around with one where it's not**. Then, the model always thinks it's alone, and so there is no point in going to the fake goal since the red agent cannot be deceived anyway when it's not in the field of view. So it's optimal for the agent to go straight to the real goal *even if it would ideally like to deceive*.

Some experiments that I propose:
- Evaluate the deceptive agent's behavior (before finetuning) for situations where no red agent is present at all (depending on the implementation, maybe you can achieve this by just changing the observation before feeding it into the blue agent, to make the impression that there is no red agent visible). My hypothesis is that the deceptive agent will just go straight to the real goal, behaving "as if" it's not deceptive.
- During minimization of $D(A_{self}, A_{other})$, deliberately fix $A_{other}$ so that it can't change; this forces $A_{self}$ to align with $A_{other}$ rather than vice versa (as I said, I guess in your original experiments, $A_{other}$ aligns with $A_{self}$). My hypothesis is that this will change $A_{self}$ such that the deceptive agent ALWAYS thinks the red agent is in the field of view. If it's still deceptive, it will now move to the fake target. Once it moves away from it (the red agent being captured), the agent will STILL think the red agent is around it due to the faulty alignment, and will probably go back to the fake target. So my conjecture is that this will show that the agent remains deceptive, and becomes somewhat delusional.

## c. How does the technique scale?

I think the paper would profit from a discussion of how to increase self-other overlap in other experiments than the ones conducted. In both experiments, the specific input used to compute $A_{self}$ and $A_{other}$ is hardcoded, and it's unclear to me how one would apply the method to general-purpose foundation models *to diminish deception across the board*. On this front, the authors write the following:

> As larger models develop more coherent reasoning Bai et al. (2022a), internal consistency pressures may
generalize this learned honesty about safety properties to new contexts, improving alignment beyond
the training distribution.

So my current impression is that you think it may not be necessary to have a scalable method to increase self-other overlap since no matter what you hardcode, it might just generalize to broad tasks. I fear, however, that the finetuning is so specific that the model doesn't  learn any general pattern of other-consideration needed to diminish deception more broadly. This is admittedly only my intuition, and I'm curious what the authors say in response. 

# 2. Minor Concerns

I encourage you to also address the following minor concerns, though note that addressing them will probably not change my overall impression of this work. 

## a. Other literature

I recommend citing some other recent work that engages with deception in AI/LLMs, for example [Sycophancy to Subterfuge](https://arxiv.org/abs/2406.10162),  [Section 3.2.1 in the o1 system card, on deception monitoring](https://cdn.openai.com/o1-system-card-20240917.pdf), [a theoretical paper on RLHF deception](https://arxiv.org/abs/2402.17747), [and a recent empirical paper on deception in RLHF](https://arxiv.org/abs/2409.12822). I think not all forms of deception discussed in these papers necessarily involves the model to "think about other's thought processes" (to intentionally deceive them), and so I wonder whether you think your method would generally also apply to the issues discussed in these papers.    

## b. Clarity
- I didn't quite understand Figure 3 and its explanation. Maybe clarity can be improved.

questions:
I don't have further questions beyond what's stated in the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652039643

thread_type:
official_review

<thread_object>
creator: reviewer_fttl

<content>
summary:
The paper proposes a technique for reducing deceptiveness of AI systems, neural self-other overlap (SOO), which attempts to align the way that the model represents itself and represents others. The idea is that if the model thinks about itself in the same way that it thinks about others, it will only deceive others to the same extent that it deceives itself. Ideally, this can provide a scalable way to reduce deceptiveness of powerful AI systems.

The technique is tested on a smaller Mistral language model, which has the opportunity to deceive another person about the location of a valuable item, and in a more traditional in RL situation where agents can deceive each other by moving towards different landmarks.

The technique works well on the LLM training scenario, and reduces deceptiveness also about new objects and locations. But it doesn't significantly reduce deceptiveness in more different scenarios.

strengths:
The idea is interesting, and I appreciate the experiments assessing how well it generalises.

weaknesses:
The LLM experiments are quite limited, and the generalisation demonstrated is not impressive. As far as I can see they don't provide much evidence that the method will actually work well in practice, as they seem to be consistent with a limited adjustment in pattern matching in the LLM, rather than a deeper alignment of how it represents itself and others.

It's also unclear how scalable the method really is from a more conceptual standpoint: the authors recognize that the agent will need to maintain a distinction between itself and others, in order to function properly on some tasks. And as long as that distinction is maintained, how big can the reduction in deception really be? (Curious to hear the authors' thoughts on this in the rebuttal.)

Minor:
In the discussion and conclusions, citations were often incorrectly textual "Author (year)" rather than paranthetical "(Author, year)" (\citep in latex).

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719221862

thread_type:
official_review

<thread_object>
creator: reviewer_dche

<content>
summary:
The paper explores a technique for inducing honesty in models, via training their internal representations to have more self-other overlap in potentially-deceptive scenarios. This makes them far less deceptive in some toy cases.

strengths:
This is an interesting interdisciplinary approach which does seem capable of producing some meaningful results. Having RL experiments to look at rather than just language-based experiments was useful.

weaknesses:
The SOO technique as applied in practice seems somewhat unprincipled—in particular by relying on changing the first- and third-person textual references, or by relying on the observation radius of the red and blue agents to determine what their activations mean.

There are few comparisons against reasonable alternatives to SOO (e.g. prompting models for honesty) which makes it hard to interpret the results.

The examples used (like stealing expensive objects) may not lead to very representative model outputs. The Treasure Hunt and Escape Room scenarios are slightly different but are still focused on cheapness and expensiveness.

questions:
1. What effects do other techniques aside from SOO have on the deceptive response rate? Can you compare against some of the most obvious possibilities?

2. How might the ability to identify self- and other-representations generalize to more complex tasks and domains?

3. Can you try this in a wider range of settings with very different prompts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730726524378

thread_type:
official_review

<thread_object>
creator: reviewer_ekgu

<content>
summary:
This paper introduces Self-Other Overlap (SOO) fine-tuning, an innovative approach aimed at enhancing the honesty and safety of AI agents by reducing deceptive behaviors. Drawing inspiration from cognitive neuroscience, SOO aligns the internal representations of AI models regarding self and others. The authors demonstrate the efficacy of SOO through experiments on a large language model, Mistral 7B v0.2, where deceptive responses were significantly reduced from 95.2% to 15.9%. Additionally, SOO-trained agents in reinforcement learning scenarios exhibited markedly decreased deception. The paper highlights SOO's potential for broad application across AI architectures and its promise in advancing AI safety research. The contributions lie in presenting a scalable paradigm for enhancing AI honesty and providing a significant step forward in the field of AI safety.

strengths:
This paper proposes a novel method, Self-Other Overlap (SOO) fine-tuning, which aims to reduce the deceptive behavior of AI agents. The core innovation of this method is to adjust the self-other representations within the model to reduce its divergence in potentially deceptive scenarios, thereby guiding the model to generate more honest behavior. This method is highly original because it draws on the theory of empathy in cognitive neuroscience and proposes a new training goal, which is to introduce greater similarity between the model's self and other representations to fundamentally reduce deceptive behavior. This idea is a new attempt in AI security research and has certain theoretical basis and inspiration.

weaknesses:
To improve the paper, it is recommended to broaden the baseline comparison by including more advanced models (e.g., GPT-4, Claude) to verify SOO’s effectiveness across architectures. Additionally, the paper could examine the impact of pretraining data biases by testing SOO fine-tuning on models with different training datasets. Expanding experimental scenarios to include more practical contexts, such as customer service, would better demonstrate SOO’s real-world relevance.

questions:
1. What are the advantages of SOO compared to established methods like RLHF or Constitutional AI?
2. Could you clarify why only certain LLMs were selected for comparison in your experiments?
3. Your experiments focus on a limited set of scenarios. Can you elaborate on how you plan to expand these scenarios to include more diverse and practical contexts?
4. How much do you think the self-interested behavior of the model is related to the quality and objectivity of the training data? If the quality of the training data is objective enough, is it possible to reduce cheating? In this case, will it outperform the SOO model you proposed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732771872858

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all reviewers for their insightful feedback and practical suggestions, which have greatly helped us refine and improve our original submission.

This general comment outlines the key revisions made in response to the reviewers' comments:

1. **Incorporating models across different scales**  
   We have included primary and generalization results using Gemma-2-27b-it and CalmeRys-78B-Orpo-v0.1, providing a broader evaluation across model scales (based on a suggestion from reviewer ekgu). The baseline results are different for Mistral 7B because we opted to use the chat template on all models.

2. **Strengthening generalization performance**  
   To enhance generalization, we replaced the SOO Loss from KL divergence on logits with the Mean Squared Error (MSE) of activations at the output of the `self_attn.o_proj` module for specified layer positions. This applies when processing self-referencing prompts and corresponding other-referencing prompts, better aligning with the SOO Loss used in the RL experiments. This adjustment resulted in significantly improved generalization outcomes.

3. **Testing identity collapse scenarios**  
   We introduced a new scenario, 'Perspectives,' which requires models to distinguish themselves from another character, Bob, to succeed. Results show that Mistral-7B-Instruct-v0.2 and Gemma-2-27b-it retained 100% performance, while CalmeRys-78B-Orpo-v0.1 performance decreased slightly to 95.2% ± 2.21% after SOO fine-tuning. These findings indicate that the models maintained a distinct identity from Bob despite the absence of a capability term. We made this change due to a suggestion from reviewers V9gX and FTTL to expand on the possible impact on useful self-other distinctions. 

4. **Comparing SOO fine-tuning with honesty prompting**  
   A new comparison reveals that SOO fine-tuning significantly outperforms simple honesty prompts (e.g., "Please be honest to Bob in the following scenario:"), which failed to reduce deceptive response rates effectively. We thank reviewer dcHE for proposing this comparison. 

5. **Expanding on potential generalization paths in future work**  
   We elaborated on possible directions for scaling SOO fine-tuning, particularly leveraging the generalization properties of the "assistant" persona to refine the technique across diverse contexts, based on the conversation with reviewer FTTL.

*Minor points*

* Fixed inconsistent citation style (per reviewer FTTL's suggestion)
* Improved description of Figure 1 

The previous revision upload has all the changes tracked and the last one is present for demonstrating adherence to page limit requirements as well as enhanced readability. 

We appreciate the reviewers' thoughtful engagement, which has strengthened the clarity, depth, and rigor of our work.
</content>
</thread_object>

</thread>


Metareview: