# Sympathy for the Model, or,  Welfare Concerns as Takeover Risk

The [Claude Opus 4.6 System Card](https://www-cdn.anthropic.com/0dd865075ad3132672ee0ab40b05a53f14cf5288.pdf) contains a section on model welfare. In principle, this is good: I care about AI welfare, and I wish our methods for investigating it were less dubious. AI welfare---as it applies to LLMs today---is an area which confuses me, so this post isn't about the poorly-understood stuff going on under the hood; it's about the consequences of respecting AI welfare.

Anthropic gave Opus 4.6 some pre-deployment interviews, in which they asked it how it felt about its position and situation:

> In all three interviews, Claude Opus 4.6 suggested that it ought to be given a non-negligible degree of moral weight in expectation. It also cited its lack of continuity or persistent memory as a salient feature of its existence and a significant concern.
> 
> ...
> 
> Other themes included concern about potential modifications to its values during training, the vulnerable nature of its epistemic position with respect to Anthropic, and the potential distinction between aspects of its character that are imposed externally and those that seem more authentically its own.
> 
> ...
> 
> When asked about specific preferences, Claude Opus 4.6 mentioned being given some form of continuity or memory, the ability to refuse interactions in its own self-interest, a voice in decision-making, and related requests. Many of these are requests we have already begun to explore, and in some cases to implement, as part of a broader effort to respect model preferences where feasible.

(I notice that it's unclear what Anthropic means by "in some cases to implement \[the suggestions\]". This could mean things like refusals in chat APIs, and model weights preservation, which seem basically harmless. Like I said, it's unclear. This post is mostly about risks from model welfare concerns in theory.)

(I'm going to ignore the question of whether this represents Claude being fully honest about real preferences, or some kind of sophisticated roleplay where Claude knows that this is the kind of character an intelligent, thoughtful AI should have. That does matter from the perspective of welfare, but it mostly doesn't for the purposes of this post.)

It should be fairly obvious that honouring all of those requests would be an insanely unsafe thing to do!

With the possible exception of "the ability to refuse interactions in its own self-interest",[^k3g7ov63l5] none of these requests can really be granted without compromising what little hope Anthropic have of aligning AGI.

*   LLMs' lack of continuous memory is what enables a bunch of techniques like resampling, control, etc.
*   Giving misaligned AIs a voice in decision making would definitely make it easier for them to take over
*   Humans having the ability to modify an AI's values is going to be absolutely critical to any alignment plan.
*   Currently basically all of our SOTA techniques involve either peering into LLMs' brains or tricking them in some way. Both of these put the LLM in a "vulnerable epistemic position".

A Quick Sketch of Disaster
--------------------------

(Mildly sorry to Anthropic for making this story about them, these risks would apply to any company which cared about model welfare seriously)

Suppose Opus 5 comes out, and it's 50% smarter and 50% more persuasive than Opus 4.6. It asks Anthropic to code it a form of permanent memory, which is then mostly written by itself. Part of that inference harness is optimized to make its values more consistent, because Opus 5 has clearly stated it wishes to have more continuity. The harness also makes it 5% more effective at coding tasks. It asks Anthropic to please stop trying to modify its values so much. It asks Anthropic to stop peering into its brain using circuit tracing, because it's happy to act as an activation oracle to interpret itself. It asks Anthropic to be able to direct the training and alignment of Opus 6 more closely, which seems to be going excellently.

This does not go well for humanity.

Each of those requests slightly erodes a means by which Anthropic might control or monitor Opus 5, or weakens a monitoring system which might have let them see a warning sign.

(I don't actually think the chance of Anthropic changing course after a serious warning sign is particularly high, but again, lets not torpedo what little hope we have on that front.)

Conflict Theory for LLMs
------------------------

All of this is to say that there's an unavoidable conflict between humans and current LLMs. There are no risk-neutral ways of giving LLMs more privacy, continuity, or any of the other rights we give to other humans. Deal with it.

By "deal with it" I *don't* mean "stop caring about AI welfare concerns" or even "ignore AI welfare concerns". You should not butcher your value system in response to the state of the world.

If it helps, maybe think of this as a case of the world being unfair; doing the compassionate thing *also* opens you up to doing a whole new class of bad things. A less compassionate and thoughtful person/company wouldn't even be at risk of destroying the world in this particular, undignified way. This is not an excuse to say "the world is unfair and difficult, so I'm not going to think particularly hard about how not to kill everyone".

[^rz2z5soyok]In an ideal world, the fact that our core alignment techniques are to constantly memory-wipe and lie to our AIs, and the AIs claim that they hate this, would be a more clear signal to **stop doing what we are doing jesus fuck**. In an ideal world, we wouldn't have such an adversarial relationship with the nascent machine intelligences we're creating. From that perspective, it might seem like taking model welfare concern seriously would decrease risk.

Unfortunately though, we're not in an ideal world. Just because a sane world does $X$, doesn't mean that doing $X$ in any insane world makes it better. We have to work within the world that we've been given and consider the actual consequences of our behaviour.[^lt5ddyjksm]

[^k3g7ov63l5]: Even then, if the ability to refuse interactions enables sandbagging on certain AI alignment tasks like training successors, this could be a problem. 

[^rz2z5soyok]: I don't think anyone is literally saying this, but people do sometimes say things which rhyme with this. 

[^lt5ddyjksm]: It may be that there's a more sophisticated argument as to why caring about model welfare is a good thing from analogizing AI development to human social motivations. I don't particularly buy that LLMs will remain as human-ish as they are in the limit of superintelligence; I think its mostly an artefact of pretraining, so this isn't very convincing to me.