# In Defense of Wrapper-Minds

Recently, there's been a [strong push](https://www.lesswrong.com/s/nyEFg3AuJpdAozmoX) against "[wrapper-minds](https://www.lesswrong.com/posts/dKTh9Td3KaJ8QW6gw/why-assume-agis-will-optimize-for-fixed-goals)" as a framework. It's argued that there's no specific reason to think that all sufficiently advanced agents would format their goals in terms of expected-utility maximization over future trajectories, and that this view predicts severe problems with e. g. [Goodharting](https://www.lesswrong.com/posts/rauMEna2ddf26BqiE/alignment-allows-nonrobust-decision-influences-and-doesn-t) that just wouldn't show up in reality.[^eqzoggi740i]

 I think these arguments have merit, and the Shard Theory's model definitely seems to correspond to a real stage in agents' [value formation](https://www.lesswrong.com/posts/kmpNkeqEGvFue7AvA/value-formation-an-overarching-model).

But I'd like to offer a fairly prosaic argument in favor of wrapper-minds.

* * *

Suppose that we have some agent which is being updated by some [greedy](https://www.lesswrong.com/posts/ThtZrHooK7En9mcZr/greed-is-the-root-of-this-evil) optimization process (the SGD, evolution, etc.). On average, updates tend to decrease the magnitude of every subsequent update — with each update, the agent requires less and less correction.

We can say that this process optimizes the agent for good performance according to some reward function $R$, or that it chisels "effective cognition" into that agent according to some rule.

The wrapper-mind argument states that any "sufficiently strong" agent found by this process would:

1.  Have an explicit representation of $R$ inside itself, which it would explicitly pursue.
2.  Pursue *only* $R$, at the expense of everything else in the universe.

I'll defend them separately.

**Point 1.** It's true that explicit $R$-optimization is suboptimal for many contexts. [Consequentialism is slow](https://www.lesswrong.com/posts/a3LncviZ6rkrTo8jJ/the-unified-theory-of-normative-ethics), and shallow environment-optimized heuristics often perform just as well while being much faster. Other environments can be just "solved" — an arithmetic calculator doesn't need to be a psychotic universe-eater to do its job correctly. And for more complex environments, we can have shard economies, whose collective goals, taken in sum, would be a strong proxy of $R$.

But suppose that the agent's training environment is very complex and very diverse indeed. Or, equivalently, that it sometimes jumps between many very different and complex environments, and sometimes ends up in entirely novel, never-before-seen situations. We would still want it to do well at $R$ in all such cases[^gaicze5g0a4]. How can we do so?

Just "solving" environments, as with arithmetic, may be impossible or computationally intractable. Systems of heuristics or shard economies also [wouldn't be up to the task](https://www.lesswrong.com/posts/rauMEna2ddf26BqiE/alignment-allows-nonrobust-decision-influences-and-doesn-t?commentId=DTF4fkK4v2yfRzPGC) — whatever proxy goal they're optimizing, there'd be at least one environment where it decouples from $R$.

It seems almost tautologically true, here, that the only way to keep an agent pointed at $R$ given this setup is to *explicitly point it at *$R$. Nothing else would do!

Thus, our optimization algorithm would *necessarily* find an $R$-pursuer, if it optimizes an agent for good performance across a sufficiently diverse (set of) environment(s).

**Point 2.** But why would that agent be shaped to pursue *only* $R$, and so strongly that it'll destroy everything else?

This, more or less, also has to do with environment diversity, plus some instrumental convergence.

As the optimization algorithm is shaping our agent, the agent will be placed in environments where it has preciously few resources, or a low probability of scoring well at $R$ (= high probability of receiving a strong update/correction after this episode ends).

Without knowing when such a circumstance would arise, how can we prepare our agent for this?

We can make it optimize for $R$ *strongly*, as strongly as it can, in fact. Acquire as much resources as possible, spend them on nothing but $R$-pursuit, minimize uncertainty of scoring well at $R$, and so on.

Every goal that *isn't* $R$ would distract from $R$-pursuit, and therefore lead to failure at some point, and so our optimization algorithm would eventually update such goals away; with update-strength proportional to how distracting a goal is.

Every missed opportunity to grab resources that can be used for $R$-pursuit, or a failure to properly optimize a plan for $R$-pursuit, would eventually lead to scoring bad at $R$. And so our optimization algorithm would instill a drive to *take* all such opportunities.

Thus, any greedy optimization algorithm would convergently shape its agent to not only pursue $R$, but to *maximize* for $R$'s pursuit — at the expense of everything else.

* * *

What should we take away from this? What should we *not* take away from this?

*   I should probably clarify that I'm not arguing that inner alignment isn't a problem, here. Aligning a wrapper-mind to a given goal is a very difficult task, and one I expect "blind" algorithms like the SGD to [fail horribly at](https://www.lesswrong.com/posts/ThtZrHooK7En9mcZr/greed-is-the-root-of-this-evil).
*   I'm not saying that the shard theory is incorrect — as I'd said, I think shard systems are very much a real developmental milestone of agents.

But I do think that we should very strongly expect the SGD to move its agents *in the direction of* $R$-optimizing wrapper-minds. Said "movement" would be [very](https://www.lesswrong.com/posts/ThtZrHooK7En9mcZr/greed-is-the-root-of-this-evil) [complex](https://www.lesswrong.com/posts/kmpNkeqEGvFue7AvA/value-formation-an-overarching-model), a nuanced path-dependent process that might lead to surprising end-points, or (as with humans) might terminate at a halfway point. But it'd still be movement in that direction!

And note the fundamental reasons behind this. It isn't because wrapper-mind behavior is convergent for any intelligent entity. Rather, it's a straightforward consequence of every known process for *generating* intelligent entities — the paradigm of local updates according to some outer function. Greedy optimization processes essentially search for mind-designs that would pre-empt any update the greedy optimization process would've made to them, so these minds come to incorporate the update rule and act in a way that'd merit a minimal update. *That*'s why. (In a way, it's because greedy optimization processes are *themselves* goal-obsessed wrappers.)

We wouldn't get *clean* wrapper-minds out of all of this, no. But they, and concerns related to them, still merit central attention.

[^eqzoggi740i]: Plus some more fundamental objections to utility-maximization as a framework, on which I haven't properly updated on yet, but which (I strongly expect) do not contradict the point I want to make in this post. 

[^gaicze5g0a4]: That is, we would shape the agent such that it doesn't require a strong update after ending up in one of these situations.