# "How conservative" should the partial maximisers be?

Due to the problem of building a [strong $V$-enhancer when we want a $U$-enhancer](https://www.lesswrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy) - and the [great difficulty in defining $U$](https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes), the utility we truly want to maximise - many people have suggested reducing the $V$-increasing focus of the AI. The idea is that, as long as the AI doesn't devote too much [optimisation power](https://www.lesswrong.com/posts/iJNK7rE5jphMSJJCa/thoughts-and-problems-with-eliezer-s-measure-of-optimization) to $V$, the $V$ and $U$ [will stay connected with each other](https://www.lesswrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy#Extremal_Goodhart), and hence a moderate increase in $V$ will in fact lead to a moderate increase in $U$.

This has lead to interest in such things as [satisficers](https://en.wikipedia.org/wiki/Satisficing) and [low-impact AIs](https://arxiv.org/abs/1705.10720), [both](https://www.lesswrong.com/posts/2qCxguXuZERZNKcNi/satisficers-want-to-become-maximisers) of [which](https://www.lesswrong.com/posts/mdQEraEZQLg7jtozn/subagents-and-impact-measures-full-and-fully-illustrated) have their problems. Those try and put an *absolute* limit on how much $V$ is optimised. The AI is not supposed to optimise $V$ above a certain limit (satisficer) or if optimising it changes too much about the world or the power of other agents (low-impact).

Another approach is to put a *relative* limit on how much an AI can push a utility function. For example, [quantilizers](https://intelligence.org/files/QuantilizersSaferAlternative.pdf) will choose randomly among the top $0< q \leq 1$ proportion of actions/policies, rather than picking the top action/policy. Then there is the approach of using [pessimism](https://www.lesswrong.com/posts/RzAmPDNciirWKdtc7/pessimism-about-unknown-unknowns-inspires-conservatism) to [make the AI more conservative](https://mkcohen-hosted-files.s3-us-west-1.amazonaws.com/Pessimism_alignmentforum.pdf). This pessimism is defined by a parametre $\beta\in(0,1)$, with $\beta\to 1$ being very pessimistic.

## Intermediate value uncertainty

The behaviours of $q$ and $\beta$ are pretty clear around the extremes. As $\beta$ and $q$ tend to $0$, the agent will behave like a $V$-maximiser. As they tend to $1$, the agent will behave randomly ($q$) or totally conservatively ($\beta$).

Thus, we expect that moving away from the extremes will improve the true $U$-performance, **and** that the conservative, $1$ end, will be less disastrous than the $V$-maximising, $0$ end (though we only know that second fact, [because of implicit assumptions we have on $U$ and $V$](https://www.lesswrong.com/posts/uL74oQv5PsnotGzt7/all-i-know-is-goodhart)).

The problem is in the middle, where the behaviour is unknown (and, since we lack a full formulation of $U$, generically unknowable). There is no principled way of setting the $q$ or the $\beta$. Consider, for example, this plot of $q$ versus $U$:

![](https://www.dropbox.com/s/18712y2gtbwwlg4/quant1.png?raw=1)

Here, the ideal $q$ is around $0.45$, but the critical thing is to keep $q$ above $0.3$: that's the point at which it falls precipitously.

Contrast now with this one:

![](https://www.dropbox.com/s/m9ixjd325klxu1q/quant2.png?raw=1)

Here, any value of $q$ above $0.2$ is essentially the same, and $q$ can be lowered as low as $0.05$ before there are any problems.

So, in the first case, we need $q$ above $0.3$, and, in the second, below $0.2$. And, moreover, it might be that the first situation appears in one world and the second in another, and both worlds are currently possible. So there's no consistent good value of $q$ we can set (and in the general case, the curve might be multi-modal, with many peaks). And note that we don't know any of these graphs (since we can't define $U$ fully). So we don't know what values to set $q$ at, have little practical guidance on what to do, but expect that some values will be disastrous.

The conservatism approach has similar problems: $\beta$ is even harder to interpret than $q$, we don't have any guidance on how to set it, and the ideal $\beta$ may vary considerably depending on the circumstance. For example, what would we want our AI to do when it finds an unexpected red button connected to nuclear weapons?

![](https://www.dropbox.com/s/3rmp89gc4ark2ne/red_button_robot.png?raw=1)

Well, that depends on whether the button starts a nuclear launch - or if it cancels one.

A future post will explore how to resolve this issue, and how to choose the conservatism parameter in a suitable way.