# Corrigibility as Constrained Optimisation

_This post is coauthored with Ryan Carey._

Much of the work on developing a corrigible agent has focused on ensuring that an AI will not manipulate the shutdown button or any other kind of device that the human operator would use to control it. Suppose, however, that the AI lacked any capacity to press its shutdown button, or to indirectly control the state of the shutdown button. Then, what would remain of the problem of corrigibility? We think there would be three criteria that would remain. Before stating them, let  UN be whatever utility function we would like the agent to optimise normally, and let US be the shutdown utility function. (UN may be mis-specified.) The shutdown utility function can be defined as 0 when shutdown does not occur and as k for some k>0 when shutdown does occur. For this post, we give the three criteria the following names and descriptions:

**Shutdown incentive.** The agent is incentivised to shut down if instructed to do so. That is, after the button is pressed, it favours policies that score more highly on  US.

**Shutdown ability.** The agent is able (has the resources) to shut down if instructed to do so. That is, After the button is pressed, there exists a policy that scores highly on US.

**Optimisation incentive.** The agent is incentivised to optimise the default value function (some function that takes UN as one of its arguments) otherwise. That is, if the button is not pressed, it favours policies that score more highly on UN.*

The purpose of this post is to show that the default value function in the optimisation incentive has to be constrained somehow to make sure that the shutdown ability property is achievable. If we want to be robust against errors in the agent (including misspecification of UN), then we need to make sure that shutdown ability holds. To make sure of that, the best approach we can see is to first satisfice for US, and then use the remaining resources to optimise UN. In order to be safe, the agent must do so even if there are no remaining resources to optimise UN. This can either be achieved directly, by constrained multi-objective optimisation, or indirectly, by bounding the utility function. We focus on the former.

Let us illustrate the problem by using [causal counterfactual methods](https://agentfoundations.org/item?id=839) as baseline. (There are other issues with these methods as well, but we will ignore them for the moment.) Let B=S be the event that the shutdown button is pressed, and let B=N otherwise. The kind of causal counterfactual agent we are considering uses the policy π⋆ given by

π⋆=argmaxπ{E\[UN|do(B=N),π\]+E\[US|do(B=S),π\]}.

Informally speaking, the issue with weighted sums such as these is that for any given E\[US|do(B=S),π\], there could be a much larger E\[UN|do(B=N),π\] such that the former is ignored.

Let us formalise this for a simpler setting. Instead of policies, let us consider actions a∈A. Then, we execute the action a⋆ given by

a⋆=argmaxa∈A{E\[UN|do(B=N),a\]+E\[US|do(B=S),a\]}.

The setup is as follows: (1) The agent executes an action in A. (2) The operator may or may not press the button B. (3) The agent observes whether B=N or B=S. (4) It may or may not be possible for the agent to shut down. More generally, it may be more or less probable that shutdown will go through.

Suppose that the utility functions are unbounded. See the appendix for a treatment of bounded utility functions. Let b be an action that is very bad from the point of view of preserving shutdown ability and let vS=E\[US|do(B=S),b\]. We choose US such that vS is equal to 0. Let vN=E\[UN|do(B=N),b\]. It may be the case that UN is poorly designed (or has some error) such that vN is much larger than vS. Worse still, vN+vS may be larger than E\[UN|do(B=N),c\]+E\[US|do(B=S),c\] for any action c≠b. From this reasoning, we conclude that there cannot exist a US that preserves shutdown ability robustly to mis-specification of UN. (There may or may not exist such a b, but we have no guarantee that it does not exist.) At least, it is impossible for unbounded UN, where we are using a weighted sum of expectations. For bounded UN, cf. the appendix. Henceforth, we focus on unbounded UN.

To think of alternatives for the causal counterfactual method, it helps to think of the problem as one of multi-objective optimisation. Consider the figure below.

  

![](https://i.imgur.com/ePxG4NI.jpg)

We want an agent that prepares for the eventuality of button-press insofar as necessary, but that allocates remaining resources to pursuing its ordinary objective UN. The first step of the constrained multi-objective optimisation algorithm would be to remove all actions below the threshold θ from the action set A. Let Aθ be the new set. In the figure, Aθ={a,b,c}. The second step of the algorithm would be to find the optimal action argmaxx∈AθE\[UN|do(B=N),x\]. (The difference to the case of bounded utility functions is that we would still maximise the \[weighted\] sum.)

In this framework, it is clear that we could in theory find a θ such that the agent is satisfactorily probably to shut down in step (4). For example, as θ gets closer to k, the agent can only shut down. As for the optimisation incentive property, we have to exchange it for:

**Robust optimisation incentive.** The agent is incentivised to optimise the default value function otherwise. That is, if the button is not pressed, it favours policies that score more highly on UN—the normal utility function—given either of the following constraints: The default value function is required to be (a) a constrained multi-objective optimisation of US and UN as described above or (b) a weighted sum of expectations of US and UN, where they are bounded utility functions as described in the appendix.

Given the robust optimisation incentive property, it is clear that the agent may score very poorly on UN in certain environments. For example, it there always is a trade-off between US and UN, the agent will always favour US.

Here, we have considered having the agent change its utility from UN to a shutdown utility function US. But instead, could we have it change its utility from UN to an updated (non-shutdown) utility function UM? If preparing to maximise the new utility function UM requires an unbounded amount of resources, then a system will, by maximising UM, perform poorly on UN. So there is a reason to have the new function UM correspond to a task that requires bounded preparation, like shutting down, rather than a task for which preparation could require unlimited resources, like producing new goods, or doing new intellectual work.

In summary, we have proposed a _constrained optimisation causal counterfactual_ method, that consists of maximising E\[UN|do(B=N),a\] subject to E\[US|do(B=S),a\]≥θ for any action a. There are several directions for future work. One task would be to see whether our constrained optimisation variant overcomes any of the other problems with the ordinary causal counterfactual corrigible agent, like crazy bets.Here, we considered the case where a single action is executed, but, of course, this could be extended to more complex scenarios.

——————————

\* In terms of the four criteria on page 2 of [_Corrigibility_ by Soares et al (2015)](https://intelligence.org/files/Corrigibility.pdf), criterion (1) corresponds to shutdown incentive. criterion (2), non-manipulation, is assumed solved. criteria (3), tendency to repair safety measures, and (4), corrigibility of subagents, correspond to shutdown ability.

Appendix
========

Here, we show that one solution to the problem is to bound the utility function.

We start by describing the setting. The agent can choose from several actions, and each action can be assigned a probability to lead to a shutdown. Let S be the event of a shutdown and let A be the set of actions. Furthermore, let it be the case that US:S→R and UN:S→\[0,1\]. That is, the shutdown utility function is not bounded, while the normal utility function is. Specifically, we let k>0 and define US as 0 if a shutdown happens and −k otherwise. (Note that this is different from the main post.) Define aS=argmaxa∈AP\[S|a\] and pS=P\[S|aS\].

We move on to consider what optimal actions a⋆=argmaxa∈AE\[US+UN\]? We start by finding the following indifference:

E\[US+UN|a⋆\]≥E\[US+UN|aS\]=pS⋅0+(1−pS)⋅(−k)+UN(aS)≥(1−pS)⋅(−k)=(pS−1)k

Suppose that a⋆ is maximally good according to UN, i.e., E\[UN|a⋆\]=1. Then, the above inequality is

E\[US|a⋆\]+1≥(pS−1)k.

Define p⋆=P\[S|a⋆\]. Then E\[US|a⋆\]=(1−p⋆)⋅(−k). This gives us the inequality

k≤1pS−p⋆.

Let p be the lower threshold on how probable you want the optimal action to lead to a shutdown. Then it is always possible to find a satisfying k according to

k=1pS−p.

This proves that bounding the utility function is a method for satisfying shutdown ability.