# Cake, or death!

Here we'll look at the famous cake or death problem teasered in the [Value loading/learning](/r/discussion/lw/f32/value_loading/) post.

Imagine you have an agent that is uncertain about its values and designed to "learn" proper values. A formula for this process is that the agent must pick an action _a_ equal to:

*   **argmax**~a∈A~ Σ~w∈W~ p(w|e,a) Σ~u∈U~ u(w)p(C(u)|w)

Let's decompose this a little, shall we? _A_ is the set of actions, so argmax of _a_ in _A_ simply means that we are looking for an action _a_ that maximises the rest of the expression. _W_ is the set of all possible worlds, and _e_ is the evidence that the agent has seen before. Hence _p(w|e,a)_ is the probability of existing in a particular world, given that the agent has seen evidence _e_ and will do action _a_. This is summed over each possible world in _W_.

And what value do we sum over in each world? Σ~u∈U~ u(w)p(C(u)|w). Here _U_ is the set of (normalised) utility functions the agent is considering. In value loading, we don't program the agent with the correct utility function from the beginning; instead we imbue it with some sort of learning algorithm (generally with feedback) so that it can deduce for itself the correct utility function. The expression _p(C(u)|w)_ expresses the probability that the utility _u_ is correct in the world _w_. For instance, it might cover statements "it's 99% certain that 'murder is bad' is the correct morality, given that I live in a world where every programmer I ask tells me that murder is bad".

The _C_ term is the correctness of the utility function, given whatever system of value learning we're using (note that some moral realists would insist that we don't need a _C_, that _p(u|w)_ makes sense directly, that we can deduce ought from is). All the subtlety of the value learning is encoded in the various _p(C(u)|w)_: this determines how the agent learns moral values.

So the whole formula can be described as:

*   For each possible world and each possible utility function, figure out the utility of that world. Weigh that by the probability that that utility is correct is that world, and by the probability of that world. Then choose the action that maximises the weighted sum of this across all utility functions and worlds.

Naive cake or death
-------------------

![](http://images.lesswrong.com/t3_f3v_4.png?v=5a8b98c565cc4eeade88fb637d91eee9)

In the initial formulation of value loading, _p(C(u)|w)_ (probability of the correctness of _u_ in world _w_) was replaced with _p(C(u)|e,a)_ (probability of the correctness of _u_ given the evidence _e_ and the action _a_). A seemingly insignificant difference; yet it lead to the first cake or death problem.

In [cake or death](http://www.youtube.com/watch?v=rZVjKlBCvhg), the agent is equally unsure between utility _u~1~_ and utility _u~2~_; hence _p(C(u~1~)|e)=p(C(u~2~)|e)=0.5_. The utility _u~1~_ gives the agent 1 utiliton every time it gives someone a cake; _u~2~_ gives the agent 1 utiliton every time it gives someone death. The agent can produce 1 cake or three deaths. It can also, for free, ask its programmer whether cake or death is better, before producing anything; this gives rise to three different worlds:

*   _w~1~_: the agent asks, and the programmer says cake.
*   _w~2~_: the agent asks, and the programmer says death.
*   _w~3~_: the agent doesn't ask.

We assume the programmer's answer completely clears up the issue. And thus after asking, the agent will do whatever the programmer recommended (and it knows this now). Since it doesn't know what the programmer will say, it has _p(C(u~1~)|e,"ask") = __p(C(u~2~)|e,"ask") = 0.5_. This gives an expected utility calculation:

*   p(w~1~|e,"ask")(p(C(u~1~)|e,"ask")u~1~(w~1~) + p(C(u~2~)|e,"ask")u~2~(w~1~)) + p(w~2~|e,"ask")(p(C(u~1~)|e,"ask")u~1~(w~2~) + p(C(u~2~)|e,"ask")u~2~(w~2~)) = 0.5\*0.5\*(u~1~(w~1~)+u~2~(w~1~)+u~1~(w~2~)+u~2~(w~2~)) = 0.25(1+0+0+3) = **1**.

If the agent doesn't ask, it will subsequently produce three deaths (as this generates 1.5 expected utilitons, while producing one cake will generate only 0.5 expected utilitons). From its current (_0.5u~1~+0.5u~2~_) perspective, this is worth 1.5 expected utilitons: so +1.5 is the expected utility gain from not asking.

Hence the agent gains from not asking.

What's going wrong here? The problem is that the agent is using its current utility function to estimate the value of its future action. At the moment, it values death or cake both at 0.5. If it asks, it runs the risk that the programmer will say "cake" and it will be forced to build cake. After hearing the answer, it will value that cake at 1, but currently it values it only at 0.5. Similarly, if the programmer says death, it will produce three deaths - which it _will_ value at 3, but _currently_ values at 1.5. Since each of these options are equally likely, it gets only (0.5+1.5)/2 = 1 utilitons from asking.

**In summary**: the naive cake-or-death problem emerges for a value learning agent when it expects its utility to change, but uses its current utility to rank its future actions.

Sophisticated cake or death: I know what you're going to say
------------------------------------------------------------

![](http://images.lesswrong.com/t3_f3v_5.png)

Using _p(C(u)|w)_ rather than _p(C(u)|e,a)_ does away with the naive cake or death problem.

Instead of having _p(C(u~1~)|e,"ask") = __p(C(u~2~)|e,"ask") = 0.5_ in all possible worlds, we have _p(C(u~1~)|w~1~)=p(C(u~2~)|w~2~) = 1_ and _p(C(u~1~)|w~2~)=p(C(u~2~)|w~1~) = 0_. Hence if it asks and gets "cake" as an answer, it will know it is in world _w~1~_, and make a cake that it will value at 1 - crucially, it currently also values cake at 1, _given that it is in world w~1~_. Similarly, it values death at 1, given that it is in world _w~2~_. So its expected utility from asking is (1+3)/2=2. This is more than the utility of not asking, and so it will ask.

The agent remains vulnerable to a more sophisticated cake-or-death problem, though. Suppose it is still uncertain between cake or death in its utility function, but it has figured out that if asked, the programmer will answer "cake". Thereafter, it will make cake. In this situation, it will only derive +1 from asking, whereas it still derives +1.5 from not asking (and doing three deaths). So it won't ask - as long as it does this, it remains in _w~3~_.

What happened here? Well, this is a badly designed _p(C(u)|w)_. It seems that it's credence in various utility function changes when it gets answers from programmers, but not from knowing what those answers are. And so therefore it'll only ask certain questions and not others (and do a lot of other nasty things), all to reach a utility function that it's easier for it to fulfil.

What we actually want, is that the agent be unable to predictably change its utility in any direction by any action (or lack of action). We want a _p(C(u)|w)_ designed so that for all actions _a_ and all putative utility functions _u_:

*   Expectation(p(C(u) | a) = p(C(u)).

So there is a "conservation of expected correctness"; if we have this, the sophisticated cake-or-death argument has no traction. This is equivalent with saying that the prior _P(C(u))_ is well defined, irrespective of any agent action.

**In summary**: the sophisticated cake-or-death problem emerges for a value learning agent when it expects its utility to change predictably in certain directions dependent on its own behaviour.