# Why you must maximize expected utility

_This post explains von Neumann-Morgenstern (VNM) axioms for decision theory_, _and what follows from them: that if you have a consistent direction in which you are trying to steer the future, you must be an expected utility maximizer. I'm writing this post in preparation for a sequence on [updateless](/lw/4g6/updateless_anthropics/) [anthropics](/lw/175/torture_vs_dust_vs_the_presumptuous_philosopher/), but I'm hoping that it will also be independently useful._

The theorems of decision theory say that if you follow certain axioms, then your behavior is described by a utility function. (If you don't know what that means, I'll explain below.) So you should have a utility function! Except, why should you want to follow these axioms in the first place?

A couple of years ago, Eliezer explained how violating one of them can [turn you into a money pump](/lw/my/the_allais_paradox/) — how, at time 11:59, you will _want_ to pay a penny to get option B instead of option A, and then at 12:01, you will _want_ to pay a penny to switch back. Either that, or the game will have ended and the option won't have made a difference.

When I read that post, I was suitably impressed, but not completely convinced: I would certainly not want to behave one way if behaving differently _always_ gave better results. But couldn't you avoid the problem by violating the axiom only in situations where it doesn't give anyone an opportunity to money-pump you? I'm not saying that would be _elegant_, but is there a reason it would be _irrational_?

It took me a while, but I have since come around to the view that you really must have a utility function, and really must behave in a way that maximizes the expectation of this function, [on pain of stupidity](/lw/oj/probability_is_in_the_mind/) (or at least that there are strong arguments in this direction). But I don't know any source that comes close to explaining the reason, the way I see it; hence, this post.

I'll use the [von Neumann-Morgenstern axioms](/lw/244/vnm_expected_utility_theory_uses_abuses_and/), which assume probability theory as a foundation (unlike the [Savage axioms](/lw/5te/a_summary_of_savages_foundations_for_probability/), which actually _imply_ that anyone following them has not only a utility function but also a probability distribution). I will assume that you already accept Bayesianism.

*

_Epistemic_ rationality is about figuring out what's true; _instrumental_ rationality is about [steering the future where you want it to go](/lw/crd/only_say_rational_when_you_cant_eliminate_the_word/). The way I see it, the axioms of decision theory tell you how to have a consistent _direction_ in which you are trying to steer the future. If my choice at 12:01 depends on whether at 11:59 I had a chance to decide differently, then perhaps I won't ever be money-pumped; but if I want to save as many human lives as possible, and I must decide between different plans that have different probabilities of saving different numbers of people, then it starts to at least seem _doubtful_ that which plan is better at 12:01 could _genuinely_ depend on my opportunity to choose at 11:59.

So how do we formalize the notion of a coherent direction in which you can steer the future?

*

Setting the stage  

--------------------

Decision theory asks what you would do if faced with choices between different sets of options, and then places restrictions on how you can act in one situation, depending on how you would act in others. This is another thing that has always bothered me: If we are talking about choices between different lotteries with small prizes, it makes some sense that we could invite you to the lab and run ten sessions with different choices, and you should probably act consistently across them. But if we're interested in the big questions, like how to save the world, then you're not going to face a series of independent, analogous scenarios. So what is the _content_ of asking what you would do if you faced a set of choices different from the one you actually face?

The real point is that you have bounded computational resources, and you can't _actually_ visualize the exact set of choices you might face in the future. A perfect Bayesian rationalist could just figure out what they _would_ do in any conceivable situation and write it down in a giant lookup table, which means that they only face a single one-time choice between different possible tables. But _you_ can't do that, and so you need to figure out general principles to follow. [A perfect Bayesian is like a Carnot engine](/lw/mt/beautiful_probability/) — it's what a theoretically perfect engine _would_ look like, so even though you can at best approximate it, it still has something to teach you about how to build a real engine.

But decision theory is _about_ what a perfect Bayesian would do, and it's annoying to have our practical concerns intrude into our ideal picture like that. So let's give our story some local color and say that _you_ aren't a perfect Bayesian, but you have a genie — that is, a powerful optimization process — that is, an AI, which _is_. (That, too, is physically impossible: AIs, like humans, can only approximate perfect Bayesianism. But we _are_ still idealizing.) Your _genie_ is able to comprehend the set of possible giant lookup tables it must choose between; _you_ must write down a formula, to be evaluated by the genie, that chooses the best table from this set, given the available information. (An unmodified human won't _actually_ be able to write down an exact formula describing their preferences, but we might be able to write down one for a paperclip maximizer.)

The first constraint decision theory places on your formula is that it must order all options your genie _might_ have to choose between from best to worst (though you might be indifferent between some of them), and then given any particular set of feasible options, it must choose the one that is least bad. In particular, if you prefer option A when options A and B are available, then you can't prefer option B when options A, B and C are available.

[**Meditation:**](http://wiki.lesswrong.com/wiki/Meditation) _Alice is trying to decide how large a bonus each member of her team should get this year. She has just decided on giving Bob the same, already large, bonus as last year when she receives an e-mail from the head of a different division, asking her if she can recommend anyone for a new project he is setting up. Alice immediately realizes that Bob would love to be on that project, and would fit the bill exactly. But she_ needs _Bob on the contract he's currently working on; losing him would be a pretty bad blow for her team._

_Alice decides there is no way that she can recommend Bob for the new project. But she still feels bad about it, and she decides to make up for it by giving Bob a larger bonus. On reflection, she finds that she genuinely feels that this is the_ right _thing to do, simply because she_ could _have recommended him but didn't. Does that mean that Alice's preferences are irrational? Or that something is wrong with decision theory?_

**Meditation:** _One kind of answer to the above and to many other criticisms of decision theory goes like this: Alice's decision isn't between giving Bob a larger bonus or not, it's between (give Bob a larger bonus unconditionally), (give Bob the same bonus unconditionally), (only give Bob a larger bonus if I could have recommended him), and so on. But if_ that _sort of thing is allowed, is there_ any _way left in which decision theory constrains Alice's behavior? If not, what good is it to Alice in figuring out what she should do?_

...  
...  
...

*

Outcomes  

-----------

My short answer is that Alice can care about anything she damn well likes. But there are a lot of things that she _doesn't_ care about, and decision theory has something to say about _those_.

In fact, deciding that some kinds of preferences should be outlawed as irrational can be dangerous: you might think that nobody in their right mind should ever care about the detailed planning algorithms their AI uses, as long as they work. But how certain are you that it's wrong to care about whether the AI has planned out your whole life in advance, in detail? (Worse: Depending on how strictly you interpret it, this injunction might even rule out [not wanting the AI to run conscious simulations of people](/lw/x4/nonperson_predicates).)

But nevertheless, I believe the "anything she damn well likes" needs to be qualified. Imagine that Alice and Carol both have an AI, and fortuitously, both AIs have been programmed with the same preferences and the same Bayesian prior (and they talk, so they also have the same posterior, because Bayesians cannot agree to disagree). But Alice's AI has taken over the stock markets, while Carol's AI has seized the world's nuclear arsenals (and is protecting them well). So Alice's AI not only doesn't want to blow up Earth, it couldn't do so _even if it wanted to_; it couldn't even bribe Carol's AI, because Carol's AI really doesn't want the Earth blown up either. And so, if it makes a difference to the AIs' preference function whether they _could_ blow up Earth if they wanted to, they have a conflict of interest.

The moral of this story is not simply that it would be _sad_ if two AIs came into conflict even though they have the same preferences. The point is that we're asking what it means to have a consistent direction in which you are trying to steer the future, and it doesn't look like our AIs are on the same bearing. Surely, a direction for steering the world should only depend on features of the _world_, not on additional information about which agent is at the rudder.

You _can_ want to not have your life planned out by an AI. But I think you should have to state your wish as a property of the world: you want _all_ AIs to refrain from doing so, not just "whatever AI happens to be executing this". And Alice can want Bob to get a larger bonus if the company could have assigned him to the new project and decided not to, but she must figure out whether _this_ is the correct way to translate her moral intuitions into preferences over properties of the world.

*

You may care about any feature of the world, but you don't in fact care about most of them. For example, there are many ways the atoms in the sun could be arranged that all add up to the same thing as far as you are concerned, and you don't have [_terminal_ preferences](/lw/l4/terminal_values_and_instrumental_values/) about which of these will be the actual one tomorrow. And though you might care about _some_ properties of the algorithms your AI is running, mostly they _really_ do not matter.

Let's define a function that takes a complete description of the world — past, present and future — and returns a data structure containing all information about the world that matters to your terminal values, and _only_ that information. (Our imaginary perfect Bayesian doesn't know exactly which way the world will turn out, but it can work with "possible worlds", complete descriptions of ways the world _may_ turn out.) We'll call this data structure an "outcome", and we require you to be indifferent between any two courses of action that will always produce the same outcome. Of course, any course of action is something that your AI would be executing in the actual world, and you are certainly allowed to care about the difference — but then the two courses of action do not lead to the same "outcome"!^1^

With this definition, I think it is pretty reasonable to say that in order to have a consistent direction in which you want to steer the world, you must be able to order these outcomes from best to worst, and always want to pick the least bad you can get.

*

Preference relations  

-----------------------

That won't be _sufficient_, though. Our genie doesn't _know_ what outcome each action will produce, it only has probabilistic information about that, and that's a complication we very much do _not_ want to idealize away (because we're trying to figure out the right way to _deal_ with it). And so our decision theory amends the earlier requirement: You must not only be indifferent between actions that always produce the same outcome, but also between all actions that only yield _the same probability distribution_ over outcomes.

This is not at all a mild assumption, though it's usually built so deeply into the definitions that it's not even called an "axiom". But we've assumed that all features of the world you care about are already encoded in the outcomes, so it does seem to me that the only reason left why you might prefer one action over another is that it gives you a better trade-off in terms of what outcomes it makes more or less likely; and I've assumed that you're already a Bayesian, so you agree that _how_ likely it makes an outcome is correctly represented by the probability of that outcome, given the action. So it certainly _seems_ that the probability distribution over outcomes should give you all the information about an action that you could _possibly_ care about. And that you should be able to order these probability distributions from best to worst, and all that.

Formally, we represent a direction for steering the world as a set ![](http://www.codecogs.com/png.latex?\mathcal{O}) of possible outcomes and a binary relation ![](http://www.codecogs.com/png.latex?\succcurlyeq) on the probability distributions over ![](http://www.codecogs.com/png.latex?\mathcal{O}) (with ![](http://www.codecogs.com/png.latex?x\succcurlyeq y) is interpreted as "![](http://www.codecogs.com/png.latex?x) is at least as good as ![](http://www.codecogs.com/png.latex?y)") which is a [total preorder](http://en.wikipedia.org/wiki/Total_preorder); that is, for all ![](http://www.codecogs.com/png.latex?x), ![](http://www.codecogs.com/png.latex?y) and ![](http://www.codecogs.com/png.latex?z):

*   If ![](http://www.codecogs.com/png.latex?x\succcurlyeq y) and ![](http://www.codecogs.com/png.latex?y\succcurlyeq z), then ![](http://www.codecogs.com/png.latex?x\succcurlyeq z)  (that is, ![](http://www.codecogs.com/png.latex?\succcurlyeq) is _transitive_); and
*   We have either ![](http://www.codecogs.com/png.latex?x\succcurlyeq y) or ![](http://www.codecogs.com/png.latex?y\succcurlyeq x) or both  (that is, ![](http://www.codecogs.com/png.latex?\succcurlyeq) is _total_).

In this post, I'll assume that ![](http://www.codecogs.com/png.latex?\mathcal{O}) is finite. We write ![](http://www.codecogs.com/png.latex?x\sim y) (for "I'm indifferent between ![](http://www.codecogs.com/png.latex?x) and ![](http://www.codecogs.com/png.latex?y)") when both ![](http://www.codecogs.com/png.latex?x\succcurlyeq y) and ![](http://www.codecogs.com/png.latex?y\succcurlyeq x), and we write ![](http://www.codecogs.com/png.latex?x\succ y) ("![](http://www.codecogs.com/png.latex?x) is strictly better than ![](http://www.codecogs.com/png.latex?y)") when ![](http://www.codecogs.com/png.latex?x\succcurlyeq y) but _not_ ![](http://www.codecogs.com/png.latex?y\succcurlyeq x). Our genie will compute the set of all actions it could possibly take, and the probability distribution over possible outcomes that (according to the genie's Bayesian posterior) each of these actions leads to, and then it will choose to act in a way that maximizes ![](http://www.codecogs.com/png.latex?\succcurlyeq). I'll also assume that the set of possible actions will always be finite, so there is always at least one optimal action.

**Meditation:** _[Omega](http://wiki.lesswrong.com/wiki/Omega) is in the neighbourhood and invites you to participate in one of its little games. Next Saturday, it plans to flip a fair coin; would you please indicate on the attached form whether you would like to bet that this coin will fall heads, or tails? If you correctly bet heads, you will win $10,000; if you correctly bet tails, you'll win $100. If you bet wrongly, you will still receive $1 for your participation._

_We'll assume that you prefer a 50% chance of $10,000 and a 50% chance of $1 to a 50% chance of $100 and a 50% chance of $1. Thus, our theory would say that you should bet heads. But there is a twist: Given recent galactopolitical events, you estimate a 3% chance that after posting its letter, Omega has been called away on urgent business. In this case, the game will be cancelled and you won't get any money, though as a consolation, Omega will probably send you some book from its rare SF collection when it returns (market value: approximately $55–$70). Our theory so far tells you nothing about how you should bet in this case, but does Rationality have anything to say about it?  
_

...  
...  
...

*

The Axiom of Independence  

----------------------------

So here's how I think about that problem: If you already _knew_ that Omega is still in the neighbourhood (but not which way the coin is going to fall), you would prefer to bet heads, and if you _knew_ it has been called away, you wouldn't care. (And what you bet has no influence on whether Omega has been called away.) So heads is either better or exactly the same; clearly, you should bet heads.

This type of reasoning is the content of the von Neumann-Morgenstern _Axiom of Independence_. Apparently, that's [the most controversial of the theory's axioms](/lw/1d5/expected_utility_without_the_independence_axiom/).

You're already a Bayesian, so you already accept that if you perform an experiment to determine whether someone is a witch, and the experiment can come out two ways, then if one of these outcomes is evidence that the person is a witch, [the other outcome must be evidence that they are _not_](/lw/ii/conservation_of_expected_evidence/). New information is allowed to make a hypothesis more likely, but not _predictably_ so; if _all_ ways the experiment could come out make the hypothesis more likely, then you should _already_ be finding it more likely than you do. The same thing is true even if only one result would make the hypothesis more likely, but the other would leave your probability estimate exactly unchanged.

The Axiom of Independence is equivalent to saying that if you're evaluating a possible course of action, and one experimental result would make it seem more attractive than it currently seems to you, while the other experimental result would at least make it seem no _less_ attractive, then you should _already_ be finding it more attractive than you do. This does _seem_ rather solid to me.

*

So what does this axiom say formally? _(Feel free to skip this section if you don't care.)_

Suppose that your genie is considering two possible actions ![](http://www.codecogs.com/png.latex?a_1) and ![](http://www.codecogs.com/png.latex?a_2) (bet heads or tails), and an event ![](http://www.codecogs.com/png.latex?E) (Omega is called away). Each action gives rise to a probability distribution over possible outcomes: E.g., ![](http://www.codecogs.com/png.latex?\mathbb{P}[i\mid a_1]) is the probability of outcome ![](http://www.codecogs.com/png.latex?i\in\mathcal{O}) if your genie chooses ![](http://www.codecogs.com/png.latex?a_1). But your genie can also compute a probability distribution _conditional on ![](http://www.codecogs.com/png.latex?E)_ , ![](http://www.codecogs.com/png.latex?\mathbb{P}[i\mid E,a_1]). Suppose that conditional on ![](http://www.codecogs.com/png.latex?E), it doesn't matter which action you pick: ![](http://www.codecogs.com/png.latex?\mathbb{P}[i\mid E,a_1] \,=\, \mathbb{P}[i\mid E,a_2]) for all ![](http://www.codecogs.com/png.latex?i\in\mathcal{O}). And finally, suppose that the probability of ![](http://www.codecogs.com/png.latex?E) doesn't depend on which action you pick: ![](http://www.codecogs.com/png.latex?%5Cmathbb{P}[E%5Cmid%20a_1]%5C,=%5C,%5Cmathbb{P}[E%5Cmid%20a_2]%5C,=:%5C,%20p), with ![](http://www.codecogs.com/png.latex?0%20%3C%20p%20%3C%201). The Axiom of Independence says that in this situation, you should prefer the distribution ![](http://www.codecogs.com/png.latex?\mathbb{P}[\;\cdot\mid a_1]) to the distribution ![](http://www.codecogs.com/png.latex?\mathbb{P}[\;\cdot\mid a_2]), and therefore prefer ![](http://www.codecogs.com/png.latex?a_1) to ![](http://www.codecogs.com/png.latex?a_2), if and only if you prefer the distribution ![](http://www.codecogs.com/png.latex?\mathbb{P}[\;\cdot\mid\neg E,a_1]) to the distribution ![](http://www.codecogs.com/png.latex?\mathbb{P}[\;\cdot\mid\neg E,a_2]).

Let's write ![](http://www.codecogs.com/png.latex?x) for the distribution ![](http://www.codecogs.com/png.latex?\mathbb{P}[\;\cdot\mid\neg E,a_1]), ![](http://www.codecogs.com/png.latex?y) for the distribution ![](http://www.codecogs.com/png.latex?\mathbb{P}[\;\cdot\mid\neg E,a_2]), and ![](http://www.codecogs.com/png.latex?z) for the distribution ![](http://www.codecogs.com/png.latex?\mathbb{P}[\;\cdot\mid E,a_1] \,=\, \mathbb{P}[\;\cdot\mid E,a_2]). (Formally, we think of these as vectors in ![](http://www.codecogs.com/png.latex?\mathbb{R}^{|\mathcal{O}|}): e.g., ![](http://www.codecogs.com/png.latex?z_i \,=\, \mathbb{P}[i\mid E,a_1]).) For all ![](http://www.codecogs.com/png.latex?i\in\mathcal{O}), we have

![](http://www.codecogs.com/png.latex?\mathbb{P}[i\mid a_1] \;=\; \mathbb{P}[\neg E\mid a_1]\cdot\mathbb{P}[i\mid\neg E,a_1] \;+\; \mathbb{P}[E\mid a_1]\cdot\mathbb{P}[i\mid E,a_1],)

so ![](http://www.codecogs.com/png.latex?\mathbb{P}[\;\cdot\mid a_1] = (1-p)x + pz), and similarly ![](http://www.codecogs.com/png.latex?\mathbb{P}[\;\cdot\mid a_2] \,=\, (1-p)y + pz). Thus, we can state the Axiom of Independence as follows:

*   ![](http://www.codecogs.com/png.latex?(1-p)x + pz \,\succcurlyeq\, (1-p)y + pz \;\iff\; x\succcurlyeq y).

We'll assume that you can't ever rule out the possibility that your AI might face this type of situation for any given ![](http://www.codecogs.com/png.latex?x), ![](http://www.codecogs.com/png.latex?y), ![](http://www.codecogs.com/png.latex?z), and ![](http://www.codecogs.com/png.latex?p), so we require that this condition hold for all probability distributions _![](http://www.codecogs.com/png.latex?x)_, _![](http://www.codecogs.com/png.latex?y)_ and _![](http://www.codecogs.com/png.latex?z)_, and for all ![](http://www.codecogs.com/png.latex?p) with ![](http://www.codecogs.com/png.latex?0<p<1).

*

Here's a [common criticism](/lw/244/vnm_expected_utility_theory_uses_abuses_and/) of Independence. Suppose a parent has two children, and one old car that they can give to one of these children. Can't they be indifferent between giving the car to their older child or their younger child, but strictly prefer throwing a coin? But let ![](http://www.codecogs.com/png.latex?x = z) mean that the younger child gets the gift, and ![](http://www.codecogs.com/png.latex?y) that the older child gets it, and ![](http://www.codecogs.com/png.latex?p = 1/2); then by Independence, if ![](http://www.codecogs.com/png.latex?x\sim y), then ![](http://www.codecogs.com/png.latex?\textstyle x \;=\; \frac12x + \frac12 z \;\sim\; \frac12y + \frac12 z), so it would seem that the parent can _not_ strictly prefer the coin throw.

In fairness, the people who find this criticism persuasive may not be Bayesians. But if _you_ think this is a good criticism: Do you think that the parent must be indifferent between throwing a coin and asking the children's crazy old kindergarten teacher which of them was better-behaved, as long as they assign 50% probability to either answer? Because if not, shouldn't you already have protested when we decided that decisions must only depend on the probabilities of different outcomes?

My own resolution is that this is another case of terminal values intruding where they don't belong. _All_ that is relevant to the parent's terminal values must _already_ be described in the outcome; the parent is allowed to prefer "I threw a coin and my younger child got the car" to "I decided that my younger child would get the car" or "I asked the kindergarten teacher and they thought my younger child was better-behaved", but if so, then these must already be different _outcomes_. The thing to remember is that it isn't a property of the _world_ that either child had a 50% probability of getting the car, and you can't steer the future in the direction of having this mythical property. It _is_ a property of the world that _the parent assigned a 50% probability_ to each child getting the car, and that _is_ a direction you can steer in — though the example with the kindergarten teacher shows that this is probably not quite the direction you actually wanted.

The preference relation is _only_ supposed to be about _trade-offs_ between probability distributions; if you're tempted to say that you want to steer the world towards one probability distribution or another, rather than one outcome or other, something has gone terribly wrong.

*

The Axiom of Continuity  

--------------------------

And… that's it. These are all the axioms that I'll ask you to accept in this post.

There is, however, one more axiom in the von Neumann-Morgenstern theory, the Axiom of Continuity. I do _not_ think this axiom is a necessary requirement on any coherent plan for steering the world; I think the best argument for it is that it doesn't make a practical difference whether you adopt it, so you might as well. But there is also a good argument to be made that if we're talking about anything _short_ of steering the entire future of humanity, your preferences _do_ in fact obey this axiom, and it makes things easier technically if we adopt it, so I'll do that at least for now.

Let's look at an example: If you prefer $50 in your pocket to $40, the axiom says that there must be _some_ small ![](http://www.codecogs.com/png.latex?\epsilon > 0) such that you prefer a probability of ![](http://www.codecogs.com/png.latex?1-\epsilon) of $50 and a probability of ![](http://www.codecogs.com/png.latex?\epsilon) of dying today to a certainty of $40. Some critics seem to see this as the ultimate _reductio ad absurdum_ for the VNM theory; they seem to think that no sane human would accept that deal.

Eliezer was surely not the first to [observe](/lw/n9/the_intuitions_behind_utilitarianism/) that this preference is exhibited each time someone drives an extra mile to save $10.

Continuity says that if you strictly prefer ![](http://www.codecogs.com/png.latex?x) to ![](http://www.codecogs.com/png.latex?y), then there is _no ![](http://www.codecogs.com/png.latex?z)_ so terrible that you wouldn't be willing to incur a small probability of it in order to (probably) get ![](http://www.codecogs.com/png.latex?x) rather than ![](http://www.codecogs.com/png.latex?y), and _no_ ![](http://www.codecogs.com/png.latex?z) so wonderful that you'd be willing to (probably) get ![](http://www.codecogs.com/png.latex?y) instead of ![](http://www.codecogs.com/png.latex?x) if this gives you some arbitrarily small probability of getting ![](http://www.codecogs.com/png.latex?z). Formally, for all ![](http://www.codecogs.com/png.latex?x), ![](http://www.codecogs.com/png.latex?y) and ![](http://www.codecogs.com/png.latex?z),

*   If ![](http://www.codecogs.com/png.latex?x\succ y), then there is an ![](http://www.codecogs.com/png.latex?\epsilon > 0) such that ![](http://www.codecogs.com/png.latex?(1-\epsilon)x + \epsilon z \;\succ\; y) and ![](http://www.codecogs.com/png.latex?x \;\succ\; (1-\epsilon)y + \epsilon z).

I think if we're talking about everyday life, we can pretty much rule out that there are things so terrible that for _arbitrarily_ small ![](http://www.codecogs.com/png.latex?\epsilon), you'd be willing to die with probability ![](http://www.codecogs.com/png.latex?1-\epsilon) to avoid a probability of ![](http://www.codecogs.com/png.latex?\epsilon) of the terrible thing. And if you feel that [it's not worth the expense](/lw/n9/the_intuitions_behind_utilitarianism/) to call a doctor every time you sneeze, you're willing to incur a _slightly_ higher probability of death in order to save some mere money. And it seems unlikely that there is _no_ ![](http://www.codecogs.com/png.latex?\epsilon) at which you'd prefer a certainty of $1 to a chance ![](http://www.codecogs.com/png.latex?\epsilon) of $100. And if you have some preference that is so slight that you wouldn't be willing to accept _any_ chance of losing $1 in order to indulge it, it can't be a very strong preference. So I think for most practical purposes, we might as well accept Continuity.

*

The VNM theorem  

------------------

If your preferences are described by a transitive and complete relation ![](http://www.codecogs.com/png.latex?\succcurlyeq) on the probability distributions over some set ![](http://www.codecogs.com/png.latex?\mathcal{O}) of "outcomes", and this relation satisfies Independence and Continuity, then you have a utility function, and your genie will be maximizing expected utility.

Here's what that means. A utility function is a function ![](http://www.codecogs.com/png.latex?u : \mathcal{O}\to\mathbb{R}) which assigns a numerical "utility" to every outcome. Given a probability distribution ![](http://www.codecogs.com/png.latex?x) over ![](http://www.codecogs.com/png.latex?\mathcal{O}), we can compute the expected value of ![](http://www.codecogs.com/png.latex?u) under ![](http://www.codecogs.com/png.latex?x), ![](http://www.codecogs.com/png.latex?\textstyle\sum_{i\in\mathcal{O}} u(i)\,x_i); this is called the _expected utility_. We can prove that there is some utility function such that for all ![](http://www.codecogs.com/png.latex?x) and ![](http://www.codecogs.com/png.latex?y), we have ![](http://www.codecogs.com/png.latex?x\succ y) if and only if the expected utility under ![](http://www.codecogs.com/png.latex?x) is greater than the expected utility under ![](http://www.codecogs.com/png.latex?y).

In other words: ![](http://www.codecogs.com/png.latex?\succ) is _completely_ described by ![](http://www.codecogs.com/png.latex?u); if you know ![](http://www.codecogs.com/png.latex?u), you know ![](http://www.codecogs.com/png.latex?\succ). Instead of programming your genie with a function that takes two outcomes and says which one is better, you might as well program it with a function that takes one outcome and returns its utility. Any coherent direction for steering the world which happens to satisfy Continuity can be reduced to a function that takes outcomes and assigns them numerical ratings.

In fact, it turns out that the ![](http://www.codecogs.com/png.latex?u) for a given ![](http://www.codecogs.com/png.latex?\succ) is "almost" unique: Given two utility functions ![](http://www.codecogs.com/png.latex?u) and ![](http://www.codecogs.com/png.latex?v) that describe the same ![](http://www.codecogs.com/png.latex?\succ), there are numbers ![](http://www.codecogs.com/png.latex?a>0) and ![](http://www.codecogs.com/png.latex?b\in\mathbb{R}) such that for all ![](http://www.codecogs.com/png.latex?i\in\mathcal{O}), ![](http://www.codecogs.com/png.latex?v(i) = au(i) + b); this is called an "affine transformation". On the other hand, it's not hard to see that for any such ![](http://www.codecogs.com/png.latex?a) and ![](http://www.codecogs.com/png.latex?b),

![](http://www.codecogs.com/png.latex?\sum_{i\in\mathcal{O}} u(i)\,x_i > \sum_{i\in\mathcal{O}} u(i)\,y_i \;\iff\; \sum_{i\in\mathcal{O}} \big(au(i) + b\big)\,x_i > \sum_{i\in\mathcal{O}} \big(au(i) + b\big)\,y_i,)

so two utility functions represent the same preference relation if and only if they are related in this way.

*

You shouldn't read _too_ much into this conception of utility. For example, it doesn't make sense to see a fundamental distinction between outcomes with "positive" and with "negative" von Neumann-Morgenstern utility — because adding the right ![](http://www.codecogs.com/png.latex?b) can make any negative utility positive and any positive utility negative, without changing the underlying preference relation. The numbers that have real meaning are ratios between differences between utilities, ![](http://www.codecogs.com/png.latex?\textstyle\frac{u(i) - u(j)}{u(k) - u(\ell)}), because these don't change under affine transformations (the ![](http://www.codecogs.com/png.latex?b)'s cancel when you take the difference, and the ![](http://www.codecogs.com/png.latex?a)'s cancel when you take the ratio). [Academian's post](/lw/244/vnm_expected_utility_theory_uses_abuses_and/#pre) has more about misunderstandings of VNM utility.

In my view, what VNM utilities represent is not _necessarily_ how _good_ each outcome is; what they represent is what trade-offs between probability distributions you are willing to accept. Now, if you strongly felt that the difference between ![](http://www.codecogs.com/png.latex?i) and ![](http://www.codecogs.com/png.latex?j) was about the same as the difference between ![](http://www.codecogs.com/png.latex?k) and ![](http://www.codecogs.com/png.latex?\ell), then you should have _a very good reason_ before you make your ![](http://www.codecogs.com/png.latex?%5Ctextstyle%5Cfrac%7Bu%28i%29%20-%20u%28j%29%7D%7Bu%28k%29%20-%20u%28%5Cell%29%7D) a huge number. But on the other hand, I think it's ultimately your responsibility to decide what trade-offs you are willing to make; I don't think you can get away with "[stating how much you value different outcomes](/lw/cod/a_novel_approach_to_axiomatic_decision_theory/6onu)" and outsourcing the rest of the job to decision theory, without ever _considering_ what these valuations should mean in terms of probabilistic trade-offs.

*

Doing without Continuity  

---------------------------

What happens if your preferences do _not_ satisfy Continuity? Say, you want to save human lives, but you're not willing to incur _any_ probability, no matter how small, of _infinitely_ many people getting tortured infinitely long for this?

I do not see a good argument that this couldn't add up to a coherent direction for steering the world. I do, however, see an argument that in this case you care so little about finite numbers of human lives that in practice, you can probably neglect this concern entirely. (As a result, I doubt that your reflective equilibrium would want to adopt such preferences. But I don't think they're _in__coherent_.)

I'll assume that your morality can still distinguish only a finite number of outcomes, and you can choose only between a finite number of decisions. It's not obvious that these assumptions are justified if we want to take into account the _possibility_ that the true laws of physics might turn out to allow for infinite computations, but even in this case _you_ and any AI _you_ build will probably still be finite (though _it_ might build a successor that isn't), so I do in fact think there is a good chance that results derived under this assumption have relevance in the real world.

In this case, it turns out that you _still_ have a utility function, in a certain sense. (Proofs for non-standard results can be found in the [math appendix](/r/discussion/lw/fq0/math_appendix_for_why_you_must_maximize_expected/) to this post. I did the work myself, but I don't expect these results to be new.) This utility function describes only the concern most important to you: in our example, only the probability of infinite torture makes a difference to expected utility; any change in the probability of saving a finite number of lives leaves expected utility unchanged.

Let's define a relation ![](http://www.codecogs.com/png.latex?x\succ_* y), read "![](http://www.codecogs.com/png.latex?x) is _much better_ than ![](http://www.codecogs.com/png.latex?y)", which says that there is nothing you wouldn't give up a little probability of in order to get ![](http://www.codecogs.com/png.latex?x) instead of ![](http://www.codecogs.com/png.latex?y) — in our example: ![](http://www.codecogs.com/png.latex?x) doesn't merely save lives compared to ![](http://www.codecogs.com/png.latex?y), it makes infinite torture less likely. Formally, we define ![](http://www.codecogs.com/png.latex?x\succ_* y) to mean that ![](http://www.codecogs.com/png.latex?x'\succ y') for all ![](http://www.codecogs.com/png.latex?x') and ![](http://www.codecogs.com/png.latex?y') "close enough" to ![](http://www.codecogs.com/png.latex?x) and ![](http://www.codecogs.com/png.latex?y) respectively; more precisely: ![](http://www.codecogs.com/png.latex?x\succ_* y) if there is an ![](http://www.codecogs.com/png.latex?\epsilon>0) such that ![](http://www.codecogs.com/png.latex?x'\succ y') for all ![](http://www.codecogs.com/png.latex?x') and ![](http://www.codecogs.com/png.latex?y') with

![](http://www.codecogs.com/png.latex?\sum_{i\in\mathcal{O}}|x'_i - x_i| \;<\; \epsilon \qquad\text{and}\qquad \sum_{i\in\mathcal{O}} |y'_i - y_i| \;<\; \epsilon.)

(Or equivalently: if there are open sets ![](http://www.codecogs.com/png.latex?U) and ![](http://www.codecogs.com/png.latex?V) around ![](http://www.codecogs.com/png.latex?x) and ![](http://www.codecogs.com/png.latex?y), respectively, such that ![](http://www.codecogs.com/png.latex?x'\succ y') for all ![](http://www.codecogs.com/png.latex?x'\in U) and ![](http://www.codecogs.com/png.latex?y'\in V).)

It turns out that if ![](http://www.codecogs.com/png.latex?\succ) is a preference relation satisfying Independence, then ![](http://www.codecogs.com/png.latex?\succ_*) is a preference relation satisfying Independence and Continuity, and there is a utility function ![](http://www.codecogs.com/png.latex?u) such that ![](http://www.codecogs.com/png.latex?x\succ_* y) iff the expected utility under ![](http://www.codecogs.com/png.latex?x) is larger than the expected utility under ![](http://www.codecogs.com/png.latex?y). Obviously, ![](http://www.codecogs.com/png.latex?x\succ_* y) implies ![](http://www.codecogs.com/png.latex?x\succ y), so whenever two options have different expected utilities, you prefer the one with the larger expected utility. Your genie is _still_ an expected utility maximizer.

Furthermore, unless ![](http://www.codecogs.com/png.latex?x\sim y) for _all_ ![](http://www.codecogs.com/png.latex?x) and ![](http://www.codecogs.com/png.latex?y), ![](http://www.codecogs.com/png.latex?u) isn't constant — that is, there are _some_ ![](http://www.codecogs.com/png.latex?x) and ![](http://www.codecogs.com/png.latex?y) with ![](http://www.codecogs.com/png.latex?x\succ_* y). (If this weren't the case, the result above obviously wouldn't tell us very much about ![](http://www.codecogs.com/png.latex?\succ)!) Being indifferent between all possible actions doesn't make for a particularly interesting direction for steering the world, if it can be called one at all, so from now on let's assume that you are not.

*

It _can_ happen that there are two distributions ![](http://www.codecogs.com/png.latex?x) and ![](http://www.codecogs.com/png.latex?y) with the same expected utility, but ![](http://www.codecogs.com/png.latex?x\succ y). (![](http://www.codecogs.com/png.latex?x) saves more lives, but the probability of eternal torture is the same.) Thus, if your genie happens to face a choice between two actions that lead to the _same_ expected utility, it must do more work to figure out which of the actions it should take. But there is some reason to expect that such situations should be _rare_.

If there are ![](http://www.codecogs.com/png.latex?N) possible outcomes, then [the set ![](http://www.codecogs.com/png.latex?\Delta_N) of probability distributions over ![](http://www.codecogs.com/png.latex?\mathcal{O})](http://en.wikipedia.org/wiki/Simplex#The_standard_simplex) is ![](http://www.codecogs.com/png.latex?(N-1))-dimensional (because the probabilities must add up to 1, so if you know ![](http://www.codecogs.com/png.latex?N-1) of them, you can figure out the last one). For example, if there are three outcomes, ![](http://www.codecogs.com/png.latex?\Delta_N) is a triangle, and if there are four outcomes, it's a tetrahedron. On the other hand, it turns out that for any ![](http://www.codecogs.com/png.latex?r\in\mathbb{R}), the set of all ![](http://www.codecogs.com/png.latex?x\in\Delta_N) for which the expected utility equals ![](http://www.codecogs.com/png.latex?r) has dimension ![](http://www.codecogs.com/png.latex?N-2) or smaller: if ![](http://www.codecogs.com/png.latex?N=3), it's a line (or a point or the empty set); if ![](http://www.codecogs.com/png.latex?N=4), it's a plane (or a line or a point or the empty set).

Thus, in order to have the same expected utility, ![](http://www.codecogs.com/png.latex?x) and ![](http://www.codecogs.com/png.latex?y) must lie on the same hyperplane — not just on a plane _very close by_, but on _exactly_ the same plane. That's not just a small target to hit, that's an infinitely small target. If you use, say, a [Solomonoff prior](/lw/dhg/an_intuitive_explanation_of_solomonoff_induction/), then it seems _very_ unlikely that two of your finitely many options just _happen_ to lead to probability distributions which yield the same expected utility.

But we are bounded rationalists, not perfect Bayesians with uncomputable Solomonoff priors. We assign heads and tails exactly the same probability, not because there is no information that would make one or the other more likely (we could try to arrive at a best guess about which side is a little heavier than the other?), but because the problem is so complicated that we simply give up on it. What if it turns out that because of this, all the _difficult_ decisions we need to make turn out to be between actions that happen to have the same expected utility?

If you do your imperfect calculation and find that two of your options seem to yield exactly the same probability of eternal hell for infinitely many people, you _could_ then try to figure out which of them is more likely to save a finite number of lives. But it seems to me that this is _not_ the best approximation of an ideal Bayesian with your stated preferences. Shouldn't you spend those computational resources on doing a _better_ calculation of which option is more likely to lead to eternal hell?

For you _might_ arrive at a new estimate under which the probabilities of hell are at least slightly different. Even if you _suspect_ that the new calculation will again come out with the probabilities exactly equal, you don't _know_ that. And therefore, can you truly in good conscience argue that doing the new calculation does not improve the odds of avoiding hell —

— _at least a teeny tiny incredibly super-small for all ordinary intents and purposes completely irrelevant bit?_

Even if it _should_ be the case that to a _perfect_ Bayesian, the expected utilities under a Solomonoff prior were exactly the same, _you_ don't know that, so how can you possibly justify stopping the calculation and saving a mere finite number of lives?

*

So there you have it. In order to have a coherent direction in which you want to steer the world, you must have a set of outcomes and a preference relation over the probability distributions over these outcomes, and this relation must satisfy Independence — or so it seems to me, anyway. And if you do, then you have a utility function, and a perfect Bayesian maximizing your preferences will always maximize expected utility.

It _could_ happen that two options have exactly the same expected utility, and in this case the utility function doesn't tell you which of these is better, under your preferences; but as a bounded rationalist, you can never _know_ this, so if you have any computational resources left that you could spend on figuring out what your true preferences have to say, you should spend them on a better calculation of the expected utilities instead.

Given this, we might as well just talk about ![](http://www.codecogs.com/png.latex?\succ_*), which satisfies Continuity as well as Independence, instead of ![](http://www.codecogs.com/png.latex?\succ); and you might as well program your genie with your utility function, which only reflects ![](http://www.codecogs.com/png.latex?\succ_*), instead of with your true preferences.

_(Note: I am not literally saying that you should not try to understand the whole topic better than this if you are_ actually _going to program a Friendly AI. This is still meant as a metaphor. I_ am, _however, saying that expected utility theory, even with boring old real numbers as utilities_, _is not to be discarded_ lightly_.)_

*

Next post: Dealing with time
----------------------------

So far, we've always pretended that you only face _one_ choice, at _one_ point in time. But not only is there a way to apply our theory to repeated interactions with the environment — there are two!

One way is to say that at each point in time, you should apply decision theory to set of actions you can perform _at that point_. Now, the actual outcome depends of course not only on what you do now, but also on what you do later; but you know that you'll still use decision theory later, so you can _foresee_ what you will do in any possible future situation, and take it into account when computing what action you should choose now.

The second way is to make a choice only once, not between the actions you can take at that point in time, but between complete _plans_ — giant lookup tables — which specify how you _will_ behave in any situation you might possibly face. Thus, you simply do your expected utility calculation _once_, and then stick with the plan you have decided on.

[**Meditation:**](/lw/fu1/why_you_must_maximize_expected_utility/81h9) _Which of these is the **right** thing to do, if you have a perfect Bayesian genie and you want steer the future in some particular direction?_ _(Does it even make a difference which one you use?)_

**[» To the mathematical appendix](/r/discussion/lw/fq0/math_appendix_for_why_you_must_maximize_expected/)**

**Notes**

^1^ The accounts of decision theory I've read use the term "outcome", or "consequence", but leave it mostly undefined; in a lottery, it's the prize you get at the end, but clearly nobody is saying decision theory should _only_ apply to lotteries. I'm not changing its role in the mathematics, and I think my explanation of it is what the term always _wanted_ to mean; I expect that other people have explained it in similar ways, though I'm not sure how similar precisely.