# Causal decision theory is unsatisfactory

_This is [crossposted](http://mindingourway.com/causal-reasoning-is-unsatisfactory/) from [my new blog](http://mindingourway.com/). I was planning to write a short post explaining how Newcomblike problems are the norm and why any sufficiently powerful intelligence built to use causal decision theory would self-modify to stop using causal decision theory in short order. Turns out it's not such a short topic, and it's turning into a short intro to decision theory._

_I've been motivating MIRI's technical agenda (decision theory and otherwise) to outsiders quite frequently recently, and I received a few comments of the form "Oh cool, I've seen lots of decision theory type stuff on LessWrong, but I hadn't understood the connection." While the intended audience of my blog is wider than the readerbase of LW (and thus, the tone might seem off and the content a bit basic), I've updated towards these posts being useful here. I also hope that some of you will correct my mistakes!_

_This sequence will probably run for four or five posts, during which I'll motivate the use of decision theory, the problems with the modern standard of decision theory (CDT), and some of the reasons why these problems are an FAI concern._

_I'll be [giving a talk](http://intelligence.org/2014/09/12/nate-soares-speaking-purdue-september-18th/) on the material from this sequence at Purdue next week._

1
=

Choice is a crucial component of reasoning. Given a set of available actions, which action do you take? Do you go out to the movies or stay in with a book? Do you capture the bishop or fork the king? Somehow, we must reason about our options and choose the best one.

Of course, we humans don't consciously weigh all of our actions. Many of our choices are made subconsciously. (Which letter will I type next? When will I get a drink of water?) Yet even if the choices are made by subconscious heuristics, they must be made somehow.

In practice, decisions are often made on autopilot. We don't weigh every available alternative when it's time to prepare for work in the morning, we just pattern-match the situation and carry out some routine. This is a shortcut that saves time and cognitive energy. Yet, no matter how much we stick to routines, we still spend _some_ of our time making hard choices, weighing alternatives, and predicting which available action will serve us best.

The study of how to make these sorts of decisions is known as _Decision Theory_. This field of research is closely intertwined with Economics, Philosophy, Mathematics, and (of course) Game Theory. It will be the subject of today's post.

2
=

Decisions about what action to choose necessarily involve _counterfactual reasoning_, in the sense that we reason about what _would_ happen if we took actions which we (in fact) will not take.

We all have some way of performing this counterfactual reasoning. Most of us can visualize what would happen if we did something that we aren't going to do. For example, consider shouting "PUPPIES!" at the top of your lungs right now. I bet you won't do it, but I _also_ bet that you can picture the results.

One of the major goals of decision theory is to formalize this counterfactual reasoning: if we had unlimited resources then how would we compute alternatives so as to ensure that we always pick the best possible action? This question is harder than it looks, for reasons explored below: counterfactal reasoning can encounter many pitfalls.

A second major goal of decision theory is this: human counterfactual reasoning sometimes runs afoul of those pitfalls, and a formal understanding of decision theory can help humans make better decisions. It's no coincidence that Game Theory was developed during the cold war!

(_My_ major goal in decision theory is to understand it as part of the process of learning how to construct a machine intelligence that reliably reasons well. This provides the motivation to nitpick existing decision theories. If they're broken then we had better learn that sooner rather than later.)

3
=

Sometimes, it's easy to choose the best available action. You consider each action in turn, and predict the outcome, and then pick the action that leads to the best outcome. This can be difficult when accurate predictions are unavailable, but that's not the problem that we address with decision theory. The problem we address is that sometimes _it is difficult to reason about what would happen if you took a given action_.

For example, imagine that you know of a fortune teller who can reliably read palms to divine destinies. Most people who get a good fortune wind up happy, while most people who get a bad fortune wind up sad. It's been experimentally verified that she can use information on palms to reliably make inferences about the palm-owner's destiny.

So... should you [get palm surgery to change your fate](http://www.thedailybeast.com/articles/2013/07/12/your-future-is-in-the-palm-of-your-surgeon-s-hand.html)?

If you're bad at reasoning about counterfactuals, you might reason as follows:

> Nine out of ten people who get a good fortune do well in life. I had better use the palm surgery to ensure a good fortune!

Now admittedly, if palm reading is shown to work, the first thing you should do is check whether you can alter destiny by altering your palms. However, _assuming_ that changing your palm doesn't drastically affect your fate, this sort of reasoning is quite silly.

The above reasoning process conflates _correlation_ with _causal control_. The above reasoner gets palm surgery because they want a good destiny. But while your palm may give _information_ about their destiny, your palm does not _control_ your fate.

If we find out that we've been using this sort of reasoning, we can usually do better by considering actions only on the basis of what they _cause_.

4
=

This idea leads us to _causal decision theory (CDT)_, which demands that we consider actions based only upon the causal effects of those actions.

Actions are considered using _causal counterfactual reasoning_. Though causal counterfactual reasoning can be formalized in many ways, we will consider graphical models specifically. Roughly, a _causal graph_ is a graph where the world model is divided into a series of nodes, with arrows signifying the causal connections between the nodes. For a more formal introduction, you'll need to consult a [textbook](http://www.amazon.com/Causality-Reasoning-Inference-Judea-Pearl/dp/0521773628). As an example, here's a causal graph for the palm-reading scenario above:

![](http://mindingourway.com/content/images/2014/Sep/palm-surgery.png)

The choice is denoted by the dotted `Surgery?` node. Your payoff is the `$` diamond. Each node is specified as a function of all nodes causing it.

For example, in a very simple deterministic version of the palm-reading scenario, the nodes could be specified as follows:

1.  `Surgery?` is a program implementing the agent, and must output either _yes_ or _no_.
2.  `Destiny` is either _good_ or _bad_.
3.  `Fortune` is always _good_ if `Surgery?` is _yes_, and is the same as `Destiny` otherwise.
4.  `$` is $100 if `Destiny` is _good_ and $10 otherwise, minus $10 if `Surgery` is _yes_. Surgery is expensive!

Now let's say that you expect even odds on whether or not your destiny is good or bad, e.g. the probability that `Destiny`=_good_ is 50%.

If the `Surgery?` node is a program that implements causal decision theory, then that program will choose between _yes_ and _no_ using the following reasoning:

*   The action node is `Surgery?`
*   The available actions are _yes_ and _no_
*   The payoff node is `$`
*   Consider the action _yes_
    *   Replace the value of `Surgery?` with a function that always returns _yes_
    *   Calculate the value of `$`
    *   We would get $90 if `Destiny`=_good_
    *   We would get $0 if `Destiny`=_bad_
    *   This is $45 in expectation.
*   Consider the action _no_
    *   Replace the value of `Surgery?` with a function that always returns _no_
    *   Calculate the value of `$`
    *   We would get $100 if `Destiny`=_good_
    *   We would get $10 if `Destiny`=_bad_
    *   This is $55 in expectation.
*   Return _no_, as that yields the higher value of `$`.

More generally, the CDT reasoning procedure works as follows:

1.  Identify your action node **A**
2.  Identify your available actions _Acts_.
3.  Identify your payoff node **U**.
4.  For each action _a_
    *   Set **A** to _a_ by replacing the value of **A** with a function that ignores its input and returns _a_
    *   Evaluate the expectation of **U** given that **A**=a
5.  Take the _a_ with the highest associated expectation of **U**.

Notice how CDT evaluates counterfactuals by setting the value of its action node in a causal graph, and then calculating its payoff accordingly. Done correctly, this allows a reasoner to figure out the causal implications of taking a specific action without getting confused by additional variables.

CDT is the academic standard decision theory. Economics, statistics, and philosophy all assume (or, indeed, _define_) that rational reasoners use causal decision theory to choose between available actions.

Furthermore, narrow AI systems which consider their options using this sort of causal counterfactual reasoning are implicitly acting like they use causal decision theory.

Unfortunately, causal decision theory is broken.

5
=

Before we dive into the problems with CDT, let's flesh it out a bit more. Game Theorists often talk about scenarios in terms of tables that list the payoffs associated with each action. This might seem a little bit like cheating, because it often takes a lot of hard work to determine what the payoff of any given action is. However, these tables will allow us to explore some simple examples of how causal reasoning works.

I will describe a variant of the classic [Prisoner's Dilemma](http://en.wikipedia.org/wiki/Prisoner's_dilemma) which I refer to as the _token trade_. There are two players in two separate rooms, one red and one green. The red player starts with the green token, and vice versa. Each must decide (in isolation, without communication) whether or not to give their token to me, in which case I will give it to the other player.

Afterwards, they may cash their token out. The red player gets $200 for cashing out the red token and $100 for the green token, and vice versa. The payoff table looks like this:

|     | Give | Keep |
| --- | --- | --- |
| Give | ( $200, $200 ) | ( $0, $300 ) |
| Keep | ( $300, $0 ) | ( $100, $100 ) |

For example, if the green player gives the red token away, and the red player keeps the green token, then the red player gets $300 while the green player gets nothing.

Now imagine a causal decision theorist facing this scenario. Their causal graph might look something like this:

![](http://mindingourway.com/content/images/2014/Sep/token-trade.png)

Let's evaluate this using CDT. The action node is `Give?`, the payoff node is `$`. We must evaluate the expectation of `$` given `Give?`=_yes_ and `Give?`=_no_. This, of course, depends upon the expected value of `TheirDecision`.

In Game Theory, we usually assume that the opponent is reasoning using something like CDT. Then we can reason about `TheirDecision` given that they are doing similar reasoning about our decision and so on. This threatens to lead to infinite regress, but in fact there are some tricks you can use to guarantee at least one equilibrium. (These are the famous [Nash equilibria](http://en.wikipedia.org/wiki/Nash_equilibrium).) This sort of reasoning requires some modifications to the simple CDT procedure given above which we're going to ignore today. Because while _most_ scenarios with multiple agents require more complicated reasoning, the token trade is an especially simple scenario where we can brush all that under the rug.

In the token trade, the expected value of `TheirDecision` doesn't matter to a CDT agent. No matter what the probability p of `TheirDecision`=_give_ happens to be, the CDT agent will do the following reasoning:

*   Change `Give?` to be a constant function returning _yes_
    *   If `TheirDecision`=_give_ then we get $200
    *   If `TheirDecision`=_keep_ then we get $0
    *   We get 200p dollars in expectation.
*   Change `Give?` to be a constant function returning _no_
    *   If `TheirDecision`=_give_ then we get $300
    *   If `TheirDecision`=_keep_ then we get $100
    *   We get 300p + 100(1-p) dollars in expectation.

Obviously, 300p+100(1-p) will be larger than 200p, _no matter what probability p is_.

A CDT agent in the token trade must have an expectation about `TheirDecision` captured by a probability p that they will give their token, and we have just shown that no matter what that p is, the CDT agent will keep their token.

When something like this occurs (where `Give?`=_no_ is better regardless of the value of `TheirDecision`) we say that `Give?`=_no_ is a "dominant strategy". CDT executes this dominant strategy, and keeps its token.

6
=

Of course, this means that each player will get $100, when they could have both recieved $200. This may seem unsatisfactory. Both players would agree that they could do better by trading tokens. Why don't they coordinate?

The classic response is that the token trade (better known as the Prisoner's Dilemma) is a game that explicitly disallows coordination. If players do have an opportunity to coordinate (or even if they expect to play the game mulitple times) then they can (and do!) do better than this.

I won't object much here, except to note that this answer is still unsatisfactory. CDT agents fail to cooperate on a one-shot Prisoner's Dilemma. That's a bullet that causal decision theorists willingly bite, but don't forget that it's still a bullet.

7
=

Failure to cooperate on the one-shot Prisoner's Dilemma is not necessarily a problem. Indeed, if you ever find yourself playing a token trade against an opponent using CDT, then you had better hold on to your token, because they surely aren't going to give you theirs.

However, CDT _does_ fail on a very similar problem where it seems insane to fail. CDT fails at the token trade _even when it knows it is playing against a perfect copy of itself._

I call this the "mirror token trade", and it works as follows: first, I clone you. Then, I make you play a token trade against yourself.

In this case, your opponent is guaranteed to pick exactly the same action that you pick. (Well, mostly: the game isn't completely symmetric. If you want to nitpick, consider that instead of playing against a copy of yourself, you must write a red/green colorblind deterministic computer program which will play against a copy of itself.)

The causal graph for this game looks like this:

![](http://mindingourway.com/content/images/2014/Sep/token-mirror.png)

Because I've swept questions of determinism and asymmetry under the rug, both decisions will be identical. The red copy should trade its token, because that's guaranteed to get it the green token (and it's the only way to do so).

Yet CDT would have you evaluate an action by considering what happens if you replace the node `Give?` with a function that always returns that action. But this intervention does not affect the opponent, which reasons the same way! Just as before, a CDT agent treats `TheirDecision` as if it has some probability of being _give_ that is independent from the agent's action, and reasons that "I always keep my token while they act independently" dominates "I always give my token while they act independently".

Do you see the problem here? CDT is evaluating its action by changing the value of its action node `Give?`, assuming that this only affects things that are _caused_ by `Give?`. The agent reasons counterfactually by considering "what if `Give?` were a constant function that always returned _yes_?" while failing to note that overwriting `Give?` in this way neglects the fact that `Give?` and `TheirDecision` are necessarily equal.

Or, to put it another way, CDT evaluates counterfactuals _assuming_ that all nodes uncaused by its action are independent of its action. It thinks it can change its action and only look at the downstream effects. This can break down when there are acausal connections between the nodes.

After the red agent has been created from the template, its decision no longer _causally_ affects the decision of the green agent. But both agents will do the same thing! There is a _logical_ connection, even though there is no causal connection. It is these logical connections that are ignored by causal counterfactual reasoning.

This is a subtle point, but an important one: the values of `Give?` and `TheirDecision` are logically connected, but CDT's method of reasoning about counterfactuals neglects this connection.

8
=

This is a know failure mode for causal decision theory. The mirror token trade is an example of what's known as a "Newcomblike problem".

Decision theorists occasionally dismiss Newcomblike problems as edge cases, or as scenarios specifically designed to punish agents for being "rational". I disagree.

And finally, eight sections in, I'm ready to articulate the original point: Newcomblike problems aren't a special case. _They're the norm._

But this post has already run on for far too long, so that discussion will have to wait until next time.