# Buridan's ass in coordination games

Consider a simple coordination game. In this game, two players (player 1 and player 2) each simultaneously choose an action, X or Y. If they both choose X, they both get 1 utility. If they both choose Y, they both get < refresh to render LaTeX > utility for some < refresh to render LaTeX > known to both players. If they choose different actions, they both get 0 utility. Which action should they each choose? Assume they get to communicate before knowing < refresh to render LaTeX > , but not after knowing < refresh to render LaTeX > .

An optimal policy pair is for each to pick X if < refresh to render LaTeX > , and Y otherwise. Unfortunately, this policy pair can break down in the presence of even a small amount of noise. Assume neither player observes < refresh to render LaTeX > , but instead each receives an independent observation < refresh to render LaTeX > (player 1 sees < refresh to render LaTeX > , player 2 sees < refresh to render LaTeX > ), each of which is drawn uniformly from the range < refresh to render LaTeX > for some small < refresh to render LaTeX > . If both players follow the policy of choosing X if and only if their _observation_ of < refresh to render LaTeX > is less than 1, then if < refresh to render LaTeX > is very close to 1, there is a significant likelihood that one player will receive an observation less than 1, while the other will receive one greater than 1. Thus, they have a significant chance of choosing different actions and receiving 0 utility. This issue is essentially the same as the one faced by [Buridan's ass](https://en.wikipedia.org/wiki/Buridan%27s_ass): both options are equally good, so there is no way to effectively decide between them.

In fact, as I will prove:

_Claim 1_: No pair of policies in which the players select their actions _independently_ given their observations can simultaneously guarantee an expected utility greater than < refresh to render LaTeX > for all < refresh to render LaTeX > .

But things change when a shared source of randomness is allowed. Then, as I will also prove:

_Claim 2_: Suppose that, after observing their observation of < refresh to render LaTeX > , each player also gets to observe a random number < refresh to render LaTeX > . Then there is a pair of policies that achieves expected utility at least < refresh to render LaTeX > regardless of < refresh to render LaTeX > .

This solution method extends to arbitrary cooperative normal-form games where players observe a perturbed version of the true utility function. This is shown in the appendix.

Why is this important? I expect progress in decision theory to come from studying very simple decision problems like this one. If it is possible to show that any solution to some simple problem has certain properties, then this usefully constrains the design space of possible decision theories. In this case, the result suggests that something like shared randomness may be required for achieving provable near-optimality even in cooperative settings.

Claim 1: impossibility of solving the game using independent randomness
=======================================================================

Fix < refresh to render LaTeX > . Let < refresh to render LaTeX > be Lesbegue measurable functions representing the two players' policies, which map < refresh to render LaTeX > to the player's probability of choosing action Y.

Define < refresh to render LaTeX > to be the function mapping < refresh to render LaTeX > to the resulting < refresh to render LaTeX > distribution.

Define < refresh to render LaTeX > . This is defined so that < refresh to render LaTeX > equals player < refresh to render LaTeX > 's overall probability of choosing action Y.

For < refresh to render LaTeX > , note that the [total variation distance](https://en.wikipedia.org/wiki/Total_variation_distance_of_probability_measures) between < refresh to render LaTeX > and < refresh to render LaTeX > is at most < refresh to render LaTeX > . Thus, since < refresh to render LaTeX > is bounded between 0 and 1, < refresh to render LaTeX > is < refresh to render LaTeX > -Lipschitz and therefore continuous.

I will now show that, for some < refresh to render LaTeX > , the players achieve expected utility at most < refresh to render LaTeX > by case analysis on < refresh to render LaTeX > :

*   If < refresh to render LaTeX > then at most 0.5 expected utility is achieved when < refresh to render LaTeX > (since at most 1 utility is achieved when player 1 chooses action X, and no utility is achieved when player 1 chooses action Y).
*   If < refresh to render LaTeX > , then at most 1.5 expected utility is achieved when < refresh to render LaTeX > (since at most 1 utility is achieved when player 1 chooses action X, and at most 2 utility is achieved when player 1 chooses action Y).
*   If neither of the two above cases hold, then by continuity of < refresh to render LaTeX > and the intermediate value theorem, there exists some < refresh to render LaTeX > with < refresh to render LaTeX > . When < refresh to render LaTeX > , since the two players choose actions independently, there is a 0.5 probability that they select the same action, ensuring that they achieve at most < refresh to render LaTeX > expected utility.

Thus, claim 1 is proven. This proof method bears resemblance to the problem faced by Buridan's ass: in the third case, some < refresh to render LaTeX > value is found so that the "policy" < refresh to render LaTeX > is equally compelled by both options, choosing between them with a 50/50 coin flip in a way that is disastrous for coordination.

Claim 2: solving the game using shared randomness
=================================================

Define < refresh to render LaTeX > . Consider a pair of policies in which player < refresh to render LaTeX > chooses Y if < refresh to render LaTeX > , and otherwise chooses X, where < refresh to render LaTeX > . Intuitively, each of these policies increases its chance of taking action Y as its observation of < refresh to render LaTeX > increases in a _smooth_ fashion, and they correlate their randomness so that they are likely to choose the same action. Now note a couple properties of this pair of policies:

1\. For < refresh to render LaTeX > , it is guaranteed that < refresh to render LaTeX > , so both players always take action Y.

2\. Since < refresh to render LaTeX > is < refresh to render LaTeX > -Lipschitz, and < refresh to render LaTeX > , the probability of the players taking different actions is at most < refresh to render LaTeX > .

I will now show that the players' expected utility is at least < refresh to render LaTeX > by case analysis:

*   Suppose < refresh to render LaTeX > . Then by property 1, both players are guaranteed to take action Y, so they achieve expected utility < refresh to render LaTeX > .
*   Suppose < refresh to render LaTeX > . By property 2, the players choose the same action with probability at least < refresh to render LaTeX > , and taking the same action yields a utility of at least 1, so they achieve expected utility at least < refresh to render LaTeX > . Due to the bound on < refresh to render LaTeX > , this is at least < refresh to render LaTeX > .

Thus, the claim is proven. By setting < refresh to render LaTeX > sufficiently low, this pair of policies yields an expected utility arbitrarily close to < refresh to render LaTeX > regardless of < refresh to render LaTeX > .

Conclusion and directions for further research
==============================================

Together, these claims show that there is a simple set of noisy coordination games (namely, the set of games described in the beginning of this post for all possible < refresh to render LaTeX > values) that is impossible to solve with only independent randomness, but which can be solved using shared randomness. Some notes on this:

*   The proof of claim 1 only used the fact that < refresh to render LaTeX > is continuous. So even without noise, if the players' policies are a continuous function of the utility < refresh to render LaTeX > , the same problem occurs.
*   A pair of Bayesians playing this coordination game who have a prior over < refresh to render LaTeX > have no need for randomization, independent or joint. The goal of achieving a high expected utility regardless of < refresh to render LaTeX > most clearly makes sense if < refresh to render LaTeX > is selected adversarially (to maximize regret). It also makes sense if the environment is hard to model in a way that makes selection of a policy with a low rate of "ties" difficult. The comparison between the shared randomness solution and the Bayesian solution is similar to the comparison between randomized Quicksort and a "Bayesian" variant of Quicksort that selects pivots based on some expected distribution of inputs. While the Bayesian solution works better than the randomized solution when the prior over the input distribution is accurate, the randomized solution is simpler, is amenable to proofs, and works well even under adversarial conditions.

Where to go from here? Roughly, my overall "plan" for formal decision theory consists of 3 steps:

1\. Solve Cartesian cooperative decision theory problems where players reason about each other using something like [reflective oracles](https://arxiv.org/abs/1508.04145).

2\. Extend the solution in 1 to Cartesian multiplayer game theory problems where players have different utility functions and reason about each other using something like reflective oracles.

3\. Extend the solution in 2 to a logically uncertain naturalized setting.

Step 1 has still not been solved. Previous writing on this includes [this post](https://agentfoundations.org/item?id=853) which studies a setting with a single memoryless player (which is similar to a set of players who have the same utility function). The post shows (redundantly with [the paper introducing the absent-minded driver problem](https://pdfs.semanticscholar.org/dd03/415f643088df006f9ad12e22f43a57adb552.pdf) as I later found out) that, if the player's policy is _globally_ optimal (i.e. it achieves the highest possible expected utility), then all actions that might be taken by that policy are CDT-optimal, assuming SIA probabilities. This second condition is a local optimality condition, so the post shows that global optimality implies local optimality.

It would be highly desirable to find a similar but different local optimality property that implies a global optimality property. That would essentially be a way of deriving collective self-interest from individual self-interest when individuals have the same utility function and common priors.

As this current post shows, the globally optimal policy is _discontinuous_ as a function of the utilities involved, and no continuous approximation always yields a near-optimal expected utility. I expect this to hinder attempts to derive global optimality from local optimality, as it implies there is a valley of bad policies between decent policies and good ones.

Introducing shared randomness here may help by preserving continuity. So a natural next step is to find useful local optimality properties in a cooperative setting where players have shared randomness.

Appendix: extension to arbitrary normal-form cooperative games
==============================================================

The solution described in the section on claim 2 can be extended to arbitrary normal-form cooperative games where the players receive only noisy observations of the payoffs. Reading this appendix (which takes up the rest of this post) is unnecessary for understanding the main point of this post.

Let < refresh to render LaTeX > be the number of players, < refresh to render LaTeX > be player < refresh to render LaTeX > 's set of actions, and < refresh to render LaTeX > be the shared unknown utility function over strategy profiles, whose minimum value is < refresh to render LaTeX > and whose maximum value is < refresh to render LaTeX > .

Fix < refresh to render LaTeX > . Let < refresh to render LaTeX > be a "perturbed" version of < refresh to render LaTeX > that player < refresh to render LaTeX > observes; it must satisfy the property that for any strategy profile < refresh to render LaTeX > , < refresh to render LaTeX > .

To define the policies, we will first number the strategy profiles arbitrarily < refresh to render LaTeX > where < refresh to render LaTeX > . Let < refresh to render LaTeX > be some number to be determined later. For < refresh to render LaTeX > , define

< refresh to render LaTeX > .

This defines, given any < refresh to render LaTeX > , a probability distribution over strategy profiles, since < refresh to render LaTeX > . Specifically, the distribution is a [softmax](https://en.wikipedia.org/wiki/Softmax_function). This distribution is more likely to select strategy profiles that < refresh to render LaTeX > considers better, but has some chance of selecting every strategy profile.

Players will sample from this distribution using a source of shared randomness, < refresh to render LaTeX > . Define

< refresh to render LaTeX >

Now note that < refresh to render LaTeX > , i.e. < refresh to render LaTeX > is distributed according to < refresh to render LaTeX > . Define policies < refresh to render LaTeX > . That is, player < refresh to render LaTeX > will play their appropriate action for strategy profile number < refresh to render LaTeX > .

Roughly, we will now show 2 properties that are sufficient to establish that these policies are near-optimal:

*   With high probability, the players play according to < refresh to render LaTeX > , i.e. for all < refresh to render LaTeX > , < refresh to render LaTeX > .
*   The strategy profile < refresh to render LaTeX > is near-optimal in expectation, i.e. < refresh to render LaTeX > is close to < refresh to render LaTeX > .

Players play according to < refresh to render LaTeX > with high probability
---------------------------------------------------------------------------

First, we will show that, for all < refresh to render LaTeX > and < refresh to render LaTeX > , < refresh to render LaTeX > is close to < refresh to render LaTeX > .

For all < refresh to render LaTeX > , since < refresh to render LaTeX > , we have

< refresh to render LaTeX > .

Therefore, for all < refresh to render LaTeX > ,

< refresh to render LaTeX > .

By identical logic,

< refresh to render LaTeX > .

Now we will bound < refresh to render LaTeX > .

< refresh to render LaTeX >

< refresh to render LaTeX >

< refresh to render LaTeX >

< refresh to render LaTeX >

< refresh to render LaTeX >

< refresh to render LaTeX >

For it to be the case that < refresh to render LaTeX > , it must be the case that for some < refresh to render LaTeX > ,

< refresh to render LaTeX >

which implies

< refresh to render LaTeX >

Due to the bound on < refresh to render LaTeX > , the measure of this hull is at most < refresh to render LaTeX > . Furthermore, there are < refresh to render LaTeX > hulls of this form (one for each < refresh to render LaTeX > value), so the total measure of these hulls is at most < refresh to render LaTeX > .

So, player < refresh to render LaTeX > chooses action < refresh to render LaTeX > with probability at least < refresh to render LaTeX > . Thus, by the union bound, the players jointly choose the actions < refresh to render LaTeX > with probability at least < refresh to render LaTeX > .

< refresh to render LaTeX > is near-optimal in expectation
----------------------------------------------------------

First we will prove a lemma.

_Softmax lemma_: For any < refresh to render LaTeX > and vector < refresh to render LaTeX > of length < refresh to render LaTeX > ,

< refresh to render LaTeX > .

_Proof_:

Let < refresh to render LaTeX > be the maximum of < refresh to render LaTeX > . Now:

< refresh to render LaTeX >

For any < refresh to render LaTeX > , we have < refresh to render LaTeX > . Therefore for any < refresh to render LaTeX > , < refresh to render LaTeX > . By exponentiating both sides, < refresh to render LaTeX > .

Applying this to the term from the previous inequality:

< refresh to render LaTeX >

At this point we have

< refresh to render LaTeX > .

< refresh to render LaTeX >

At this point the implication is straightforward. Since < refresh to render LaTeX > is distributed according to < refresh to render LaTeX > , we have

< refresh to render LaTeX > .

Proving the result from these facts
-----------------------------------

At this point we have:

*   Players play according to < refresh to render LaTeX > with probability at least < refresh to render LaTeX > .
*   < refresh to render LaTeX > .

The first fact lets us quantify the expected difference between < refresh to render LaTeX > and < refresh to render LaTeX > . Since these random variables are equal with probability at least < refresh to render LaTeX > , and their difference is bounded by < refresh to render LaTeX > , their expected difference is at most < refresh to render LaTeX > .

Combining this with the second inequality:

< refresh to render LaTeX >

To minimize < refresh to render LaTeX > , set

< refresh to render LaTeX >

This yields:

< refresh to render LaTeX >

Now note that

< refresh to render LaTeX >

Due to the fact that < refresh to render LaTeX > and < refresh to render LaTeX > ,

< refresh to render LaTeX > .

Because for any < refresh to render LaTeX > , < refresh to render LaTeX > , we have

< refresh to render LaTeX > .

By combining inequalities and equalities so far:

< refresh to render LaTeX > .

The expected suboptimality goes to 0 as < refresh to render LaTeX > approaches 0.