Abstract: A set of players delegate playing a game to a set of representatives, one for each player. We imagine that each player trusts their respective representative's strategic abilities. Thus, we might imagine that per default, the original players would simply instruct the representatives to play the original game as best as they can. In this paper, we ask: are there safe Pareto improvements on this default way of giving instructions? That is, we imagine that the original players can coordinate to tell their representatives to only consider some subset of the available strategies and to assign utilities to outcomes differently than the original players. Then can the original players do this in such a way that the payoff is guaranteed to be weakly higher than under the default instructions for all the original players? In particular, can they Pareto-improve without probabilistic assumptions about how the representatives play games? In this paper, we give some examples of safe Pareto improvements. We prove that the notion of safe Pareto improvements is closely related to a notion of outcome correspondence between games. We also show that under some specific assumptions about how the representatives play games, finding safe Pareto improvements is NP-complete.

Safe Pareto Improvements for Delegated Game Playing

INTRODUCTION Between Aliceland and Bobbesia lies a sparsely populated desert. Until recently, neither of the two countries had any interest in the desert. However, geologists have recently discovered that it contains large oil reserves. Now, both Aliceland and Bobbesia would like to annex the desert, but they worry about a military conflict that would ensue if both countries insist on annexing. Table  1  models this strategic situation as a normal-form game. The strategy DM (short for "Demand with Military") denotes a military invasion of the desert, demanding annexation. If both countries send their military with such an aggressive mission, the countries fight a devastating war. The strategy RM (for "Refrain with Mility") denotes yielding the territory to the other country, but building defenses to prevent an invasion of one's original territories. Alternatively, the countries can choose to not raise a military force at all, while potentially still demanding control of the desert by sending only its leader (DL, short for "Demand with Leader"). In this case, if both countries demand the desert, war does not ensue. Finally, they could neither demand nor build up a mility (RL). If one of the two countries has their military ready and the other does not, the militarized country will know and will be able to invade the other country. In game-theoretical terms, militarizing therefore strictly dominates not militarizing. Instead of making the decision directly, the parliaments of Aliceland and Bobbesia appoint special commissions for making this strategic decision, led by Alice and Bob, respectively. The parliaments can instruct these representatives in various ways. They can explicitly tell them what to do -for example, Aliceland could directly tell Alice to play DM. However, we imagine that the parliaments trust the commissions' judgments more than they trust their own and hence they might prefer to give an instruction of the type, "make whatever demands you think are best for our country" (perhaps contractually guaranteeing a reward in proportion to the utility of the final outcome). They might not know what that will entail, i.e., how the commissions decide what demands to make given that instruction. However -based on their trust in their representatives -they might still believe that this leads to better outcomes than giving an explicit instruction. We will also imagine these instructions are (or at least can be) given publicly and that the commissions are bound (as if by a contract) to follow these instructions. In particular, we imagine that the two commissions can see each other's instructions. Thus, in instructing their commissions, the countries play a game with bilateral precommitment. When instructed to play a game as best as they can, we imagine that the commissions play that game in the usual way, i.e., without further abilities to credibly commit or to instruct subcommittees and so forth. It may seem that without having their parliaments ponder equilibrium selection, Aliceland and Bobbesia cannot do better than leave the game to their representatives. Unfortunately, in this default equilibrium, war is still a possibility. Even the brilliant strategists Alice and Bob may not always be able to resolve the difficult equilibrium selection problem to the same pure Nash equilibrium. In the literature on commitment devices and in particular the literature on program equilibrium, important ideas have been proposed for avoiding such bad outcomes. Imagine for a moment that Alice and Bob will play a Prisoner's Dilemma (rather than the Demand Game of Table  1 ). Then the default of (Defect, Defect) can be Pareto-improved upon. Both original players (Aliceland and Bobbesia) can use the following instruction for their representatives: "If the opponent's instruction is equal to this instruction, Cooperate; otherwise Defect."  [11, 15, 25]  Then it is a Nash equilibrium for both players to use this instruction. In this equilibrium, (Cooperate, Cooperate) is played and it is thus Pareto-optimal and Pareto-better than the default.  In cases like the Demand Game, it is more difficult to apply this approach to improve upon the default of simply delegating the choice. Of course, if one could calculate the expected utility of submitting the default instructions, then one could similarly commit the representatives to follow some (joint) mix over the Pareto-optimal outcomes ((RM, DM), (DM, RM), (RM, RM), (DL, DL), etc.) that Pareto-improves on the default expected utilities. However, we will assume that the original players are unable or unwilling to form probabilistic expectations about how the representatives play the Demand Game, i.e., about what would happen with the default instructions. If this is the case, then this type of Pareto-improvement on the default is unappealing. The goal of this paper is to show and analyze how even without forming probabilistic beliefs about the representatives, the original players can Pareto-improve on the default equilibrium. We will call such improvements safe Pareto improvements (SPIs). We here briefly give an example in the Demand Game. The key idea is for the original players to instruct the representatives to select only from {DL, RL}, i.e., to not raise a military. Further, they tell them to disvalue the conflict outcome (DL, DL) as they would disvalue the original conflict outcome of war in the default equilibrium. Overall, this means telling them to play the game of Table  2 . (Again, we could imagine that the instructions specify Table  2  to be how Aliceland and Bobbesia financially reward Alice and Bob.) Importantly, Aliceland's instruction to play that game must be conditional on Bobbesia also instructing their commission to play that game, and vice versa. Otherwise, one of the countries could profit from deviating by instructing their representative to always play DM or RM (or to play by the original utility function). The game of Table  2  is isomorphic to the DM-RM part of the original Demand Game of Table  1 . Of course, the original players know neither how the original Demand Game nor the game of Table  2  will be played by the representatives. However, since these games are isomorphic, one should arguably expect them to be played isomorphically. For example, one should expect that (RM, DM) would be played in the original game if and only if (RL, DL) would be played in the modified game. However, the conflict outcome (DM, DM) is replaced in the new game with the outcome (DL, DL). This outcome is harmless (Pareto-optimal) for the original players. Contributions. Our paper generalizes this idea to arbitrary normalform games and is organized as follows. In Section 2, we introduce some notation for games and multivalued functions that we will use throughout this paper. In Section 3, we introduce the setting of delegated game playing for this paper and define and motivate the concept of safe Pareto improvements in more detail. In Section we briefly review the the concepts of program games and program equilibrium and show that SPIs can be implemented as program equilibria. In Section 5, we introduce a notion of outcome correspondence between games. This relation expresses the original players' beliefs about similarities between how the representatives play different games. For example, in our example, the Demand Game of Table  1  (arguably) corresponds to the game of Table  2  in that the representatives (arguably) would play (DM, DM) in the original game if and only if they play (DL, DL) in the new game, and so forth. We also show some basic results (reflexivity, transitivity, etc.) about the outcome correspondence relation on games. In Section 6 we show that the notion of outcome correspondence is central to deriving SPIs. In particular, we show that a game Γ 𝑠 is an SPI on another game Γ if and only if there is a Pareto-improving outcome correspondence relation between Γ 𝑠 and Γ. To derive SPIs, we need to make some assumptions about outcome correspondence, i.e., about which games are played in similar ways by representatives. We give two very weak assumptions of this type in Section 7. The first is that the representatives play isomorphic games isomorphically. The second is that the representatives' play is invariant under the removal of strictly dominated strategies. For example, we assume that in the Demand Game the representatives only play DM and RM. Moreover we assume that we could remove DL and RL from the game and the representatives would still play the same strategies as in the original Demand Game with certainty. Our SPI for the Demand Game can be proven using these assumptions. Section 8 shows that determining whether there exists an SPI based on these assumptions is NP-complete. Section 9 considers a different setting in which we allow the original players to let the representatives choose from newly constructed strategies whose corresponding outcomes map arbitrarily onto feasible payoff vectors from the original game. In this new setting, finding SPIs can be done in polynomial time. We conclude by discussing the problem of selecting between different SPIs on a given game (Section 10) and giving some ideas for directions for future work (Section 11).
 PRELIMINARIES 2.1 Games We here recall some basic definitions from game theory. An 𝑛-player game is a tuple (𝐴, u) of a set 𝐴 = 𝐴 1 × ... × 𝐴 𝑛 of (pure) strategy profiles (or outcomes) and a function u : 𝐴 → R 𝑛 that assigns to each outcome a utility for each player. Instead of, (𝐴, u) will also write (𝐴 1 , ..., 𝐴 𝑛 , 𝑢 1 , ..., 𝑢 𝑛 ). We say that 𝑎 𝑖 ∈ 𝐴 𝑖 strictly dominates 𝑎 ′ 𝑖 ∈ 𝐴 𝑖 if for all 𝑎 −𝑖 ∈ 𝐴 −𝑖 , 𝑢 𝑖 (𝑎 𝑖 , 𝑎 −𝑖 ) > 𝑢 𝑖 (𝑎 ′ 𝑖 , 𝑎 −𝑖 ). For any given game Γ = (𝐴, u), we will call any game Γ ′ = (𝐴 ′ , u ′ ) a subset game of Γ if 𝐴 ′ 𝑖 ⊆ 𝐴 𝑖 for 𝑖 = 1, ..., 𝑛. Note that a subset game may assign different utilities to outcomes than the original game. For any set of strategies 𝑆, we denote by Γ − 𝑆 ((𝐴 1 − 𝑆) × ...., (𝐴 𝑛 − 𝑆), u) the game that arises from Γ by removing the strategies 𝑆 for all players. We say that some utility vector y ∈ R 𝑛 is a Pareto-improvement on (or is Pareto-better than) y ′ ∈ R 𝑛 if 𝑦 𝑖 ≥ 𝑦 ′ 𝑖 for 𝑖 = 1, ..., 𝑛. We will also denote this by y ≥ y ′ . Note that, contrary to convention, we allow y = y ′ . Whenever we require one of the inequalities to be strict, we will say that y is a strict Pareto improvement on y ′ . In a given game, we will also say that an outcome a is a Paretoimprovement on another outcome a ′ if u(a) ≥ u(a ′ ). We say that y is Pareto-optimal or Pareto-efficient relative to some 𝑆 ⊂ R 𝑛 if there is no element of 𝑆 that strictly Pareto-dominates y. The Demand Game of Table  1  is an example of a game that we will use throughout this paper. As noted earlier, DM and RM strictly dominate DL and RL. The game of Table  2  is a subset game of the Demand Game.
 Multivalued functions For sets 𝑀 and 𝑁 , a multi-valued function Φ : 𝑀 ⊸ 𝑁 is a function which maps each element 𝑚 ∈ 𝑀 to a set Φ(𝑚) ⊆ 𝑁 . For a subset 𝑄 ⊆ 𝑀, we define Φ(𝑄) 𝑚 ∈𝑄 Φ(𝑚). Note that Φ(𝑄) ⊆ 𝑁 and that Φ(∅) = ∅. For any set 𝑀, we define the identity function id 𝑀 : 𝑀 ⊸ 𝑀 : 𝑚 → {𝑚}. Also, for two sets 𝑀, 𝑁 , we define all 𝑀,𝑁 : 𝑀 ⊸ 𝑁 : 𝑚 ↦ → 𝑁 . We define the inverse Φ −1 : 𝑁 ⊸ 𝑀 : 𝑛 ↦ → {𝑚 ∈ 𝑀 | 𝑛 ∈ Φ(𝑚)}. Note that Φ −1 (∅) = ∅ for any multi-valued function Φ. For sets 𝑀, 𝑁 , 𝑄 and functions Φ : 𝑀 ⊸ 𝑁 , Ψ : 𝑁 ⊸ 𝑄, we define the composite Ψ • Φ : 𝑀 ⊸ 𝑄 : 𝑚 ↦ → Ψ(Φ(𝑚)). As with regular functions, composition of multi-valued functions is associative. We say that Φ : 𝑀 ⊸ 𝑁 is single-valued if |Φ(𝑚)| = 1 for all 𝑚 ∈ 𝑀. Whenever a multi-valued function is single-valued, we can apply many of the terms for regular functions. For example, we will take injectivity, surjectivity, and bijectivity for single-valued functions to have the usual meaning. We will never apply these notions to non-single-valued functions.
 DELEGATION AND SAFE PARETO IMPROVEMENTS We consider a setting in which a given game Γ is played through what we will call representatives. For example, the representatives could be humans whose behavior is determined or incentivized by some contract à la the principal-agent literature  [14] . We imagine that one way in which the representatives can be instructed is to in turn play a subset game Γ 𝑠 = (𝐴 𝑠 1 ⊆ 𝐴 1 , ..., 𝐴 𝑠 𝑛 ⊆ 𝐴 𝑛 , u 𝑠 ) of the original game, without necessarily specifying a strategy or algorithm for solving such a game. We emphasize, again, that u 𝑠 is allowed to be a vector of entirely different utility functions. For any subset game Γ 𝑠 , we denote by Π(Γ 𝑠 ) the outcome that arises if the representatives play the subset game Γ 𝑠 of Γ. Because in many games, it is not clear what the right choice is, the original players might be uncertain about Π(Γ 𝑠 ) for many games Γ 𝑠 . We will therefore model each Π(Γ 𝑠 ) as a random variable. The original players trust their representatives to the extent that we take Π(Γ) to be a default way for the game to played for any Γ. For example, in the Game of Chicken, it is not clear what the right action is. Thus, if one can simply delegate the decision to someone with more relevant expertise, that is the first option one would consider. We are interested in whether and how the original players can jointly Pareto-improve on the default. Of course, one option is to compute the expected utilities in the default (E [u(Π(Γ))]) and then let the representatives play a distribution over outcomes whose expected utility exceeds that default expected utility. However, this is unrealistic if Γ is a complex game with multiple Nash equilibria. For one, the precise point of delegation is that the original players are unable or unwilling to properly evaluate Γ. Second, there is no widely agreed upon, universal procedure for selecting an action in the face of equilibrium selection problems. We address this problem in a typical way. Essentially, we require of any attempted improvement that it incurs no regret in the worstcase. That is, we are interested in subset games Γ 𝑠 that are Pareto improvements with certainty under weak and purely qualitative assumptions about Π. Definition 3.1. Let Γ 𝑠 be a subset game of Γ. We say Γ 𝑠 is a safe Pareto improvement (SPI) on Γ if u(Π(Γ 𝑠 )) ≥ u(Π(Γ)) with certainty. We say that Γ 𝑠 is a strict SPI if furthermore, there is a player 𝑖 s.t. 𝑢 𝑖 (Π(Γ 𝑠 )) > 𝑢 𝑖 (Π(Γ 𝑠 )) with positive probability.
 PROGRAM EQUILIBRIUM So far, we have been vague about the details of the strategic situation that the original players face in instructing their representatives. From what set of actions can they choose? How can they jointly let the representatives play some new subset game Γ 𝑠 ? Are SPIs Nash equilibria of the meta game played by the representatives? In this section, we briefly describe one way to fill this gap by discussing the concept of program games and program equilibrium  [3, 6, 8, 17, 25] . This section is essential to understanding why SPIs are relevant. However, the remaining technical content of this paper does not rely on this section and the main ideas presented here are straightforward from previous work. For more detail, see Appendix A. For any game Γ = (𝐴, u), the program equilibrium literature considers the following meta game. First, each player 𝑖 chooses from a set of computer programs. Each program then receives as input a vector containing everyone else's chosen program. Each player 𝑖's program then returns an action from 𝐴 𝑖 , player 𝑖's set of actions in Γ. Together these actions then form an outcome a ∈ 𝐴 of the original game. Finally, the utilities u(a) are realized according to the utility function of Γ. The meta game can be analyzed like any other game. Its Nash equilibria are called program equilibria. Importantly, the program equilibria can implement payoffs not implemented by any Nash equilibria of Γ itself. For example, in the Prisoner's Dilemma, both players can submit a program that says: "If the opponent's chosen computer program is equal to this computer program, Cooperate; otherwise Defect. "  [11, 15, 25]  This is a program equilibrium which implements mutual cooperation. In the setting for our paper, we similarly imagine that each player 𝑖 can choose from a set of programs that in turn choose from 𝐴 𝑖 . However, the types of program that we have in mind here are more sophisticated than those typically considered in the program equilibrium literature. Specifically we imagine that the programs are executed by intelligent representatives who are themselves able to competently choose an action for player 𝑖 in any given game Γ 𝑠 , without the original player having to describe how this choice is to be made. The original player may not even understand much about this program other than that it generally plays well. Thus, in addition to the elementary instructions used in a typical computer program (branches, comparisons, arithmetic operations, etc.), we allow player 𝑖 to use an instruction "Play Π 𝑖 (Γ 𝑠 )" in the program she submits. To jointly let the representatives play, e.g., the SPI Γ 𝑠 of Table  2  on the Demand Game of Table  1 , the representatives can both use an instruction that says, "If the opponent's chosen program is analogous to this one, play Π 𝑖 (Γ 𝑠 ); otherwise play DM". Assuming some minimal rationality requirements on the representatives (i.e., on how "play Π 𝑖 (Γ 𝑠 )" is implemented), this is a Nash equilibrium. More generally, we can prove the following result (see Appendix A for a proof). Theorem 4.1. Let Γ be a game and Γ 𝑠 be an SPI of Γ. Now consider a program game on Γ, where each player 𝑖 can choose from a set of computer programs that output actions for Γ. In addition to the normal kind of instructions, we allow the use of the command "play Π 𝑖 (Γ ′ )" for any subset game Γ ′ of Γ. Finally, assume that Π(Γ) guarantees each player 𝑖 at least that player's minimax utility (a.k.a. threat point) in the base game Γ. Then Π(Γ 𝑠 ) is played in a program equilibrium, i.e., in a Nash equilibrium of the program game. As an alternative to having the original players choose contracts separately, we could imagine the use of jointly signed contracts which only come into effect once signed by all players [cf.  12, 16] . Also compare earlier work by Sen  [24]  and  [21] , which we discuss in Appendix B.
 OUTCOME CORRESPONDENCE BETWEEN GAMES In this section, we introduce a notion of outcome correspondence, which we will see is essential to constructing SPIs. Definition 5.1. Consider two games Γ = (𝐴 1 , ..., 𝐴 𝑛 , u) and Γ ′ = (𝐴 ′ 1 , ..., 𝐴 ′ 𝑛 , u ′ ). We write Γ ∼ Φ Γ ′ for Φ : 𝐴 ⊸ 𝐴 ′ if Π(Γ ′ ) ∈ Φ(Π(Γ)) with certainty. Note that Γ ∼ Φ Γ ′ is a statement about Π, i.e., about how the representatives choose. Whether such a statement holds generally depends on the specific representatives being used. In Section 7, we describe two general circumstances under which it seems plausible that Γ ∼ Φ Γ ′ . For example, if two games Γ and Γ ′ are isomorphic, then one might expect Γ ∼ Φ Γ ′ , where Φ is constructed from the 𝑛 isomorphisms of the particular action spaces. We now state some basic facts about the relation ∼, many of which we will use throughout this paper. Lemma 5.2. Let Γ = (𝐴, u), Γ ′ = (𝐴 ′ , u ′ ), Γ = ( Â, û) and Φ, Ξ : 𝐴 ⊸ 𝐴 ′ , Ψ : 𝐴 ′ ⊸ Â. (1) Reflexivity: Γ ∼ id 𝐴 Γ, where id 𝐴 : 𝐴 ⊸ 𝐴 : a ↦ → {a}. (2) Symmetry: If Γ ∼ Φ Γ ′ , then Γ ′ ∼ Φ −1 Γ. ( 3 ) Transitivity: If Γ ∼ Φ Γ ′ and Γ ′ ∼ Ψ Γ, then Γ ∼ Ψ•Φ Γ. (4) If Γ ∼ Φ Γ ′ and Φ(a) ⊆ Ξ(a) for all a ∈ 𝐴, then Γ ∼ Ξ Γ ′ . ( 5 ) Γ ∼ all 𝐴,𝐴 ′ Γ ′ , where all 𝐴,𝐴 ′ : 𝐴 ⊸ 𝐴 ′ : a ↦ → 𝐴 ′ . ( 6 ) If Γ ∼ Φ Γ ′ and Φ(a) = ∅, then Π(Γ) ≠ a with certainty. For completness, we prove these in Appendix C. Items 1-3 show that ∼ has properties resembling those of an equivalence relation. Note, however, that since ∼ is not a binary relationship, ∼ itself cannot be an equivalence relation in the usual sense. Item 4 states that we can make an outcome correspondence claim less precise and it will still hold true. Item 5 states that in the extreme, it is always Γ ∼ all 𝐴,𝐴 ′ Γ ′ , where all 𝐴,𝐴 ′ is the trivial, maximally imprecise outcome correspondence function that confers no information. Item 6 shows that ∼ can be used to express the elimination of outcomes, i.e., the belief that a particular outcome (or strategy) will never occur.
 SAFE PARETO IMPROVEMENTS THROUGH OUTCOME CORRESPONDENCE We now show that as advertised, outcome correspondence is closely tied to SPIs. The following theorem shows not only how outcome correspondences can be used to find (and prove) SPIs. It also shows that any SPI requires an outcome correspondence relation with what we will call a Pareto-improving correspondence function. Theorem 6.1. Let Γ = (𝐴, u) be a game and Γ 𝑠 = (𝐴 𝑠 , u 𝑠 ) be a subset game of Γ. Then Γ 𝑠 is an SPI on Γ if and only if there is Φ such that Γ ∼ Φ Γ 𝑠 and for all a ∈ 𝐴 it is for all a 𝑠 ∈ Φ(a) the case that u(a 𝑠 ) ≥ u(a). Proof. ⇐: By definition, Π(Γ 𝑠 ) ∈ Φ(Π(Γ)) with certainty. Hence, for 𝑖 = 1, 2, 𝑢 𝑖 (Π(Γ 𝑠 )) ∈ 𝑢 𝑖 (Φ(Π(Γ))) with certainty. Hence, by as- sumption about Φ, with certainty, 𝑢 𝑖 (Π(Γ 𝑠 )) ≥ 𝑢 𝑖 (Π(Γ)). ⇒: Assume that 𝑢 𝑖 (Π(Γ)) ≥ 𝑢 𝑖 (Π(Γ 𝑠 )) with certainty for 𝑖 = 1, 2. We define Φ : 𝐴 → 𝐴 𝑠 : a ↦ → {a 𝑠 ∈ 𝐴 𝑠 | u(a 𝑠 ) ≥ u(a)}. It is immediately obvious that Φ is Pareto-improving as required. Also, whenever Π(Γ) = a and Π(Γ 𝑠 ) = a 𝑠 for any a ∈ 𝐴 and a 𝑠 ∈ 𝐴 𝑠 , it is (by assumption) with certainty u(a 𝑠 ) ≥ u(a). Thus, by definition of Φ, it holds that a 𝑠 ∈ Φ(a). We conclude that Γ ∼ Φ Γ 𝑠 as claimed. □ Note that the theorem concerns weak SPIs and therefore allows the case where with certainty u(Π(Γ)) = u(Π(Γ 𝑠 )). To show that some Γ 𝑠 is a strict SPI, we need additional information about which outcomes occur with positive probability. We now illustrate how outcome correspondences can be used to derive the SPI for the Demand Game from the introduction as per Theorem 6.1. Of course, at this point we do not have any assumptions about when games are equivalent. We will introduce some in the following section. Nevertheless, we can already sketch the argument. Let Γ be the Demand Game of Table  1 . First, it seems plausible that Γ is in some sense equivalent to Γ ′ , where Γ ′ = Γ − {DL, RL} is the game that results from removing DL and RL for both players from Γ. Again, strict dominance could be given as an argument. We can formalize this as Γ ∼ Φ Γ ′ , where Φ(𝑎 1 , 𝑎 2 ) = {(𝑎 1 , 𝑎 2 )} if 𝑎 1 , 𝑎 2 ∈ {DM, RM} and Φ(𝑎 1 , 𝑎 2 ) = ∅ otherwise. In a second step, it seems plausible that Γ ′ ∼ Ψ Γ 𝑠 , where Γ 𝑠 is the game of Table  2  and Ψ is the isomorphism between Γ ′ and Γ 𝑠 . Finally, we can use transitivity to obtain Γ ∼ Ψ•Φ Γ 𝑠 . To see that Ψ • Φ is Paretoimproving for the original utility functions of Γ, notice that Φ does not change utilities at all. Ψ maps the conflict outcome (DM, DM) onto the outcome (DL, DL), which is better for both original players. Other than that, Ψ, too, does not change the utilities. Hence, Ψ • Φ is Pareto-improving. By Theorem 6.1, Γ 𝑠 is therefore an SPI on Γ. In principle, Theorem 6.1 does not hinge on Π(Γ) and Π(Γ 𝑠 ) resulting from playing games. An analogous result holds for any random variables over 𝐴 and 𝐴 𝑠 . In particular, this means that Theorem 6.1 applies also if the representatives receive other kinds of instructions (cf. Section 4). However, it seems hard to establish nontrivial outcome correspondences between Π(Γ) and other types of instructions. Still, the use of more complicated instructions can be used to derive different kinds of SPIs. For example, if there are different game SPIs, then the original players could tell their representatives to randomize between them in a coordinated way.
 ASSUMPTIONS ABOUT OUTCOME CORRESPONDENCE To make any claims about how the original players should play the meta-game, i.e., about what instructions they should submit, we have to make assumptions about how the representatives choose and (by Theorem 6.1) about outcome correspondence in particular. We here make two fairly weak assumptions. The first is that the representatives play two isomorphic games isomorphically. Assumption 7.1. Let Γ = (𝐴, u) and Γ ′ = (𝐴 ′ , u ′ ) be two games for which there are single-valued bijections Φ 𝑖 : 𝐴 𝑖 → 𝐴 ′ 𝑖 for 𝑖 = 1, ..., 𝑛 such that u(𝑎 1 , ..., 𝑎 𝑛 ) = u ′ (Φ 1 (𝑎 1 ), ..., Φ 𝑛 (𝑎 𝑛 )) for all a ∈ 𝐴. Then for one tuple Φ of such bijections, Γ ∼ Φ Γ ′ . Similar desiderata have been discussed in the context of equilibrium selection, e.g., by Harsanyi and Selten  [10, Chapter 3.4 ]. In fact, they consider a generalization in which the utilities are allowed to be linear transformations of each other. Although this generalization is extremely plausible, we omit it here for simplicity. One could criticize Assumption 7.1 by referring to focal points (introduced by Schelling [22, pp. 54-58]) as an example where context and labels of strategies matter. A possible response might be that in games where context plays a role, that context should be included as additional information and not be considered part of (𝐴, u). Assumption 7.1 would then either not apply to such games with (relevant) context or would require one to, in some way, translate the context along with the strategies. However, in this paper we will not formalize context, and assume that there is no decisionrelevant context. Assumption 7.2. Let Γ = (𝐴, u) be an arbitrary 𝑛-player game where 𝐴 1 , ..., 𝐴 𝑛 are pairwise disjoint, and ã𝑖 ∈ 𝐴 𝑖 be strictly dominated by some other strategy â𝑖 ∈ 𝐴 𝑖 . Then Γ ∼ Φ Γ − { ã𝑖 }, where for all 𝑎 −𝑖 ∈ 𝐴 −𝑖 , Φ( ã𝑖 , 𝑎 −𝑖 ) = ∅ and Φ(𝑎 𝑖 , 𝑎 −𝑖 ) = {(𝑎 𝑖 , 𝑎 −𝑖 )} whenever 𝑎 𝑖 ≠ ã𝑖 . Assumption 7.2 expresses that representatives should never play strictly dominated strategies. Moreover, it states that we can remove strictly dominated strategies from a game and the resulting game will be played in the same way by the representatives. For example, this implies that when evaluating a strategy 𝑎 𝑖 , the representatives do not take into account how many other strategies 𝑎 𝑖 strictly dominates. Assumption 7.2 also allows (via Transitivity of ∼ as per Lemma 5.2.3) the iterated removal of strictly dominated strategies. The notion that we can (iteratively) remove strictly dominated strategies is common in game theory  [13, 18, 19 , Section 2.9, Chapter 12] and has rarely been questioned. It is also implicit in the solution concept of Nash equilibrium -if a strategy is removed by iterated strict dominance, that strategy is played in no Nash equilibrium. However, like the concept of Nash equilibrium, the elimination of strictly dominated strategies becomes implausible if the game is not played in the usual way. In particular, for Assumption 7.2 to hold, we will in most games Γ have to assume that the representatives cannot in turn make credible precommitments (or delegate to further subrepresentatives) or play the game iteratively  [2] . With  Γ. Further, if 𝑃 (Π(Γ)=(DM, DM)) > 0, Γ 𝑠 is a strict SPI.
 COMPUTING SAFE PARETO IMPROVEMENTS In this section, we ask how computationally costly it is for the original players to identify for a given game Γ a non-trivial SPI Γ 𝑠 . In particular, we ask whether a given game Γ has a non-trivial SPI that can be proved using only Assumptions 7.1 and 7.2, Transitivity (Lemma 5.2.3) and Theorem 6.1. Formally: Definition 8.1. The SPI decision problem consists in deciding for any given Γ, whether there is a sequence of outcome correspondences Φ 1 , ..., Φ 𝑘 and a sequence of subset games Γ 0 = Γ, Γ 1 , ..., Γ 𝑘 of Γ s.t.: (1) (Non-triviality:) If we fully reduce Γ 𝑘 and Γ using iterated strict dominance (Assumption 7.2), the two resulting games are not equal. (Of course, they are allowed to be isomorphic.) (2) For 𝑖 = 1, ..., 𝑘, Γ 𝑖−1 ∼ Φ 𝑖 Γ 𝑖 is valid by a single application of either Assumption 7.1 or Assumption 7.2. (3) For all a ∈ 𝐴, and whenever a 𝑠 ∈ (Φ 𝑘 • Φ 𝑘−1 • ... • Φ 1 ) (a), it is the case that 𝑢 (a 𝑠 ) ≥ u(a). For the strict SPI decision problem, we further require: (4.) There is a player 𝑖 and an outcome a that survives iterated elimination of strictly dominated strategies from Γ s.t. 𝑢 𝑖 ((Φ 𝑘 • Φ 𝑘−1 • ... • Φ 1 )(a)) > 𝑢 𝑖 (a). Many variants of this problem may be considered. For example, we might generalize it to allow imposing additional properties on the SPI. This will generally not change the computational complexity of the problem. One may also wish to compute all SPIs, or -in line with multi-criteria optimization  [7, 27]  -all SPIs that cannot in turn be safely improved upon. However, in general there may exist exponentially many such SPIs. To retain any hope of developing an efficient algorithm, one would therefore have to first develop a more efficient representation scheme [cf.  20  The full proof is tedious (see Appendix E), but the main idea is simple. To find an SPI on Γ based on Assumptions 7.1 and 7.2, one has to first iteratively remove all strictly dominated actions to obtain a reduced game Γ ′ , which the representatives would play the same as the original game. This can be done in polynomial time. One then has to map the actions Γ ′ onto the original Γ in such a way that each outcome in Γ ′ is mapped onto a weakly Pareto-better outcome in Γ. Our proof of NP-hardness works by reducing from the subgraph isomorphism problem, where the payoff matrices of Γ ′ , Γ represent the adjacency matrices of the graphs. Besides being about a specific set of assumptions about ∼, note that Theorem 8.2 and Proposition 8.3 also assume that the utility function of the game is represented explicitly in normal form as a payoff matrix. If we changed the game representation (e.g., to boolean circuits, extensive form game trees, quantified boolean formulas, or even Turing machines), this can affect the complexity of the SPI problem [cf. 9]. In fact, even reducing a game using strict dominance by pure strategies -which contributes only insignificantly to the complexity of the SPI problem for normal-form games -is difficult in some game representations [4, Section 6].
 SAFE PARETO IMPROVEMENTS UNDER IMPROVED COORDINATION In this section, we imagine that the players are able to simply invent new token strategies with new payoffs that arise from mixing existing feasible payoffs. To define this formally, we first define for any game Γ = (𝐴, u), C(Γ) u(Δ(𝐴)) = a∈𝐴 𝑝 a u(a) ∀a∈𝐴:𝑝 a ∈[0, 1], a∈𝐴 𝑝 a = 1 to be the set of feasible coordinated payoff vectors of Γ, which is exactly the convex closure of u(𝐴), i.e., of the deterministically achievable utilities of the original game. For any game Γ, we then imagine that in addition to subset games, the players can let the representatives play a perfect-coordination token game (𝐴 𝑠 , u 𝑠 , u 𝑒 ), where for all 𝑖, 𝐴 𝑠 𝑖 ∩𝐴 𝑖 = ∅ and 𝑢 𝑠 𝑖 : 𝐴 𝑠 → R are arbitrary utility functions to be used by the representatives and u 𝑒 : 𝐴 𝑠 → C(Γ) are the utilities that the original players assign to the token strategies. The instruction (𝐴 𝑠 , u 𝑠 , u 𝑒 ) lets the representatives play the game (𝐴 𝑠 , u 𝑠 ) as usual. However, the strategies 𝐴 𝑠 are imagined to be meaningless token strategies which do not resolve the given game Γ. Once some token strategies a 𝑠 are selected, these are translated into some probability distribution over 𝐴, i.e., over outcomes of the original game, thus giving rise to (expected) utilities u 𝑒 (a 𝑠 ) ∈ C(Γ). These distributions and thus utilities are specified by the original players. We here imagine in our definition of C(Γ) that these distributions over 𝐴 could require the representatives to correlate their choices for the original game for any given a 𝑠 . Definition 9.1. Let Γ be a game. A perfect-coordination SPI for Γ is a perfect-coordination token game (𝐴 𝑠 , u 𝑠 , u 𝑒 ) for Γ s.t. u 𝑒 (Π(𝐴 𝑠 , 𝑢 𝑠 )) ≥ u(Π(Γ)) with certainty. We call (𝐴 𝑠 , u 𝑠 , u 𝑒 ) a strict perfect-coordination SPI if there furthermore is a player 𝑖 for whom 𝑢 𝑒 𝑖 (Π(𝐴 𝑠 , 𝑢 𝑠 )) > 𝑢 𝑖 (Π(Γ)) with positive probability. As an example, imagine that Γ is just the DM-RM subset game of the Demand Game of Table  1 . Then, intuitively, an SPI under improved coordination could consist of the original players telling the representatives, "Play as if you were playing the DM-RM subset game of the Demand Game, but whenever you find yourself playing (DM, DM), randomize [according to some given distribution] between the other (Pareto-optimal) outcomes instead". Formally, 𝐴 𝑠 1 = { D, R}, 𝐴 𝑠 2 = { D, R} would then consist of tokenized versions of the original strategies. The utility functions 𝑢 𝑠 1 , 𝑢 𝑠 2 are then simply the same as in the original Demand Game except that they are applied to the token strategies. E.g., u 𝑠 ( D, R) = (2, 0). The utilities for the original players remove the conflict outcome. For example, the original players might specify u 𝑒 ( D, D) = (1, 1), representing that the representatives are supposed to play (RM, RM) in the ( D, D) case. For all other outcomes ( â1 , â2 ), it must be the case that u 𝑒 ( â1 , â2 ) = u 𝑠 ( â1 , â2 ) because the other outcomes cannot be Pareto-improved upon. As with our earlier SPIs for the Demand Game, Assumption 7.1 implies that Γ ∼ Φ Γ 𝑠 , where Φ maps the original conflict outcome (DM, DM) onto the Pareto-optimal ( D, D). Relative to the SPIs considered up until now, these new types of instructions put significant additional requirements on how the representatives interact. They now have to engage in a two-round process of first choosing and observing one another's token strategies and then playing the corresponding distribution over outcomes from the original game. Further, it must be the case that this additional coordination does not affect the payoffs of the original outcomes. The latter may not be the case in, e.g., the Game of Chicken. That is, we could imagine a Game of Chicken in which coordination is possible but that the rewards of the game change if the players do coordinate. After all, the underlying story in the Game of Chicken is that the positive reward (admiration from peers) is attained precisely for accepting a grave risk. With these more powerful ways to instruct representatives, we can now replace individual outcomes of the default game ad libitum. For example, in the reduced Demand Game, we singled out the outcome (DM, DM) as Pareto-suboptimal and replaced it by a Pareto-optimal outcome, while keeping all other outcomes the same. This allows us to construct SPIs in many more games then before. Definition 9.2. The strict full-coordination SPI decision problem consists in deciding for any given Γ whether under Assumption 7.1 there is a perfect-coordination SPI Γ 𝑠 for Γ. Lemma 9.3. For a given 𝑛-player game Γ and payoff vector y ∈ R 𝑛 , it can be decided by linear programming and thus in polynomial time whether y is Pareto-optimal in C(Γ). For completeness the linear program is given in Appendix F. Based on Lemma 9.3, Algorithm 1 decides whether there is a strict perfect-coordination SPI for a given game Γ. It is easy to see that this algorithm runs in polynomial time (in the size of, e.g., the normal form representation of the game). It is also correct: if it returns True, simply replace the Pareto-suboptimal outcome while keeping all other outcomes the same; if it returns False, then all outcomes are Pareto-optimal within C(Γ) and so there can be no strict SPI. We summarize this result in the following proposition. From the problem of deciding whether there are strict SPIs under improved coordination at all, we move on to the question of what different perfect-coordination SPIs there are. In particular, one might ask what the cost is of only considering safe Pareto improvements relative to acting on a probability distribution over Π(Γ) and the resulting expected utilities E [u(Π(Γ))]. We start with a lemma that directly provides a characterization. So far, all the considered perfect-coordination SPIs (𝐴 𝑠 , u 𝑠 , u 𝑒 ) for a game (𝐴, u) have consisted in letting the representatives play a game (𝐴 𝑠 , u 𝑠 ) that is isomorphic to the original game, but Pareto-improves (from the original players' perspectives, i.e., u 𝑒 ) at least one of the outcomes. It turns out that we can restrict attention to this very simple type of SPI under improved coordination. Lemma 9.5. Let Γ = ({𝑎 is also an SPI on Γ, with E u(Π(Γ 𝑠 )) Π(Γ)=a = E u(Π(Γ ′ )) Π(Γ)=a for all a ∈ 𝐴 and consequently E [u(Π(Γ 𝑠 ))] = E [u(Π(Γ ′ ))]. We prove this in Appendix G. Because of this result, we will focus on these particular types of SPIs, which simply create an isomorphic game with different (Pareto-better) utilities. Note, however, that without assigning exact probabilities to the distributions of Π(Γ), Π(Γ ′ ), the original players will in general not be able to construct a Γ 𝑠 that satisfies the expected payoff equalities. For this reason, one could still conceive of situations in which a different type of SPI would be chosen by the original players and the original players are unable to instead choose an SPI of the type described in Lemma 9.5. Lemma 9.5 directly implies a characterization of the expected utilities that can be achieved with perfect-coordination SPIs. Of course, this characterization depends on the exact distribution of Π(Γ). We omit the statement of this result. However, we state the following implication. Corollary 9.6. Under Assumption 7.1, the set of Pareto improvements that are safely achievable with perfect coordination {E[u(Γ ′ )] | Γ ′ is perfect-coordination SPI on Γ} is a convex polygon. Because of this result, one can also efficiently optimize convex functions over the set of perfect-coordination SPIs. Even without referring to the distribution Π(Γ), many interesting questions can be answered efficiently. For example, we can efficiently identify the perfect-coordination SPI that maximizes the minimum improvements across players and outcomes a ∈ 𝐴. In the following, we aim to use Lemma 9.5 and Corollary 9.6 to give maximally strong positive results about what Pareto improvements can be safely achieved, without referring to exact probabilities over Π(Γ). To keep things simple, we will do this only for the case of two players. To state our results, we first need some notation: We use PF(C) y ∈ C ∄y ′ ∈C, 𝑖 ∈ {1, ..., 𝑛} : y ′ ≥ y, 𝑦 ′ 𝑖 > 𝑦 to denote the Pareto frontier of a convex polygon C (or more generally convex, closed set). For any real number 𝑥 ∈ R, we use 𝜋 𝑖 (𝑥, C(Γ)) to denote the y ′ ∈ C(Γ) which maximizes 𝑦 ′ −𝑖 under the constraint 𝑦 ′ 𝑖 = 𝑥. (Recall that we consider 2-player games, so 𝑦 ′ −𝑖 is a single real number.) Note that such a y ′ exists if and only if 𝑥 is 𝑖's utility in some feasible payoff vector. We first state our result formally. Afterwards, we will give a graphical explanation of the result, which we believe is easier to understand. Then there is an SPI under improved coordination Γ 𝑠 such that E [u(Π(Γ 𝑠 ))] = y. B) If there is no element in C(Γ) which Pareto-dominates all of supp(Π(Γ)) and if y is Pareto-dominated by an element each of 𝐿 1 and 𝐿 3 as defined above, then there is a perfect-coordination SPI Γ 𝑠 such that E [u(Π(Γ 𝑠 ))] = y. We now illustrate the result graphically. We start with Case A, which is illustrated in Figure  1 . The Pareto-frontier is the solid line in the north and east. The points marked x indicate outcomes in supp(Π(Γ)). The point marked by a filled circle indicates the expected value of the default equilibrium E [u(Π(Γ))]. For some y ∈ R 2 to be a Pareto-improvement, it must be to the north-east of the filled circle. The vertical dashed lines starting at the two extreme x marks illustrate the application of 𝜋 1 to project 𝑥 min/max 1 onto the Pareto frontier. The dotted line between these two points is 𝐿 1 . Similarly, the horizontal dashed lines starting at x marks illustrate the application of 𝜋 2 to project 𝑥 min/max 2 onto the Pareto frontier. The line segment between these two points is 𝐿 3 . In this case, this line segments lies on the Pareto frontier. The set 𝐿 2 is simply that part of the Pareto frontier, which Pareto-dominates all elements of supp(Π(Γ)), i.e., the part of the Pareto frontier to the north-east  between the two intersections with the northern horizontal dashed line and eastern vertical dashed line. E [𝑢 1 (Π(Γ 𝑠 ))] E [𝑢 1 (Π(Γ)))] E [𝑢 2 (Π(Γ)))] E [𝑢 2 (Π(Γ 𝑠 ))] E [𝑢 1 (Π(Γ 𝑠 ))] E [𝑢 1 (Π(Γ))] E [𝑢 2 (Π(Γ))]) E [𝑢 2 (Π(Γ))] Case B of Theorem 9.7 is depicted in Figure  2 . Note that here the two line segments 𝐿 1 and 𝐿 3 intersect. To ensure that a Pareto improvement is safely achievable, the theorem requires that it is below both of these lines. Theorem 9.7 is proven by re-mapping each of the outcomes of the original game as per Lemma 9.5. For example, the projection of the default equilibrium E [u(Π(Γ))] (i.e., the filled circle) onto 𝐿 1 is obtained as an SPI by projecting all the outcomes (i.e., all the x marks) onto 𝐿 1 . In Case A, any utility vector y ∈ 𝐿 2 that Paretoimproves on all outcomes of the original game can be obtained by re-mapping all outcomes onto y. Other kinds of y are handled similarly. For a full proof, see Appendix H. As a corollary of Theorem 9.7, we can see that all (potentially unsafe) Pareto improvements in the DM-RM subset game of the Demand Game of Table  1  are equivalent to some perfect-coordination SPI. However, this is not always the case: Proposition 9.8. There is a game Γ = (𝐴, u), representatives Π that satisfy Assumptions 7.1 and 7.2, and an outcome a ∈ 𝐴 s.t. 𝑢 𝑖 (a) > E [𝑢 𝑖 (Π(Γ))] for all players 𝑖, but there is no perfectcoordination SPI (𝐴 𝑠 , u 𝑠 , u 𝑒 ) s.t. for all players 𝑖, E 𝑢 𝑒 𝑖 (Π(𝐴 𝑠 , u 𝑠 )) = 𝑢 𝑖 (a). We prove this in Appendix I.
 THE SPI SELECTION PROBLEM In the Demand Game, there happens to be a single non-trivial SPI. However, in general (even without the type of coordination assumed in Section 9) there may be multiple incomparable SPIs that result in different payoffs for the players. If multiple SPIs are available, the original players would be left with the difficult decision of which SPI to demand in their instruction. This difficulty of choosing what SPI to demand cannot be denied. However, we would here like to emphasize that players can profit from the use of SPIs even without addressing this SPI selection problem. To do so, a player picks an instruction that is very compliant ("dove-ish") w.r.t. what SPI is chosen, e.g., one that simply goes with whatever SPI the other players demand as long as that SPI cannot further be safely Pareto-improved upon. In many cases, all such SPIs benefit both players. For example, SPIs in bargaining scenarios like the Demand Game remove the conflict outcome, which benefits all parties. Thus, a player can expect a safe improvement even under such maximally compliant demands on the selected SPI.
 CONCLUSION AND FUTURE DIRECTIONS Safe Pareto improvements are a promising new idea for delegating strategic decision making. To conclude this paper, we discuss some ideas for further research on SPIs. Straightforward technical questions arise in the context of the complexity results of Section 8. First, what impact on the complexity does varying the assumptions have? Our NP-completeness proof is easy to generalize at least to some other types of assumptions. It would be interesting to give a generic version of the result. We also wonder whether there are plausible assumptions under which the complexity changes in interesting ways. Second, one could ask how the complexity changes if we use more sophisticated game representations (see the remarks at the end of that section). Third, one could impose additional restrictions on the sought SPI. For example, some of the players may be unable to have their representative maximize arbitrary utility functions. We could then ask whether there is an SPI in which only a given subset of the players adopt different utility functions and restrictions on the set of available strategies. Fourth, we could restrict the games under consideration. Are there games in which it becomes easy to decide whether there is an SPI? It would also be interesting to see what real-world situations can already be interpreted as utilizing SPIs, or could be Pareto-improved upon using SPIs.
 A PROOF OF THEOREM 4.1 -PROGRAM EQUILIBRIUM IMPLEMENTATIONS OF SAFE PARETO IMPROVEMENTS This paper considers the meta-game of delegation. SPIs are a proposed way of playing these games. However, throughout most of this paper, we do not analyze the meta-game directly as a game using the typical tools of game theory. We here fill that gap and in particular prove Theorem 4.1, which shows that SPIs are played in Nash equilibria of the meta game, assuming sufficiently strong contracting abilities. As noted, this result is essential. However, since it is mostly an application of existing ideas from the literature on program equilibrium, we left a detailed treatment out of the main text. A program game for Γ = (𝐴, u) is defined via a set PROG = PROG 1 ×...×PROG 𝑛 and a non-deterministic mapping exec : PROG 1 × ... × PROG 𝑛 ⇝ 𝐴. We obtain a new game with action sets PROG and utility function 𝑈 : PROG → R 𝑛 : c ↦ → E [u(exec(c))] . (1) Though this definition is generic, one generally imagines in the program equilibrium literature that for all 𝑖, PROG 𝑖 consists of computer programs in some programming language, such as Lisp, that take as input vectors in PROG and return an action 𝑎 𝑖 . The function exec on input c ∈ PROG then executes each player 𝑖's program 𝑐 𝑖 on c to assign 𝑖 an action. The definition implicitly assumes that PROG only contains programs that halt when fed one another as input. A program equilibrium is then simply a Nash equilibrium of the program game. For the present paper, we add the following feature to the underlying programming language. A program can call a "black box subroutine" Π 𝑖 (Γ ′ ) for any subset game Γ ′ of Γ, where Π 𝑖 (Γ ′ ) is a random variable over 𝐴 ′ 𝑖 and Π(Γ ′ ) = (Π 1 (Γ ′ ), ..., Π 𝑛 (Γ ′ )). We need one more definition. For any game Γ and player 𝑖, we define Player 𝑖's threat point (a.k.a. minimax utility) 𝑣 Γ 𝑖 as 𝑣 Γ 𝑖 = min 𝝈 −𝑖 ∈ 𝑗 ≠𝑖 Δ(𝐴 𝑗 ) max 𝜎 𝑖 ∈Δ(𝐴 𝑖 ) 𝑢 𝑖 (𝜎 𝑖 , 𝝈 −𝑖 ). (2) In words, 𝑣 Γ 𝑖 is the minimum utility that the players other than 𝑖 can force onto 𝑖, under the assumption that 𝑖 reacts optimally to their strategy. We further will use minimax (𝑖, 𝑗) ∈ Δ(𝐴 𝑗 ) to denote the strategy for Player 𝑗 that is played in the minimizer 𝜎 −𝑖 of the above. Of course, in general, there might be multiple minimizers 𝜎 −𝑖 . In the following, we will assume that the function minimax breaks such ties in some consistent way, such that for all 𝑖, (minimax (𝑖, 𝑗)) 𝑗 ∈ {1,...,𝑛 }−{𝑖 } ∈ arg min 𝝈 −𝑖 ∈ 𝑗 ≠𝑖 Δ(𝐴 𝑗 ) max 𝜎 𝑖 ∈Δ(𝐴 𝑖 ) 𝑢 𝑖 (𝜎 𝑖 , 𝝈 −𝑖 ). (3) Note that for 𝑛 = 2, each player's threat point is computable in polynomial time via linear programming; and that by the minimax theorem  [28] , the threat point is equal to the maximin utility, i.e., 𝑣 Γ 𝑖 = max 𝜎 𝑖 ∈Δ(𝐴 𝑖 ) min 𝜎 −𝑖 ∈Δ(𝐴 −𝑖 ) 𝑢 𝑖 (𝜎 𝑖 , 𝜎 −𝑖 ), (4) so 𝑣 Γ 𝑖 is also the minimum utility that Player 𝑖 can guarantee for herself under the assumption that the opponent sees her mixed strategy and reacts in order to minimize Player 𝑖's utility. Tennenholtz'  [25]  main result on program games is the following: Theorem A.1  (Tennenholtz 2004 [25] ). Let Γ = (𝐴, u) be a game and let x ∈ u 𝑛 𝑖=1 Δ(𝐴 𝑖 ) be a (feasible) payoff vector. If 𝑥 𝑖 ≥ 𝑣 Γ 𝑖 for 𝑖 = 1, ..., 𝑛, then x is the utility of some program equilibrium of a program game on Γ. Throughout the rest of this section, our goal is to use similar ideas as Tennenholtz did for Theorem A.1 to construct for any SPI Γ 𝑠 on Γ, a program equilibrium that results in the play of Π(Γ 𝑠 ). As noted in the main text, the Player 𝑖's instruction to her representative to play the game Γ 𝑠 will usually be conditional on the other player telling her representative to also play her part of Γ 𝑠 and and vice versa. After all, if Player 𝑖 simply tells her representative to maximize 𝑢 𝑠 𝑖 from 𝐴 𝑠 𝑖 regardless of Player −𝑖's instruction, then Player −𝑖 will often be able to profit from deviating from the Γ 𝑠 instruction. For example, in the safe Pareto improvement on the Demand Game, each player would only want their representative to choose from {DL, RL} rather than {DM, DM} if the other player's representative does the same. It would then seem that in a program equilibrium in which Π(Γ 𝑠 ) is played, each program 𝑐 𝑖 would have to contain a condition of the type, "if the opponent code plays as in Π(Γ 𝑠 ) against me, I also play as I would in Π(Γ 𝑠 )." But in a naive implementation of this, each of the programs would have to call the other, leading to an infinite recursion. In the literature on program equilibrium, various solutions to this problem have been discovered. We here use the general scheme proposed by Tennenholtz  [25] , because it is the simplest. We could similarly use the variant proposed by Fortnow  [8] , techniques based on Löb's theorem  [3, 6] , or 𝜖-grounded mutual simulation  [17]  or even (meta) Assurance Game preferences (see Appendix B). In our equilibrium, we let each player submit code as sketched in Algorithm 2. Roughly, each player uses a program that says, "if everyone else submitted the same source code as this one, then play Π(Γ 𝑠 ). Otherwise, if there is a player 𝑗 who submits a different source code, punish player 𝑗 by playing her minimax strategy". Note that for convenience, Algorithm 2 receives the player number 𝑖 as input. This way, every player can use the exact same source code. Otherwise the original players would have to provide slightly different programs and in line 2 of the algorithm, we would have to use a more complicated comparison, roughly: "if 𝑐 𝑗 ≠ 𝑐 𝑖 are the same, except for the player index used".   )) = 𝑣 𝑖 ≤ E [𝑢 𝑖 (Π(Γ))] ≤ E 𝑢 𝑖 (Π(Γ 𝑠 )) = E [𝑢 𝑖 (exec(c))] as required. □ Theorem 4.1 follows immediately.
 B A DISCUSSION OF WORK BY SEN (1974) AND RAUB (1990) ON PREFERENCE ADAPTATION GAMES We here discuss Raub's  [21]  paper in some detail, which in turn elaborates on an idea by Sen  [24] . Superficially, Raub's setting seems somewhat similar to ours, but we here argue that it should be thought of as closer to the work on program equilibrium and bilateral precommitment. In Sections 1, 3 and 4, we briefly discuss multilateral commitment games, which have been discussed before in various forms in the game-theoretic literature. Our paper extends this setting by allowing instructions that let the representatives play a game without specifying an algorithm for solving that game. On first sight, it appears that Raub pursues a very similar idea. Translated to our setting, Raub allows that as an instruction, each player 𝑖 chooses a new utility function 𝑢 𝑠 𝑖 : 𝐴 → R, where 𝐴 is the set of outcomes of the original game Γ. Given instructions 𝑢 𝑠 1 , ..., 𝑢 𝑠 𝑛 , the representatives then play the game (𝐴, u 𝑠 ). In particular, each representative can see what utility functions all the other representatives have been instructed to maximize. However, what utility function representative 𝑖 maximizes is not conditional on any of the instructions by other players. In other words, the instructions in Raub's paper are raw utility functions without any surrounding control structures, etc. Raub then asks for equilibria u 𝑠 of the meta-game that Pareto-improve on the default outcome. To better understand how Raub's approach relates to ours, we here give an example of the kind of instructions Raub has in mind. (Raub uses the same example in his paper.) As the underlying game Γ, we take the Prisoner's Dilemma. Now the main idea of his paper is that the original players can instruct their representatives to adopt so-called Assurance Game preferences. In the Prisoner's Dilemma, this means that the representatives prefer to cooperate if the other representative cooperates, and prefer to defect if the other player defects. Further, they prefer mutual cooperation over mutual defection. An example of such Assurance Game preferences is given in Table  3 . (Note that this payoff matrix resembles the classic Stag Hunt studied in game theory.) The Assurance Game preferences have two important properties. (1) If both players tell their representatives to adopt Assurance Game preferences, (Cooperate, Cooperate) is a Nash equilibrium. (Defect, Defect) is a Nash equilibrium as well. However, since (Cooperate, Cooperate) is Pareto-better than (Defect, Defect), the original players could reasonably expect that the representatives play (Cooperate, Cooperate). (  2 ) Under reasonable assumptions about the rationality of the representatives, it is a Nash equilibrium of the meta-game for both players to adopt Assurance Game preferences. If Player 1 tells her representative to adopt Assurance Game preferences, then Player 2 maximizes his utility by telling his representative to also maximize Assurance Game preferences. After all, representative 1 prefers defecting if representative 2 defects. Hence, if Player 2 instructs his representative to adopt preferences that suggest defecting, then he should expect representative to defect as well. The first important difference between Raub's approach and ours is related to item 2. We have ignored the issue of making SPIs Γ 𝑠 Nash equilibria of our meta game. As we have explained in Section 4 and Appendix A, we imagine that this is taken care of by additional bilateral commitment mechanisms that are not the focus of this paper. For Raub's paper, on the other hand, ensuring mutual cooperation to be stable in the new game Γ 𝑠 is arguably the key idea. Still, we could pursue the approach of the present paper even when we limit assumptions to those that consist only of a utility function. The second difference is even more important. Raub assumes that -as in the PD -the default outcome of the game (Π(Γ) in the formalism of this paper) is known. (Less significantly, he also assumes that it is known how the representatives play under assurance game preferences.) Of course, the key feature of the setting of this paper is that the underlying game Γ might be difficult (through equilibrium selection problems) and thus that the original players might be unable to predict Π(Γ). These are the reasons why we cite Raub in our section on bilateral commitment mechanisms. Arguably, Raub's paper could be seen as very early work on program equilibrium, except that he uses utility functions as a programming language for representative. In this sense, Raub's Assurance Game preferences are analogous to the program equilibrium schemes of Tennenholtz  [25] , Oesterheld  [25] , Barasz et al.  [3]  and van der Hoek et al.  [26] , ordered in increasing order of similarity of the main idea of the scheme. C PROOF OF PROPOSITION 5.2 Lemma 5.2. Let Γ = (𝐴, u), Γ ′ = (𝐴 ′ , u ′ ), Γ = ( Â, û) and Φ, Ξ : 𝐴 ⊸ 𝐴 ′ , Ψ : 𝐴 ′ ⊸ Â. (1) Reflexivity: Γ ∼ id 𝐴 Γ, where id 𝐴 : 𝐴 ⊸ 𝐴 : a ↦ → {a}. (2) Symmetry: If Γ ∼ Φ Γ ′ , then Γ ′ ∼ Φ −1 Γ. ( 3 ) Transitivity: If Γ ∼ Φ Γ ′ and Γ ′ ∼ Ψ Γ, then Γ ∼ Ψ•Φ Γ. (4) If Γ ∼ Φ Γ ′ and Φ(a) ⊆ Ξ(a) for all a ∈ 𝐴, then Γ ∼ Ξ Γ ′ . ( 5 ) Γ ∼ all 𝐴,𝐴 ′ Γ ′ , where all 𝐴,𝐴 ′ : 𝐴 ⊸ 𝐴 ′ : a ↦ → 𝐴 ′ . (6) If Γ ∼ Φ Γ ′ and Φ(a) = ∅, then Π(Γ) ≠ a with certainty. Proof. 1. By reflexivity of equality, Π(Γ) = Π(Γ) with certainty. Hence, Π(Γ) ∈ id 𝐴 (Π(Γ)) by definition of id 𝐴 . Therefore, Γ ∼ id 𝐴 Γ by definition of ∼, as claimed. 2. Γ ∼ Φ Γ ′ means that Π(Γ ′ ) ∈ Φ(Π(Γ)) with certainty. Thus, Π(Γ) ∈ {a∈𝐴 | Π(Γ ′ )∈Φ(a)} = Φ −1 (Π(Γ ′ )), where equality is by the definition of the inverse of multivalued functions. We conclude (by definition of ∼) that Γ ′ ∼ Φ −1 Γ as claimed. 3. If Γ ∼ Φ Γ ′ , Γ ′ ∼ Ψ Γ, then by definition of ∼, (i) Π(Γ ′ ) ∈ Φ(Π(Γ)) and (ii) Π( Γ) ∈ Ψ(Π(Γ ′ )) , both with certainty. The former (i) implies {Π(Γ ′ )} ⊆ Φ(Π(Γ)). Hence, Ψ(Π(Γ ′ )) = Ψ({Π(Γ ′ )}) ⊆ Ψ(Φ(Π(Γ))). With ii, it follows that Π( Γ) ∈ Ψ(Φ(Π(Γ))) with certainty. By definition, Γ ∼ Ψ•Φ Γ as claimed. 4. It is Π(Γ ′ ) ∈ Φ(Π(Γ)) ⊆ Ξ(Π(Γ)) with certainty. Thus, by definition Γ ∼ Ξ Γ ′ . 5. By definition of Π, it is Π(Γ ′ ) ∈ 𝐴 ′ with certainty. By definition of all 𝐴,𝐴 ′ , it is all 𝐴,𝐴 ′ (Π(Γ)) = 𝐴 ′ with certainty. Hence, Π(Γ ′ ) ∈ all 𝐴,𝐴 ′ (Π(Γ)) with certainty. We conclude that Γ ∼ all 𝐴,𝐴 ′ Γ ′ as claimed. 6. With certainty, Π(Γ ′ ) ∈ Φ(Π(Γ)) (by assumption). Also, with certainty Π(Γ ′ ) ∉ ∅. Hence, Φ(Π(Γ)) ≠ ∅ with certainty. We conclude that Π(Γ) ≠ a with certainty. □ We here prove Theorem 8.2.
 D EXAMPLES Throughout this section, we use the following lemma. Lemma E.1. Let Φ, Ψ be isomorphisms between Γ,Γ ′ . If Φ is (strictly) Pareto-improving, then so is Ψ. This will allow us to conclude from the existence of a Paretoimproving isomorphism Φ that there is Pareto-improving Ψ s.t. Γ ∼ Ψ Γ ′ by Assumption 7.1, even if there are multiple isomorphisms between Γ, Γ ′ . Proof. Let Γ = (𝐴, u), Γ ′ = (𝐴 ′ , u ′ ). Then for all a ∈ 𝐴, u(a) = u ′ (Ψ(a)) = u(Φ −1 (Ψ(a)) ≤ u(Φ(Φ −1 (Ψ(a)))) = u(Ψ(a)). Furthermore, if Φ is strictly Pareto-improving for some ã ∈ 𝐴, then by bijectivity of Φ, Ψ, there is a ∈ 𝐴 s.t. Φ −1 (Ψ(a)) = ã. For this a, the inequality above is strict and therefore u(a) < u(Ψ(a)). □ We start by showing that the SPI problem is in NP at all. The following algorithm can be used to determine whether there is a safe Pareto improvement: Reduce the given game Γ until it can be reduced no further to obtain some subset game Γ ′ = (𝐴 ′ , u). Then non-deterministically select injections Φ 𝑖 : 𝐴 ′ 𝑖 → 𝐴 𝑖 . If Φ = (Φ 1 , ..., Φ 𝑛 ) is (strictly) Pareto-improving (as required in Theorem 6.1), return True with the solution Γ 𝑠 defined as follows: The set of action profiles is defined as 𝐴 𝑠 = 𝑖 Φ 𝑖 (𝐴 ′ 𝑖 ). The utility functions are 𝑢 𝑠 𝑖 : 𝐴 𝑠 → R : a 𝑠 ↦ → (𝑢 𝑖 (Φ −1 1 (𝑎 𝑠 1 ), ..., Φ −1 𝑛 (𝑎 𝑠 𝑛 ))) 𝑖=1,...,𝑛 . (6) Otherwise, return False. It is easy to see that this algorithm runs in non-deterministic polynomial time. Furthermore, with Lemma E.1 it is easy to see that if this algorithm finds a solution Γ 𝑠 , that solution is indeed a safe Pareto improvement. It is left to show that if there is a safe Pareto improvement via a sequence of Assumption 7.1 and 7.2 outcome correspondences, then the algorithm indeed finds a safe Pareto improvement. To prove this fact, we prove a few simple lemmata. First, one might worry that the algorithm only ever finds sequences of outcome correspondences that start with a number of reductions and end with a single isomorphism step. Perhaps some safe Pareto improvements can only be found by considering very different sequences? The following two lemmata show that this is not an issue, i.e., that it is sufficient to consider sequences that start with a number of reductions and end in a single isomorphism step. Lemma E.2. Let Γ ∼ Φ iso Γ by Assumption 7.1 and Γ ∼ Φ red Γ by Assumption 7.2. Then there are Γ ′ , Ψ red , Ψ iso s.t. Γ ∼ Ψ red Γ ′ by Assumption 7.2, Γ ′ ∼ Ψ iso Γ by Assumption 7.1 and Ψ iso • Ψ red = Φ red • Φ iso . Intuitively, this means that isomorphism steps as per Assumption 7.1 and reduction steps as per Assumption 7.2 commute. Instead of first applying Assumption 7.1 and then Assumption 7.2 to a game, we can also apply Assumption 7.2 first and then Assumption 7.1 to obtain the same game Γ in both cases. Proof. We construct Γ ′ , Ψ red , Ψ iso as follows. First, Γ ′ = (𝐴 ′ 1 , ..., 𝐴 ′ 𝑛 , u ′ ), where 𝐴 ′ 𝑖 = (Φ iso 𝑖 ) −1 ( Ã𝑖 ) and 𝑢 ′ 𝑖 = 𝑢 𝑖 | 𝐴 ′ . Next, we define Ψ red = (Φ iso ) −1 • Φ red • Φ iso (7) and Ψ iso = Φ iso | 𝐴 ′ 1 ×𝐴 ′ 2 . (8) We now need to show that these satisfy the consequents of the lemma. First, it is Ψ iso • Ψ red = Φ iso | 𝐴 ′ 1 ×𝐴 ′ 2 • (Φ iso ) −1 • Φ red • Φ iso = Φ iso | 𝐴 ′ 1 ×𝐴 ′ 2 • (Φ iso ) −1 • Φ red • Φ iso = Φ red • Φ iso as claimed. Note that the second step uses the associativity of • on multivalued functions. The third step uses the fact that Φ iso is a single-valued bijection, which means that Φ iso • Φ iso −1 = id. Of course, Φ iso is here restricted to 𝐴 ′ 1 × 𝐴 ′ 2 , but this is not a problem, because 𝐴 ′ 𝑖 = (Φ iso 𝑖 ) −1 ( Ã𝑖 ) and à is the codomain of Φ red • Φ iso . Hence, the restriction is inconsequential. Second, we need to show that it is indeed Γ ′ ∼ Ψ iso Γ by Assumption 7.1. (Note that because Ψ iso • Ψ red = Φ red • Φ iso , it really must be Γ ′ ∼ Ψ iso Γ rather than Γ ′ ∼ Ψiso Γ for some different isomorphism Ψiso ≠ Ψ iso .) First, it is easy to show that Ψ iso decomposes into Ψ iso 1 , ..., Ψ iso 𝑛 as required because Φ iso decomposes. Further, Ψ iso is a single-valued injection because Φ iso is a single-valued bijection. It is surjective because its codomain is defined as the image of its domain. Now let (𝑎 ′ 1 , 𝑎 ′ 2 ) ∈ 𝐴 ′ 1 × 𝐴 ′ 2 . It is u ′ (a ′ ) = u(a ′ ) = û(Φ iso (a ′ )) = ũ(Φ iso (a ′ )) = ũ(Ψ iso (a ′ )), as required. Finally, we have to show that Γ ∼ Ψ red Γ ′ by Assumption 7.2. We leave this as an exercise to the reader. □ Lemma E.3. Let Γ 1 ∼ Φ 1 ... ∼ Φ 𝑘−1 Γ 𝑘 , (9) where each outcome correspondence is due to a single application of Assumption 7.1 or 7.2. Then there is a sequence Γ ′2 , ..., Γ ′𝑚 with 𝑚 ≤ 𝑘 − 1 such that Γ 1 ∼ Ψ 1 Γ ′2 ∼ Ψ 2 Γ ′3 ∼ Ψ 3 ... ∼ Ψ 𝑚−1 Γ ′𝑚 (10) all by single applications of Assumption 7.2, Γ ′𝑚 ∼ Ξ Γ 𝑘 by a single application of Assumption 7.1, and Φ 𝑘−1 • Φ 𝑘−2 • ... • Φ 1 = Ξ • Ψ 𝑚−1 • ... • Ψ 1 . (11) Proof. Start with the initial sequence of line 9. We can iteratively apply Lemma E.2 to obtain a new sequence of the same length in which one first applies only Assumption 7.2 and then only Assumption 7.1 while obtaining the same composite outcome correspondence function. We can summarize all the applications of Assumption 7.1 into a single step applying that assumption. □ A second potential worry about our algorithm is that it reduces the game completely and only then looks for a Pareto-improving isomorphism step. Perhaps in some cases one has to only partially reduce and then look for a Pareto-improving isomorphism step? The next lemma shows that the answer to this is no and that one can restrict oneself to sequences that fully reduce. Lemma E.4 shows that it is enough to consider isomorphism steps from fully reduced versions of Γ. A third worry might be that even so, elimination via Assumption 7.2 might be path-dependent and therefore we have to consider the resulting games from multiple paths. However, iterated elimination of strictly dominated strategies is known to be path-independent  [1, 19] . Proposition E.5. If there is a safe Pareto improvement for a given game, then the above algorithm applied to that game returns True. Lemma E.4. Let Γ = (𝐴, u), Γ𝑎 = ( Â𝑎 , u), Γ𝑏 = ( Â𝑏 , u) such that Â𝑏 𝑖 ⊆ Â𝑎 𝑖 ⊆ 𝐴 𝑖 for 𝑖 = 1, ..., Proof. Let us say there is a sequence of outcome correspondences as per Assumptions 7.1 and 7.2 that show Γ ∼ Φ Γ 𝑠 for Pareto-improving Φ. Then by Lemma E.3, there is Γ ′ such that Γ ∼ Ψ red Γ ′ via an arbitrary number of applications of Assumption 7.2 and Γ ′ ∼ Ψ iso Γ 𝑠 via a single application of Assumption 7.1. Because of the path-independence of iterated removal of strictly dominated strategies, Γ ′ contains (as a subset game with equal utility functions) the unique Γ 𝑟 arising from full iterated removal as per Assumption 7.2. By Lemma E.4, there is a Pareto-improving outcome correspondence Γ 𝑟 ∼ Ψiso Γ 𝑠𝑟 as per Assumption 7.2. By construction, our algorithm finds (guesses) this Pareto-improving outcome correspondence. □ Overall, we have now shown that our non-deterministic polynomial-time algorithm is correct and therefore that the SPI problem is in NP. Note that the correctness of other algorithms can be proven using very similar ideas. For example, instead of first reducing and then finding an isomorphism, one could first find an isomorphism, then reduce and then (only after reducing) test whether the overall outcome correspondence function is Paretoimproving. One advantage of reducing first is that there are fewer isomorphisms to test if the game is smaller. In particular, the number of possible isomorphisms is exponential in the number of strategies in the reduced game Γ ′ but polynomial in everything else. Hence, by implementing our algorithm deterministically, we obtain the following positive result. We now proceed to showing that the safe Pareto improvement problem is NP-hard. We will do this by reducing the subgraph isomorphism problem to the (two-player) safe Pareto improvement problem. We start by briefly describing one version of that problem here. A (simple, directed) graph is a tuple (𝑛, 𝑎 : {1, ..., 𝑛} × {1, ..., 𝑛} → B), where 𝑛 ∈ N and B {0, 1}. We call 𝑎 the adjacency function of the graph. Since the graph is supposed to be simple and therefore free of self-loops (edges from one vertex to itself), we take the values 𝑎( 𝑗, 𝑗) for 𝑗 ∈ {1, ..., 𝑛} to be meaningless. For given graphs 𝐺 = (𝑛, 𝑎),𝐺 ′ = (𝑛 ′ , 𝑎 ′ ) a subgraph isomorphism from 𝐺 to 𝐺 ′ is an injection 𝜙 : {1, ..., 𝑛} → {1, ...𝑛 ′ } such that for all 𝑗 ≠ 𝑙 𝑎( 𝑗, 𝑙) ≤ 𝑎 ′ (𝜙 ( 𝑗), 𝜙 (𝑙)). (12) In words, a subgraph isomorphism from 𝐺 to 𝐺 ′ identifies for each node in 𝐺 a node in 𝐺 ′ s.t. if there is an edge from node 𝑗 to node 𝑙 in 𝐺, there must also be an edge in the same direction between the corresponding nodes 𝜙 ( 𝑗), 𝜙 (𝑙) in 𝐺 ′ . Another way to say this is that we can remove some set of (𝑛 ′ − 𝑛) nodes and some edges from 𝐺 ′ to get a graph that is just a relabeled (isomorphic) version of 𝐺. Given two graphs 𝐺, 𝐺 ′ , the subgraph isomorphism problem consists in deciding whether there is a subgraph isomorphism 𝜙 between 𝐺, 𝐺 ′ . The problem is well-known to be NP-complete  [5, Theorem 2] . Lemma E.6. The subgraph isomorphism problem is reducible in linear time with linear increase in problem instance size to the safe Pareto improvement problem. As a consequence, the safe Pareto improvement problem is NP-hard. Proof. We conduct our proof only for the strict safe Pareto improvement problem. Reducing to the non-strict safe Pareto improvement problem is a little easier and can be done using a subset of the ideas in this proof. So take graphs 𝐺 = (𝑛, 𝑎) and Ĝ = ( n, â). We will transform these step-wise into a single game. First, we define the games Γ 𝑎 = (𝐴 1 , 𝐴 2 , 𝑢 1 , 𝑢 2 ) and Γ𝑎 = ( Â1 , Â2 , û1 , û2 ), where 𝐴 1 = 𝐴 2 = {1, ..., 𝑛}, Â1 = Â2 = {1, ..., n}, 𝑢 1 ( 𝑗, 𝑙) = 𝑢 2 ( 𝑗, 𝑙) = 𝑎( 𝑗, 𝑙) for all 𝑗, 𝑙 ∈ {1, ..., 𝑛} with 𝑗 ≠ 𝑙 and 𝑢 1 ( 𝑗, 𝑗) = 𝑢 2 ( 𝑗, 𝑗) = 2. We analogously define û1 = û2 based on â. Setting the utility functions is the main idea of the entire proof, of course, and will become clearer below. Setting the utilities 𝑢 1 ( 𝑗, 𝑗) = 𝑢 2 ( 𝑗, 𝑗) = 2 is to ensure that Pareto-improving mappings Φ between Γ 𝑎 and Γ𝑎 satisfy Φ 1 ( 𝑗) = Φ 2 ( 𝑗) for all 𝑗, and thus directly relate to subgraph isomorphisms. Next, we add dummy strategies to Γ 𝑎 , Γ𝑎 , to obtain two purposes. We want to remove exact equivalences and allow only strict Pareto improvements; and we want to remove the possibility of reducing either of these games via Assumption 7.2. In particular, we consider Γ 𝑏 as follows: 𝐴 𝑏 𝑖 = {1, ..., 2𝑛}; u 𝑏 ( 𝑗, 𝑙) = u 𝑎 ( 𝑗, 𝑙) if 𝑗, 𝑙 ∈ {1, ..., 𝑛}, u 𝑏 ( 𝑗, 𝑛+𝑗) = u 𝑏 (𝑛+𝑗, 𝑛) = (3, 3) for 𝑗 ∈ {1, .., 𝑛}, u 𝑏 ( 𝑗, 𝑙) = (−1, −1) otherwise. We define Γ𝑏 analogously, except that utilities of  (3, 3)  are to be replaced by  (4, 4) . Finally, we construct from Γ 𝑏 , Γ𝑏 a single game Γ 𝑐 . Roughly, the idea is for Γ 𝑐 to contain as subset games both Γ 𝑏 and Γ𝑏 , but to reduce to Γ 𝑏 via Assumption 7.2. We construct Γ 𝑐 thus: 𝐴 𝑐 𝑖 = ({𝐷 } × 𝐴 𝑏 𝑖 ) ∪ ({𝐶} × Â𝑏 𝑖 ) and u 𝑐 ((𝐶, â1 ), (𝐶, â2 )) = û𝑏 ( â1 , â2 ) for all â1 ∈ Â𝑏 1 , â2 ∈ Â𝑏 2 u 𝑐 ((𝐷, 𝑎 1 ), (𝐷, 𝑎 2 )) = u 𝑏 (𝑎 1 , 𝑎 2 ) for all 𝑎 1 ∈ 𝐴 𝑏 1 , 𝑎 2 ∈ 𝐴 𝑏 2 𝑢 𝑐 𝑖 ((𝐷, 𝑎 𝑖 ), (𝐶, â−𝑖 )) = 5 for all 𝑎 𝑖 ∈ 𝐴 𝑏 𝑖 , â−𝑖 ∈ Â𝑏 2 𝑢 𝑐 −𝑖 ((𝐷, 𝑎 𝑖 ), (𝐶, â−𝑖 )) = −5 for all 𝑎 𝑖 ∈ 𝐴 𝑏 𝑖 , â−𝑖 ∈ Â𝑏 2 . It is easy to show that this reduction can be computed in linear time and that it also increases the problem instance size only linearly. It is left to prove the correctness of the reduction. We start by showing that if there is a subgraph isomorphism from 𝐺 to Ĝ, then there is also a safe (strict) Pareto improvement via a sequence of outcome correspondence as per Assumptions 7.1 and 7.2. So let 𝜙 be that subgraph isomorphism. Then we need to construct a series of outcome correspondences as per Assumptions 7.1 and 7.2. First notice that Γ 𝑐 ∼ Ξ Γ 𝑐,𝐷 by Assumption 7.2, where Γ 𝑐,𝐷 is the subset game of Γ 𝑐 that contains only the strategies of type (𝐷, 𝑗) for both players. We now show that Ψ is a Pareto-improving isomorphism between Γ 𝑐,𝐷 and Γ = ( Ã1 , Ã2 , ũ1 , ũ2 ), where we define Ã1 = Ã2 = {𝐶} × (𝜙 ({1, ..., 𝑛}) ∪ { n + 𝜙 ( 𝑗) | 𝑗 = 1, ..., 𝑛}),  (13)  for 𝑖 = 1, 2 as Ψ 𝑖 (𝐷, 𝑗) = (𝐶, 𝜙 ( 𝑗)) if 𝑗 ∈ {1, ..., 𝑛} and Ψ 𝑖 (𝐷, 𝑗) = (𝐶, n +𝜙 ( 𝑗 −𝑛)) otherwise, and the utility function is simply defined as ũ𝑖 ( ã1 , ã2 ) = 𝑢 𝑖 (Ψ −1 𝑖 ( ã1 , ã2 )). Again, it will then follow from Lemma E.1 that in particular it is Γ 𝑐,𝐷 ∼ Ψ ′ Γ via Assumption 7.1 for some potentially different, but still Pareto-improving Ψ ′ . It is easy to see that Ψ 𝑖 is surjective for 𝑖 = 1, 2. From the fact that 𝜙 is injective and that 𝜙 never returns values greater than n, it follows that Ψ 𝑖 is also injective for 𝑖 = 1, 2. Finally, Ψ maintains utilities by definition: ũ𝑖 (Ψ(𝑎 1 , 𝑎 2 )) = 𝑢 𝑖 (Ψ −1 (Ψ(𝑎 1 , 𝑎 2 ))) = 𝑢 𝑖 (𝑎 1 , 𝑎 2 ). ( ) 14 With Transitivity, it is left to show that Ψ is Pareto-improving for the original players, i.e., for u 𝑐 . For this we distinguish a number of different cases. If 𝑗, 𝑙 ∈ {1, ..., 𝑛} and 𝑗 ≠ 𝑙, then for 𝑖 = 1, 2 The cases ( 𝑗, 𝑙) ∈ {𝑛 + 1, ..., 2𝑛} × {1, ..., 𝑛} work analogously. This concludes our proof that if a graph isomorphism exists, there also exists a strict SPI as per Assumptions 7.1 and 7.2. It is left to prove that if there is a safe Pareto improvement for Γ 𝑐 , then there also exists a graph isomorphism. So let Γ 𝑐 ∼ Ξ Γ for some Γ, via some Pareto-improving outcome correspondence function Ξ. By our earlier results (Proposition E.5), this means that there is a sequence of outcome correspondences that first fully reduces Γ to Γ 𝑐,𝐷 and then applies Assumption 7.1 to get Γ 𝑐,𝐷 ∼ Ψ Γ via some Pareto-improving Ψ. To construct a subgraph isomorphism, we must now realize some facts about the structure of Ψ that all follow from Ψ being utility-increasing: (1) Ψ ({𝐷 } × {1, ..., 2𝑛}) 2 ⊆ ({𝐶} × {1, ..., 2 n}) 2 : For Ψ to be strict, there has to be some overlap, i.e., there has to be (𝐷, 𝑗) s. By an analogous argument it can further be shown that Ψ 𝑖 (𝐷, 𝑗) ∈ {𝐶} × {1, ..., 2 n} for all 𝑗 = 1, ..., 2𝑛. (2) For every 𝑗 ∈ {1, ..., 𝑛}, it is Ψ 𝑖 (𝐷, 𝑗) ∈ {𝐶} × {1, ..., n} for 𝑖 = 1, 2. That is, Ψ 𝑖 maps the "non-dummy" strategies (which correspond to nodes in the original graph) onto "nondummy" strategies. We will show this by showing the contrapositive, i.e., that if this were not the case, then Ψ would not be Pareto-improving. < (0, 0) ≤ u 𝑐 ((𝐷, 𝑗), (𝐷, 𝑙)). (3) For all 𝑗 ∈ {1, ..., 𝑛} and 𝑖 ∈ {1, ..., 𝑛} it is Ψ 𝑖 (𝐷, 𝑗 + 𝑛) = Ψ −𝑖 (𝐷, 𝑗) + n, where addition is defined to operate only on the second entry like the subtraction defined above. We again prove the contrapositive, i.e., if this is not true then Ψ is not Pareto-improving. So let us assume that there is 𝑗 ∈ {1, ..., 𝑛} and 𝑖 ∈ {1, ..., 𝑛} s.t. Ψ 𝑖 (𝐷, 𝑗 + 𝑛) ≠ Ψ −𝑖 (𝐷, 𝑗). Then it would be u 𝑐 (Ψ 𝑖 (𝐷, 𝑗 + 𝑛), Ψ −𝑖 (𝐷, 𝑗)) ≤ (2, 2) < (3, 3) = u 𝑐 ((𝐶, 𝑗 + 𝑛), (𝐶, 𝑗)). (4) Finally, we prove that Ψ 1 = Ψ 2 . We first show that for 𝑗 ∈ {1, ..., 𝑛} it is Ψ 1 (𝐷, 𝑗) = Ψ 2 (𝐷, 𝑗). We do this again by showing the contrapositive.  = n + Ψ 1 (𝐷, 𝑗) = Item 3 Ψ 2 (𝐷, 𝑛 + 𝑗), where the middle equality is due to the equality we have already proven. Given these, we can define our graph isomorphism as 𝜙 : {1, ..., 𝑛} → {1, ..., n} : 𝑗 ↦ → 𝜋 2 (Ψ 1 (𝐷, 𝑗)),  (15)  where 𝜋 2 just maps pairs (𝐶, 𝑗) onto the second entry 𝑗. This is well-defined because of item 2. Note that because Ψ is an injection, so is 𝜙. Hence, 𝜙 is a subgraph isomorphism as desired. □ F PROOF OF LEMMA 9.3 Lemma 9.3. For a given 𝑛-player game Γ and payoff vector y ∈ R 𝑛 , it can be decided by linear programming and thus in polynomial time whether y is Pareto-optimal in C(Γ). For an introduction to linear programming, see, e.g., Schrijver  [23] . In short, a linear program is a specific type of constrained optimization problem that can be solved efficiently. Algorithm 1 : 4 . 14 An algorithm for deciding the strict perfectcoordination SPI problem.Data: Game Γ, set supp(Π(Γ) 1 for a ∈ supp(Π(Γ)) do 2 if u(a) is Pareto-suboptimal within C(Γ) Assuming supp(Π(Γ)) is known and that Assumption 7.1 holds, it can be decided in polynomial time whether there is a strict perfect-coordination SPI.
 Theorem 9 . 7 . 97 Make Assumption 7.1. Let Γ be a two-player game. Let y ∈ R 2 be some potentially unsafe Pareto improvement onE [u(Π(Γ))]. For 𝑖 = 1, 2, let 𝑥 min/max 𝑖 = min/max 𝑢 𝑖 (supp(Π(Γ))). Then: A) Ifthere is some element in C(Γ) which Pareto-dominates all of supp(Π(Γ)) and if y is Pareto-dominated by an element of at least one of the following three sets: • 𝐿 1 the line segment between 𝜋 1 (𝑥 min 1 , PF(C(Γ)) and 𝜋 1 (𝑥 max 1 , PF(C(Γ)); • 𝐿 2 the segment of the curve PF(C(Γ)) between 𝜋 1 (𝑥 max 1 , PF(C(Γ)))) and 𝜋 2 (𝑥 max 2 , PF(C(Γ)))); • 𝐿 3 the line segment between 𝜋 2 (𝑥 max 2 , PF(C(Γ)) and 𝜋 2 (𝑥 min 2 , PF(C(Γ)).
 Figure 1 : 1 Figure 1: This figure illustrates Theorem 9.7, Case A.
 Figure 2 : 2 Figure 2: This figure illustrates Theorem 9.7, Case B.
 Algorithm 2 : 2 A program equilibrium implementation of an SPI Γ 𝑠 of Γ. Data: Everybody's source code c, my index 𝑖 1 for 𝑗 ∈ {1, ..., 𝑛} − {𝑖} do 2 if 𝑐 𝑗 ≠ 𝑐 𝑖 then 3 Play minimax (𝑖, 𝑗); 4 Play Π 𝑖 (Γ 𝑠 ); Proposition A.2. Let Γ be a game and let Γ 𝑠 be an SPI on Γ. Let c be the program profile consisting only of Algorithm 2 for each player. Assume that Π(Γ) guarantees each player at least threat point utility in expectation. Then c is a program equilibrium and apply(c) = Π(Γ 𝑠 ).
 𝑢 𝑐𝑖 ((𝐷, 𝑗), (𝐷, 𝑙)) = 𝑎( 𝑗, 𝑙) ≤ â(𝜙 ( 𝑗), 𝜙 (𝑙))= û𝑐 𝑖 ((𝐶, 𝜙 ( 𝑗)), (𝐶, 𝜙 (𝑙))) = û𝑐 𝑖 (Ψ 1 (𝐷, 𝑗), Ψ 2 (𝐷, 𝑙)). For 𝑗 ∈ {1, ..., 𝑛}, it is u 𝑐 ((𝐷, 𝑗), (𝐷, 𝑗)) = (2, 2) = û𝑐 ((𝐶, 𝜙 ( 𝑗)), (𝐶, 𝜙 ( 𝑗))) = û𝑐 (Ψ 1 (𝐷, 𝑗), Ψ 2 (𝐷, 𝑗)).For 𝑗, 𝑙 ∈ {𝑛 + 1, ..., 2𝑛}, it isu 𝑐 ((𝐷, 𝑗), (𝐷, 𝑙)) = u 𝑏 ( 𝑗, 𝑙) = (−1, −1) = û𝑏 ( n + 𝜙 ( 𝑗 − 𝑛), n + 𝜙 (𝑙 − 𝑛)) = u 𝑐 (Ψ 1 (𝐷, 𝑗), Ψ 2 (𝐷, 𝑙)).If ( 𝑗, 𝑙) ∈ {1, ..., 𝑛} × {𝑛 + 1, ..., 2𝑛} and 𝑙 = 𝑗 + 𝑛, then u 𝑐 ((𝐷, 𝑗), (𝐷, 𝑙)) = (2, 2)< (3, 3) = u 𝑐 ((𝐶, Φ( 𝑗)), (𝐶, Φ( 𝑗) + n)) = u 𝑐 (Ψ 1 (𝐷, 𝑗), Ψ 2 (𝐷, 𝑙)).If ( 𝑗, 𝑙) ∈ {1, ..., 𝑛} × {𝑛 + 1, ..., 2𝑛} but 𝑙 ≠ 𝑗 + 𝑛, thenu 𝑐 ((𝐷, 𝑗), (𝐷, 𝑙)) = u 𝑏 ( 𝑗, 𝑙) = (−1, −1)= û𝑏 (𝜙 ( 𝑗), n + 𝜙 (𝑙 − 𝑛))= u 𝑐 (Ψ 1 (𝐷, 𝑗), Ψ 2 (𝐷, 𝑙)).
 5 < 5 t. Ψ 𝑖 (𝐷, 𝑗) ∈ {𝐶} × {1, ..., 2 n}. But for Ψ to be Paretoimproving for 𝑖, it has to be for all (𝐷, 𝑙) with 𝑙 = 1, ..., 2𝑛 the case that Ψ −𝑖 (𝐷, 𝑙) ∈ {𝐶} × {1, ..., 2 n} since otherwise it would be 𝑢 𝑖 (Ψ 𝑖 (𝐷, 𝑗), Ψ −𝑖 (𝐷, 𝑙)) = −𝑢 𝑖 ((𝐷, 𝑗), (𝐷, 𝑙)).
 So assume there is 𝑗 ∈ {1, ..., 𝑛} and 𝑖 ∈ {1, 2} with Ψ 𝑖 (𝐷, 𝑗) ∈ {𝐶} × { n + 1, ..., 2 n}. Because Ψ −𝑖 is injective, there is an 𝑙 ∈ {1, ..., 𝑛} s.t. Ψ −𝑖 (𝐷, 𝑙) ≠ Ψ 𝑖 (𝐷, 𝑗) − n, where we define (𝐷, 𝑘) − n (𝐷, 𝑘 − n). Then for that 𝑙 it is u(Ψ 𝑖 (𝐷, 𝑗), Ψ −𝑖 (𝐷, 𝑙)) = (−1, −1)
 1 ( 1 For all 𝑗, 𝑙 ∈ {1, ..., 𝑛} with 𝑗 ≠ 𝑙 it isâ(𝜙 ( 𝑗), 𝜙 (𝑙)) = û𝑎 1 (𝜙 ( 𝑗), 𝜙 (𝑙)) = û𝑐 1 ((𝐶, 𝜙 ( 𝑗)), (𝐶, (𝜙 (𝑙)))) = Item 1 û𝑐 1 (Ψ 1 (𝐷, 𝑗), Ψ 1 (𝐷, 𝑙)) = Item 4 û𝑐 1 (Ψ 1 (𝐷, 𝑗), Ψ 2 (𝐷, 𝑙)) ≥ 𝑢 𝑐 1 ((𝐷, 𝑗), (𝐷, 𝑙)) = 𝑢 𝑎 𝑗, 𝑙) =𝑎( 𝑗, 𝑙).
 Table 1 : 1 The Demand Game
 Table 2 : 2 A safe Pareto improvement for the Demand Game 4,
 Assumptions 7.1 and 7.2 we can finally state our example SPIs formally: Our proofs are in Appendix D. Proposition (Example) 7.1. Let Γ be the Prisoner's Dilemma and Γ 𝑠 = (𝐴 𝑠 1 , 𝐴 𝑠 2 , 𝑢 𝑠 1 , 𝑢 𝑠 2 ) be any subset game of Γ with 𝐴 𝑠 1 = 𝐴 𝑠 2 = {Cooperate}. Then under Assumption 7.2, Γ 𝑠 is a strict SPI on Γ. Proposition (Example) 7.2. Let Γ be the Demand Game of Table 1 and Γ 𝑠 be the subset game described in Table 2. Under Assumptions 7.1 and 7.2, Γ 𝑠 is an SPI on
 , Sect. 16.4]. Theorem 8.2. The (strict) SPI decision problem is NP-complete, even for 2-player games. Proposition 8.3. For games Γ with |𝐴 1 | + ... + |𝐴 𝑛 | = 𝑚 that can be reduced (via iterative application of Assumption 7.2) to a game Γ ′ with |𝐴 ′ 1 | + ... + |𝐴 ′ 𝑛 | = 𝑙, the (strict) SPI decision problem can be solved in 𝑂 (𝑚 𝑙 ).
 1 1 , ..., 𝑎 𝑙 1 1 }, ..., {𝑎 1 𝑛 , ..., 𝑎 𝑙 𝑛 𝑛 }, u) be any game. Let Γ ′ be a perfect-coordination SPI on Γ. Then we can define u 𝑒 with values in C(Γ) such that under Assumption 7.1 the game Γ 𝑠 = Â1 { â1 1 , ..., â𝑙 1 1 }, ..., Â𝑛 { â1 𝑛 , ..., â𝑙 𝑛 𝑛 }, û : ( â𝑖 1 1 , ..., â𝑖 𝑛 𝑛 ) ↦ → u(𝑎 𝑖 1 1 , ..., 𝑎 𝑖 𝑛 𝑛 ), u 𝑒
 Table 3 : 3 Assurance Game preferences for the Prisoner's Dilemma Proof. By inspection of Algorithm 2, we see that exec(c) = Π(Γ 𝑠 ). It is left to show that c is a Nash equilibrium. So let 𝑖 be any player and 𝑐 ′ 𝑖 ∈ PROG 𝑖 − {𝑐 𝑖 }. We need to show that E 𝑢 𝑖 (exec(c −𝑖 , 𝑐 ′ 𝑖 )) ≤ E [𝑢 𝑖 (exec(c))]. Again, by inspection of c, exec(c −𝑖 , 𝑐 ′ 𝑖 ) is the threat point of Player 𝑖. Hence, E 𝑢 𝑖 (exec(c −𝑖 , 𝑐 ′ 𝑖 Player 2 Cooperate Defect Cooperate 4, 4 1, 3 Player 1 Defect 3, 1 2, 2
 Proposition (Example) 7.1. Let Γ be the Prisoner's Dilemma and Γ 𝑠 = (𝐴 𝑠 1 , 𝐴 𝑠 2 , 𝑢 𝑠 1 , 𝑢 𝑠 2 ) be any subset game of Γ with 𝐴 𝑠 1 = 𝐴 𝑠 2 = {Cooperate}. Then under Assumption 7.2, Γ 𝑠 is a strict SPI on Γ.Proof. By applying Assumption 7.2 twice and Transitivity once, Γ ∼ Φ Γ−{Cooperate}, where Φ(Defect, Defect) = {(Defect, Defect)} and Φ(𝑎 1 , 𝑎 2 ) = ∅ for all (𝑎 1 , 𝑎 2 ) ≠ (Defect, Defect). By Lemma 5.2.5, we further obtain Γ − {Cooperate} ∼ all Γ 𝑠 , where Γ 𝑠 is as described in the proposition. Hence, by transitivity, Γ ∼ all•Φ Γ 𝑠 . It is easy to verify that the function all • Φ is Pareto-improving. Proposition (Example) 7.2. Let Γ be the Demand Game of Table1and Γ 𝑠 be the subset game described in Table2. Under Assumptions 7.1 and 7.2, Γ 𝑠 is an SPI on Γ. Further, if 𝑃 (Π(Γ)=(DM, DM)) > 0, Γ 𝑠 is a strict SPI.Proof. Let (𝐴 1 , 𝐴 2 , 𝑢 1 , 𝑢 2 ) = Γ. We can repeatedly apply Assumption 7.2 to eliminate from Γ the strategies DL and RL for both players. We can then apply Lemma 5.2.3 (Transitivity) to obtain 𝐺 ∼ Φ Ĝ = ({DM, RM}, {DM, RM}, 𝑢 1 , 𝑢 2 ), whereΦ(𝑎 1 , 𝑎 2 ) = {(𝑎 1 , 𝑎 2 )} if 𝑎 1 , 𝑎2 ∈ {DM, RM} Next, by Assumption 7.1, Γ ∼ Ψ Γ 𝑠 , where Ψ 𝑖 (DM) = DL and Ψ 𝑖 (RM) = RL for 𝑖 = 1, 2. We can then apply Lemma 5.2.3 (Transitivity) again, to infer Γ ∼ Ψ•Φ Γ 𝑠 . It is easy to verify that for all (𝑎 1 , 𝑎 2 ) ∈ 𝐴 1 × 𝐴 2 , it is for all (𝑎 𝑠 1 , 𝑎 𝑠 2 ) ∈ Ψ(Φ(Γ 𝑠 )) the case that u(𝑎 𝑠 1 , 𝑎 𝑠 2 ) ≥ u(𝑎 1 , 𝑎 2 ). □ E PROOF OF THEOREM 8.2 □ D.2 Proof of Proposition (Example) 7.2 ∅ otherwise . (5) D.1 Proof of Proposition (Example) 7.1
 𝑛. If there is a subset game Γ𝑎 = ( Ã𝑎 , ũ𝑎 ) of Γ such that Γ𝑎 ∼ Φ Γ𝑎 by Assumption 7.1, then Γ𝑏 ∼ Φ | Â𝑏 Γ𝑏 , Note that if the correspondence function Φ is Pareto-improving, so is Φ| Â𝑏 . where Γ𝑏 = (Φ 1 ( Â𝑏 1 ), ..., Φ 𝑛 ( Â𝑏 𝑛 ), ũ𝑎 ).
 Proposition 8.3. For games Γ with |𝐴 1 | + ... + |𝐴 𝑛 | = 𝑚 that can be reduced (via iterative application of Assumption 7.2) to a game Γ ′ with |𝐴 ′ 1 | + ... + |𝐴 ′ 𝑛 | = 𝑙, the (strict) SPI decision problem can be solved in 𝑂 (𝑚 𝑙 ).
 So assume Ψ 1 (𝐷, 𝑗) ≠ Ψ 2 (𝐷, 𝑗). Recall that by item 2, it is Ψ 1 (𝐷, 𝑗), Ψ 2 (𝐷, 𝑗) ∈ {𝐶}×{1, ..., n}. Hence, u(Ψ 1 (𝐷, 𝑗), Ψ 2 (𝐷, 𝑗)) ≤ (1, 1) < (3, 3) = u((𝐷, 𝑗), (𝐷, 𝑗)),contradicting the assumption that Ψ is Pareto-improving. Finally, for 𝑗 ∈ {1, ..., 𝑛} it is Ψ 1 (𝐷, 𝑛 + 𝑗) = Item 3 n + Ψ 2 (𝐷, 𝑗)
			 Proc. of the 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2021), U. Endriss, A. Nowé, F. Dignum, A. Lomuscio (eds.), May 3-7, 2021, Online. © 2021 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). All rights reserved.