SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fivos Kalogiannis
fkalogia@uci.edu
The works of (Daskalakis et al., 2009, 2022; Jin et al., 2022; Deng et al., 2023) indicate that computing Nash equilibria in multi-player Markov games is a computationally hard task. This fact raises the question of whether or not computational intractability can be circumvented if one focuses on specific classes of Markov games. One such example is two-player zero-sum Markov games, in which efficient ways to compute a Nash equilibrium are known. Inspired by zero-sum polymatrix normal-form games (Cai et al., 2016), we define a class of zero-sum multi-agent Markov games in which there are only pairwise interactions described by a graph that changes per state. For this class of Markov games, we show that an  $\epsilon$ -approximate Nash equilibrium can be found efficiently. To do so, we generalize the techniques of (Cai et al., 2016), by showing that the set of coarse-correlated equilibria collapses to the set of Nash equilibria. Afterwards, it is possible to use any algorithm in the literature that computes approximate coarse-correlated equilibria Markovian policies to get an approximate Nash equilibrium.
Multi-agent reinforcement learning (MARL) is the discipline that is concerned with strategic interactions between agents who find themselves in a dynamically changing environment. Early aspects of MARL can be traced back to, as early as, the initial text on two-player zero-sum stochastic/Markov games (Shapley, 1953). Today, Markov games have been established as the theoretical framework for MARL (Littman, 1994). The connection between game theory and MARL has lead to several recent cornerstone results in benchmark domains in AI (Bowling et al., 2015; Brown and Sandholm, 2019, 2018; Brown et al., 2020; Silver et al., 2017; Moravčík et al., 2017; Perolat et al., 2022; Vinyals et al., 2019). The majority of the aforementioned breakthroughs relied on computing Nash equilibria (Nash, 1951) in a scalable and often decentralized manner. Although the theory of single agent reinforcement learning (RL) has witnessed an outstanding progress (e.g., see (Agarwal et al., 2020; Bertsekas, 2000; Jin et al., 2018; Li et al., 2021; Luo et al., 2019; Panait and Luke, 2005; Sidford et al., 2018; Sutton and Barto, 2018), and references therein), the landscape of multi-agent settings eludes a thorough understanding. In fact, guarantees for provably efficient computation of Nash equilibria remain limited to either environments in which agents strive to coordinate towards a shared goal (Chen et al., 2022; Claus and Boutilier, 1998; Ding et al., 2022; Fox et al., 2022; Leonardos et al., 2021; Maheshwari et al., 2022; Wang and Sandholm, 2002; Zhang et al., 2021) or fully competitive such as two-player zero-sum games (Cen et al., 2021; Condon, 1993; Daskalakis et al., 2020; Sayin et al., 2021, 2020; Wei et al., 2021) to name a few. Part of the lack of efficient algorithmic results in MARL is the fact that computing approximate Nash equilibria in (general-sum) games is computationally intractable (Daskalakis et al., 2009; Rubinstein, 2017; Chen et al., 2009; Etessami and Yannakakis, 2010) even when the games have a single state, i.e., normal-form two-player games.
We aim at providing a theoretical framework that captures an array of real-world applications with multiple agents — which admittedly correspond to a big portion of all modern applications. A recent contribution that computes NE efficiently in a setting that combines both collaboration and competition, (Kalogiannis et al., 2022), concerns adversarial team Markov games, or competition between an adversary and a group of uncoordinated agents with common rewards. Efficient algorithms for computing Nash equilibria in settings that include both cooperation and competition are far fewer and tend to impose assumptions that are restrictive and difficult to meet in most applications (Bowling, 2000; Hu and Wellman, 2003). The focus of our work is centered around the following question:
Are there any other settings of Markov games that encompass both competition and coordination while maintaining the tractability of Nash equilibrium computation?  $(\star)$
Inspired by contemporary works in algorithmic game theory and specifically zero-sum polymatrix normal-form games (Cai et al., 2016), we focus on the problem of computing Nash equilibria in zero-sum polymatrix Markov games. Informally, a polymatrix Markov game is a multi-agent Markov decision process with  $n$  agents, state-space  $S$ , action space  $\mathcal{A}_k$  for agent  $k$ , a transition probability model  $\mathbb{P}$  and is characterized by a graph  $\mathcal{G}_s(\mathcal{V},\mathcal{E}_s)$  which is potentially different in every state  $s$ . For a fixed state  $s$ , the nodes of the graph  $\mathcal{V}$  correspond to the agents, and the edges  $\mathcal{E}_s$  of the graph are two-player normal-form games (different per state). Every node/agent  $k$  has a fixed set of actions  $\mathcal{A}_k$  and chooses a strategy from this set to play in all games corresponding to adjacent edges. Given an action profile of all the players, the node's reward is the sum of its rewards in all games on the edges adjacent to it. The game is globally zero-sum if, for all strategy profiles, the rewards of all players add up to zero. Afterwards, the process transitions to a state  $s'$  according to  $\mathbb{P}$ . In a more high-level description, the agents interact over a network whose connections change at every state.
Our results. We consider a zero-sum polymatrix Markov game with the additional property that a single agent (not necessarily the same) controls the transition at each state, i.e., the transition model is affected by a single agent's actions for each state  $s$ . These games are known as switching controller Markov games. We show that we can compute in time poly  $(|S|, n, \max_{i \in [n]} |\mathcal{A}_i|, 1 / \epsilon)$  an  $\epsilon$ -approximate Nash equilibrium. The proof relies on the fact that zero-sum polymatrix Markov games with a switching controller have the following important property: the marginals of a coarse-correlated equilibrium constitute a Nash equilibrium (see Section 3.2). We refer to this phenomenon as equilibrium collapse. This property was already known for zero-sum polymatrix normal-form games by Cai et al. (2016) and our results generalize the aforementioned work for Markov games. As a corollary, we get that any algorithm in the literature that guarantees convergence to approximate coarse-correlated equilibria Markovian policies—e.g., (Daskalakis et al., 2022)—can be used to get approximate Nash equilibria. Our contribution also unifies previous results that where otherwise only applicable to the settings of single and switching-control two-player zero-sum games, or zero-sum polymatrix normal-form games. Finally, we show that the equilibrium collapsing phenomenon does not carry over if there are two or more controllers per state (see Section 3.3).
Technical overview. In order to prove our results, we rely on nonlinear programming and, in particular, nonlinear programs whose optima coincide with the Nash equilibria for a particular Markov game (Filar et al., 1991; Filar and Vrieze, 2012). Our approach is analogous to the one used by (Cai et al., 2016) which uses linear programming to prove the collapse of the set of CCE to the set of NE. Nevertheless, using the duality of linear programming in our case is not possible since a Markov game introduces nonlinear terms in the program. It is noteworthy that we do not need to invoke (Lagrangian) duality or an argument that relies on stationary points of a Lagrangian function. Rather, we use the structure of the zero-sum polymatrix Markov games with a switching controller to conclude the relation between a correlated policy and the individual policies formed by its marginals in terms of the individual utilities of the game.
1.1 Importance of zero-sum polymatrix Markov games
Strategic interactions of agents over a network is a topic of research in multiple disciplines that span computer science (Easley and Kleinberg, 2010), economics (Schweitzer et al., 2009), control theory (Tipsuwan and Chow, 2003), and biology (Szabó and Fath, 2007) to name a few.
In many environments where multiple agents interact with each other, they do so in a localized manner. That is, every agent is affected by the set of agents that belong to their immediate "neighborhood". Further, it is quite common that these agents will interact independently with each one of their
neighbors; meaning that the outcome of their total interactions is a sum of pairwise interactions rather than interactions that depend on joint actions. Finally, players might remain indifferent to actions of players are not their neighbors.
To illustrate this phenomenon we can think of multiplayer e-games (e.g., CS:GO, Fortnite, League of Legends, etc) where each player interacts through the same move only with players that are present on their premises and, in general, the neighbors cannot combine their actions into something that is not a mere sum of their individual actions (i.e., they rarely can "multiply" the effect of the individual actions). In other scenarios, such as strategic games played on social networks (e.g., opinion dynamics) agents clearly interact in a pairwise manner with agents that belong to their neighborhood and are somewhat oblivious to the actions of agents who they do not share a connection with.
With the proposed model we provide the theoretical framework needed to reason about such strategic interactions over dynamically changing networks.
From the literature of Markov games, we recognize the settings of single controller (Filar and Raghavan, 1984; Sayin et al., 2022; Guan et al., 2016; Qiu et al., 2021) and switching controller (Vrieze et al., 1983) Markov games to be one of the most related to ours. In these settings, all agents' actions affect individual rewards, but in every state one particular player (single controller), or respectively a potentially different one (switching controller), controls the transition of the environment to a new state. To the best of our knowledge, prior to our work, the only Markov games that have been examined under this assumption are either zero-sum or potential games.
Further, we manage to go beyond the dichotomy of absolute competition or absolute collaboration by generalizing zero-sum polymatrix games to their Markovian counterpart. In this sense, our work is related to previous works of Cai et al. (2016); Anagnostides et al. (2022); Ao et al. (2022) which show fast convergence to Nash equilibria in zero-sum polymatrix normal-form games for various no-regret learning algorithms including optimistic gradient descent.
Notation. We define  $[n] := \{1, \dots, n\}$ . Scalars are denoted using lightface variables, while, we use boldface for vectors and matrices. For simplicity in the exposition, we use  $O(\cdot)$  to suppress dependencies that are polynomial in the parameters of the game. Additionally, given a collection  $\mathbf{x}$  of policies or strategies for players  $[n]$ ,  $\mathbf{x}_{-k}$  denotes the policies of every player excluding  $k$ .
In its most general form, a Markov game (MG) with a finite number of  $n$  players is defined as a tuple  $\Gamma(H, \mathcal{S}, \{\mathcal{A}_k\}_{k \in [n]}, \mathbb{P}, \{r_k\}_{k \in [n]}, \gamma, \rho)$ . Namely,
-  $H \in \mathbb{N}_{+}$  denotes the time horizon, or the length of each episode,
-  $S$ , with cardinality  $S \coloneqq |\mathcal{S}|$ , stands for the state space,
-  $\{\mathcal{A}_k\}_{k\in [n]}$  is the collection of every player's action space, while  $\mathcal{A} := \mathcal{A}_1\times \dots \times \mathcal{A}_n$  denotes the joint action space; further, an element of that set—a joint action—is generally noted as  $\pmb{a} = (a_{1},\ldots ,a_{n})\in \mathcal{A}$ .
-  $\mathbb{P} := \{\mathbb{P}_h\}_{h \in [H]}$  is the set of all transition matrices, with  $\mathbb{P}_h : \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$ ; further,  $\mathbb{P}_h(\cdot | s, a)$  marks the probability of transitioning to every state given that the joint action  $a$  is selected at time  $h$  and state  $s$  — in infinite-horizon games  $\mathbb{P}$  does not depend on  $h$  and the index is dropped.
-  $r_k \coloneqq \{r_{k,h}\}$  is the reward function of player  $k$  at time  $h$ ;  $r_{k,h}: \mathcal{S}, \mathcal{A} \to [-1,1]$  yields the reward of player  $k$  at a given state and joint action — in infinite-horizon games,  $r_{k,h}$  is the same for every  $h$  and the index is dropped.
- a discount factor  $\gamma > 0$ , which is generally set to 1 when  $H < \infty$ , and  $\gamma < 1$  when  $H \to \infty$ .
- an initial state distribution  $\rho \in \Delta (\mathcal{S})$
Policies and value functions. We will define stationary and nonstationary Markov policies. When the horizon  $H$  is finite, a stationary policy equilibrium need not necessarily exist even for a single-agent MG, i.e., a Markov decision process; in this case, we seek nonstationary policies. For the case of infinite-horizon games, it is folklore that a stationary Markov policy Nash equilibrium always exists.
We note that a policy is Markovian when it depends on the present state only. A nonstationary Markov policy  $\pi_{k}$  for player  $k$  is defined as  $\pi_{k} := \{\pi_{k,h} : S \to \Delta(\mathcal{A}_{k}), \forall h \in [H]\}$ . It is a sequence of mappings of states  $s$  to a distribution over actions  $\Delta(\mathcal{A}_{k})$  for every timestep  $h$ . By  $\pi_{k,h}(a|s)$  we will denote the probability of player  $k$  taking action  $a$  in timestep  $h$  and state  $s$ . A Markov policy is said to be stationary in the case that it outputs an identical probability distribution over actions whenever a particular state is visited regardless of the corresponding timestep  $h$ .
Further, we define a nonstationary Markov joint policy  $\sigma \coloneqq \{\pi_h,\forall h\in [H]\}$  to be a sequence of mappings from states to distributions over joint actions  $\Delta (\mathcal{A})\equiv \Delta (\mathcal{A}_1\times \dots \times \mathcal{A}_n)$  for all times steps  $h$  in the time horizon. In this case, the players can be said to share a common source of randomness, or that the joint policy is correlated.
A joint policy  $\pi$  will be said to be a product policy if there exist policies  $\pi_k: [H] \times S \to \Delta(\mathcal{A}_k)$ ,  $\forall k \in [n]$  such that  $\pi_h = \pi_{1,h} \times \dots \times \pi_{n,h}$ ,  $\forall h \in [H]$ . Moreover, given a joint policy  $\pi$  we let a joint policy  $\pi_{-k}$  stand for the marginal joint policy excluding player  $k$ , i.e.,
$$
\pi_ {- k, h} (\boldsymbol {a} | s) = \sum_ {a ^ {\prime} \in \mathcal {A} _ {k}} \pi_ {h} (a ^ {\prime}, \boldsymbol {a} | s), \forall h \in [ H ], \forall s \in \mathcal {S}, \forall \boldsymbol {a} \in \mathcal {A} _ {- k}.
$$
By fixing a joint policy  $\pi$  we can define the value function of any given state  $s$  and timestep  $h$  for every player  $k$  as the expected cumulative reward they get from that state and timestep  $h$  onward,
$$
V _ {k, h} ^ {\boldsymbol {\pi}} (s _ {1}) = \mathbb {E} _ {\boldsymbol {\pi}} \left[ \sum_ {\tau = h} ^ {H} \gamma^ {\tau - 1} r _ {k, \tau} (s _ {\tau}, \boldsymbol {a} _ {\tau}) | s _ {1} \right] = \boldsymbol {e} _ {s _ {1}} ^ {\top} \sum_ {\tau = h} ^ {H} \left(\gamma^ {\tau - 1} \prod_ {\tau = h} ^ {h} \mathbb {P} _ {\tau} (\boldsymbol {\pi} _ {\tau})\right) \boldsymbol {r} _ {k, \tau} (\boldsymbol {\pi} _ {\tau}).
$$
Depending on whether the game is of finite or infinite horizon we get the followin displays,
- In finite-horizon games,  $\gamma = 1$ , the value function reads.
$$
V _ {k, h} ^ {\boldsymbol {\pi}} (s _ {1}) = \boldsymbol {e} _ {s _ {1}} ^ {\top} \sum_ {\tau = h} ^ {H} \left(\prod_ {\tau^ {\prime} = h} ^ {s _ {\tau}} \mathbb {P} _ {\tau^ {\prime}} (\boldsymbol {\pi} _ {\tau^ {\prime}})\right) \boldsymbol {r} _ {k, \tau} (\boldsymbol {\pi} _ {\tau}),
$$
- In infinite-horizon games, the value function of each state is,
$$
V _ {k} ^ {\boldsymbol {\pi}} (s _ {1}) = \boldsymbol {e} _ {s _ {1}} ^ {\top} (\mathbf {I} - \gamma \mathbb {P} (\boldsymbol {\pi})) ^ {- 1} \boldsymbol {r} (\boldsymbol {\pi}).
$$
Where  $\mathbb{P}_h(\pi_h),\mathbb{P}(\pi)$  and  $\pmb {r}_h(\pmb {\pi}_h),\pmb {r}(\pmb {\pi})$  denote the state-to-state transition probability matrix and expected per-state reward vector for a given policy  $\pi_h$  or  $\pmb{\pi}$  accordingly. Additionally,  $e_{s_1}$  is an all-zero vector apart of a value of 1 in its  $s_1$ -th position. Also, we denote  $V_{k,h}^{\pi}(\rho) = \sum_{s\in S}\rho (s)V_{k,h}^{\pi}(s)$ .
Best-response policies. Given an arbitrary joint policy  $\sigma$ , we define the best-response policy of a player  $k$  to be a policy  $\pi_k^\dagger \coloneqq \{\pi_{k,h}^\dagger, \forall h \in [H]\}$ , such that it is a maximizer of  $\max_{\pi_k'} V_{k,1}^{\pi_k' \times \sigma_{-k}}(s_1)$ . Additionally, we will use the following notation  $V_{k,h}^{\dagger,\sigma_{-k}}(s) \coloneqq \max_{\pi_k'} V_{k,h}^{\pi_k' \times \sigma_{-k}}(s)$ .
Equilibrium notions. Having defined what a best-response is, it is then quite direct to define different notions of equilibria for Markov games.
Definition 2.1 (CCE). We will say that a joint (potentially correlated) policy  $\sigma \in \Delta (\mathcal{A})^{H\times S}$  is an  $\epsilon$ -approximate coarse-correlated equilibrium if it holds that, for an  $\epsilon >0$
$$
V _ {k, 1} ^ {\dagger , \sigma - k} (s _ {1}) - V _ {k, 1} ^ {\sigma} (s _ {1}) \leq \epsilon , \forall k \in [ n ]. \tag {CCE}
$$
Further, we will define a Nash equilibrium policy,
Definition 2.2 (NE). A joint, product policy  $\pi \in \prod_{k\in [n]}\Delta (\mathcal{A}_k)^{H\times S}$  is an  $\epsilon$ -approximate Nash equilibrium if it holds that, for an  $\epsilon >0$
$$
V _ {k, 1} ^ {\dagger , \pi_ {- k}} (s _ {1}) - V _ {k, 1} ^ {\pi} (s _ {1}) \leq \epsilon , \forall k \in [ n ]. \tag {NE}
$$
It is quite evident that an approximate Nash equilibrium is also an approximate coarse-correlated equilibrium while the converse is not generally true. For infinite-horizon games the definitions are analogous and are deferred to the appendix.
2.2 Our setting
We focus on the setting of zero-sum polymatrix switching-control Markov games. This setting encompasses two major assumptions related to the reward functions in every state  $\{r_k\}_{k\in [n]}$  and the transition kernel  $\mathbb{P}$ . The first assumption imposes a zero-sum, polymatrix structure on  $\{r_k\}_{k\in [n]}$  for every state and directly generalizes zero-sum polymatrix games for games with multiple states.
Assumption 1 (Zero-sum polymatrix games). The reward functions of every player in any state  $s$  are characterized by a zero-sum, polymatrix structure.
Polymatrix structure. For every state  $s$  there exists an undirected graph  $\mathcal{G}_s(\mathcal{V},\mathcal{E}_s)$  where,
- the set of nodes  $\mathcal{V}$  coincides with the set of agents  $[n]$ ; the  $k$ -th node is the  $k$ -th agent,
- the set of edges  $\mathcal{E}_s$  stands for the set of pair-wise interactions; each edge  $e = (k,j), k,j \in [n], k \neq j$  stands for a general-sum normal-form game played between players  $k, j$  and which we note as  $\left(r_{kj}(s,\cdot,\cdot), r_{jk}(s,\cdot,\cdot)\right)$  with  $r_{kj}, r_{jk}: \mathcal{S} \times \mathcal{A}_k \times \mathcal{A}_j \to [-1,1]$ .
Moreover, we define  $\mathrm{adj}(s,k)\coloneqq \{j\in [n]\mid (k,j)\in \mathcal{E}_s\} \subseteq [n]$  to be the set of all neighbors of an arbitrary agent  $k$  in state  $s$ . The reward of agent  $k$  at state  $s$  given a joint action  $\pmb{a}$  depends solely on interactions with their neighbors,
$$
r _ {k, h} (s, \boldsymbol {a}) = \sum_ {j \in \operatorname {a d j} (k)} r _ {k j, h} (s, a _ {k}, a _ {j}), \forall h \in [ H ], \forall s \in \mathcal {S}, \forall \boldsymbol {a} \in \mathcal {A}.
$$
Further, the zero-sum assumption implies that,
$$
\sum_ {k} r _ {k, h} (s, \boldsymbol {a}) = 0, \quad \forall h \in [ H ], \forall s \in \mathcal {S}, \forall \boldsymbol {a} \in \mathcal {A}. \tag {1}
$$
In the infinite-horizon setting, the subscript  $h$  can be dropped.
A further assumption (switching-control) is necessary in order to ensure the desirable property of equilibrium collapse.
Assumption 2 (Switching-control). In every state  $s \in S$ , there exists a single player (not necessarily the same), or controller, whose actions determine the probability of transitioning to a new state.
The function  $\operatorname{argctrl} : S \to [n]$  returns the index of the player who controls the transition probability at a given state  $s$ . On the other hand, the function  $\operatorname{ctrl} : S \times \mathcal{A} \to \mathcal{A}_{\mathrm{argctrl}(s)}$  gets an input of a joint action  $\mathbf{a}$ , for a particular state  $s$ , and returns the action of the controller of that state,  $a_{\mathrm{argctrl}(s)}$ .
Remark 1. It is direct to see that Markov games with a single controller and turn-based Markov games (Daskalakis et al., 2022), are special case of Markov games with switching controller.
In this section we provide the main results of this paper. We shall show the collapsing phenomenon of coarse-correlated equilibria to Nash equilibria in the case of zero-sum, single switching controller polymatrix Markov games. Before we proceed, we provide a formal definition of the notion of collapsing.
Definition 3.1 (CCE collapse to NE). Let  $\sigma$  be any  $\epsilon$ -CCE policy of a Markov game. Moreover, let the marginal policy  $\pi^{\sigma} := (\pi_1^{\sigma}, \dots, \pi_n^{\sigma})$  be defined as:
$$
\pi_ {k} ^ {\sigma} (a | s) = \sum_ {\boldsymbol {a} _ {- k} \in \mathcal {A} _ {- k}} \sigma (a, \boldsymbol {a} _ {- k} | s), \forall k, \forall s \in \mathcal {S}, \forall a \in \mathcal {A} _ {k}.
$$
If  $\pi^{\sigma}$  is a  $O(\epsilon)$ -NE equilibrium for every  $\sigma$  then we say the set of approximate CCE's collapses to that of approximate NE's.
We start with the warm-up result that the set of CCE's collapses to the set of NE's for two-player zero-sum Markov games.
3.1 Warm-up: equilibrium collapse in two-player zero-sum MG's
Since we focus on two-player zero-sum Markov games, we simplify the notation by using  $V_{h=1}^{\cdot}(s) \coloneqq V_{2,1}^{\cdot}(s)$  — i.e., player 1 is the minimizing player and player 2 is the maximizer. We show the following theorem:
Theorem 3.1 (Collapse in two-player zero-sum MG's). Let a two-player zero-sum Markov game  $\Gamma'$  and an  $\epsilon$ -approximate CCE policy of that game  $\sigma$ . Then, the marginalized product policies  $\pi_1^\sigma, \pi_2^\sigma$  form a  $2\epsilon$ -approximate NE.
Proof. Since  $\sigma$  is an  $\epsilon$ -approximate CCE joint policy, by definition it holds that for any  $\pi_1$  and any  $\pi_2$ ,
$$
V _ {h = 1} ^ {\boldsymbol {\sigma} _ {- 2} \times \boldsymbol {\pi} _ {2}} (s _ {1}) - \epsilon \leq V _ {h = 1} ^ {\boldsymbol {\sigma}} (s _ {1}) \leq V _ {h = 1} ^ {\boldsymbol {\pi} _ {1} \times \boldsymbol {\sigma} _ {- 1}} (s _ {1}) + \epsilon .
$$
Due to Claim A.1, the latter is equivalent to the following inequality,
$$
V _ {h = 1} ^ {\boldsymbol {\pi} _ {1} ^ {\sigma} \times \boldsymbol {\pi} _ {2}} (s _ {1}) - \epsilon \leq V _ {h = 1} ^ {\sigma} (s _ {1}) \leq V _ {h = 1} ^ {\boldsymbol {\pi} _ {1} \times \boldsymbol {\pi} _ {2} ^ {\sigma}} (s _ {1}) + \epsilon .
$$
Plugging in  $\pi_1^\sigma, \pi_2^\sigma$  alternatingly, we get the inequalities:
$$
\left\{ \begin{array}{l} V _ {h = 1} ^ {\pi_ {1} ^ {\sigma} \times \pi_ {2}} (s _ {1}) - \epsilon \leq V _ {h = 1} ^ {\sigma} (s _ {1}) \leq V _ {h = 1} ^ {\pi_ {1} ^ {\sigma} \times \pi_ {2} ^ {\sigma}} (s _ {1}) + \epsilon \\ V _ {h = 1} ^ {\pi_ {1} ^ {\sigma} \times \pi_ {2} ^ {\sigma}} (s _ {1}) - \epsilon \leq V _ {h = 1} ^ {\sigma} (s _ {1}) \leq V _ {h = 1} ^ {\pi_ {1} \times \pi_ {2} ^ {\sigma}} (s _ {1}) + \epsilon \end{array} \right.
$$
The latter leads us to conclude that for any  $\pi_1$  and any  $\pi_2$ ,