SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 Team Double Oracle (Mix-and-Match)
Result: TMECor  
Input: Initial population  $\Pi_{\mathsf{T}}^{0}, \Pi_{\mathsf{O}}^{0}$   
repeat {for  $t = 0,1,\ldots$  } if Team DO-MM then  $(\mu_{\mathsf{T}}^{r}, \mu_{\mathsf{O}}^{r}) \gets \text{NE in restricted game } G(\mathcal{P}(\Pi_{\mathsf{T}}^{t}), \mathcal{P}(\Pi_{\mathsf{O}}^{t}))$  else  $(\mu_{\mathsf{T}}^{r}, \mu_{\mathsf{O}}^{r}) \gets \text{NE in restricted game } G(\Pi_{\mathsf{T}}^{t}, \Pi_{\mathsf{O}}^{t}) \{\text{Team DO}\}$ $\Pi_{\mathsf{T}}^{t+1} \gets \Pi_{\mathsf{T}}^{t} \cup \{\mathbb{BR}_{\mathsf{T}}(\mu_{\mathsf{O}}^{r})\}$ $\Pi_{\mathsf{O}}^{t+1} \gets \Pi_{\mathsf{O}}^{t} \cup \{\mathbb{BR}_{\mathsf{O}}(\mu_{\mathsf{T}}^{r})\}$  until  $u_{\mathsf{T}}(\mu_{\mathsf{T}}^{r}, \mu_{\mathsf{O}}^{r}) = u_{\mathsf{T}}(\mathbb{BR}_{\mathsf{T}}(\mu_{\mathsf{O}}), \mu_{\mathsf{O}}^{r}) = u_{\mathsf{T}}(\mu_{\mathsf{T}}^{r}, \mathbb{BR}_{\mathsf{O}}(\mu_{\mathsf{T}}))$  Return:  $(\mu_{\mathsf{T}}^{r}, \mu_{\mathsf{O}}^{r})$
terminating. Second, in extensive form games, the number of pure joint strategies is exponential in the size of the game. Lastly, computing a best response in team games is computationally intractable in general (Celli & Gatti, 2018).
Despite these issues, team DO is a promising method for the following reasons. First, it naturally outputs a measure of exploitability for free. This is useful to track progress of the algorithm, especially in large team games where computing exploitability is difficult. Second, team DO often terminates well before expanding all joint pure strategies. Indeed, as shown in our experiments below, we find that team DO terminates in a very low number of iterations for all of the games we tested. When team DO terminates in a small number of iterations, it is often much faster than existing methods. Lastly, in the same way that double oracle provides a foundation for PSRO, team DO provides a natural way of scaling up to large games by replacing the best response operator with deep reinforcement learning with Team PSRO. We describe this approach in the next section when we describe our main contribution, Team PSRO.
The main bottleneck for the tabular algorithm consists of the computation of the best response policy of each team. Taking for example the side of team  $\top$ , that task corresponds to solving the optimization problem
$$
\begin{array}{l} \mathbb{B}\mathbb{R}_{\mathsf{T}}(\mu_{\mathsf{O}}):= \operatorname *{arg  max}_{\mu_{\mathsf{T}}\in \Delta (\Pi_{\mathsf{T}1}\times \Pi_{\mathsf{T}2})}u_{\mathsf{T}}(\mu_{\mathsf{T}},\mu_{\mathsf{O}}) \\ = \underset {\mu_ {T} \in \Delta \left(\Pi_ {T 1} \times \Pi_ {T 2}\right)} {\arg \max } \left\{\sum_ {z \in Z} \hat {u} _ {T} (z) \binom {\sum_ {\pi_ {T 1} \in \Pi_ {T 1} (z)} \mu_ {T} \left(\pi_ {T 1}, \pi_ {T 2}\right)} {\pi_ {T 2} \in \Pi_ {T 2} (z)} \binom {\sum_ {\pi_ {O 1} \in \Pi_ {O 1} (z)} \mu_ {O} \left(\pi_ {O 1}, \pi_ {O 2}\right)} {\pi_ {O 2} \in \Pi_ {O 2} (z)} \right\}. \tag {4} \\ \end{array}
$$
While the above optimization problem is computationally intractable in the general case (Celli & Gatti, 2018), several computational approaches that can scale up to moderately sized games are known. Celli & Gatti (2018) propose simultaneous row and column generation using an integer-programming-based pricing oracle. Additionally, Farina et al. (2018a); Zhang et al. (2020); Farina et al. (2021) propose approaches based on integer programming, while Zhang & Sandholm (2021); Zhang et al. (2022) propose approaches based on tree decompositions. Finally, Farina & Sandholm (2020) propose a algorithm with worst-case polynomial-time guarantees in a subclass of games called triangle-free. As of today, even the best among these approaches can scale only up to games with a few tens of thousands of terminal sequences, and quickly run out of time or memory beyond that scale.
In our tabular experiments, we implemented a variant of the pricing algorithm of Celli & Gatti (2018) to compute best responses for the teams. The algorithm we implemented constructs an integer program to capture the domain of feasible  $\mu_{\mathsf{T}}$  in the best-response problem (4). We chose the algorithm by Celli & Gatti (2018) since it only requires a linear number of integer variables in the size of the game tree (this is in contrast, for example, with other integer-programming-based approaches that use a quadratic number of integer variables to achieve a tighter formulation, see e.g. Farina et al. (2021); other approaches such as Zhang & Sandholm (2021); Zhang et al. (2022) use a worst-case exponential number of variables to construct an exact linear programming relaxation of the optimization problem). Pseudocode for the implemented best-response oracle is available in the Appendix.
4.2 Tabular Team Double Oracle Mix-and-Match
In this section we introduce team double oracle mix-and-match (Team DO-MM), which builds on Team DO by adding additional joint policies to the population for free. Similarly to Team DO, Team DO-MM maintains a population of pure joint strategies for each team  $\Pi_{\top}$ ,  $\Pi_{\circ}$ . Each pure strategy in a team's population is a pure joint strategy for each player on that team  $\mu_{\top} = (\pi_{\top 1}, \pi_{\top 2})$ . In Team DO-MM, however, we construct a restricted game in an expanded space of pure strategies that considers all possible mixtures of policies from each joint best response. Contrasted with Team DO, Team DO-MM considers pure strategies that mix and match policies from different best responses of the players within each team. Our restricted game  $G(\mathcal{P}(\Pi_{\top}), \mathcal{P}(\Pi_{\circ}))$  is a two-player zero-sum normal-form game that takes as actions the pure strategies  $\mu_{\top} \in \mathcal{P}(\Pi_{\top})$  for the team and  $\mu_{\circ} \in \mathcal{P}(\Pi_{\circ})$  for the opponent team. Similar to Team DO, every iteration, the NE  $(\mu_{\top}^{r}, \mu_{\circ}^{r})$  of the restricted game  $G(\mathcal{P}(\Pi_{\top}^{t}), \mathcal{P}(\Pi_{\circ}^{t}))$  is computed. Next, a best response to this NE in the original game  $\mathbb{BR}_{\circ}(\mu_{\top}^{r})$  is computed and added to the population. Team DO-MM also terminates when the best response for both players does not improve over the Nash value of the restricted game. Because upon termination no best response can improve on the Nash value, Team DO-MM inherits the convergence guarantee of Team DO to TMECor. Team DO-MM is shown as a one-line change in Algorithm 1.
Proposition 1. Tabular Team DO and Team DO-MM with exact best responses converge to a TMECor.
Proof. Proof is contained in the Appendix.
4.3 Team PSRO
We now describe our main contribution, team PSRO and team PSRO Mix-and-Match. All existing tabular algorithms cannot scale to a game the size of Google Research Football. And existing algorithms such as self-play do not minimize exploitability, even though they might do well sometimes in practice. Team PSRO scales up Team DO to large games by swapping out an exact joint best response  $\mathbb{BR}_{\top}(\mu_{\mathrm{O}}^{r})$  for an approximate joint best response  $\beta_{\top}$  that is trained through reinforcement learning. Instead of learning a policy for each player individually, as in PSRO, Team PSRO trains a joint best response every iteration. This joint best response is a pair of policies that are trained using cooperative RL. In particular, we use MAPPO to train the joint best response. As in PSRO, once the joint best response for a team is trained, it is added to that team's population  $\Pi_{\top}^{t}$  and a two-player zero-sum normal form restricted game  $G(\Pi_{\top}^{t}, \Pi_{\mathrm{O}}^{t})$  is created. The actions of this normal form game correspond to pure joint strategies in the population and the payoffs correspond to the estimated expected value when one team chooses one joint policy and the other team chooses another joint policy  $u_{\top}(\beta_{\top}(\mu_{\mathrm{O}}), \mu_{\mathrm{O}}^{r})$ . Payoff values in the restricted game are estimated by sampling rollouts from both teams' joint policies. Team PSRO is described in Algorithm 2.
Note that when the joint best responses achieve expected value within  $\epsilon$  of the optimal best response, then upon convergence Team PSRO will output an approximate TMECor. Even when the cooperative reinforcement learning is not within  $\epsilon$  of the optimal best response, we find that in practice the reward that joint best responses achieve against the restricted NE tends to go down over time. We call this evaluation metric of the sum of the reward that both joint best responses achieve against the restricted NE approximate exploitability because the this is a lower bound on the exact exploitability. The better the cooperative RL algorithm is in team PSRO, the more accurately the approximate exploitability will match true exploitability. As mentioned previously, a key benefit of Team PSRO is that it outputs approximate exploitability as a byproduct of the algorithm. If the RL algorithm is very strong and Team PSRO converges, then one can be reasonably sure that the strategy produced is strong.
The MAPPO algorithm we use contains a centralized critic value function that has access to the history of the game. Since we train the joint best response centrally but execute decentrally, MAPPO allows the critic to see the partner's cards at training time to reduce variance in the update. Another MAPPO trick we use is to share the parameters of the neural network between partners in a joint best response. This has been found to help stabilize training. Although we use MAPPO as our joint RL best response, in principle any cooperative RL algorithm could be used to learn an approximate joint best response.
Algorithm 2 Team PSRO (Mix-and-Match)
Result: Approximate TMECorr  
Input: Initial population  $\Pi_{\mathsf{T}}^{0}, \Pi_{\mathsf{O}}^{0}$   
repeat {for  $t = 0,1,\ldots$  }  
if Team-PSRO-MM then  
 $(\mu_{\mathsf{T}}^{r}, \mu_{\mathsf{O}}^{r}) \gets \text{NE in restricted game } G(\mathcal{P}(\Pi_{\mathsf{T}}^{t}), \mathcal{P}(\Pi_{\mathsf{O}}^{t}))$   
else  
 $(\mu_{\mathsf{T}}^{r}, \mu_{\mathsf{O}}^{r}) \gets \text{NE in restricted game } G(\Pi_{\mathsf{T}}^{t}, \Pi_{\mathsf{O}}^{t})$   
for  $m$  iterations do  
Update joint best response  $\beta_{\mathsf{T}}$  toward  $\mathbb{BR}_{\mathsf{T}}(\mu_{\mathsf{O}}^{r})$  via cooperative RL  
Update joint best response  $\beta_{\mathsf{O}}$  toward  $\mathbb{BR}_{\mathsf{O}}(\mu_{\mathsf{T}}^{r})$  via cooperative RL  
 $\Pi_{\mathsf{T}}^{t+1} \gets \Pi_{\mathsf{T}}^{t} \cup \{\beta_{\mathsf{T}}\}$ $\Pi_{\mathsf{O}}^{t+1} \gets \Pi_{\mathsf{O}}^{t} \cup \{\beta_{\mathsf{O}}\}$   
until  $u_{\mathsf{T}}(\beta_{\mathsf{T}}(\mu_{\mathsf{O}}), \mu_{\mathsf{O}}^{r}) + u_{\mathsf{O}}(\mu_{\mathsf{T}}^{r}, \beta_{\mathsf{O}}(\mu_{\mathsf{T}})) \leq u_{\mathsf{T}}(\mu_{\mathsf{T}}^{r}, \mu_{\mathsf{O}}^{r}) + u_{\mathsf{O}}(\mu_{\mathsf{T}}^{r}, \mu_{\mathsf{O}}^{r}) + \epsilon$   
Return:  $(\mu_{\mathsf{T}}^{r}, \mu_{\mathsf{O}}^{r})$
4.4 Team PSRO Mix-and-Match
Similarly to how Team DO-MM builds on Team DO by adding mixed-and-matched pure strategies, Team PSRO Mix-and-Match (Team PSRO-MM) builds on Team PSRO by adding mixed-and-matched deep RL policies. As in Team PSRO, once the joint best response for a team is trained, it is added to that team's population  $\Pi_{\mathsf{T}}^{t}$ . Similarly to Team DO-MM, an expanded two-player zero-sum normal form restricted game  $G(\mathcal{P}(\Pi_{\mathsf{T}}^{t}), \mathcal{P}(\Pi_{\mathsf{O}}^{t}))$  is created. The actions of this normal form game correspond to mixed-and-matched joint strategies in the population and the payoffs correspond to the estimated expected value when one team chooses one joint policy and the other team chooses another joint policy  $u_{\mathsf{T}}(\beta_{\mathsf{T}}(\mu_{\mathsf{O}}), \mu_{\mathsf{O}}^{r})$ . Payoff values in the restricted game are estimated by sampling rollouts from both teams' joint policies. Team PSRO-MM is described in Algorithm 2.
Figure 1: Experimental results in tabular games
5.1 Team DO Experiments
We run Team DO and Team DO-MM on two small games, Kuhn poker and Liar's dice, and present the results in Figure 1. Although we do not know of any existing regret bounds for double oracle techniques, these results demonstrate that in practice double oracle methods work well and can find approximate TMECor in a reasonable number of iterations. Notably, Team DO-MM is able to achieve faster convergence compared to Team DO in the number of iterations. However, we found that since calculating best responses in this game does not take much time, computing the extra payoff values caused Team DO-MM to converge much slower in wall clock time compared to Team DO.
Additionally, both Team-DO methods outperform Fictitious Team Play (Farina et al., 2018b), which adds team best responses to the opponent average strategy every iteration. In large games, as shown below, finding a best response requires much more time than evaluating the expected value of two team policies, so Team DO-MM converges faster in wall clock time as well. We include a detailed analysis of wall clock time on these tabular experiments in the appendix.
5.2 Team PSRO Experiments
(a) Elo
(b) Goal Diff. vs. AI
Figure 2: Team PSRO Results on Google Research Football. Both Team PSRO and Team PSRO-MM outperform self play, with Team PSRO-MM performing the best.
(c) Relative Pop. Performance
We demonstrate that Team PSRO can scale to large games by training it on the benchmark game of Google Research Football (GRF) (Kurach et al., 2020). Because there do not exist any other game-theoretic techniques for approaching TMECor in large games, we only compare to self-play. Although we do not know of any theoretical guarantees that self-play will converge to TMECor, methods based on self-play (Berner et al., 2019b; Liu et al., 2019; Jaderberg et al., 2019) have achieved impressive performance on large team games. As a result, we believe that self-play reinforcement learning is a strong benchmark to beat. In these experiments we use the four-player version of GRF and modify it so that each player only observes the information of his own team, the ball, and three closest opponents' information (which means that there are always two opponents that are not observable for a player). We also include results on the perfect-information version of the game in the Appendix and find that the same pattern holds where Team PSRO-MM is better than Team PSRO which is better than self-play. We also include further training details in the Appendix.
We use the same reinforcement learning algorithm for both Team PSRO and self play, namely MAPPO with a centralized critic and shared parameters for a team. We allow the centralized critic to see all players' information. Self play reinforcement learning continually trains one joint policy for one team against the other joint policy for the other team and does not make use of a population like Team PSRO. In Figure 2, we see that Team PSRO and Team PSRO-MM are both able to outperform self-play across all three measures of Elo, average goal difference against the built-in AI, and relative population performance (Balduzzi et al., 2019). Furthermore, Team PSRO-MM outperforms Team PSRO with a small additional cost of extra rollouts to evaluate the matrix payoffs. Since this is such a large game, the time needed to compute the BRs is much more than the time needed to evaluate two team policies, so the extra time is negligible.
6.1 Limitations
As with any double oracle algorithm, the main limitation of our algorithm is that it suffers from a lack of practical guarantees. It is possible that all pure strategies of the game must be expanded before termination, and indeed we find that in Kuhn poker, even using an exact best response plateaus indefinitely. With that being said, sometimes exploitability can be too strict of a measure if the exact best response is hard to find. In the case of team games, where finding an exact best response is NP-Hard, perhaps being approximately unexploitable against a reinforcement learning best response is a reasonable goal, and one that we achieve with Team-PSRO.
6.2 Future Work
There are many future directions to build on this work. First, we are interested in continuing to scale up to large games, with the goal of becoming superhuman at bridge. A second direction of future work involves studying the problem of cooperative reinforcement learning directly, where improvements should directly transfer to our setting. A third research direction is transferring over new PSRO algorithms such as Anytime PSRO (McAleer et al., 2022b), and NXDO (McAleer et al., 2021). Related efforts in solving team games using regret minimization and subgame solving are possibly complimentary to our approach.
This material is based on work supported by the Vannevar Bush Faculty Fellowship ONR N00014-23-1-2876, National Science Foundation grants RI-2312342 and RI-1901403, ARO award W911NF2210266, and NIH award A240108S001.
Anagnostides, I., Kalogiannis, F., Panageas, I., Vlatakis-Gkaragkounis, E.-V., and McAleer, S. Algorithms and complexity for computing nash equilibria in adversarial team games. Economics and Computation (EC), 2023.
Balduzzi, D., Garnelo, M., Bachrach, Y., Czarnecki, W., Perolat, J., Jaderberg, M., and Graepel, T. Open-ended learning in symmetric zero-sum games. International Conference on Machine Learning (ICML), 2019.
Basilico, N., Celli, A., Nittis, G. D., and Gatti, N. Team-maxmin equilibrium: efficiency bounds and algorithms. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp. 356-362, 2017.
Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019a.
Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019b.
Bowling, M., Burch, N., Johanson, M., and Tammelin, O. Heads-up limit hold'em poker is solved. Science, 347(6218), January 2015.
Brown, N. and Sandholm, T. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science, pp. eaao1733, Dec. 2017.
Celli, A. and Gatti, N. Computational results for extensive-form adversarial team games. In AAAI Conference on Artificial Intelligence (AAAI), pp. 965-972, 2018.
Farina, G. and Sandholm, T. Polynomial-time computation of optimal correlated equilibria in two-player extensive-form games with public chance moves and beyond. In *ArXiv preprint*, 2020.
Farina, G., Celli, A., Gatti, N., and Sandholm, T. Ex ante coordination and collusion in zero-sum multi-player extensive-form games. In Advances in Neural Information Processing Systems, pp. 9638-9648, 2018a.
Farina, G., Celli, A., Gatti, N., and Sandholm, T. Ex ante coordination and collusion in zero-sum multi-player extensive-form games. Advances in Neural Information Processing Systems, 31, 2018b.
Farina, G., Celli, A., Gatti, N., and Sandholm, T. Connecting optimal ex-ante collusion in teams to extensive-form correlation: Faster algorithms and positive complexity results. In International Conference on Machine Learning, 2021.
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019b.
von Stengel, B. and Koller, D. Team-maxmin equilibria. Games and Economic Behavior, 21(1-2): 309-321, 1997.
Wang, Y., Han, B., Wang, T., Dong, H., and Zhang, C. {DOP}: Off-policy multi-agent decomposed policy gradients. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=6FqKiVAdI3Y.
Wellman, M. P. Methods for empirical game-theoretic analysis. AAAI conference on artificial intelligence, 2006.
Xu, Z., Liang, Y., Yu, C., Wang, Y., and Wu, Y. Fictitious cross-play: Learning global nash equilibrium in mixed cooperative-competitive games. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pp. 1053–1061, 2023.
Yao, J., Liu, W., Fu, H., Yang, Y., McAleer, S., Fu, Q., and Yang, W. Policy space diversity for non-transitive games. Conference on Neural Information Processing Systems (NeurIPS), 2023.
Yu, C., Velu, A., Vinitsky, E., Wang, Y., Bayen, A., and Wu, Y. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.
Zhang, B. H. and Sandholm, T. Team correlated equilibria in zero-sum extensive-form games via tree decompositions. arXiv preprint arXiv:2109.05284, 2021.
Zhang, B. H., Farina, G., and Sandholm, T. Team belief dag form: A concise representation for team-correlated game-theoretic decision making. arXiv preprint arXiv:2202.00789, 2022.
Zhang, B. H., Farina, G., Anagnostides, I., Cacciamani, F., McAleer, S. M., Haupt, A. A., Celli, A., Gatti, N., Conitzer, V., and Sandholm, T. Computing optimal equilibria and mechanisms via learning in zero-sum extensive-form games. Conference on Neural Information Processing Systems (NeurIPS), 2023.
Zhang, Y. and An, B. Computing team-maxmin equilibria in zero-sum multiplayer extensive-form games. In AAAI Conference on Artificial Intelligence (AAAI), pp. 2318-2325, 2020a.
Zhang, Y. and An, B. Converging to team-maxmin equilibria in zero-sum multiplayer games. In International Conference on Machine Learning (ICML), 2020b.
Zhang, Y., An, B., and Černý, J. Computing ex ante coordinated team-maxmin equilibria in zero-sum multiplayer extensive-form games. In ArXiv preprint:2009.12629, 2020.
Proof. Note that by Equation 3 and the stopping condition in Algorithm 1, when Team DO(-MM) terminates, the meta-NE  $(\mu_{\mathrm{T}}^{r},\mu_{\mathrm{O}}^{r})$  is a TMECor. What is left to show is that Team DO(-MM) eventually terminates. Team DO(-MM) eventually terminates because in every iteration it either adds a new strategy for at least one player or it terminates. To see this, consider the case where no new strategies are added for either player in an iteration. Then the best response value for each player is no greater than the value of their meta-NE against the opponent, so Equation 3 is satisfied and Team DO(-MM) terminates in a TMECor.
B Tabular Best-Response Oracle
In this section we give the formulation of the team-best-response oracle we implemented in the tabular case.
As discussed in the body (Section 4.1), the algorithm is based on integer programming and implements the algorithm introduced by Celli & Gatti (2018). The integer programming formulation uses the
fact that by convexity, the optimal solution  $\mu^{*} \in \Delta(\Pi_{\mathsf{T}1} \times \Pi_{\mathsf{T}2})$  in (4) puts all mass on exactly a pair  $(\pi_{\mathsf{T}1}^{*}, \pi_{\mathsf{T}2}^{*}) \in \Pi_{\mathsf{T}1} \times \Pi_{\mathsf{T}2}$ . With that, for any  $z \in Z$  we can write
$$
\sum_{\substack{\pi_{T1}\in \Pi_{T1}(z)\\ \pi_{T2}\in \Pi_{T2}(z)}}\mu_{T}^{*}(\pi_{T1},\pi_{T2}) = \mathbf{1}[\pi_{T1}^{*}\in \Pi_{T1}(z)]\cdot \mathbf{1}[\pi_{T2}^{*}\in \Pi_{T2}(z)] = \pi_{T1}^{*}[\sigma_{T1}(z)]\cdot \pi_{T2}^{*}[\sigma_{T2}(z)].
$$
Hence, we can rewrite (4) as
$$
\mathbb{B}\mathbb{R}_{\mathsf{T}}(\mu_{\mathsf{O}}) = \operatorname *{arg  max}_{(\pi_{\mathsf{T}1}^{*},\pi_{\mathsf{T}2}^{*})\in \Pi_{\mathsf{T}1}\times \Pi_{\mathsf{T}2}}\left\{\sum_{z\in Z}\hat{u}_{\mathsf{T}}(z)\pi_{\mathsf{T}1}^{*}[\sigma_{\mathsf{T}1}(z)]\pi_{\mathsf{T}2}^{*}[\sigma_{\mathsf{T}2}(z)]\left(\sum_{\substack{\pi_{\mathsf{O}1}\in \Pi_{\mathsf{O}1}(z)\\ \pi_{\mathsf{O}2}\in \Pi_{\mathsf{O}2}(z)}}\right)\right\} .
$$