Title: Policy Space Diversity for Non-Transitive Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: Policy Diversity

Policy-Space Response Oracles

Nash Equilibrium

Multi-agent Reinforcement Learning
Abstract: Policy-Space Response Oracles (PSRO) is an influential algorithm framework for approximating a Nash Equilibrium (NE) in multi-agent non-transitive games. Many previous studies have been trying to promote policy diversity in PSRO. A major weakness with existing diversity metrics is that a more diverse (according to their diversity metrics) population does not necessarily mean (as we proved in the paper) a better approximation to a NE. To alleviate this problem, we propose a new diversity metric, the improvement of which guarantees a better approximation to a NE. Meanwhile, we develop a practical and well-justified method to optimize our diversity metric using only state-action samples. By incorporating our diversity regularization into the best response solving of PSRO, we obtain a new PSRO variant, \textit{Policy Space Diversity} PSRO (PSD-PSRO). We present the convergence property of PSD-PSRO. Empirically, extensive experiments on single-state games, Leduc, and Goofspiel demonstrate that PSD-PSRO is more effective in producing significantly less exploitable policies than state-of-the-art PSRO variants.

Policy Space Diversity for Non-Transitive Games

Abstract

Policy-Space Response Oracles (PSRO) is an influential algorithm framework for approximating a Nash Equilibrium (NE) in multi-agent non-transitive games. Many previous studies have been trying to promote policy diversity in PSRO. A major weakness in existing diversity metrics is that a more diverse (according to their diversity metrics) population does not necessarily mean (as we proved in the paper) a better approximation to a NE. To alleviate this problem, we propose a new diversity metric, the improvement of which guarantees a better approximation to a NE. Meanwhile, we develop a practical and well-justified method to optimize our diversity metric using only state-action samples. By incorporating our diversity regularization into the best response solving in PSRO, we obtain a new PSRO variant, Policy Space Diversity PSRO (PSD-PSRO). We present the convergence property of PSD-PSRO. Empirically, extensive experiments on various games demonstrate that PSD-PSRO is more effective in producing significantly less exploitable policies than state-of-the-art PSRO variants. The experiment code is available at https://github.com/nigelyaoj/policy-space-diversity-psro.

1 Introduction

Most real-world games demonstrate strong non-transitivity [9], where the winning rule follows a cyclic pattern (e.g., the strategy cycle in Rock-Paper-Scissors) [6, 3]. A common objective in solving non-transitive games is to find a Nash Equilibrium (NE), which has the best worst-case performance in the whole policy space. Traditional algorithms, like simple self-play, fail to converge to a NE in games with strong non-transitivity [26]. Recently, many game-theoretic methods have been proposed to approximate a NE in such games. For example, Counterfactual Regret Minimization (CFR) [54] minimizes the so-called counterfactual regret. Neural fictitious self play [18, 19] extends the classical game-theoretic approach, Fictitious Play (FP) [4], to larger games using Reinforcement Learning (RL) to approximate a Best Response (BR). Another well-known algorithm is Policy-Space Response Oracles (PSRO) [26], which generalizes the double oracle approach [38] by adopting a RL subroutine to approximate a BR.

Improving the performance of PSRO on approximating a NE is an active research topic, and many PSRO variants have been proposed so far, which generally fall into three categories. The first category [36, 45, 12, 29] aims to improve the training efficiency at each iteration. For instance, pipeline-PSRO [36] trains multiple BRs in parallel at each iteration. Neural population learning [29] enables fast transfer learning across policies via representing a population of policies within a single conditional model. The second category [37, 53] incorporates no-regret learning into PSRO, which solves an unrestricted-restricted game with a no-regret learning method to guarantee the decrease of exploitability of the meta-strategy across each iteration. The third category [2, 41, 32, 33]

promotes policy diversity in the population, which is usually implemented by incorporating a diversity regularization term into the BR solving in the original PSRO.

Despite achieving promising improvements over the original PSRO, the theoretical reason why the diversity metrics in existing diversity-enhancing PSRO variants [2, 41, 32, 33] help PSRO in terms of approximating a NE is unclear. More specifically, those diversity metrics are 'justified' in the sense that adding the corresponding diversity-regularized BR strictly enlarges the gamescape . However, as we prove and demonstrate later in the paper, a population with a larger gamescape is neither a sufficient nor a necessary condition for a better approximation (we will define the precise meaning later) to a NE. One fundamental reason is that gamescape is a concept that varies significantly according to the choice of opponent policies. In contrast, exploitability (the distance to a NE) measures the worst-case performance that is invariant to the choice of opponent policies.

In this paper, we seek for a new and better justified diversity metric that improves the approximation of a NE in PSRO. To achieve this, we introduce a new concept, named Population Exploitability (PE), to quantify the strength of a population. The PE of a population is the optimal exploitability that can be achieved by selecting a policy from its Policy Hull (PH), which is simply a complete set of polices that are convex combinations of individual polices in the population. In addition, we show that a larger PH means a lower PE. Based on these insights, we make the following contributions:

We point out a major and common weakness of existing diversity-enhancing PSRO variants: their goal of enlarging the gamescape of the population in PSRO is somewhat deceptive to the extent that it can lead to a weaker population in terms of PE. In other words, a more diverse (according to their diversity metrics) population ⇒ a larger gamescape ⇏ closer to a full game NE.

We develop a new diversity metric that encourages the enlargement of a population's PH. In addition, we develop a practical and well-justified method to optimize our diversity metric using only state-action samples. We then incorporate our diversity metric (as a regularization term) into the BR solving in the original PSRO and obtain a new algorithm: Policy Space Diversity PSRO (PSD-PSRO). Our method PSD-PSRO establishes the causality: a more diverse (according to our diversity metric) population ⇒ a larger PH ⇒ a lower PE ⇒ closer to a full game NE.

We prove that a full game NE is guaranteed once PSD-PSRO is converged. In contrast, it is not clear, in other state-of-the-art diversity-enhancing PSRO variants [2, 41, 32, 33], whether a full game NE is found once they are converged in terms of their optimization objectives. Notably, PSRO rN [2] is not guaranteed to find a NE once converged [36].

2 Notations and Preliminary

2.1 Extensive-form Games, NE, and Exploitability

Extensive-form games are used to model sequential interaction involving multiple agents, which can be defined by a tuple ⟨N , S , P, A , u ⟩ . N = { 1 , 2 } denotes the set of players (we focus on the two-player zero-sum games). S is a set of information states for decision-making. Each information state node s ∈ S includes a set of actions A ( s ) that lead to subsequent information states. The player function P : S → N ∪ { c } , with c denoting chance, determines which player takes action in s . We use s i , S i = { s ∈ S| P ( s ) = i } , and A i = ∪ s ∈S i A ( s ) to denote player i 's state, set of states, and set of actions respectively. We consider games with perfect recall , where each player remembers the sequence of states to the current state.

Aplayer's behavioral strategy is denoted by π i ( s ) ∈ ∆( A ( s )) , ∀ s ∈ S i , and π i ( a | s ) is the probability of player i taking action a in s . A strategy profile π = ( π 1 , π 2 ) is a pair of strategies for each player, and we use π -i to refer to the strategy in π except π i . u i ( π ) = u i ( π i , π -i ) denotes the payoff for player i when both players follow π . The BR of player i to the opponent's strategy π -i is denoted by BR ( π -i ) = arg max π ′ i u i ( π ′ i , π -i ) . The exploitability of strategy profile π is defined as:

When E ( π ) = 0 , π is a NE of the game.

2.2 Meta-Games, PH, and PSRO

Meta-games are introduced to represent games at a higher level. Denoting a population of mixed strategies for player i by Π i := { π 1 i , π 2 i , ... } , the payoff matrix on the joint population Π = Π i × Π -i is denoted by M Π i , Π -i , where M Π i , Π -i [ j, k ] := u i ( π j i , π k -i ) . The meta-game on Π and M Π i , Π -i is simply a normal-form game where selecting an action means choosing which π i to play for player i . Accordingly, we use σ i ( σ i is called a meta-strategy and could be, e.g., playing [ π 1 i , π 2 i ] with probability [0 . 5 , 0 . 5] ) to denote a mixed strategy over Π i , i.e., σ i ∈ ∆ Π i . A meta-policy σ i over Π i can be viewed as a convex combination of polices in Π i , and we define the PH of a population H (Π i ) as the set of all convex combinations of the policies in Π i . Meta-games are often open-ended in the sense that there exist an infinite number of mixed strategies and that new policies will be successively added to Π i and Π -i respectively. We give a summary of notations in Appendix A.

PSRO operates on meta-games and consists of two components: an oracle and a meta-policy solver. At each iteration t , PSRO maintains a population of policies, denoted by Π t i , for each player i . The joint meta-policy solver first computes a NE meta-policy σ t on the restricted meta-game represented by M Π t i , Π t -i . Afterwards, for each player i , the oracle computes an approximate BR (i.e., π t +1 i ) against the meta-policy σ t -i : π t +1 i ∈ BR ( σ t -i ) . The new policy π t +1 i is then added to its population ( Π t +1 i = Π t i ∪ { π t +1 i } ), and the next iteration starts. In the end, PSRO outputs a meta-policy NE on the final joint population as an approximation to a full game NE.

2.3 Previous Diversity Metrics for PSRO

Effective diversity [2] measures the variety of effective strategies (strategies with support under a meta-policy NE) and uses a rectifier to focus on how these effective strategies beat each other. Let ( σ ∗ i , σ ∗ -i ) denote a meta-policy NE on M Π i , Π -i . The effective diversity of Π i is:

where ⌊ x ⌋ + := x if x ≥ 0 else 0 .

Expected Cardinality [41], inspired by the determinantal point processes [35], measures the diversity of a population Π i as the expected cardinality of the random set Y sampled according to det ( L Π ) :

where | Y | is the cardinality of Y , and L Π = M Π i , Π -i M T Π i , Π -i .

Convex Hull Enlargement [32] builds on the idea of enlarging the convex hull of all row vectors in the payoff matrix:

where π ′ i is the new strategy to add, and m is the payoff vector of policy π ′ i against each opponent policy in Π -i : m [ j ] = u i ( π ′ i , π j -i ) .

Occupancy Measure Mismatching [32] considers the state-action distribution ρ π ( s, a ) induced by a joint policy π . When considering adding a new policy π ′ i , the corresponding diversity metric is:

where π ′ i is the new policy to add; ( σ ∗ i , σ ∗ -i ) is a meta-policy NE on M Π i , Π -i , and D f is a general f -divergence between two distributions. It is worth noting that Equation 5 only considers the difference between two policies ( π ′ i and σ ∗ i ), instead of π ′ i and Π i . In practice, this diversity metric is used together with the convex hull enlargement in [32].

Unified Diversity Measure [33] offers a unified view on existing diversity metrics and is defined as:

where f takes different forms for different existing diversity metrics; λ m is the eigenvalues of [ K ( ϕ m , ϕ n )] | Π i |×| Π i | ; K ( · , · ) is a predefined kernel function; and ϕ m is the strategy feature for the m -th policy in Π i . It is worth mentioning that only payoff vectors in M Π i , Π -i were investigated for the strategy feature of the new diversity metric proposed in [33].

3 A Common Weakness of Existing Diversity-Enhancing PSRO Variants

As shown in last section, all previous diversity-enhancing PSRO variants [2, 41, 32, 33] try to enlarge the gamescape of Π i , which is the convex hull of the rows in the empirical payoff matrix:

where m j is the j -th row vector in M Π i , Π -i . However, the gamescape of a population depends on the choice of opponent policies, and two policies with the same payoff vector are not necessarily the same. Moreover, enlarging the gamescape without careful tuning would encourage the current player to deliberately lose to the opponent to get 'diverse' payoffs. We suspect this might be the reason why the optimization of the gamescape is activated later in the training procedure in [32]. More importantly, it is not theoretically clear from previous diversity-enhancing PSRO variants why enlarging the gamescape would help in approximating a full game NE in PSRO.

To rigorously answer the question whether a diversity metric is helpful in approximating a NE in PSRO, we need a performance measure to monitor the progress of PSRO across iterations in terms of finding a full game NE. In other words, we need to quantify the strength of a population of policies. Previously, the exploitability of a meta NE of the joint population is usually employed to monitor the progress of PSRO. Yet, as demonstrated in [37], this exploitability may increase after an iteration. Intuitively, a better alternative is the exploitability of the least exploitable mixed strategy supported by a population. We define this exploitability as the population exploitability :

Definition 3.1. For a joint population Π = Π i × Π -i , let ( σ ∗ i , σ ∗ -i ) be a meta NE on M Π i , Π -i . The relative population performance of Π i against Π -i [2] is:

The population exploitability of the joint population Π is defined as:

where Ω i is the full set of all possible mixed strategies of player i .

We notice that PE is equal to the sum of negative population effectivity defined in [32]. Yet, we prefer PE as it is more of a natural extension to exploitability in Equation 1. Some properties of PE and its relation to PH are presented in the following.

Proposition 3.2. Considering a joint population Π = Π i × Π -i , we have:

PE (Π) ≥ 0 , ∀ Π .

For another joint population Π ′ , if H (Π i ) ×H (Π 2 ) ⊆ H (Π ′ i ) ×H (Π ′ -i ) , then PE (Π) ≥ PE (Π ′ ) .

If Π i = { π i } and Π -i = { π -i } , then PE (Π) = E ( π ) , where π = ( π i , π -i ) .

∃ π = ( π i , π -i ) ∈ H (Π i ) ×H (Π -i ) s.t. E ( π ) = PE (Π) = min π ′ ∈H (Π i ) ×H (Π -i ) E ( π ′ ) .

Let ( σ ∗ i , σ ∗ -i ) denote an arbitrary NE of the full game. PE (Π) = 0 if and only if ( σ ∗ i , σ ∗ -i ) ∈ H (Π i ) ×H (Π -i ) .

The proof is in Appendix B.2. Once we use PE to monitor the progress of PSRO, we have:

Proposition 3.3. The PE of the joint population Π t at each iteration t in PSRO is monotonically decreasing and will converge to 0 in finite iterations for finite games. Once PE (Π T ) = 0 , a meta NE on Π T is a full game NE.

The proof is in Appendix B.3. From Proposition 3.2 and 3.3, we are convinced that PE is indeed an appropriate performance measure for populations of polices. Using PE, we can now formally present why enlarging the gamescape of the population in PSRO is somewhat deceptive:

Theorem 3.4. The enlargement of the gamescape is neither sufficient nor necessary for the decrease of PE. Considering two populations ( Π 1 i and Π 2 i ) for player i and one population Π -i for player -i , and denoting Π j = Π j i × Π -i , j = 1 , 2 we have

The proof of Theorem 3.4 is in Appendix B.5, where we provide concrete examples.

In other words, enlarging the gamescape in either short term or long term does not necessarily lead to a better approximation to a full game NE.

4 Policy Space Diversity PSRO

In this section, we develop a new diversity-enhancing PSRO variant, i.e., PSD-PSRO. In contrast to methods that enlarge the gamescape , PSD-PSRO encourages the enlargement of PH of a population, which helps reduce a population's PE (according to Proposition 3.2). In addition, we develop a well-justified state-action sampling method to optimize our diversity metric in practice. Finally, we present the convergence property of PSD-PSRO and discuss its relation to the original PSRO.

4.1 A New Diversity Regularization Term for PSRO

Our purpose of promoting diversity in PSRO is to facilitate the convergence to a full game NE. We follow the conventional scheme in previous diversity-enhancing PSRO variants [41, 32, 33], which introduces a diversity regularization term to the BR solving in PSRO. Nonetheless, our diversity regularization encourages the enlargement of PH of the current population, which is in contrast to the enlargement of gamescape in previous methods [2, 41, 32, 33]. We thus name our diversity metric policy space diversity . Intuitively, the larger the PH of a population is, the more likely it will include a full game NE. More formally, a larger PH means a lower PE (Proposition 3.2), which means our diversity metric avoids the common weakness (Section 3) of existing ones.

Recall that the PH of a population is simply the complete set of polices that are convex combinations of individual polices in the population. To quantify the contribution of a new policy to the enlargement of the PH of the current population, a straightforward idea is to maximize a distance between the new policy and the PH. Such a distance should be 0 for any policy that belongs to the PH and greater than 0 otherwise. Without loss of generality, the distance between a policy and a PH could be defined as the minimal distance between the policy and any policy in the PH. We can now write down the diversity regularized BR solving objective in PSD-PSRO, where at each iteration t for player i we add a new policy π t +1 i by solving:

where σ t -i is the opponent's meta NE policy at the t -th iteration, λ is a Lagrange multiplier, and dist( · , · ) is a distance function (will be specified in the next subsection) between two polices.

4.2 Diversity Optimization in Practice

To be able to optimize our diversity metric (the right part in Equation 10) in practice, we need to encode a policy into some representation space and specify a distance function there. Such a representation should be a one-to-one mapping between a policy and its representation. Also, to ensure that enlarging the convex hull in the representation space results in the enlargement of the PH, we require the representation to satisfy the linearity property. Formally, we have the following definition:

Definition 4.1. A fine policy representation for our purpose is a function ρ : Π i → R N i , which satisfies the following two properties:

(bijection) For any representation ρ ( π i ) , there exists a unique behavior policy π i whose representation is ρ ( π i ) , and vice-versa.

(linearity) For any two policies ( π j i and π k i ) and α ( 0 ≤ α ≤ 1 ), the following holds:

where απ j i +(1 -α ) π k i means playing π j i with probability α and π k i with probability (1 -α ) .

Existing diversity metrics explicitly or implicitly define a policy representation [33]. For instance, the gamescape -based methods [2, 41, 32, 33] represent a policy using its payoff vector against the opponent's population. Yet, this representation is not a fine policy representation as it is not a bijection (different policies can have the same payoff vector). The (joint) occupancy measure, which is a fine policy representation , is usually used to encode a policy in the RL community [48, 20, 32].The f -divergence is then employed to measure the distance between two policies [32, 24, 16]. However, computing the f -divergence based on the occupancy measure is usually intractable and often in practice roughly approximated using the prediction of neural networks [32, 24, 16].

Instead, we use another fine policy representation , i.e., the sequence-form representation [21, 11, 31, 27, 1], which was originally developed for representing a policy in multi-agent games. We then define the distance between two policies using the Bregman divergence, which can be further simplified to a tractable form and optimized using only state-action samples in practice.

The sequence-form representation of a policy remembers the realization probability of reaching a state-action pair. We follow the definition in [11, 31], where the sequence form representation x i ∈ X i ⊆ [0 , 1] |S i ×A i | of π i is a vector:

where τ ( s, a ) is a trajectory from the beginning to ( s, a ) . By the perfect-recall assumption, there is a unique τ that leads to ( s, a ) . The policy π i can be written as π i ( a | s ) = x i ( s, a ) / ∥ x i ( s ) ∥ 1 , where x i ( s ) is ( x i ( s, a 1 ) , . . . , x i ( s, a n )) with a 1 , . . . , a n ∈ A ( s ) . Unlike the payoff vector representation or the occupancy measure representation, x i is independent of the opponent's policy as well as the environmental dynamics. Therefore, it should be more appropriate in representing a policy for the diversity optimization. Without loss of generality and following [21, 11, 31], we define the distance dist( π i , π ′ i ) between two policies as the Bregman divergence on the sequence form representation B d ( x i ∥ x ′ i ) , which can be further written in terms of state-action pairs (the derivation is presented in Appendix B.1) in the following:

where B d s ( π i ( s ) ∥ π ′ i ( s )) is the Bregman divergence between π i ( s ) and π ′ i ( s ) . In our experiment, we let B d s ( π i ( s ) ∥ π ′ i ( s )) = ∑ a π i ( a | s ) log π i ( a | s ) /π ′ i ( a | s ) = KL( π i ( s ) ∥ π ′ i ( s )) , i.e., the KL divergence. In previous work [21, 11, 31], the coefficient β s usually declines monotonically as the length of the sequence increases. In our case, we make β s depend on an opponent's policy b -i : β s = ∏ ˜ s -i , ˜ a ∈ τ ( s,a ) b -i ( ˜ a | ˜ s -i ) 2 . This weighting method allows us to estimate the distance using the sampled average KL divergence and avoids importance sampling:

where s i ∼ π i , b -i means sampling player i 's information states from the trajectories that are collected by playing π i against b -i . As a result, we can rewrite the diversity regularized BR solving objective in PSD-PSRO as follows:

Now we provide a practical way to optimize Equation 14. By regarding the opponent and chance as the environment, we can use the policy gradient method [47, 43, 44] in RL to train π t +1 i . Denote the probability of generating τ by π i ( τ ) , and the payoff of player i for the trajectory τ by R ( τ ) . Let π min i = arg min π k i ∈H (Π t i ) E s i ∼ π i ,b -i [KL( π i ( s i ) ∥ π k i ( s i ))] be the policy in H (Π t i ) that minimizes the distance and let R KL ( τ ) = E s i ∼ τ [KL( π i ( s i ) ∥ π min i ( s i ))] .

The gradient of the first term u ( π i , σ t -i ) in Equation 14 with respect to π i can be written as:

The gradient of the diversity term in Equation 14 with respect to π i can be written as:

where we use the property that ∇ E [ KL ( π i ( s i ) | π min i ( s i )] ∈ ∂ min π k i E [ KL ( π i ( s i ) | π k i ( s i )] , whose correctness can be shown from the following proposition:

Proposition 4.2. For any local Lipschitz continuous function f ( x, y ) , assume ∀ x, min y f ( x, y ) exists, then ∂ x f ( x, y ) | y ∈ arg min f ( x,y ) ∈ ∂ x min y f ( x, y ) , where ∂f is the generalized gradient [7].

Proof. According to Theorem 2.1 (property (4) in [7], the result is immediate, as ∂ x min y f ( x, y ) is the convex hull of { ∂ x f ( x, y ) | y ∈ arg max g ( x, y ) } .

Combining the above two equations, we have,

According to Equation 17, we can see that optimizing π t +1 i requires maximizing two types of rewards: R ( τ ) and R KL ( τ ) . This can be done by averaging the gradients using samples that are generated by playing π i against σ t -i and b -i separately. The last term in Equation 17 is easily estimated by sampling states in the sampled trajectories via playing π i against b -i . For training efficiency, we simply set b -i = σ t -i in our experiments, although other settings of b -i are possible, e.g., the uniform random policy. We also want to emphasize that we optimize π t +1 i for each iteration t , and σ t -i is fixed during an iteration and thus can be viewed as a part of the environment. Finally, to estimate the distance between π i and π min i , we should ideally compute π min i exactly first by solving a convex optimization problem. In practice, we find sampling the policies in H (Π t i ) and using the minimal sampled distance already gives us satisfactory performance. The pseudo-code of PSD-PSRO is provided in Appendix C.

4.3 The Convergence Property of PSD-PSRO

We first show how the PH evolves in the original PSRO:

Proposition 4.3. Before the PE of the joint population reaches 0 , adding a BR to the meta NE policy from last iteration in PSRO will strictly enlarge the PH of the current population.

The proof is in Appendix B.3. From Proposition 4.3, we can see that adding a BR serves one way (an implicit way) of enlarging the PH and hence reducing the PE. In contrast, the optimization of our diversity term in Equation 14 aims to explicitly enlarge the PH. In other words, adding a diversity regularized BR in PSD-PSRO serves as a mixed way of enlarging the PH. More formally, we have the following theorem:

Theorem 4.4. (1) In PSD-PSRO, before the PE of the joint population reaches 0 , adding an optimal solution in Equation 14 will strictly enlarge the PH and hence reduce the PE. (2) Once the PH can not be enlarged (i.e., PSD-PSRO converges) by adding an optimal solution in Equation 14, the PE reaches 0, and PSD-PSRO finds a full game NE.

The proof is in Appendix B.6. In terms of convergence property, one significant benefit of PSD-PSRO over other state-of-the-art diversity-enhancing PSRO variants [2, 41, 32, 33] is the convergence of PH guarantees a full game NE in PSD-PSRO. Yet, it is not clear in those papers whether a full game NE is found once the PH of their populations converge. Notably, PSRO rN [2] is not guaranteed to find a NE once converged [36]. In practice, we expect a significant performance improvement of PSD-PSRO over PSRO in approximating a NE. As for different games, there might exist different optimal trade-offs between 'exploitation' (adding a BR) and 'exploration' (optimizing our diversity metric) in enlarging the PH and reducing the PE. In other words, PSD-PSRO generalizes PSRO (a PSD-PSRO instance when λ = 0) in ways of enlarging the PH and reducing the PE.

5 Related Work

Diversity has been widely studied in evolutionary computation [13], with a central focus that mimics the natural evolution process. One of the ideas is novelty search [28], which searches for policies that lead to novel outcomes. By hybridizing novelty search with a fitness objective, quality-diversity [42] aims for diverse behaviors of good qualities. Despite these methods achieving good empirical results [8, 23], the diversity metric is often hand-crafted for different tasks.

Promoting diversity is also intensively studied in RL. By adding a distance regularization between the current policy and a previous policy, a diversity-driven approach has been proposed for good exploration [22]. Unsupervised learning of diverse policies [10] has been studied to serve as an effective pretraining mechanism for downstream RL tasks. A diversity metric based on DPP [39] has been proposed to improve exploration in population-based training. Diverse behaviors were learned in order to improve generalization ability for test environments that are different from training [25]. A diversity-regularized collaborative exploration strategy has been proposed in [40]. Reward randomization [49] has been employed to discover diverse strategies in multi-agent games. Trajectory diversity has been studied for better zero-shot coordination in a multi-agent environment [34]. Quality-similar diversity has been investigated in [51].

Diversity also plays a role in game-theoretic methods. Smooth FP [17] adds a policy entropy term when finding a BR. PSRO rN [2] encourages effective diversity, which considers amplifying the strength over the weakness in a policy. DPP-PSRO [41] introduces a diversity metric based on DPP and provides a geometric interpretation of behavioral diversity. BD&RD-PSRO [32] combines the occupancy measure mismatch and the diversity on payoff vectors as a unified diversity metric. UDM [33] summarizes existing diversity metrics, by providing a unified diversity framework. In both opponent modeling [15] and opponent-limited subgame solving [30], diversity has been shown to have a large impact on the performance.

AlphaStar888

AlphaStar888

PSRO

PSRO

100

P-PSRO

100

P-PSRO

PSRO-rN

PSRO-rN

DPP-PSRO

Population Exploitability

DPP-PSRO

BD&RD-PSRO

BD&RD-PSRO

Exploitability

PSD-PSRO(OurS)

PSD-PSRO(OurS)

10-1

10-1

102

0

25

50

75

100

125

150

175

200

0

25

50

75

100

125

150

175

200

Iterations

Iterations

+

PSRO

P-PSRO

PSRO-rN

DPP-PSRO

BD&RD-PSRO

PSD-PSRO

X

Exp: 40.88 ± 2.18

Exp: 2.79 ± 0.97

Exp: 44.79 ± 3.57

Exp: 2.49 ± 0.39

Exp: 3.97 ± 1.39

Exp: 1.68± 0.52

6 Experiments

The main purpose of the experiments is to compare PSD-PSRO with existing state-of-the-art PSRO variants in terms of approximating a full game NE. The baseline methods include PSRO [26], PipelinePSRO (P-PSRO) [36], PSRO rN [2], DPP-PSRO [41], and BD&RD-PSRO [32]. The benchmarks consist of single-state games (AlphaStar888 and non-transitive mixture game) and complex extensive games (Leduc poker and Goofspiel). For AlphaStar888 and Leduc poker, we report the exploitability of the meta NE and the PE of the joint population through the training process. For the non-transitive mixture game, we illustrate the 'diversity' of the population and report the final exploitability . For Goofspiel where the exact exploitability is intractable, we report the win rate between the final agents. In addition, we illustrate how the PH evolves for each method using the Disc game [2] in Appendix D.3, where PSD-PSRO is more effective at enlarging the PH and approximating a NE. An ablation study on λ of PSD-PSRO in Appendix D.2 reveals that, for different benchmarks, an optimal trade-off between 'exploitation' and 'exploration' in enlarging the PH to approximate a NE usually happens when λ is greater than zero. Error bars or stds in the results are obtained via 5 independent runs. In Appendix D.3, we also investigate the time cost of calculating our policy space diversity. More details for the environments and hyper-parameters are given in Appendix E.

AlphaStar888 is an empirical game generated from the process of solving Starcraft II [50], which contains a payoff table for 888 RL policies. be viewed as a zero-sum symmetric two-player game where there is only one state s 0 . In s 0 , there are 888 legal actions. Any mixed strategy is a discrete probability distribution over the 888 actions. Hence, the distance function in Equation 14 for AlphaStar888 reduces to the KL divergence between two 888-dim discrete probability distributions.

In Figure 1, we can see that PSD-PSRO is more effective at reducing both the exploitability and PE than other methods.

Non-Transitive Mixture Game consists of 7 equally-distanced Gaussian humps on the 2D plane. Each strategy can be represented by a point on the 2D plane, which is equivalent to the weights (the likelihood of that point in each Gaussian distribution) that each player puts on the humps. The optimal strategy is to stay close to the center of the Gaussian humps and explore all the distributions. In Figure 2, we show the exploration trajectories for different methods during training, where PSRO and PSRO rN get trapped and fail in this non-transitive game. In contrast, PSD-PSRO tends to find diverse strategies and explore all Gaussians. Also, the exploitability of the meta NE of the final population is significantly lower in PSD-PSRO than others.

LeducPoker

LeducPoker

2 × 100

PSRO

PSRO

P-PSRO

P-PSRO

PSRO-rN

PSRO-rN

PSD-PSRO (Ours)

Population Exploitability

PSD-PSRO (Ours)

Exploitability

100

0.8

0.6

6 × 10-1

0.4

0.3

4 × 101

3× 10-1

0.2

0

50

100

Episodes (1e4)

150

200

250

0

50

100

Episodes (1e4)

150

200

250

Leduc Poker is a simplified poker [46], where the deck consists of two suits with three cards in each suit. Each player bets one chip as an ante, and a single private card is dealt to each player. Since DPP-PSRO cannot scale to the RL setting [32] and the code of BD&RD-PSRO for complex games is not available, we compare PSD-PSRO only to P-PSRO, PSRO rN and PSRO. As demonstrated in Figure 3, PSD-PSRO is more effective at reducing both the exploitability and PE.

Goofspiel is commonly used as a large-scale multi-stage simultaneous move game. Goofspiel features strong non-transitivity, as every pure strategy can be exploited by a simple counter-strategy. We compare PSD-PSRO with PSRO, P-PSRO, and PSRO rN on Goofspiel with 5 point cards and 8 point cards settings. In the game with 5 point cards setting, due to the relatively small game size, we can calculate the exact exploitability . We report the results in Appendix D.1, in which we see that PSD-PSRO reduces the exploitability more effectively than other methods. In the game with 8 point cards setting, the game size is too large to show exact exploitability for each iteration. In this setting, we provide a comparison among final solutions produced by different methods. We report the win rate between each two methods in Table 1, where we can see that PSD-PSRO consistently beats existing methods with a 62% win rate on average.

7 Conclusions and Limitations

In this paper, we point out a major and common weakness of existing diversity metrics in previous diversity-enhancing PSRO variants, which is their goal of enlarging the gamescape does not necessarily result in a better approximation to a full game NE. Based on the insight that a larger PH means a lower PE (a better approximation to a NE), we develop a new diversity metric ( policy space diversity ) that explicitly encourages the enlargement of a population's PH. We then develop a practical method to optimize our diversity metric using only state-action samples, which is derived based on the Bregman divergence on the sequence form of policies. We incorporate our diversity metric into the BR solving in PSRO to obtain PSD-PSRO. We present the convergence property of PSD-PSRO, and extensive experiments demonstrate that PSD-PSRO is significantly more effective in approximating a NE than state-of-the-art PSRO variants.

The diversity regularization term λ in PSD-PSRO plays an important role in balancing the 'exploitation' and 'exploration' in terms of enlarging the PH to approximate a NE. In this paper, other than showing that different problems have different optimal settings of λ , we have not discussed about the guidance of choosing λ optimally. Although there has been related work on how to adapt a diversity regularization term using online bandits [39], future work is still needed on how our policy space diversity could benefit the approximation of a full game NE most. Another interesting direction of future work is to extend PSD-PSRO to larger scale games, such as poker [5], Mahjong [14], and dark chess [52].

References

Yu Bai, Chi Jin, Song Mei, and Tiancheng Yu. Near-optimal learning of extensive-form games with imperfect information. In International Conference on Machine Learning , pages 1337-1382. PMLR, 2022.

David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech Czarnecki, Julien Perolat, Max Jaderberg, and Thore Graepel. Open-ended learning in symmetric zero-sum games. In International Conference on Machine Learning , pages 434-443. PMLR, 2019.

David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. The mechanics of n-player differentiable games. In International Conference on Machine Learning , pages 354-363. PMLR, 2018.

George W Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation , 13(1):374, 1951.

Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science , 359(6374):418-424, 2018.

Ozan Candogan, Ishai Menache, Asuman Ozdaglar, and Pablo A Parrilo. Flows and decompositions of games: Harmonic and potential games. Mathematics of Operations Research , 36(3):474-503, 2011.

Frank H Clarke. Generalized gradients and applications. Transactions of the American Mathematical Society , 205:247-262, 1975.

Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. Robots that can adapt like animals. Nature , 521(7553):503-507, 2015.

Wojciech M Czarnecki, Gauthier Gidel, Brendan Tracey, Karl Tuyls, Shayegan Omidshafiei, David Balduzzi, and Max Jaderberg. Real world games look like spinning tops. In Advances in Neural Information Processing Systems , volume 33, pages 17443-17454, 2020.

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations , 2018.

Gabriele Farina, Christian Kroer, and Tuomas Sandholm. Optimistic regret minimization for extensive-form games via dilated distance-generating functions. Advances in neural information processing systems , 32, 2019.

Xidong Feng, Oliver Slumbers, Ziyu Wan, Bo Liu, Stephen McAleer, Ying Wen, Jun Wang, and Yaodong Yang. Neural auto-curricula in two-player zero-sum games. In Advances in Neural Information Processing Systems , volume 34, pages 3504-3517, 2021.

David B Fogel. Evolutionary computation: toward a new philosophy of machine intelligence . John Wiley &Sons, 2006.

Haobo Fu, Weiming Liu, Shuang Wu, Yijia Wang, Tao Yang, Kai Li, Junliang Xing, Bin Li, Bo Ma, Qiang Fu, et al. Actor-critic policy optimization in a large-scale imperfect-information game. In International Conference on Learning Representations , 2021.

Haobo Fu, Ye Tian, Hongxiang Yu, Weiming Liu, Shuang Wu, Jiechao Xiong, Ying Wen, Kai Li, Junliang Xing, Qiang Fu, et al. Greedy when sure and conservative when uncertain about the opponents. In International Conference on Machine Learning , pages 6829-6848. PMLR, 2022.

Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248 , 2017.

Drew Fudenberg and David K Levine. Consistency and cautious fictitious play. Journal of Economic Dynamics and Control , 19(5-7):1065-1089, 1995.

Johannes Heinrich, Marc Lanctot, and David Silver. Fictitious self-play in extensive-form games. In International conference on machine learning , pages 805-813. PMLR, 2015.

Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfect-information games. arXiv preprint arXiv:1603.01121 , 2016.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems , 29, 2016.

Samid Hoda, Andrew Gilpin, Javier Pena, and Tuomas Sandholm. Smoothing techniques for computing nash equilibria of sequential games. Mathematics of Operations Research , 35(2):494-512, 2010.

Zhang-Wei Hong, Tzu-Yun Shann, Shih-Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Yi Lee. Diversity-driven exploration strategy for deep reinforcement learning. In Advances in neural information processing systems , volume 31, 2018.

Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science , 364(6443):859-865, 2019.

Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. arXiv e-prints , pages arXiv-1911, 2019.

Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. In Advances in Neural Information Processing Systems , volume 33, pages 8198-8210, 2020.

Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In Advances in neural information processing systems , volume 30, 2017.

Chung-Wei Lee, Christian Kroer, and Haipeng Luo. Last-iterate convergence in extensive-form games. Advances in Neural Information Processing Systems , 34:14293-14305, 2021.

Joel Lehman and Kenneth O Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation , 19(2):189-223, 2011.

Siqi Liu, Luke Marris, Daniel Hennes, Josh Merel, Nicolas Heess, and Thore Graepel. Neupl: Neural population learning. arXiv preprint arXiv:2202.07415 , 2022.

Weiming Liu, Haobo Fu, Qiang Fu, and Yang Wei. Opponent-limited online search for imperfect information games. In International Conference on Machine Learning . PMLR, 2023.

Weiming Liu, Huacong Jiang, Bin Li, and Houqiang Li. Equivalence analysis between counterfactual regret minimization and online mirror descent. In Proceedings of the 39th International Conference on Machine Learning , volume 162, pages 13717-13745, 2022.

Xiangyu Liu, Hangtian Jia, Ying Wen, Yujing Hu, Yingfeng Chen, Changjie Fan, Zhipeng Hu, and Yaodong Yang. Towards unifying behavioral and response diversity for open-ended learning in zero-sum games. In Advances in Neural Information Processing Systems , volume 34, pages 941-952, 2021.

Zongkai Liu, Chao Yu, Yaodong Yang, Zifan Wu, Yuan Li, et al. A unified diversity measure for multiagent reinforcement learning. In Advances in Neural Information Processing Systems , 2022.

Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In International Conference on Machine Learning , pages 7204-7213. PMLR, 2021.

Odile Macchi. The fermion process-a model of stochastic point process with repulsive points. In Transactions of the Seventh Prague Conference on Information Theory, Statistical Decision Functions, Random Processes and of the 1974 European Meeting of Statisticians , pages 391-398. Springer, 1977.

Stephen McAleer, John B Lanier, Roy Fox, and Pierre Baldi. Pipeline psro: A scalable approach for finding approximate nash equilibria in large games. In Advances in neural information processing systems , volume 33, pages 20238-20248, 2020.

Stephen McAleer, Kevin Wang, Marc Lanctot, John Lanier, Pierre Baldi, and Roy Fox. Anytime optimal psro for two-player zero-sum games. arXiv preprint arXiv:2201.07700 , 2022.

H Brendan McMahan, Geoffrey J Gordon, and Avrim Blum. Planning in the presence of cost functions controlled by an adversary. In Proceedings of the 20th International Conference on Machine Learning (ICML-03) , pages 536-543, 2003.

Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. In Advances in Neural Information Processing Systems , volume 33, pages 18050-18062, 2020.

Zhenghao Peng, Hao Sun, and Bolei Zhou. Non-local policy optimization via diversity-regularized collaborative exploration. arXiv preprint arXiv:2006.07781 , 2020.

Nicolas Perez-Nieves, Yaodong Yang, Oliver Slumbers, David H Mguni, Ying Wen, and Jun Wang. Modelling behavioural diversity for learning in open-ended games. In International Conference on Machine Learning , pages 8514-8524. PMLR, 2021.

Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI , page 40, 2016.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Max Smith, Thomas Anthony, and Michael Wellman. Iterative empirical game solving via single policy best response. In International Conference on Learning Representations , 2020.

Finnegan Southey, Michael P Bowling, Bryce Larson, Carmelo Piccione, Neil Burch, Darse Billings, and Chris Rayner. Bayes' bluff: Opponent modelling in poker. arXiv preprint arXiv:1207.1411 , 2012.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999.

Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear programming. In Proceedings of the 25th international conference on Machine learning , pages 1032-1039, 2008.

Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Shaolei Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. In International Conference on Learning Representations , 2020.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

Shuang Wu, Jian Yao, Haobo Fu, Ye Tian, Chao Qian, Yaodong Yang, Qiang Fu, and Yang Wei. Qualitysimilar diversity via population based reinforcement learning. In The Eleventh International Conference on Learning Representations , 2022.

Brian Zhang and Tuomas Sandholm. Subgame solving without common knowledge. Advances in Neural Information Processing Systems , 34:23993-24004, 2021.

Ming Zhou, Jingxiao Chen, Ying Wen, Weinan Zhang, Yaodong Yang, Yong Yu, and Jun Wang. Efficient policy space response oracles. arXiv preprint arXiv:2202.00633 , 2022.

Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. In Advances in neural information processing systems , volume 20, 2007.

A Notation

We provide the notation in Table 2.

B Theoretical Analysis

B.1 Derivation of Bregman Divergence

We define the Bregman divergence on the sequence-form representation of the policies. Given a strictly convex function d defined on R |S i ×A i | , a Bregman divergence is defined as

for any x i , x ′ i ∈ X i . Intuitively, Bregman divergence is the gap between d ( x i ) and its first-order approximation at x ′ i . It is non-negative (zero is achieved if and only if x i = x ′ i ) and strictly convex in its first argument. So, it can be used to measure the difference between x i and x ′ i . Examples of Bregman divergence include the Squared Euclidean distance, which is generated by the l 2 function, and the Kullback-Leibler (KL) divergence, which is generated by the negative entropy function. For sequence-form policies, we use the dilated Distance-Generating Function (dilated DGF) [21, 11, 31]:

where β s > 0 is a hyper-parameter and d s is a strictly convex function defined on R |A ( s ) | , for example, the negative entropy function. It is proven that for the dilated DGF (Lemma D.2 in [31] ), we have the Bregman divergence

The result allows us to compute the Bergman divergence using the behavior policy instead of explicitly using the sequence-form representation, which is prohibited in large-scale games. Formally, we define the distance of two policies π i , π ′ i as

B.2 Proof of Proposition 3.2

Proof. Recall that the PE of joint policy set Π in two-player zero-sum games can be written into

since P -i (Π ′ -i , Π 1 ) = -P i (Π i , Π ′ 2 ) .

We can check this property directly, since

We first note that H (Π 1 ) × H (Π 2 ) ⊆ H (Π ′ i ) × H (Π ′ -i ) means H (Π 1 ) ⊆ H (Π ′ i ) and H (Π 2 ) ⊆ H (Π ′ -i ) .

H (Π 2 ) ⊆ H (Π ′ 2 ) suggests that ∀ Φ i ∈ Ω i , we have P i (Φ i , Π 2 ) ≥ P i (Φ i , Π ′ 2 ) , which is a property of relative population performance [2]. Take the maximum for both sides, we have,

Similarly, using the condition H (Π 1 ) ⊆ H (Π ′ i ) , we have,

Combining Equation 22 and 23, we obtain PE (Π) ≥ PE (Π ′ )

By the monotonicity of PE, we have

Since Π -i = { π -i } contains only one policy, by the definition of relative population performance, we have,

Substituting Equation 25 into Equation 24, we obtain

By Equation 24 and the definition of the relative population performance, there exists NE ( σ i , π -i ) in game M Ω i , Π -i and NE ( π i , σ -i ) in game M Π i , Ω -i s.t.

By the definition of NE, we have,

We now prove ( π i , π -i ) is also the least exploitable policy in the joint policy set. ∀ π ′ = ( π ′ i , π ′ -i ) ∈ Π i × Π -i , we have,

where the inequality is from the monotonicity of PE.

Hence, ( π i , π -i ) is the least exploitable policy in the Policy Hull of the population.

Necessity If ( σ ∗ i , σ ∗ -i ) is the NE of the game, we can regard the { σ ∗ i } and { σ ∗ -i } as the populations which contain a single policy. Then by property 2 , we have,

Note that by property 1 , we have PE (Π) ≥ 0 , so we have PE (Π) = 0 .

Sufficiency Using the conclusion from 3, It is easy to check the sufficiency. If PE (Π) = 0 , there exists σ = ( σ i , σ -i ) s.t. E ( σ ) = 0 , which means σ is the NE of the game.

B.3 Proof of Proposition 3.3

Proof. Recall that at each iteration t , the PSRO calculates the NE ( σ t i , σ t -i ) of the meta-game M Π t i , Π t -i , then adds the new policy π t = ( BR ( σ t -i ) , BR ( σ t i )) to the population.

If PE (Π t ) > 0 , then by the second property in Proposition 3.2, we know the population does not contain the NE of the game. Therefore, since ( σ t i , σ t -i ) ∈ Π t , it is not the NE of the full game, which means at least one of the conditions holds: (1) BR ( σ t -i ) / ∈ H (Π t i ) ; (2) BR ( σ t i ) / ∈ H (Π t -i ) , which suggests the enlargement of the Policy Hull.

In finite games, since the PSRO adds pure BR at each iteration, this Policy Hull extension stops at some iteration, saying T , in which case we have BR ( σ T -i ) ∈ H (Π T i ) and BR ( σ T i ) ∈ H (Π T -i ) (by above analysis). Let ( π T +1 i , π T +1 -i ) := ( BR ( σ T -i ) , BR ( σ T i )) , we claim that ( π T +1 i , π T +1 -i ) is

the NE of the whole game. To see this, note that ( σ T i , σ T -i ) is the NE of the game M Π T i , Π T -i and π T +1 i ∈ H (Π T i ) , hence we have,

On the other hand, since π T +1 is the BR to σ T

i -i , we have,

Combining Equation 29 and 30, we obtain u ( π T +1 i , σ T -i ) = u ( σ T i , σ T -i ) , which suggests σ T i is also the BR of σ T -i . Similarly, we can prove that σ T -i is the BR of σ T i , and hence ( σ T i , σ T -i ) is the NE of the game.

Finally, by the fourth property in Proposition 3.2, we have PE (Π T ) = 0 .

B.4 Proof of Proposition 4.3

This is an intermediate result in the proof of 3.3 (Appendix B.3).

B.5 Proof of Theorem 3.4

Proof. We prove this theorem by providing the counterexamples. Let's first focus on the following two-player zero-sum game:

The game is inspired by Rock-Paper-Scissors. We add high-level strategies { R ′ , P ′ , S ′ } to player i in the original R-P-S game. The high-level strategy can beat the low-level strategy with the same type.

Let Π i = { S, R ′ , P ′ } and Π -i = { R,P, 1 2 R + 1 2 P } , where 1 2 R + 1 2 P means playing [ R,P ] with probability [ 1 2 , 1 2 ] . The payoff matrix M Π i , Π -i is

It is in player i 'turn to choose his new strategy. Considering two candidates π 1 i = S ′ and π 2 -i = 1 2 R + 1 2 R ′ (There are many choices for π 2 i to construct the counterexample, we just take one for example).

The payoff of π 1 i against Π -i is ( -1 , 1 , 0) , the same as the first row in M Π i , Π -i The payoff of π 2 i is ( 1 2 , -1 , -1 4 ) , which is out of the gamescape of Π i .

Denote Π 1 i = Π i ∪ { π 1 i } , Π 2 i = Π i ∪ { π 2 i } . By the analysis above, we have

Hence, to promote the diversity defined on the payoff matrix and aim to enlarge the gamescape [2, 41], we should choose π 2 i to add. However, in this case, we show adding π 1 i is more helpful to decrease PE.

Denote Π 1 = Π 1 i × Π -i and Π 2 = Π 2 i × Π -i , by Equation 24 and a few calculation, we have,

i.e., PE (Π 1 ) < PE (Π 2 ) . We can see that adding a new policy following the guidance of enlarging gamescape may not be the best choice to decrease PE.

Back to the Proposition 3.4, since we have found an example that,

hence,

Rewrite the example in Equation 35, we have,

Rename the superscripts (exchange the identity of 1 , 2 in the superscript), we obtain,

Combine Equation 36 and 38, we complete the proof.

Remark We offer another example in a symmetric game, proving that this proposition is also valid in this type of game.

Current policy set Π i = Π -i = { A,B } , the payoff is

Considering π 1 i = C, π 2 i = D , the payoff of π 1 i = C against Π -i is (0 . 5 , -0 . 5) , contained in GS (Π i | Π -i ) , while the payoff of π 2 i = D against Π -i is (1 , 1) , which is out of GS (Π i | Π -i ) .

Denote Π 1 i = Π i ∪ { π 1 i } , Π 2 i = Π i ∪ { π 2 i } , Π 1 = Π 1 i × Π -i and Π 2 = Π 2 i × Π -i , we have GS (Π 1 i | Π -i ) ⫋ GS (Π 2 i | Π -i ) . However,

Hence, similar to the above analysis, we can prove the Proposition 3.4 in symmetric games.

B.6 Proof of Theorem 4.4

Proof. Similar to Appendix B.3, we denote the meta-strategy at iteration t as ( σ t i , σ t -i ) . At iteration t of PSRO, if PE (Π t ) > 0 , then from Appendix B.3, we know that at least one of the conditions holds: (1) BR ( σ t -i ) / ∈ H (Π t i ) ; (2) BR ( σ t i ) / ∈ H (Π t -i ) . Without loss of generality, we assume

Then ∀ ˆ σ i ∈ H (Π t i ) and λ > 0 ,

where the first equation is because ˆ σ i ∈ H (Π t i ) and the last two inequalities is due to the definition of BR and Equation 42 respectively.

Algorithm 1 PSD-PSRO

From Equation 43, we know that any policy in H (Π t i ) will not be considered as the optimal solution in Equation 14, since they are dominated by at least one of the policy that out of H (Π t i ) , i.e., the BR. Hence, before PE reaches 0, solving Equation 14 will find a policy not in H (Π t i ) , which guarantees the enlargement of the Policy Hull. Once we can not enlarge the Policy Hull by adding the optimal solution in Equation 14, the PE of the Policy Hull must be 0 , otherwise the optimal solution will be outside of the Policy Hull.

C Algorithm for PSD-PSRO

We provide the algorithm for PSD-PSRO in Algorithm 1, which generates the algorithm of PSRO to incorporate the estimation of the diversity in Equation 14.

D Additional Study

D.1 PE on Leduc

In Figure 4, we report the exploitability of the meta NE on Goofspiel with 5 point cards setting.

D.2 Ablation Study on Diversity Weight

We conduct the ablation study on the sensitivity of the diversity weight λ on the Non-Transitive Mixture Game and Leduc Poker. In Table 3 and Table 4, we show the exploitability of the final population against different diversity weights in Non-Transitive Mixture Game and Leduc Poker, respectively. We can see that the λ can significantly affect the training convergence and the final exploitability . A large or small λ can lead to high exploitability . This suggests that properly combing the BR (implicitly enlarges the Policy Hull) and PSD (explicitly enlarges the Policy Hull) is critical to improving the efficiency at reducing the exploitability .

Goofspiel (5 point cards)

1.0

PSRO

P-PSRO

0.9

PSRO-rN

PSD-PSRO(Ours)

Exploitability

0.8

0.7

0.6

0.5

0

50

100

150

200

250

Episodes (1e4)

D.3 The Visualization of PH

we demonstrate the expansion of PH on the Disc game [2]. In Disc game, each pure strategy is represented by a point in the circle. Due to the linearity in the payoff function, a mixed strategy is equivalent to a pure strategy and thus can be equivalently visualized. Figure 5 shows the expansion of PH on the Disc game [2]. It demonstrates that PSD-PSRO is more effective than other PSRO variants at enlarging the PH.

D.4 Time Consumption in PSD-PSRO

We run an experiment on Leduc poker to estimate the time portion of different parts in our algorithm PSD-PSRO. We can see from Table 5 that the majority of time in PSD-PSRO is in BR solving and computing the payoff matrix, which is shared by PSRO and all its variants. The time spent on calculating our diversity metric is not significant, which accounts for 7 . 9% computational time.

E Benchmark and Implementation Details

E.1 AlphaStar888

AlphaStar888 is an empirical game generated from the process of solving Starcraft II [50], which contains a payoff table for 888 RL policies. The occupancy measure is reduced to the action distribution as the game can be regarded as a single-state Markov Game. We choose diversity weight λ to be 0 . 85 in this game.

E.2 Non-Transitive Mixture Game

This game consists 7 equally-distanced Gaussian humps on the 2D plane. We represent each strategy by a point on the 2D plane, which is equivalent to the weights (the likelihood of that point in each Gaussian distribution) that each player puts on the humps. The payoff of the game which contains

iter10

iter15

iter20

PSD-PSRO

DPP-PSRO

Unit Circle

iter10

iter15

iter20

PSD-PSRO

PSRO

Unit Circle

iter10

iter15

iter20

PSD-PSRO

P-PSRO

Unit Circle

iter10

iter15

iter20

PSD-PSRO

BD&RD-PSRO

Unit Circle

iter10

iter15

iter20

PSD-PSRO

PSRO-rN

Unit Circle

both non-transitive and transitive components is π T i S π -i + 1 ∑ 7 k =1 ( π k i -π k -i )

We choose diversity weight λ to be 1 . 9 in this game.

E.3 Leduc Poker

Since the DPP-PSRO needs evolutionary updates and cannot scale to the RL setting, and the code for BD&RD-PSRO on complex games, where f-divergence on occupancy measure is approximated using prediction errors of neural networks, is not available, we compare PSD-PSRO with P-PSRO, PSRO rN and PSRO in this benchmark. We implement the PSRO framework with Nash solver, using PPO as the oracle agent. Hyper-parameters are shown in Table 6.

E.4 Goofspiel

In game theory and artificial intelligence, Goofspiel is commonly used as an example of a multi-stage simultaneous move game. In two-player Goofspiel, each player has one suit of cards, which is ranked A (low), 2 , ..., 10 , J , Q , K (high). Another suit of cards serves as the prize (competition cards). Play proceeds in a series of rounds. The players make sealed bids for the top (face up) prize by selecting a card from their hand (keeping their choice secret from their opponent). Once these cards are selected, they are simultaneously revealed, and the player making the highest bid takes the competition card. If tied, the competition card is discarded. The cards used for bidding are discarded, and play continues

with a new upturned prize card. After 13 rounds, there are no remaining cards and the game ends. Typically, players earn points equal to the sum of the ranks of cards won (i.e. A is worth one point, 2 is worth two points, etc., J 11, Q 12, and K 13 points). Goofspiel demonstrates high non-transitivity. Any pure strategy in this game has a simple counter-strategy where the opponent bids one rank higher, or as low as possible against the King bid. As an example, consider the strategy of matching the upturned card value mentioned in the previous section. The final score will be 78 - 13 with the K being the only lost prize. we focus on the simple versions of this game: goofspiel with 5 point cards and 8 point cards settings.

Similar to the setting in Leduc poker, we compare PSD-PSRO with P-PSRO, PSRO rN , and PSRO in this benchmark. We implement the PSRO framework with Nash solver, using DQN as the Oracle agent. We use Linear Programming to solve NE from the payoff table. As the game size of Goofspiel is larger than Leduc poker, we improve the capacity of the model by setting the hidden dimension in MLP to be 512 and the episodes for each BR training to be 3 × 10 4 . Other settings are similar to Leduc poker in Table 6.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the strategy exploration problem faced in learning-based game-solving algorithms. Their posed solution is to expand the empirical game with policies that are both approximate best responses to the current solution and maximally expands the policy hull. This method is motivated by an analysis on the exploitability of the population captured by the empirical game. They show a variety of performance benefits across a suite of games.\n\nSTRENGTHS:\nOverall, this paper is well-rounded. The framing is clear and the related work very nicely characterizes a variety of the preceding work. While the contributed ideas of population performance and exploitability are lightly different than previous work, I think the interpretation brought forward by the small difference is rather nice. Moreover, this work treats the soft and difficult topic of diversity formally, which is a nice change of pace compared to preceding work focused on heuristics. They provide both an exact formulation of their metric, and a practical version that can be used in gradient optimization.\n\nWEAKNESSES:\nThe major weakness of this work is the confusion of empirical games and what they refer to as the \"meta game\". The crucial difference being that empirical games have only estimates of the true payoffs. PSRO uses an empirical game, but meta games may be useful for best-case convergence guarantees. Unfortunately this confuses and invalidates a variety of claims throughout the paper that mix up their usage. This is easy to correct, and I am very supportive of the paper post this correction.\n\nAnother general point of weakness is the recurring claim of the problems with gamescapes. They are right, as they showed, that each iteration isn't gauranteed to lead to a monotonically improving game solution. However, this does not mean that the gamescape analysis wouldn't have long-term benefits. I suggest lightening the critique of gamescapes, and instead focus criticism only on their specific application of per-iteration policy exploration.  \n\nI would have liked to see some analysis related to diversity and/or settings of their algorithm in the results.\n\nQUESTIONS:\n- Convex Hull Enlargement and Occupancy Measure Mismatching appear to be functionally different from the other diversity measures because they focus on the impact of including another policy. I was wondering if you thought about how your methods or the others might be alternatively used between these two perspectives? \n- Population exploitability appears to be similar to the metrics defined in Lanctot et al.. \"Population-based Evaluation in Repeated Rock-Paper-Scissors as a Benchmark for Multiagent Reinforcement Learning.\" Might be worth discussing as concurrent/previous work.\n- L159-160: There's nothing ill-posed with the idea of enlarging a gamescape generally. I would recommend refining this claim to focus on your point: for diversity. \n- L165-166: Exploitability increasing in the short-term might be OK if it facilitates better long-term exploration. I would suggest revising this claim to focus on when future iterations occurring are uncertain---I believe AnytimePSRO makes an argument like this.\n- L197: If you're defining the concept of fine policy representation could you make it more clear please?\n- L218: There is something potentially confusing about your definition of the sequence-form. Shouldn't it either include all possible trajectories to (s,a) or be only the probability chain of actions since perfect recall makes all paths unique? A quick comment about this in the paper I think would be very nice.\n- L245: This method requires a design choice of which opponent distribution diversity will be measured against. Could the authors comment on different settings they tried? It might be also useful to discuss how this is both a limit and potential advantage of this method when compared to other diversity metrics. You are both potentially more limited in your measure with a bad choice of opponent, or able to better measure by including behaviors outside of the empirical game. \n- For the AlphaStar888 experiments could you explain how you computed the diversity metrics given only the empirical game? The metrics are all dependent on the policies that are not directly available. \n- L323: I would be interested in having the details of what makes BD&RD-PSRO to tune and anything you might have learned from the attempted reimplementation in the appendix. \n- For Goofspiel, if you can easily get them, could you please also provide approximate regret curves? This can be estimated using the combined empirical game across methods.\n- For Goofspiel, are you using the final solutions produced by the algorithms to compare or regret? It's also not clear how long this experiment has been run and the respective population sizes.\n\nLIMITATIONS:\nThere is no focused discussion on either the limitations or broader impact of this work within the paper. \n\nThe last paragraph, presumably their limitation sections, quickly discuss the role of trading-off diversity in game solving. I would have preferred to see discussions related:\n - Setting the opponent policy for policy comparisons\n - How the selection of the aforementioned opponent biases the measure of diversity\n - How changes in the opponent policy will require much larger computational costs\n - Results are partially on games that are not exactly solved, and the true relative performance of methods may differ\n - etc..", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

comments: SUMMARY:
This paper considers PSRO framework for approximating a NE in multi-agent non-stransitive games. The authors points out that the existing diversity matrics cannot guarantee that a more diverse policy space leads to a better approximation of the NE. To solve this issue, this paper proposes a new diversity metric and a corresponding algorithm, PSD-PSRO, which enjoys better convergence property.

STRENGTHS:
The paper is very well-organized and easy to follow. It provides a nice overview of related literature and states its contribution clearly.

WEAKNESSES:
My major concern is the correctness of the result. In specific, I think the derivation of the gradient caculation in equation (15) might be incorrect. The derivation from line 484 to 485 in the supplementary material might need some reconsideration. I particular I don't think the first equation actually holds because the $\pi_i^{\min}$ should be a function of $\pi_i$. It might even be questionable whether this function is differentiable.

I'm also curious about the computational complexity of the algorithm proposed in the paper. It seems that compared with other diversity metrics, the new metrics proposed is harder to compute because it needs to solve a minimization problem. What are the computational time for the numerical examples? And how does it compare with other methods? In Section D.3, the authors only provides the percentage, but it would be better to have a comparison among different methods.

QUESTIONS:
See the Weakness section

LIMITATIONS:
See the Weakness section
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2023-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors sought to justify the benefit of diversity by investigating properties of an expanding policy hull (PH), as supported by a set of policies in the context of population learning using Policy Space Response Oracle (PSRO). The authors discussed the limitations of expanding gamescape as a proxy for exploration in the policy space and reviewed several works on inducing diversity in prior works. Finally, the authors proposed their own diversity regularisation techniques and showed evidence that this approach leads to faster convergence to full game NE compared to baselines.\n\nSTRENGTHS:\n* originality / significance\nI find the discussion on what to focus on when it comes to defining policy space diversity interesting, thorough and well motivated. There have been several works on this topics in the literature (as surveyed by the authors) but prior works presented different definitions and motivations for the need for policy space diversity which this work attempts to address. Having a unified language to discuss diversity with the goal of game-solving is sorely needed by the community. \n\n* quality / clarity \nEmpirical results of PSD-PSRO appear thorough, though I have a few clarifying questions on the results. I would encourage the authors to revise the presentation to make the experimental protocol more clear and convincing. The theoretical justification for policy hull expansion appears sound.\n\nWEAKNESSES:\n* Clarity: the authors presented prior works from the angle of NE game-solving. However I believe this is not necessarily the stated goal in all prior works? For instance, PSRO_rN was proposed as a way to expand gamescape (rather than a method that solves for a full game NE at faster convergence rate). This should be clear from their discussion on the disc game where focusing on NE game solving would lead to a contracted gamescape, i.e. the final solution would not contain pure strategies that would be interesting to represent as they optimally exploit certain opponents (as compared to the NE mixed strategy). I would encourage the authors to revisit the stated motivation for these prior works and re-state their motivations more precisely.\n\n* Scope: the scope of this work could be stated more precisely? Several prior works on population diversity explicitly focused on symmetric two-player zero-sum games (i.e. gamescape was defined in this regime). The authors appear to state their results generally (e.g. L103, $\\Pi_i$, $\\Pi_{-i}$), do you aim to make statements about n-player general-sum games in this paper? If so, how should one interpret e.g. Equation 7? Should I interpret $\\sigma_{-i}$ as the joint distribution across all coplayers of player $i$? What would be the dimensions of the relative population performance in that case? \n\n* Preciseness: in several places the authors seem to imply that vanilla PSRO would not converge to a full game NE while there are convergence guarantees when using suitable best-response solvers. Perhaps revisit these statements to make it clear that the intention is to offer faster convergence rate? E.g. L266 and L174.\n\nQUESTIONS:\n* Eq 14: how is the min operator implemented as there are infinite such mixed policies? If each such sample corresponds to a mixture-policy mixed at the start of an episode, would you draw multiple samples from the same distribution in the population simplex?  \n\n* Policy Hull: is the definition of Policy Hull the same as Population Simplex described in https://arxiv.org/pdf/2205.15879.pdf? I.e. they both correspond to all convex mixtures supported by a set of policies? If so, would simplex-NeuPL be a useful representation scheme when searching for diverse BR policies? If not, could you clarify their differences? \n\n* L269: ... PSD-PSRO over PSRO in approximating a NE. Is this statement empirical or do you expect better convergence rate in theory which can be stated more explicitly?\n\n* Fig 1 (Left): nit: could you specify finer-grained y-axis for these figure? \n\n* L302: instances of Goofspiel as implemented in OpenSpiel could be solved exactly, could the authors clarify which game setting is used? Could similar analysis be done in more restricted setting such that we can show exact convergence in Goofspiel? \n\n* AlphaStar888: the dataset presents the payoff between 888 RL agents, but the diversity regularisation term seem to suggest needing access to state action in the underlying game. Could you clarify how diversities between these RL policies are computed?  \n\n* Fig 3: exploitability in Leduc Poker seems to be reducing still (at 0.2 currently), would it reduce further to lower level? Are there computational bottleneck that would prevent further training? How are best-responses solved in this experiment? By RL? Or by exact solvers from OpenSpiel? \n\nMy reservation in accepting this work is primarily due to my confusion as stated above, I would encourage the authors to further clarify their empirical results and experimental settings.\n\nLIMITATIONS:\nThis work is methodological and does not present immediate potential for negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how to promote policy diversity in two-player zero-sum games to better converge to a Nash Equilibrium (NE) under the algorithm framework of policy-space response oracles (PSRO). While previous works propose various diversity metrics to enlarge the gamescape, this paper shows that the enlargement of gamescape does not necessarily contribute to a better approximation of NE. Therefore, the authors propose a new diversity metric in policy space, namely KL divergence, to alleviate this issue. Theoretical results show that the proposed algorithm, PSD-PSRO, enjoys good convergence properties to NE. Experiment results in several common benchmarks for PSRO demonstrate the strong empirical performance of PSD-PSRO.\n\nSTRENGTHS:\n+ The main logic and presentation of this paper is very clear. Personally, I like the writing paradigm of first analyzing drawbacks of previous approaches, motivating the algorithm, and then showing strong theoretical and empirical results with the proposed algorithm.\n\n+ This paper points out an important issue of the main-stream diversity-motivated PSRO algorithms, which in my opinion is a solid contribution to the community. To resolve this issue, the proposed algorithm is quite simple (with KL divergence) but effective.\n\nWEAKNESSES:\n+ The defined diversity metric in Eq.14 is simple in form but very complex in derivation (from line215 to line234). The fancy derivation process essentially prevents readers from obtaining the algorithm insights. I will ask detailed questions in the next section.\n\nQUESTIONS:\n+ I'm not quite familiar with the notation in two-player zero-sum games. In Eq.11, xi seems to be the occupancy measure over the entire state-action space. Why is the state transition probability omitted here?\n\n+ In line221 the authors say that Eq.11 is independent of environment dynamics, but why? Eq.11 involves the trajectory \\tau.\n\n+ From my point of view, the derivation process from line215 to line234 tells the reader that KL divergence essentially measures the differernce of policies w.r.t. the sequence representation, but in my opinion this is not that straightforward. If this is generally true in RL (or two-player zero-sum games), I think there should be previous works illustrating this point as a theorem. In line230, why \\beta_s can depend on the opponent policy? Depending on the opponnet policy is intuitively true, but I didn't see any theoretical evidences that it should be this way. Why does setting \\beta_s in this way not changing the property that Eq.13 still measures policy differences in terms of sequence representation?\n\nLIMITATIONS:\nThe authors have addressed several limitations in Sec.7. I didn't see any additional major limiatations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}