SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jian Yao $^{1*}$ , Weiming Liu $^{1*}$ , Haobo Fu $^{1*}$ , Yaodong Yang $^{2}$ , Stephen McAleer $^{3}$ , Qiang Fu $^{1}$ , Wei Yang $^{1}$
Tencent AI Lab, Shenzhen, China
2Peking University, Beijing, China
3Carnegie Mellon University
Policy-Space Response Oracles (PSRO) is an influential algorithm framework for approximating a Nash Equilibrium (NE) in multi-agent non-transitive games. Many previous studies have been trying to promote policy diversity in PSRO. A major weakness in existing diversity metrics is that a more diverse (according to their diversity metrics) population does not necessarily mean (as we proved in the paper) a better approximation to a NE. To alleviate this problem, we propose a new diversity metric, the improvement of which guarantees a better approximation to a NE. Meanwhile, we develop a practical and well-justified method to optimize our diversity metric using only state-action samples. By incorporating our diversity regularization into the best response solving in PSRO, we obtain a new PSRO variant, Policy Space Diversity PSRO (PSD-PSRO). We present the convergence property of PSD-PSRO. Empirically, extensive experiments on various games demonstrate that PSD-PSRO is more effective in producing significantly less exploitable policies than state-of-the-art PSRO variants. The experiment code is available at https://github.com/nigelyaoj/policy-space-diversity-psro.
Most real-world games demonstrate strong non-transitivity [9], where the winning rule follows a cyclic pattern (e.g., the strategy cycle in Rock-Paper-Scissors) [6, 3]. A common objective in solving non-transitive games is to find a Nash Equilibrium (NE), which has the best worst-case performance in the whole policy space. Traditional algorithms, like simple self-play, fail to converge to a NE in games with strong non-transitivity [26]. Recently, many game-theoretic methods have been proposed to approximate a NE in such games. For example, Counterfactual Regret Minimization (CFR) [54] minimizes the so-called counterfactual regret. Neural fictitious self-play [18, 19] extends the classical game-theoretic approach, Fictitious Play (FP) [4], to larger games using Reinforcement Learning (RL) to approximate a Best Response (BR). Another well-known algorithm is Policy-Space Response Oracles (PSRO) [26], which generalizes the double oracle approach [38] by adopting a RL subroutine to approximate a BR.
Improving the performance of PSRO on approximating a NE is an active research topic, and many PSRO variants have been proposed so far, which generally fall into three categories. The first category [36, 45, 12, 29] aims to improve the training efficiency at each iteration. For instance, pipeline-PSRO [36] trains multiple BRs in parallel at each iteration. Neural population learning [29] enables fast transfer learning across policies via representing a population of policies within a single conditional model. The second category [37, 53] incorporates no-regret learning into PSRO, which solves an unrestricted-restricted game with a no-regret learning method to guarantee the decrease of exploitability of the meta-strategy across each iteration. The third category [2, 41, 32, 33]
*Equal contribution. Correspondence to: Haobo Fu (haobofu@tencent.com).
promotes policy diversity in the population, which is usually implemented by incorporating a diversity regularization term into the BR solving in the original PSRO.
Despite achieving promising improvements over the original PSRO, the theoretical reason why the diversity metrics in existing diversity-enhancing PSRO variants [2, 41, 32, 33] help PSRO in terms of approximating a NE is unclear. More specifically, those diversity metrics are 'justified' in the sense that adding the corresponding diversity-regularized BR strictly enlarges the gamescape. However, as we prove and demonstrate later in the paper, a population with a larger gamescape is neither a sufficient nor a necessary condition for a better approximation (we will define the precise meaning later) to a NE. One fundamental reason is that gamescape is a concept that varies significantly according to the choice of opponent policies. In contrast, exploitability (the distance to a NE) measures the worst-case performance that is invariant to the choice of opponent policies.
In this paper, we seek for a new and better justified diversity metric that improves the approximation of a NE in PSRO. To achieve this, we introduce a new concept, named Population Exploitability (PE), to quantify the strength of a population. The PE of a population is the optimal exploitability that can be achieved by selecting a policy from its Policy Hull (PH), which is simply a complete set of polices that are convex combinations of individual polices in the population. In addition, we show that a larger PH means a lower PE. Based on these insights, we make the following contributions:
- We point out a major and common weakness of existing diversity-enhancing PSRO variants: their goal of enlarging the gamescape of the population in PSRO is somewhat deceptive to the extent that it can lead to a weaker population in terms of PE. In other words, a more diverse (according to their diversity metrics) population  $\Rightarrow$  a larger gamescape  $\nRightarrow$  closer to a full game NE.
- We develop a new diversity metric that encourages the enlargement of a population's PH. In addition, we develop a practical and well-justified method to optimize our diversity metric using only state-action samples. We then incorporate our diversity metric (as a regularization term) into the BR solving in the original PSRO and obtain a new algorithm: Policy Space Diversity PSRO (PSD-PSRO). Our method PSD-PSRO establishes the causality: a more diverse (according to our diversity metric) population  $\Rightarrow$  a larger PH  $\Rightarrow$  a lower PE  $\Rightarrow$  closer to a full game NE.
- We prove that a full game NE is guaranteed once PSD-PSRO is converged. In contrast, it is not clear, in other state-of-the-art diversity-enhancing PSRO variants [2, 41, 32, 33], whether a full game NE is found once they are converged in terms of their optimization objectives. Notably,  $\mathrm{PSRO}_{rN}$  [2] is not guaranteed to find a NE once converged [36].
2 Notations and Preliminary
2.1 Extensive-form Games, NE, and Exploitability
Extensive-form games are used to model sequential interaction involving multiple agents, which can be defined by a tuple  $\langle \mathcal{N},\mathcal{S},P,\mathcal{A},u\rangle$ $\mathcal{N} = \{1,2\}$  denotes the set of players (we focus on the two-player zero-sum games).  $\mathcal{S}$  is a set of information states for decision-making. Each information state node  $s\in S$  includes a set of actions  $\mathcal{A}(s)$  that lead to subsequent information states. The player function  $P:\mathcal{S}\to \mathcal{N}\cup \{c\}$ , with  $c$  denoting chance, determines which player takes action in  $s$ . We use  $s_i$ ,  $\mathcal{S}_i = \{s\in \mathcal{S}|P(s) = i\}$ , and  $\mathcal{A}_i = \cup_{s\in \mathcal{S}_i}\mathcal{A}(s)$  to denote player  $i$ 's state, set of states, and set of actions respectively. We consider games with perfect recall, where each player remembers the sequence of states to the current state.
A player's behavioral strategy is denoted by  $\pi_i(s) \in \Delta(\mathcal{A}(s)), \forall s \in S_i$ , and  $\pi_i(a|s)$  is the probability of player  $i$  taking action  $a$  in  $s$ . A strategy profile  $\pi = (\pi_1, \pi_2)$  is a pair of strategies for each player, and we use  $\pi_{-i}$  to refer to the strategy in  $\pi$  except  $\pi_i$ .  $u_i(\pi) = u_i(\pi_i, \pi_{-i})$  denotes the payoff for player  $i$  when both players follow  $\pi$ . The BR of player  $i$  to the opponent's strategy  $\pi_{-i}$  is denoted by  $\mathcal{BR}(\pi_{-i}) = \arg \max_{\pi_i'} u_i(\pi_i', \pi_{-i})$ . The exploitability of strategy profile  $\pi$  is defined as:
$$
\mathcal {E} (\pi) = \frac {1}{2} \sum_ {i \in \mathcal {N}} \left[ \max  _ {\pi_ {i} ^ {\prime}} u _ {i} \left(\pi_ {i} ^ {\prime}, \pi_ {- i}\right) - u _ {i} \left(\pi_ {i}, \pi_ {- i}\right) \right]. \tag {1}
$$
When  $\mathcal{E}(\pi) = 0$ ,  $\pi$  is a NE of the game.
2.2 Meta-Games, PH, and PSRO
Meta-games are introduced to represent games at a higher level. Denoting a population of mixed strategies for player  $i$  by  $\Pi_i \coloneqq \{\pi_i^1, \pi_i^2, \ldots\}$ , the payoff matrix on the joint population  $\Pi = \Pi_i \times \Pi_{-i}$  is denoted by  $\mathbf{M}_{\Pi_i, \Pi_{-i}}$ , where  $\mathbf{M}_{\Pi_i, \Pi_{-i}}[j, k] \coloneqq u_i(\pi_i^j, \pi_{-i}^k)$ . The meta-game on  $\Pi$  and  $\mathbf{M}_{\Pi_i, \Pi_{-i}}$  is simply a normal-form game where selecting an action means choosing which  $\pi_i$  to play for player  $i$ . Accordingly, we use  $\sigma_i$  ( $\sigma_i$  is called a meta-strategy and could be, e.g., playing  $[\pi_i^1, \pi_i^2]$  with probability [0.5, 0.5]) to denote a mixed strategy over  $\Pi_i$ , i.e.,  $\sigma_i \in \Delta_{\Pi_i}$ . A meta-policy  $\sigma_i$  over  $\Pi_i$  can be viewed as a convex combination of policies in  $\Pi_i$ , and we define the PH of a population  $\mathcal{H}(\Pi_i)$  as the set of all convex combinations of the policies in  $\Pi_i$ . Meta-games are often open-ended in the sense that there exist an infinite number of mixed strategies and that new policies will be successively added to  $\Pi_i$  and  $\Pi_{-i}$  respectively. We give a summary of notations in Appendix A.
PSRO operates on meta-games and consists of two components: an oracle and a meta-policy solver. At each iteration  $t$ , PSRO maintains a population of policies, denoted by  $\Pi_i^t$ , for each player  $i$ . The joint meta-policy solver first computes a NE meta-policy  $\sigma^t$  on the restricted meta-game represented by  $\mathbf{M}_{\Pi_i^t,\Pi_{-i}^t}$ . Afterwards, for each player  $i$ , the oracle computes an approximate BR (i.e.,  $\pi_i^{t + 1}$ ) against the meta-policy  $\sigma_{-i}^t$ :  $\pi_i^{t + 1} \in \mathcal{BR}(\sigma_{-i}^t)$ . The new policy  $\pi_i^{t + 1}$  is then added to its population  $(\Pi_i^{t + 1} = \Pi_i^t \cup \{\pi_i^{t + 1}\})$ , and the next iteration starts. In the end, PSRO outputs a meta-policy NE on the final joint population as an approximation to a full game NE.
2.3 Previous Diversity Metrics for PSRO
**Effective diversity** [2] measures the variety of effective strategies (strategies with support under a meta-policy NE) and uses a rectifier to focus on how these effective strategies beat each other. Let  $(\sigma_{i}^{*}, \sigma_{-i}^{*})$  denote a meta-policy NE on  $\mathbf{M}_{\Pi_i, \Pi_{-i}}$ . The effective diversity of  $\Pi_i$  is:
$$
\operatorname {D i v} \left(\Pi_ {i}\right) = \sigma_ {i} ^ {* T} \left\lfloor \mathbf {M} _ {\Pi_ {i}, \Pi_ {- i}} \right\rfloor + \sigma_ {- i} ^ {*}, \tag {2}
$$
where  $\lfloor x\rfloor_{+}\coloneqq x$  if  $\mathrm{x}\geq 0$  else  $0$
Expected Cardinality [41], inspired by the determinantal point processes [35], measures the diversity of a population  $\Pi_{i}$  as the expected cardinality of the random set  $\mathbf{Y}$  sampled according to  $det(\mathcal{L}_{\Pi})$ :
$$
\operatorname {D i v} \left(\Pi_ {i}\right) = \mathbb {E} _ {\mathbf {Y} \sim \mathbb {P} _ {\mathcal {L} _ {\Pi}}} \left[ | \mathbf {Y} | \right] = \operatorname {T r} \left(\mathbf {I} - \left(\mathcal {L} _ {\Pi} + \mathbf {I}\right) ^ {- 1}\right), \tag {3}
$$
where  $|\mathbf{Y}|$  is the cardinality of  $\mathbf{Y}$ , and  $\mathcal{L}_{\Pi} = \mathbf{M}_{\Pi_i,\Pi_{-i}}\mathbf{M}_{\Pi_i,\Pi_{-i}}^T$ .
Convex Hull Enlargement [32] builds on the idea of enlarging the convex hull of all row vectors in the payoff matrix:
$$
\operatorname {D i v} \left(\Pi_ {i} \cup \left\{\pi_ {i} ^ {\prime} \right\}\right) = \min  _ {\mathbf {1} ^ {T} \beta = 1, \beta \geq 0} \left\| \mathbf {M} _ {\Pi_ {i}, \Pi_ {- i}} ^ {T} \beta - \mathbf {m} \right\|, \tag {4}
$$
where  $\pi_i'$  is the new strategy to add, and  $\mathbf{m}$  is the payoff vector of policy  $\pi_i'$  against each opponent policy in  $\Pi_{-i}$ :  $\mathbf{m}[j] = u_i(\pi_i', \pi_{-i}^j)$ .
Occupancy Measure Mismatching [32] considers the state-action distribution  $\rho_{\pi}(s,a)$  induced by a joint policy  $\pi$ . When considering adding a new policy  $\pi_i'$ , the corresponding diversity metric is:
$$
\operatorname {D i v} \left(\Pi_ {i} \cup \left\{\pi_ {i} ^ {\prime} \right\}\right) = D _ {f} \left(\rho_ {\left(\pi_ {i} ^ {\prime}, \sigma_ {- i} ^ {*}\right)} \right\rvert   \left\| \rho_ {\left(\sigma_ {i} ^ {*}, \sigma_ {- i} ^ {*}\right)}\right), \tag {5}
$$
where  $\pi_i^\prime$  is the new policy to add;  $(\sigma_{i}^{*},\sigma_{-i}^{*})$  is a meta-policy NE on  $\mathbf{M}_{\Pi_i,\Pi_{-i}}$ , and  $D_{f}$  is a general  $f$ -divergence between two distributions. It is worth noting that Equation 5 only considers the difference between two policies ( $\pi_i^\prime$  and  $\sigma_i^*$ ), instead of  $\pi_i^\prime$  and  $\Pi_i$ . In practice, this diversity metric is used together with the convex hull enlargement in [32].
Unified Diversity Measure [33] offers a unified view on existing diversity metrics and is defined as:
$$
\operatorname {D i v} \left(\Pi_ {i}\right) = \sum_ {m = 1} ^ {| \Pi_ {i} |} f \left(\lambda_ {m}\right), \tag {6}
$$
where  $f$  takes different forms for different existing diversity metrics;  $\lambda_{m}$  is the eigenvalues of  $[K(\phi_m,\phi_n)]_{|\Pi_i|\times |\Pi_i|};K(\cdot ,\cdot)$  is a predefined kernel function; and  $\phi_{m}$  is the strategy feature for the  $m$ -th policy in  $\Pi_{i}$ . It is worth mentioning that only payoff vectors in  $\mathbf{M}_{\Pi_i,\Pi_{-i}}$  were investigated for the strategy feature of the new diversity metric proposed in [33].
3 A Common Weakness of Existing Diversity-Enhancing PSRO Variants
As shown in last section, all previous diversity-enhancing PSRO variants [2, 41, 32, 33] try to enlarge the gamescape of  $\Pi_{i}$ , which is the convex hull of the rows in the empirical payoff matrix:
$$
\mathcal {G S} (\Pi_ {i} | \Pi_ {- i}) := \left\{\sum_ {j} \alpha_ {j} \mathbf {m} _ {j}: \boldsymbol {\alpha} \geq 0, \boldsymbol {\alpha} ^ {T} \mathbf {1} = 1 \right\},
$$
where  $\mathbf{m}_j$  is the  $j$ -th row vector in  $\mathbf{M}_{\Pi_i,\Pi_{-i}}$ . However, the gamescape of a population depends on the choice of opponent policies, and two policies with the same payoff vector are not necessarily the same. Moreover, enlarging the gamescape without careful tuning would encourage the current player to deliberately lose to the opponent to get 'diverse' payoffs. We suspect this might be the reason why the optimization of the gamescape is activated later in the training procedure in [32]. More importantly, it is not theoretically clear from previous diversity-enhancing PSRO variants why enlarging the gamescape would help in approximating a full game NE in PSRO.
To rigorously answer the question whether a diversity metric is helpful in approximating a NE in PSRO, we need a performance measure to monitor the progress of PSRO across iterations in terms of finding a full game NE. In other words, we need to quantify the strength of a population of policies. Previously, the exploitability of a meta NE of the joint population is usually employed to monitor the progress of PSRO. Yet, as demonstrated in [37], this exploitability may increase after an iteration. Intuitively, a better alternative is the exploitability of the least exploitable mixed strategy supported by a population. We define this exploitability as the population exploitability:
Definition 3.1. For a joint population  $\Pi = \Pi_{i} \times \Pi_{-i}$ , let  $(\sigma_{i}^{*}, \sigma_{-i}^{*})$  be a meta NE on  $\mathbf{M}_{\Pi_i, \Pi_{-i}}$ . The relative population performance of  $\Pi_{i}$  against  $\Pi_{-i}$  [2] is:
$$
\mathcal {P} _ {i} \left(\Pi_ {i}, \Pi_ {- i}\right) = \sigma_ {i} ^ {* T} \mathbf {M} _ {\Pi_ {i}, \Pi_ {- i}} \sigma_ {- i} ^ {*}. \tag {7}
$$
The population exploitability of the joint population  $\Pi$  is defined as:
$$
\mathcal {P E} (\Pi) = \frac {1}{2} \sum_ {i = 1, 2} \max  _ {\Pi_ {i} ^ {\prime} \subseteq \Omega_ {i}} \mathcal {P} _ {i} \left(\Pi_ {i} ^ {\prime}, \Pi_ {- i}\right), \tag {8}
$$
where  $\Omega_{i}$  is the full set of all possible mixed strategies of player  $i$ .
We notice that PE is equal to the sum of negative population effectiveness defined in [32]. Yet, we prefer PE as it is more of a natural extension to exploitability in Equation 1. Some properties of PE and its relation to PH are presented in the following.
Proposition 3.2. Considering a joint population  $\Pi = \Pi_{i} \times \Pi_{-i}$ , we have:
1.  $\mathcal{PE}(\Pi)\geq 0,\forall \Pi$
2. For another joint population  $\Pi'$ , if  $\mathcal{H}(\Pi_i) \times \mathcal{H}(\Pi_2) \subseteq \mathcal{H}(\Pi_i') \times \mathcal{H}(\Pi_{-i}')$ , then  $\mathcal{PE}(\Pi) \geq \mathcal{PE}(\Pi')$ .
3. If  $\Pi_i = \{\pi_i\}$  and  $\Pi_{-i} = \{\pi_{-i}\}$ , then  $\mathcal{PE}(\Pi) = \mathcal{E}(\pi)$ , where  $\pi = (\pi_i, \pi_{-i})$ .
4.  $\exists \pi = (\pi_i,\pi_{-i})\in \mathcal{H}(\Pi_i)\times \mathcal{H}(\Pi_{-i})$  s.t.  $\mathcal{E}(\pi) = \mathcal{P}\mathcal{E}(\Pi) = \min_{\pi^{\prime}\in \mathcal{H}(\Pi_{i})\times \mathcal{H}(\Pi_{-i})}\mathcal{E}(\pi^{\prime})$
5. Let  $(\sigma_i^*, \sigma_{-i}^*)$  denote an arbitrary NE of the full game.  $\mathcal{PE}(\Pi) = 0$  if and only if  $(\sigma_i^*, \sigma_{-i}^*) \in \mathcal{H}(\Pi_i) \times \mathcal{H}(\Pi_{-i})$ .
The proof is in Appendix B.2. Once we use PE to monitor the progress of PSRO, we have:
Proposition 3.3. The  $PE$  of the joint population  $\Pi^t$  at each iteration  $t$  in PSRO is monotonically decreasing and will converge to 0 in finite iterations for finite games. Once  $\mathcal{P}\mathcal{E}(\Pi^T) = 0$ , a meta NE on  $\Pi^T$  is a full game NE.
The proof is in Appendix B.3. From Proposition 3.2 and 3.3, we are convinced that PE is indeed an appropriate performance measure for populations of polices. Using PE, we can now formally present why enlarging the gamescape of the population in PSRO is somewhat deceptive:
Theorem 3.4. The enlargement of the gamescape is neither sufficient nor necessary for the decrease of PE. Considering two populations  $(\Pi_i^1$  and  $\Pi_i^2)$  for player  $i$  and one population  $\Pi_{-i}$  for player  $-i$ , and denoting  $\Pi^j = \Pi_i^j\times \Pi_{-i}$ ,  $j = 1,2$  we have
$$
\begin{array}{l} \mathcal {G S} \left(\Pi_ {i} ^ {1} | \Pi_ {- i}\right) \subsetneq \mathcal {G S} \left(\Pi_ {i} ^ {2} | \Pi_ {- i}\right) \nRightarrow \mathcal {P E} \left(\Pi^ {1}\right) \geq \mathcal {P E} \left(\Pi^ {2}\right) \\ \mathcal {P E} (\Pi^ {1}) \geq \mathcal {P E} (\Pi^ {2}) \nRightarrow \mathcal {G S} (\Pi_ {i} ^ {1} | \Pi_ {- i}) \subsetneq \mathcal {G S} (\Pi_ {i} ^ {2} | \Pi_ {- i}) \tag {9} \\ \end{array}
$$
The proof of Theorem 3.4 is in Appendix B.5, where we provide concrete examples.
In other words, enlarging the gamescape in either short term or long term does not necessarily lead to a better approximation to a full game NE.
4 Policy Space Diversity PSRO
In this section, we develop a new diversity-enhancing PSRO variant, i.e., PSD-PSRO. In contrast to methods that enlarge the gamescape, PSD-PSRO encourages the enlargement of PH of a population, which helps reduce a population's PE (according to Proposition 3.2). In addition, we develop a well-justified state-action sampling method to optimize our diversity metric in practice. Finally, we present the convergence property of PSD-PSRO and discuss its relation to the original PSRO.
4.1 A New Diversity Regularization Term for PSRO
Our purpose of promoting diversity in PSRO is to facilitate the convergence to a full game NE. We follow the conventional scheme in previous diversity-enhancing PSRO variants [41, 32, 33], which introduces a diversity regularization term to the BR solving in PSRO. Nonetheless, our diversity regularization encourages the enlargement of PH of the current population, which is in contrast to the enlargement of gamescape in previous methods [2, 41, 32, 33]. We thus name our diversity metric policy space diversity. Intuitively, the larger the PH of a population is, the more likely it will include a full game NE. More formally, a larger PH means a lower PE (Proposition 3.2), which means our diversity metric avoids the common weakness (Section 3) of existing ones.
Recall that the PH of a population is simply the complete set of polices that are convex combinations of individual polices in the population. To quantify the contribution of a new policy to the enlargement of the PH of the current population, a straightforward idea is to maximize a distance between the new policy and the PH. Such a distance should be 0 for any policy that belongs to the PH and greater than 0 otherwise. Without loss of generality, the distance between a policy and a PH could be defined as the minimal distance between the policy and any policy in the PH. We can now write down the diversity regularized BR solving objective in PSD-PSRO, where at each iteration  $t$  for player  $i$  we add a new policy  $\pi_i^{t+1}$  by solving:
$$
\pi_ {i} ^ {t + 1} = \arg \max  _ {\pi_ {i}} \left\{u \left(\pi_ {i}, \sigma_ {- i} ^ {t}\right) + \lambda \min  _ {\pi_ {i} ^ {k} \in \mathcal {H} \left(\Pi_ {i} ^ {t}\right)} \operatorname {d i s t} \left(\pi_ {i}, \pi_ {i} ^ {k}\right) \right\}, \tag {10}
$$
where  $\sigma_{-i}^{t}$  is the opponent's meta NE policy at the  $t$ -th iteration,  $\lambda$  is a Lagrange multiplier, and  $\mathrm{dist}(\cdot ,\cdot)$  is a distance function (will be specified in the next subsection) between two polices.
4.2 Diversity Optimization in Practice
To be able to optimize our diversity metric (the right part in Equation 10) in practice, we need to encode a policy into some representation space and specify a distance function there. Such a representation should be a one-to-one mapping between a policy and its representation. Also, to ensure that enlarging the convex hull in the representation space results in the enlargement of the PH, we require the representation to satisfy the linearity property. Formally, we have the following definition:
Definition 4.1. A fine policy representation for our purpose is a function  $\rho: \Pi_i \to R^{N_i}$ , which satisfies the following two properties:
- (bijection) For any representation  $\rho(\pi_i)$ , there exists a unique behavior policy  $\pi_i$  whose representation is  $\rho(\pi_i)$ , and vice-versa.
- (linearity) For any two policies  $(\pi_i^j$  and  $\pi_i^k)$  and  $\alpha$ $(0\leq \alpha \leq 1)$ , the following holds:
$$
\rho \left(\alpha \pi_ {i} ^ {j} + (1 - \alpha) \pi_ {i} ^ {k}\right) = \alpha \rho \left(\pi_ {i} ^ {j}\right) + (1 - \alpha) \rho \left(\pi_ {i} ^ {k}\right),
$$
where  $\alpha \pi_i^j + (1 - \alpha) \pi_i^k$  means playing  $\pi_i^j$  with probability  $\alpha$  and  $\pi_i^k$  with probability  $(1 - \alpha)$ .
Existing diversity metrics explicitly or implicitly define a policy representation [33]. For instance, the gamescape-based methods [2, 41, 32, 33] represent a policy using its payoff vector against the opponent's population. Yet, this representation is not a fine policy representation as it is not a bijection (different policies can have the same payoff vector). The (joint) occupancy measure, which is a fine policy representation, is usually used to encode a policy in the RL community [48, 20, 32]. The  $f$ -divergence is then employed to measure the distance between two policies [32, 24, 16]. However, computing the  $f$ -divergence based on the occupancy measure is usually intractable and often in practice roughly approximated using the prediction of neural networks [32, 24, 16].
Instead, we use another fine policy representation, i.e., the sequence-form representation [21, 11, 31, 27, 1], which was originally developed for representing a policy in multi-agent games. We then define the distance between two policies using the Bregman divergence, which can be further simplified to a tractable form and optimized using only state-action samples in practice.