SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wei Fu $^{1b}$ , Weihua Du $^{*1}$ , Jingwei Li $^{*1}$ , Sunli Chen $^{1}$ , Jingzhao Zhang $^{12}$ , Yi Wu $^{12\#}$
$^{1}$  IIIS, Tsinghua University,  $^{2}$  Shanghai Qi Zhi Institute
fuwth17@gmail.com, #jxwuyi@gmail.com
In complex reinforcement learning (RL) problems, policies with similar rewards may have substantially different behaviors. It remains a fundamental challenge to optimize rewards while also discovering as many diverse strategies as possible, which can be crucial in many practical applications. Our study examines two design choices for tackling this challenge, i.e., diversity measure and computation framework. First, we find that with existing diversity measures, visually indistinguishable policies can still yield high diversity scores. To accurately capture the behavioral difference, we propose to incorporate the state-space distance information into the diversity measure. In addition, we examine two common computation frameworks for this problem, i.e., population-based training (PBT) and iterative learning (ITR). We show that although PBT is the precise problem formulation, ITR can achieve comparable diversity scores with higher computation efficiency, leading to improved solution quality in practice. Based on our analysis, we further combine ITR with two tractable realizations of the state-distance-based diversity measures and develop a novel diversity-driven RL algorithm, State-based Intrinsic-reward Policy Optimization (SIPO), with provable convergence properties. We empirically examine SIPO across three domains from robot locomotion to multi-agent games. In all of our testing environments, SIPO consistently produces strategically diverse and human-interpretable policies that cannot be discovered by existing baselines.
A consensus in deep learning (DL) is that different local optima have similar mappings in the functional space, leading to similar losses to the global optimum [62, 56, 40]. Hence, via stochastic gradient descent (SGD), most DL works only focus on the final performance without considering which local optimum SGD discovers. However, in complex reinforcement learning (RL) problems, the policies associated with different local optima can exhibit significantly different behaviors [10, 35, 64]. Thus, it is a fundamental problem for an RL algorithm to not only optimize rewards but also discover as many diverse strategies as possible. A pool of diversified policies can be further leveraged towards a wide range of applications, including the discovery of emergent behaviors [34, 3, 61], generating diverse dialogues [30], designing robust robots [12, 25, 19], and enhancing human-AI collaboration [38, 9, 11].
Obtaining diverse RL strategies requires a quantitative method for measuring the difference (i.e., diversity) between two policies. However, how to define such a measure remains an open challenge. Previous studies have proposed various diversity measures, such as comparing the difference between the action distributions generated by policies [60, 38, 74], computing probabilistic distances between the state occupancy of different policies [43], or measuring the mutual information between states and
policy identities [16]. However, it remains unclear which measure could produce the best empirical performance. Besides, the potential pitfalls of these measures are rarely discussed.
In addition to diversity measures, there are two common computation frameworks for discovering diverse policies, including population-based training (PBT) and iterative learning (ITR). PBT directly solves a constrained optimization problem by learning a collection of policies simultaneously, subject to policy diversity constraints [52, 38, 9]. Although PBT is perhaps the most popular framework in the existing literature, it can be computationally challenging [48] since the number of constraints grows quadratically with the number of policies. The alternative framework is ITR, which iteratively learns a single policy that is sufficiently different from previous policies [43, 74]. ITR is a greedy relaxation of the PBT framework and it largely simplifies the optimization problem in each iteration. However, the performance of the ITR framework has not been theoretically analyzed yet, and it is often believed that ITR can be less efficient due to its sequential nature.
We provide a comprehensive study of the two aforementioned design choices. First, we examine the limitations of existing diversity measures in a few representative scenarios, where two policies outputting very different action distributions can still lead to similar state transitions. In these scenarios, state-occupancy-based measures are not sufficient to truly reflect the underlying behavior differences of the policies either. By contrast, we observe that diversity measures based on state distances can accurately capture the visual behavior differences of different policies. Therefore, we suggest that an effective diversity measure should explicitly incorporate state distance information for the best practical use. Furthermore, for the choice of computation framework, we conduct an in-depth analysis of PBT and ITR. We provide theoretical evidence that ITR, which has a simplified optimization process with fewer constraints, can discover solutions with the same reward as PBT while achieving at least half of the diversity score. This finding implies that although ITR is a greedy relaxation of PBT, their optimal solutions can indeed have comparable qualities. Furthermore, note that policy optimization is much simplified in ITR, which suggests that ITR can result in much better empirical performances and should be preferred in practice.
Following our insights, we combine ITR and a state-distance-based diversity measure to develop a generic and effective algorithm, State-based Intrinsic-reward Policy Optimization (SIPO), for discovering diverse RL strategies. In each iteration, we further solve this constrained optimization problem via the Lagrange method and two-timescale gradient descent ascent (GDA) [31]. We theoretically prove that our algorithm is guaranteed to converge to a neighbor of  $\epsilon$ -stationary point. Regarding the diversity measure, we provide two practical realizations, including a straightforward version based on the RBF kernel and a more general learning-based variant using Wasserstein distance.
We evaluate SIPO in three domains ranging from single-agent continuous control to multi-agent games: Humanoid locomotion [42], StarCraft Multi-Agent Challenge [57], and Google Research Football (GRF) [26]. Our findings demonstrate that SIPO surpasses baselines in terms of population diversity score across all three domains. Remarkably, our algorithm can successfully discover 6 distinct human-interpretable strategies in the GRF 3-vs-1 scenario and 4 strategies in two 11-player GRF scenarios, namely counter-attack, and corner, without any domain-specific priors, which are beyond the capabilities of existing algorithms.
Diversity in RL. It has been shown that policies trained under the same reward function can exhibit significantly different behaviors [10, 35]. Merely discovering a single high-performing solution may not suffice in various applications [12, 64, 25]. As such, the discovery of a diverse range of policies is a fundamental research problem, garnering attention over many years [44, 13, 28]. Early works are primarily based on multi-objective optimization [45, 55, 39, 47, 54], which assumes a set of reward functions is given in advance. In RL, this is also related to reward shaping [46, 2, 15, 61]. We consider learning diverse policies without any domain knowledge.
Population-based training (PBT) is the most popular framework for diverse solutions by jointly learning separate policies. Representative works include evolutionary computation [65, 37, 52], league training [64, 23], computing Hessian matrix [51] or constrained optimization with a population diversity measure [38, 73, 29, 36, 9]. An improvement is to learn a latent variable policy instead of separate ones. Prior works have incorporated different domain knowledge to design the latent code,
such as action clustering [66], agent identities [29, 21] or prosocial level [53, 3]. The latent variable can be also learned in an unsupervised fashion, such as in DIYAN [16] and its variants [25, 49]. Zahavy et al. [71] learns diverse policies with hard constraints on rewards to ensure the derived policies are (nearly) optimal, potentially hindering policies with disparate reward scales. On the other hand, our method prioritizes diversity and fully accepts sub-optimal strategies.
Iterative learning (ITR) simplifies PBT by only optimizing a single policy in each iteration and forcing it to behave differently w.r.t. previously learned ones [43, 60, 74]. While some ITR works require an expensive clustering process before each iteration [72] or domain-specific features [70], we consider domain-agnostic ITR in an end-to-end fashion. Besides, Pacchiano et al. [50] learns a kernel-based score function to iteratively guide policy optimization. The score function is conceptually similar to SIPO-WD but is applied to a parallel setting with more restricted expressiveness power.
Diversity Measure. Most previous works considered diversity measures on action distribution and state occupancy. For example, measures such as Jensen-Shannon divergence [38] and cross-entropy [74] are defined over policy distributions to encourage different policies to take different actions on the same state, implicitly promoting the generation of diverse trajectories. Other measures such as maximum mean discrepancy [43] maximize the probability distance between the state distributions induced by two policies. However, these approaches can fail to capture meaningful behavior differences between two policies in certain scenarios, as we will discuss in Section 4.1. There also exist specialized measures, such as cross-play rewards [9], which are designed for cooperative multi-agent games. It is worth noting that diversity measures are closely related to exploration criteria [4, 20, 6, 27] and skill discovery [8, 32, 24], where a diversity surrogate objective is often introduced to encourage broad state coverage. However, this paper aims to explicitly discover mutually distinct policies. Our diversity measure depends on a function that computes the distance between states visited by two policies.
Notation: We consider POMDP [59] defined by  $\mathcal{M} = \langle S, \mathcal{A}, \mathcal{O}, r, P, O, \nu, H \rangle$ .  $\mathcal{S}$  is the state space.  $\mathcal{A}$  and  $\mathcal{O}$  are the action and observation space.  $r : S \times \mathcal{A} \to \mathbb{R}$  is the reward function.  $O : S \to \mathcal{O}$  is the observation function.  $H$  is the horizon.  $P$  is the transition function. At timestep  $h$ , the agent receives an observation  $o_h = O(s_h)$  and outputs an action  $a_h \in \mathcal{A}$  w.r.t. its policy  $\pi : \mathcal{O} \to \triangle (\mathcal{A})$ . The RL objective  $J(\pi)$  is defined by  $J(\pi) = \mathbb{E}_{(s_h, a_h) \sim (P, \pi)}\left[\sum_{h=1}^{H} r(s_h, a_h)\right]$ . The above formulation can be naturally extended to cooperative multi-agent settings, where  $\pi$  and  $R$  correspond to the joint policy and the shared reward. We follow the standard POMDP notations for conciseness. Our method will be evaluated in both single-agent tasks and complex cooperative multi-agent scenarios. Among them, multi-agent environments encompass a notably more diverse range of potential winning strategies, and hence offer an apt platform for assessing the effectiveness of our method. Moreover, in this paper, we assume access to object-centric information and features rather than pure visual observations to simplify our discussion. We remark that although we restrict the scope of this paper to states, our method can be further extended to high-dimensional inputs (e.g. images, see App. B.4.1) or tabular MDPs via representation learning [68, 14].
Finally, to discover diverse strategies, we aim to learn a set of  $M$  policies  $\{\pi_i\}_{i=1}^M$  such that all of these policies are locally optimal under  $J(\cdot)$  but mutually distinct subject to some diversity measure  $D(\cdot, \cdot): \triangle \times \triangle \to \mathbb{R}$ , which captures the difference between two policies.
Existing Diversity Measures: We say a diversity measure  $D$  is defined over action distribution if it can be written as
$$
D \left(\pi_ {i}, \pi_ {j}\right) = \mathbb {E} _ {s \sim q (s)} \left[ \tilde {D} _ {\mathcal {A}} \left(\pi_ {i} (\cdot | s) \| \pi_ {j} (\cdot | s)\right) \right], \tag {1}
$$
where  $q$  is an occupancy measure over states,  $\tilde{D}_{\mathcal{A}}:\triangle \times \triangle \to \mathbb{R}$  measures the difference between action distributions.  $\tilde{D}_{\mathcal{A}}$  can be any probability distance as defined in prior works [60, 38, 74, 52].
Denote the state occupancy of  $\pi$  as  $q_{\pi}$ . We say a diversity measure is defined over state occupancy if it can be written as
$$
D \left(\pi_ {i}, \pi_ {j}\right) = \tilde {D} _ {\mathcal {S}} \left(q _ {\pi_ {i}} \| q _ {\pi_ {j}}\right), \tag {2}
$$
Figure 1: (left) A grid-world environment with 5 different optimal policies. Intuitively,  $D(\pi_1, \pi_2) < D(\pi_1, \pi_3)$  and  $D(\pi_3, \pi_4) < D(\pi_3, \pi_5)$ . However, action-based measures can give  $D_{\mathcal{A}}(\pi_1, \pi_2) \geq D_{\mathcal{A}}(\pi_1, \pi_3)$  and state-occupancy-based measures can give  $D(\pi_3, \pi_4) = D(\pi_3, \pi_5)$ .
Table 1: Diversity measures of the grid-world example. Computation details can be found in App. B.
human	action-based	state-distance-based
KL	JSD1	JSD0/EMD	L2norm	L2norm	EMD
D(π1, π2)	small	+∞	log 2	1/2	√7	2√2	5.7
D(π1, π3)	large	+∞	log 2	1/8	1	2√6	11.3
which can be realized as an integral probability metric [43]. We remark that  $q_{\pi}$  is usually intractable. In addition to diversity measures, we present two popular computation frameworks for this purpose.
Population-Based Training (PBT): PBT is a straightforward formulation by jointly learning  $M$  policies  $\{\pi_i\}_{i=1}^M$  subject to pairwise diversity constraints, i.e.,
$$
\max  _ {\{\pi_ {i} \}} \sum_ {i = 1} ^ {M} J \left(\pi_ {i}\right) \text {s . t .} D \left(\pi_ {j}, \pi_ {k}\right) \geq \delta , \forall j, k \in [ M ], j \neq k, \tag {3}
$$
where  $\delta$  is a threshold. In our paper, we consistently refer to the aforementioned computation framework as "PBT", rather than adjusting hyperparameters [22]. Despite a precise formulation, PBT poses severe optimization challenges due to mutual constraints.
Iterative Learning (ITR): ITR is a greedy approximation of PBT by iteratively learning novel policies. In the  $i$ -th  $(1 \leq i \leq M)$  iteration, ITR solves
$$
\pi_ {i} ^ {\star} = \arg \max  _ {\pi_ {i}} J \left(\pi_ {i}\right) \text {s . t .} D \left(\pi_ {i}, \pi_ {j} ^ {\star}\right) \geq \delta , \forall 1 \leq j <   i. \tag {4}
$$
$\pi_j^\star$  is recursively defined by the above equation. Compared with PBT, ITR trades off wall-clock time for less required computation resources (e.g., GPU memory) and performs open-ended training (i.e., the population size  $M$  does not need to be fixed at the beginning of training).
4 Analysis of Existing Diversity-Discovery Approaches
In this section, we conduct both quantitative and theoretical analyses of existing approaches to motivate our method. We first discuss diversity measures in Sec. 4.1 and then compare computation frameworks, namely PBT and ITR, in Sec. 4.2.
4.1 A Common Missing Piece in Diversity Measure: State Distance
The perception of diversity among humans primarily relies on the level of dissimilarity within the state space, which is measured by a distance function. However, the diversity measures outlined in Eq. (1) and Eq. (2) completely fail to account for such crucial information. In this section, we provide a detailed analysis to instantiate this observation with concrete examples and propose a novel diversity measure defined over state distances.
First, we present a synthetic example to demonstrate the limitations of current diversity measures. Our example consists of a grid-world environment with a single agent and grid size  $N_G$ . The agent starts at the top left of the grid-world and must navigate to the bottom right corner, as shown in Fig. 1. While  $N_G$  can be large in general, we illustrate with  $N_G = 5$  for simplicity. We draw five distinct policies, denoted as  $\pi_1$  through  $\pi_5$ , which differ in their approach to navigating the grid-world. Consider  $\pi_1$ ,  $\pi_2$ , and  $\pi_3$  first. Although humans may intuitively perceive that policies  $\pi_1$  and  $\pi_2$ , which move along the diagonal, are more similar to each other than to  $\pi_3$ , which moves along the boundary, diversity measures based on actions can fail to reflect this intuition, as shown in Table 1. Then, let's switch to policies  $\pi_3$ ,  $\pi_4$ , and  $\pi_5$ . We find that state-occupancy-based diversity measures are unable to differentiate between  $\pi_4$  and  $\pi_5$  in contrast to  $\pi_3$ . This is because the states visited by  $\pi_3$  are entirely disjoint from those visited by both  $\pi_4$  and  $\pi_5$ . However, humans would judge  $\pi_5$  to be more distinct from  $\pi_3$  than  $\pi_4$  because both  $\pi_3$  and  $\pi_4$  tend to visit the upper boundary.
Figure 2: A counter-example for action-based diversity measure: in a football game, we can achieve a high diversity score by simply asking a single idle player to output random actions, which does not affect the high-level gameplay strategy at all.
Figure 3: Illustration of the learning process of PBT and ITR in a 2-D navigation environment with 4 modes. PBT will not uniformly converge to different landmarks as computation can be either too costly or unstable. By contrast, ITR repeatedly excludes a particular landmark, such that policy in the next iteration can continuously explore until a novel landmark is discovered.
Next, we consider a more realistic and complicated multi-agent football scenario, i.e., the Google Research Football [26] environment, in Fig. 2, where an idle player in the backyard takes an arbitrary action without involving in the attack at all. Although the idle player stays still with no effect on the team strategy, action-based measures can produce high diversity scores. This example underscores a notable issue. If action-based measures are leveraged to optimize diversity, the resultant policies can produce visually similar behavior. While it can be possible to exclude idle actions by modifying task rewards, it requires domain-specific hacks and engineering efforts. The issue of idle actions exists even in such popular MARL benchmarks. Similar issues have also been observed in previous works [38].
To summarize, existing measures suffer from a significant limitation — they only compare the behavior trajectories implicitly through the lens of action or state distribution without explicitly measuring state distance. Specifically, action-based measures fail to capture the behavioral differences that may arise when similar states are reached via different actions. Similarly, state occupancy measures do not quantify the degree of dissimilarity between states. To address this limitation, we propose a new diversity measure that explicitly takes into account the distance function in state space:
$$
D \left(\pi_ {i}, \pi_ {j}\right) = \mathbb {E} _ {\left(s, s ^ {\prime}\right) \sim \gamma} \left[ g \left(d \left(s, s ^ {\prime}\right)\right) \right], \tag {5}
$$
$d$  is a distance metric over  $S \times S$ .  $g: \mathbb{R}^{+} \to \mathbb{R}$  is a monotonic cost function.  $\gamma \in \Gamma(q_{\pi_i}, q_{\pi_j})$  is a distribution over state pairs.  $\Gamma(q_{\pi_i}, q_{\pi_j})$  denotes the collection of all distributions on  $S \times S$  with marginals  $q_{\pi_i}$  and  $q_{\pi_j}$  on the first and second factors respectively. The cost function  $g$  is a notation providing a generalized and unified definition. It also contributes to training stability by scaling the raw distance. We highlight that Eq. (5) computes the cost on individual states before taking expectation, and therefore prevents information loss of taking the average over the entire trajectory (e.g. the DvD score [52]). We also note that states are consequences of performed actions. Hence, a state-distance-based measure also implicitly reflects the (meaningful) differences in actions between two policies. We compute two simple measures based on state distance, i.e., the  $L_2$  norm and the Earth Moving Distance (EMD), for the grid-world example and present results in Table 1. These measures are consistent with human intuition.
4.2 Computation Framework: Population-Based or Iterative Learning?
We first consider the simplest motivating example to intuitively illustrate the optimization challenges. Let's assume that  $\pi_i$  is a scalar,  $J(\pi_i)$  is linear in  $\pi_i$ , and  $D(\pi_i, \pi_j) = |\pi_i - \pi_j|$ . In our definition, where  $M$  denotes the number of diverse policies, PBT involves  $\Theta(M^2)$  constraints in a single linear programming problem while ITR involves  $\mathcal{O}(M)$  constraints in each of  $M$  iterations. Given that the complexity of linear programming is a high-degree polynomial (higher than 2) of the number of constraints, solving PBT is harder (and probably slower) than solving ITR in a total of  $M$  iterations,
despite PBT being parallelized. This challenge can be more severe in RL due to complex solution space and large training variance.
Although ITR can be optimized efficiently, it remains unclear whether ITR, as a greedy approximation of PBT, can obtain solutions of comparable rewards. Fig. 4 shows the worst case in the 1-D setting when the ITR solutions (green) can indeed have lower rewards than the PBT solution (red) subject to the same diversity constraint. However, we will show in the next theorem that ITR is guaranteed to have no worse rewards than PBT by trading off half of the diversity.
Figure 4: 1-D worst case of ITR. With threshold  $\delta$ , ITR finds solutions with inferior rewards. However, ITR can find optimal solutions if the threshold is halved.
Theorem 4.1. Assume  $D$  is a distance metric. Denote the optimal value of Eq.(3) as  $T_{1}$ . Let  $T_{2} = \sum_{i=1}^{M} J(\tilde{\pi}_{i})$  where
$$
\tilde {\pi} _ {i} = \arg \max  _ {\pi_ {i}} J \left(\pi_ {i}\right) s. t. D \left(\pi_ {i}, \tilde {\pi} _ {j}\right) \geq \delta / 2, \forall 1 \leq j <   i \tag {6}
$$
for  $i = 1,\dots ,M$  then  $T_{2}\geq T_{1}$
Please see App. E.1 for the proof. The above theorem provides a quality guarantee for ITR. The proof can be intuitively explained by the 1-D example in Fig. 4, where green points represent the worst case with threshold  $\delta$  and blue points represent the solutions with threshold  $\delta /2$ . Thm. 4.1 shows that, for any policy pool derived by PBT, we can always use ITR to obtain another policy pool, which has the same rewards and comparable diversity scores.
Empirical Results: We empirically compare PBT and ITR in a 2-D navigation environment with 1 agent and  $N_{L}$  landmarks in Fig. 3. The reward is 1 if the agent successfully navigates to a landmark and 0 otherwise. We train  $N_{L}$  policies using both PBT and ITR to discover strategies toward each of these landmarks. More details can be found in App. D. Table 2 shows the number of discovered landmarks by PBT and ITR. ITR performs consistently better than PBT even in this simple example. We intuitively
illustrate the learning process of PBT and ITR in Fig. 3. ITR, due to its computation efficiency, can afford to run longer iterations and tolerate larger exploration noises. Hence, it can converge easily to diverse solutions by imposing a large diversity constraint. PBT, however, only converges when the exploration is faint, otherwise it diverges or converges too slowly.
Table 2: The number of discovered landmarks across 6 seeds with standard deviation in the bracket.
setting	PBT	ITR
NL=4	2.0 (1.0)	3.5 (0.5)
NL=5	2.2 (0.9)	4.5 (0.5)
4.3 Practical Remark