SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Based on the above analyses, we suggest ITR and diversity measures based on state distances be preferred in RL applications. We also acknowledge that, by the no-free-lunch theorem, they cannot be universal solutions and that trade-offs may still exist (see discussions in Sec.7 and App.F). Nonetheless, in the following sections, we will show that the effective implementation of these choices can lead to superior performances in various challenging benchmarks. We hope that our approach will serve as a starting point and provide valuable insights into the development of increasingly powerful algorithms for potentially more challenging scenarios.
In this section, we develop a diversity-driven RL algorithm, State-based Intrinsic-reward Policy Optimization (SIPO), by combining ITR and state-distance-based measures. SIPO runs  $M$  iterations to discover  $M$  distinct policies. At the  $i$ -th iteration, we solve equation (4) by converting it into unconstrained optimization using the Lagrange method. The unconstrained optimization can be written as:
$$
\min  _ {\pi_ {i}} \max  _ {\lambda_ {j} \geq 0, 1 \leq j <   i} - J \left(\pi_ {i}\right) - \sum_ {j = 1} ^ {i - 1} \lambda_ {j} \left(D _ {\mathcal {S}} \left(\pi_ {i}, \pi_ {j} ^ {\star}\right) - \delta\right) \tag {7}
$$
$\lambda_{j}$ $(1\leq j < i)$  are Lagrange multipliers.  $\{\pi_j^\star \}_{j = 1}^{i - 1}$  are previously obtained policies. We adopt two-timescale Gradient Descent Ascent (GDA) [31] to solve the above minimax optimization, i.e., performing gradient descent over  $\pi_{i}$  and gradient ascent over  $\lambda_{j}$  with different learning rates. In our algorithm, we additionally enforce the dual variables  $\lambda_{j}$  to be bounded (i.e., in an interval  $[0,\Lambda ]$  for a large number  $\Lambda$ ), which plays an important role both in the theoretical analysis and in empirical convergence. However,  $D_{S}(\pi_{i},\pi_{j}^{\star})$  cannot be directly optimized w.r.t.  $\pi_{i}$  through gradient-based methods because it depends on the states traversed by  $\pi$ , rather than its output (e.g. actions). Therefore, we cast  $D_{S}(\pi_{i},\pi_{j}^{\star})$  as the cumulative sum of intrinsic rewards, specifically the intrinsic return. This allows us to leverage policy gradient techniques for optimization. The pseudocode of SIPO can be found in App. G.
An important property of SIPO is the convergence guarantee. We present an informal illustration in Thm. 5.1 and present the formal theorem with proof in App. E.2.
Theorem 5.1. (Informal) Under continuity assumptions, SIPO converges to an  $\epsilon$ -stationary point.
Remark: We assumed that the return  $J$  and the distance  $D_{S}$  are smooth in policies. In practice, this is true if (1) policy and state space are bounded and (2) reward function and system dynamics are continuous in the policy. (Continuous functions are bounded over compact spaces.) The key step is to analyze the role of the bounded dual variables  $\lambda$ , which achieves an  $\frac{1}{\Lambda}$ -approximation of constraint without hurting the optimality condition.
Instead of directly defining  $D_{\mathcal{S}}$ , we define intrinsic rewards as illustrated in Sec. 5, such that  $D_{\mathcal{S}}(\pi_i,\pi_j^{\star}) = \mathbb{E}_{s_h\sim \mu_{\pi_i}}\left[\sum_{h = 1}^{H}r_{\mathrm{int}}(s_h;\pi_i,\pi_j^{\star})\right]$ .
RBF Kernel: The most popular realization of Eq. (5) in machine learning is through kernel functions. Herein, we realize Eq. (5) as an RBF kernel on states. Formally, the intrinsic reward is defined by
$$
r _ {\text {i n t}} ^ {\mathrm {R B F}} \left(s _ {h}; \pi_ {i}, \pi_ {j} ^ {\star}\right) = \frac {1}{H} \mathbb {E} _ {s ^ {\prime} \sim \mu_ {\pi_ {j} ^ {\star}}} \left[ - \exp \left(- \frac {\| s _ {h} - s ^ {\prime} \| ^ {2}}{2 \sigma^ {2}}\right) \right] \tag {8}
$$
where  $\sigma$  is a hyperparameter controlling the variance.
Wasserstein Distance: For stronger discrimination power, we can also realize Eq. (5) as  $L_{2}$ -Wasserstein distance. According to the dual form [63], we define
$$
r _ {\text {i n t}} ^ {\mathrm {W D}} \left(s _ {h}; \pi_ {i}, \pi_ {j} ^ {\star}\right) = \frac {1}{H} \sup  _ {\| f \| _ {L} \leq 1} f \left(s _ {h}\right) - \mathbb {E} _ {s ^ {\prime} \sim \mu_ {\pi_ {j} ^ {\star}}} \left[ f \left(s ^ {\prime}\right) \right] \tag {9}
$$
where  $f: S \to \mathbb{R}$  is a 1-Lipschitz function. This realization holds a distinct advantage due to its interpretation within optimal transport theory [63, 1]. Unlike distances that rely solely on specific summary statistics such as means, Wasserstein distance can effectively quantify shifts in state distributions and remains robust in the presence of outliers [63]. We implement  $f$  as a neural network and clip parameters to  $[-0.01, 0.01]$  to ensure the Lipschitz constraint. Note that  $r_{\mathrm{int}}^{\mathrm{WD}}$  incorporates representation learning by utilizing a learnable scoring function  $f$  and is more flexible in practice. We also show in App. B.4 that  $r_{\mathrm{int}}^{\mathrm{WD}}$  is robust to different inputs, including states with random noises and RGB images.
We name SIPO with  $r_{\mathrm{int}}^{\mathrm{RBF}}$  and  $r_{\mathrm{int}}^{\mathrm{WD}}$  SIPO-RBF and SIPO-WD respectively.
Implementation: To incorporate temporal information, we stack the recent 4 global states to compute intrinsic rewards and normalize the intrinsic rewards to stabilize training. In multi-agent environments, we learn an agent-ID-conditioned policy [17] and share the parameter across all agents. Our implementation is based on MAPPO [69] with more details in App. D.
We evaluate SIPO across three domains that exhibit multi-modality of solutions. The first domain is the humanoid locomotion task in Isaac Gym [42], where diversity can be quantitatively assessed by well-defined behavior descriptors. We remark that the issues we addressed in Sec. 4.1 may not be present in this task where the action space is small and actions are highly correlated with states. Further, we examine the effectiveness of SIPO in two much more challenging multi-agent domains, StarCarft Multi-Agent Challenge (SMAC) [57] and Google Research Football (GRF) [26], where well-defined behavior descriptors are not available and existing diversity measures may produce misleading diversity scores. We provide introductions to these environments in App. C.
First, we show that SIPO can efficiently learn diverse strategies and outperform several baseline methods, including DIPG [43], SMERL [25], DvD [52], and RSPO [74]. Then, we qualitatively demonstrate the emergent behaviors learned by SIPO, which are both visually distinguishable and human-interpretable. Finally, we perform an ablation study over the building components of SIPO and show that both the diversity measure, ITR, and GDA are critical to the performance.
All algorithms run for the same number of environment frames on a desktop machine with an RTX3090 GPU. Numbers are average values over 5 seeds in Humanoid and SMAC and 3 seeds in GRF with standard deviation shown in brackets. More algorithm details can be found in App. D. Additional visualization results can be found on our project website (see App. A).
6.1 Comparison with Baseline Methods
Humanoid Locomotion. Following Zhou et al. [74], we train a population of size 4. We assess diversity by the pairwise distance of joint torques, a widely used behavior descriptor in recent Quality-Diversity works [67]. Torque states are not included as the input of diversity measures and we only use them for evaluation to ensure a fair comparison. Results are shown in Table 3. We can see that both variants of SIPO can outperform all baseline methods except that SIPO-RBF achieves comparable performance with RSPO, even if RSPO explicitly encourages the output of different actions/forces.
Table 3: Pairwise distance of joint torques (i.e., diversity scores) in the humanoid locomotion task.
SIPO-RBF	SIPO-WD	RSPO
0.53(0.17)	0.71(0.23)	0.53(0.05)
DIPG	DvD	SMERL
0.12(0.04)	0.40(0.22)	0.01(0.00)
SMAC Following Zhou et al. [74], we run SIPO and all baselines on an easy map,  $2m\_ vs\_ 1z$ , and a hard map,  $2c\_ vs\_ 64zg$ , both across 4 iterations. We merge all trajectories produced by the policy collection and incorporate a  $k$ -nearest-neighbor state entropy estimation [58] to assess diversity. Intuitively, a more diverse population should have a larger state entropy value. We set  $k = 12$  following Liu and Abbeel [32] and show results in Table 4. On these maps, two agents are both involved in the attack. Therefore, RSPO, which incorporates an action-based cross
Table 4: State entropy estimated by  $k$ -nearest-neighbor in SMAC. ( $k = 12$ )
2m_vs_1z	2c_vs_64zg
SIPO-RBF	0.038(0.002)	0.072(0.003)
SIPO-WD	0.036(0.001)	0.056(0.003)
RSPO	0.032(0.003)	0.070(0.001)
DIPG	0.032(0.002)	0.056(0.004)
SMERL	0.028(0.002)	0.042(0.002)
DvD	0.030(0.002)	0.057(0.003)
entropy measure, can perform well across all baselines. However, SIPO explicitly compares the distance between resulting trajectories and can even outperform RSPO, leading to the most diverse population.
GRF We consider three academy scenarios, specifically  $3\nu 1$ , counterattack (CA), and corner. The GRF environment is more challenging than SMAC due to the large action space, more agents, and the existence of duplicate actions. We determine a population size  $M = 4$  by balancing resources and wall-clock time across different baselines. Table 5 compares the number of distinct policies (in terms of ball-passing routes, see App. B.3) discovered in the population. Due to the strong adversarial power of our diversity measures and the application of GDA, SIPO is the most efficient and robust — even in the challenging 11-vs-11 corner and CA scenario, SIPO can effectively discover different winning strategies in just a few iterations across different seeds. By contrast, baselines suffer from learning instability in these challenging environments and tend to discover policies with slight distinctions. We also calculate the estimated state entropy as we did in SMAC. However, we find that this metric cannot distinguish fine-grained ball-passing behaviors in GRF (check our discussions in App. B).
Remark: In GRF experiments, when  $M$  is small, even repeated training with different random seeds (PG) is a strong baseline (see Table 5). Hence, the numbers are actually restricted in a small interval (with a lower bound equal to PG results and an upper bound equal to  $M = 4$ ), which makes the improvements by SIPO seemingly less significant. However, achieving clear improvements in these challenging applications remains particularly non-trivial. With a population size  $M = 10$ , SIPO clearly outperforms baselines by consistently discovering one or more additional strategies.
6.2 Qualitative Analysis
For SMAC, we present heatmaps of agent positions in Fig. 5. The heatmaps clearly show that SIPO can consistently learn novel winning strategies to conquer the enemy. Fig. 6 presents the learned
Figure 5: Heatmaps of agent positions in SMAC across 4 iterations with SIPO-RBF.
Figure 6: Learning curves and discovered strategies by SIPO-WD in the  $3\nu 1$  scenario over 7 iterations. Strategies of seed 1 are shown.
behavior by SIPO in the GRF  $3\nu 1$  scenario of seed 1. We can observe that agents have learned a wide spectrum of collaboration strategies across merely 7 iterations. The strategies discovered by SIPO are both diverse and human-interpretable. In the first iteration, all agents are involved in the attack such that they can distract the defender and obtain a high win rate. The 2nd and the 6th iterations demonstrate an efficient pass-and-shoot strategy, where agents quickly elude the defender and score a goal. In the 3rd and the 7th iterations, agents learn smart "one-two" strategies to bypass the defender, a prevalent tactic employed by human football players. We note that NONE of the baselines have ever discovered this strategy across all runs, while SIPO is consistently able to derive such strategies for all random seeds. Visualization results in CA and corner scenarios can be found in App. B.
We apply these changes to SIPO-WD:
- fix-L: Fixing the multiplier  $\lambda_{i}$  instead of applying GDA.
Table 5: Number of distinct strategies in GRF discovered by different methods in terms of the ball-passing route. Details of the evaluation protocol can be found in App. B.3.
Population Size M	ours	basielines	random
SIPO-RBF	SIPO-WD	DIPG	SMERL	DvD1	RSPO	PG
3v1	4	3.0 (0.8)	3.0 (0.0)	2.7 (0.5)	1.3 (0.5)	3.0 (0.8)	2.0 (0.0)	2.7 (0.5)
CA	4	3.3 (0.5)	3.0 (0.8)	2.3 (0.5)	1.3 (0.5)	-	2.0 (0.0)	1.7 (0.5)
corner	4	2.7 (0.5)	3.0 (0.8)	1.7 (0.5)	1.0 (0.0)	-	1.6 (0.5)	2.0 (0.8)
3v1	10	4.3 (0.5)	5.7 (0.5)	3.7 (0.5)	-	-	2.3 (0.5)	-
${}^{1}$  Training DvD in  ${CA}$  and corner or with  $M = {10}$  requires  $> {24}\mathrm{{GB}}$  GPU memory,which exceeds our memory limit.
-  $CE$ : The intrinsic reward is replaced with cross-entropy, i.e.,  $r_{\mathrm{int}}^{\mathrm{CE}}(s_h, a_h) = -\log \pi_j^\star(a_h \mid s_h)$ , where  $\pi_j^\star$  denotes a previously discovered policy. Additionally, GDA is still applied.
- filter: Optimizing the extrinsic rewards on trajectories that have intrinsic returns exceeding  $\delta$  and optimizing intrinsic rewards defined by Eq. (9) for other trajectories [74].
-  $PBT$ : Simultaneously training  $M$  policies with  $M(M - 1) / 2$  constraints (i.e., directly solving Eq. (3)) with intrinsic rewards defined by Eq. (9) and GDA.
We report the number of visually distinct policies discovered by these methods in Table 6. Comparison between SIPO and CE demonstrates that the action-based cross-entropy measure may suffer from duplicate actions in GRF and produce nearly identical behavior by overly exploiting duplicate actions, especially in the CA and corner scenarios with 11
Table 6: # distinct strategies of ablations in GRF.
ours	fix-L	CE	filter	PBT
3v1	3.0 (0.0)	1.0 (0.0)	2.7 (0.5)	1.3 (0.5)	2.7 (0.5)
CA	3.0 (0.8)	-1	2.3 (0.8)	1.0 (0.0)	-2
corner	3.0 (0.8)	-1	1.7 (0.5)	1.0 (0.0)	-2
1 Not converged.
2 Training requires  $>24\mathrm{GB}$  memory and exceeds our memory limit.
agents. Besides, the fixed Lagrange coefficient, the filtering-based method, and PBT are all detrimental to our algorithm. These methods also suffer from significant training instability. Overall, the state-distance-based diversity measure, ITR, and GDA are all critical to the performance of SIPO.
We tackle the problem of discovering diverse high-reward policies in RL. First, we demonstrate concrete failure cases of existing diversity measures and propose a novel measure that explicitly compares the distance in state space. Next, we present a thorough comparison between PBT and ITR and show that ITR is much easier to optimize and can derive solutions with comparable quality to PBT. Motivated by these insights, we combine ITR with a state-distance-based diversity measure to develop SIPO, which has provable convergence and can efficiently discover a wide spectrum of human-interpretable strategies in a wide range of environments.
Limitations: First, we assume direct access to an object-centric state representation. When such a representation is not available (e.g., image-based observations), representation learning becomes necessary and algorithm performance can be affected by the quality of the learned representations. Second, because ITR requires sequential training, the wall clock time of SIPO can be longer than the PBT alternatives when fixing the total number of training samples. The acceleration of ITR remains an open challenge.
Future Directions: Besides addressing the above limitations, we suggest three additional future directions based on our paper. First, a consensus on the best algorithmic formulation of distinct solutions in RL remains elusive. It is imperative to understand diversity in a more theoretical manner. Second, while this paper focuses on single-agent and cooperative multi-agent domains, extending SIPO to multi-agent competitive games holds great potential. Finally, although SIPO/ITR enables open-ended training, it is worth studying how to determine the optimal population size to better balance resources and the diversity of the resulting population.
This project is partially supported by 2030 Innovation Megaprojects of China (Programme on New Generation Artificial Intelligence) Grant No. 2021AAA0150000.
[1] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 214–223. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/arjovsky17a.html.
A Project Website
Check https://sites.google.com/view/diversity-sipo for GIF demonstrations.
B.1 More Qualitative Results
Figure 7: Visualization of learned behaviors in GRF CA across a single training trial.
Player Direction
Figure 8: Visualization of learned behaviors in GRF corner.
We show additional visualization results in Fig. 7, Fig. 8, and Fig. 9. Corresponding GIF visualizations can be found on our project website.
B.2 Task Performance Evaluation
The evaluation win rates of the demonstrated visualization results in SMAC and GRF are shown in Table 8. Evaluated episode returns in Humanoid are shown in Table 9.
We also present the diversity score and average rewards achieved by baselines in Table 10. These numerical values are averaged across the entire population for a clear comparison. The tabulated data highlights the varying trade-offs between task performance and diversity exhibited by different
Table 7:  $k$  -nearest neighbor state entropy estimation in GRF. Population size  $M = 4$  .
ours	basielines	PG (random seeding)
SIPO-RBF	SIPO-WD	DIPG	SMERL1	DvD2	RSPO1
3v1	0.009(0.000)	0.012(0.000)	0.010(0.001)	0.011(0.002)	0.010(0.000)	0.011(0.001)	0.009(0.001)
CA	0.037(0.000)	0.031(0.006)	0.036(0.002)	-	-	0.034(0.001)	0.039(0.001)
Corner	0.028(0.001)	0.031(0.001)	0.030(0.002)	-	-	-	0.028(0.002)
1 The learned policy in some iterations cannot even collect a single winning trajectory, so we are unable to compute their diversity score.
${}^{2}$  Training DvD in CA and corner requires  $> {24}\mathrm{{GB}}$  GPU memory,which exceeds our memory limit.
algorithms. It is noteworthy that SIPO, in particular, displays an adeptness at training a notably more diverse population while upholding a reasonably moderate level of task performance.
Table 8: Evaluation win rate (%) of the demonstrated visualization results in SMAC and GRF.
SMAC	GRF
2m1z	2c64zg	3v1	CA	corner
π1	100.0(0.0)	98.1(2.1)	92.3(6.2)	48.2(10.4)	78.2(16.2)
π2	99.6(0.9)	100.0(0.0)	82.1(8.4)	43.8(42.2)	57.0(37.7)
π3	100.0(0.0)	96.9(3.3)	90.7(1.1)	54.7(30.6)	55.7(20.8)
π4	99.6(0.6)	98.6(2.4)	63.6(45.0)	17.2(30.0)	30.7(29.0)
π5	-	-	85.4(9.1)	-	-
π6	-	-	93.2(1.9)	-	-
π7	-	-	64.6(32.5)	-	-
B.3 Evaluation Metric and Protocol for Diversity
B.3.1 Humanoid
The Humanoid locomotion task is well-studied in the Quality-Diversity (QD) community, enabling the application of well-defined behavior descriptors (BD) to assess diversity scores. While domain-agnostic metrics like DvD scores can also be applied, we consider domain-specific BDs to be more appropriate and accurate for evaluation in this setting.
B.3.2 SMAC
Complex multi-agent tasks like SMAC lack well-defined BDs. Hence, domain-agnostic diversity measures such as the state-entropy measure should be applied. Moreover, different SMAC winning strategies tend to visit different areas of the map, which can be usually captured by the state-entropy measure.
Iteration 2
Figure 9: Visualization of learned behaviors in Humanoid.
Table 9: Episode returns in Humanoid.
SIPO-RBF	SIPO-WD	SIPO-WD (visual)
π1	4863.9(970.3)	3909.4(533.4)	4761.3(107.8)
π2	3746.5(488.0)	3784.2(481.2)	4349.3(169.0)
π3	3092.0(805.0)	3770.4(674.4)	4724.3(946.5)
π4	2332.8(519.8)	3589.6(387.4)	3819.7(588.7)
Table 10: Reward/diversity of all baselines. The reward metric in SMAC and GRF are evaluation win rate (\%). The evaluation metrics of diversity used in humanoid, SMAC, GRF are the joint torque distance, state entropy (1e-3), and the number of different ball-passing routes, respectively. It is noteworthy that SIPO, in particular, displays an adeptness at training a notably more diverse population while upholding a reasonably moderate level of task performance.
Task/Scenario	SIPO-RBF	SIPO-WD	DIPG	RSPO	SMERL	DvD	PPO
humanoid	3508 / 0.53	3763 / 0.71	5191 / 0.12	1455 / 0.53	4253 / 0.01	4498 / 0.40	5299 / -
SMAC 2m1z	100 / 38	100 / 36	100 / 32	100 / 32	100 / 28	100 / 30	100 / -
SMAC 2c64zg	99 / 72	93 / 56	99 / 70	85 / 56	100 / 42	100 / 57	100 / -
GRF 3v1 (first 4)	93 / 3.0	82 / 3.0	93 / 2.7	94 / 2.0	91 / 1.3	83 / 3.0	92 / 2.7
GRF CA	70 / 3.3	41 / 3.0	46 / 2.3	76 / 2.0	45 / 1.3	-	50 / 1.7
GRF Corner	72 / 2.7	56 / 3.0	75 / 1.7	23 / 1.6	67 / 1.0	-	71 / 2.0
B.3.3 GRF
In our initial study of the GRF task, diversity was evaluated using the  $k$ -nearest-neighbor state entropy estimation as in SMAC (see Table 7). However, we observed a significant difference between the computed scores and visualized behaviors. Further investigation revealed that state entropy can sometimes report fake diversity in GRF. For example, the ball-moving route is highly fine-grained between nearby players in the counter-attack (CA) scenario, and additional passes may not change the state entropy significantly. Instead, agents' positions play a crucial role in this scenario, where different shooting positions can introduce substantial state variance and lead to a higher entropy score. As an example, readers can refer to the replays of SIPO-RBF (4 iterations of seed 2) and PG (seed 2, 1002, 2002, and 3002), where SIPO-RBF discovers four distinct passing strategies, while PG keeps passing the ball to the same player. Nevertheless, the state entropy of PG (0.0397) is higher than that of SIPO-RBF (0.0378).
Hence, we counted the number of distinct policies according to their ball-passing routes, such as passing the ball to different players or shooting with different players, to evaluate diversity in GRF. To quantify these differences, we extracted the positions of the ball and the players in the field and calculated the nearest ally player ID to the ball across a winning episode. We then removed timesteps where the nearest distance was above a pre-defined threshold of 0.03. Typically, these timesteps correspond to instances when the ball is being transferred among players, making the nearest player ID irrelevant. Next, we removed consecutive duplicate player IDs from the resulting sequence to obtain a concise and informative embedding of the ball passing route. By comparing the lengths of their respective embeddings and verifying that the player IDs in each embedding are identical, we determined whether the two policies exhibit similar behavior.
We acknowledge that existing diversity measures may not be applicable in GRF, and hence we opted for this novel approach to evaluate diversity. Additionally, we experimented with using raw observations, which include ball ownership information provided by the game engine, but found it to be highly inaccurate based on our visualization.
B.4 Additional Ablation Studies
B.4.1 Input to the Diversity Measure
Vectorized States in Google Research Football We perform an additional ablation study over the input of our diversity measure in GRF  $3\nu l$  scenario with SIPO-WD. We consider the following kinds of state input besides the default state input we adopted in Sec. 6:
- full observation (named full, 115 dims);
- default state input with random noises of the same dimension (named random, 36 dims).
The numbers of visually distinct strategies are listed in Table 11. The performance of full and random is similarly good. The result implies that the learnable discriminator can automatically filter out irrelevant states to some extent, and that SIPO-WD performs relatively robust w.r.t. different state input of the diversity measure.
Table 11: State input ablation. The table shows the number of distinct strategies in GRF  $3\nu l$ .
SIPO-WD	full	random
3v1	3.0 (0.0)	3.0 (0.8)	3.0 (0.0)