Title: Iteratively Learn Diverse Strategies with State Distance Information
Venue: NeurIPS 2023
Accepted: yes
Keywords: diverse behavior

multi-agent reinforcement learning

deep reinforcement learning
Abstract: In complex reinforcement learning (RL) problems, policies with similar rewards may have substantially different behaviors. It remains a fundamental challenge to optimize rewards while also discovering as many *diverse* strategies as possible, which can be crucial in many practical applications. Our study examines two design choices for tackling this challenge, i.e., *diversity measure* and *computation framework*. First, we find that with existing diversity measures, visually indistinguishable policies can still yield high diversity scores. To accurately capture the behavioral difference, we propose to incorporate the state-space distance information into the diversity measure. In addition, we examine two common computation frameworks for this problem, i.e., population-based training (PBT) and iterative learning (ITR). We show that although PBT is the precise problem formulation, ITR can achieve comparable diversity scores with higher computation efficiency, leading to improved solution quality in practice. Based on our analysis, we further combine ITR with two tractable realizations of the state-distance-based diversity measures and develop a novel diversity-driven RL algorithm, *State-based Intrinsic-reward Policy Optimization* (SIPO), with provable convergence properties. We empirically examine SIPO across three domains from robot locomotion to multi-agent games. In all of our testing environments, SIPO consistently produces strategically diverse and human-interpretable policies that cannot be discovered by existing baselines.

Iteratively Learn Diverse Strategies with State Distance Information

Abstract

In complex reinforcement learning (RL) problems, policies with similar rewards may have substantially different behaviors. It remains a fundamental challenge to optimize rewards while also discovering as many diverse strategies as possible, which can be crucial in many practical applications. Our study examines two design choices for tackling this challenge, i.e., diversity measure and computation framework . First, we find that with existing diversity measures, visually indistinguishable policies can still yield high diversity scores. To accurately capture the behavioral difference, we propose to incorporate the state-space distance information into the diversity measure. In addition, we examine two common computation frameworks for this problem, i.e., population-based training (PBT) and iterative learning (ITR). We show that although PBT is the precise problem formulation, ITR can achieve comparable diversity scores with higher computation efficiency, leading to improved solution quality in practice. Based on our analysis, we further combine ITR with two tractable realizations of the state-distance-based diversity measures and develop a novel diversity-driven RL algorithm, State-based Intrinsic-reward Policy Optimization (SIPO), with provable convergence properties. We empirically examine SIPO across three domains from robot locomotion to multi-agent games. In all of our testing environments, SIPO consistently produces strategically diverse and human-interpretable policies that cannot be discovered by existing baselines.

1 Introduction

A consensus in deep learning (DL) is that different local optima have similar mappings in the functional space, leading to similar losses to the global optimum [62, 56, 40]. Hence, via stochastic gradient descent (SGD), most DL works only focus on the final performance without considering which local optimum SGD discovers. However, in complex reinforcement learning (RL) problems, the policies associated with different local optima can exhibit significantly different behaviors [10, 35, 64]. Thus, it is a fundamental problem for an RL algorithm to not only optimize rewards but also discover as many diverse strategies as possible. A pool of diversified policies can be further leveraged towards a wide range of applications, including the discovery of emergent behaviors [34, 3, 61], generating diverse dialogues [30], designing robust robots [12, 25, 19], and enhancing human-AI collaboration [38, 9, 11].

Obtaining diverse RL strategies requires a quantitative method for measuring the difference (i.e., diversity ) between two policies. However, how to define such a measure remains an open challenge. Previous studies have proposed various diversity measures, such as comparing the difference between the action distributions generated by policies [60, 38, 74], computing probabilistic distances between the state occupancy of different policies [43], or measuring the mutual information between states and

policy identities [16]. However, it remains unclear which measure could produce the best empirical performance. Besides, the potential pitfalls of these measures are rarely discussed.

In addition to diversity measures, there are two common computation frameworks for discovering diverse policies, including population-based training (PBT) and iterative learning (ITR). PBT directly solves a constrained optimization problem by learning a collection of policies simultaneously, subject to policy diversity constraints [52, 38, 9]. Although PBT is perhaps the most popular framework in the existing literature, it can be computationally challenging [48] since the number of constraints grows quadratically with the number of policies. The alternative framework is ITR, which iteratively learns a single policy that is sufficiently different from previous policies [43, 74]. ITR is a greedy relaxation of the PBT framework and it largely simplifies the optimization problem in each iteration. However, the performance of the ITR framework has not been theoretically analyzed yet, and it is often believed that ITR can be less efficient due to its sequential nature.

We provide a comprehensive study of the two aforementioned design choices. First, we examine the limitations of existing diversity measures in a few representative scenarios, where two policies outputting very different action distributions can still lead to similar state transitions. In these scenarios, state-occupancy-based measures are not sufficient to truly reflect the underlying behavior differences of the policies either. By contrast, we observe that diversity measures based on state distances can accurately capture the visual behavior differences of different policies. Therefore, we suggest that an effective diversity measure should explicitly incorporate state distance information for the best practical use. Furthermore, for the choice of computation framework, we conduct an in-depth analysis of PBT and ITR. We provide theoretical evidence that ITR, which has a simplified optimization process with fewer constraints, can discover solutions with the same reward as PBT while achieving at least half of the diversity score. This finding implies that although ITR is a greedy relaxation of PBT, their optimal solutions can indeed have comparable qualities. Furthermore, note that policy optimization is much simplified in ITR, which suggests that ITR can result in much better empirical performances and should be preferred in practice.

Following our insights, we combine ITR and a state-distance-based diversity measure to develop a generic and effective algorithm, State-based Intrinsic-reward Policy Optimization (SIPO) , for discovering diverse RL strategies. In each iteration, we further solve this constrained optimization problem via the Lagrange method and two-timescale gradient descent ascent (GDA) [31]. We theoretically prove that our algorithm is guaranteed to converge to a neighbor of ϵ -stationary point. Regarding the diversity measure, we provide two practical realizations, including a straightforward version based on the RBF kernel and a more general learning-based variant using Wasserstein distance.

We evaluate SIPO in three domains ranging from single-agent continuous control to multi-agent games: Humanoid locomotion [42], StarCraft Multi-Agent Challenge [57], and Google Research Football (GRF) [26]. Our findings demonstrate that SIPO surpasses baselines in terms of population diversity score across all three domains. Remarkably, our algorithm can successfully discover 6 distinct human-interpretable strategies in the GRF 3-vs-1 scenario and 4 strategies in two 11-player GRF scenarios, namely counter-attack, and corner, without any domain-specific priors, which are beyond the capabilities of existing algorithms.

2 Related Work

Diversity in RL. It has been shown that policies trained under the same reward function can exhibit significantly different behaviors [10, 35]. Merely discovering a single high-performing solution may not suffice in various applications [12, 64, 25]. As such, the discovery of a diverse range of policies is a fundamental research problem, garnering attention over many years [44, 13, 28]. Early works are primarily based on multi-objective optimization [45, 55, 39, 47, 54], which assumes a set of reward functions is given in advance. In RL, this is also related to reward shaping [46, 2, 15, 61]. We consider learning diverse policies without any domain knowledge.

Population-based training (PBT) is the most popular framework for diverse solutions by jointly learning separate policies. Representative works include evolutionary computation [65, 37, 52], league training [64, 23], computing Hessian matrix [51] or constrained optimization with a population diversity measure [38, 73, 29, 36, 9]. An improvement is to learn a latent variable policy instead of separate ones. Prior works have incorporated different domain knowledge to design the latent code,

such as action clustering [66], agent identities [29, 21] or prosocial level [53, 3]. The latent variable can be also learned in an unsupervised fashion, such as in DIYAN [16] and its variants [25, 49]. Zahavy et al. [71] learns diverse policies with hard constraints on rewards to ensure the derived policies are (nearly) optimal, potentially hindering policies with disparate reward scales. On the other hand, our method prioritizes diversity and fully accepts sub-optimal strategies.

Iterative learning (ITR) simplifies PBT by only optimizing a single policy in each iteration and forcing it to behave differently w.r.t. previously learned ones [43, 60, 74]. While some ITR works require an expensive clustering process before each iteration [72] or domain-specific features [70], we consider domain-agnostic ITR in an end-to-end fashion. Besides, Pacchiano et al. [50] learns a kernelbased score function to iteratively guide policy optimization. The score function is conceptually similar to SIPO-WD but is applied to a parallel setting with more restricted expressiveness power.

Diversity Measure. Most previous works considered diversity measures on action distribution and state occupancy. For example, measures such as Jensen-Shannon divergence [38] and crossentropy [74] are defined over policy distributions to encourage different policies to take different actions on the same state, implicitly promoting the generation of diverse trajectories. Other measures such as maximum mean discrepancy [43] maximize the probability distance between the state distributions induced by two policies. However, these approaches can fail to capture meaningful behavior differences between two policies in certain scenarios, as we will discuss in Section 4.1. There also exist specialized measures, such as cross-play rewards [9], which are designed for cooperative multi-agent games. It is worth noting that diversity measures are closely related to exploration criteria [4, 20, 6, 27] and skill discovery [8, 32, 24], where a diversity surrogate objective is often introduced to encourage broad state coverage. However, this paper aims to explicitly discover mutually distinct policies. Our diversity measure depends on a function that computes the distance between states visited by two policies.

3 Preliminary

Notation: We consider POMDP [59] defined by M = ⟨S , A , O , r, P, O, ν, H ⟩ . S is the state space. A and O are the action and observation space. r : S × A → R is the reward function. O : S → O is the observation function. H is the horizon. P is the transition function. At timestep h , the agent receives an observation o h = O ( s h ) and outputs an action a h ∈ A w.r.t. its policy

[ ∑ ] .

The above formulation can be naturally extended to cooperative multi-agent settings, where π and R correspond to the joint policy and the shared reward. We follow the standard POMDP notations for conciseness. Our method will be evaluated in both single-agent tasks and complex cooperative multi-agent scenarios. Among them, multi-agent environments encompass a notably more diverse range of potential winning strategies, and hence offer an apt platform for assessing the effectiveness of our method. Moreover, in this paper, we assume access to object-centric information and features rather than pure visual observations to simplify our discussion. We remark that although we restrict the scope of this paper to states, our method can be further extended to high-dimensional inputs (e.g. images, see App. B.4.1) or tabular MDPs via representation learning [68, 14].

Finally, to discover diverse strategies, we aim to learn a set of M policies { π i } i M =1 such that all of these policies are locally optimal under J ( · ) but mutually distinct subject to some diversity measure D ( · , · ) : △×△→ R , which captures the difference between two policies.

Existing Diversity Measures: We say a diversity measure D is defined over action distribution if it can be written as

where q is an occupancy measure over states, ˜ D A : △×△→ R measures the difference between action distributions. ˜ D A can be any probability distance as defined in prior works [60, 38, 74, 52].

Denote the state occupancy of π as q π . We say a diversity measure is defined over state occupancy if it can be written as

Agent

Goal

which can be realized as an integral probability metric [43]. We remark that q π is usually intractable. In addition to diversity measures, we present two popular computation frameworks for this purpose.

Population-Based Training (PBT): PBT is a straightforward formulation by jointly learning M policies { π i } i M =1 subject to pairwise diversity constraints, i.e.,

̸

where δ is a threshold. In our paper, we consistently refer to the aforementioned computation framework as "PBT", rather than adjusting hyperparameters [22]. Despite a precise formulation, PBT poses severe optimization challenges due to mutual constraints.

Iterative Learning (ITR): ITR is a greedy approximation of PBT by iteratively learning novel policies. In the i -th ( 1 ≤ i ≤ M ) iteration, ITR solves

π ⋆ j is recursively defined by the above equation. Compared with PBT, ITR trades off wall-clock time for less required computation resources (e.g., GPU memory) and performs open-ended training (i.e., the population size M does not need to be fixed at the beginning of training).

4 Analysis of Existing Diversity-Discovery Approaches

In this section, we conduct both quantitative and theoretical analyses of existing approaches to motivate our method. We first discuss diversity measures in Sec. 4.1 and then compare computation frameworks, namely PBT and ITR, in Sec. 4.2.

4.1 A Common Missing Piece in Diversity Measure: State Distance

The perception of diversity among humans primarily relies on the level of dissimilarity within the state space, which is measured by a distance function. However, the diversity measures outlined in Eq. (1) and Eq. (2) completely fail to account for such crucial information. In this section, we provide a detailed analysis to instantiate this observation with concrete examples and propose a novel diversity measure defined over state distances.

First, we present a synthetic example to demonstrate the limitations of current diversity measures. Our example consists of a grid-world environment with a single agent and grid size N G . The agent starts at the top left of the grid-world and must navigate to the bottom right corner, as shown in Fig. 1. While N G can be large in general, we illustrate with N G = 5 for simplicity. We draw five distinct policies, denoted as π 1 through π 5 , which differ in their approach to navigating the grid-world. Consider π 1 , π 2 , and π 3 first. Although humans may intuitively perceive that policies π 1 and π 2 , which move along the diagonal, are more similar to each other than to π 3 , which moves along the boundary, diversity measures based on actions can fail to reflect this intuition, as shown in Table 1. Then, let's switch to policies π 3 , π 4 , and π 5 . We find that state-occupancy-based diversity measures are unable to differentiate between π 4 and π 5 in contrast to π 3 . This is because the states visited by π 3 are entirely disjoint from those visited by both π 4 and π 5 . However, humans would judge π 5 to be more distinct from π 3 than π 4 because both π 3 and π 4 tend to visit the upper boundary.

Idle Player

CURIE

ARCHIMEDINHO

Training Progress

Iteration 1

Iteration 2

Iteration 3

Iteration 4

Epoch 10

Epoch 20

Epoch 30

PBT

ITR

Next, we consider a more realistic and complicated multi-agent football scenario, i.e., the Google Research Football [26] environment, in Fig. 2, where an idle player in the backyard takes an arbitrary action without involving in the attack at all. Although the idle player stays still with no effect on the team strategy, action-based measures can produce high diversity scores. This example underscores a notable issue. If action-based measures are leveraged to optimize diversity, the resultant policies can produce visually similar behavior. While it can be possible to exclude idle actions by modifying task rewards, it requires domain-specific hacks and engineering efforts. The issue of idle actions exists even in such popular MARL benchmarks. Similar issues have also been observed in previous works [38].

To summarize, existing measures suffer from a significant limitation - they only compare the behavior trajectories implicitly through the lens of action or state distribution without explicitly measuring state distance . Specifically, action-based measures fail to capture the behavioral differences that may arise when similar states are reached via different actions. Similarly, state occupancy measures do not quantify the degree of dissimilarity between states. To address this limitation, we propose a new diversity measure that explicitly takes into account the distance function in state space:

d is a distance metric over S × S . g : R + → R is a monotonic cost function. γ ∈ Γ( q π i , q π j ) is a distribution over state pairs. Γ( q π i , q π j ) denotes the collection of all distributions on S × S with marginals q π i and q π j on the first and second factors respectively. The cost function g is a notation providing a generalized and unified definition. It also contributes to training stability by scaling the raw distance. We highlight that Eq. (5) computes the cost on individual states before taking expectation, and therefore prevents information loss of taking the average over the entire trajectory (e.g. the DvD score [52]). We also note that states are consequences of performed actions. Hence, a state-distance-based measure also implicitly reflects the (meaningful) differences in actions between two policies. We compute two simple measures based on state distance, i.e., the L 2 norm and the Earth Moving Distance (EMD), for the grid-world example and present results in Table 1. These measures are consistent with human intuition.

4.2 Computation Framework: Population-Based or Iterative Learning?

We first consider the simplest motivating example to intuitively illustrate the optimization challenges. Let's assume that π i is a scalar, J ( π i ) is linear in π i , and D ( π i , π j ) = | π i -π j | . In our definition, where M denotes the number of diverse policies, PBT involves Θ( M 2 ) constraints in a single linear programming problem while ITR involves O ( M ) constraints in each of M iterations. Given that the complexity of linear programming is a high-degree polynomial (higher than 2) of the number of constraints, solving PBT is harder (and probably slower) than solving ITR in a total of M iterations,

despite PBT being parallelized . This challenge can be more severe in RL due to complex solution space and large training variance.

Although ITR can be optimized efficiently, it remains unclear whether ITR, as a greedy approximation of PBT, can obtain solutions of comparable rewards. Fig. 4 shows the worst case in the 1-D setting when the ITR solutions (green) can indeed have lower rewards than the PBT solution (red) subject to the same diversity constraint. However, we will show in the next theorem that ITR is guaranteed to have no worse rewards than PBT by trading off half of the diversity.

δ/2

δ/2

δ/2

δ/2

optimal solution

solution found by ITR

with threshold

δ/2

worst-case solution found

by ITR with threshold

δ

J(π)

π

1

2

1

2

Theorem 4.1. Assume D is a distance metric. Denote the optimal value of Eq.( 3) as T 1 . Let T 2 = ∑ M i =1 J (˜ π i ) where

Please see App. E.1 for the proof. The above theorem provides a quality guarantee for ITR. The proof can be intuitively explained by the 1-D example in Fig. 4, where green points represent the worst case with threshold δ and blue points represent the solutions with threshold δ/ 2 . Thm. 4.1 shows that, for any policy pool derived by PBT, we can always use ITR to obtain another policy pool, which has the same rewards and comparable diversity scores .

Empirical Results: We empirically compare PBT and ITR in a 2-D navigation environment with 1 agent and N L landmarks in Fig. 3. The reward is 1 if the agent successfully navigates to a landmark and 0 otherwise. We train N L policies using both PBT and ITR to discover strategies toward each of these landmarks. More details can be found in App. D. Table 2 shows the number of discovered landmarks by PBT and ITR. ITR performs consistently better than PBT even in this simple example. We intuitively

illustrate the learning process of PBT and ITR in Fig. 3. ITR, due to its computation efficiency, can afford to run longer iterations and tolerate larger exploration noises. Hence, it can converge easily to diverse solutions by imposing a large diversity constraint. PBT, however, only converges when the exploration is faint, otherwise it diverges or converges too slowly.

4.3 Practical Remark

Based on the above analyses, we suggest ITR and diversity measures based on state distances be preferred in RL applications. We also acknowledge that, by the no-free-lunch theorem, they cannot be universal solutions and that trade-offs may still exist (see discussions in Sec.7 and App.F). Nonetheless, in the following sections, we will show that the effective implementation of these choices can lead to superior performances in various challenging benchmarks. We hope that our approach will serve as a starting point and provide valuable insights into the development of increasingly powerful algorithms for potentially more challenging scenarios.

5 Method

In this section, we develop a diversity-driven RL algorithm, State-based Intrinsic-reward Policy Optimization (SIPO) , by combining ITR and state-distance-based measures. SIPO runs M iterations to discover M distinct policies. At the i -th iteration, we solve equation (4) by converting it into unconstrained optimization using the Lagrange method. The unconstrained optimization can be written as:

λ j ( 1 ≤ j < i ) are Lagrange multipliers. { π ⋆ j } i -1 j =1 are previously obtained policies. We adopt two-timescale Gradient Descent Ascent (GDA) [31] to solve the above minimax optimization, i.e., performing gradient descent over π i and gradient ascent over λ j with different learning rates. In our algorithm, we additionally enforce the dual variables λ j to be bounded (i.e., in an interval [0 , Λ] for a large number Λ ), which plays an important role both in the theoretical analysis and in empirical convergence. However, D S ( π i , π ⋆ j ) cannot be directly optimized w.r.t. π i through gradient-based methods because it depends on the states traversed by π , rather than its output (e.g. actions). Therefore, we cast D S ( π i , π ⋆ j ) as the cumulative sum of intrinsic rewards, specifically the intrinsic return. This allows us to leverage policy gradient techniques for optimization. The pseudocode of SIPO can be found in App. G.

An important property of SIPO is the convergence guarantee. We present an informal illustration in Thm. 5.1 and present the formal theorem with proof in App. E.2.

Theorem 5.1. (Informal) Under continuity assumptions, SIPO converges to an ϵ -stationary point.

Remark: We assumed that the return J and the distance D S are smooth in policies. In practice, this is true if (1) policy and state space are bounded and (2) reward function and system dynamics are continuous in the policy. (Continuous functions are bounded over compact spaces.) The key step is to analyze the role of the bounded dual variables λ , which achieves an 1 Λ -approximation of constraint without hurting the optimality condition.

Instead of directly defining D S , we define intrinsic rewards as illustrated in Sec. 5, such that D S ( π i , π ⋆ j ) = E s h ∼ µ π i [ ∑ H h =1 r int ( s h ; π i , π ⋆ j ) ] .

RBF Kernel: The most popular realization of Eq. (5) in machine learning is through kernel functions. Herein, we realize Eq. (5) as an RBF kernel on states. Formally, the intrinsic reward is defined by

where σ is a hyperparameter controlling the variance.

Wasserstein Distance: For stronger discrimination power, we can also realize Eq. (5) as L 2 -Wasserstein distance. According to the dual form [63], we define

where f : S → R is a 1 -Lipschitz function. This realization holds a distinct advantage due to its interpretation within optimal transport theory [63, 1]. Unlike distances that rely solely on specific summary statistics such as means, Wasserstein distance can effectively quantify shifts in state distributions and remains robust in the presence of outliers [63]. We implement f as a neural network and clip parameters to [ -0 . 01 , 0 . 01] to ensure the Lipschitz constraint. Note that r WD int incorporates representation learning by utilizing a learnable scoring function f and is more flexible in practice. We also show in App. B.4 that r WD int is robust to different inputs, including states with random noises and RGB images.

We name SIPO with r RBF int and r WD int SIPO-RBF and SIPO-WD respectively.

Implementation: To incorporate temporal information, we stack the recent 4 global states to compute intrinsic rewards and normalize the intrinsic rewards to stabilize training. In multi-agent environments, we learn an agent-ID-conditioned policy [17] and share the parameter across all agents. Our implementation is based on MAPPO [69] with more details in App. D.

6 Experiments

We evaluate SIPO across three domains that exhibit multi-modality of solutions. The first domain is the humanoid locomotion task in Isaac Gym [42], where diversity can be quantitatively assessed by well-defined behavior descriptors. We remark that the issues we addressed in Sec. 4.1 may not be present in this task where the action space is small and actions are highly correlated with states. Further, we examine the effectiveness of SIPO in two much more challenging multi-agent domains, StarCarft Multi-Agent Challenge (SMAC) [57] and Google Research Football (GRF) [26], where well-defined behavior descriptors are not available and existing diversity measures may produce misleading diversity scores. We provide introductions to these environments in App. C.

First, we show that SIPO can efficiently learn diverse strategies and outperform several baseline methods, including DIPG [43], SMERL [25], DvD [52], and RSPO [74]. Then, we qualitatively demonstrate the emergent behaviors learned by SIPO, which are both visually distinguishable and human-interpretable . Finally, we perform an ablation study over the building components of SIPO and show that both the diversity measure, ITR, and GDA are critical to the performance.

All algorithms run for the same number of environment frames on a desktop machine with an RTX3090 GPU. Numbers are average values over 5 seeds in Humanoid and SMAC and 3 seeds in GRF with standard deviation shown in brackets. More algorithm details can be found in App. D. Additional visualization results can be found on our project website (see App. A).

6.1 Comparison with Baseline Methods

Humanoid Locomotion. Following Zhou et al. [74], we train a population of size 4 . We assess diversity by the pairwise distance of joint torques, a widely used behavior descriptor in recent Quality-Diversity works [67]. Torque states are not included as the input of diversity measures and we only use them for evaluation to ensure a fair comparison. Results are shown in Table 3. We can see that both variants of SIPO can outperform all baseline methods except that SIPO-RBF achieves comparable performance with RSPO, even if RSPO explicitly encourages the output of different actions/forces.

SMAC Following Zhou et al. [74], we run SIPO and all baselines on an easy map, 2m_vs_1z , and a hard map, 2c_vs_64zg , both across 4 iterations. We merge all trajectories produced by the policy collection and incorporate a k -nearest-neighbor state entropy estimation [58] to assess diversity. Intuitively, a more diverse population should have a larger state entropy value. We set k = 12 following Liu and Abbeel [32] and show results in Table 4. On these maps, two agents are both involved in the attack. Therefore, RSPO, which incorporates an action-based cross-

Table 4: State entropy estimated by k -nearest-neighbor in SMAC. ( k = 12 )

entropy measure, can perform well across all baselines. However, SIPO explicitly compares the distance between resulting trajectories and can even outperform RSPO, leading to the most diverse population.

GRF We consider three academy scenarios, specifically 3v1 , counterattack ( CA ), and corner . The GRF environment is more challenging than SMAC due to the large action space, more agents, and the existence of duplicate actions. We determine a population size M = 4 by balancing resources and wall-clock time across different baselines. Table 5 compares the number of distinct policies (in terms of ball-passing routes, see App. B.3) discovered in the population. Due to the strong adversarial power of our diversity measures and the application of GDA, SIPO is the most efficient and robust - even in the challenging 11-vs-11 corner and CA scenario, SIPO can effectively discover different winning strategies in just a few iterations across different seeds. By contrast, baselines suffer from learning instability in these challenging environments and tend to discover policies with slight distinctions. We also calculate the estimated state entropy as we did in SMAC. However, we find that this metric cannot distinguish fine-grained ball-passing behaviors in GRF (check our discussions in App. B).

Remark: In GRF experiments, when M is small, even repeated training with different random seeds (PG) is a strong baseline (see Table 5). Hence, the numbers are actually restricted in a small interval (with a lower bound equal to PG results and an upper bound equal to M = 4 ), which makes the improvements by SIPO seemingly less significant. However, achieving clear improvements in these challenging applications remains particularly non-trivial. With a population size M = 10 , SIPO clearly outperforms baselines by consistently discovering one or more additional strategies.

6.2 Qualitative Analysis

For SMAC, we present heatmaps of agent positions in Fig. 5. The heatmaps clearly show that SIPO can consistently learn novel winning strategies to conquer the enemy. Fig. 6 presents the learned

2c_vs_64zg

2m_vs_1z

policy 1

policy 3

policy 4

25

25

25

25

0.9

1.0

20

20

20 -

yposition

20

density

15

15

15

15

0.3

10

10 -

10

10

0.2

0.1

5

5

10

15

20

25

5

5

10

15

20

25

5

5

10

15

20

25

5

5

10

1520

25

0.0

xposition

policy 1

policy3

policy 4

30

30

30

30

0.9

1.0

25 -

25

25

y position

252015

20

20

20

density

15

15

10

10

10

10

5

5

0.2

0.1

0+

0

10

20

30

0+

0

10

20

30

0+

0

10

20

30

+0

0

xposition

10

20

30

0.0

Attacker

Bal  Direction

Player Direction

1

G

3

2

1

Bal  Direction

Player Direction

1

G

3

2

1

Bal  Direction

Player Direction

1

G

3

2

1

Bal  Direction

Player Direction

1

G

3

2

1

Bal  Direction

Player Direction

1

G

3

2

1

Bal  Direction

Player Direction

1

G

3

2

1

Bal  Direction

Player Direction

1

G

3

2

1

Bal  Direction

Player Direction

1

G

3

2

1

iteration 1

iteration 2

iteration 3

iteration 4

iteration 5

iteration 6

iteration 7

1

1

Ball Direction

Player Direction

1.0

Rate

0.5

Win

0.0

0.0

0.2

0.61

0.8

1.01

1.41

Environment Erames (1.0e8)

behavior by SIPO in the GRF 3v1 scenario of seed 1. We can observe that agents have learned a wide spectrum of collaboration strategies across merely 7 iterations. The strategies discovered by SIPO are both diverse and human-interpretable . In the first iteration, all agents are involved in the attack such that they can distract the defender and obtain a high win rate. The 2nd and the 6th iterations demonstrate an efficient pass-and-shoot strategy, where agents quickly elude the defender and score a goal. In the 3rd and the 7th iterations, agents learn smart 'one-two' strategies to bypass the defender, a prevalent tactic employed by human football players. We note that NONE of the baselines have ever discovered this strategy across all runs, while SIPO is consistently able to derive such strategies for all random seeds. Visualization results in CA and corner scenarios can be found in App. B.

6.3 Ablation Study

We apply these changes to SIPO-WD:

fix-L : Fixing the multiplier λ i instead of applying GDA.

CE : The intrinsic reward is replaced with cross-entropy, i.e., r CE int ( s h , a h ) = -log π ⋆ j ( a h | s h ) , where π ⋆ j denotes a previously discovered policy. Additionally, GDA is still applied.

filter : Optimizing the extrinsic rewards on trajectories that have intrinsic returns exceeding δ and optimizing intrinsic rewards defined by Eq. (9) for other trajectories [74].

PBT : Simultaneously training M policies with M ( M -1) / 2 constraints (i.e., directly solving Eq. (3)) with intrinsic rewards defined by Eq. (9) and GDA.

We report the number of visually distinct policies discovered by these methods in Table 6. Comparison between SIPO and CE demonstrates that the action-based cross-entropy measure may suffer from duplicate actions in GRF and produce nearly identical behavior by overly exploiting duplicate actions, especially in the CA and corner scenarios with 11

1 Not converged.

2 Training requires > 24GB memory and exceeds our memory limit.

agents. Besides, the fixed Lagrange coefficient, the filtering-based method, and PBT are all detrimental to our algorithm. These methods also suffer from significant training instability. Overall, the state-distance-based diversity measure, ITR, and GDA are all critical to the performance of SIPO.

7 Conclusion

We tackle the problem of discovering diverse high-reward policies in RL. First, we demonstrate concrete failure cases of existing diversity measures and propose a novel measure that explicitly compares the distance in state space. Next, we present a thorough comparison between PBT and ITR and show that ITR is much easier to optimize and can derive solutions with comparable quality to PBT. Motivated by these insights, we combine ITR with a state-distance-based diversity measure to develop SIPO, which has provable convergence and can efficiently discover a wide spectrum of human-interpretable strategies in a wide range of environments.

Limitations: First, we assume direct access to an object-centric state representation. When such a representation is not available (e.g., image-based observations), representation learning becomes necessary and algorithm performance can be affected by the quality of the learned representations. Second, because ITR requires sequential training, the wall clock time of SIPO can be longer than the PBT alternatives when fixing the total number of training samples. The acceleration of ITR remains an open challenge.

Future Directions: Besides addressing the above limitations, we suggest three additional future directions based on our paper. First, a consensus on the best algorithmic formulation of distinct solutions in RL remains elusive. It is imperative to understand diversity in a more theoretical manner. Second, while this paper focuses on single-agent and cooperative multi-agent domains, extending SIPO to multi-agent competitive games holds great potential. Finally, although SIPO/ITR enables open-ended training, it is worth studying how to determine the optimal population size to better balance resources and the diversity of the resulting population.

References

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 214-223. PMLR, 06-11 Aug 2017. URL https://proceedings.mlr.press/v70/ arjovsky17a.html .

Monica Babes, Enrique Munoz de Cote, and Michael L. Littman. Social reward shaping in the prisoner's dilemma. In Lin Padgham, David C. Parkes, Jörg P. Müller, and Simon Parsons, editors, 7th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2008), Estoril, Portugal, May 12-16, 2008, Volume 3 , pages 1389-1392. IFAAMAS, 2008. URL https://dl.acm.org/citation.cfm?id=1402880 .

Bowen Baker, Ingmar Kanitscheider, Todor M. Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=SkxpxJBKwS .

Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems , 29, 2016.

Yura Burda and Harri Edwards, Oct 2018. URL https://openai.com/blog/ reinforcement-learning-with-prediction-based-rewards/ .

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894 , 2018.

Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B. Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=ETBc_ MIMgoX .

Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning , pages 1317-1327. PMLR, 2020.

Rujikorn Charakorn, Poramate Manoonpong, and Nat Dilokthanakul. Generating diverse cooperative agents by learning incompatible policies. In ICML 2022 Workshop AI for AgentBased Modelling , 2022.

Jack Clark and Dario Amodei, Dec 2016. URL https://openai.com/blog/ faulty-reward-functions/ .

Brandon Cui, Andrei Lupu, Samuel Sokota, Hengyuan Hu, David J Wu, and Jakob Nicolaus Foerster. Adversarial diversity in hanabi. In The Eleventh International Conference on Learning Representations , 2023.

Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. Robots that can adapt like animals. Nature , 521(7553):503-507, May 2015. ISSN 0028-0836, 1476-4687. doi: 10.1038/nature14422. URL http://www.nature.com/articles/nature14422 .

Kalyanmoy Deb and Amit Saha. Finding multiple solutions for multimodal optimization problems using a multi-objective evolutionary approach. In Martin Pelikan and Jürgen Branke, editors, Genetic and Evolutionary Computation Conference, GECCO 2010, Proceedings, Portland, Oregon, USA, July 7-11, 2010 , pages 447-454. ACM, 2010. doi: 10.1145/1830483.1830568. URL https://doi.org/10.1145/1830483.1830568 .

Fei Deng, Ingook Jang, and Sungjin Ahn. Dreamerpro: Reconstruction-free model-based reinforcement learning with prototypical representations. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 4956-4975. PMLR, 2022. URL https://proceedings.mlr.press/v162/deng22a.html .

Sam Devlin and Daniel Kudenko. Theoretical considerations of potential-based reward shaping for multi-agent systems. In Liz Sonenberg, Peter Stone, Kagan Tumer, and Pinar Yolum, editors, 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2011), Taipei, Taiwan, May 2-6, 2011, Volume 1-3 , pages 225-232. IFAAMAS, 2011. URL http: //portal.acm.org/citation.cfm?id=2030503&CFID=69153967&CFTOKEN=38069692 .

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=SJx63jRqFm .

Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. Revisiting some common practices in cooperative multi-agent reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 6863-6877. PMLR, 17-23 Jul 2022. URL https: //proceedings.mlr.press/v162/fu22d.html .

Dibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with goal conditioned policies. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https: //openreview.net/forum?id=Hye9lnCct7 .

Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature communications , 12(1):1-12, 2021.

Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning , pages 2681-2691. PMLR, 2019.

Siyi Hu, Chuanlong Xie, Xiaodan Liang, and Xiaojun Chang. Policy diagnosis via measuring role diversity in cooperative multi-agent RL. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 9041-9071. PMLR, 2022. URL https://proceedings. mlr.press/v162/hu22c.html .

Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population Based Training of Neural Networks, November 2017. URL http://arxiv.org/abs/1711.09846 . arXiv:1711.09846 [cs].

Max Jaderberg, Wojciech M. Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel. Human-level performance in 3D multiplayer games with population-based reinforcement learning. Science , 364(6443):859-865, May 2019. ISSN 0036-8075, 1095-9203. doi: 10.1126/science.aau6249. URL https://www.science.org/ doi/10.1126/science.aau6249 .

Zheyuan Jiang, Jingyue Gao, and Jianyu Chen. Unsupervised skill discovery via recurrent skill training. Advances in Neural Information Processing Systems , 35:39034-39046, 2022.

Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent RL. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 5d151d1059a6281335a10732fc49620e-Abstract.html .

Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly. Google research football: A novel reinforcement learning environment. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 4501-4510. AAAI Press, 2020. URL https://ojs.aaai.org/index.php/ AAAI/article/view/5878 .

Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric P. Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. CoRR , abs/1906.05274, 2019. URL http://arxiv.org/abs/1906.05274 .

Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from underspecified data. CoRR , abs/2202.03418, 2022. URL https://arxiv.org/abs/2202. 03418 .

Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 3991-4002, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 20aee3a5f4643755a79ee5f6a73050ac-Abstract.html .

Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In Jian Su, Xavier Carreras, and Kevin Duh, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016 , pages 1192-1202. The Association for Computational Linguistics, 2016. doi: 10.18653/v1/d16-1127. URL https: //doi.org/10.18653/v1/d16-1127 .

Tianyi Lin, Chi Jin, and Michael I. Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 6083-6093. PMLR, 2020. URL http://proceedings.mlr.press/v119/ lin20a.html .

Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems , 34:18459-18473, 2021.

Iou-Jen Liu, Unnat Jain, Raymond A. Yeh, and Alexander G. Schwing. Cooperative exploration for multi-agent deep reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 6826-6836. PMLR, 2021. URL http://proceedings.mlr.press/v139/liu21j.html .

Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, and Thore Graepel. Emergent coordination through competition. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=BkG8sjR5Km .

Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, S. M. Ali Eslami, Daniel Hennes, Wojciech M. Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, Noah Y. Siegel, Leonard Hasenclever, Luke Marris, Saran Tunyasuvunakool, H. Francis Song, Markus Wulfmeier, Paul Muller, Tuomas Haarnoja, Brendan D. Tracey, Karl Tuyls, Thore Graepel, and Nicolas Heess. From motor control to team play in simulated humanoid football. Sci. Robotics , 7(69), 2022. doi: 10.1126/scirobotics.abo0235. URL https://doi.org/10.1126/scirobotics.abo0235 .

Xiangyu Liu, Hangtian Jia, Ying Wen, Yujing Hu, Yingfeng Chen, Changjie Fan, Zhipeng Hu, and Yaodong Yang. Towards unifying behavioral and response diversity for open-ended learning in zero-sum games. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 941-952, 2021. URL https://proceedings.neurips. cc/paper/2021/hash/07bba581a2dd8d098a3be0f683560643-Abstract.html .

Qian Long, Zihan Zhou, Abhinav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang. Evolutionary population curriculum for scaling multi-agent reinforcement learning. In International Conference on Learning Representations , 2020.

Andrei Lupu, Hengyuan Hu, and Jakob N. Foerster. Trajectory diversity for zero-shot coordination. In Frank Dignum, Alessio Lomuscio, Ulle Endriss, and Ann Nowé, editors, AAMAS '21: 20th International Conference on Autonomous Agents and Multiagent Systems, Virtual Event, United Kingdom, May 3-7, 2021 , pages 1593-1595. ACM, 2021. doi: 10.5555/3463952.3464170. URL https://www.ifaamas.org/Proceedings/aamas2021/ pdfs/p1593.pdf .

Pingchuan Ma, Tao Du, and Wojciech Matusik. Efficient continuous pareto exploration in multi-task learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 6522-6531. PMLR, 2020. URL http://proceedings.mlr.press/v119/ ma20a.html .

Tengyu Ma. Why Do Local Methods Solve Nonconvex Problems?, March 2021. URL http: //arxiv.org/abs/2103.13462 . arXiv:2103.13462 [cs, math, stat].

Marlos C. Machado, Marc G. Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 5125-5133. AAAI Press, 2020. URL https://ojs.aaai.org/index.php/AAAI/article/view/5955 .

Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning, 2021.

Muhammad A. Masood and Finale Doshi-Velez. Diversity-inducing policy gradient: Using maximum mean discrepancy to find a set of diverse policies. In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019 , pages 5923-5929. ijcai.org, 2019. doi: 10.24963/ijcai. 2019/821. URL https://doi.org/10.24963/ijcai.2019/821 .

B.L. Miller and M.J. Shaw. Genetic algorithms with dynamic niche sharing for multimodal function optimization. In Proceedings of IEEE International Conference on Evolutionary Computation , pages 786-791, Nagoya, Japan, 1996. IEEE. ISBN 978-0-7803-2902-7. doi: 10.1109/ICEC.1996.542701. URL http://ieeexplore.ieee.org/document/542701/ .

Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. CoRR , abs/1504.04909, 2015. URL http://arxiv.org/abs/1504.04909 .

Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml , volume 99, pages 278-287, 1999.

Olle Nilsson and Antoine Cully. Policy gradient assisted MAP-Elites. In Proceedings of the Genetic and Evolutionary Computation Conference , pages 866-875, Lille France, June 2021. ACM. ISBN 978-1-4503-8350-9. doi: 10.1145/3449639.3459304. URL https://dl.acm. org/doi/10.1145/3449639.3459304 .

Shayegan Omidshafiei, Karl Tuyls, Wojciech M Czarnecki, Francisco C Santos, Mark Rowland, Jerome Connor, Daniel Hennes, Paul Muller, Julien Pérolat, Bart De Vylder, et al. Navigating the landscape of multiplayer games. Nature communications , 11(1):1-17, 2020.

Takayuki Osa, Voot Tangkaratt, and Masashi Sugiyama. Discovering diverse solutions in deep reinforcement learning by maximizing state-action-based mutual information. Neural Networks , 152:90-104, 2022. doi: 10.1016/j.neunet.2022.04.009. URL https://doi.org/10.1016/j. neunet.2022.04.009 .

Aldo Pacchiano, Jack Parker-Holder, Yunhao Tang, Krzysztof Choromanski, Anna Choromanska, and Michael Jordan. Learning to score behaviors for guided policy optimization. In International Conference on Machine Learning , pages 7445-7454. PMLR, 2020.

Jack Parker-Holder, Luke Metz, Cinjon Resnick, Hengyuan Hu, Adam Lerer, Alistair Letcher, Alexander Peysakhovich, Aldo Pacchiano, and Jakob N. Foerster. Ridge rider: Finding diverse solutions by following eigenvectors of the hessian. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips. cc/paper/2020/hash/08425b881bcde94a383cd258cea331be-Abstract.html .

Jack Parker-Holder, Aldo Pacchiano, Krzysztof Marcin Choromanski, and Stephen J. Roberts. Effective diversity in population based reinforcement learning. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ d1dc3a8270a6f9394f88847d7f0050cf-Abstract.html .

Alexander Peysakhovich and Adam Lerer. Consequentialist conditional cooperation in social dilemmas with imperfect information. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/forum?id=BkabRiQpb .

Thomas Pierrot, Valentin Macé, Félix Chalumeau, Arthur Flajolet, Geoffrey Cideron, Karim Beguir, Antoine Cully, Olivier Sigaud, and Nicolas Perrin-Gilbert. Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization. In Proceedings of the Genetic and Evolutionary Computation Conference , pages 1075-1083, July 2022. doi: 10.1145/3512290. 3528845. URL http://arxiv.org/abs/2006.08505 . arXiv:2006.08505 [cs].

Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. Quality Diversity: A New Frontier for Evolutionary Computation. Frontiers in Robotics and AI , 3, July 2016. ISSN 2296-9144. doi: 10.3389/frobt.2016.00040. URL http://journal.frontiersin.org/Article/10.3389/ frobt.2016.00040/abstract .

Tim Roughgarden, editor. Beyond the Worst-Case Analysis of Algorithms . Cambridge University Press, 2020. ISBN 9781108637435. doi: 10.1017/9781108637435. URL https://doi.org/ 10.1017/9781108637435 .

Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In Edith Elkind, Manuela Veloso, Noa Agmon, and Matthew E. Taylor, editors, Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, Montreal, QC, Canada, May 13-17, 2019 , pages 2186-2188. International Foundation for Autonomous Agents and Multiagent Systems, 2019. URL http://dl.acm.org/citation.cfm?id=3332052 .

Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Nearest neighbor estimates of entropy. American journal of mathematical and management sciences , 23(3-4):301-321, 2003.

Matthijs TJ Spaan. Partially observable markov decision processes. In Reinforcement Learning , pages 387-414. Springer, 2012.

Hao Sun, Zhenghao Peng, Bo Dai, Jian Guo, Dahua Lin, and Bolei Zhou. Novel policy seeking with constrained optimization. arXiv preprint arXiv:2005.10696 , 2020.

Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Shaolei Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview. net/forum?id=lvRTC669EY_ .

Luca Venturi, Afonso S. Bandeira, and Joan Bruna. Neural networks with finite intrinsic dimension have no spurious valleys. CoRR , abs/1802.06384, 2018. URL http://arxiv.org/ abs/1802.06384 .

Cédric Villani. Optimal transport: old and new , volume 338. Springer, 2009.

Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature , 575(7782):350-354, November 2019. ISSN 0028-0836, 1476-4687. doi: 10.1038/ s41586-019-1724-z. URL http://www.nature.com/articles/s41586-019-1724-z .

Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Poet: open-ended coevolution of environments and their optimized solutions. In Proceedings of the Genetic and Evolutionary Computation Conference , pages 142-151, 2019.

Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. RODE: learning roles to decompose multi-agent tasks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=TTUVg6vkNjK .

Shuang Wu, Jian Yao, Haobo Fu, Ye Tian, Chao Qian, Yaodong Yang, QIANG FU, and Yang Wei. Quality-similar diversity via population based reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023.

Yifan Wu, George Tucker, and Ofir Nachum. The laplacian in RL: learning representations with efficient approximations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https: //openreview.net/forum?id=HJlNpoA5YQ .

Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955 , 2021.

Tom Zahavy, Brendan O'Donoghue, Andre Barreto, Volodymyr Mnih, Sebastian Flennerhag, and Satinder Singh. Discovering diverse nearly optimal policies with successor features. arXiv preprint arXiv:2106.00669 , 2021.

Tom Zahavy, Yannick Schroecker, Feryal M. P. Behbahani, Kate Baumli, Sebastian Flennerhag, Shaobo Hou, and Satinder Singh. Discovering policies with domino: Diversity optimization maintaining near optimality. CoRR , abs/2205.13521, 2022. doi: 10.48550/arXiv.2205.13521. URL https://doi.org/10.48550/arXiv.2205.13521 .

Yunbo Zhang, Wenhao Yu, and Greg Turk. Learning novel policies for tasks. In International Conference on Machine Learning , pages 7483-7492. PMLR, 2019.

Rui Zhao, Jinming Song, Hu Haifeng, Yang Gao, Yi Wu, Zhongqian Sun, and Yang Wei. Maximum entropy population based training for zero-shot human-ai coordination. arXiv preprint arXiv:2112.11701 , 2021.

Zihan Zhou, Wei Fu, Bingliang Zhang, and Yi Wu. Continuously discovering novel strategies via reward-switching policy optimization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=hcQHRHKfN_ .

A Project Website

Check https://sites.google.com/view/diversity-sipo for GIF demonstrations.

B Additional Results

B.1 More Qualitative Results

1

1

2

3

G

2

3

4

1

1

2

3

4

G

2

3

4

1

1

2

3

4

G

2

3

4

1

1

2

3

4

G

2

3

4

4

1

1

Attacker

Ball Direction

Player Direction

1

3

5

4

7

6

8

2

6

7

8

2

3

4

5

G

1

9

9

3

5

4

7

6

8

2

6

7

8

2

3

4

5

G

1

9

9

1

1

3

5

4

7

6

8

2

6

7

8

2

3

4

5

G

1

9

9

1

3

5

4

7

6

8

2

6

7

8

2

3

4

5

G

1

9

9

iteration 2

iteration 3

iteration 4

iteration 1

1

1

Attacker

Ball Direction

Player Direction

Waiting

1.0

Rate

Win I

0.5

0.0

1

3

4

Environment Frarmes (1.0e10)

Weshow additional visualization results in Fig. 7, Fig. 8, and Fig. 9. Corresponding GIF visualizations can be found on our project website.

B.2 Task Performance Evaluation

The evaluation win rates of the demonstrated visualization results in SMAC and GRF are shown in Table 8. Evaluated episode returns in Humanoid are shown in Table 9.

We also present the diversity score and average rewards achieved by baselines in Table 10. These numerical values are averaged across the entire population for a clear comparison. The tabulated data highlights the varying trade-offs between task performance and diversity exhibited by different

algorithms. It is noteworthy that SIPO, in particular, displays an adeptness at training a notably more diverse population while upholding a reasonably moderate level of task performance.

B.3 Evaluation Metric and Protocol for Diversity

B.3.1 Humanoid

The Humanoid locomotion task is well-studied in the Quality-Diversity (QD) community, enabling the application of well-defined behavior descriptors (BD) to assess diversity scores. While domainagnostic metrics like DvD scores can also be applied, we consider domain-specific BDs to be more appropriate and accurate for evaluation in this setting.

B.3.2 SMAC

Complex multi-agent tasks like SMAC lack well-defined BDs. Hence, domain-agnostic diversity measures such as the state-entropy measure should be applied. Moreover, different SMAC winning strategies tend to visit different areas of the map, which can be usually captured by the state-entropy measure.

Iteration 1

Iteration 3

Iteration 2

Iteration 4

B.3.3 GRF

In our initial study of the GRF task, diversity was evaluated using the k -nearest-neighbor state entropy estimation as in SMAC (see Table 7). However, we observed a significant difference between the computed scores and visualized behaviors. Further investigation revealed that state entropy can sometimes report fake diversity in GRF. For example, the ball-moving route is highly fine-grained between nearby players in the counter-attack (CA) scenario, and additional passes may not change the state entropy significantly. Instead, agents' positions play a crucial role in this scenario, where different shooting positions can introduce substantial state variance and lead to a higher entropy score. As an example, readers can refer to the replays of SIPO-RBF (4 iterations of seed 2) and PG (seed 2, 1002, 2002, and 3002), where SIPO-RBF discovers four distinct passing strategies, while PG keeps passing the ball to the same player. Nevertheless, the state entropy of PG (0.0397) is higher than that of SIPO-RBF (0.0378).

Hence, we counted the number of distinct policies according to their ball-passing routes, such as passing the ball to different players or shooting with different players, to evaluate diversity in GRF. To quantify these differences, we extracted the positions of the ball and the players in the field and calculated the nearest ally player ID to the ball across a winning episode. We then removed timesteps where the nearest distance was above a pre-defined threshold of 0 . 03 . Typically, these timesteps correspond to instances when the ball is being transferred among players, making the nearest player ID irrelevant. Next, we removed consecutive duplicate player IDs from the resulting sequence to obtain a concise and informative embedding of the ball passing route. By comparing the lengths of their respective embeddings and verifying that the player IDs in each embedding are identical, we determined whether the two policies exhibit similar behavior.

We acknowledge that existing diversity measures may not be applicable in GRF, and hence we opted for this novel approach to evaluate diversity. Additionally, we experimented with using raw observations, which include ball ownership information provided by the game engine, but found it to be highly inaccurate based on our visualization.

B.4 Additional Ablation Studies

B.4.1 Input to the Diversity Measure

Vectorized States in Google Research Football We perform an additional ablation study over the input of our diversity measure in GRF 3v1 scenario with SIPO-WD. We consider the following kinds of state input besides the default state input we adopted in Sec. 6:

full observation (named full , 115 dims);

default state input with random noises of the same dimension (named random , 36 dims).

The numbers of visually distinct strategies are listed in Table 11. The performance of full and random is similarly good. The result implies that the learnable discriminator can automatically filter out irrelevant states to some extent, and that SIPO-WD performs relatively robust w.r.t. different state input of the diversity measure.

RGB Images in Locomotion Tasks We run SIPO-WD in the visual Humanoid task based on Isaac Gym [42]. The training protocol is similar to the state-only version (i.e., the input of policy and intrinsic rewards are both locomotion states of the Humanoid) except that we stack recent 4 RGB camera observations ( 84 × 84 ) as the input of intrinsic rewards in Eq. 9. We adopt the training code developed in Isaac Gym and the default PPO configuration. The backbone of the discriminator is composed of 4 convolutional layers with kernel size 3, stride 2, padding 1, and [16, 32, 32, 32] channels. Then the feature is passed to an MLP with 1 hidden layer and 256 hidden units. The activation function is leaky ReLU with slope 0 . 2 . We also compute the pairwise distance of joint torques as in the state-only version and show the result in Table 12. Visualizations are shown in Fig. 10. SIPO-WD can also learn meaningful diverse behaviors with RGB images as the state input thanks to the learnable Wasserstein discriminator. This implies that our algorithm can be naturally extended to high-dimensional states and incorporated with advances in representation learning, which may be a potential future direction.

B.4.2 Combining State- and Action-based Diversity Measures

Based on SIPO-RBF, we introduce additional action information by directly concatenating the global state, used for diversity calculation, with the one-hot encoded actions of all agents within the GRF domain. Table 13 presents the outcomes, indicating the number of policies obtained. For scenarios with a limited number of agents, the action-augmented variant demonstrates comparable performance. However, when the agent count increases (as evident in the 11-agent cases of CA and corner), the incorporation of actions can introduce misleading diversity, detracting from the authenticity of the outcomes.

B.5 How to Adjust Constraint-Related Hyperparameters

Three hyperparameters are essential in the implementation of the intrinsic reward r int: the threshold δ , the intrinsic reward scale factor α , and the variance factor σ in r RBF int . These parameters differ under different domains and must be adjusted individually. We find proper parameters by running two

ΓffΓEΓfiΓΛΓOΓHΓIΓWΓıΓOΓHΓJΓΛΓMΓXΓPΓSΓLΓQΓJ

ΓffΓGΓfiΓΛΓKΓDΓQΓGΓıΓEΓDΓOΓDΓQΓFΓHΓGΓΛΓPΓLΓQΓFΓLΓQΓJ

ΓffΓDΓfiΓΛΓΛΓQΓRΓUΓPΓDΓOΓΛΓUΓXΓQΓQΓLΓQΓJ

ΓffΓFΓfiΓΛΓΛΓUΓLΓJΓKΓWΓıΓOΓHΓJΓΛΓMΓXΓPΓSΓLΓQΓJ

iterations without constraints and get two similar policies π 0 and π 1 . We record r int during training π 1 and the trend is shown in Fig. 11. Not surprisingly, r int gradually decreases as training proceeds.

Threshold We set δ = c 1 D S ( π 0 , π 1 ) . We try several different c 1 ∈ { 1 , 1 . 2 , 1 . 4 , 1 . 6 , 1 . 8 , 2 . 0 } and find that c 1 = 1 . 2 or 1 . 4 are universal proper solutions for all the experimental environments.

Intrinsic Scale Factor We need to balance the intrinsic reward r int and the original reward J so that neither of the two rewards can dominate the training process. Empirically, the maximums of the two rewards should be in the same order of magnitude. i.e., max π J ( π ) = α × c 2 λ max δ , where c 2 = O (1) . When c 2 is too large, the new-trained policy π j will oscillate near the boundary of D ( π i , π j ) = δ for some pre-trained policy p i . Conversely, when c 2 is too small, the intrinsic reward r int cannot yield diverse strategies. In experiments, we set c 2 = 1 . 0 .

Variance Factor We sweep the variance factor across { 1 e -3 , 5 e -3 , 1 e -2 , 2 e -2 , 1 e -3 } by training π 1 and observe the trend of intrinsic rewards. We find the steepest trend and select the corresponding σ . Empirically, we find that our algorithm performs robustly well when σ 2 = 0 . 02 .

The δ and α of GRF and SMAC are listed in Table 14.

B.6 Computation of Action-Based Measures in the Grid-World Example

We consider the policies illustrated in Fig. 12. These policies are all optimal since these actions only include 'right' and 'down' and actions on non-visited states can be arbitrary. We only mark actions on states visited by any of these 3 policies and actions on other states can be considered the same.

B.6.1 Action-Distribution-Based Measures

Action-distribution-based diversity measures can be defined as

where ˜ D ( · , · ) : △×△→ R is a measure over action distributions and q : △ ( S ) is a state distribution. Here, we consider q to be the joint state distribution visited by π i and π j .

KL Divergence KL divergence is defined by

When π j ( a | s ) = 0 at any state s , KL divergence is + ∞ . Since the trajectories of these policies have disjoint states, D KL A ( π 1 , π 2 ) = D KL A ( π 1 , π 3 ) = + ∞ . Similar results can be obtained for cross-entropy.

JSD γ JSD γ was defined in [38] and we consider two special cases when γ = 0 and γ = 1 .

As illustrated by [38], JSD 0 measures the expected number of times two policies will 'disagree' by selecting different actions. On trajectories induced by π 1 and π 2 , there are 4 + 4 states that π 1 disagrees with π 2 ( π 1 and π 2 are symmetric) and D JSD 0 A ( π 1 , π 2 ) = 8 / 16 = 1 / 2 . Similarly, π 1 and π 3 only disagree at the initial state, therefore we have D JSD 0 A ( π 1 , π 3 ) = 2 / 16 = 1 / 8 .

JSD 1 is defined by

̸

Since each of the policies considered only induces a single trajectory and π i ( τ j ) = 0 ( i = j ) , we can easily compute

Wasserstein Distance Wasserstein distance or Earth Moving Distance (EMD) is 1 if two policies disagree on a state and 0 otherwise. Therefore, it equals to D JSD 0 A .

B.6.2 Action Norm

We embed the action 'right' as vector [1 , 0] since it increases the x-coordinate by 1 and the action 'down' as vector [0 , -1] since it decreases the y-coordinate by 1. This embedding can be naturally extended to a continuous action space with velocity actions. Following [52], we compute the action norm over a uniform distribution on states. We can see that there are 7 states where π 1 and π 2 perform differently and 1 state (the initial state) where π 1 and π 3 perform differently. Therefore, we can get D ( π 1 , π 2 ) = √ 7 and D ( π 1 , π 3 ) = 1 .

B.6.3 State-Distance-Based Measures

State L 2 Norm Similar to action L 2 norm, we concatenate the coordinates instead of actions as the embedding and compute the L 2 norm between embedding.

Wasserstein Distance Wasserstein distance is tractable in the grid-world example. We consider 7 states (except the initial and final states) in each trajectory and compute the pair-wise distance as matrix C . Then we solve the following linear programming

where ⊙ means element-wise multiplication, 1 is a all-one vector, a = [ 1 T , 0 T ] T and b = [ 0 T , 1 T ] T is the marginal state distribution of each policy.

C Environment Details

C.1 Details of the 2D Navigation Environment

The navigation environment has an agent circle with size a and 4 landmark circles with size b . We pre-specify a threshold c and constrain that the distance of final states reaching different landmarks must be larger than c . Correspondingly, landmark circles are randomly initialized by constraining the pairwise distance between centers to be larger than a threshold c +2( a + b ) such that the final-state constraint is valid. An episode ends if the agent touches any landmarks, i.e., the distance between the center of the agent and the center of the landmark d < a + b , or 1000 timesteps have elapsed. The observation space includes the positions of the agent and all landmarks, which is a 10-dimensional vector. The action space is a 2-dimensional vector, which is the agent velocity. The time interval is set to be ∆ t = 0 . 1 , i.e., the next position is computed by x t +1 = x t +∆ t · v . The reward is 1 if the agent touches the landmark and 0 otherwise.

C.2 Details of Environments

We provide training configurations and environment introductions below and refer readers to our project website in App. A for visualizations of these environments.

Humanoid We use the Humanoid environment in IsaacGym [42] with default observation and action spaces. The input of intrinsic rewards or diversity measure is the observation without all torque states.

SMAC We adopt the SMAC environment in the MAPPO codebase 2 with the same configuration as Yu et al. [69]. The input of intrinsic rewards or diversity measure is the state of all allies, including positions, health, etc.

On the 'easy' map 2m_vs_1z , two marines must be controlled to defeat a Zealot. The marines can attack from a distance, while the Zealot's attacks are limited to close range. A successful strategy involves alternating the marines' attacks to distract the Zealot. On the 'hard' map 2c_vs_64zg , two colossi must be controlled by the agents to fight against 64 zergs. The colossi have a wider attack range and can move over cliffs. Strategies on this map may include hit-and-run tactics, waiting in corners, or dividing and conquering enemies. The level of difficulty is determined by the learning performance of existing MARL algorithms. Harder maps require more exploration and training steps.

GRF We adopt the 'simple115v2' representation as observation with both 'scoring' and 'checkpoint' reward. The reward is shared across all agents. The input of intrinsic rewards or diversity measure is the position and velocity of all attackers and the ball. All policies are trained to control the left team to score against built-in bots.

academy_3_vs_1_with_keeper : In this scenario, a team of three players (left) tries to score a goal against a single defender and a goalkeeper. The left team starts with the ball and has to dribble past the defender and the goalkeeper to score a goal.

academy_counterattack_easy : In this scenario, the left team starts with the ball in the front yard and tries to score a goal against several defenders. All eleven players in the left players can be controlled.

academy_corner : In this scenario, the left team tries to score a goal from a corner kick. The right team defends the goal and tries to prevent the left team from scoring. All eleven players in the left players can be controlled.

D Implementation Details

D.1 2D Navigation

We apply PPO with Lagrange multipliers to optimize the policy and hyperparameters are summarized in Table 15. D ( π i , π j ) is simply taken as the L 2 distance of the final state reached by π i and π j , i.e., D ( π i , π j ) = ∥ s π i H -s π j H ∥ 2 . The applied algorithm is the same as SIPO (see Appendix G) except that the intrinsic reward is only computed at the last timestep.

D.2 SIPO

In the i -th iteration ( 1 ≤ i ≤ M ), we learn an actor and a critic with i separate value heads to accurately predict different return terms, including i -1 intrinsic returns for the diversity constraints and the environment reward. We include all practical tricks mentioned in [69] because we find them all critical to algorithm performance. We use separate actor and critic networks, both with hidden size 64 and a GRU layer with hidden size 64. The common hyperparameters for SIPO, baselines, and ablations are listed in Table 16. Other environment-specific parameters, such as PPO epochs and mini-batch size, are all the same as [69]. Besides, Table 14 and Table 17 lists some extra hyperparameters for SIPO.

D.3 Baselines

We re-implement all baselines with PPO based on the MAPPO [69] project. All algorithms run for the same number of environment frames. Specific hyperparameters for baselines can be found in Appendix D.3.

SMERL SMERL trains a latent-conditioned policy that can robustly adapt to new scenarios. It promotes diversity by maximizing the mutual information between states and the latent variable. We implement SMERL with PPO, where the actor and the critic take as the input the concatenation of observation and a one-hot latent variable. The discriminator is a 2-layer feed-forward network with 64 hidden units. The learning rate of the discriminator is the same as the learning rate of the critic network. The input of the discriminator is the same as the input we use for SIPO-WD. The critic has 2 value heads for an accurate estimation of intrinsic return. Since SMERL trains a

single latent-conditioned policy, we train SMERL for M × more environment steps, such that total environment frames are the same. The scaling factor of intrinsic rewards is 0 . 1 and the threshold for diversification is [0 . 81 , 0 . 45 , 0 . 72] ( 0 . 9 × [0 . 9 , 0 . 5 , 0 . 8] ) for '3v1', 'counterattack', and 'corner' respectively.

DvD DvD simultaneously trains a population of policies to maximize the determinant of a kernel matrix based on action difference. We concatenate the one-hot actions along a trajectory as the behavioral embedding. The square of the variance factor, i.e., σ 2 in the RBF kernel, is set to be the length of behavioral embedding. We also use the same Bayesian bandits as proposed in the original paper. Training DvD in 'counterattack' and 'corner' exceeds the GPU memory and we exclude the results in the main body.

DIPG DIPG iteratively maximizes the maximum mean discrepancy (MMD) distance between the state distribution of the current policy and previously discovered policies. For DIPG, we follow the open-source implementation 3 . We set the same variance factor in the RBF kernel as SIPO-RBF and apply the same state as the input of the RBF kernel. We sweep the coefficient of MMD loss among { 0 . 1 , 0 . 5 , 0 . 9 } and find 0 . 1 the most appropriate (larger value will cause training instability). We use the same method to save archived trajectories as SIPO and the input of the RBF kernel is the same as the input we use for SIPO-RBF. To improve training efficiency, we only back-propagate the MMD loss at the first PPO epoch.

RSPO RSPO iteratively discovers diverse policies by optimizing extrinsic rewards on novel trajectories while optimizing diversity on other trajectories. The diversity measure is defined as the action-cross entropy along the trajectory. For RSPO, we follow the opensource implementation 4 and use the same hyperparameters on the SMAC 2c_vs_64zg map in the original paper for GRF experiments.

TrajDi TrajDi was originally designed for cooperative multi-agent domains to facilitate zeroshot coordination. It defines a generalized Jensen-Shanon divergence objective between policy action distributions. Then this objective and rewards are simultaneously optimized via populationbased training. We tried TrajDi in SMAC and GRF. We sweep the action discount factor among { 0 . 1 , 0 . 5 , 0 . 9 } and the coefficient of TrajDi loss among { 0 . 1 , 0 . 01 , 0 . 001 } . However, TrajDi fails to converge in the '3v1' scenario and exceeds the GPU memory in the 'counterattack' and 'corner' scenarios. Therefore, we exclude the performance of TrajDi in the main body.

Domino We have meticulously re-implemented Domino within our codebase according to the appendix of Zahavy et al. [71]. We execute the algorithm in the Humanoid locomotion task, employing the robot state (excluding torques) for successor feature computation. Despite our earnest efforts to optimize Domino's performance, our findings reveal its comparable performance to SMERL, illustrated by a minimal diversity score of 0.01. Therefore, we exclude the performance of Domino in the main body.

APT APT maximized the nearest neighbor state-entropy estimation for skill discovery. While we also adopted this metric for diversity evaluation, there is a fundamental distinction in formulation. APT optimizes state entropy within a single policy, whereas our method, SIPO, targets the joint entropy of a population of policies. It is okay for each single policy within the population to have low state entropy. To employ APT's objective of discovering diverse policies, training a population of agents concurrently is required. The algorithm should optimize the estimated entropy over states visited by all policies. Yet, this approach mandates large-scale k-NN computation (k=12) over substantial batches, leading to significant computational inefficiency. Despite our dedicated efforts, we didn't finish a single training trial of APT within 48 hours (in contrast to other PBT baselines, e.g. DvD, which completes training in less than 8 hours).

D.4 Ablation Study Details

For the three ablation studies: fix-L, CE, and filter, we list the specific hyperparameters here:

fix-L: we set the Lagrange multiplier to be 0 . 2 ;

CE: the threshold is 3 . 800 and the intrinsic reward scale factor is 1 / 1000 of that in the WD setting;

filter: all the hyperparameters in the setting are the same as those in the WD setting.

E Proofs

E.1 Proof of theorem 4.1

Theorem 4.1. Assume D is a distance metric. Denote the optimal value of Problem 3 as T 1 . Let T 2 = ∑ M i =1 J (˜ π i ) where

for i = 1 , . . . , M , then T 2 ≥ T 1 .

Proof. Suppose the optimal solution of Problem 3 is π 1 , π 2 , ..., π M satisfying J ( π 1 ) ≥ J ( π 2 ) ≥ ... ≥ J ( π M ) and the optimal solution of Problem 6 is ˜ π 1 , ˜ π 2 , ..., ˜ π M satisfying J (˜ π 1 ) ≥ J (˜ π 2 ) ≥ ... ≥ J (˜ π M ) .

Assume the contrary that Thm. 4.1 is not true, which means ∑ M i =1 J ( π i ) = T 1 > T 2 = ∑ M i =1 J (˜ π i ) . Then we choose the smallest number N ≤ M that satisfies

By T 1 > T 2 we know that N exists. In addition, because Problem 6 solves unconstrained RL in the first iteration, we know that ˜ π 1 = arg max π J ( π ) and then J ( π 1 ) ≤ J (˜ π 1 ) . Therefore, N ≥ 2 .

Suppose J ( π N ) ≤ J (˜ π N ) . Then we have

Contradicting the fact that N is the smallest number satisfies that equation.

Hence, we know that J ( π N ) > J (˜ π N ) . Then

Consider the optimization problem of ˜ π N :

This optimization does not find { π 1 , . . . , π N } but find ˜ π N , which means that for each π i , 1 ≤ i ≤ N , there exists 1 ≤ j i < N such that D ( π i , ˜ π j i ) < δ/ 2 . Otherwise, we will get the solution of the above problem as π i instead of ˜ π N .

̸

By the Pigeonhole Principle, we know that there exist two indexes i 1 ∈ [ N ] and i 2 ∈ [ N ] ( i 1 = i 2 ) such that j i 1 = j i 2 = ˆ j . Then we have

where the inequality follows by the triangle inequality of the distance function.

It contradict with the fact that D ( π i 1 , π i 2 ) ≥ δ in Problem 3.

Therefore, we prove the theorem ∑ M i =1 J ( π i ) = T 1 ≤ T 2 = ∑ M i =1 J (˜ π i ) .

E.2 Proof of Theorem 5.1

In this section, we consider the i -th iteration of SIPO illustrated in Eq. (4). For the sake of simplicity, we use a ≤ λ ≤ b for vector λ to denote each component of λ satisfies a ≤ λ i ≤ b , where a, b ∈ R . We use π to denote the policy we are optimizing, and π j (1 ≤ j < i ) to denote a previously obtained policy. We denote the Lagrange function as L ( π, λ ) = -J ( π ) -∑ i -1 j =1 λ j ( D ( π, π j ) -δ ) .

To prove Theorem 5.1, we consider the following two optimization problems:

and where Λ = 1 ϵ 0 and ϵ 0 > 0 is sufficiently small.

We make the following assumptions to prove this theorem:

Assumption E.2. ∀ λ ≥ 0 , L ( · , λ ) is l -smooth and ζ -Lipschitz.

We may notice that solving the optimization problem (11) is hard because its domain is unbounded. Therefore, we make some approximations and consider the bounded optimization problem (12). First, we prove the following lemma about the value function J :

Lemma E.3. J ( π i ) ≤ J (˜ π i ) .

Proof. As the domain of λ in Eq. 12 is smaller than Eq. (11), we have L ( π i , λ ) ≥ L (˜ π i , ˜ λ ) .

By the fundamental property of Lagrange duality, we know that L achieves its optimal value when λ = 0 and the optimal value is -J ( π i ) .

By the optimality of (˜ π i , ˜ λ ⋆ ) , we know that

Then we have

Then we prove the distance between optimal policy ˜ π i in problem (12) and optimal policy π i in problem (11) is very small:

Lemma E.4. Under Assumption E.1, D (˜ π i , π j ) ≥ δ -ϵ 0 , ∀ 1 ≤ j < i .

Proof. We prove this by contradiction.

Suppose there exists 1 ≤ j 0 < i , D (˜ π i , π j 0 ) < δ -ϵ 0 . Then we choose ˆ λ such that

̸

By the Assumption E.1, Eq. (13), and Λ = 1 ϵ 0 , we have

That is a contradiction. So we have proved that

From the deduction above, we get the following approximation lemma:

Lemma E.5. Denote the optimal solution of Eq. 11 and Eq. 12 as ( π i , λ ) and (˜ π i , ˜ λ ) respectively. Then we have the following approximation about the optimal value and distance:

Proof. This lemma follows directly by Lemma E.3 and Lemma E.4.

Therefore, it is reasonable to consider the constrained optimization problem (12) instead of primal problem (11) because we have proved that the optimal value doesn't get smaller and the distance of policy is ϵ 0 -approximation of the primal problem. Finally we use the conclusion in the paper [31] to analysis the convergence of problem (12):

Lemma E.6. ([31], Theorem 4.8) Under Assumption E.2, solving Eq. (12) via two-timescale GDA with learning rate η π = Θ( ϵ 4 /l 3 ζ 2 Λ 2 ) and η λ = Θ(1 /l ) requires

iterations to converge to an ϵ -stationary point π ⋆ i , where C 1 and C 2 are the constants that depend on the distance between the initial point and the optimal point.

Theorem 5.1. Under assumptions E.1 and E.2 and learning rate with learning rate η π = Θ( ϵ 4 /l 3 ζ 2 Λ 2 ) and η λ = Θ(1 /l ) , SIPO converges to an ϵ -stationary point with convergence rate O ( l 3 ζ 2 Λ 2 C 1 ϵ 6 + l 3 Λ 2 C 2 ϵ 4 ) .

Proof. We consider the following constraint nonconvex-concave optimization:

Following Lemma E.6, we know that the Two-Timescale GDA algorithm converges to an ϵ -stationary point π ∗ i .

From the above deduction, the Two-Timescale GDA algorithm requires O ( l 3 ζ 2 Λ 2 C 1 ϵ 6 + l 3 Λ 2 C 2 ϵ 4 ) iterations with learning rate η π = Θ( ϵ 4 /l 3 ζ 2 Λ 2 ) and η λ = Θ(1 /l ) to converge to an ϵ -stationary point with convergence rate.

F Discussion

F.1 The Failure Case of State-Distance-Based Diversity Measures

A failure case of state-distance-based diversity measures may be when the state space includes many irrelevant features . These features cannot reflect behavioral differences. If we run SIPO in such an environment, the learned strategies may be only diverse w.r.t these features and have little visual distinction. Like the famous noisy TV problem [5], the issue of irrelevant features is intrinsically challenging for general RL applications, which cannot be resolved by using action-based or state-occupancy-based diversity measures either.

Thanks to the advantages we discussed in the paper, we generally find that state-distance-based measures can be preferred in challenging RL problems. Meanwhile, since the state dimension can be much higher than actions, it is possible that RL optimization over states may be accordingly more difficult than actions. In practice, we can design a feature selector for those most relevant features for visual diversity and run diversity learning over the filtered features. In SMAC and GRF, we utilize the agent features (excluding enemies) as the input of diversity constraint without further modifications, as discussed in Appendix D. We remark that even after filtering, the agent features remain high-dimensional while our algorithm still works well. Note that using a feature selector is a common practice in many existing domains, such as novelty search [12], exploration [33], and curriculum learning [7]. There are also works studying how to extract useful low-dimensional features from observations [68, 18], which are orthogonal to our focus.

F.2 The Distance Metric

In Sec. 5, we adopt the two most popular implementations in the machine learning literature, i.e., RBF kernel and Wasserstein distance, while it is totally fine to adopt alternative implementations. For example, we can learn state representations (e.g. auto-encoder, Laplacian, or successor feature) and utilize pair-wise distance or norms as a diversity measure. Similar topics have been extensively discussed in the exploration literature [68, 41]. We leave them as our future directions.

G Pseudocode of SIPO

The pseudocode of SIPO is shown in Algorithm 1.

Algorithm 1 SIPO (red for SIPO-RBF and blue for SIPO-WD)

,

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nReinforcement Learning (RL) algorithms commonly learn a distinct policy that is responsible for a distinct behavior. Learning different, diverse behaviors generally is a difficult task in RL. This paper proposes a new algorithm State-based Intrinsic-reward Policy Optimization (SIPO) that can learn diverse, human interpretable behaviors.\nThe paper first discusses some of the current diversity measures, and based on the insight, it proposes the new ITR-based method, even though PBT is the choice when it comes to learning diverse behaviors. The method is efficient and can learn diverse behaviors. The effectiveness is demonstrated in simulated environments.\n\nSTRENGTHS:\n- well written. Motivation and contributions made clear\n- the paper covers a good amount of related work, even though there are a few missing, in my opinion (see Weaknesses).\n- Nice evaluations w.r.t. diversity\n\nWEAKNESSES:\n- The big field of unsupervised reinforcement learning also covers a lot of similarity/dissimilarity measures for formulating an intrinsic reward. Even though some of the works are mentioned (e.g. DIAYN), there are several other works that should be mentioned as they are using different objectives and hence differently measure the similarity/dissimilarity:\n\t- https://arxiv.org/pdf/1906.05274.pdf\n\t- http://proceedings.mlr.press/v139/yarats21a/yarats21a.pdf\n\nDuring a Google search, I discovered another work that also seems to have an interesting objective to learn different solutions:\n\t- https://proceedings.mlr.press/v164/celik22a/celik22a.pdf\n\n- weak motivation for the necessity of the state distance (see Questions)\n\nQUESTIONS:\n- The motivation behind Figure 2 is unclear to me even when reading the text. I think this figure definitely needs more description and explanation. There are also no numbers indicating what the measured distances are. \n\n- Different works have approximated the state entropy for discovering new skills. E.g. the APT [1]. It would be interesting to see how the used diversity measures compare against it. Can the authors assess, how different the solutions would be when using the approximated state entropy compared to the proposed metrics? \n\n- The humanoid locomotion task in Section 6.1 is used to compare the diversity of learned policies. How do the methods compare w.r.t. performance?  \n\n[1]: https://openreview.net/forum?id=fIn4wLS2XzU\n\nLIMITATIONS:\nThe paper explicitly addresses limitations but doesn't state anything about potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of learning diverse policies for a given RL task. First, it studies pros and cons of various formulations in two different dimensions: (i) How to measure diversity, for which it considers measures based on action or state distribution and state distances, (ii) how to compute the diverse policies, either with a joint method or an iterative one. Then, it combines the findings into a method, called SIPO, which implements iterative learning through gradient descent ascent with state distance incentives. Finally, it empirically evaluates SIPO on a variety of domains, including locomotion tasks, Star-Craft, and Google Research Football.\n\nSTRENGTHS:\n- Relevance of the problem. Several recent works have addressed the problem of learning diverse policies in RL, in both supervised and unsupervised settings;\n- Methodology. Interesting and reasonably efficient methodology that is based on a new state-metric perspective on the problem of learning diverse policies;\n- Experimental results. Promising empirical results on challenging domains.\n\nWEAKNESSES:\n- Motivation. The introduced objective lacks a strong theoretical ground that is mostly present in previous works with alternative diversity objectives;\n- Related works. The comparison between SIPO and some relevant related works, especially Zahavy et al., 2022, is not really upfront;\n- Experiments robustness and significance. The experiments consider arbitrary performance metric and average results over 5 or 3 seeds, which is hardly enough to get statistical significance.\n\nQUESTIONS:\nI have one crucial concern about this paper, which is otherwise sound and interesting. The superiority of diversity measures based on state distances is only informally motivated, through examples rather than theoretical justification. Instead, the diversity measures based on action distribution or state occupancy have been both linked to important theoretical properties, such as reward robustness for the former (e.g., Husain et al., Regularized policies are reward robust, 2021) or bounded sub-optimality for the latter (e.g., Kakade and Langford, Approximately optimal approximate reinforcement learning, 2022). Diversity based on state distances currently lacks similar theoretical ground, which also leaves an existential question on the motivation of the paper? Why we want to learn diverse strategies in the first place? Is it to be robust to reward perturbations? Is to learn reusable skills? Is it something else?\n\nThe concern reported above somewhat propagates to the experimental evaluation. If the ultimate purpose of the training is not clear, how can we evaluate one method against another? Indeed, the experiments are based on arbitrary diversity scores. SIPO may be better than baselines on those, but it is hard to reach some conclusion beyond qualitative justifications. Perhaps a good option would be to evaluate SIPO also on more consequential benchmarks, such as testing the reward robustness of the trained set of policies, or showing that the latter is a good starting point for fine-tuning to a different task.\n\nFor this main reason, I am currently providing a borderline evaluation, while I am open to change my score if the authors can provide more formal justification for their objective function in the rebuttal.\n\nI provide below additional comments on the paper.\n\n**Objective function**\n\nOne key aspect that Section 4.1 seems to overlook is that all of the provided motivations (especially in lines 158-169) make sense only when the environment's states lie on a metric space. In several domains, such as tabular MDPs with symbolic state representations or vision-based tasks, defining a proper metric over the states might be even more challenging than learning diverse policies alone. I think this limitation should be explicitly reported everywhere in the paper, including the introduction and abstract.\n\nThe motivation reported in lines 170-173 appears to be somewhat weak too, at least in domains where external rewards are present, and the concept of idle actions may be incorporated in the rewards.\n\nCan the authors clarify the role of the cost function $g$ into the diversity score reported in Eq. 5?\n\n**Experiments**\n\nWhy are the authors reporting experiments in multi-agent domains, even if they present a method for learning diverse policies in a single-agent setting? This choice looks somewhat odd, and it shall be motivated.\n\nI am worried about the statistical significance of the reported performance results, since most of them are obtained as average over 5 or 3 seeds. While I can understand that running experiments in such complex domains is costly, the standard to claim experimental robustness has increased lately. Moreover, the results report average and standard deviation. On the one hand, confidence intervals would be more meaningful than standard deviation, on the other hand only it is not enough to have best average performance to say that one method outperforms another when the intervals are overlapping.\n\n**Related Works**\n\nThis work looks in a way orthogonal to Zahavy et al. (2022), in which the authors present a method to learn diverse near-optimal policies by maximizing diversity with an hard reward constraint. While their concept of diversity is different, it would be interesting to discuss pros and cons of the two alternative solutions in depth, and perhaps compare SIPO with Domino in the experimental campaign.\n\nAnother work that is somewhat relevant to the topic is (Mutti et al., Reward-free policy space compression for reinforcement learning). Their aim is to learn a diverse set of policies from which any reward function can be approximately optimized, and they also present a gradient descent ascent procedure for this purpose. The fact that their work does not account for rewards is an important difference, but the authors can perhaps consider discussing this work in the paper as well. \n\nOne key benefit that I see in the SIPO solution w.r.t. prior works, is that SIPO shall enjoy favorable computational complexity, whereas both Zahavy et al. and Mutti et al. need to solve a non-convex non-concave optimization problem.\n\n**Minor**\n\n- The preliminaries are framing the problem as POMDP. I do not understand why partial observability is introduced, and then never used in the paper at all.\n- Figure 4, IL is instead ITR?\n- Theorem 4 says that ITR can achieve the same rewards as PBT, but only with an inferior diversity. This trade-off between computation and diversity could be better highlighted.\n- One of the evaluation metric is the entropy estimator via k-NN, for which (Liu & Abbeel, 2021) is mentioned. Note that those entropy estimators have been presented before (e.g., Singh et al., Nearest neighbor estimates of entropy, 2001). Moreover, if the goal is to maximize the state entropy, why not using APT instead of SIPO?\n\nLIMITATIONS:\nThe limitations are reported in the final sentences of the paper. Perhaps, some limiting aspects could be discussed in additional length.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the challenge of optimizing rewards while discovering diverse strategies in complex reinforcement learning problems. This paper examines two design choices for tackling this challenge: diversity measure and computation framework. By incorporating state-space distance information into the diversity measure, the behavioral differences between policies are accurately captured. Besides, two common computation frameworks: population-based training (PBT) and iterative learning (ITR) are compared, and it shows that ITR can achieve comparable diversity scores with higher computation efficiency. Based on above analysis, a novel diversity-driven RL algorithm named State-based Intrinsic-reward Policy Optimization (SIPO) is proposed. The authors evaluate SIPO across three environments and demonstrate that it consistently produces diverse policies that cannot be discovered by existing baselines.\n\nSTRENGTHS:\nThis paper introduces a state-based population diversity measurement and transfers it into a shaping reward in practice for computational conveniency. By comparison with PBT and ITR, it is concluded that ITR is able to achieve higher performance. Additionally, experiment environments including single-agent reinforcement learning and multi-agent reinforcement learning all shows the effectiveness of the proposed method. In a way, this paper is well-written and presents some contributions to the field of reinforcement learning. The authors provide clear explanations of their methodology and experiment results, making the paper easy to track.\n\nWEAKNESSES:\nIn this paper, the proposed state-based diversity measurement is one of key innovations. By combining with ITR framework, it achieves satisfactory performance. However, there still exist some weaknesses remained to be polished up.\n\n(1) The comprehensiveness of literature reviews remains to be improved. \n\n(2) The motivating example in Sec4.1 shows the limitation of previous action-based diversity measurement. However, the reason why state-distance based measurement can overcome the problem is not explained adequately.\n\n(3) The experiment part seems to be insufficient. Some SOTA diversity enhancement methods in RL and MARL are not analyzed or summarized. Besides, the assessment criteria of diversity seems unfair. \n\n(4) It claims that the heatmaps of agent positions in SMAC show obvious strategic differences. However, all four policies seem to explore similar areas in the map.\n\n(5) While the main body of the paper is well-written, there is space for improvement. I defer some of my issues in the appendix to \"Questions\".\n\nQUESTIONS:\nQ1: Would you improve the literature reviews from the aspect of adding some well-known or SOTA related works (e.g. [1] [2] [3] [4] [5])?\n\nQ2: Why can state-distance based measurement overcome the shortcomings of action-based measurements? It is suggested to make more discussion theoretically.\n\nQ3: Can SIPO be extended to algorithms in two-player zero-sum game scenario, like self-play, PSRO?\n\nQ4: What will be like if combining state-based diversity measurement and action-based diversity measurement? It is suggested to add an study in experiment part.\n\nQ5: Since the number of population size is fixed in PBT or ITR, the determination of it is still an open question as we known. Would you add a theoretical or empirical analysis on it?\n\nQ6: Comparison criteria of diversity seems unfair to baselines in Humanoid Locomotion and SMAC, since all baselines’ optimization objective are action-based. An intuitive evaluation criteria is the quantifiable behavioral difference between agents. Could you make more analysis on it?\n\nQ7: As can be seen, the heatmaps in Fig7 seems to show that the positions explored by the four agents are not significantly different. Would you add additional explanations on it?\n\nQ8: Minor: abbreviation in Fig4 (i.e., “IL”) is not been explained.\n\n[1] Liu, Z., Yu, C., Yang, Y., Wu, Z., & Li, Y. (2022). A Unified Diversity Measure for Multiagent Reinforcement Learning. Advances in Neural Information Processing Systems, 35, 10339-10352.\n\n[2] Balduzzi, D., Garnelo, M., Bachrach, Y., Czarnecki, W., Perolat, J., Jaderberg, M., & Graepel, T. (2019, May). Open-ended learning in symmetric zero-sum games. In International Conference on Machine Learning (pp. 434-443). PMLR.\n\n[3] Hu, S., Xie, C., Liang, X., & Chang, X. (2022, June). Policy diagnosis via measuring role diversity in cooperative multi-agent rl. In International Conference on Machine Learning (pp. 9041-9071). PMLR.\n\n[4] Masood, M. A., & Doshi-Velez, F. (2019). Diversity-inducing policy gradient: Using maximum mean discrepancy to find a set of diverse policies. arXiv preprint arXiv:1906.00088.\n\n[5] Li, C., Wang, T., Wu, C., Zhao, Q., Yang, J., & Zhang, C. (2021). Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 3991-4002.\n\nLIMITATIONS:\nThe authors have concluded the limitation of the methodology from the aspect of state representation and acceleration problem.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a solution to the problem of finding diverse policies for complex (multi-agent) reinforcement learning (RL) environments. The paper is presented as a joint study on diversity metrics and learning frameworks. For the former, the authors show the limitations of common diversity metrics like action-distribution diversity and state-occupancy based metrics and argue for metrics that incorporate state-distances for training mutually-distinct policies instead. For the latter, population-based (PBT) and iterative training (ITR) approaches are considered, where PBT is presented as a more formally suitable framework, that in reality is limited by its pairwise diversity constraints. ITR as a reasonable relaxation of PBT, on the other hand, is found to be effective in combination with the author’s proposed (’meaningful’) state-distance diversity. The combination if ITR and the proposed metric is incorporated into the State-Based Intrinsic-Reward Policy Optimization (SIPO) algorithm and evaluated on the single-agent human locomotion domain, as well as the multi-agent SMAC and GRF environments. SIPO is shown to outperform related baselines and is presented to be more capable of producing visually distinct / humanly intuitive policies compared to its counterparts.\n\nSTRENGTHS:\n- The paper is well researched in related work. While the evaluated frameworks PBT and ITR or the concept of state-distance in itself are not novel (but well cited and explained), the combination of ITR with the proposed state-dissimilarity diversity -- realized either as RBF-Kernel or with Wasserstein-Distance — are novel as far as I can tell.\n- Both the study on the limitations and the evaluation are presented in-depth and read quite intuitive. I am not too familiar with formal proofs of convergence criteria but the technical aspects in the paper itself do appear to be sound. The evaluation is reasonably complex, covering both complex single-agent and two multi-agent domains against reasonable baseline models. \n- The paper is very nicely written and well organized. The logical structure of motivation, related work, background, analytic study of the frameworks in combination with the metric and the presentation of the SIPO algorithm and its evaluation is easy to follow and intuitively explained. Small examples to illustrate the argumentation help lighten an otherwise densely formulated reasoning. I also find the visualizations to be nicely done.\n- The author’s SIPO approach seems to produce some very diverse strategies, which also appear intuitively interpretable. While diversity in itself is very important for exploration and an important Problem to solve for RL in general, I find the convergence towards interpretable strategies to be very intriguing. The paper is not only nicely understandable and informative on the matter of diversity related approaches, the evaluation is rather strong and the supplementary material has enough content for another paper by itself. I have no real complaints about this work, very nicely done.\n\nWEAKNESSES:\n- Besides providing a reasonably complex evaluation, there could have been more than 5 seeds used for the evaluation.\n- I would also like to see at least some classic total-reward metrics / comparisons for these domains, as the metrics chosen here to highlight the performances of the SIPO algorithm (pairwise distances, est. state entropy, agent position distributions, nr. distinct strategies) seem a bit cherry-picked to argue in favor of diversity only. While there are some relevant plots in the Appendix (and some of the toy-examples reason with rewards), I would encourage to showcase at last one such classic performance comparison against the baselines in the main-paper itself.\n\nQUESTIONS:\n- In 4.2 it is mentioned that PBT only converges when the exploration is faint. Is this claim shown somewhere or is this based on empirical evidence?\n- Could you provide some motivation on why you are casting $D_s(\\pi_i, \\pi_j^*)$ as intrinsic reward, and why RBF Kernel and the Wasserstein Distance were chosen for the diversity? Why does L_2 WSD provide ‘stronger discriminative power’(l263)?\n- Increasingly more MARL approaches start to require a DEC-POMDP for independent / localized actions without a fully observable state. How would you judge the SIPO’s transferability to such a formalization?\n\nLIMITATIONS:\nThere are two larger limitations mentioned, for once the assumption of continuity for convergence and the access to an object-centric state-representation. Both are openly disclosed and discussed in the paper. ‘The acceleration of ITR remains an open challenge’ is also a valid outlook, that should be addressed in the future.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}