Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality
Authors: Tom Zahavy
Yannick Schroecker
Feryal Behbahani
Kate Baumli
Sebastian Flennerhag
Shaobo Hou
Satinder Singh
Venue: ICLR 2023 poster (2023)

Abstract: In this work we propose a Reinforcement Learning (RL) agent that can discover complex behaviours in a rich environment with a simple reward function. We define diversity in terms of state-action occupancy measures, since policies with different occupancy measures visit different states on average. More importantly, defining diversity in this way allows us to derive an intrinsic reward function for maximizing the diversity directly. Our agent, DOMiNO, stands for Diversity Optimization Maintaining Near Optimally. It is based on maximizing a reward function with two components: the extrinsic reward and the diversity intrinsic reward, which are combined with Lagrange multipliers to balance the quality-diversity trade-off. Any RL algorithm can be used to maximize this reward and no other changes are needed. We demonstrate that given a simple reward functions in various control domains, like height (stand) and forward velocity (walk), DOMiNO discovers diverse and meaningful behaviours. We also perform extensive analysis of our approach, compare it with other multi-objective baselines, demonstrate that we can control both the quality and the diversity of the set via interpretable hyperparameters, and show that the set is robust to perturbations of the environment.

Published as a conference paper at ICLR 2023
DISCOVERING POLICIES WITH DOMINO: DIVERSITY OPTIMIZATION MAINTAINING NEAR OPTIMALITY
Tom Zahavy, Yannick Schroecker, Feryal Behbahani, Kate Baumli, Sebastian Flennerhag, Shaobo Hou and Satinder Singh DeepMind, London
ABSTRACT
In this work we propose a Reinforcement Learning (RL) agent that can discover complex behaviours in rich environment with simple reward function. We define diversity in terms of state-action occupancy measures, since policies with different occupancy measures visit different states on average. More importantly, defining diversity in this way allows us to derive an intrinsic reward function for maximizing the diversity directly. Our agent, DOMiNO, stands for Diversity Optimization Maintaining Near Optimally. It is based on maximizing reward function with two components: the extrinsic reward and the diversity intrinsic reward, which are combined with Lagrange multipliers to balance the quality-diversity trade-off. Any RI. algorithm can be used to maximize this reward and no other changes are needed. We demonstrate that given a simple reward functions in various control domains, like height (stand) and forward velocity (walk), DOMINO discovers diverse and meaningful behaviours. We also perform extensive analysis of our approach, compare it with other multi-objective baselines, demonstrate that we can control both the quality and the diversity of the set via interpretable hyperparameters, and show that the set is robust to perturbations of the environment.
INTRODUCTION
As we make progress in Artificial Intelligence, our agents get to interact with richer and richer environments. This means that we cannot expect such agents to come to fully understand and control all of their environment. Nevertheless, given an environment that rich enough, we would like to build agents that are able to discover complex behaviours even if they are only provided with simple reward function. Once a reward is specified, most existing RL algorithms will focus on finding the single best policy for maximizing it. However, when the environment is rich enough, there may be many qualitatively (optimal or near-optimal) different policies for maximising the reward, even if is simple. Finding such diverse of policies may help an RL agent to become more robust to changes, to construct a basis of behaviours, and to generalize better to future tasks.
Our focus in this work is agents that find creative and new ways maximize the reward, which is closely related to Creative Problem Solving (Osborn, 1953): the mental process searching for an original and previously unknown solution a problem. Previous work on this topic has been done in the field of Quality-Diversing (QD), which comprises of two main families of algorithms: MAP-Elites (Mouret & Clune, 2015; Cully al., 2015) and novelty search with local competition (Lehman & Stanley, 2011). Other work has been done in the RL community and typically involves combining the extrinsic reward with diversity intrinsic reward (Gregor et al., 2017: Eysenbach al., 2019). Our work shares similar objective to these excellent previous work, and proposes new class of algorithms as we will soon explain. Due space considerations we further discuss the connections to related work in Appendix A.
Our main contribution is a new framework for maximizing the diversity of RL policies in the space of state-action occupancy measures. Intuitively speaking, state-action occupancy dx measures how often a policy π visits each state-action pair. Thus, policies with diverse state-action occupancies induce different trajectories. Moreover, for such diverse set there exist rewards (down stream tasks) for which some policies are better than others (since the value function is a linear product between the state occupancy and the reward). But most importantly, defining diversity in terms of

---

Published as a conference paper at ICLR 2023
state occupancies allows to duality tools from convex optimization and propose discovery algorithm that is based, completely, on intrinsic reward maximization. Concretely, one can come up with different diversity objectives, and use the gradient of these objectives as intrinsic reward. Building on these results, we show how to use existing RI. code bases for diverse policy discovery. The only change needed is to to provide the agent with diversity objective and its gradient.
To demonstrate the effectiveness of our framework, we propose propose DOMINO, a method for Diversity Optimization that Maintains Nearly Optimal policies. DOMINO trains set of policies using a policy-specific, weighted combination of the extrinsic reward and an intrinsic diversity reward. The weights are adapted using Lagrange multipliers to guarantee that each policy near-optimal. We propose two novel diversity objectives: a repulsive pair-wise force that motivates policies to have distinct expected features and a Van Der Waals force, which combines the repulsive force with an attractive one and allows us to specify the degree of diversity in the set. We emphasize that our framework is more general, and we encourage others propose new diversity objectives and use their gradients as intrinsic rewards.
To demonstrate the effectiveness DOMiNO, we perform experiments the DeepMind Control Suite (Tassa et al., 2018) and show that given simple reward functions like height and forward velocity, DOMiNO discovers qualitatively diverse and complex locomotion behaviors (Fig. lb). We analyze our approach and compare it to other multi-objective strategies for handling the QD trade-off. Lastly, we demonstrate that the discovered set is robust to perturbations of the environment and the morphology of the avatar. We emphasize that the focus of experiments is validating and getting confidence in this new and exciting approach; we do not explicitly compare it with other work nor argue that one works better than the other.
Epdaa Tagrase
Dannig Avrrge
Ruminng foerage
MIP
(b)
(a)
Figure 1: (a) DOMINO's architecture: The agent learns a set of diverse high quality policies via a single latent-conditioned actor-critic network with intrinsic and extrinsic value heads. Dashed arrows signify training objectives. (b) DOMiNO's æ: Near optimal diverse policies in staner.stand corresponding to standing on both legs, standing on either leg, lifting the other leg forward and backward, spreading the legs and stamping. Not only are the policies different from each other, they also achieve high extrinsic reward in standing (see values top of each policy).
2 PRELIMINARIES AND NOTATION
In this work, we will express objectives in terms of the state occupancy measure dn Intuitively speaking, dz measures how often a policy æ visits each state-action pair. As we will soon see, the classic RL objective of reward maximization can be expressed as a linear product between the reward vector and the state occupancy. In addition, in this work we will formulate diversity maximization via an objective that is nonlinear function of the state occupancy. While it might seem unclear which reward should be maximized to solve such an objective, we take inspiration from Convex MDPs (Zahavy et al., 2021b) where one such reward is the gradient of the objective with respect d,.
We begin with some formal definitions. In RL an agent interacts with an environment and seeks to maximize its cumulative reward. We consider two cases, the average reward case and the discounted case. The Markov decision process (Puterman, 1984, MDP) is defined by the tuple (S, A, P. R) for the average reward case and by the tuple (S, A, P. R. y, Do) for the discounted case. We assume an infinite horizon, finite state-action problem. Initially, the state of the agent is sampled according 80 Do. At time I,, given state 8t, the agent selects action a according its policy x =8s::),
2

---

Published as a conference paper at ICLR 2023
receives reward r. R(st; at) and transitions to new state 81+1 P(., 0t). We consider two performance metrics, given by e0 = lim / =E==n" Ne. t = ( I for the average reward case and discounted case respectively. The goal of the agent is to find a policy that maximizes exx or vzz Let (* =
be the probability measure over states at time / under policy æ, then the state occupancy measure dx is given as ds, a) lim +EΣ (s. s) (s, a), and di (s.a) a) (1 >) > 2 >'P=(>> = (s, a), for the average reward case and the discounted case respectively. With these, we can rewrite the RL objective in as a linear function of the occupancy measure ea E., r(s, a)d, a), where K is the set of admissible distributions (Zahavy et al., 2021b). Next, consider objective of the form:
min f(dx), e,
(1)
where f K. + R is nonlinear function. Sequential decision making problems that take this form include Apprenticeship Learning (AL) and pure exploration, among others (see Appendix A). In the remainder of this section, we briefly explain how to solve Eq. (1) using RL methods when the function f is convex. We begin with rewriting Eq. (1) using Fenchel duality as
min f(dx) max min (A dx (2)) d,
(2)
AEA d, EK
where is the closure of (sub-)gradient space {0f(d))ld+ E KC), which is compact and convex (Abernethy & Wang, 2017), and f is the Fenchel conjugate of the function f.
Eq. (2) presents a zero-sum max-min game between two players, the policy player Alg, and the cost player Algx. We can see that from the perspective of the policy player, the objective is linear minimization problem in d, Thus, intuitively speaking, the goal of the policy player is to maximize the negative cost reward from (Abernethy& & Wang, 2017), which uses two online learning algorithms. The policy player Alg, generates a sequence of policies fnr kee by maximizing a sequence of negative costs --X" as rewards that are produced by the cost player Algx- In this paper, the policy player uses an online RL algorithm and the cost player uses the Follow the Leader (FTL.) algorithm. This implies that the cost time & is given as
-A To solve Eq. (2), we employ the meta algorithm
===Vo (
In other words, to solve an RL problem with a convex objective function (Eq. (1)), the policy player maximizes a non stationary reward that at time k corresponds to the negative gradient of the objective function f w.r.t - When the function f is convex, it is guaranteed that the average state occupancy of these polices, dk R Σ E dx. converges to an optimal solution to Eq. (1), iee. dk + d= E arg mind,ex fddz (Zahavy et al., 2021b).
Features and expected features. We focus on the case where each state-action pair is associated with some observable features Φ(s, a) E Rd For example, in the DM control suite (Tassa al., 2018), these features correspond to the positions and velocities of the body joints being controlled by the agent. In other cases, we can learn é with a neural network. Similar to value functions, which represent the expectation of the reward under the state occupancy, we define expected features as ( a) = ('_'da'dd,(ss Ø(s', a') E IRd, Note that in the special case of one-hot feature vectors, the expected features coincide with the state occupancy. The definition of t/r depends on the state occupancy we consider. In the discounted case, v2 E Rd is also known as Successor Features (SFs) as defined in (Barreto et al., 2017b; 2020). In the average case, vave Rd represents the expected features under the policy's 's stationary distribution and therefore it has the same value for all the state action pairs. Similar definitions were suggested in (Mehta et al., 2008; Zahavy al., 2020b).
DISCOVERING DIVERSE NEAR-OPTIMAL POLICIES
We now introduce Diversity Optimization Maintaining Near Optimality, or, DOMINO, which discovers a set of n policies m (ni) by solving the optimization problem:
max Diversity(I")) s.t dy -e 2 av,; VT 11", 11"
where o. is the value of the optimal policy. In other words, we are looking for a set of policies
that are diverse from each other as possible, defined as Diversity: (R/S)IAljr R. In addition,
3

---

Published as a conference paper at ICLR 2023
constrain the policies in the set to be nearly optimal. To define optimalltyy we introduce hyperparameter a E [0, 1), such that a policy is said to be near optimal if achieves a value that is at least ox.. In practice, we "fix" the Lagrange multiplier for the first policy μ = 1. so this policy only receives extrinsic reward, and use the value of this policy to estimate ve (v² = wy)'. Notice that this estimate is changing through training.
Before we dive into the details, we briefly explain the main components of DOMiNO. Building on Section 2 and, in particular, Eq. (3), we find policies that maximize the diversity objective by maximizing its gradient as a reward signal, i.e., r V d; Diversity(d.,......
We propose the first candidate for this objective and derive an analytical formula for the associated reward in Eq. (5). We then move to investigate mechanisms for balancing the quality-diversiy trade-off. In Eq. (7) we extend the first objective and combine it with a second, attractive force (Eq. (7)), taking inspiration from the Van Der Waals (VDW) force. The manner in which we combine these two forces allows us to control the degree of diversity in the set. In Section 3.1 and Section 3.2 we investigate other mechanisms for balancing the QD trade-off. We look at methods that combine the two rewards via coefficients oe and c² such that each of the policies, π ......,,'' x", is maximizing reward r that is a linear combination of the extrinsic reward Te and intrinsic reward ryy i.e. (s,, a) rere(s (s, a) + (s, a). In Section 3.1 we focus on the method of Lagrange multipliers, which adapts the coefficients online in order to solve Eq. (4) and compare it with other multi-objective baselines in Section 3.2.
A repulsive force.
We now present an objective that motivates policies to visit different states on average. It does so by leveraging the information about the policies' long-term behavior available in their expected features, and motivating the state occupancies to be different from each other. For that reason, we refer to this objective as a repulsive force (Eq. (5)).
How do we compute a set of policies with maximal distances between their expected features? To answer this question, we first consider the simpler scenario where there are only two policies in the set and consider the following objective max,* x= || 1/ This objective is related to the objective of Apprenticeship Learning (AL; Abbeel & Ng. 2004), i.., solving the problem ming Ilvl 1 where ve are the feature expectations of an expert. Both problems use the euclidean norm in the feature expectation space to measure distances between policies. Since we are interested in diversity, we are maximizing this objective, while AL aims to minimize it.
Next, we investigate how to measure the distance of a policy from the set of multiple policies, II" First we introduce the Hausdorff distance (Rockafellar, 1970) that measures how far two subsets D, C of a metric space are from each other: Dist D, C) = min ||c - d|3 /3. In other words, two sets are far from each other in the Hausdorff distance if every point of either set is far from all the points of the other set. Building on this definition, we can define the distance from an expected features vector s to the set of the other expected features vectors as mingye 1| ل 11.. This equation gives us the distance between each individual policy and the other policies in the set. Maximizing it across the policies in the set, gives us our first diversity objective:
max 0.5 Σn_ min da
(5)
In order to compute the associated diversity reward, we compute the gradient rt Va Diversity(d), ... d=)). To do so, we begin with a simpler case where there are only two policies, i.e., r Vaz 1 0²
) such that r(s, a) = o(s a) (*²) v) This reward was first derived by Abbeel & Ng (2004), but here is with an opposite sign since we care about maximizing it. Lastly, for a given policy n¹, we define by
Vay [E (' (s.) a) (sd ((s, a) = = ---(- (v)
ji the index of the policy with the closest expected features to T², iee. j = arg min 11 - I
Using the definition of jr, we get² that V mingzi 111/
w = Vay || u/o5" [13, and that
a))s, = o(8,a) a) (1b² 3i))
'We also experimented with variation where oi max, ef It performed roughly the same which makes sense since for most of the time max, of = vol (as policy O only maximizes extrinsic reward). II the rare case that the arg min has more than one solution, the gradient is not defined, but we can still use Eq. (6) a reward.
4

---

Published as a conference paper at ICLR 2023
The Van Der Waals force. Next, propose a second diversity objective that allows to control the degree of diversity in the set via hyperparameter Eg. As we will soon see, once a set of policies will satisfy diversity degree of lo. the intrinsic reward will be zero, allowing the policies to focus on maximizing the extrinsic reward (similar to clipping mechanism). The objective is inspired from molecular physics, and specifically, by how atoms in crystal lattice self-organize themselves at equal distances from each other. This phenomenon is typically explained as an equilibrium between two distance dependent forces operating between the atoms known as the Van Der Waals (VDW) forces; one force that is attractive and another that is repulsive.
The VDW force typically characterized by a distance in which the combined force becomes repulsive rather than attractive (see, for example, (Singh. 2016)). This distance is called the VDW contact distance, and we denote it by lo. In addition, we denote by & = || - e ll2 the Hausdorff' distance for policy i. With this notation, we define our second diversity objective
maa
0.5g
0.2(6%/6)
Repuisive
Artractive
We can see that Eq. (7) is a polynomial in e,, composed of two forces with opposite signs and different powers. The different powers determine when each force dominates the other. For example, when the expected features are close to each other (f, << tu), the repulsive force dominates, and when (&, >> (o) the attractive force dominates. The gradient (and hence, the reward) given by
(s, a) (1- (ls/o)) (os, a) (v' (.
(8)
We note that the coefficients in Eq. (7) are chosen to simplify the reward in Eq. (8). Lee,, since the reward is the gradient of the objective, after differentiation the coefficients are 1 in Eq. (8). Inspecting Eq. (8) we can see that when the expected features are organized at the VDW contact distance lo. the objective is maximized and the gradient is zero. We note that the powers (and the coefficients) of the attractive and repulsive forces in Eq. (7) were chosen to give a simple expression for the reward in Eq. (8) but one could consider other combinations of powers and coefficients.
3.1 CONSTRAINED MDPs
At the core of our approach is solution to the CMDP Eq. (4). There exist different methods for solving CMDPs and we refer the reader to (Altman, 1999) and (Szepesvári, 2020) for treatments of the subject at different levels of abstraction. In this work we will focus reduction of CMDPs to MDPs via gradient updates, known as Lagrangian methods (Borkar, 2005).
Most of the literature CMDPs has focused on linear objectives and linear constraints. In Section 2, we discussed how to solve an unconstrained convex RL problem of the form of Eq. (1) as a saddle point problem. We now extend these results to the case where the objective is convex and the constraint is linear, i.e. min f(zz), subject to g(dx) I O, where f denotes the diversity objective and g is a linear function of the form (d+)) = ore d. re defining the constraint. Solving this problem is equivalent to solving the following problem:
min max d, EK a20
f(d.) + pg(dz) min max X dx - (²) + p(av, dx re), d, EK M20,A
(9)
where the equality follows from Fenchel duality as before. Similar to Section we use the FTL algorithm for the À player (Eq. (3)). This implies that the cost at iteration k, X² is equivalent to the gradient of the diversity objective, which we denote by ra. Eq. (9) involves vector, X- pure, linearly interacting with d,. Thus, intuitively speaking, minimizing Eq. (9) from the perspective of the policy player is equivalent to maximizing reward ra + pra-
The objective for the Lagrange multiplier is to maximize Eq. (9), equivalently plaz," dx- re). Intuitively speaking, when the policy achieves an extrinsic value that satisties the constraint, the Lagrange multiplier/ y decreases (putting a smaller weight on the extrinsic component of the reward) and it increases otherwise. More formally, we can solve the problem in Eq. (9) as a three-player game. In this case the policy player controls d, as before, the cost player chooses À using Eq. (3), and the Lagrange player chooses with gradient descent. Proving this statement is out of the scope of this work, but shall investigate it empirically.
5

---

Published as a conference paper at ICLR 2023
3.2 MULTI-OBIECTIVE ALTERNATIVES
We conclude this section by discussing two alternative approaches for balancing the QD trade-off, which we later compare empirically with the CMDP approach. First, consider a linear combination method that combines the diversity objective with the extrinsic reward as
max Cadx + CaDiversity( iitiverrstyy.............. (()) 1
(10)
where Cas Ca are fixed weights that balance the diversity objective and the extrinsic reward. We note that the solution of such linear combination MDP cannot be a solution to a CMDP in general. Le., it is not possible to find the optimal dual variables j4", plug them into Eq. (9) and simply solve the resulting (unconstrained) MDP. Such an approach ignores the fact that the dual variables must be a "best-response" to the policy and is referred to as the "scalarization fallacy" in (Szepesvári, 2020). We now outline few potential advantages for using CMDPs. First, the CMDP formulation guarantees that the policies that we find are near optimal (satisfy the constraint). Secondly, the weighting coefficient in linear combination MDPs has to be tuned, where in CMDPs it is adapted. This is particularly important in the context of maximizing diversity while satisfying reward. Next, consider a hybrid approach that combines linear combination MDP with CMDP as
max max(O, ax." d² re) + caDiversity(d,..... .d)). 11
We denote by I' an indicator function for the event in which the constraint is not satisfied for policy n², Le.. I = if d' Te < ox"" and O, otherwise. With this notation the reward is given by Reverse SMERL: r'(ss a) ['re(s, +)+ +arr' a) In other words, the agent maximizes weighted combination of the extrinsic reward and the diversity reward when the constraint is violated and only the diversity reward when the constraint is satisfied. Kumar et al. (2020) proposed a similar approach where the agent maximizes weighted combination of the rewards when the constraint is satisfied, and only the extrinsic reward otherwise: SMERL: r' a) = re a) + ca(1 - I') (s, a) Note that these methods come with an additional hyperparameter cæ which balances the two objectives linear combination MDP, in addition to the optimality ratio a.
4 EXPERIMENTS
Our experiments are designed to validate and get confidence in the DOMINO agent. We emphasize that we do not explicitly compare DOMiNO with previous work nor argue that one works better than the other. Instead, we address the following questions: (a) Can DOMiNO discover diverse policies that are near optimal? see Fig. 2, Appendix C.I, Fig. 1b and the videos in the supplementary. (b) Can DOMINO balance the QD trade-off? see Fig. 2, Fig. 2 & 3. (c) Do the discovered policies enable robustness and fast adaptation to perturbations of the environment? (see Fig. 4).
Environment. We conducted most of our experiments on domains from the DM Control Suite (Tassa et al., 2018), 8), standard continuous control locomotion tasks where diverse near-optimal policies should naturally correspond to different gaits. Due to space considerations, we present Control Suite results on the walker.stand task. In the supplementary. however, we present similar results for walker walk and BiPedal walker from OpenAl Gym (Brockman et al., 2016) suggesting that the method generalizes across different reward functions and domains. We also include the challenging dog domain with 38 actions and 223 223-2iimaee -ii--mnnsso observation space, which is one of the more challenging domains in control suite.
Agent. Fig. la shows an overview of DOMiNO's components and their interactions, instantiated in an actor-critic agent. While acting, the agent samples a new latent variable = E [1, n] uniformly at random at the start of each episode. We train all the policies simultaneously and provide this latent variable as an input. For the average reward state occupancy, the agent keeps an empirical running average for each latent policy of the rewards g and features (either from the environment Pats or torso embedding @embeddsng) ways encountered, where the average is taken as ž = agaa + (1 a.) / Σi= ,'((s,, a.) with decay factor azg. Varying a. can make the estimation more online (small Od, as used for the constraint), or less online (large Odd as needed for Eq. (4)). The agent uses Jave to compute the diversity reward as described in Eq. (6). o is used to optimize the Lagrange multiplier μ for each policy in Eq. (9) which is then used to weight the quality and diversity advantages for the policy gradient update. Pseudo code and further implementation details, as well as treatment of the discounted state occupancy, can be found in Appendix B.

---

Published as a conference paper at ICLR 2023
Quality and diversity. To measure diversity qualitatively, we present "motion figures" by discretizing the videos (details in the Appendix) that give a fair impression of the behaviors. The videos, associated with these figures, can be found in the supplementary as well. Fig. 1b presents ten polices discovered by DOMiNO with the repulsive objective and the optimality ratio set to 0.9. The policies are ordered from top-left to bottom right, so policy 1, which only maximizes extrinsic reward and sets the constraint, is always at the top left. The policies exhibit different types of standing: standing on both legs, standing on either leg, lifting the other leg forward and backward, spreading the legs and stamping. Not only are the policies different from each other, they also achieve high extrinsic reward in standing (see values on top of each policy visualization). Similar figures for other domains can be found in Appendix C.I.
To further study the QD trade-off, we use scatter plots, showing the episode return on the y-axis, and the diversity score, corresponding to the Hausdorff distance (Eq. (5)), on the x-axis. The top-right corner of the diagram, therefore, represents the most diverse and highest quality policies. Each figure presents a sweep over one two hyperparameters and we use the color and marker to indicate the values. In all of our experiments, we report 95% confidence intervals. In the scatter plots, they correspond to 5 seeds and are indicated by the crosses surrounding each point.
Fig. 2 (left) presents the results for DOMINO with the repulsive reward in the walker. domain. We can observe that regardless of the set size, DOMINO achieves roughly the same extrinsic reward across different optimality ratios (points with the same color obtain the same y-value). This implies that the constraint mechanism is working as expected across different set sizes and optimality ratios. In addition, we can inspect how the QD trade-off is affected when changing the optimality ratio a for sets of the same size (indicated in the figures with light lines). This observation can be explained by the fact that for lower values of a, the volume of the constrained set is larger, and therefore, it is possible to find more diversity within it. This behavior is consistent across different set sizes, though it is naturally more difficult to find a set of diverse policies as the set size gets larger (remember that we measure the distance to the closest policy in the set).
We present the same investigation for the VDW reward in Fig. 2 (center). Similar to the repulsive reward, we can observe that the constraint is satisfied, and that reducing the optimality ratio allows for more diversity. Fig. 2 (right) shows how different values of to affect the QD trade-off for a set of size 10. We can observe that the different combinations of Eo and a are organized as a grid in the QD scatter, suggesting that setting these two interpretable hyperparameters. Lastly, Fig. C8 in the end of Appendix C.2 presents results for an additional experiment where we use only the VDW reward without the constraints.
control both the level of optimality and the degree of diversity by
Fig. 3 compares the QD balance yielded by DOMINO to the alternative strategies described in Section 3.2. Specifically, look at DOMiNO's Lagrangian method, the linear combination of the objectives (Eq. (10)), and the two hybrid strategies, SMERL and Reverse SMERL for a set of ten policies in walker.stand. Note that in all cases we are using DOMiNO's s repulsive diversity objective, and the comparison is strictly about strategies for combining the quality and diversity objectives. The plot for each strategy shows how the solution to the QD tradeoff varies according to the relevant hyperparameters for that strategy, namely, the optimality ratio for DOMINO, the fixed constant ce for the linear combination strategy (we implicitly set Ca - Ce), and both a and constant ea for the hybrid approaches (in the hybrid plots, Co value is labeled as marker, while is indicated by color).
For the linear combination, shown on the right, the C. parameter proves iel-behaved and choppy. quickly jumping from the extreme of all diversity no quality to the opposite, without smooth interim.
Repuisive DCNINO (set size
Van der Wasls DOMINO (set size
Van der Vasiss DOMA (o
ato
p.s
41.55
4.0
+555
0.9
500
Revard
600
700
700
700
p.
.0.55
4.75
-.00
1125-15
4.5--45
08
300
4.75
4.0
+3.25
06
502
0:75
4.0
.2.35
05
0.6
0.8
1.0
1.2
1.4
2.5
0.7
0.8 Diversity
0.8
1.0
3.2
3.4
Diveniity
Divesity
Figure 2: DOMINO QD results in walker.stan. Left: Set size vs. optimality ratio (a) with the repulsive reward. Center: Set size vs. a with the VDW reward (6g = 1). Right: Target diversity (b)) vs. a with the VDW reward.
7

---

Published as a conference paper at ICLR 2023
Unear
2000
3000
45
500
400
Iiverity
Figure 3: DOMiNO's Lagrangian method finds only solutions that push the upper right boundary of quality and diversity, and varies in smooth, interpretable way with its only hyperparameter, a, contrasted with the jumpy nature of the linear combination hyperparameter ce. and the redundancy of the hyperparameters in the hybrid methods.
In contrast, the DOMiNO approach of solving the CMDP directly for the I Lagrange multiplier yields solutions that push along the upper right diagonal boundary, finding the highest diversity (farthest right) set of policies for a given optimality ratio (color), varying smoothly along this line a varies. Another advantage of DOMiNO's 's approach is that it only finds such QD-optimal solutions, where, in contrast, SMERL (left), when appropriately tuned, can also yield some solutions along the upper-right QD border, but often finds sub-optimal solutions, and therefore must be tuned further with cJ to find the best solutions. We further explore the difficulty tuning SMERL in the supplementary (Fig. C6) and find that the best for 10 policies provides solutions with no diversity for other set sizes.
Feature analysis The choice of feature space used for optimizing diversity can have a huge impact on the kind of diverse behavior learned. In environments where the observation space is high dimensional and less structured (e.g. pixel observations), computing diversity using the raw features may not lead to meaningful behavior. As specified in Section 3 the feature space used to compute diversity in our Control Suite experiments throughout the paper corresponds to the positions and velocities of the body joints returned observations by the environment. We show that it is feasible to use learned embedding space instead. As a proof of principle we use the output of the torso network as a learned embedding described in Section 4 for computing our diversity metric.
Table 1 compares the diversity measured in raw ob- servation features (first row) and embedding fea- tures (second row) in the saak domain. Columns indicate the feature space that was used compute the diversity objective during training averaged across 20 seeds. Inspecting the table, we can see that agents trained to optimize diversity in the learned embedding space and agents that directly optimize diversity in the observation space achieve comparable diversity if measured in either space, indicating that learned embeddings can feasibly be used to achieve meaningful diversity.
Oahs
Gemhedding
Observation Embedding
1.21 = 0.05
1.01 = 0.05
2.14 = 0.09
2.35 = 0.09
Table
k-shot adaptation. We motivated qualitative diversity by saying that diverse solutions can be robust and allow for rapid adaptation to new tasks and changes in the environment. Here we validate this claim in k-shot adaptation experiment: we train set of diverse high quality policies on a canonical benchmark task, then test their ability to adapt to environment and agent perturbations. These include four kinematies and dynamics perturbations from the Real World RL suite (Dulac-Arood et al., 2019), and a fifth perturbation inspired by a "motor failure" condition (Kumar al., 2020) which, every 50 steps and starting at T 10, disables action-inputs for the first two joints for a fixed amount of time. In Fig. 4, we present the results in the walker. and walker.stand domains (rows). Columns correspond to perturbation types and the x-axis corresponds to the perturbation.
Closing the loop:
K-shot adaptation is measured in the following manner. For each perturbed environment, and each method, we first execute k = 10 environment trajectories with each policy. Then, for each method we select the policy that performs the best in the set. We then evaluate this policy for 40 more trajectories and measure the average reward of the selected policy metthod. The y-axis in each figure measures /ussed/ neeiie; where Thmseline measures the reward in the perturbed environment of an RL. baseline agent that was trained with a single policy to maximize the extrinsic reward in the original task. We note that the baseline was trained with the same RL algorithm, but without diversity, and it matches the state-of-the-art in each training domain (it is almost optimal). We found the relative metric to be the most informative overall, but it might be misleading in situations where all the methods perform roughly the same (e.g. in Fig. 4 bottom right). For that reason, we also include figure with the raw rewards r meethod Thaseline in the supplementary (Fig. E10). Lastly, we repeat this process across 20 training seeds, and report the average with 95% Confidence Interval (CI).
8

---

Published as a conference paper at ICLR 2023
1.50
1.00
1.50
SMERI
0.50
1.20
1.00
1.00
0.75
Lineer
direct
0.50
1.35
2.75
7.40
1.35
1.50
1.50
1.00
1.20
3.65
1.30
2.00
1.25
2.55
20
6.0
6.5
6.7
1.
Thigh length
Torso lesgi
Jean damping
Conbutt tration
Figure 4: K-shot adaptation Control Suite. We report mean episode return (95% CI) on held-out test tasks relative to the performance of single policy trained on extrinsic rewards. While not invariant to sudden changes in the environment, DOMiNO more robust to a variety of perturbations.
We compare the following methods: DOMINO, SMERL, Linear Combination, and No diversity, where all the diversity methods use the diversity reward from Eq. (6) with 10 policies. Since we are treating the perturbed environments as hold out tasks, we selected the hyper parameters for each method based on the results in Fig. 3, i.e.ee chose the configuration that was the most qualitatively diverse the upper-right most corner of Fig. 3). Concretely, for DOMiNO and SMERL = 0.9, for SMERI. = 0.5 and for linear combination Ce 0.7. More K-shot adaptation curves with other hyper parameter values can be found in Appendix E. The No diversity method is similar to Tbaseline- but uses 10 policies that all maximize the extrinsic reward (instead of a single policy).
Inspecting Fig. we can see that for small perturbations, DOMiNO retains the performance of the baseline. However, as the magnitude of the perturbation increases, the performance of DOMiNO is much higher than the baseline (by a factor of 1.5 2.0). This observation highlights that a diverse set of policies found by DOMiNO is much more capable at handling changes the environment and can serve as a strong starting point for recovering optimal behavior. As we have shown in 3.2, other approaches managing the trade-off between quality and diversity such as SMERL much more sensitive to the choice of able to find useful, diverse set of policies with some effort, it is difficult match DOMINO's performance across all pertubations and tasks, See the supplementary material for further comparison of DOMiNO with SMERL and linear combination over more hyper parameters. We also include video that illustrates how the individual policies adapt to the environment perturbations.
and require significant tuning. While SMERL is
5 CONCLUSIONS
In this work we proposed DOMiNO, an algorithm for discovering diverse behaviors that maintain optimality. We framed the problem as a CMDP in the state occupancies of the policies in the set and developed an end-to-end differentiable solution to it based on reward maximization.
We note that the diversity maximization objectives we explored are not convex RL problems; the opposite problem of minimizing the distance between state occupancies that is often used for Ap- prenticeship Learning is convex. Nevertheless, we explored empirically the application of convex RL algorithms to the convex problem of maximizing diversity. In experiments that the policies discovered by DOMiNO, or, DOMINO'S π, are diverse and maintain optimality. We then explored how DOMiNO balances the QD trade-off and compared it with linear combination baselines. Our results suggest that DOMiNO can control the degree of quality and diversity via interpretable hyperparameters, while other baselines struggle to capture both.
demonstrated
In particular, in Fig. 2 (right) we showed that DOMINO indeed solves the problem it is designed to solve: it satisfies the constraint and achieves the diversity specified by the VDW reward (&o). This is perhaps not surprising, the (non convex) optimization of Deep Neural Networks often builds on convex optimization algorithms to great success. Nevertheless we believe that our results make an important demonstration of this concept to the study of diverse skill discovery.
In our K-shot experiments we demonstrated that DOMiNO's can adapt to changes in the environ- ment. An exciting direction for future work is to use DOMINO in never ending RL setting, where the environment changes smoothly over time, and see maintaing a set of QD diverse policies will make it more resilient to such changes.

---

Published as a conference paper at ICLR 2023
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1. ACM, 2004.
Jacob Abemethy and Jun-Kun Wang. On frank-wolfe and equilibrium computation. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL httt::::/pprccceeee /ttpg/nnnei...../p////////000000 cccpppper/2017/FLL Pappee..........pp........
Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
Safa Alver and Doina Precup. Constructing a good behavior basis for transfer using generalized policy updates. arXiv preprint arXiv:21 2021.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038, 2020.
Andre Barreto, Will Dabney, Remi Munos, Jonathan Hunt, Tom Schaul, Hado van Has- selt, and David Silver. Successor features for transfer in reinforcement learning. In Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- neurips 9b8c05 Paper pdf.
ciates, Inc., 2017a. URL https: http://pooeeeencceceeeeo mewpeeddddinnnnnnnnn..reee.en wwwprriissccppppppeeppe wwtppaper/201//Eile 35Cdb08
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Advances in neural information processing systems, pp. 4055 4065, 2017b.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pp. 501- 510. PMLR, 2018.
André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Shibl Mourad, David Silver, Doina Precup, et The option keyboard: Combining skills in reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences, 2020.
Kate Baumli, David Warde-Farley Steven Hansen, and Volodymyr Mnih. Relative variational intrinsic control. Proceedings of the AAAI Conference on Artificial Intelligence, 35:6732 6740. May 2021. URL https: /ojs aaai. org/ index. wwtpp/pppppppaaaaaa http:///t..aaattteeeeeeeee www.tattttttttteeeeeeeeeeeetttee 16832..
Stay Belogolovsky, Philip Korsunsky, Shie Mannor, Chen Tessler, and Tom Zahavy. Inverse rein- forcement learning in contextual mdps. Machine Learning, pp. 1-40, 2021.
Shalabh Bhatnagar and Lakshmanan. An online actor-critic algorithm with function approximation for constrained markov decision processes. Journal of Optimization Theory and Applications, 153 (3):688-700 7088-708.77 2012.
Vivek S Borkar. An actor-critic algorithm for constrained markov decision processes. Systems & control letters, 54(3):207-213, 2005.
Greg Brockman, Vicki Cheung.
Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 100.::160060114000 2016.
Dan A. Calian, Daniel J Mankowitz, Tom Zahavy, Zhongwen Xu, Junhyuk Oh, Nir Levine, and Timothy Mann. Balancing constraints and rewards with meta-gradiae d4pg. In International Conference on Learning Representations, 2021. URL https: : / oppnnevveww net forum? id= 90
10

---

Published as a conference paper at ICLR 2023
Proceedings of the Genetic and Evolutionary Computation Conference, pp. 81-89, 2019. Antoine Cully. Autonomous skill discovery with qqulity-diverree and unsupervised descriptors. In
Antoine Cully and Yiannis Demiris. Quality and diversity optimization: A unifying modular
framework. IEEE Transactions Evolutionary Computation, 22(2):245-22) 259, 2017.
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. Robots that can adapt like animals. Nature, 521(7553):503-5077 2015.
Gabriel Dulac-Arnold Daniel Mankowitz, and Todd Hester. Challenges real-world reinforcement learning. arXiv preprint 1rX1v:1904.12991, 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pp. 1407-1416. PMLR, 2018.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward funetion. In International Conference Learning Representations, 2019. URL https: http:////ppeeeeeeeeeeee net/
JR
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The information geometry of unsupervised reinforcement learning. arXiv preprint arXiv:2 10.02279,,, 2021.
Yannis Blet-Berliac, Johan Ferret, Olivier Pietquin. Philippe Preux, and Matthieu Geist. Adversarially guided actor-critic. arXiv preprint arXiv:2102.043777 2021.
Tanmay Gangwani, Jian Peng, and Yuan Zhou Harnessing distribution ratio estimators for learning agents with quality and diversity. arXiv preprint arXiv:201 2222000..0214.4 2020.
Matthicu Geist, Julien Pérolat, Mathieu Laurière, Romuald Elie, Sarah Perrin, Olivier Bachem, Rémi Munos, and Olivier Pietquin. Concave utility reinforcement lcarning: the mean-field game viewpoint. arXiv preprint arXiv:2106.03787, 2021.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. International Conference on Learning Representations, Workshop Track, 2017. URL httpss://oppeeeeiieee net/forum F=
David Ha. Reinforcement learning for improving agent design. arXiv preprint 17x.1718100003779 2018.
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference Machine Learning, pp. 2681-2691. PMLR, 2019.
Matteo Hessel, Manuel Kroiss, Aidan Clark, lurii Kemaev, John Quan, Thomas Keck, Fabio Viola, and Hado van Hasselt. Podracer architectures for scalable reinforcement learning. arXiv preprint /04k:22/04006272, 2021.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. arXiv preprint 17X1::160..03476. 2016.
Zhang-War Wei Hong, Tzu- Yun Shann, Shih-Yan Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun- Yi Lee. Diversity-driven exploration strategy for deep reinforcement learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. pp. 10510-10521, 2018.
Sandy Huang, Abbas Abdolmaleki, Philemon Brakel, Steven Bohez, Nicolas Heess, Martin Ried- miller, et al. Explicit pareto front optimization for constrained reinforcement learning. 2020.
Saurabh Kumar, Aviral Kumar, Sergey Levine. and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems, 33, 2020.
Joel Lehman and Kenneth O Stanley. Evolving diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, pp. 21 1-2188 2011.
11

---

Published as a conference paper at ICLR 2023
Yang Liu, Prajit Ramachandran, Qiang Liu, and Jian Peng. Stein variational policy gradient. In 33rd Conference on Uncertainty in Artificial Intelligence, UAI 2017,0011
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7204-7213. PMLR, 18-24 Jul 2021. URL https: http::://ppocceeiigg wwr. mwr.pprsss/v1399
lupu2la. html.
Muhammad A Masood and Finale Doshi Velez. Diversity-inducing policy gradient: Using maximum mean discrepancy to find set of diverse policies. arXiv preprint arXiv:/906 000.00088, 2019.
Neville Mehta, Sriraam Natarajan, Prasad Tadepalli, and Alan Fern. Transfer in variable-reward hierarchical reinforcement learning. Machine Learning, 73(3):289, 2008.
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04900 2015.
Mirco Mutti, Riccardo De Santi, Piersilvio De Bartolomeis, and Marcello Restelli. Challenging common assumptions convex reinforcement learning. arXiv preprint arXiv:2202.01511 2022.
Geraud Nangue Tasse, Steven James, and Benjamin Rosman. A boolean task algebra for reinforcement learning. Advances in Neural Information Processing Systems, 33:9497-9507, 2020.
Olle Nilsson and Antoine Cully. Policy gradient assisted map-elites. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 866-875, 2021.
Alex F Osborn. Applied imagination. Scribner's, 1953.
Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances Neural Information Processing Systems, 33, 2020.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference machine learning, pp. 2778-2787. PMLR, 2017.
Zhenghao Peng, Hao Sun, and Bolei Zhou. Non-local policy optimization via diversity-reeullarized collaborative exploration. arXiv preprint arXiv:2006.07781, 2020.
Thomas Pierrot, Valentin Macé, Felix Chalumeau, Arthur Flajolet, Geoffrey Cideron, Karim Beguir, Antoine Cully, Olivier Sigaud, and Nicolas Perrin-Gilbers Diversity policy gradient for sample efficient quality-diversit optimization. ICLR Workshop on Agent Learning Open-Endedness, 2022.
Justin K Pugh, Lisa Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40, 2016.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 1984.
Ralph Tyrell Rockafellar. Convex analysis. Princeton university press, 1970.
Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off-policy actor-critic with shared experience replay. In International Conference on Machine Learning, pp. 8545-8554. PMLR, 2020.
Lior Shani, Tom Zahavy, and Shie Mannor. Online apprenticeship learning. arXiv:2102.06924, 2021.
arXiv preprint
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-awan unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
URL https: http:::::////peeeiie/ee net. mnet/forree forum? ddH
12

---

Published as a conference paper at ICLR 2023
Ashok K. Singh. Chapter structure, synthesis, and application of nanoparticles. In Ashok K. Singh (ed.), Engineered Nanoparticles, pp. 19-76. Academic Press, Boston, 2016. ISBN 978-0- 12-801406-6. doi: https://doi.oog/100111666666 0200778800011-000000666000000000 sci eccienceiin rect. com/ sci ence cannc//rtiin e 0101 4066000029.
URL https: //www.
e/
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. International Conference on Machine Learning, pp. 9133-9143, PMLR, 2020.
Hao Sun, Zhenghao Peng, Bo Dai, Jian Guo, Dahua Lin, and Bolei Zhou. Novel policy seeking with constrained optimization. arXiv preprint arXiv:2005. 10696 2020.
Csaba Szepesvári. Constrained mdps and the reward hypothesis. Musings about machine learning and other things (blog). 2020. URL const ra mad and www.re hewtte--ypottessiis html.
hol..oooppo...m....ooo.o......o. www.slloo.......o..o...o........ com/ 2020/03/
Danesh Tarapore, Jeff Clune, Antoine Cully, and Jean-Baptiste Mouret. How do different encodings influence the performance of the map-elites algorithm? In Proceedings of the Genetic and Evolutionary Computation Conference 2016, pp. 173-180, 2016.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 18Xiv:1801.000 00000,0 2018
Geraud Nangue Tasse, Steven James, and Benjamin Rosman. Generalisation in lifelong reinforcement learning through logical composition. In International Conference on Learning Representations, 2021.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations, 2019, URL htttts:///peeeee////// n wwwoorruunidsskfffeeeeeee
Bryon Tjanaka, Matthew C Fontaine, Julian Togelius, and Stefanos Nikolaidis. Approximating gradi- ents for differentiable quality diversity in reinforcement learning. arXiv preprint arXiv:2202.03666, 2022.
Tamara Ulrich and Lothar Thiele. Maximizing population diversity in single-objective optimization. In Proceedings of the 13th annual conference Genetic and evolutionary computation, pp. 641-648, 2011.
Vassilis Vassiliades, Konstantinos Chatzilygeroudis, and Jean-Baptiste Mouret. Scaling up map-elites using centroidal voronoi tessellations. arXiv preprint arXiv:1610.05729, 2016.
Tom Zahavy, Alon Cohen, Haim Kaplan, and Yishay Mansour. Apprenticeship learning via frank- wolfe. AAAI, 2020, 2020a.
Tom Zahavy, Alon Cohen, Haim Kaplan, and Yishay Mansour. Average reward reinforcement learning with unknown mixing times. The Conference on Uncertainty in Artificial Intelligence (UAI), 2020b.
Tom Zahavy, Andre Barreto, Daniel Mankowitz, Shaobo Hou, Brendan O'Donoguue, Iurii Kemaev, and Satinder Singh. Discovering set of policies for the worst case reward. In International Conference on Learning Representations, 2021a. URL https: http:////ppeeeeee/eeeeee net/ forum? id-PUkAW265dy5.
Tom Zahavy, Brendan O' O'Dongguue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex MDPs. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.). Advances in Neural Information Processing Systems, 2021b. URL https: btttps://penrreview net./ Forum? 2 d=RIndveeA--TR
Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari, and Mengdi Wang. Varia- tional policy gradient method for reinforcement learning with general utilities. arXiv preprint 05Xi5 2020.
13

---

Published as a conference paper at ICLR 2023
Yunbo Zhang, Wenhao Yu, and Greg Turk. Learning novel policies for tasks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7483-7492. PMLR. 09-15 Jun 2019. URL ht:t:: / proceed ngs mll. press/ v zhang] 9q html.
Zihan Zhou, Wei Fu, Bingliang Zhang, and Yi Wu. Continuously discovering novel strategies via reward-switching policy optimization. arXiv preprint urXhv:220..02246 2022.
14

---

Published as a conference paper at ICLR 2023
APPENDIX
A RELATED WORK
QD. quality-Diversii optimization is a type of evolutionary algorithm that aims at generating large collections of diverse solutions that are all nigh-performing (Pugh et al., 2016; Cully & Demiris, 2017. QD). It comprises two main families of approaches: MAP-Elitee (Cully et al., 2015; Mouret & Clune, 2015) and novelty search with local competition (Lehman & Stanley, 2011, NSLC), which both distinguish and maintain policies that are different in the behaviour space. The main difference is how the policy collection is implemented, either as a structured grid or an unstructured collection, respectively. The behaviour space is either handcrafted based on domain knowledge (Cully al., 2015; Tarapore et al.. 2016) or learned from data (Cully, 2019). Some other works combine evolutionary QD with RL (Pierrot et al., 2022; Tjanaka et al., 2022; Nilsson & Cully, 2021). Further references can be found on the QD webpage.
Our paper share its goals with those of the QD formulation, but it differs from this line of work in the type of optimization used to find the diverse high quality policies; while QD algorithms focus on evolution, our approach is based on reward maximization. This is feasible thanks to the introduction of Lagrange multipliers and because we apply the Fenchel transform to the diversity objective as we explained in Section 2.
MDPs with general objectives. Our work builds on recent advancements in the study of MDPs with an objective that is convex in the state occupancy and includes Apprenticeship Learning (Abbeel & Ng, 2004; Zahavy et al., 2020a; Shani et al., 2021; Belogolovsky et al., 2021) and maximum state entropy exploration (Hazan et al., 2019). Recently, few papers were published on solving general convex MDPs (Hazan et al., 2019; Zhang et al., 2020; Geist et al., 2021; Zahavy et al., 2021b; Mutti al., 2022).
Our work considers an objective that is not convex but uses the same algorithms that were developed for the convex case as we explained in Section 2. It might surprise some readers that the same algorithms perform well in the non convex case, nevertheless this is what we found empirically. We also note that it is also the case that convex optimization algorithms are found to be useful for optimizing deep neural nets, which are not convex. That said, we hope to further consider the implications of optimizing non convex MDPs with Convex MDP techniques in future work.
Intrinsic rewards. Intrinsic rewards have been used for discovering diverse skills. The most common approach is to define diversity in terms of the discriminability of different trajectory-spectte quantities and to use these ideas to maximize the Mutual information between states and skills (Gregor et al., 2017: Eysenbach et al., 2019; Sharma et al., 2020; Baumli et al., 2021). Other works implicitly induce diversity to learn policies that maximize the set robustness to the worst-possible reward (Kumar et al., 2020: Zahavy et al., 2021a), and other add diversity as a regularizer when maximizing the extrinsic reward (Hong et al., 2018; Masood & Doshi Velez, 2019; Peng al., 2020; Sun et al., 2020; Zhang et al., 2020; Sharma et al., 2020; Badia et al., 2020; Pathak et al., 2017).
It has been recently suggested that the Mutual Information (MI) between policies and states is equivalent to the average KL. divergence between state occupancies (Zahavy et al., 2021 Eysenbach et al., 2021). The MI is minimized in GAIL for AL (Ho & Ermon, 2016) and maximized for diversity in VIC and DIAYN (Eysenbach et al., 2019; Gregor et al., 2017). Our framework for modeling diversity objectives and deriving intrinsic rewards from them generalizes these results (which were specific to the mutual information) and allows one to introduce new objectives for diversity maximization and easily derive intrinsic rewards for them.
Lastly, Marker-Holder et al. (2020) measure diversity between behavioral embeddings of policies. The advantage that the behavioral embeddings can be defined via a differential function directly on the parameters of the policy, which makes the diversity objective differentiable w.r.t to the diversity measure. Our approach is similar in the sense that state-occupancies are also behavioral embeddings;
the state occupancy characterizes the policy perfectly as T(S, a) = E. d=(sa) (in states that are visited with non zero probability). This representation is not differentiable w.r.t the policy parameters, and we overcome this hurdle by deriving a reward signal that leads to maximizing diversity in this representation.
(s.a,)
15

---

Published as a conference paper at ICLR 2023
Using Successor Features to discover policies. In a related line of work, Successor Features used to discover and control policies (Tasse et al., 2021; Nangue Tasse et al., 2020; Zahavy et al., 2021a;a; Alver & Precup, 2021; Barreto et al., 2017a; 2018; 2019). In this line of work, there is sequence of rounds, and a new policy is discovered in each round by maximizing a stationary reward signal (the objective for each new policy is a standard RL problem). The rewards can be random vectors, one hot vectors, or an output of an algorithm (for example, an algorithm can be designed to increase the diversity of the set).
The main difference from this line of work and our paper is that we train all the policies in parallel via shared latent architecture. This approach has the advantage that is more sample efficient (since we solve a single problem for all the policies and not a new problem for each policy). In addition, some of the iterative approaches can be viewed as solutions for convex MDPs. That is, it can be shown that by optimizing a sequence of policies and defining a new reward in each round as the function of the previous state occupancies, the procedure converges to a solution for convex MDP (Abbeel & Ng. 2004; Zahavy et al., 2021a). However, in DOMINO we train all the policies in parallel to maximize single objective via a non stationary reward as we described in Section 2.
Constrained MDPs and multi objective QD. Constrained MDPs are popular model with vast literature; we refer the reader to (Altman, 1999; Szepesvári, 2020) for more details. The most popular approach for solving CMDPs is to use Lagrange multipliers Section 3.1. The main advantage with this approach is that the CMDP is reduced to an MDP with a non stationary reward, which makes it possible to existing RL algorithms analayze and implement a solution for CMDP (Borkar, 2005; Bhatnagar & Lakshmanan, 2012; Tessler et al., 2019; Stooke et al., 2020; Calian et al., 2021; Huang et al., 2020).
A different approach for solving CMDPs is SMERL (Kumar et al., 2020) which we analyzed in Section 3.2. SMERL and the reverse SMERL variation performed quite similar to the Lagrange multipliers approach, but required to be properly tuned. Another related idea is the reward switching mechanism (Zhou et al., 2022), which switches between extrinsic and intrinsic rewards via trajectory- based novelty measurement during the optimization process.
There are also approaches that aim to solve QD problems defined as constraint optimization problems. For example, the NOAH algorithm (Ulrich & Thiele. 2011) is an evolutionary algorithm which determines a maximally diverse set solutions whose objective values are below a provided objective barrier. It does so by iteratively switching between objective value and set-diversity optimization while automatically adapting a constraint on the objective value until reaches the barrier. The challenge in using this kind of algorithm in the RL setup is that it requires solving many RL problems (for different barriers). The Lagrange multipliers approach, on the other hand, allows us to optimize all the policies in parallel, together with the Lagrange multipliers, which is more computationally efficient.
In (Zhang et al., 2019) the authors propose to compute two gradients, one for the reward and one for the diversity objective, and then to update the policy in the direction of the angular bisector of the two objectives.
We also note that there had been many works that considered using a linear combination of rewards to balance the QD trade-off (Hong et al., 2018; Masood & Doshi- 2019; Parker-Holder et al., 2020; Gangwani et al., 2020; Peng et al., 2020). However, as we note before, these methods cannot find solutions to CMDPs (in general) due to the scalarization fallacy (Szepesvári, 2020). In addition, we found empirically that they do not find good solutions to the QD trade-off and instead find policy sets that are either diverse or of high quality, and not find solutions with more flexible balance.
Diversity in Multi Agent RI. There has also been lot of work in multi agent RL on discovering diverse sets of policies. In this case, the idea typically to discover diverse high quality policies and then to train Best Response policy that exploits all of the discovered policies. For example, Lupu et al. (2021) study zero-shot coordination. They propose train a common best response to a population of agents, which they regulate to be diverse. The diversity objective defined in terms of trajectories and use the
divergence to optimize it. Liu et al. (2017) consider diversity in Markov games. They proposed state- occupaacyy based diversity objectives via the shared state-action occupancy of all the players in the game (in addition to a second response based diversity metric that is specific to multi-agent domains). They then use an empowerment based reward to increase the diversity of this state occupancy. This is very interesting idea but somewhat orthogonal
16

---

Published as a conference paper at ICLR 2023
to our work since focus on the single agent scenario. In addition that, our approach generalizes the class of objectives, which typically focused on mutual information, to a more general class of diversity functions and to show that there is easy way derive an intrinsic reward for them. We also proposed two new diversity objectives based on these ideas (the Hausdorff and the VDW). We believe that this is exciting direction as it will allow others propose other objectives and then simply differentiate them to get reward.
Repulsive forces. In related line of work. Vassiliades et al. (2016) suggested to Voronoi tessellation to partition the feature space of the MAP-Elite algorithm to regions of equal size and Liu al. (2017) proposed Stein Variational Policy Gradient with repulsive and attractive components. Flet-Berliac et al. (2021) introduced an ad adversary that mimics the actor in an actor critic agent, and then added repulsive force between the agent and the adversary. Using VDW force to control diversity is novel to the best of our knowledge.
17

---

Published as a conference paper at ICLR 2023
B IMPLEMENTATION DETAILS
Distributed agent Acting and learning are decoupled, with multiple actors gathering data in parallel from a batched stream of environments, and storing their trajectories, including the latent variable 2, in replay buffer and queue from which the learner can sample a mixed batch of online and replay trajectories (Schmitt et al., 2020; Hessel et al., 2021). The latent variable 2 is sampled uniformly at random in [1, ni] during acting at the beginning of each new episode. The learner differentiates the loss function as described in Algorithm I, and uses the optimizer (specified in Table 2) to update the network parameters and the Lagrange multipliers (specified in Table 3). Lastly, the learner also updates the and moving averages as described in Algorithm
Initialization When training begins we initialize the network parameters as well as the Lagrange multipliers: μ² = a (0.5), Vi E [2, n1,, where σ -1 is the inverse of the Sigmoid function J = 1; and the moving averages: gayg = 0., Vi E [1, nl], Javg = I/d, Vi E [1, Here n is the number of policies and d is the dimension of the features
Bounded Lagrange multiplier To ensure the Lagrange multiplier does not get too large so as to increase the magnitude of the extrinsic reward and destabilize learning, use a bounded Lagrange multiplier (Stooke et al., 2020) by applying Sigmoid activation on μe so the effective reward is a convex combination of the diversity and the extrinsic rewards: r(s, a) = (u)))r a) +11 (o0--o(w)))) (rr'))r' and the objective for is a (()) - dz re).
Average state occupancy The empirical feature averages used for experiments in the main text are good, though imperfect due to the bias from samples before the policy mixes. In our experiments, however, since the mixing time for the DM Control Suite much shorter than the episode length T, the bias is small (~ 5%).
Discounted state oceupancy For a more scalable solution, as mentioned in Section 2, we can instead predict successor features using an additional network head as shown in Fig. C7a. Similar to value learning, we use V-trace (Espeholt al., 2018) targets for training successor features. state occupancy case we also use the extrinsic value function of each policy vg (Fig. la) to estimate dæ Te, instead of the running average v We show experimental results for this setup in Fig. C7b.
discounted
Loss functions. Instead of learning single value head for the combined reward, our network has two value heads, one for diversity reward and one for extrinsic reward.
We use V-trace (Espeholt et al., 2018) to compute td-errors and advantages for each of the value heads using the "vtrace td eгror and advantage" function implemented here https: / /gi thub com/ deepmind//i zlax/ /blbb/ /////materrr rlax/ /twp///s//////////////////////e/ vtraace py. The value loss for each head is the squared & loss of the td-errors, and the combined value loss for the network the sum of these two losses: td + td..
In addition to that, our network has policy head that is trained with policy gradient loss as implemented in https: /tte:://gttttt com/ / hn/rlaandrrax /bwbb/bb///// hob/mmae /aaaaaateeee strester//ee rlax/ laxx_src// policy_ hetllygrrdeentttte py). When training the policy, combine the intrinsic and extrinsic advantages 6 = ( + (1- o ())))) (see the Weight cumulants function in Appendix B. which has the same effect combining the reward. However, found that having value heads is more stable as each value can have different scale.
The final loss of the agent is weighted sum of the value loss the policy loss and the entropy regularization loss, and the weights can be found in Table 2.
Algorithm also returns Lagrange loss function, designed to force the policies achieve value that is at least a times the value of the first policy (which only maximizes extrinsic reward), where a the optimally ratio (Table 3). We update the Lagrange multipliers f using the optimizer specified in Table 3 but keep the multiplier of the first policy fixed 4d" = 1.
Lastly, Algorithm I also updates the moving averages.
18

---

Published as a conference paper at ICLR 2023
Algorithm 1: Loss function
Parameters:
Network parameters 0, Lagrange multipliers μa, moving averages ( ===1
i=1
Data: m trajectories { size T, T3 = { og.rg 02 22. [(ag[=)))) s==1 where
((a) is the probability assigned to ag in state z by the behaviour policy p(alz)) Forward pass: [T(a)>=)) v (x())), va(=))3 t Network((T))
Compute extrinsic td-errors and advantages: tde (e(2)) < V-trace with extrinsic
reward r. and extrinsic critic (x(22)
Compute intrinsic reward: r=[ from " og with Eq. (6) or (8)
Compute intrinsic td-errors and advantages: tda(x)), 5x(x2) + V-trace with intrinsic
reward rg and intrinsic critic va(az)
Combine ad adnatages:: 8(xz) a ((')) + (1 o (1)))a())
Weighted loss:
Σ by (tds + (da(x))²) + d log( + bea Entropy(T (x
Lagrange loss:
a Σ
o (o - a
Update moving averages:
2019 ad
avv ad g00g
Jayz x²
add
Juny
Jave
(1 ag
gavg
)or
return Weighted loss, Lagrange loss,
(
Javz
I 1==1"
B. FUNCTIONS
19

---

Published as a conference paper at ICLR 2023
def intrinsic (pwi,, sfs, latents, attrattive_owwer-3. sepulsive_power-O.. attractive_coeef-o0 target_d=l.); """Compuess a diversity reward using successor features.
Args:
phi: features [tbf].
sfs: avg successor features [1f) or
predicted, discounted successor features [tbfl]. latents: [tbl].
attractive_power. the power of the attractive force. repulsive_powee the power the repulsive force.
10
11
attractive_coet convex mixing of attractive repulsive forces target_d(\ell_) desired target distance between the sfs. When cepracttve__oeff--] objective, i.e., the gradient (the reward) is zero. Returns:
12
target_d is the minimizer of the
15
16
intrinsic_roward
17
# If sfs are predicted have extra leading dims. if jnp.ndim (sfs) == 4:
20
sfs swapaess (sfs, 2, 3) tbf1 -> tblf (to match shape of avg sf)
siatte_ddst__n__ = jax. vmap (jax. vmap ) matmul in Lambda x, y: jnp. einsum tbl, tbif->tbi', x, y)
23
elif inw. (sfs) 2:
24
compute_dist_
compute_distance matmul_ff = jnp.matmul
25
else:
raise ValueError (1 Invalid shape for argument 'sfs'.')
28
1, f shapeel--
29
# Computes an tb 1xl matrix where each row, corresponding to a latent, is a hot vector indicating the index of the latent with the closest sfs
30
dists compute_dist_fn(itt
sfs)
31
dists +- jnp.eye jap.max (dists)
32
learest__atents_matte - jax. wax.mn..ce__hn.
jnp.argmin (dists, axis=-2), nur_classes-1)
Computes [tb1] vector with the nearest latent each latent latents
35
nearest_latens latents = matmul fn (latents, nearest nearest_ matrix)
# Compute psi i-psi_j
psi_diff matmul_fn (latents nearest_latents, sfs) thf
38
norm_ciff jnp.sqrt (jnp. (jnp. square (psi_diff), axis=-1)) target_d
39
о (1. attractivo_cocfff wwfff itteeatttttcctve_poeeeee powerrowerr norm form_difffaee reward = c jnp. sum (phi psi_diff, axis=-1) reward
40
a - attractive_coet
43
44
def 12aist (x, y):
45
"""Returns the L2 distance between pair of inputs."" ""o
return jnp.sqrt (jnp. sum sinp..quureeee }
def 48
(x, y, dist_fn-12dist):
49
"*"Returns the distance between each pair of the two collections of inputs. ***
return jax. vmap (jax. vmap (dist_fn) (None, 01), (0, None) (x, y)
Listing Intrinsic Reward
20

---

Published as a conference paper at ICLR 2023
def weight cotullnts (lagrange, latents, extrinsic -etanssiccummaass,
intzinsic_oumulaatts)
"*"Weights cumulants using the Lagrange multiplier.
Args:
lagrange: lagrange [1].
latents: latents [tbl].
extrinsic_cumulane [tb].
Intrinsic_cumulatts: [tb].
10
11
Returns:
12
extrinsic reward r_e and intrinsic_reward r_d.
sig Lag_aag jax.nn mn sigmoid (lagrange)
15
latent sig_lagrange
imp.matmul (latents, sig [gg_lagrange)
16
# No diversity rewards for latent 0, only maximizo extrinsic reward
17
Intrinsic_oumulante + (1 latentsl;, 0])
return latent_sig aatent_sig_lagrange)
intrinsic_cumulants +
satent_sig_lagranee extrinsicc_ummlll
Listing 2: Weight cumulants
def lagrangian (lagrange, r, "**loss function for the lagrange multiplier.
Args:
lagrange: lagrange [1].
moving averages reward [1].
optimality_ration [1].**M
1_ = jax. nn. sigmoid
return jnp.sum (1_ (r r[0] serimmaitt_ratiin
Listing lagrange loss function
B.2 MOTION FIGURES
Our "motion figures" were created in the following manner. Given trajectory of frames that composes a video fi.....fr we first trim and sub sample the trajectory into point of interest in time: fa..... fntm. We always sub figures in a figure). We then sub sample frames from the trimmed sequence at frequency 1/p: fas fatp- fnt2p- After that, take the maximum over the sequence and present this "max" image. In Python for example, this simply corresponds to
the same trimming across the same set of policies (the
n=400, m=30, p-3 indices range (n, n+m, im np. max [indices])
This creates the effect of motion in single figure since the object has higher values than the background.
B.3 HYPERPARAMETERS
The hyperparameters in Table 2 are shared across all environments except the BiPedal Domain the learning rate is set 10-5 and the learner frames 5 x 10T, We report the DOMINO specific hyperparameters in Table 3.
21

---

Published as a conference paper at ICLR 2023
Hyperparameter
Value
Replay capacity
5 x 10a
Learning rate
10-4
Learner frames
2 10?
Discount factor
0.99
bee Entropy regularization weight
0.01
lx Policy loss weight
1.0
Value loss weight
1.0
Replay batch size
600
Online batch size
Sequence length Optimizer
40
RMSprop
Table 2: General hyperparameters
Hyperparameter
Control Suite
BiPedal Walker
a Optimality ratio Lagrange initialization
0.9
0.7
0.5
0.5
Lagrange learning rate
10-3
10
Lagrange optimizer
Adam
Adam
aav
decay factor od
0.9
0.999
decay factor ad
0.99
0.9999
Table 3: DOMINO hyperparameters
C ADDITIONAL EXPERIMENT RESULTS
C.1 MOTION FIGURES
We now present motion figures, similar to Fig. lb, but in other domains (see Fig. 6-9). The videos, associated with these figures can be found in separate .zip file. Each figure presents ten polices discovered by DOMiNO and their associated rewards (in white text) with the repulsive objective and optimality ratio set to 0.9. As we can see, the policies exhibit different gaits.
Next to each figure, we also present the distances between the expected features of the discovered policies measured by the & norm. addition, in each row we use a dark black frame to indicate the the index of the policy with the closest expected features to n² i.e., in the i-th row we highlight the j-th column such that j = j = arg min || 26
22

---

Published as a conference paper at ICLR 2023
Usslamue
Figure Cl: QD in walker.stand. Left: 10 policies and corresponding rewards. Right: distances in l
norm between the Successor features of the policies.
25
Figure C2: QD in walker walk. Left:
10 policies and corresponding rewards. Right: distances & norm between the Successor features of the policies.
Figure C3: QD in Left: 10 policies and corresponding rewards. Right: distances in &
norm between the Successor features of the policies.
Figure C4: QD in dog. walk. Left:
10 policies and corresponding rewards. Right: distances in 82 norm between the Successor features of the policies.
23

---

Published as a conference paper at ICLR 2023
C.2 ADDITIONAL QUALITY DIVERSITY RESULTS
We now present additional experimental results evaluating the trade-off between quality and diversity using the scatter plots introduced in Fig. 2. y-axis shows the episode return while the diversity score, corresponding to the Hausdorff distance (Eq. (5)), is on the x-axis. The top-right corner of the diagram represents the most diverse and highest quality policies. Each figure presents sweep over one or two hyperparameters and we use the color and a marker to indicate the values. In all of our scatter plots, we report 95% confidence intervals, corresponding to seeds, which are indicated by the crosses surrounding each point.
Quality Diversity: walker.walk In Fig. C5 we show experimental results for DOMiNO in the walker. walk domain. Consistent with Fig. which shows similar results walker. we show that our constraint mechanism is working as expected across different set sizes and optimality ratios across different tasks.
Repulsivo DONSNO (set size
Wan der Vasil DONNN (++
-20
-075
800
703
2.75
2.0
-225
03
800
Rewerd
n00
23
4075
200
-.25
-25
07
600
503
600
-05
-35
+4.25
-25
06
400
500
500
455
400
0.5
1.2
0.4
0.8
20
Diversity
Diversit
Figure C5: QD Scaling results on walker. task. Left: Number of policies vs. optimality ratio in walker.walk with the repulsive reward and (center) with the VDW reward. Right: Optimality ratio vs. VDW target distance Eg.
Quality Diversity: SMERL vs DOMINO In Fig. C6 we show further experimental results in walker.stand for SMERL in comparison to DOMiNO. When SMERL is appropriately tuned (here for the 10 policies configuration), it can find some solutions along the upper-right QD border; however we find that the best Ca does not transfer to other configurations. The choice of ca that enables the agent find set of 10 diverse policies produces sets without diversity for any other set size.
SMERL (set size vs. n))
Repuisive DOMINO (set size vs. a)
1000
0.8
Reward
600
0.7
400
10
0.6
0
1.0
1.5
2.0
0.5
1.0
1.5
2.0
Diversity
Diversity
Figure C6: Scaling SMERL (left) vs. DOMiNO (right) on Walker.Stand. Set size is indicated with marker, color corresponds to optimality ratio a. The ca for SMERL is set to 0.5, which was tuned using set size of 10 policies (see 3, left). This choice does not scale well to any other set size, where regardless of optimality ratios, all policies only optimize for extrinsic reward, at the expense of diversity.
Discounted State Occupancy We run the same experiments reported in Fig. 2 with DOMINO's Lagrangian method and report the results in Fig. C7b. As can be observed, using predicted discounted features does not make any significant difference in performance. Since the mixing time for the DM Control Suite much shorter than the episode length T, the bias in the empirical feature averages is small.
24

---

Published as a conference paper at ICLR 2023
obs
Lpdate constraint Lggrange
AG
Discounted DOMINO
МEP
Torso
Diversity Reward
50
0.9
Acebokinn
800
50.0
20
0.8
Reward
700
400
220
10
0.7
600
20
0.6
V2
500
50
400
0.5
0.8
1.0
1.2
1.4
1.6
1.8
Diversity
(h)
Figure C7: (a) DOMiNO with a discounted state oceupancy. An additional network head is trained to predict successor features 1/7, which are used instead of the average features glang to compute the diversity reward. The discounted, extrinsic value is used constraint instead of the averaged rewards. Dashed lines signify training objectives. (b) Number of policies vs. optimality ratio in walker.stand with DOMiNO, consistent with Fig.
The VDW reward without constraints Lastly, we study how the VDW reward can help to balance the QD trade off without using the constrained mechanism at (r Fa Tes where ra the VDW reward). Fig. C8 presents such study where we compare the VDW contact distance lo and the size of the set. We can see that setting to to different values indeed gives us this degree of diversity. For example, the purple points correspond to to 0.25 and they are all scattered at Diversity values (x-axis) 0.25. We can also see that for the same lg, increasing the set size decrease performance, and in fact, for large values of to is only for small set sizes (of size 10) that it is possible to find sets with that degree of diversity without hurting performance. However this phenomena diminishes for lower values of fo. This expected since if the "volume" of sub optimal policies is fixed, then for lower values of to is possible to get more policies inside
902
303
702
VOW ausuns
002
202
oo
203
02
7.4
0.8
03
1.0
12
Divesity
Figure C8: DOMINO QD results in walker.stand. Set size vs. VDW distance (lo) with the VDW reward
Hyper parameter study of the powers used in the VDW reward Eq. (8)
As we explained below Eq. (8), the powers in Eq. (7) are chosen to simplify the reward in Eq. (8). The only restriction is that the power of the repulsive force will be lower then of the attractive force. To verify this statement we performed the following hyper parameter study where we changed the attractive power in 3,4,5 and the repulsive power in 0,1,2 (bold indicates the values used in the main paper). The results can be found Fig. C9a and Fig. C9b, showing that these hyper parameters have an insignificant effect on diversity and reward. The only impact that found the speed of
25

---

Published as a conference paper at ICLR 2023
convergence of the diversity metric where there three under performing eurves (lower) in Fig. C9b corresponding to the larger attractive force power (2) but still converged at the end to the same level of performance.
(b)
Figure C9: Hyper parameter study of the powers used in the VDW reward on the (a) Extrinsic reward and (b) Diversity.
26

---

Published as a conference paper at ICLR 2023
D LIMITATIONS
Diversity increasing by decreasing a Inspecting Fig. C5, Fig. C7b and Fig. 2. we can observe that the diversity score increases for lower optimality ratios. Recall that the optimality ratio specifies feasibility region in the state-occupancy space (the set of all a-optimal policies). Thus, the size of this space increases as decreases, and observe more diverse sets for smaller values of a. This intuition was correct in most of our experiments, but not always (e.g.. Fig. C5).
One possible explanation is that the Lagrange multipliers solution is seeking for the lowest value of that satisfies the constraint (so that we can get more diversity), i.e., finds solutions that satisfy the constraint almost with equality: of au (instead of vy > aw²). The size of the level sets (e" = ax²)) do not necessarily increase with lower values of (while the feasibility sets ea 2 av do). Another explanation is that in walker. walk (Fig. C5) it might be easier to find diverse walking (e.g., = 0.9) than diverse "half walking" (e.g., = 0.5). This might be explained by "half walking" being less stable (it is harder find diverse modes for it).
Features Another possible limitation of our approach is that diversity is defined via the environment features. We partially addressed this concern in Table where we showed that possible to learn diverse high quality policies with our approach using the embedding of NN as features. In future work we plan to scale our approach to higher dimensional domains and study which auxiliary losses should be added to learn good representations for diversity.
E
ADDITIONAL K-SHOT EXPERIMENTS
E. GENERAL COMMENTS
We note that while we tried to recreate a similar condition (Kumar al., 2020), the tasks are not directly comparable due to significant differences in the simulators that have been used as well as the termination conditions the base task.
For CI, we use boosted CI with nested sampling as implemented in the bootstrap function here, which reflects the amount of training seeds and the amount of evaluation seeds per training seed.
E.2 CONTROL SUITE
Next, we report additional results for K-shot adaptation in the control suite. In Fig. E1O we report the absolute values achieved by each method obtained (in the exact same setup as in Fig. 4). That is, report
report mmethod for each method (instead of Tmethod/ Maaaood//rssllin /baseline in Fig. 4). Additionally, Thaseline, which the "Single policy baseline" (blue) in Fig. R10. Inspecting Fig. E10, we can see that all the methods deteriorate in performance as the magnitude of the perturbation increases. However, the performance of DOMiNO (orange) deteriorates slower than that of the other methods. We can also that the performance of the no diversity baseline is similar when it learns 10 policies (red) and a single policy (blue), which indicates that when the algorithm maximize only the extrinsic reward, it finds the same policy again and again with each of the policies.
Sbhn
SMIRI No diversity Unear
200
2000
200
200
Walk
400
30
00 Thigh langth
3
30
>
6
2
5
6.2
0.005
0.001
Motor fallure Iduration
Torsa bength
Joint damping
Contect friciir
Figure E10: K-shot adaptation in Control Suite, similar to Figure 4, but reporting absolute rather than relative returns.
Next, we inspect a wider range of hyper parameters for the SMERL and linear combination methods. Concretely, for DOMiNO and SMERL 0.9, for SMERL ca E [0.5, 1, and for linear
27

---

Published as a conference paper at ICLR 2023
combination ce [0.1, 0.3, 0.4, 0.5, 0.6, 0.660.7.. 0.7, 0.9] and all methods are trained with 10 policies. These values correspond to the values that we considered in Fig. 3.
that the best methods overall are DOMINO and SMERL (with ca 1). We can also see that DOMiNO and SMERL consistently outperform the linear combination baseline for many hyper parameter values. This is consistent with our results in Fig. 3 which suggest that the linear combination method tends to be either diverse or high performing and fails to capture a good balance in between. Lastly, it reasonable that SMERL and DOMINO perform similar since they are both valid solutions to the same CMDP. However, SMERL comes an with additional hyper parameter ed that may be tricky tune in some situations. For example, trying to tune ed based on the results in the vanilla domain (picking the upper-right most point in Fig. 3) led us to choose cd 0.5 for SMERL, instead of The Lagrange multipliers formulation DOMiNO does not have this challenge as it does not have an extra hyper parameter.
Inspecting Fig. E11
1000
Single baseline
Noun eetums Stand
(0.1)
Linear (0.2)
Lincar (0.3)
Lincar 4)
400
10
Linear (0.9)
Linear (0.7)
Mean returna Wak
SMEAL le-5.0
SMEII (om0.5)
200
200
200
0.02
0.005
0,011
Moter fallure (dlaation))
Thigh
Torso length
Joint damping
Figure E11: K-shot in Control Suite, similar to Figure but reporting wider range of hyper parameters for SMERL and linear combination.
E.3 VARIABLE
Next, in Fig. E12 we performed an ablation study on the impact that the total number of trajectories K has selecting the best policy in the set. Since K is the total number of trajectories, it is not always perfectly divided by the number of policies (chosen to be 10 for this experiment). Thus, we randomly distribute the modulo between the policies. Fig. E12 suggests that using more trajectories clearly helps, but it also suggests that performance saturates early on and that it is possible to use much smaller K (around 30) and still outperform the "single extrinsic policy baseline".
700
800
825
Baseline
750
38000
700
800
700
775
600
600
650
750
800
750
700
700
700
08575
700
Moan
600
$50
660
650
500
525
50
10
50
100
100
10
50
100
10
100
Fallure
Langth
Friction
Figure B12: K-shot adaptation ablation. K, the number of total trajectories (x-axis) vs the performance of the best policy the set (y-axis). Baseline corresponds to the performance of the single extrinsic only policy.
28

---

Published as a conference paper at ICLR 2023
E.4 BIPEDALWALKES
For the BipedalWalker environment, we either perturb the morphology or the terrain. To perturb the morphology, we follow (Ha, 2018) and specify a set of re-scaling factors. Specifically, each leg is made up of two rectangles, with pre-defined width and height parameters: leg, ((w) /t)), [w]. 11)) leg, = ((w), h2), (wg, hz)). To generate a perturbed morphology. we define a scaling range [0, withing which we uniformly sample scaling factors e, of ~ I-1, m/. for i = 1,2 and j = 2. perturbed environment is defined by re-scaling the default parameters: leg, = (((1 4. (1 of M)), ((1+ (1 (L+2)))))) and leg, (((1+ (1++))h)), ((1+ ((1+ (1+z2)43)))
The values for this perturbations can be found in Table 4.
Perturbation type
Perturbation scale parameter values (1))
Morphology
0., 00 0.15, 0.20, 0.25, 0.30, 0.35
Stumps (height, width)
0., 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.
Pits (width)
0., 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.
Stairs (height, width)
0., 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.
Table 4: Bipedal perturbation scale values
For terrain changes, we selectively enable one of three available obstacles available in the OpenAI Gym implementation: stumps, pits, or stairs. For each obstacle, we specify a perturbation interval [0, n). This interval determines the upper bounds on the obstacles height and width when the environment generates terrain for an episode. For details see the "Hardcore" implementation of the BiPedal environment. Note that for stairs, we fixed the upper bound on the number of steps the environment can generate in one go to 5.
To evaluate adaptation, we first train 10 agents independently on the "BiPedalwalker-vy" environment, which only uses flat terrain. To evaluate the trained agents we sample random perturbations of the environment. Specifically, for each type of perturbation (morphology, pits, stumps, stairs) and for each value of the scale parameter 7) we randomly sample 30 perturbations. We then run each option for 40 episodes; adaptation takes the form of using the first 10 episodes to estimate the option with highest episode return, which is then used for evaluation the remaining 30 episodes.
BiPedal Walker
400
No diversity
With diversity
-100
0.0
0.1
0.2
0.3
0.0
0.3
0.6
0.9
Morphology scale
Stump size
BiPedal Walker
400
-100
0.0
0.6
0.0
0.3
0.9
Pit size
Stairs step size
Figure E13: K-shot adaptation in BiPedal walker
Fig. F15 shows that, while performance degrades as the morphology is deformed, DOMiNO exhibits greater adaptability as evidenced by less severe degradation of performance. In terms of morphology, we find gradual decline in performance the Control Suite, diversity is beneficial and helps the agent adapt while not being impervious to these changes. In terms of terrain perturbations, these have more abrupt impact the agent's performance. While diversity does not prevent significant drop in performance, it is still beneficial when adapting stumps and pits and does not negatively impact performance in the case of stairs.
we increase the degree of deformation. Similar to
29

---

Published as a conference paper at ICLR 2023
F ADDITIONAL FIGURES FOR THE REBUTTAL
360
300
250-
206
000
400
36-
150
EPC loare fume
076-
250-
400
am.
76-
46-
220
200
150
100
500
200
100
300-
200
100
Figure F14: Reward learning curves (episode reward number of environment steps) corresponding to the results in Figure 2, center. Sub figures correspond to different optimally ratio (0.5 to 0.9 up to bottom), colors correspond to values of Eu.
30

---

Published as a conference paper at ICLR 2023
88-
03-
28-
84-
28-
03
66-
85
Figure F15: Diversity learning curves (episode reward number of environment steps) corresponding to the results in Figure 2, center. Sub figures correspond to different optimally ratio (0.5 to 0.9 up to bottom), colors correspond values of fn.
31


Reviews:
[{"review_id": "h-ux17qig1", "reviewer": "Reviewer_9fff", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a method for training diverse populations fo agents, using any RL algorithm, through a carefully constructed adaptive reward function.", "summary_of_the_review": "I'm recommending a weak accept for this paper due to the novel framework for optimizing diversity based methods, which appears to be practical for optimizing the objectives it was tested on. ", "strength_and_weaknesses": "Strengths: \n* Reformulating the problem of promoting diversity through Fenchel duality is novel and interesting\n* Using this reformulation to recast the problem as a multi-agent RL problem allows flexibility of the method as algorithms change\n\nWeaknesses:\n* It is difficult to understand the quality of the diversity metrics without more extensive qualitative evaluation.  One could expect a state-occupancy based diversity measure could be degenerate in the same way the noisy-TV problem effects surprise based exploration.  If put into a best buy full of TVs, it seems like each policy could learn to look at a different TV, ignoring many non-TV looking behaviors.\n* Sometimes multi-agent and single-agent problem formulations are confused, for instance in Section 3.3 the reduction is to a 3-player game, so it is not a reduction to an MDP, but to a Markov Game.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dhFHKSfW275", "reviewer": "Reviewer_PkBm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates how to learn near optimal policies with diverse gaits by augmenting a simple reward function in the domain with a diversity objective, which can be implemented as either constrained optimization, or as a multi-objective reward. The contributions of the paper include a diversity objective that repurposes the reward function from Abbeel & Ng (2004) to repel policies based on their expected feature similarity. Additionally, a second diversity objective is presented inspired by the Van Der Waals force, which extends the repulsive force to have a repulsive and attractive term---this is appealing because once learned policies are diverse enough, the Van Der Waals force is zero, and policies can purely maximize rewards.\n\nThe authors propose a framework for optimizing these objectives using Constrained MDPs, and provide ablations for a corresponding multi-objective variation of the objectives.The resulting method is called DOMiNO, and solves a constrained optimization problem, where the aim is to find a near optimal set of policies (where near optimal is defined in terms of recovering a fraction $\\alpha$ of the performance of the optimal policy), while maximizing the diversity objective.\n", "summary_of_the_review": "Overall, this paper studies an important problem in reinforcement learning: finding policies with diverse gaits that effectively balance diversity with task performance. The proposed method has appealing qualities: higher diversity, insensitivity to hyperparameters, and good task performance once diversity is attained. However, four significant weaknesses outweigh the paper’s strengths and consequently I cannot recommend the paper for acceptance in its current form. I am willing to reconsider my evaluation in light of new information and encourage the authors to address my concerns or provide clarifications where necessary.\n", "strength_and_weaknesses": "Strengths Of The Paper:\n\n1)\n\nThe diversity objective inspired by the Van Der Waals force is appealing because it encourages the set of policies to be diverse enough (as specified by a new hyperparameter referred to as the Van Der Waals contact distance), but not too diverse as to hinder reward maximization.\n\n2)\n\nProvided videos of the resulting policies show that diverse behaviors are indeed learned in all domains, which is particularly impressive in the DM-Control dog.walk domain.\n\n3)\n\nAblations show that the method is not particularly sensitive to the size of the set of policies over which optimization is performed, which is encouraging since tuning this hyperparameter could otherwise require knowledge about the number of different gaits the agent can learn in advance.\n\nWeaknesses Of The Paper:\n\n1)\n\nResults on K-shot adaptation when baselines are tuned for performance (see Figure E10 in the appendix), appear to show that SMERL with $\\alpha=1.0$ performs as well as DOMiNO in all environments, and occasionally performs better such as in “Walk+Torso length”. The authors note that SMERL has the additional hyperparameter of $c_d$, while DOMiNO does not since the method uses constrained optimization, which can be seen as adaptively turning $c_d$.\n\n2)\n\nK-shot adaptation is a misleading title since in practice, for a set of 50 policies, if 10 trajectories are collected for each policy, this results in the agent collecting 500 trajectories. In the spirit of k-shot evaluation, this experiment could be made stronger if results were reported as K varies. Additionally, the authors are encouraged to be clear about how many environment transitions are collected in all K-shot adaptation experiments in the paper for reproducibility.\n\n3)\n\nWhile the ablations presented are sound, and the proposed framework is novel, the empirical results in the paper are weak: the proposed method, DOMiNO, does not perform distinguishably better than its alternatives, especially the SMERL implementation. The additional work of using constrained optimization, including a “cost player us[ing] the Follow the Leader (FTL) algorithm” (Page 3) does not provide a benefit over the simpler SMERL, which uses standard RL. Evaluation on more tasks could provide evidence to counter this interpretation, and I encourage the authors to provide additional results and clarifications on the comparison to SMERL.\n\n4)\n\nThe proposed diversity objectives are not compared to other diversity objectives in the literature, such as DIAYN (Eysenbach et al., 2019), and MaxEnt RL (Haarnoja et al., 2018), which makes it hard to judge the utility this paper provides over existing methods for diversity maximization. It could be the case that the proposed method better controls the tradeoff between maximizing rewards and maximizing diversity (which may hinder reward maximization) via the Van Der Waals force. However, the paper is missing such a comparison, and it should be added.\n\nSuggestions For The Paper:\n\n1)\n\nOn page 5, the authors write “one could consider other combinations of powers and coefficients” for the diversity objective defined in Equation 7. While not necessary, and not currently negatively affecting my review (see the listed weaknesses above), the paper could be made stronger if an ablation of different powers and coefficients was given in the Appendix.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "agzw6IuMCap", "reviewer": "Reviewer_vof2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of learning a set of diverse policies that are simultaneously nearly-optimal for a given reward function. In their formulation, the diverse near-optimal policies are learned by maximizing an objective function related to diversity, while near-optimality is ensured through a linear constraint on the value of each policy. Crucially, the diversity between the policies is measured through a distance (either L2 or Van Der Waals force) of the expected state-action features they induce. The paper proposes to optimize the introduced problem through a Lagrangian relaxation that can be seen as a three-player game. Finally, the paper provides an experimental analysis in the DeepMind Control Suite to show that\n- the proposed approach leads to the discovery of diverse nearly-optimal behaviors, where near-optimality and diversity can be traded-off in various ways;\n- the proposed Lagrangian approach dominates a simple scalarization of the diversity and near-optimality objectives;\n- the set of diverse nearly-optimal policies allows for efficient k-shot adaptation to reward perturbations.", "summary_of_the_review": "The idea proposed in this paper is interesting and original, and it has interesting connections with several previous works in the literature. On the one hand, it can be seen as a formalization of skills discovery methods when the reward constraint is set to void, whereas it is related to constrained convex RL otherwise, and to multi-objective RL if we scalarize the combination of the objective and constraint. It has also technical similarities with apprenticeship learning and state-based imitation learning.\n\nHowever, the provided theoretical ground for the approach is somewhat weak: It is not clear if the proposed optimization problem is tractable, and what kind of guarantees can we have optimizing it. The algorithmic contribution is also relegated to tiny bits of the paper, while it should be a main contribution to my understanding.\n\nOverall, my current evaluation for this work is borderline, but I am open to raise my score if the authors could address my comments above.", "strength_and_weaknesses": "*After discussion*\n\nHaving read the authors' response and other reviews, I am changing my score from 5 to 6.\n\n----\n\n*Strengths*\n- (Novelty and relevance) The paper introduces an interesting objective that measures the diversity between the policies through a direct distance in the space of the state-action distributions. While previous works have tackled diverse skills discovery, this direct formulation is, to the best of my knowledge, novel;\n- (Experimental analysis) The proposed approach is evaluated in challenging domains, and the experiments' section includes nice visualizations and interesting findings.\n\n*Weaknesses*\n- (Theoretical ground) The theoretical ground for the proposed framework is weak. Especially, the paper proposes to maximize non-concave objectives, and convergence guarantees for the approach are questionable;\n- (Algorithm) The paper dedicates little space to the presentation of the algorithm, which should be a relevant contribution per se given the novelty of the problem, and the corresponding technical challenges (e.g., how the objective function can be estimated from samples); \n- (Comparison with previous works) Whereas the authors made clear that the experimental analysis is not intended to demonstrate the superiority against any baseline, previous works addressed both reward robustness as well as diverse skills discovery, and even an indirect comparison could have been telling.\n\n*Detailed Comments*\n\nOptimization Problem\n\n(C1, Major) The discovery of near-optimal policies is formulated through a maximization problem of a non-concave function, e.g., the L2-distance between the expected state-action features. While the objective is at least convex, and thus the gradient ascent procedure may work fine within the space, it is unclear what happens at the boundaries. This problem has been pointed out before, such as in (Eysenbach et al., 2021), which is mentioned in the paper. Especially, I am wondering whether the optimization problem proposed in the paper is tractable even when the underlying MDP is fully known. Can the authors assess the computational complexity of the problem?\n\n(C2) Also related to the previous comment: The optimal set of policies is attained by deterministic Markovian policies in this setting? To my intuition, if I am just maximizing the L2-distance between the occupancies of two policies, I should converge to deterministic policies. However, it is not obvious to me what happens with more than two policies and the value constraint.\n\n(C3, Major) Even ignoring the potential intractability of the problem, I am wondering what kind of game-theoretic guarantees do we have if all the policies are optimized simultaneously via gradient ascent. Have the authors some ground to say that the optimization would converge to some notion of equilibrium?\n\n(C4) The paper proposes two different objective formulation, either the L2-distance or Van Der Waals force. Can the authors better motivate why these two alternative have been chosen over other distances/divergences? Do we necessarily need a proper distance for the objective function? KL-divergence, which is not symmetric, could work as well?\n\n(C5, Minor) The problem is formulated through a maximization of the diversity objective under the value constraint. Do the authors considered also an alternative formulation though a set of problems in which the objective is the value of a policy under a diversity constraint? Would this formulation change the nature of the problem to make it (possibly) tractable?\n\nMethodology\n\n(C6) The methodology is not clearly explained in the main text, as it is mostly relegated to the Figure 1a, in which the architecture is presented, and a few lines at the end of Sec. 3.3 and the Agent paragraph in Sec. 4. Can the authors describe the methodology in details, and what kind of peculiar challenges it has to address?\n\n(C7) Especially, I am wondering how problematic it can be to estimate the objective function in practice. Estimating a distance/divergence through density estimations is known to be cumbersome in continuous/high-dimensional domains. Do the authors validated their estimators empirically?\n\n(C8, Minor) Previous works in imitation learning (e.g., Schroeker and Isbell, State aware imitation learning, 2017; Schroeker et al., Generative predecessor models for sample-efficient imitation learning, 2018) have considered the gradient of the L2 distance between state-action occupancies. Clearly, they follow a gradient descent procedure instead of the gradient ascent proposed in this paper to maximize diversity, but the gradient formula could still have some common traits.\n\nExperiments\n\n(C9, Major) Whereas the authors explained that they do not intend to compare their approach to any baseline, it is quite hard to actually evaluate the results, as we do not know how far the obtained policies are from the global optimum. Especially, \n- Can the authors show the value of the objective during training, to understand whether the algorithm is actually improving the value of the objective with a monotonic trend?\n- Can the authors show that the algorithm is converging, or do they just stop the training at some point while the policies would continuously change?  \n\nRelated Works\n\n(C10) While there is not a clear baseline for the proposed approach, as the problem formulation is novel to my knowledge, there exist previous methods that addresses similar problems. Especially,\n- Maximum entropy regularization for RL is known to provide policies that are reward-robust (see Husain et al., Regularized policies are reward robust, 2021);\n- The proposed objective with a void value constraint is similar in nature to the problem of (unsupervised) diverse skills discovery (e.g., Eysenbach et al., 2019).\n\n(C11, Minor) Another recent work (Mutti et al., Reward-free policy space compression for reinforcement learning, 2022) comes up with an optimization problem that have some similarities with the one proposed here, although the underlying motivation is different.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "E2S0y4gj_0i", "reviewer": "Reviewer_oggL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of jointly learning a diverse set of (nearly) optimal policies within a single RL environment. The paper follows the SMERL paper and adopts a novel mathematical framework based on convex MDP and Fenchel duality. The derived algorithm shows that we can simply solve the constrained optimization by designing an intrinsic reward based on the successor feature of the learning policy and the gradient of the diversity measure.", "summary_of_the_review": "This is a technically interesting paper with solid experiments, although there still exist some flaws. Assuming the paper will be improved by the authors, my current perspective is leaning towards acceptance since I did learn something from this paper.", "strength_and_weaknesses": "## Strength\nI do enjoy reading the paper from a technical perspective and feel really interested to see the conclusion that the derived algorithm can be simply built upon the gradient of diversity measure, which is novel in my opinion. \n\nThe experiments are sufficient and solid from my perspective. The section contains sufficient ablation studies, emergent behaviors as well as the full suites of experiments in the original SMERL paper. \n\n## Weakness\nMy major concerns are about the presentation of this paper. Some of the details are listed below.\n\n1. The title is definitely a bad one. I do understand the space is pretty limited, but I still think the authors could have done a better job to come up with a more informative title, i.e., at least including the terms \"diverse\" and \"nearly optimal\". It is really a misleading title, although I do know this will be addressed in the camera-ready version. \n\n2. I have to express my disagreement with the statement \"we do not explicitly compare it with other work nor argue that one works better than the other.\" This is really an awkward statement in a paper, particularly knowing that it is even presented twice! I could understand that there could be some reason, but I do believe there are better ways to position this work in the literature. Note that you do compare with SMERL and outperform it! It is completely okay to say this work is better than SMERL, isn't it? Also, wouldn't it be natural to include a few SMERL follow-ups as additional baselines for improved soundness? The only tricky part of baselines that I could imagine would be those diversity methods on a swapped objective, i.e., maximizing the extrinsic reward with a diversity constraint. This line of research is related but, in fact, parallel to this work and will result in a completely different solution set. More interestingly, the algorithmic framework here will no longer hold if the diversity and reward objectives are swapped. So, I would suggest the authors spend some texts discussing the difference between the two lines of work (i.e., SMERL v.s. diversity MARL) in the main paper for better paper positioning rather than repeatedly leaving such an awkward execuse. \n\n3. **A technical flaw in VDW objective**. I want to point out the derivation holds under the assumption of convex MDP. However, in equation (7), the diversity metric induced by the Van der Waals force isn't really convex over the entire domain --- it is only convex in a sub-space. Although in practice, it seems that this doesn't really matter much, I do think this part should be fixed or at least be discussed further. \n\n4. **novelty and positioning**: A large body of the content, particularly the algorithmic framework, directly follows [1]. In the current draft, the whole related work section is deferred to the appendix without a careful discussion of the relationship between this work and [1]. To me, it looks like the authors adopt the framework from [1] and apply it to the SMERL setting. I do think this should be explicitly presented in the introduction section. \n\n[1] Reward is enough for convex MDPs, Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, Satinder Singh, NeurIPS 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster