Discovering Creative Behaviors through DUPLEX: Diverse Universal Features for Policy Exploration
Authors: Borja G. León
Francesco Riccio
Kaushik Subramanian
Peter R. Wurman
Peter Stone
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Policy Diversity
Generalization
Primary area: reinforcement_learning
TL;DR: Novel algorithm for learning diverse near-optimal policies capable of generalizing within and out-of distribution

Abstract: The ability to approach the same problem from different angles is a cornerstone of human intelligence that leads to robust solutions and effective adaptation to problem variations. In contrast, current RL methodologies tend to lead to policies that settle on a single solution to a given problem, making them brittle to problem variations. Replicating human flexibility in reinforcement learning agents is the challenge that we explore in this work. We tackle this challenge by extending state-of-the-art approaches to introduce DUPLEX, a method that explicitly defines a diversity objective with constraints and makes robust estimates of policies’ expected behavior through successor features. The trained agents can (i) learn a diverse set of near-optimal policies in complex highly-dynamic environments and (ii) exhibit competitive and diverse skills in out-of-distribution (OOD) contexts. Empirical results indicate that DUPLEX improves over previous methods and successfully learns competitive driving styles in a hyper-realistic simulator (i.e., GranTurismo ™ 7) as well as diverse and effective policies in several multi-context robotics MuJoCo simulations with OOD gravity forces and height limits. To the best of our knowledge, our method is the first to achieve diverse solutions in complex driving simulators and OOD robotic contexts. DUPLEX agents demonstrating diverse behaviors can be found at https://ai.sony/publications/Discovering-Creative-Behaviors-through-DUPLEX-Diverse-Universal-Features-for-Policy-Exploration/.

Discovering Creative Behaviors through DUPLEX: Diverse Universal Features for Policy Exploration
Borja G. Leon" Iconic bor corjjaeico ai
Francesco Riccio
Sony AI ricciolsony......oo.....o.o..... francesco.riccio
francesco.
com
Kaushik Subramanian Sony AI
Peter R. Wurman
Sony Al
kaushik. kaushik. subramaniannaony com
peter wwweerwwwaannnnnncooom com
Peter Stone
Sony Al
The University of Texas at Austin peter. www.eesonn..n.ocooomm
Abstract
The ability approach the same problem from different angles is comnesstone human intelligence that leads to robust solutions and effective adaptation problem variations. In contrast, current RL methodologies tend to lead policies that settle on a single solution to a given problem, making them brittle problem variations. Replicating human flexibility in reinforcement learning agents is the challenge that we explore in this work. We tackle this challenge by extending state-of-the-arr approaches to introduce DUPLEX, method that explicitly defines a diversity objective with constraints and makes robust estimates of policies' expected behavior through successor features. The trained agents can (i) learn diverse set of near-optimal policies in complex environments and (ii) exhibit competitive and diverse skills in out-of-distribun (OOD) contexts. Empirical results indicate that DUPLEX improves over previous methods and successfully learns competitive driving styles in a hyper-realistic simulator (i.e., GrannTurismo" 7) as well as diverse and effective policies in several multi-context roboties MuloCo simulations with OOD gravity forces and height limits. To the best of our knowledge, our method is the first achieve diverse solutions in complex driving simulators and OOD robotic contexts. DUPLEX agents demonstrating diverse behaviors can be found https: //ai httpreeeee-naaaaaaarraurer-r--o- Explorationo DUPLEX-
Introduction
In non-stationary complex environments, reinforcement learning (RL) 1 agents compelled exhibit flexible and diverse behaviors to robustly adapt to different scenarios and interact with other actors 3. To this end, a growing community researching methodologies to train agents that, unlike conventional RL [4,5], can solve tasks with a diverse set of near-optimal strategies [6]. Such methodologies are explicitly crafted to enhance the exploration of the state-action space, equipping
"Wor done Sony AI.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Training contexts
DUPLEX
cnlicc exinnsic reward
context
distriution
[<)-C
cffic
Bo
C₁
policy
Torso
Intrnsic reward
Ch
(2) (onchorta])
SUccessor
features
K
Input
encodings
Environment
Figure 1: DUPLEX data flow while training three policies over a set of two contexts. At each iteration, we provide three inputs our multi-policy agent: (i) context vector describing task requirements and environment dynamics in the current episode c; (ii) an encoding of the policy used in the episode z; (iii) and the current state of the environment The critic network returns estimates for the intrinsic and extrinsic rewards and successor features to drive diverse behavior discovery. Finally, the algorithm samples policies in II uniformly and rolls them out collect more experience.
agents with the capability to discover qualitatively diverse solutions for given task distribution. For example, when planning a route from home to work, a human commuter might identify one route that uses the highway and one that sticks to side streets. While using the highway may be optimal in expectation, using the alternative route may be called for different contexts that influence the traffic report, or weather forecast.
Existing approaches struggle to generalize to highly dynamic environments and are designed to learn in single-task settings 3 7 To alleviate this, we introduce Diverse Universal features for PoLicy EXploration (DUPLEX), an algorithm that trains an agent to optimize set of diverse policies over different contexts. In this work, contexts comprise two components: task requirement and description of the current environment dynamics. To better ground such concepts, let us refer to the previous commuter example. In this case, would refer to task requirements time-to mome-tto-oopltionn On the other hand, factors such as the weather forecast and traffic report would form part of the environment dynamics DUPLEX builds top of the strengths of diversity learning 66 in RL and to introduce a novel methodology that preserves both performance
universal estimators (UE) S [9] and diversity in highly dynamic environments and multi-con human-level task execution. For example, if the commuter is human that injures their leg, they can immediately balance on one leg and even jump forward without using the injured leg. We aim transfer this adaptability to agents in different contexts. To this end, we adopt the Contextual Markov Decision Process (CMDP) framework [IO), where different episodes correspond to different contexts. As shown in Figure ! DUPLEX is designed to receive three inputs representing the encoding of the context, policy, and state. Contexts are represented feature vectors and uniformly sampled from discrete set of predefined task requirements and dynamics settings (c.g., weather condition); the i-th policy in the set of policies comes directly from the environment observation. Then, the critic output is split into different heads support the estimation of the extrinsic and intrinsic rewards, and successor features (SFs) [T]. We refer the reward coming from the environment as the extrinsic reward, while the metric computed to promote diversity the intrinsic reward. It is worth noting that, at the beginning each training episode, we uniformly sample a context from C' and a policy from IT to collect experience. DUPLEX then iteratively trains the set of policies to maximize both the distance in their successor features (guided by intrinsic rewards) and the task performance (given by the extrinsic rewards). We rely on SFs to measure distances as they, by definition, represent the state features that given policy is expected to experience along trajectory, and thus, they are intuitive way to quantify policy diversity.
settings which is key to enabling
selected and passed to the model (e.g., different routes); and the state
Hence, our main contribution is DUPLEX, novel algorithm that contributes to diversity learning in RL by improving on previous work to better preserve the diversity vs. near-optimality trade-off in highly-dynamic environments and multi-context settings. We evaluate our approach the real-time, physically realistic, car-racing simulator GranTurissmo" 7 (GT) [22 and on two multi-task MuJoCo environments with changing dynamics (Walker2D, Ant) [3 Our experimental results indicate that

---

DUPLEX improves over previous ofate-of-the-art RL diversity baselines [6] and UE baselines 8 9 In fact, DUPLEX is the only algorithm learn diverse competitive policies in GT and outperforms other baselines when evaluated in out-of training distribution (OOD) contexts. Moreover, we conduct a detailed ablation study that isolates the effects of each of DUPLEX'ss main components.
2 Related Work
In this work we build on top of two main bodies of research: diversity learning and universal function approximators. We organize related work accordingly.
Diversity. Diversity learning is gaining attention within the research community, due to the im- portant benefits it yields to autonomous agents. Such benefits include generalization [14], explo- ration 15 16], creativity [3) and self-play (author?) 22. In this work, we extend the application of diversity learning highly-dynamic environments and multi-context settings. A central precursor to our work is DOMINO [6]. DOMINO employs RL to maximize extrinsic and intrinsic diversity rewards, which are combined with Lagrange multipliers and Van der Waals (VdW) forces balance diversity and performance. DOMiNO has two variants: one bases its diversity objective on the average of input features which grants stability at the cost of storing a moving average for every task - limiting scalability. The second variant uses the critic to estimate SFs. which solves the scalability issue at the cost of instability due to the added learned target. Intuitively, such limitations prevent DOMiNO from being effective at discovering diverse behaviors in CMDPs. Further, the Lagrange multipliers and VdW forces can limit diversity in complex environments where creative behaviors need significant exploration before providing satisfying returns
[17] Such limitations
become more evident in highly dynamic environments such as GT (see Section 5 Our algorithm is
instead designed to tackle these limitations.
Quality-Diversity (QD) is another active line of work within diversity learning that involves evolu- tionary methods. QD aims to generate large collections of diverse and high-performing solutions, primarily through evolutionary optimization 18 19]. In QD, diversity is measured through a de- scriptor space defined by the user (20). Some of the main approaches in QD include MAP-Eliter
21 (22] local competition
23 and more recently those that combine RL with QD
24 25 26).
Closer to our research, (author?) 27 introduce a variant for MAP-Eliter to learn diverse behaviors for multiple tasks within the training distribution. Our framework diverges from QD in several ways. First, QD algorithms emphasize diversity through evolutionary strategies, while our method relies on maximizing reward objective. Second, QD requires the user to define diversity descriptor, while we only restrict diversity to be in the near-optimal space. Third, one of our main goals is finding competitive diverse behaviors for OOD tasks and dynamics, an objective that has not been studied in Quality-Diveriity (QD).
Universal Function Approximators. To make more robust estimations of the policies' expected behaviors, and thus to better quantify diversity, our work pivots around universal function approxima- tor (UE). UE research is grounded on factoring the value estimates separating states from tasks g and policies [2 Notably, (author?) (9] improve the formalization of successor features (TI) an estimation of state-action visitation, by conditioning their estimation on both task and policies. Still, approaches based on SFs are brittle in complex domains [29] 30 requiring sophisticated network
architectures [31] 32] to work effectively. Akin to (author?) B3 we adopt similar approach to
transferring learning of SFs to continuous domains by combining them with SAC [4. However, we achieve an improved estimation of the expected SFs by employing the average of the critic outputs and by adding the entropy term to the SFs learning objective. We find that the addition of the entropy term is novel component that DUPLEX carries with and it is key to improving robustness and performance (see Appendix B
3 Background and Notation
We briefly introduce the main building blocks of DUPLEX and the notation we adopt. First, we review basic concepts of multi-task RI. and explain how it can be represented by the contextal-MDDP framework. Next, we describe how universal estimators are used to enhance generalization across contexts (especially when context enumeration is impractical).
3

---

Multitask RL. We consider training a multi-policy RL agent that solves CMDP [00] represented as a tuple M AS. P. R. Y, (C)) where S,A are the state, action spaces respectively. R is the reward function, P is the unknown transition function, jss is the initial state distribution conditioned on the context, and μCc is the context distribution. We use context с = {u, w) to summarize both information about the particular dynamics u of that environment (e.g., the effect of gravity), and information about the task w in the current episode (e.g., position constraints). Every episode starts with a context ~ MC and an initial state so uus( c). Then, at each time step i, the agent selects an action d, according to its policy a(. 88,, c), receives a reward r: R(ss, de, c) and transitions to the next state $1+1 P(- 8t, Qe. c). Policies are characterized by their state-action occupancy, i.e., how often a policy visits a state-action action pair. We consider two state-action occupancy metrics: dx. (s, a) = lim 1 Ei= Pa (st = 5) <(s, a, c) for the average T
occupancy metric and dG. (s.a) (1- -ー) Lt= 7'p (st s)m(s, a, c) for the discounted case - where IP is the probability measure of states at induced by ι in c. The objective of our algorithm is to find set of diverse policies that maximize the expected return in every context maxd, EK Ex,,e r(s, a, c)da, (s, a). where K. is the set of admissible distributions [34.
Universal estimators. When aiming to solve multiple tasks, we follow the common approach of decomposing the input of the neural network to facilitate transfer learning between tasks and policies. Methods that are relevant for our work include "Universal Value Function Approximators" (UVFA) [8] which add a task- desk-descripto vector w as input to a value function approximator parameterized by 0, Ve(s, a, w). If Ve is smooth w.r.t w, then Vi is expected to generalize across tasks within the training task space. Akin to previous works [T] 6) we assume that every state-action pair is correlated with observable features known as "cumulants" áfs, a, Rd. These cumulants can either be given through relevant properties within the state or be learned. In a similar fashion to value functions, we define expected features Uh, (s,a)
(s.)), Ø(s a', c) Rd which we will refer to as average (1)47)) or discounted (1²7) expected features when using days and dy respectively. In the discounted case, vp7 are also known as "successor features" (SFs). This latter formalization is key to our work and also to USFA (9) general framework that combines UVFA-like task decomposition with the SFs, and exploits generalized policy improvement algorithm (GPI) (author?) [I]] Importantly, USFA disentangles both tasks and policies by giving as input a vector z that is a representation of the current policy z = e(TT), where e an encoding function in our approach e encodes policies as one-hot vectors z.
4 Learning Near-Optimal Diverse Behaviors in Multi-Goal Continuous Settings
DUPLEX a method designed to provide diverse solutions to complex tasks by simultaneously maximizing both the true reward and dissimilarity within set of policies II across the different environments of a CMDP. Thus, according to previous work [6] we define diversity as:
Definition 4.1 (Diversity) Diversity is a metric of dissimilarity among policies in set II with a common goal. Formally, if Hh, and WH are a function of state-occupanc of relevant features of any two policies in 11, then their dissimilarity is given by llv., L,,, non-zero value of this norm indicates dissimilarity, with larger values indicating greater divergence between the policies. Mathematically, diversity is defined as the sum of the minimum L2 dissimilarity norms in II:
Diversity(II) 2 (II)
Σ Val.aell (1=
min Ilpm 4ay 112
Using this definition, we formulate our learning problem and describe DUPLEX's s main contribu- tions in the following sections. We first extend the formulation of diversity learning to context- conditioned environments, then we introduce novel mechanism to more efficiently control the trade-off, and finally, we describe how to improve SFs estimation.
FFor example, in car racing scenario features could correspond to relevant information for the car such as the course velocity if has crashed into another

---

4.1 Context-conditioned Diversity Learning
Similar to (author?) [6] our objective is to maximize diversity within a set of policies II. We express the distances between policies in II in terms of expected features th, which are function of their state-occupancy (1 (0). We measure v distances following Def. [4.1 [4d and use a context-based Hausdorff distance (35 to enforce context-conditioned diversity within IT. We aim at training an RI.
agent that, given context discovers set of n. near-optimal policies II = {=]] j= by successfully
optimizing:
max Diversity (II) s.t ds. Te 2 pūer VT E 11
(1)
where dx is the occupancy metric of a context-conditionel policy, re is the environment reward (or extrinsic reward), p [o, 1] is a pappmm-e defining the near-optimality region that we refer to as optimalit--ratio, and é. is the value of a target policy. In practice, the target policy refers to a policy in IT that ignores the diversity objective and is trained exclusively to maximize the extrinsic rewards. As in a policy т. According to Definition 4.1 diversity is a distance over set of occupancy metrics Diversity: {IR (9] Al } I and Eq. is defined maximize the diversity of the occupancy metrics and preserve the near-optimality of the policies in II.
[6], the policy value e expresses the expected reward accumulated by
To promote diversity between different policies, we adopt the repulsive reward from 66 and extend it to be conditioned on the context. More formally, given the tuple (s, a, x², x',c) and a cumulant function Φ, the diversity reward that we maximize for is:
(s, a, c) = Φ( a, c) (Wre - thrg)
where the are the expected features from policy nO E IT at (s, c) and म.. E II refers to the policy with the closest expected features to T' in (s, a, c) according to Def Intuitively, since contexts e ~ PC are fixed for each training episode, this reward encourages the algorithm to train policies that visit different state-action pairs within each context c, thus, promoting context-conditioned diversity.
4.1.1 Stabilising Diversity across Different Domains
Eq. expands DOMiNO's repulsive force to operate across different contexts. However, the scale of intrinsic rewards varies greatly through different environments (i.c., different contexts will inherently yield different successor features). We need to stabilize fluctuations of such rewards when working with diverse contexts. We introduce two constraints to the learning objective that modulate the intrinsic reward ry: dynamic intrinsic reward factor x that scales ra to target a factor of the moving average of the general extrinsic value and a soft- soft-lower bound that limits the search of diverse policies to near-optimal subspace. We then define the DUPLEX intrinsic reward as:
ri X.
where x and À are computed independently by the algorithm.
Dynamic intrinsic reward factor. X scales ra proportionally to the sum of extrinsic values of policies in IT. Formally, ra X &Vewg = Σ
iim1 o with v'_... = &ravg v'n.. + - (l--owww),, where, Dewg E [0, 1] weights the contribution of the extrinsic value of the i-th policy and its immediate extrinsic reward. Finally, re. is the extrinsic reward at when the agent acts under policy i. Intuitively, we are scaling the intrinsic rewards according to the average extrinsic value that the set of policies is achieving. Hence, at each algorithmic iteration, x is updated follows:
X = = + (1- (1
where ax is the update rate of X. and X' Vewee/ //dww/ (1 (1-p) is the target value of the update, where Udave is the return value based on diversity intrinsic reward ra while Vewwy is based on the extrinsic reward r. instead. Through the optimality ratio p. X minimizes the domain dependency by scaling the intrinsic rewards as factor of the extrinsic objective and the optimality ratio p. Regarding the update rate ax- once we converged to a stable value, we kept fixed across all experiments and domains.
5

---

Soft-lower bound. While preserves relationship between extrinsic and intrinsic rewards, it does not prevent policies in II from exploring regions of the search space that are too far from the near- optimality regions of the target policies. To limit this possibility, we explored Lagrangian-constraindd optimization but found it unsatisfactory in complex domains like GT. Hence, we introduce bound the near-optimal subspace for each policy using:
{ 0h
"
A
(5)
|Beave +
where (*) = e1(1+e-kkr and 3 E [0, 1] is a hyper-paramett indicating the reward region we interested in exploring, regulates how "soft" is the bound, is the number of policies, is small constant to prevent division by zero, and Vewwg is the average extrinsic value. Intuitively, limits the exploration of diverse behaviors to near-optimal area defined by the threshold B. We find that introducing sigmoid- based limit provides a more stable solution than Lagrangian constrained optimization (See Figure 2p. Similarly ox. once we found stable value for k, it remained fixed for all experiments.
4.2 Estimating Successor Features
Since rely d to determine policy diversity (Def. 4.1p. it is fundamental to estimate reliably, Additionally, as mentioned in Section 3 is not tractable to keep ग for each context and task since they may be infinite. Thus building top of UE, exploit discounted expected features (SFs) (s, a, 2, c) and incorporate extra head in the critic to estimate SFs (), (6,a,22, 2,
However, in related work [30, we found that the critic struggles to correctly estimate xf in complex settings. Specifically, since intrinsic rewards are bootstrapped over the expected difference of successor features, the accuracy in the SFs estimation plays major role. Taking inspiration from the SAC algorithm [4. we incorporate an entropy term in the expected features objective to account for the stochastic component within the learned policies.
For policy i context c, our SFs at 8, are computed as:
w ( (st, ar, c) c) Φ. + Exg
" an (T
(6)
A=++1
where a is the entropy weight. Intuitively, unless the critic is confident that the actor will visit a state-action pair more frequently than others, the network will be encouraged to estimate that the policy will maximize the entropy. We formulate our temporal difference loss for the critic output 7 account for the entropy term as:
[] (s,,, c,, v(o s(,)
TD#(07 (+ (+))) (c.0))-TE
E
(7)
where, similar SAC, we have two critic estimates j = 2). but instead taking the minimum of the two, our target is obtained from the average of the estimates. Then, we define v(á, 8', s',c,22 22 as:
v s ,,, = ((t) + avg Voun, (8', (o',",,)) a log ('.c) (=1,2
(8)
with ă'. - <55-11' c).. The motivation lies in the different impacts of overestimation of values and successor features. When predicting values, overestimation directly influences the policy since is trained to maximize advantage. However, in DUPLEX, SFs assume a distinct role: the policy is not incentivized to select actions with the highest successor features but rather those that differ the most from the SFs of the remaining policies (subject not sacrificing too much performance, cquation] D. Consequently, precision becomes of greater significance for y², and taking the average emerges as more reliable solution.
The introduction of entropy improves stability in the estimation of SFs through most of the training process and, as we report in the experimental section, it supports OOD generalization. However, as learning stabilizes, we typically encounter phases where the estimated difference between policies rapidly increases its order of magnitude. For this reason, we also introduce a fixed upper bound to ox when estimating SFs. Details on the upper bound, DUPLEX pseudocode, and hyper-parameters are included in Appendix C and D

---

20
Reward
Num. active policies
28
26
24
DUPLEX (8=0.97)
aug
22
peliyy
DUPLEX (8=0.0)
DONINO_NVD (Vg
DOMINO (avg. p=0.01)
GRSAC
20
0.0
0.2
0.4
diversity
0.6
0.8
1.0
spech
30
79
285
240
26
(a) Rewards vs Diversity
(b) Active Policies
Lap time
350
(8-0.97)
DUPLEX (8=0.0)
DOMINO (avg. pm0.01)
249
022
09:54
policy Index
(c) Lap Times
Figure 2: Results in GT. is the reward- diversity trade off, DUPLEX/DOMINO best bese__peliccttyy__peetppeeee refers to max, (T E II) of every run, while DUPLEX/DOMINO _aa refers to avg., (л E II) of that run. Differently from DOMINO, DUPLEX is able generate both competitive and diverse behaviors. Figure b) illustrates the number of policies that finish laps (active). Figure c) min lap times of the diverse policies. Note that the number of active policies is important showease that all policies actively searching for near-optimal diverse hehaviors through different epochs. Applying our soft lower bound (3 + 0) is key focusing diversity in the region of interest.
Summarizing, DUPLEX enhances diversity learning in context-conditione environments through four key components: (i) dynamic intrinsic reward factor (x) that balances diversity with extrinsic rewards and avoids environment-specifn tuning: (ii) soft lower bound (X) that constrains policies to optimize for diversity only within a target near-optimal region; (iii) entropy regularization (a.)) in successor feature estimation improve robustness; and (iv) averaging over critic estimates for computing diversity rewards. In Section 5 we demonstrate how these features improve both performance and diversity of the learned policies.
5 Experiments
The objectives of our experiments are to demonstrate that DUPLEX: 1) is the first successful method in learning diverse driving styles in compared against previous state-of-the. -ftte-f--he-art diversity approaches in RL, yields a better performance vs. diversity trade-off in canonical physics simulators (Figure 3); 3) exhibits diverse and effective behaviors in OOD dynamics and tasks (Figure 4): and finally, 4) improves on previous baselines in GT
physics simulators such GT (Figure when
estimating SFs (Figure 5) and 6). All results in MuloCO are obtained from five training experiments are more computationally demanding requiring a minimum of seven days to converge, so the results are obtained from three training runs.
Baselines. Our main baseline DOMiNO (6), which is the state-of-the-art diversity learning in RL. Then, since DOMiNO is not designed for OOD generalization, our experiments in OOD settings include as baselines the two most popular frameworks within UE literature: UVFA & and USFA 19.. Note that, throughout the section and given the two versions of DOMiNO, we will refer to
7

---

Results from max, (rr E (])
Results from avg, (I7 (1)
7000
SaPuEX ant p-0.
DUTLEX pmbls
8000
DOMIO ped)
SCwBK 2-05
5000
DOMBO pio
Reward
8000
Reward
4000
pno VoRL
p.0
9000
D. DJPEE pwd
OUFEDA 2
8000
2000
2000
DONIRO
1000
DONIOO yeu HH
DONIRO prb d0010
DOMIN pn pwbli VOML
0.0
0.2
0.8 10
Diversity
Diversity
Figure 3: Walker single-task results. Both DUPLEX and DOMiNO do not estimate SFs and use On the left, we show results from the best policy each variant while on the right, the average reward across all policies. Dots and lines represent mean and standard deviation, respectively. It is worth reporting that the vanilla SAC implementation scores 5576. 256 783 when evaluated the same training epoch.
MulkiTask Walker Treining
MulliTask Walker OOD Dynamics
MultiTask Walker OOD Tasks
1500
2500
6000
2000
Reward
4000
noward
1000
1000
DUPLEX
1000
2000
500
0.8
Diversity
Diversity
Diversity
MultiTask Ant Training
MultiTask Ant OOD Dynamics
MultiTask Ant OOD Tasks
3500
DUPLEX
EX
DUPLEX
3000
INVA
Reword
4000
E 2500
gawara
2500
2000
DUPLEX
2000
2000
UVEA
USSTA
1500
1500
DCMBD
-2000
6.0
0.2
0-4
0.5
0.8
1000
0.50
0.05
0.10
0.15 0.20 0.23 0.30
1000
0.00
0.05
0.30
0.15
0.30
Diversity
Diversity
Diversity
a) Training
b) Zero-shot (Test)
Figure 4: Results in MuloCo Walker (Top) and Ant (bottom) multitask environments. On the y-axis, we report the reward accumulated by the agent while on the x- x-axis normalized diversity according to Def. D.. Ideally, we want to be as close as possible to the top right region. OOD dynamics refers to gravities that are at least 40% stronger or weaker than the strongest and weakest gravities seen in training, respectively. OOD tasks represent tasks where the agent needs to walk forward below a height 20% lower than the lowest height seen in training. Only DUPLEX and UVFA are evaluated in the Test since the other algorithms already failed in training.
DOMINO any as the version of the algorithm using averaged cumulants as successor features, and as DOMiNO.sat est as the version of the algorithm that estimates SFs.
Benchmarks. We use three benchmarks. The first is a racing track in GT, where the goal is have different driving styles while completing fast laps. Achieving diverse and competitive driving styles in such realistic simulator [12 can unlock multiple applications both in gaming and self-driving Second, set of experiments is conducted in the MuloCo's 's Walker2D and Ant environments where we compare DUPLEX against DOMiNOavg in the same environments used in 6). Finally, in the last benchmark, we evaluate our approach in a multi-context version of MuJoCo's Walker2D and Ant environments. We extend the default versions of these scenarios to include three-dimensional context where the inputs represent: the gravity coefficient and, the upper and lower height thresholds we would like the agent to respect while moving. The goal of this benchmark is to find diverse policies in OOD contexts while guaranteeing competitive performance in task execution. Note that,
8

---

Ablations with yd from max, E (
Ablations with y from avg,
7000
6000
4000
Reward
5000
Reward
1500
3000
4000
2500
DURLES whood byp LAgccive
DJPLE with enterfl
2000
DU oin
DUNDS witeed obacth
Donna was ranoe,
2000
DunLEs wan gom
DiN 10 witeet Soedhing
DUNES eit
0/4 Diversity
0.5
Diversity
Figure 5: Ablation of DUPLEX in Multitask Walker. Reward and diversity scores of the best trained policy are reported on the left, while averaged values of all policies are on the right.
to have fair comparison against the baselines, only the last benchmark configures DUPLEX to estimate SFs through v/7, Additional details about the metrics and the environments are reported in Appendix[A]
Results in GranTurismo" TM 7. As in 12 we use QRSAC as the base RL algorithm for both DOMiNO ang and DUPLEX. Figure 2 (top-left) shows that only DUPLEX learns diverse policies in the near-optimal region. Figure [ illustrates how this diversity translates into diverse lap times. Moreover, Figure bi) provides evidence of the benefits that our lower bound /9 formulation carries with it. When using 3 = O, the most diverse policies are "busy" finding different ways to not finish the track. Furthermore, we see DUPLEX results oppose to DOMiNO that instead collapses all policies to a similar behavior even when using an extremely forgiving value of the optimality ratio p. It is worth mentioning, that DOMINO fails to provide diversity even if implements Lagrangian constraint optimization objective that should support more adaptability. For example, when configuring p 0.01, DOMINO should consider as "acceptable" any diverse policy achieving 1 1 of the value of the target policy Beweg
But still, the algorithm struggles to find diverse policies.
Results in MuJoCo. To further validate the contribution of DUPLEX diversity learning, include a direct comparison of DUPLEX and DOMiNO while configuring the Walker2D envi- ronment that the authors present in (6). Within this set of experiments, both algorithms have been configured to work with ج which fosters the opportunity to ablate the benefits of our constrained ry (Eq. 3 isolated from the impact of estimating SFs. Figure 3 includes the results from both frameworks with multiple optimality ratios p. Moreover, to provide an exhaustive evaluation, we also configure DOMiNO with various Van Der Waal (VdW) distance hyper- parammaareetrss. Such parameter modulates the desired distance between all the policies in II (see [6] for details). Finally, DUPLEX able to find better Pareto-frontier for the quality-diversity objective (Figure right), while achieving better near-optimal policies than the baseline (Figure 3 left).
Results in multi-context MuJoCo. Figure 4 reports the results of our baselines in multi-context scenarios when evaluated within- and out-of training distribution. In the within-distribution setting (Figure 4 left-hand side) only DUPLEX shows good performance and diversity while USFA and DOMINO fail at both learning competitive policies and providing diversity. As we report in the USFA ablation B). DUPLEX succeeds such benchmark due to the entropy term introduced to make the estimation of SFs more robust (Eq. 7 Additionally, note that USFA and UVFA do not optimize for diversity, and their diversity score is expected to be close to zero. Then, Figure f (right-hand side) reports the generalization effectiveness of DUPLEX and UVFA when operating out-of training distribution. Note that USFA and DOMiNO do not qualify in this setting as they failed to learn already in the training set. We observe that in all the OOD scenarios DUPLEX outperforms UVFA and, most importantly, it covers a bigger range of diversity values while completing the tasks.
Ablation of successor-feature estimation. Figure illlssrrtte the contribution of 1 and the dif- ferent features incorporated by DUPLEX. We have evidence that all features that compose DUPLEX are key to guaranteeing robust learning of diverse behaviors in highly-dynane environments and

---

multi-context settings, Notably, USFA can also effectively solve multi-content MuJoCo environments when akin DUPLEX. adds the entropy term (see Appendix
Conclusions
DUPLEX provides a powerful technique that enables learning of diverse near-optimal behaviors, Our experimental session shows that (differently from other baselines) DUPLEX (i) learns effective diverse behaviors in hyper-realistic complex domains (such as GranTurismo) and (ii) generalizes to OOD in challenging multi-context MuJoCo environments by demonstrating competitive diverse behaviors.
Limitations and Future Directions. There potential refine and enhance the methodology presented in this work. Nevertheless, we perceive the current limitations of DUPLEX not as setbacks, but as exciting opportunities that pave the way for new research avenues. For example. (1) can we generate more than twenty diverse policies and still guarantee meaningful diversity? (2) DUPLEX carries an additional cost due to the need to compute SF distances of each policy pair in our set, thus how can we improve sample efficiency and keep a low computational footprint? (3) DUPLEX does not impose any exploration strategy on each policy, and thus, can we control in what measure particular policy will be different? And finally, (4) can we combine the strengths of different on a single solution? This latter question is very interesting to us. We suppose that once diverse policies have been learned, each of them retains different useful skills (e.g. different commuting routes) that can be combined dynamically to reconstruct a single agent capable of acting optimally and tackling unseen portions of the state-space.
Finally, the presented approach holds sigmificant promise for advancing research in diverse policy learning, scalability, and efficient transfer and exploration [36 37 across varied contexts. We note also that RI. algorithms may also risk reinforcing biases and require careful implementation to ensure transparency and fairness. Diversity-learning algorithms might result in training policies with potentially unethical behaviors that are harder to predict. These are reasons that motivate us to pursue controllability while learning diverse policies in this and future works.
References
[1] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[2] Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level starcraft using multi-agent reinforcement learning. Nature, 575(7782)3350-3544 2019.
[3] Tom Zahavy, Vivek Veeriah, Shaobo Hou, Kevin Waugh, Matthew Lai, Edouard Leurent, Nenad Tomasev, Lisa Schut, Demis Hassabis, and Satinder Singh. Diversifying Al: Towards Creative Chess with AlphaZero. August 2023. MrXIv:2308.09177 [cs].
[4] Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässsan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1856-1865. PMLR, 2018
[5] Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1406-1415 PMLR, 2018
[6] Tom Zahavy, Yannick Schroecker, Feryal M. P. Behbahani, Kate Baumli, Sebastian Flennerhag, Shaobo Hou, and Satinder Singh. Discovering policies with domino: Diversity optimization maintaining near optimality. The Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
10

---

[7] Wei Fu, Weihua Du Jingwei Li, Sunli Chen, Jingzhao Zhang, and Yi Wu. Iteratively learn diverse strategies with state distance information. Advances in Neural Information Processing Systems, 36, 2023.
[8] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In International conference on machine learning, pages 1312-1320. PMLR, 2015.
[9] Diana Borsa, André Barreto, John Quan, Daniel J. Mankowitz, Hado van Hasselt, Rémi Munos, David Silver, and Tom Schaul. Universal successor features approximators. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[10] Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv: 1.X2::2500202259, 2015.
[11] André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning, Advances in neural information processing systems, 30, 2017.
[12] Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subrama- nian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896):223-22) 2022.
[13] Emanuel Todorov. Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5033- IEEE, 2012.
[14] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is All You Need: Learning Skills without a Reward Function. September 2018.
[15] Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier. Hierarchical Skills for Efficient Exploration. November 2021.
[16] Wojciech M Czarnecki, Gauthier Gidel, Brendan Tracey, Karl Tuyls, Shayegan Omidshafici. David Balduzzi, and Max Jaderberg. Real world games look like spinning tops. Advances in Neural Information Processing Systems, 33:17443-17453 2020.
[17] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzami, Heinrich Küttler, John P. Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado Hasselt, David Silver, Timothy P. Lillicrap, Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp, and Rodney Tsing. Starcraft II: A new challenge for reinforcement learning. CoRR, abs/1 708.04782. 2017.
[18] Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers Robotics AI, 3:40, 2016.
[19] Luca Grillotti and Antoine Cully. Relevance-guided unsupervised discovery of abilities with quality-diversity algorithms. In Jonathan E. Fieldsend and Markus Wagner, editors, GECCO '22: Genetic and Evolutionary Computation Conference, Boston, Massachusens, USA, July 9- 13, 2022, pages 77-85, ACM, 2022.
[20] Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean- Sann-Bapisste Mouret. Robots that can adapt like animals. Nature, 521(7553):533-507 May 2015, arXiv:1407.3501 [cs, q-bio].
[21] Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. CoRR, abs/1500.044009 2015.
[22] Manon Flageat, Bryan Lim, and Antoine Cully. Multiple hands make light work: Enhanc- ing quality and diversity using map-elites with multiple parallel evolution strategies. CORR, Abs/2303.06133) 2023.

---

[23] Joel Lehman and Kenneth O. Stanley. Evolving a diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, GECCO '11, pages 211-218, New York, NY, USA, July 2011 1. Association for Computing Machinery.
[24] Olle Nilsson and Antoine Cully. Policy gradient assisted map-elites. In Francisco Chicano and Krzysztof Krawiec, editors, GECCO '1: Genetic and Evolutionary Computation Conference, Lille, France, July 10-14, 2021, pages 866-875. ACM, 2021.
[25] Maxence Faldor, Félix Chalumeau, Manon Flageat, and Antoine Cully. Map-elites with descriplor-condiiionee gradients and archive distillation into a single policy. In Sara Silva and Luís Paquete, editors, Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2023, Lisbon, Portugal, July 15-19, 2023, pages 138-146. ACM, 2023.
[26] Bryon Tjanaka, Matthew C. Fontaine, Julian Togelius, and Stefanos Nikolaidis. Approximating gradients for differentiable quality diversity in reinforcement learning. In Jonathan E. Fieldsend and Markus Wagner, editors, GECCO '22: Genetic and Evolutionary Computation Conference, Boston, Massachusetts, USA, July 9 13, 2022, pages 100--1111. ACM, 2022.
[27] Timothee Anne and Jean- Baptttttttte Mouret. Multi-tast multi-behavion map-elites. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation, pages 111-114, 2023,
[28] Chen Ma, Dylan R. Ashley, Junfeng Wen, and Yoshua Bengio. Universal successor features for transfer reinforcement learning. CoRR, abs/2001. 0.402,, 2020.
[29] Wilka Carvalho, Andrew K. Lampinen, Kyriacos Nikiforou, Felix Hill, and Murray Shanahan. Feature-attending recurrent modules for generalization in reinforcement learning. CoRR, abs/21 12.0869,,, 2021.
[30] Wilka Carvalho, Angelos Filos, Richard L.. Lewis, Honglak Lee, and Satinder Singh. Composing task knowledge with modular successor feature approximators. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- view.net, 2023.
[31] Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murray Shanahan, Guillaume Lajoie, Michael Mozer, and Yoshua Bengio. Learning combine top-down and bottom-up signals in recurrent neural networks with attention over modules. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 6972-6986. PMLR, 2020.
[32] Borja G. León, Murray Shanahan, and Francesco Belardinelli. In nutshell, the human asked for this: Latent goals for following temporal specifications. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[33] Melissa Mozifian, Dieter Fox, David Meger, Fabio Ramos, and Animesh Garg. Generalizing successor features to continuous domains for multi-task learning. 2021.
[34] Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex MDPs. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances Neural Information Process- ing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 25746 257599, 2021.
[35] R. Tyrrell Rockafellar. Convex Analysis. Princeton Landmarks in Mathematics and Physics. Princeton University Press, 1970.
[36] Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Ef- fective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33: 1800- 2020.
[37] Zhenghao Peng. Hao Sun, and Bolei Zhou. Non-local policy optimization via diversity- regularized collaborative exploration. arXiv preprint arXiv:2006.0778.. 2020.
12

---

[38] Vivek S Borkar. An actor-critic algorithm for constrained markov decision processes. Systems control letters, 54(3):207-213, 2005.
[39] Eitan Altman. Constrained Markov decision processes, volume 7. CRC press, 1999.
[40] Csaba Szepesvári. Constrained MDPs and the reward hypothesis. Musings about machine learning and other things (blog), 2020.
13

---

A Environment details
In MuJoCo environments, we measure diversity directly the agent's observations, which is represented as a feature vector containing the angles and velocities of the agent's joints. In GT, diversity is measured through the most characterizing descriptors of a driving style: action intensities (brake, throttle, and steering values), wheel angles, and distances to the track edges.
Note that these diversity features were chosen to align with each environment's 'ss objectives. Neverthe- less, we can maximize the diversity of any feature in the observation. Such an arbitrary selection of features intuitively leads to different optimization objectives and thus learned strategies. For instance, in GT, we could maximize diversity on the frequency of hitting other cars, and this would lead to population of policies that are all driving as fast as possible while exploring the spectrum of aggressiveness: ranging from a very timid behavior (letting opponents pass to avoid contact) to a very aggressive one (hitting cars intentionally if the collision does not result in significant speed loss).
We employ three types of environments to conduct our experiments, We use the canonical MuJoCo benchmark for single-task experiments; then the TimeTTalTimmTTralll track in GranTurismo" 7' game for the GT experiments; and finally a MuJoCo wrapper environment for the multi-context experiments. Such wrapper augments the observation of the agent by adding context vector of three elements = (g. I, u) representing the gravity coefficient g. the lower height the agent should walk below of I and the upper height the agent should walk above of u. It is worth noticing that we excluded context configurations that would not be possible to solve, i.e. episodes were designed to let the agent learn to either walk low jump high. At training time, in the Walker2D environment, the context features (g. I, u) take values in E [6. 15). 7 E [0.8. 1.2] and u E [1.25, 1.8]. While in the Ant environment features take value in g E [6, 15). E [0.35, 0.8] and u E [0.9, 1.2].
In the OOD scenarios, we used two out-of-distribution contexts for the Walker2D and the Ant environments respectively. In the former, we used two values of the gravity coefficient and lower
bound constraint, namely co = (3, 0.6, None) and c1 = (24, 0.6, N None), while in the latter we used co 0.2, None) and C (24, 0.2, None)
B Ablation in USFA
Multflasit Walker
Tisk
7002
7000
UKFA with Brnn
USFA with H Ray
6000
Venils
5002
4000
0007
2000
USA with mmivg
1003
-3000
USe with avgis
USFA
1.00
0003
0.010
0.005
0023 0.520
-2000
0.002
0.002
0.500
0.004
0005
Diversity
Figure 6: Ablation on USFA, we see that incorporating entropy to the learning objective of the successor features estimator as in DUPLEX yields a significant boost to accumulate rewards.
In this section, we ablate the addition of the entropy term to the SFs objective function. In particular, we extrapolate such component from the others introduced by DUPLEX, and we evaluate the entropy term in isolation within the original USFA framework. The assumption that we validate with this ablation study is that Equation 7 has a broader impact and that it makes the estimation of SFs more robust. Figure 6 illustrates our findings, we report that the entropy term enables effective learning in multi-context MuJoCo environments and significantly improves performance w.r.t. vanilla USFA.
Importantly, Figure 6 illustrates that, using avg( (6,, 2) instead of the min(@o, 2).. is not beneficial for USFA. Note that differently from DUPLEX, USFA infers extrinsic rewards from SFs. Thus, in USFA an overestimation of SFs (i.c., overestimating how often the policy visits some state-action
14

---

Algorithm DUPLEX
I: Initialize dataset with random exploration, set up critics 0 and policy parameters w. and initialize moving
averages {veg) ) -= 0 reward weight 0. 1 and step =
2: while converged do
Generate more data samples
4: Sample batch of m transitions (ty where a transition is t. = (s',a',d, ,d, oe,,'),, 22,
ax" 5: Use critic to get Bey> fit
and (s, a, 2) for every x² E TI. Where &b avg to
h=1,2
6:
Compute (s) from vi 4, 6(s,, with Equation Compute extrinsic temporal difference (TD) errors and q-values: TD.. Qe For sampled get TD error with entropy on successor features with Equation TDs Compute intrinsic TD errors and q-values: TD,, Qr
8:
9:
10:
Combine q-values Q = (x)
-xx)))r
11:
if step critic warming-up period then
Policy loss:
12:
V min Qo, (8, T- ,l, 21)) log (-w e ,², =0)|2² ,2,,))))) M==12
Calculate weighted critic loss: b.. TD.) + by (TD#) if Constrained optimization then
14:
> Lagrange loss
15:
Σ Zo(k')(nn (Vive poavg)
1=1
16:
Update
17:
Update A and w
18:
Update vigue Ovann Neave
Ovann Dri,ts v'dave Ovann Vdaaz Oveav Dra, X with Eq.
step +
pairs) yields an overestimation of rewards. Such an overestimation motivates our design choice of using the ang of two critics DUPLEX, In fact, since DUPLEX relies on the distance between SFs to estimate intrinsic rewards, we achieve a more accurate representation of intrinsic rewards by taking the average of the two estimates.
C Training DUPLEX
DUPLEX follows actor-critic approach, where policy network determines actions to take while critic network estimates values and, the case of multiple contexts, also successor features. Figure [ depicts the workflow of our algorithm. The networks have three independent input encoders for the state, policy, and context respectively. Since we use SAC as the base RL algorithm, we employ two critic networks and take the min of the two for value estimates 7 and the average for J (Eq.
The training procedure of DUPLEX is described in Alg. After warming up to collect data, we start sampling transitions t and the learning routine (Line Since we need estimate the distance between all policies to compute the intrinsic rewards, broadcast the input of the critic to have the SF estimates for all policies in every transition and do a forward pass through the critic (Line 5). Then, we obtain with Eq. 3 and compute the lemporal-diiferreee errors (Line - Line 9 Similarly, after a short warm-up period, we compute the policy loss (Line 12). Then, depending on if we enable update the Lagrange multipliers (Line [5}. Finally,
constrained optimization, network weights and the average policy values to update the dynamic factor (Line 17 and Line [8].
update the
Upper bound on ax. As described in Section 4, when the critic learns to estimate SFs, we typically observe events where the estimated difference between policies rapidly increases its order of mag- nitude. Such a shift in the magnitude of expected distances results in rapid increase of intrinsic rewards, larger value estimates, and ultimately, the divergence of the crities output. To address such magnitude shifts, we modify the update rate of
if [Vaaaau > Se/Ueeve! Cksefault otherwise
0x
(
Intuitively, we want Ty to motivate the agent to explore increasingly diverse behaviors without stagnating the extrinsic rewards. For this reason, need ax to be very low value use le-5
15

---

across all our experiments. An exception is made when a sudden drastic change in aa could cause the critic to diverge. We automatically detect those changes through a threshold x11ra% where ey is scalar, and perform one-step update to prevent from destabilizing the learning process. Similarly to other and benchmarks.
once we found stable value for <x remained fixed for all experiments
Constrained optimization. Within the algorithm framework, allow Langrangian constraint optimization [38 to be contingent to specific domains. When constrained optimization is configured, we establish multi goaa objective (author?) [39 40 using Lagrange multipliers K, whose objective is to maximize n(pē, dx Te). Intuitively, this objective yields a multiplier that decreases in value when the policy satisfying the optimality constraint, and increases otherwise. However, we empirically find that while Lagrangian constrained optimization might be beneficial in canonical domains, not effective in complex environments such as GT and fails to find competitive diverse behaviors. Nevertheless, Equation (4) and make our less reliant on constraint optimization than previous approaches. For these reasons, we implement constrained optimization in DUPLEX by keeping fixed by default and making it learnable an optional feature.
Remark C.1 DUPLEX is designed so that only two hyper-parameters significantly impact its perfor- mance: p and B. The former regulates how much the user wants to weight the intrinsic rewards w.r.t the extrinsic reward, while the latter defines the near-optimality region that the agent should explore when learning diverse policies.
D Hyper-parameters and Computational Analysis
D.1 Hyper-parameters
Table Dlists DUPLEX hyper-pane paremmearsssssss and neural network configuration.
Table 1: Hyper-parameters of DUPLEX.
HYPER-PARAMETER
MUJoco
GT
N
10
10
[0.9.0.7]
[0.88, 0.8]
0.5
0.97
Ox
1e
1e
Ex
20
20
FROM SIGMOID
0.9999
0,9999
0.999
0.999
SAMPLE RATE
0.3
0.3
R LEARNING RATE
le-2
SAC/QRSAC LEARNING RATE
3e
2.5c
TARGET CRITIC TARGET SOFT UPDATE RATE
0.005
0.005
CLIP GRADIENT GLOBAL NORM
OBS ENCODER SIZE
256
2048
CONTEXT ENCODER SIZE
64
POLICY ENCODER SIZE
256
256
TORSO SIZE
2x256
4x2048
CRITIC HEAD HIDDEN LAYER SIZE
256
2048
SAC ENTROPY LEARNING RATE
1e
FIXED ENTROPY
D.2 Computational Analysis
Given the additional components that DUPLEX requires to stabilize training, we report in the following table the computational cost of the update function of the core learning algorithm while varying the number of used policies. We compare our approach to DOMiNO and vanilla SAC and demonstrate that the additional cost negligible when considering related work, and does not increase with the number of policies. Execution times of the selected algorithms reported in milliscconds
16

---

and have been evaluated on the Walker2D MuJoCo environment rolled out on 13th Gen Intel(R) Core(TM) 19-13900HX and NVIDIA GeForce RTX 4080 Laptop GPU.
Table 2: Execution time of the algorithms update function. Time is reported in milliseconds when computing diversity for 2, 4, 10, and 20 policies simultaneously.
ALGORITHM
2
10
20
DUPLEX
55.78 + 1.51
55.72 + 1.82
55.21 + 1.37
56.73 + 2.06
DOMINO
52.64 + 1.88
53.68 + 1.22
54.64 + 1.36
56.08 + 1.63
SAC
41.01 2.05
E Hardware
We ran our experiments on an internal cluster designed for distributed training. Each job has been configured with a trainer (to update the models) and N rollout workers (to collect data).
MuJoCo. In this environment with GPU. The former uses 7.7 vCPUs and 8Gi of RAM, while the latter uses vCPUs and 2Gi of RAM. The duration of each experiment is two days.
trainer and one rollout worker of them equipped
GT. this environment, each job configured with one trainer and ten rollout workers. The trainer is equipped with 7.7 vCPUs, Nvidia V100, and 55Gi of RAM, while each rollout worker features 2000m vCPUs, 3328Mi RAM, and PlayStation4. The duration of each experiment is seven days.
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
We listed and described claims made in this work both in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations of framework Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The research described in this paper does not make theoretical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our main contribution is the introduction of a novel iterative algorithm: DUPLEX. We provide (i) flow diagrams to illustrate how the building blocks are connected; (ii) describe each component and equation in detail in the methodological section; (iii) outline the pseudocode of DUPLEX to improve readability; and (iv) details of our benchmarks both in Section 5 and Appendix A along with the hyper-parameter used in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
19

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer:
[No]
Justification:
The data and code used for empirical results and analysis are proprictary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips. Sliaps.cccpubliee wwtppbbiccgguueeeeeoooooocppcped
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include all the details reproduce our results in Section 5 and Appendix A Additional details, including the hyper-paramen used, are available in (Appendix D)
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report average results and error bars of our experiments. Figure 2(b) and Figure represent an exception as we report the performance of the most competitive agent for each baseline.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include details about the compute resources and hardware setup in Ap- pendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, we believe the research DUPLEX and empirical analysis conform to the NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no direct socictal impact of the work described in the paper. We focus learning diverse policies in dynamic domains advance the field of RI..
Guidelines:
21

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does include the release of data or models and as such does not pose risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have provided citations and references to existing MuloCo and GT assets, including GT's trademark in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
22

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papprsaipphaeeoooccaaapppaapppse thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not include the release of any new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The research described in this paper does not include any work with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: IRB approval was not required as the research does not include any work with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "2q90BwUaiO", "reviewer": "Reviewer_zFsC", "summary": "introduced a method to enhance diversity in RL policies by using successor features for robust behavior estimation. Experiments show DUPLEX outperforms state-of-the-art baselines, achieving diverse, competitive policies in GranTurismoTM 7 and multi-task MuJoCo environments, even in out-of-distribution contexts.", "questions": "All my questions are mentioned in the weaknesses section.", "limitations": "Thoroughly discussed in Conclusions.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The ability to learn multiple strategies for a given task is meaningful and can shed light on future decision-making agents' research.\n\n2. The experimental results show that the proposed method outperforms previous works in complex driving simulators and OOD robotic tasks.", "weaknesses": "I am not familiar with work related to universal function approximators, but I am knowledgeable about quality diversity and RL-related work. Therefore, I will focus on raising questions about the latter.\n\n1. In 4.1.1, the authors mention, \"We explored Lagrangian-constrained optimization but found it unsatisfactory in complex domains like GT.\" Although this statement is experimentally validated, many past QD RL works have used Lagrangian-constrained optimization successfully[1,2]\n, making this claim counterintuitive to me. Could the authors compare their results with these works or explain in detail the reasons behind their findings? Furthermore, could they mathematically explain the impact of multiplying the reward function by lambda on the optimization objective?\n\n2. In the experimental section, the right side of Figure 3 shows that to improve the diversity score by 0.15, the reward decreases to 5000 points, which is not very high for the walker task. I have not run diversity strategy experiments under this setting and thus lack a reference point. However, I feel the increase in the diversity score is insufficient. Could the authors run n instances of SAC/PPO, each with different seeds, and stop at around 5000 points to report the diversity scores as a reference? Otherwise, the Baseline (DOMiNO) scores in the lower left of Figure 3 make me suspect the authors have not correctly implemented the baseline algorithm. The same issue appears in Figure 4, where DOMiNO is in the lower left, and I would like a reasonable explanation from the authors.\n\n3. There are several minor errors in the paper, which, while partially understandable in context, collectively make the paper difficult to understand:\n- $V_{d_{avg}}$ is not defined (line 196).\n- $V_{e_{avg}}$ is first mentioned in line 191 but introduced only in line 207.\n- The SF estimator on the left side of Equation 7 includes $\\gamma$, but the right side does not.\n- $y$ in Equation 8 is not a function of $c$, yet $c$ is used on the right side.\n- The $z$ in Equation 8’s right side is in the superscript, where $\\gamma$ was used earlier.\n- The legend in Figures 3 and 5 is too small.\n- The difference between the left and right of Figure 5 is not well-explained.\n\n\n4. **The authors did not submit their code.**\n\n5. Lastly, as a reader, I am more interested in the impact of multi-tasking on diversity strategies. For example, as mentioned in [3], the goal is for strategies to be as similar as possible across different tasks, which is contrary to the diversity score. Have the authors found similar results, and how did they address this? \n\n[1] Kumar, Saurabh, et al. \"One solution is not all you need: Few-shot extrapolation via structured maxent rl.\" Advances in Neural Information Processing Systems 33 (2020): 8198-8210.\n\n[2] Chen, Wentse, et al. \"DGPO: discovering multiple strategies with diversity-guided policy optimization.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 10. 2024.\n\n[3] Teh, Yee, et al. \"Distral: Robust multitask reinforcement learning.\" Advances in neural information processing systems30 (2017).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the feedback you provided on our submission. \n\n\n**In 4.1.1, the authors …**\n\nPlease note that our results conform to the literature [2, 3, DOMiNO] when evaluating the Lagragian-constrained optimization in canonical MuJoCo environments and we found it to outperform the non-lagrangian counterparts. However, when considering a complex domain like Gran Turismo, where the difference between a competitive and an unsatisfactory policy could be a few hundred milliseconds (when evaluating the lap time in completing a track) and state transitions obey complex dynamic interactions, we found that Lagrangian multipliers are not as beneficial. Also, note that this kind of domain is not considered in [1, 2] or in DOMiNO.\nIn complex domains, we report that Lagragian-constrained optimization fails to preserve a good performance vs diversity trade-off. This is evident in Figure (2, a) where DOMiNO is only optimizing for performance, disregarding the diversity component of the training objective. Mathematically, Lagragian-constrained optimization is known to be unstable [4] while searching for a good compromise across different constraints. \nWhen evaluating Lagragian-constrained optimization in DUPLEX, we found that rapid changes in the multipliers are not beneficial when learning complex environmental dynamics and the agent would either optimize for performance or diversity. This observation led us to introduce the soft-lower bound in DUPLEX – which as we report in Figure (2, b-c) outperforms other baselines.\n\n**In the experimental section...**\n\nWe welcome the reviewer's suggestion and we are working towards including the diversity score of a vanilla SAC baseline in our experimental section. It is worth noting that we are normalizing diversity scores with respect to the most diverse set of policies across domains to facilitate comparison. The reported increment is then a squashed score but, as we report in Figure 3., it is important to note that DUPLEX can provide a more stable performance vs. diversity trade-off.  While we cannot provide the requested baseline yet, we can share data demonstrating the correctness of the DOMiNO implementation:\n\n- In Figure 3, the average reward and mean diversity of DOMiNO are presented for two different optimality ratios (OR). With an OR of 0.9, DOMiNO achieves an average reward of 6140 ± 94.34 and a mean diversity of 0.0215 ± 0.0086. At an OR of 0.5, the average reward is 5568 ± 698.70, with a mean diversity of 0.0372 ± 0.029. For comparison, we evaluated USFA in a single-task Walker scenario. USFA, which uses SAC and lacks diversity incentives, achieved an average reward of 6280 ± 555 and a mean diversity of 0.003 ± 0.001. USFA results are also based on 5 independent runs. These findings indicate that DOMiNO produces policies that are 7 to 12 times more diverse than those of USFA. The plot normalizes these differences, which appear minimal when compared with DUPLEX.\n- In our code-base, both DOMiNO and DUPLEX use the same utility functions; core computation of the intrinsics reward; Lagrange-constrained optimization, and SAC as the base algorithm.\n- Figure 4 illustrates the performance of different methods on a multi-context benchmark. As mentioned in Section 4.2, with infinite environments, we cannot use the average of inputs to compute Successor Features (SFs) and must estimate them instead. This introduces instability, causing both SFs and diversity rewards to chase correlated errors. DOMiNO lacks the stabilizing mechanisms introduced with DUPLEX, resulting in its failure to learn competitive and diverse policies, similar to USFA. Figure 5 highlights the importance of these mechanisms, showing that without them, DUPLEX would also fail. Appendix B demonstrates that incorporating entropy regularization into USFA (as in DUPLEX) significantly improves its results.\n\n**Minor corrections**\n\nThank you for the pointer, we integrated these corrections to the main body of the paper.\n\n**Code.**\n\nWe cannot share the code to this date. We included a detailed pseudo-code description of our algorithm, described our environment domains, and included the hyper-parameters used for training DUPLEX agents. We believe this to be the key information needed to reproduce our experiments but we are happy to include further details otherwise.\n\n**Lastly, as a reader…**\n\nThe approach presented in [3] exploits policy distillation to extrapolate common skills from strategies learned across the different tasks. As pointed out by the reviewer, the diversity score instead serves a different purpose. That is, in contrast to [3], our training objective is based on encouraging diversity subject to the performance of the target policy. Such diversity is especially important in generalizing to OOD tasks, as illustrated in Figure 4 b). This is because policies can converge to strategies that share fundamental skills but learn different strategies that can be useful to unseen tasks. For example, in our Walker2D environment with changing gravity, a policy that applies higher torques to the joints has a higher chance to succeed under higher gravitational force conditions.\n\nFinally, demonstrating near-optimal performance while combining diversity and multitask generalization has direct applications to the GT player community, increasing replayability and engagement for users by racing against diverse and competitive opponents. Being robust towards OOD tasks and environments makes these agents more applicable to new game functionalities and updates.\n\nThank you for your feedback. Let us know if you have any further questions.\n\n[4] Moskovitz, Ted, et al. \"Reload: Reinforcement learning with optimistic ascent-descent for last-iterate convergence in constrained mdps.\" International Conference on Machine Learning. PMLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xFmRwmfPLx", "reviewer": "Reviewer_Eqtv", "summary": "The authors introduce DUPLEX, a method that defines a diversity objective with constraints and uses successor features to robustly estimate policies' expected behavior. DUPLEX allows agents to:\n1.\tLearn a diverse set of near-optimal policies in complex, highly-dynamic environments.\n2.\tExhibit competitive and diverse skills in out-of-distribution (OOD) contexts.\nThe authors claim that DUPLEX is the first method to achieve diverse solutions in both complex driving simulators and OOD robotic contexts.", "questions": "See Weaknesses", "limitations": "Authors have mentioned the limitations in the paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "a. The diversity objective is commendable as it promotes the development of a set of policies that are diverse enough without hindering reward maximization.\n\nb. The paper is very well-written and organized. The logical structure of motivation, related work, background, and analytic study of the frameworks, combined with the metric and the presentation of the algorithm and its evaluation, is easy to follow and intuitively explained. The inclusion of small examples to illustrate the arguments helps lighten otherwise densely formulated reasoning.", "weaknesses": "•\tThe paper does not compare its diversity objective to other existing diversity objectives in the literature, such as DIAYN (Eysenbach et al., 2019) and MaxEnt RL (Haarnoja et al., 2018). This omission makes it difficult to judge why this particular diversity measure should be considered over the others.\n\n•\tThe paper builds upon the Domino paper, but it is not clear how it differs from Domino since both papers use extrinsic rewards and another metric computed to promote diversity, i.e., intrinsic reward.\n\n•\tAnother fundamental question arises: If a model can generate stochastic policies with high rewards, will it be considered diverse compared to the deterministic policy model? Is diversity essentially a measure of generating high-reward stochastic policies?\n\n•\tI would urge authors to upload code and some visualizations with comparison of the method to Domino during rebuttal period. \n\n\nOverall a good paper and I would be happy to discuss more with authors. I will be willing increase my score if my questions are answered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and feedback you provided on our submission. \n \n**The paper does not compare its diversity objective to other existing diversity objectives in the literature, such as DIAYN (Eysenbach et al., 2019) and MaxEnt RL (Haarnoja et al., 2018). This omission makes it difficult to judge why this particular diversity measure should be considered over the others.**\n\nIncluding algorithmic baselines can greatly enhance a scientific contribution. For our experimental section, we selected baselines that we believe best support the claims of our work. We consider DOMiNO to be the state-of-the-art method in reward-based diversity learning and thus encompasses other baselines.  In contrast to DIAYN and MaxEnt RL (and DOMiNO), DUPLEX is also being evaluated in OOD multi-task generalization. In such a setting it would not be a valid data point for the comparison against standard diversity learning algorithms. To support the contribution of DUPLEX in such settings, we rely on state-of-the-art frameworks for multi-task learning such as UVFA and USFA.\n\n**The paper builds upon the Domino paper, but it is not clear how it differs from Domino since both papers use extrinsic rewards and another metric computed to promote diversity, i.e., intrinsic reward.**\n\nAs the reviewer correctly points out, DUPLEX’s core definition of diversity and diversity reward computation is aligned with DOMiNO. However, we extended the training problem to include context-based scenarios and enhanced the robustness of training diverse policies. We achieved this by adding new components, which are briefly outlined here (and detailed Section 4):\n- *Dynamic intrinsic reward factor* to modulate the ratio between diversity and extrinsic rewards, alleviating the reward tuning that DOMiNO requires across different environments\n- *Soft-lower bound* to limit the region of interest for the diverse policies. \n- *Entropy regularization term in SFs estimation* to improve the estimation of SFs in multi-task environments.\n- *Use of the average of the critic estimates*. Differently than taking the min of the critics as it is common with value estimation, we find that to compute diversity rewards is beneficial to take the average of the estimated SFs.\n\nTo further highlight the novel components that DUPLEX introduces, we added a brief paragraph summarizing key contributions at the end of Section 4. \n\n**Another fundamental question arises: If a model can generate stochastic policies with high rewards, will it be considered diverse compared to the deterministic policy model? Is diversity essentially a measure of generating high-reward stochastic policies?**\n\nIn this work, according to Definition 4.1. we consider (deterministic and stochastic) policies to be “diverse” if they feature diverse state-action occupancies ). A stochastic policy will visit different state-action pairs than a deterministic one. Hence these two policies can be considered diverse. When introducing a near-optimal constraint, we are still searching for policies that feature diverse state-action occupancies, but we constrain the search within a particular area of the search-space, which is defined by Equation 1.\n\nIn other words, the reward obtained by the agent (or model) has no impact on the computation of the diversity metric, but since DUPLEX enforces that all the policies have to simultaneously optimize for the true objective as well, it yields the ability to maximize discounted cumulative rewards (high-rewards).\n\n**I would urge authors to upload code and some visualizations with comparison of the method to Domino during rebuttal period.**\n\nWe cannot share the code to this date. We included a detailed pseudo-code description of our algorithm, described our environment domains, and included the hyper-parameters used for training DUPLEX agents.  We believe this to be the key information needed to reproduce our experiments but we are happy to include further details otherwise.\n\nWe are working towards including visualizations of Domino vs DUPLEX in our site https://sites.google.com/view/duplex-submission-2436/home but, we will likely add them after the discussion period ends.\n\nIn the meantime, we can describe our observations from completed experiments. DOMiNO typically produces policies with subtle differences, as seen in Figure 1(b) of [2], where each policy raises the leg slightly more than the previous one. Occasionally, by sufficiently reducing the optimality ratio, we may observe two “main behaviors” that are visually distinguishable, with other policies showing small variations around them. In contrast, DUPLEX tends to produce significantly more diverse populations of near-optimal policies that are visually distinct. This greater diversity yields better trade-offs, as shown in Figures 2, 3, and 4.\n\nThank you for your useful feedback. We will be happy to respond to any further questions you might have.\n\n[1] Zahavy, Tom, et al. \"Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality.\" The Eleventh International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8ha3k1DGWa", "reviewer": "Reviewer_9gQo", "summary": "This paper proposes to use successor feature (an embedding of state-action pair) to boost the behavior diversity of a population of policies.\n\nSeveral tricks are proposed to compute the diversity intrinsic reward. These include using the running average of the extrinsic reward to scale the intrinsic reward and using a dynamic weight $\\lambda$ to ensure the task performance is not dropped too much.\n\nExperiments on GranTurismo 7 and Mujoco show that the method can learn diverse yet high-performing policies, which also shows certain OOD capacity.", "questions": "What does \"closest expected features\" in Line 178 mean? A feature is a vector, how you get the closest vector of another vector? Are you computing the L1/L2 norm as the distance between features?\n\nClarity issues:\n* Typo in Line 198 \"or the update\"\n* There should be a $i$ notation in $r_d(s, a, c)$ in Eq 2 to indicate that reward is for policy $i$.\n* Equation 7 has an undefined $y$.\n\nWhat does the braces in Eq 5 means?\n\nHow $\\phi(s, a, c)$ is computed? The output of a neural network? In Appendix Algorithm Line 5 you say \"use critic to get\" it, what does this mean? Please describe the detailed architecture of the critic network.\n\nIt seems the method use QRSAC as the RL algorithm. Will you recompute the intrinsic reward for each sampled batch during training? Or you store the intrinsic reward as it is to the replay buffer?\n\nMissing citations to relevant works:\n\n* Exploration by Random Network Distillation\n* Non-local policy optimization via diversity-regularized collaborative exploration", "limitations": "Limitations are addressed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The proposed method is simple.\n\nThe detailed ablations on each part of the proposed method is sufficient.\n\nThe behavior in GranTurismo 7 is interesting and it's indeed diverse yet competent.", "weaknesses": "There is a strong assumption of the method: the context vector is given by the environment. The claim on context-conditioned MDP doesn't seem to be interesting to me as considering those context vector as part of the observation space will not change anything. Thereafter the claim in Line 174-175: \"We adopt the repulsive reward ... and extend it to be conditioned on the context\" does not mean a lot and it suggests the technical novelty of the method against Zahavy is limited.\n\nThe method is not scalable as you need to compute Equation 2 and Equation 3 for every state action pair to collect the intrinsic reward. That means you need to iterate over all agents to get their successor feature.\n\nNo code is provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and appreciate the feedback provided.  \n\n**There is a strong assumption of the method: the context vector is given by the environment.**\n\nWe agree that not all environments are designed to provide a context vector. However, we comply with related work in zero-shot generalization [1] where contextual information is provided as a separate input that is procedurally generated by the environment. That said, we believe that extending our work to domains with hidden contextual information is a promising future direction and will include it in the conclusion section. However, it is out of the scope of this paper.\n\n[1] Kirk, Robert, et al. \"A survey of zero-shot generalization in deep reinforcement learning.\" Journal of Artificial Intelligence Research 76 (2023): 201-264.\n\n**The claim on context-conditioned MDP…**\n\nWe agree that only extending Equation 2 to contextual-MDPs is not a significant contribution and thus it is not listed as a main contribution. To have a fair comparison, we also provide contextual information to DOMiNO when evaluating it.\n\nTo highlight the novelty, we list the technical contributions briefly here. These are detailed in Section 4 and the ablation studies.\nDynamic intrinsic reward factor to modulate the ratio between diversity and extrinsic rewards, alleviating the reward tuning that DOMiNO requires across different environments\nSoft-lower bound to limit the region of interest for the diverse policies. \nEntropy regularisation term in SFs estimation to improve the estimation of SFs in multi-task environments.\nUsing the average of the critic estimates is beneficial when estimating SFs and at the same time computing diversity\n\nTo further highlight the novel components that DUPLEX introduces, we added a brief paragraph summarizing key contributions at the end of Section 4. As evidenced in the ablations, all these components have a significant impact with respect to DOMiNO and enable our agent to achieve competitive performance in hyper-realistic domains and in multi-task environments.\n\n**The method is not scalable…**\n\nPlease note that Equation 2 and Equation 3 are not computed separately for each pair of policies but instead are vectorized through matrix multiplication, and thus scalable. We note that the equations are computationally equivalent to the equations in Zahavy’s method (which provides pseudocode on how to compute these equations as a matrix). We specified in Appendix C that we do not increase computational complexity with respect to previous baselines.\n\n**No code is provided.**\n\nWe cannot share the code to this date. We included a detailed pseudo-code description of our algorithm, described our environment domains, and included the hyper-parameters used for training DUPLEX agents.  We believe this to be the key information needed to reproduce our experiments but we are happy to include further details otherwise.\n\n**What does \"closest expected features\" …**\n\nAccording to Definition 4.1, we compute distances among policies as the L2-norm distance between their feature vectors. We added a reference to Definition 4.1 on line 178.\n\n**Clarity issues**\n\nWe thank the reviewer for the detailed review. We have integrated the recommended corrections into the equations. Note that the definition of `y` follows in Equation 8 and we have updated the text to better describe this connection.\n\n**What does the braces in Eq 5 means?**\n\nBraces indicate that the soft-lower bound is defined as a vector of n = size(\\Pi) components, one for each policy. We corrected a typo in the equation that as currently defined, would consider n + 1 policies.\n\n**How  ϕ(s,a,c) is computed? The …**\n\nWe compute expected features ψ by predicting the expectation of the cumulants ϕ through the estimation of the critic network. And we achieve that by adding an extra head to the critic network.\n\nMore specifically, cumulants ϕ are a part of the input vector where we want to maximize diversity. By default, we use the whole environment observation as ϕ. Nevertheless, if we want to tease out diversity in particular features, as in the case of GT, we only use of subset of the observations to define ϕ. For example, in the aggressiveness scenario shown here https://sites.google.com/view/duplex-submission-2436/home, only the part of the observation that characterizes collisions among cars is included in ϕ, and as a result, the agent trains policies with different levels of aggressiveness.\n\nExpected features ψ are directly estimated by an additional head of the critic network (Line 8) and the error is obtained through Equation 7. The critic architecture is detailed in Figure 1. and the size of the expected features vector is equal to the number of cumulants. We included this description in Table 1.\n\n**It seems the method use QRSAC…**\n\nWe use SAC in the canonical MuJoCo environment for repeatability of the results and use QRSAC in GT as it represents the state-of-the-art in that domain.\n\nSince the intrinsic reward depends on the transitions sampled from the environment, we compute the reward signal for each batch. Nevertheless, this operation is vectorized through matrix multiplication and it is not iteratively computed, which would make the approach impractical. \n\n**Missing citations to relevant works**\n\nWe are happy to include these references, we also think that exploration is key when learning diversity and these citations represent a promising future direction to further improve our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8F10brwRAT", "reviewer": "Reviewer_VFg8", "summary": "This is a work in the Reinforcement Learning domain, particularly relevant to policy exploration. They propose a method to better preserve the trade-off between the exploration diversity and near-optimality. This strategy utilizes a diversity objective with defined constraints such that it enforces the trained agent learns a diverse set of near optimal policies and could exhibit competitive and diverse skills in out-of-distribution environment. Two experimental domains are used to showcase the superiority of the proposed method.", "questions": "* Regarding the diversity definition in Def 4.1, can you give an example of whether/how different modalities of observed features (e.g., visual or textual) could have a different impact on the diversity formula? or it does not matter (no effect).\n\n* What is the relative additional cost introduced by computing SF distance of each policy pair?", "limitations": "I like how the author clarified their limitations in section 6. I am very interested in learning more explanations on question #3 and #4.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The problem the author studies is an interesting and challenging direction. There are many related work and the author has clarified the motivation and their own contribution pretty clearly. \n* The proposed method is technically novel and sound.\n* The experiments performed are comprehensive.", "weaknesses": "* Relying on multiple hyper-parameters (introduced by equations 3-5, probably hard to scale across the domain).\n* Some of the paragraphs has heavy notations and might be more readable if the author could give more intuitions on why it came out like that. For example in section 4.1.1 and section 4.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their interest in our work and the feedback provided. Below are the detailed responses:\n\n**Relying on multiple hyper-parameters (introduced by equations 3-5, probably hard to scale across the domain).**\n\nAs indicated in lines 207-209 and 843-845, most of DUPLEX hyper-parameters do scale across domains used in our evaluations. As reported in Table 1, apart from network size and learning rate, to guarantee diversity, we only need to adjust `\\rho` and `\\beta`. It is worth mentioning that we want to expose such hyper-parameters to the user to be able to specify the performance vs. diversity trade-off according to the use case.\n\n**Some of the paragraphs has heavy notations and might be more readable if the author could give more intuitions on why it came out like that. For example in section 4.1.1 and section 4.2.**\n\nReviewer XDei expressed a similar comment and suggested adding a preview paragraph describing how the components interact with each other at the beginning of Section 4. We, therefore, redirect this comment to that answer as well.\n\n**Regarding the diversity definition in Def 4.1, can you give an example of whether/how different modalities of observed features (e.g., visual or textual) could have a different impact on the diversity formula? or it does not matter (no effect).**\n\nThis is an interesting observation. We believe that different modalities do not alter Definition 4.1, which holds valid. We can always compute L2-norm distances of tensors. \n\nHowever, under different modalities, it is important to understand how to design cumulants and guarantee that the agents focus on important features. Similar images can represent different things and the same concept can be expressed with very different words. For example, in GT, few pixels can draw the line between a mispredicted collision and an actual collision between cars.\n\nOne approach is to preprocess visual and textual information with an encoder, or we could design modality-invariant cumulants. Nevertheless, guaranteeing execution under different modalities is an interesting aspect that is mandatory to generalize DUPLEX to diverse domains, and will be a topic in our future work.\n\n**What is the relative additional cost introduced by computing SF distance of each policy pair?**\nWe will compute the execution time and report the additional cost of estimating SFs per batch iteration. It will be included in Section D. The computation of the distance of all the samples in the batch is done through matrix multiplication, not iteratively.\n\n**I like how the author clarified their limitations in section 6. I am very interested in learning more explanations on questions #3 and #4.**\n\nWe appreciate your comment and your interest in our future work. Below we detail questions 3 and 4:\n\nQ.3) DUPLEX does not impose any exploration strategy on each policy …\n\nBy design, DUPLEX does not bias exploration within the near-optimal regions and policies can converge to any local-minima in such space as long as the diversity constraint is satisfied. As a result, two consecutive experiments within the same domain may return with a different set of diverse policies. However, for some use-cases it is reasonable to guarantee that some behavior is always discovered. For instance, in Walker2D, there is always a policy that learns to jump by only using the left leg. In GT, an aggressive policy is often learned. In future work, we could explore how to differently mask cumulants for each policy in the set and bias exploration. Alternatively, we could design SF vectors that represent desired behaviors and use such vectors as behavior baselines for each policy.\n\nQ.4) can we combine the strengths of different on a single solution? …\n\nIntuitively, even though the target policy is configured to only maximize the extrinsic rewards, the other policies (due to the added exploration) could converge to behaviors that are more efficient in specific state transitions. We believe that enhancing the target policy with situational corrections coming from the auxiliary policies can improve the overall performance of the agent.\n\nIn early experiments, we explored generalized policy improvement (GPI) [2], which intuitively, uses the critic estimations to select, at each step, the best action over a set of policies. However, empirically, we did not find it to improve the performance of our agent. We hypothesize that the imperfect information of the critic becomes especially relevant in continuous-action environments. Moreover, our continuous settings might demand GPI to be executed with a longer horizon, and a “simple” step-wise composition of diverse policies might not be sufficient. \n\nFinally, we believe that improving accuracy in the critic estimations and taking advantage of the added exploration of diverse learning approaches is an exciting line of work and is necessary to advance the research field.\n\n[1] Wurman, Peter R., et al. \"Outracing champion Gran Turismo drivers with deep reinforcement learning.\" Nature 602.7896 (2022): 223-228.\n[2] Barreto, Andre, et al. \"Transfer in deep reinforcement learning using successor features and generalized policy improvement.\" International Conference on Machine Learning. PMLR, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "25uZlayp6x", "reviewer": "Reviewer_XDei", "summary": "This paper proposes an algorithm, DUPLEX, that learns diverse yet high-performance policies for context Markov decision process. DUPLEX extends the previous SOTA DOMiNO by introducing context in the MDP and extending the diversity objective as context conditioned. DUPLEX also introduces three tricks to stabilize the training: a dynamic scaling parameter to balance extrinsic and intrinsic reward, a lower bound to encourage high-performance policies, and a SAC-inspired entropy term to stabilize successor feature estimation. The experiment results show DUPLEX can learn diverse and high-performance policies on both highly complex GranTurismo environment as well as standard MuJoCo domains with OOD physics. In general, although the paper shows strong performance with its proposed method, I have concerns over the clarify of the paper and the evaluation.", "questions": "Please see the weaknesses for questions.", "limitations": "The authors did not discuss the potential negative societal impact of the work. I hope the authors consider including them in the rebuttal.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors motivate the need for *diverse* high-performance policies well. I agree it is an important subject for research. \n2. The paper presents several novel algorithmic ideas to stabilize the diversity-seeking policy training, including balancing the extrinsic and intrinsic rewards, a lower bound for encouraging being close to near-optimality regions, and a entropy term for stabling successor feature estimation. \n3. The paper tests with the highly complex GranTurismo task and achieve impressive results. The authors also manage to show experiment results on MuJoCo standard tasks and ablation results, which help reproducibility.", "weaknesses": "1. The paper’s writing can be improved. \n\n1.1. (Line 42-43) It seems to have a logical gap on why diversity in “multi-context setting” is key to enabling human-level task execution. As a result, I do not see why we should learn diverse behaviors on envs with different contexts (e.g., OOD environments). \n\n1.2. (Line 132) the paper claims to find a set of diverse policies that maximizes the expected return in every context, but that does not seem to be enforced (not to mention there could be infinite c, which makes empirically showing that the policies are maximizing the return in *every context* impossible). \n\n1.3. The Section 3 discussion about universal estimators seems disconnected with the intro and other texts around it. I do not understand from the description how universal estimators relate to the goal of finding diverse policies. \n\n1.4. While I was able to follow Section 4, I find it hard. I suggest authors give a “preview” in the beginning of the section, and/or a diagram showing how different pieces of novel algorithms are connected together to form DUPLEX. It will also help motivate each algorithm component better. \n\n1.5. The design of Equation 5 seems a mystery. I suggest authors provide stronger justification on the design, as well as exploring alternative, simpler design to achieve similar effect. \n\n1.6. Similarly, the motivation for using an entropy term to stabilize successor feature estimation seems weak to me. Entropy term encourages the policy/value to respect uncertainties and multi-modalities in policy/value estimation, but SF estimation is unstable because of the Q-learning’s nature of chasing-its-own-tail learning and the difficulty to explore the entire space of hard problems such as GranTurismo. I do not see how they are related. \n\n2. The features, as introduced in line 141 of the paper as “cumulants”, are essential to the core of the algorithm as involved in successor feature estimation and the diversity definition 4.1. However, the paper does not discuss how the features or cumulants are selected for any of the experiments, which makes me very concerned about reproducibility and how the algorithm works without very sophisticated feature choices. I strongly recommend the authors provide more details regarding it and if possible, ablation studies on different feature choices. Many of the experiment results could be trivial if DUPLEX is optimizing the defined diversity with the features, while baseline algorithms are optimizing different definitions of diversity and are evaluated with DUPLEX diversity. \n\nMinor comments\n1. Line 39 a period is missing. \n2. While I agree successor features are good representation of a policy, I think it is also necessary to discuss the limitation of it – it ignores the order of features, which in some situations are essential. For example, successor features consider the state sequence a,b,c,a,b,c as the same as c,b,a,c,b,a, which is not necessarily always true.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback, it has been very useful to strengthen our work. \n\n**(Ln 42-43) It seems to have a..**\n\nHumans adapt OOD contexts continuously. For example, if a person injures their leg, they can immediately balance on one leg and even jump forward without using the injured leg. We aim to transfer this adaptability to agents in different contexts. For instance, a bipedal robot learning to walk should explore various walking modalities to be ready for unseen situations. This example is included in Line 43.\n\n**(Ln 132) the paper claims…**\n\nWe agree that it is not possible to empirically prove that DUPLEX finds a set of near-optimal diverse policies for every context. However, we do not claim this. Instead, our training objective is designed to promote configurations that maximize the expected return across diverse policies in each context. Specifically, we assert that DUPLEX finds diverse policies and generalizes better to OOD contexts than previous state-of-the-art methods while maximizing the training objective (Line 132). We revised Line 131 to state: “The objective of our algorithm is to find a set of diverse policies that maximize the expected return in every context.”\n\n**The Section 3 discussion...**\n\nOur goal is to find diverse policies that can be generalized to different contexts. We make use of function approximators to handle cases where it would not be possible to enumerate all possible contexts that an agent will encounter. We replaced line 118 with:\n```\nWe briefly introduce the main building blocks of DUPLEX and the notation we use. First, we review basic concepts of multi-task reinforcement learning (RL) and explain how it can be represented by the contextual-MDP framework. Next, we describe the universal estimators used to enhance generalization across contexts, in cases where it would be hard to enumerate all possible contexts.\n```\n\n**While I was able to follow Section 4…**\n\nWe welcome the suggestion of the reviewer to improve clarity. We will lighten part of the preview at the beginning of Section 4.1.1 and introduce it at the beginning of Section 4 by describing how the different components are connected and interact.\n\n\n**The design of Equation 5 …**\n\nWe explored various designs for Equation 5. Initially, we tried Lagrangian-constrained optimization but found it to be unstable, especially in domains like GT. Next, we tested a simple hard-threshold approach with `\\beta \\dot v_{target}`, but this made the reward signal discontinuous and more unstable. Ultimately, we used a sigmoid function to modulate the lambda parameter, as shown in Equation 5. This approach provided a continuous reward signal and effectively penalized deviations from near-optimality. Normalizing by `|v_{target} + l|` helped us avoid fine-tuning the sigmoid slope and maintain consistent hyperparameters across domains. We included this description in Appendix C.\n\n\n**The motivation for using an entropy…**\n\nInstability in RL often arises from predicting a moving target. Additionally, the complexity of training can increase depending on the application and task. We found that standard successor feature (SF) prediction is inadequate in multi-context settings.\n\nWe hypothesized that adding an entropy term to SF estimation helps the critic to account for domain uncertainty, similar to the motivation behind entropy in standard RL algorithms like SAC. This approach stabilizes estimation and prevents divergence, as shown in Figure 5. We further explore this in Appendix B and Figure 6, where we demonstrate that adding an entropy term also improves the standard USFA framework, achieving a 3x improvement in multi-context MuJoCo domains.\n\n**The features…**\n\nDefinition 4.1 offers a general definition of policy diversity, aligning with related work [2] and it does not introduce a novel diversity metric (we clarified this in the paper). That said, to evaluate DUPLEX against other baselines, we avoided feature engineering and, in all MuJoCo experiments, used the entire observation vector for cumulants without selecting specific features.\n\nIn GT,  we selected features that better characterize a driving style: action intensities (brake, throttle, and steering values), wheel angles, and distances to the track edges and cars. It is important to mention that in this type of domain, the observation vector includes hundreds of features and it might be difficult to tease out behaviors that can be visually appreciated to the naked eye. Initially, we did use the whole observation also in GT, and DUPLEX was indeed able to find much better diversity trade-offs, but, it was finding diversity also over less relevant features such as how much dirt there is on the tires, or how much tires are on tarmac and how many on the grass. \n\n**Minor comments…**\n\nWe appreciate the reviewer's comments. While we acknowledge the limitations of SFs, we did not find them to hinder our ability to discover diverse behaviors. This is supported by the fact that our approach remains effective even when using averaged cumulant values along a trajectory. That said, it is an interesting future direction that could lead to improving the accuracy of our agent's predictions. As a first step, we could explore n-step returns when computing the prediction errors. We included this observation in the conclusions when discussing limitations.\n\n**Negative societal impact:**\n\nWe will add: RL algorithms optimize decision-making in various domains. However, they also risk reinforcing biases and require careful implementation to ensure transparency and fairness. Diversity-learning algorithms might result in training policies with potentially unethical behaviors that are harder to predict. These are the reasons that motivate us to pursue controllability while learning diverse policies (Section 6, Question 4).\n\nWe thank the reviewer for their feedback, let us know if you have any further questions.\n\n[1] Haarnoja, Tuomas et al. 2017.\n[2] Zahavy, Tom, et al. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for the detailed feedback and insightful comments. We appreciate the time and effort you have invested in this review which has significantly strengthened our contribution.
We considered each point highlighted by the individual reviewers and made several revisions to address their concerns. Here is an overview of the major changes:

- **Clarity**: We have revised the paper to address all the individual concerns raised by the reviewers. These changes have enhanced the overall presentation and made the paper clearer.
- **Related Work and Discussions**: We added the suggested citations and expanded our discussion on limitations and future work for DUPLEX.
- **Canonical RL baselines**: We have highlighted the differences with respect to non-diversity-optimizing baselines and provided additional data points in standard environments.
- **Diversity Rewards**: We have emphasized that the core computation of the diversity rewards for both DOMiNO and DUPLEX are the same.
- **Universal Estimators**: We have provided a clearer explanation of the motivation behind universal estimators in Section 4.1 to enhance understanding and clarity.
- **Code**: Unfortunately, we are unable to share the implementation at this time. The code includes proprietary content and sharing it will infringe our copyright.
- **DOMiNO Visualizations**: we are working towards including additional visualizations of DOMiNO to illustrate its performance and behavior more effectively. In the mean time, we added more data-points and insights into the answers to the rewards.

Thank you once again for your valuable feedback. We are confident that these revisions have strengthened our paper.

Meta-review: Overall, reviewers were mildly positive about this paper, with several thinking the topic is important. Most seemed satisfied with the experimental evaluation. They mentioned "detailed ablations ... is sufficient", the "behavior in GranTurismo 7 is interesting and it's indeed diverse yet competent", and "the experiments performed are comprehensive". Overall, the contribution seems solid enough and worth accepting. Below, I expand on my reasoning below by summarizing the final opinions and justifications of the reviewers.

Reviewer zFsC's concerns were addressed, although they did not acknowledge that their final 3 concerns were addressed. It appears that their remaining concern 2. about needing a vanilla SAC baseline was not addressed. The authors mentioned that they are working on it but that it is not complete. This concern stemmed mainly from a doubt about the integrity of the implementation of another baseline, DOMiNO. The authors clarified that their implementation is built on top of DOMiNO, which lends more credibility to the integrity of its implementation. 

Reviewer Eqtv did not give us an update after the author response. After reading their concerns and author response again, the concerns seem plausibly resolved, except for the concern about the absence of the implementation code.

Reviewer 9gQo considers the paper complete and sound, but had concerns about the weakness of the claims and extent of the novelty. The authors responded to these points, and I found their responses reasonable, adding support for the extent of the novelty.

Reviewer VFg8 did not have any major concerns (they recommended weak accept), and stated that they maintain this score after reading the author response.

Reviewer XDei recommended borderline accept, and stated their major concerns were answered/mitigated by the author response.
Decision: Accept (poster)