Discovering Creative Behaviors through DUPLEX: Diverse Universal Features for Policy Exploration

Authors: borja g leon
francesco  riccio
kaushik  subramanian
peter r wurman
peter  stone
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 10:25:45

Abstract: The ability to approach the same problem from different angles is a cornerstone of human intelligence that leads to robust solutions and effective adaptation to problem variations. In contrast, current RL methodologies tend to lead to policies that settle on a single solution to a given problem, making them brittle to problem variations. Replicating human flexibility in reinforcement learning agents is the challenge that we explore in this work. We tackle this challenge by extending state-of-the-art approaches to introduce DUPLEX, a method that explicitly defines a diversity objective with constraints and makes robust estimates of policies’ expected behavior through successor features. The trained agents can (i) learn a diverse set of near-optimal policies in complex highly-dynamic environments and (ii) exhibit competitive and diverse skills in out-of-distribution (OOD) contexts. Empirical results indicate that DUPLEX improves over previous methods and successfully learns competitive driving styles in a hyper-realistic simulator (i.e., GranTurismo ™ 7) as well as diverse and effective policies in several multi-context robotics MuJoCo simulations with OOD gravity forces and height limits. To the best of our knowledge, our method is the first to achieve diverse solutions in complex driving simulators and OOD robotic contexts. DUPLEX agents demonstrating diverse behaviors can be found at https://ai.sony/publications/Discovering-Creative-Behaviors-through-DUPLEX-Diverse-Universal-Features-for-Policy-Exploration/.

Main paper: 
 introduction: 
 In non-stationary complex environments, reinforcement learning (RL) (Sutton & Barto, 2018) agents are compelled to exhibit flexible and diverse behaviors to robustly adapt to different scenarios and interact with other actors (Vinyals et al., 2019; Zahavy et al., 2023b) . To this end, a growing community is researching methodologies to train agents that, unlike conventional RL (Haarnoja et al., 2018; Espeholt et al., 2018) , can solve tasks with a diverse set of near-optimal strategies (Zahavy At each iteration, we provide three inputs to our multi-policy agent: (i) a context vector describing task requirements and environment dynamics in the current episode c; (ii) an encoding of the policy used in the episode z; (iii) and the current state of the environment s. The critic network returns estimates for the intrinsic and extrinsic rewards and successor features to drive diverse behavior discovery. Finally, the algorithm samples policies in Π uniformly and rolls them out to collect more experience. et al., 2023a) . Such methodologies are explicitly crafted to enhance the exploration of the state-action space, equipping agents with the capability to discover qualitatively diverse solutions for a given task distribution. For example, when planning a route from home to work, a human commuter might identify one route that uses the highway and one that sticks to side streets. While using the highway may be optimal in expectation, using the alternative route may be called for different contexts that influence the traffic report, or weather forecast.Existing approaches struggle to generalize to highly dynamic environments and are designed to learn in single-task settings (Zahavy et al., 2023b; Fu et al., 2023) . To alleviate this, we introduce Diverse Universal features for PoLicy EXploration (DUPLEX), an algorithm that trains an agent to optimize a set of diverse policies over different contexts. In this work, contexts comprise two components: a task requirement and a description of the current environment dynamics. To better ground such concepts, let us refer to the previous commuter example. In this case, we would refer to task requirements as time-to-completion. On the other hand, factors such as the weather forecast and traffic report would form part of the environment dynamics DUPLEX builds on top of the strengths of diversity learning (Zahavy et al., 2023a) in RL and universal estimators (UE) (Schaul et al., 2015; Borsa et al., 2019) to introduce a novel methodology that preserves both performance and diversity in highly dynamic environments and multi-context settings -which is key to enabling human-level task execution. For example, if the commuter is a human that injures their leg, they can immediately balance on one leg and even jump forward without using the injured leg. We aim to transfer this adaptability to agents in different contexts. To this end, we adopt the Contextual Markov Decision Process (CMDP) framework (Hallak et al., 2015b) , where different episodes correspond to different contexts. As shown in Figure 1 , DUPLEX is designed to receive three inputs representing the encoding of the context, policy, and state. Contexts are represented as feature vectors and uniformly sampled from a discrete set of predefined task requirements and dynamics settings (e.g., weather condition); the i-th policy in the set of policies Π is selected and passed to the model (e.g., different routes); and the state comes directly from the environment observation. Then, the critic output is split into different heads to support the estimation of the extrinsic and intrinsic rewards, and successor features (SFs) (Barreto et al., 2017) . We refer to the reward coming from the environment as the extrinsic reward, while the metric computed to promote diversity is the intrinsic reward. It is worth noting that, at the beginning of each training episode, we uniformly sample a context from C and a policy from Π to collect experience. DUPLEX then iteratively trains the set of policies to maximize both the distance in their successor features (guided by intrinsic rewards) and the task performance (given by the extrinsic rewards). We rely on SFs to measure distances as they, by definition, represent the state features that a given policy is expected to experience along a trajectory, and thus, they are an intuitive way to quantify policy diversity. Hence, our main contribution is DUPLEX, a novel algorithm that contributes to diversity learning in RL by improving on previous work to better preserve the diversity vs. near-optimality trade-off in highly-dynamic environments and multi-context settings. We evaluate our approach on the real-time, physically realistic, car-racing simulator GranTurismo ™ 7 (GT) (Wurman et al., 2022) and on two multi-task MuJoCo environments with changing dynamics (Walker2D, Ant) (Todorov et al., 2012) . Our experimental results indicate that DUPLEX improves over previous state-of-the-art RL diversity baselines (Zahavy et al., 2023a) and UE baselines (Schaul et al., 2015; Borsa et al., 2019) . In fact, DUPLEX is the only algorithm to learn diverse competitive policies in GT and outperforms other baselines when evaluated in out-of training distribution (OOD) contexts. Moreover, we conduct a detailed ablation study that isolates the effects of each of DUPLEX's main components. 
related work: 
 In this work we build on top of two main bodies of research: diversity learning and universal function approximators. We organize related work accordingly.Diversity. Diversity learning is gaining attention within the research community, due to the important benefits it yields to autonomous agents. Such benefits include generalization (Eysenbach et al., 2018) , exploration (Gehring et al., 2021; Czarnecki et al., 2020) , creativity (Zahavy et al., 2023b) , and self-play Vinyals et al. (2019) . In this work, we extend the application of diversity learning to highly-dynamic environments and multi-context settings. A central precursor to our work is DOMiNO Zahavy et al. (2023a) . DOMiNO employs RL to maximize extrinsic and intrinsic diversity rewards, which are combined with Lagrange multipliers and Van der Waals (VdW) forces to balance diversity and performance. DOMiNO has two variants: one bases its diversity objective on the average of input features which grants stability at the cost of storing a moving average for every task -limiting scalability. The second variant uses the critic to estimate SFs, which solves the scalability issue at the cost of instability due to the added learned target. Intuitively, such limitations prevent DOMiNO from being effective at discovering diverse behaviors in CMDPs. Further, the Lagrange multipliers and VdW forces can limit diversity in complex environments where creative behaviors need significant exploration before providing satisfying returns (Vinyals et al., 2017) . Such limitations become more evident in highly dynamic environments such as GT (see Section 5). Our algorithm is instead designed to tackle these limitations.Quality-Diversity (QD) is another active line of work within diversity learning that involves evolutionary methods. QD aims to generate large collections of diverse and high-performing solutions, primarily through evolutionary optimization (Pugh et al., 2016; Grillotti & Cully, 2022) . In QD, diversity is measured through a descriptor space defined by the user (Cully et al., 2015) . Some of the main approaches in QD include MAP-Elites (Mouret & Clune, 2015; Flageat et al., 2023) , local competition (Lehman & Stanley, 2011) and more recently those that combine RL with QD (Nilsson & Cully, 2021; Faldor et al., 2023; Tjanaka et al., 2022) . Closer to our research, Anne & Mouret (2023) introduce a variant for MAP-Elites to learn diverse behaviors for multiple tasks within the training distribution. Our framework diverges from QD in several ways. First, QD algorithms emphasize diversity through evolutionary strategies, while our method relies on maximizing a reward objective. Second, QD requires the user to define a diversity descriptor, while we only restrict diversity to be in the near-optimal space. Third, one of our main goals is finding competitive diverse behaviors for OOD tasks and dynamics, an objective that has not been studied in Quality-Diversity (QD).Universal Function Approximators. To make more robust estimations of the policies' expected behaviors, and thus to better quantify diversity, our work pivots around universal function approximator (UE). UE research is grounded on factoring the value estimates separating states from tasks (Schaul et al., 2015) and policies (Ma et al., 2020) . Notably, Borsa et al. (2019) improve the formalization of successor features (Barreto et al., 2017) , an estimation of state-action visitation, by conditioning their estimation on both task and policies. Still, approaches based on SFs are brittle in complex domains (Carvalho et al., 2021 (Carvalho et al., , 2023)) , requiring sophisticated network architectures (Mittal et al., 2020; León et al., 2022) to work effectively. Akin to Mozifian et al. (2021) , we adopt a similar approach to transferring learning of SFs to continuous domains by combining them with SAC (Haarnoja et al., 2018) . However, we achieve an improved estimation of the expected SFs by employing the average of the critic outputs and by adding the entropy term to the SFs learning objective. We find that the addition of the entropy term is a novel component that DUPLEX carries with and it is key to improving robustness and performance (see Appendix B).Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discussed the limitations of our framework in Section 6. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
background and notation: 
 We briefly introduce the main building blocks of DUPLEX and the notation we adopt. First, we review basic concepts of multi-task RL and explain how it can be represented by the contextual-MDP framework. Next, we describe how universal estimators are used to enhance generalization across contexts (especially when context enumeration is impractical).Multitask RL. We consider training a multi-policy RL agent that solves a CMDP (Hallak et al., 2015a) represented as a tuple M = ⟨S, A, P, R, γ, µ C , µ S ⟩, where S, A are the state, action spaces respectively, R is the reward function, P is the unknown transition function, µ S is the initial state distribution conditioned on the context, and µ C is the context distribution. We use context c ≡ {u, w} to summarize both information about the particular dynamics u of that environment (e.g., the effect of gravity), and information about the task w in the current episode (e.g., position constraints). Every episode starts with a context c ∼ µ C and an initial state s 0 ∼ µ S (• | c). Then, at each time step t, the agent selects an action a t according to its policy π(• | s t , c), receives a reward r t ∼ R(s t , a t , c) and transitions to the next state s t+1 ∼ P (• | s t , a t , c). Policies are characterized by their state-action occupancy, i.e., how often a policy visits a state-action pair. We consider two state-action occupancy metrics:d avg πc (s, a) = lim T →∞ 1 T ET t=1 P πc (s t = s) π(s, a, c) for the average occupancy metric and d γ πc (s, a) = (1 -γ)E ∞ t=1 γ t P πc (s t = s) π(s, a, c) for the discounted case -where P is the probability measure of states at t induced by π in c. The objective of our algorithm is to find a set of diverse policies that maximize the expected return in every context max dπ c ∈K s,a,c r(s, a, c)d πc (s, a), where K is the set of admissible distributions (Zahavy et al., 2021) .Universal estimators. When aiming to solve multiple tasks, we follow the common approach of decomposing the input of the neural network to facilitate transfer learning between tasks and policies. Methods that are relevant for our work include "Universal Value Function Approximators" (UVFA) (Schaul et al., 2015) , which add a task-descriptor vector w as input to a value function approximator parameterized by θ, V θ (s, a, w). If V θ is smooth w.r.t. w, then V θ is expected to generalize across tasks within the training task space. Akin to previous works (Barreto et al., 2017; Zahavy et al., 2023a) , we assume that every state-action pair is correlated with observable features known as "cumulants" ϕ(s, a, c) ∈ R d . These cumulants can either be given through relevant properties within the state2 or be learned. In a similar fashion to value functions, we can define expected features ψ πc (s, a) = E s ′ ,a ′ ∼dπ c (s,a) ϕ(s ′ , a ′ , c) ∈ R d , which we will refer to as average (ψ avg ) or discounted (ψ γ ) expected features when using d avg πc and d γ πc respectively. In the discounted case, ψ γ are also known as "successor features" (SFs). This latter formalization is key to our work and also to USFA (Borsa et al., 2019) , a general framework that combines UVFA-like task decomposition with the SFs, and exploits generalized policy improvement algorithm (GPI) Barreto et al. (2017) . Importantly, USFA disentangles both tasks and policies by giving as input a vector z that is a representation of the current policy z = e(π), where e is an encoding function -in our approach e encodes policies as one-hot vectors z.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We include details about the compute resources and hardware setup in Appendix E.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
learning near-optimal diverse behaviors in multi-goal continuous settings: 
 DUPLEX is a method designed to provide diverse solutions to complex tasks by simultaneously maximizing both the true reward and dissimilarity within a set of policies Π across the different environments of a CMDP. Thus, according to previous work (Zahavy et al., 2023a) we define diversity as:Definition 4.1 (Diversity) Diversity(Π) is a metric of dissimilarity among policies in a set Π with a common goal. Formally, if ψ πi and ψ πj are a function of state-occupancy of relevant features of any two policies in Π, then their dissimilarity is given by ||ψ πi -ψ πj ||. A non-zero value of this norm indicates dissimilarity, with larger values indicating greater divergence between the policies.Mathematically, diversity is defined as the sum of the minimum L2 dissimilarity norms in Π:Diversity(Π) = 1 2 size(Π) • ∀πi,πj ∈Π, i!=j min ||ψ πi -ψ πj || 2 2Using this definition, we formulate our learning problem and describe DUPLEX's main contributions in the following sections. We first extend the formulation of diversity learning to contextconditioned environments, then we introduce a novel mechanism to more efficiently control the diversity-performance trade-off, and finally, we describe how to improve SFs estimation.Similar to Zahavy et al. (2023a) , our objective is to maximize diversity within a set of policies Π. We express the distances between policies in Π in terms of expected features ψ, which are a function of their state-occupancy (Borsa et al. (2019) ; Carvalho et al. (2023) ). We measure ψ distances following Def. 4.1 and use a context-based Hausdorff distance (Rockafellar, 1970) to enforce context-conditioned diversity within Π. We aim at training an RL agent that, given a context c, discovers a set of n near-optimal policies Π = π i c n i=1 by successfully optimizing:max Π Diversity (Π) s.t d πc • r e ≥ ρv e , ∀π c ∈ Π (1)where d πc is the occupancy metric of a context-conditioned policy, r e is the environment reward (or extrinsic reward), ρ ∈ [0, 1] is a hyper-parameter defining the near-optimality region that we refer to as optimality-ratio, and ve is the value of a target policy. In practice, the target policy refers to a policy in Π that ignores the diversity objective and is trained exclusively to maximize the extrinsic rewards. As in Zahavy et al. (2023a) , the policy value v expresses the expected reward accumulated by a policy π. According to Definition 4.1, diversity is a distance over a set of occupancy metrics Diversity : R |S||A||C| n → R and Eq. 1 is defined to maximize the diversity of the occupancy metrics and preserve the near-optimality of the policies in Π.To promote diversity between different policies, we adopt the repulsive reward from Zahavy et al. (2023a) and extend it to be conditioned on the context. More formally, given the tuple ⟨s, a, π i , c⟩ and a cumulant function ϕ, the diversity reward that we maximize for is:EQUATIONwhere ψ π i c are the expected features from policy π i ∈ Π at (s, a, c) and πi c ∈ Π refers to the policy with the closest expected features to π i in (s, a, c) according to Def.4.1. Intuitively, since contexts c ∼ µ C are fixed for each training episode, this reward encourages the algorithm to train policies that visit different state-action pairs within each context c, thus, promoting context-conditioned diversity.Eq. 2 expands DOMiNO's repulsive force to operate across different contexts. However, the scale of intrinsic rewards varies greatly through different environments (i.e., different contexts will inherently yield different successor features). We need to stabilize fluctuations of such rewards when working with diverse contexts. We introduce two constraints to the learning objective that modulate the intrinsic reward r I : a dynamic intrinsic reward factor χ that scales r d to target a factor of the moving average of the general extrinsic value and a soft-lower bound λ that limits the search of diverse policies to a near-optimal subspace. We then define the DUPLEX intrinsic reward as:r I = λ • χ • r d (3)where χ and λ are computed independently by the algorithm.Dynamic intrinsic reward factor. χ scales r d proportionally to the sum of extrinsic values of policies in Π. Formally,r d ∝ v eavg = 1 n n i=1 v i eavg with v i eavg = α vavg v i eavg +(1-α vavg )r i e,twhere, α vavg ∈ [0, 1] weights the contribution of the extrinsic value of the i-th policy and its immediate extrinsic reward. Finally, r i e,t is the extrinsic reward at t when the agent acts under policy i. Intuitively, we are scaling the intrinsic rewards according to the average extrinsic value that the set of policies is achieving. Hence, at each algorithmic iteration, χ is updated as follows:χ t = α χ χ ′ t + (1 -α χ )χ (t-1) (4)where α χ is the update rate of χ, andχ ′ = |v eavg /v davg |(1 -ρ)is the target value of the update, where v davg is the return value based on diversity intrinsic reward r d while v eavg is based on the extrinsic reward r e instead. Through the optimality ratio ρ, χ minimizes the domain dependency by scaling the intrinsic rewards as a factor of the extrinsic objective and the optimality ratio ρ. Regarding the update rate α χ , once we converged to a stable value, we kept it fixed across all experiments and domains.Soft-lower bound. While χ preserves a relationship between extrinsic and intrinsic rewards, it does not prevent policies in Π from exploring regions of the search space that are too far from the nearoptimality regions of the target policies. To limit this possibility, we explored Lagrangian-constrained optimization but found it unsatisfactory in complex domains like GT. Hence, we introduce λ to bound the near-optimal subspace for each policy using:EQUATIONwhere σ k (x) = 1/(1 + e -kx ) and β ∈ [0, 1] is a hyper-parameter indicating the reward region we are interested in exploring, k regulates how "soft" is the bound, n is the number of policies, l is a small constant to prevent division by zero, and v eavg is the average extrinsic value. Intuitively, λ limits the exploration of diverse behaviors to a near-optimal area defined by the threshold β. We find that introducing a sigmoid-based limit provides a more stable solution than Lagrangian-constrained optimization (See Figure 2 ). Similarly to α χ , once we found a stable value for k, it remained fixed for all experiments.Since we rely on ψ to determine policy diversity (Def. 4.1), it is fundamental to estimate ψ reliably. Additionally, as mentioned in Section 3, it is not tractable to keep ψ avg for each context and task since they may be infinite. Thus building on top of UE, we exploit discounted expected features (SFs) ψ γ (s, a, z, c) and incorporate an extra head in the critic to estimate SFs ψγ (s, a, z, c).However, as in related work (Carvalho et al., 2023) , we found that the critic struggles to correctly estimate ψ γ in complex settings. Specifically, since intrinsic rewards are bootstrapped over the expected difference of successor features, the accuracy in the SFs estimation plays a major role.Taking inspiration from the SAC algorithm (Haarnoja et al., 2018) , we incorporate an entropy term in the expected features objective to account for the stochastic component within the learned policies.For policy i in context c, our SFs at s t are computed as:EQUATIONwhere α H is the entropy weight. Intuitively, unless the critic is confident that the actor will visit a state-action pair more frequently than others, the network will be encouraged to estimate that the policy will maximize the entropy. We formulate our temporal difference loss for the critic output ψγ to account for the entropy term as:TDH( ψγ θ j (π z c )) = E (s,a)∼π z c 1 2 ψγ θ j (s, a, c, z) -y(ϕ, s ′ , c, z) 2 (7)where, similar to SAC, we have two critic estimates j = {1, 2}, but instead of taking the minimum of the two, our target is obtained from the average of the estimates. Then, we define y(ϕ, s ′ , c, z) as:EQUATIONwith ã′ z ∼ π z ω (•|s ′ , c). The motivation lies in the different impacts of overestimation of values and successor features. When predicting values, overestimation directly influences the policy since it is trained to maximize advantage. However, in DUPLEX, SFs assume a distinct role: the policy is not incentivized to select actions with the highest successor features but rather those that differ the most from the SFs of the remaining policies (subject to not sacrificing too much performance, equation 1). Consequently, precision becomes of greater significance for ψ γ , and taking the average emerges as a more reliable solution.The introduction of entropy improves stability in the estimation of SFs through most of the training process and, as we report in the experimental section, it supports OOD generalization. However, as learning stabilizes, we typically encounter phases where the estimated difference between policies rapidly increases its order of magnitude. For this reason, we also introduce a fixed upper bound to α χ when estimating SFs. Details on the upper bound, DUPLEX pseudocode, and hyper-parameters are included in Appendix C and D.Summarizing, DUPLEX enhances diversity learning in context-conditioned environments through four key components: (i) dynamic intrinsic reward factor (χ) that balances diversity with extrinsic rewards and avoids environment-specific tuning; (ii) soft lower bound (λ) that constrains policies to optimize for diversity only within a target near-optimal region; (iii) entropy regularization (α H ) in successor feature estimation to improve robustness; and (iv) averaging over critic estimates for computing diversity rewards. In Section 5 we demonstrate how these features improve both performance and diversity of the learned policies. 
experiments: 
 The objectives of our experiments are to demonstrate that DUPLEX: 1) is the first successful method in learning diverse driving styles in highly-dynamic physics simulators such as GT (Figure 2 ); 2) when Baselines. Our main baseline is DOMiNO (Zahavy et al., 2023a) , which is the state-of-the-art of diversity learning in RL. Then, since DOMiNO is not designed for OOD generalization, our experiments in OOD settings include as baselines the two most popular frameworks within UE literature: UVFA (Schaul et al., 2015) and USFA (Borsa et al., 2019) . Note that, throughout the section and given the two versions of DOMiNO, we will refer to DOMiNO avg as the version of the algorithm using averaged cumulants as successor features, and as DOMiNO est as the version of the algorithm that estimates SFs.Benchmarks. We use three benchmarks. The first is a racing track in GT, where the goal is to have different driving styles while completing fast laps. Achieving diverse and competitive driving styles in such realistic simulator Wurman et al. (2022) can unlock multiple applications both in gaming and self-driving. Second, a set of experiments is conducted in the MuJoCo's Walker2D and Ant environments where we compare DUPLEX against DOMiNO avg in the same environments used in Zahavy et al. (2023a) . Finally, in the last benchmark, we evaluate our approach in a multi-context version of MuJoCo's Walker2D and Ant environments. We extend the default versions of these scenarios to include a three-dimensional context c where the inputs represent: the gravity coefficient and, the upper and lower height thresholds we would like the agent to respect while moving. The goal of this benchmark is to find diverse policies in OOD contexts while guaranteeing competitive performance in task execution. Note that, to have a fair comparison against the baselines, only the last benchmark configures DUPLEX to estimate SFs through ψ γ . Additional details about the metrics and the environments are reported in Appendix A.Results in GranTurismo ™ 7. As in Wurman et al. (2022) , we use QRSAC as the base RL algorithm for both DOMiNO avg and DUPLEX. Figure 2 (top-left) shows that only DUPLEX learns diverse policies in the near-optimal region. Figure 2 (c) illustrates how this diversity translates into diverse lap times. Moreover, Figure 2 (b) provides evidence of the benefits that our lower bound β formulation carries with it. When using β = 0, the most diverse policies are "busy" finding different ways to not finish the track. Furthermore, we see DUPLEX results oppose to DOMiNO that instead collapses all policies to a similar behavior even when using an extremely forgiving value of the optimality ratio ρ. It is worth mentioning, that DOMiNO fails to provide diversity even if it implements a Lagrangian constraint optimization objective that should support more adaptability. For example, when configuring ρ = 0.01, DOMiNO should consider as "acceptable" any diverse policy achieving 1% of the value of the target policy veavg . But still, the algorithm struggles to find diverse policies.Results in MuJoCo. To further validate the contribution of DUPLEX to diversity learning, we include a direct comparison of DUPLEX and DOMiNO avg while configuring the Walker2D environment that the authors present in Zahavy et al. (2023a) . Within this set of experiments, both algorithms have been configured to work with ψ avg which fosters the opportunity to ablate the benefits of our constrained r I (Eq. 3) isolated from the impact of estimating SFs. Figure 3 includes the results from both frameworks with multiple optimality ratios ρ. Moreover, to provide an exhaustive evaluation, we also configure DOMiNO avg with various Van Der Waal (VdW) distance hyper-parameters. Such a parameter modulates the desired distance between all the policies in Π (see Zahavy et al. (2023a) for details). Finally, DUPLEX is able to find a better Pareto-frontier for the quality-diversity objective (Figure 3 right), while achieving better near-optimal policies than the baseline (Figure 3 Results in multi-context MuJoCo. Figure 4 reports the results of our baselines in multi-context scenarios when evaluated within-and out-of training distribution. In the within-distribution setting (Figure 4 left-hand side) only DUPLEX shows good performance and diversity while USFA and DOMiNO fail at both learning competitive policies and providing diversity. As we report in the USFA ablation (B), DUPLEX succeeds in such benchmark due to the entropy term introduced to make the estimation of SFs more robust (Eq. 7). Additionally, note that USFA and UVFA do not optimize for diversity, and their diversity score is expected to be close to zero. Then, Figure 4 (right-hand side) reports the generalization effectiveness of DUPLEX and UVFA when operating out-of training distribution. Note that USFA and DOMiNO do not qualify in this setting as they failed to learn already in the training set. We observe that in all the OOD scenarios DUPLEX outperforms UVFA and, most importantly, it covers a bigger range of diversity values while completing the tasks.Ablation of successor-feature estimation. Figure 5 illustrates the contribution of ψ δ and the different features incorporated by DUPLEX. We have evidence that all features that compose DUPLEX are key to guaranteeing robust learning of diverse behaviors in highly-dynamic environments and multi-context settings. Notably, USFA can also effectively solve multi-context MuJoCo environments when akin to DUPLEX, adds the entropy term (see Appendix B). 
conclusions: 
 DUPLEX provides a powerful technique that enables learning of diverse near-optimal behaviors. Our experimental session shows that (differently from other baselines) DUPLEX (i) learns effective diverse behaviors in hyper-realistic complex domains (such as GranTurismo) and (ii) generalizes to OOD in challenging multi-context MuJoCo environments by demonstrating competitive diverse behaviors.Limitations and Future Directions. There is potential to refine and enhance the methodology presented in this work. Nevertheless, we perceive the current limitations of DUPLEX not as setbacks, but as exciting opportunities that pave the way for new research avenues. For example, (1) can we generate more than twenty diverse policies and still guarantee meaningful diversity? (2) DUPLEX carries an additional cost due to the need to compute SF distances of each policy pair in our set, thus how can we improve sample efficiency and keep a low computational footprint? (3) DUPLEX does not impose any exploration strategy on each policy, and thus, can we control in what measure a particular policy will be different? And finally, (4) can we combine the strengths of different on a single solution? This latter question is very interesting to us. We suppose that once diverse policies have been learned, each of them retains different useful skills (e.g. different commuting routes) that can be combined dynamically to reconstruct a single agent capable of acting optimally and tackling unseen portions of the state-space.Finally, the presented approach holds significant promise for advancing research in diverse policy learning, scalability, and efficient transfer and exploration (Parker-Holder et al., 2020; Peng et al., 2020) across varied contexts. We note also that RL algorithms may also risk reinforcing biases and require careful implementation to ensure transparency and fairness. Diversity-learning algorithms might result in training policies with potentially unethical behaviors that are harder to predict. These are reasons that motivate us to pursue controllability while learning diverse policies in this and future works. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: Yes, we believe the research on DUPLEX and empirical analysis conform to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper does include the release of data or models and as such does not pose a risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have provided citations and references to existing MuJoCo and GT assets, including GT's trademark in Section 1.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not include the release of any new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: IRB approval was not required as the research does not include any work with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In MuJoCo environments, we measure diversity directly on the agent's observations, which is represented as a feature vector containing the angles and velocities of the agent's joints. In GT, diversity is measured through the most characterizing descriptors of a driving style: action intensities (brake, throttle, and steering values), wheel angles, and distances to the track edges.Note that these diversity features were chosen to align with each environment's objectives. Nevertheless, we can maximize the diversity of any feature in the observation. Such an arbitrary selection of features intuitively leads to different optimization objectives and thus learned strategies. For instance, in GT, we could maximize diversity on the frequency of hitting other cars, and this would lead to a population of policies that are all driving as fast as possible while exploring the spectrum of aggressiveness: ranging from a very timid behavior (letting opponents pass to avoid contact) to a very aggressive one (hitting cars intentionally if the collision does not result in a significant speed loss). In the OOD scenarios, we used two out-of-distribution contexts for the Walker2D and the Ant environments respectively. In the former, we used two values of the gravity coefficient and a lower bound constraint, namely c 0 = ⟨3, 0.6, N one⟩ and c 1 = ⟨24, 0.6, N one⟩, while in the latter we used c 0 = ⟨3, 0.2, N one⟩ and c 1 = ⟨24, 0.2, N one⟩ B Ablation in USFA In this section, we ablate the addition of the entropy term to the SFs objective function. In particular, we extrapolate such component from the others introduced by DUPLEX, and we evaluate the entropy term in isolation within the original USFA framework. The assumption that we validate with this ablation study is that Equation 7 has a broader impact and that it makes the estimation of SFs more robust. Figure 6 illustrates our findings, we report that the entropy term enables effective learning in multi-context MuJoCo environments and significantly improves performance w.r.t. vanilla USFA.Importantly, Figure 6 illustrates that, using avg(ϕ θ1,2 ) instead of the min(ϕ θ1,2 ), is not beneficial for USFA. Note that differently from DUPLEX, USFA infers extrinsic rewards from SFs. Thus, in USFA an overestimation of SFs (i.e., overestimating how often the policy visits some state-action Algorithm 1 DUPLEX 1: Initialize dataset with random exploration, set up critics θ and policy parameters ω. and initialize moving averages {v π i avg } n i=1 = 0 reward weight κ = 0, χ = 1 and step = 0 2: while not converged do 3:Generate more data samples 4:Sample batch of m transitions {tj} m j=1 , where a transition is tj = (s j , a j , c j , ϕ(s j , a j , c j ), z j , r j e , s ′j ) 5:Use critic to get ṽπ i e j , ṽπ i i j and ψγ (s, a, c, z) for every π z ∈ Π. Where ψ = avg h=1,2 ψθ h 6: Compute rI (s) from ψ, z, ϕ(s, a, c) with Equation 3 7:Compute extrinsic temporal difference (TD) errors and q-values: TDe, Qe 8:For sampled z get TD error with entropy on successor features with Equation 7: TDH 9:Compute intrinsic TD errors and q-values: TDI , QI 10:Combine q-values Q = σ(κ)Qe + (1 -σ(κ))QI 11:if step > critic warming-up period then ▷ Policy loss:12: ∇ω j min h=1,2Q θ i (s, πω(•|s j , c j , z j )) -α log πω πω(•|s j , c j , z j ) s j , c j , z j 13: Calculate weighted critic loss: bv(TDe + TDi) + b ψ (TDH ) 14:if Constrained optimization then ▷ Lagrange loss 15:n i=1 σ(κ i )(v i avg -ρvavg) 16:Update κ 17:Update θ and ω 18:Updatev i eavg = αv avg v i eavg + (1 -αv avg )r i e,t , v i davg = αv avg v i davg + (1 -αv avg )r i d,t, χ with Eq. 4 and step + = 1 pairs) yields an overestimation of rewards. Such an overestimation motivates our design choice of using the avg of two critics in DUPLEX. In fact, since DUPLEX relies on the distance between SFs to estimate intrinsic rewards, we achieve a more accurate representation of intrinsic rewards by taking the average of the two estimates.DUPLEX follows an actor-critic approach, where a policy network determines actions to take while a critic network estimates values and, in the case of multiple contexts, also successor features. Figure 1 depicts the workflow of our algorithm. The networks have three independent input encoders for the state, policy, and context respectively. Since we use SAC as the base RL algorithm, we employ two critic networks and take the min of the two for value estimates ṽγ and the average for ψγ (Eq. 8).The training procedure of DUPLEX is described in Alg. 1. After warming up to collect data, we start sampling transitions t and the learning routine (Line 4). Since we need to estimate the distance between all policies to compute the intrinsic rewards, we broadcast the input of the critic to have the SF estimates for all policies in every transition and do a forward pass through the critic (Line 5). Then, we obtain r I with Eq. 3 and compute the temporal-difference errors (Line 7 -Line 9). Similarly, after a short warm-up period, we compute the policy loss (Line 12). Then, depending on if we enable constrained optimization, we update the Lagrange multipliers (Line 15). Finally, we update the network weights and the average policy values to update the dynamic factor χ (Line 17 and Line 18).Upper bound on α χ . As described in Section.4, when the critic learns to estimate SFs, we typically observe events where the estimated difference between policies rapidly increases its order of magnitude. Such a shift in the magnitude of expected distances results in a rapid increase of intrinsic rewards, larger value estimates, and ultimately, the divergence of the critics output. To address such magnitude shifts, we modify the update rate of χ:αχ = 1 if |v davg | > ϵχ|ve avg | αdefault otherwise (9)Intuitively, we want r I to motivate the agent to explore increasingly diverse behaviors without stagnating the extrinsic rewards. For this reason, we need α χ to be a very low value -we use 1e-5 across all our experiments. An exception is made when a sudden drastic change in ψ could cause the critic to diverge. We automatically detect those changes through a threshold ϵ χ |r avg e |, where ϵ χ is a scalar, and perform a one-step update to prevent r I from destabilizing the learning process. Similarly to other hyper-parameters, once we found a stable value for ϵ χ it remained fixed for all experiments and benchmarks.Constrained optimization. Within the algorithm framework, we allow Langrangian constraint optimization Borkar (2005) to be contingent to specific domains. When constrained optimization is configured, we establish a multi-goal objective Altman (1999) ; Szepesvári (2020) using Lagrange multipliers κ, whose objective is to maximize κ(ρv e -d π • r e ). Intuitively, this objective yields a multiplier κ that decreases in value when the policy is satisfying the optimality constraint, and increases otherwise. However, we empirically find that while Lagrangian constrained optimization might be beneficial in canonical domains, it is not effective in complex environments such as GT and fails to find competitive diverse behaviors. Nevertheless, Equation ( 4) and 5 make our r i less reliant on constraint optimization than previous approaches. For these reasons, we implement constrained optimization in DUPLEX by keeping κ fixed by default and making it learnable as an optional feature.Remark C.1 DUPLEX is designed so that only two hyper-parameters significantly impact its performance: ρ and β. The former regulates how much the user wants to weight the intrinsic rewards w.r.t. the extrinsic reward, while the latter defines the near-optimality region that the agent should explore when learning diverse policies. Given the additional components that DUPLEX requires to stabilize training, we report in the following table the computational cost of the update function of the core learning algorithm while varying the number of used policies. We compare our approach to DOMiNO and vanilla SAC and demonstrate that the additional cost is negligible when considering related work, and does not increase with the number of policies. Execution times of the selected algorithms are reported in milliseconds and have been evaluated on the Walker2D MuJoCo environment rolled out on a 13th Gen Intel(R) Core(TM) i9-13900HX and NVIDIA GeForce RTX 4080 Laptop GPU. We ran our experiments on an internal cluster designed for distributed training. Each job has been configured with a trainer (to update the models) and N rollout workers (to collect data).MuJoCo. In this environment we use one trainer and one rollout worker -none of them equipped with a GPU. The former uses 7.7 vCPUs and 8Gi of RAM, while the latter uses 1 vCPUs and 2Gi of RAM. The duration of each experiment is two days.GT. In this environment, each job is configured with one trainer and ten rollout workers. The trainer is equipped with 7.7 vCPUs, 1 Nvidia V100, and 55Gi of RAM, while each rollout worker features 2000m vCPUs, 3328Mi of RAM, and 1 PlayStation4. The duration of each experiment is seven days.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We listed and described claims made in this work both in the abstract and the introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: There is no direct societal impact of the work described in the paper. We focus on learning diverse policies in dynamic domains to advance the field of RL.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).For example, in a car racing scenario features could correspond to relevant information for the car such as the course velocity or if a car has crashed into another 


Decision: Overall, reviewers were mildly positive about this paper, with several thinking the topic is important. Most seemed satisfied with the experimental evaluation. They mentioned "detailed ablations ... is sufficient", the "behavior in GranTurismo 7 is interesting and it's indeed diverse yet competent", and "the experiments performed are comprehensive". Overall, the contribution seems solid enough and worth accepting. Below, I expand on my reasoning below by summarizing the final opinions and justifications of the reviewers.

Reviewer zFsC's concerns were addressed, although they did not acknowledge that their final 3 concerns were addressed. It appears that their remaining concern 2. about needing a vanilla SAC baseline was not addressed. The authors mentioned that they are working on it but that it is not complete. This concern stemmed mainly from a doubt about the integrity of the implementation of another baseline, DOMiNO. The authors clarified that their implementation is built on top of DOMiNO, which lends more credibility to the integrity of its implementation. 

Reviewer Eqtv did not give us an update after the author response. After reading their concerns and author response again, the concerns seem plausibly resolved, except for the concern about the absence of the implementation code.

Reviewer 9gQo considers the paper complete and sound, but had concerns about the weakness of the claims and extent of the novelty. The authors responded to these points, and I found their responses reasonable, adding support for the extent of the novelty.

Reviewer VFg8 did not have any major concerns (they recommended weak accept), and stated that they maintain this score after reading the author response.

Reviewer XDei recommended borderline accept, and stated their major concerns were answered/mitigated by the author response.

Reviews: review: paper_summary: paper_summary: introduced a method to enhance diversity in RL policies by using successor features for robust behavior estimation. Experiments show DUPLEX outperforms state-of-the-art baselines, achieving diverse, competitive policies in GranTurismoTM 7 and multi-task MuJoCo environments, even in out-of-distribution contexts.
strength_weakness: strength_weakness: I am not familiar with work related to universal function approximators, but I am knowledgeable about quality diversity and RL-related work. Therefore, I will focus on raising questions about the latter.

1. In 4.1.1, the authors mention, "We explored Lagrangian-constrained optimization but found it unsatisfactory in complex domains like GT." Although this statement is experimentally validated, many past QD RL works have used Lagrangian-constrained optimization successfully[1,2]
, making this claim counterintuitive to me. Could the authors compare their results with these works or explain in detail the reasons behind their findings? Furthermore, could they mathematically explain the impact of multiplying the reward function by lambda on the optimization objective?

2. In the experimental section, the right side of Figure 3 shows that to improve the diversity score by 0.15, the reward decreases to 5000 points, which is not very high for the walker task. I have not run diversity strategy experiments under this setting and thus lack a reference point. However, I feel the increase in the diversity score is insufficient. Could the authors run n instances of SAC/PPO, each with different seeds, and stop at around 5000 points to report the diversity scores as a reference? Otherwise, the Baseline (DOMiNO) scores in the lower left of Figure 3 make me suspect the authors have not correctly implemented the baseline algorithm. The same issue appears in Figure 4, where DOMiNO is in the lower left, and I would like a reasonable explanation from the authors.

3. There are several minor errors in the paper, which, while partially understandable in context, collectively make the paper difficult to understand:
- $V_{d_{avg}}$ is not defined (line 196).
- $V_{e_{avg}}$ is first mentioned in line 191 but introduced only in line 207.
- The SF estimator on the left side of Equation 7 includes $\gamma$, but the right side does not.
- $y$ in Equation 8 is not a function of $c$, yet $c$ is used on the right side.
- The $z$ in Equation 8’s right side is in the superscript, where $\gamma$ was used earlier.
- The legend in Figures 3 and 5 is too small.
- The difference between the left and right of Figure 5 is not well-explained.


4. **The authors did not submit their code.**

5. Lastly, as a reader, I am more interested in the impact of multi-tasking on diversity strategies. For example, as mentioned in [3], the goal is for strategies to be as similar as possible across different tasks, which is contrary to the diversity score. Have the authors found similar results, and how did they address this? 

[1] Kumar, Saurabh, et al. "One solution is not all you need: Few-shot extrapolation via structured maxent rl." Advances in Neural Information Processing Systems 33 (2020): 8198-8210.

[2] Chen, Wentse, et al. "DGPO: discovering multiple strategies with diversity-guided policy optimization." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 10. 2024.

[3] Teh, Yee, et al. "Distral: Robust multitask reinforcement learning." Advances in neural information processing systems30 (2017).
questions: questions: All my questions are mentioned in the weaknesses section.
limitations: limitations: Thoroughly discussed in Conclusions.
paper_summary: paper_summary: The authors introduce DUPLEX, a method that defines a diversity objective with constraints and uses successor features to robustly estimate policies' expected behavior. DUPLEX allows agents to:
1.	Learn a diverse set of near-optimal policies in complex, highly-dynamic environments.
2.	Exhibit competitive and diverse skills in out-of-distribution (OOD) contexts.
The authors claim that DUPLEX is the first method to achieve diverse solutions in both complex driving simulators and OOD robotic contexts.
strength_weakness: strength_weakness: •	The paper does not compare its diversity objective to other existing diversity objectives in the literature, such as DIAYN (Eysenbach et al., 2019) and MaxEnt RL (Haarnoja et al., 2018). This omission makes it difficult to judge why this particular diversity measure should be considered over the others.

•	The paper builds upon the Domino paper, but it is not clear how it differs from Domino since both papers use extrinsic rewards and another metric computed to promote diversity, i.e., intrinsic reward.

•	Another fundamental question arises: If a model can generate stochastic policies with high rewards, will it be considered diverse compared to the deterministic policy model? Is diversity essentially a measure of generating high-reward stochastic policies?

•	I would urge authors to upload code and some visualizations with comparison of the method to Domino during rebuttal period. 


Overall a good paper and I would be happy to discuss more with authors. I will be willing increase my score if my questions are answered.
questions: questions: See Weaknesses
limitations: limitations: Authors have mentioned the limitations in the paper.
paper_summary: paper_summary: This paper proposes to use successor feature (an embedding of state-action pair) to boost the behavior diversity of a population of policies.

Several tricks are proposed to compute the diversity intrinsic reward. These include using the running average of the extrinsic reward to scale the intrinsic reward and using a dynamic weight $\lambda$ to ensure the task performance is not dropped too much.

Experiments on GranTurismo 7 and Mujoco show that the method can learn diverse yet high-performing policies, which also shows certain OOD capacity.
strength_weakness: strength_weakness: There is a strong assumption of the method: the context vector is given by the environment. The claim on context-conditioned MDP doesn't seem to be interesting to me as considering those context vector as part of the observation space will not change anything. Thereafter the claim in Line 174-175: "We adopt the repulsive reward ... and extend it to be conditioned on the context" does not mean a lot and it suggests the technical novelty of the method against Zahavy is limited.

The method is not scalable as you need to compute Equation 2 and Equation 3 for every state action pair to collect the intrinsic reward. That means you need to iterate over all agents to get their successor feature.

No code is provided.
questions: questions: What does "closest expected features" in Line 178 mean? A feature is a vector, how you get the closest vector of another vector? Are you computing the L1/L2 norm as the distance between features?

Clarity issues:
* Typo in Line 198 "or the update"
* There should be a $i$ notation in $r_d(s, a, c)$ in Eq 2 to indicate that reward is for policy $i$.
* Equation 7 has an undefined $y$.

What does the braces in Eq 5 means?

How $\phi(s, a, c)$ is computed? The output of a neural network? In Appendix Algorithm Line 5 you say "use critic to get" it, what does this mean? Please describe the detailed architecture of the critic network.

It seems the method use QRSAC as the RL algorithm. Will you recompute the intrinsic reward for each sampled batch during training? Or you store the intrinsic reward as it is to the replay buffer?

Missing citations to relevant works:

* Exploration by Random Network Distillation
* Non-local policy optimization via diversity-regularized collaborative exploration
limitations: limitations: Limitations are addressed.
paper_summary: paper_summary: This is a work in the Reinforcement Learning domain, particularly relevant to policy exploration. They propose a method to better preserve the trade-off between the exploration diversity and near-optimality. This strategy utilizes a diversity objective with defined constraints such that it enforces the trained agent learns a diverse set of near optimal policies and could exhibit competitive and diverse skills in out-of-distribution environment. Two experimental domains are used to showcase the superiority of the proposed method.
strength_weakness: strength_weakness: * Relying on multiple hyper-parameters (introduced by equations 3-5, probably hard to scale across the domain).
* Some of the paragraphs has heavy notations and might be more readable if the author could give more intuitions on why it came out like that. For example in section 4.1.1 and section 4.2.
questions: questions: * Regarding the diversity definition in Def 4.1, can you give an example of whether/how different modalities of observed features (e.g., visual or textual) could have a different impact on the diversity formula? or it does not matter (no effect).

* What is the relative additional cost introduced by computing SF distance of each policy pair?
limitations: limitations: I like how the author clarified their limitations in section 6. I am very interested in learning more explanations on question #3 and #4.
paper_summary: paper_summary: This paper proposes an algorithm, DUPLEX, that learns diverse yet high-performance policies for context Markov decision process. DUPLEX extends the previous SOTA DOMiNO by introducing context in the MDP and extending the diversity objective as context conditioned. DUPLEX also introduces three tricks to stabilize the training: a dynamic scaling parameter to balance extrinsic and intrinsic reward, a lower bound to encourage high-performance policies, and a SAC-inspired entropy term to stabilize successor feature estimation. The experiment results show DUPLEX can learn diverse and high-performance policies on both highly complex GranTurismo environment as well as standard MuJoCo domains with OOD physics. In general, although the paper shows strong performance with its proposed method, I have concerns over the clarify of the paper and the evaluation.
strength_weakness: strength_weakness: 1. The paper’s writing can be improved. 

1.1. (Line 42-43) It seems to have a logical gap on why diversity in “multi-context setting” is key to enabling human-level task execution. As a result, I do not see why we should learn diverse behaviors on envs with different contexts (e.g., OOD environments). 

1.2. (Line 132) the paper claims to find a set of diverse policies that maximizes the expected return in every context, but that does not seem to be enforced (not to mention there could be infinite c, which makes empirically showing that the policies are maximizing the return in *every context* impossible). 

1.3. The Section 3 discussion about universal estimators seems disconnected with the intro and other texts around it. I do not understand from the description how universal estimators relate to the goal of finding diverse policies. 

1.4. While I was able to follow Section 4, I find it hard. I suggest authors give a “preview” in the beginning of the section, and/or a diagram showing how different pieces of novel algorithms are connected together to form DUPLEX. It will also help motivate each algorithm component better. 

1.5. The design of Equation 5 seems a mystery. I suggest authors provide stronger justification on the design, as well as exploring alternative, simpler design to achieve similar effect. 

1.6. Similarly, the motivation for using an entropy term to stabilize successor feature estimation seems weak to me. Entropy term encourages the policy/value to respect uncertainties and multi-modalities in policy/value estimation, but SF estimation is unstable because of the Q-learning’s nature of chasing-its-own-tail learning and the difficulty to explore the entire space of hard problems such as GranTurismo. I do not see how they are related. 

2. The features, as introduced in line 141 of the paper as “cumulants”, are essential to the core of the algorithm as involved in successor feature estimation and the diversity definition 4.1. However, the paper does not discuss how the features or cumulants are selected for any of the experiments, which makes me very concerned about reproducibility and how the algorithm works without very sophisticated feature choices. I strongly recommend the authors provide more details regarding it and if possible, ablation studies on different feature choices. Many of the experiment results could be trivial if DUPLEX is optimizing the defined diversity with the features, while baseline algorithms are optimizing different definitions of diversity and are evaluated with DUPLEX diversity. 

Minor comments
1. Line 39 a period is missing. 
2. While I agree successor features are good representation of a policy, I think it is also necessary to discuss the limitation of it – it ignores the order of features, which in some situations are essential. For example, successor features consider the state sequence a,b,c,a,b,c as the same as c,b,a,c,b,a, which is not necessarily always true.
questions: questions: Please see the weaknesses for questions.
limitations: limitations: The authors did not discuss the potential negative societal impact of the work. I hope the authors consider including them in the rebuttal.

Comments: title: Follow up
Follow up
Follow up
comment: Thank you for addressing most of my concerns. I have raised my score to 5. We recommend that the authors work on improving the paper's writing, as highlighted in our initial review, and include an explanation of the motivation behind the design of the heuristic lambda in the next version of the paper to enhance clarity.
Thank you to the authors for answering my question about the entropy term. I think "unless the critic is confident that the actor will visit a state-action pair more frequently than others, the network will be encouraged to estimate that the policy will maximize the entropy" is a great intuition to explain the motivation of introducing entropy in SF. I suggest making that clear in future paper revisions.
Thank you to the reviewer for their response and increasing their score.  Regarding the addition of the entropy term:

Indeed incorporating entropy to the critic estimators is not the same as incentivising the actor to maximize entropy. Our statement citing SAC was meant to credit our source of inspiration when ideating what would be done to master the multi-task challenge (as seen in Figures 4 and 5 and Appendix B, without this only UVFA, that does estimate SFs was able to learn competitive policies in this environment). 

Our intuition is that by maximizing the entropy in the estimation of what the actor will do, i.e. max entropy in SFs estimation, minimizes the critics’ underlying biases. That is, unless the critic is confident that the actor will visit a state-action pair more frequently than others, the network will be encouraged to estimate that the policy will maximize the entropy. 

The question that arises is the generality of the observation that the entropy maximization by default is beneficial. This is an interesting question since the underlying policies are also maximizing entropy, i.e., giving equal probabilities to all the actions when a dominant one is not available. Future research could further investigate this direction evaluating the performance of  USFA with other RL policies whose actor is not maximizing the entropy, e.g., DQNs.
Thank you for your response and updating your score. Regarding your concerns on the technical novelty and significance of our contribution, please note:

- Research significance: 
  - Our research is the first one to learn end-to-end competitive diverse policies in hyper realistic racing simulators and to show visibly diverse driving styles. As reported in the paper, previous state-of-art for diversity in RL failed to provide any visually perceptive degree of diversity while guaranteeing competitive behaviors.

   - Demonstrating near-optimal performance while combining diversity and multitask generalization has direct applications to the GT player community, increasing replayability and engagement for users by racing against diverse and competitive opponents. For instance, we invite the reviewer to check https://sites.google.com/view/duplex-submission-2436/home#h.57pf96h6i4l7 in the first GranTurismo video in the link, and go to the timestamp 1:34 to contrast policies 0 and 1 in their overtake maneuver. It can be seen how the different policies that were so far driving quite similarly take very different strategies to overtake. Having opponents that use such diverse complex tactics can be very rewarding for players and has practical applications for self-play in RL.  Moreover, being robust towards OOD tasks and environments makes these agents more applicable to new game functionalities and updates.

   - We are first to introduce an RL algorithm that provides multiple and diverse solutions in OOD tasks and environment dynamics. As demonstrated in Figure 4, where DUPLEX generalizes better than UVFA and USFA, diversity can be helpful in finding better solutions to OOD tasks and dynamics.

- Technical novelty:
   - Our algorithm builds upon previous state-of-the-art methods from two disconnected bodies of research in RL, several simple yet effective heuristics, and a novel objective function for one of the main components. Simplicity in our method should not hinder the merit of our work, since we prove how our algorithm outperforms related work, and how all of the DUPLEX components play a fundamental role that compounds to achieve the final result (see ablations in Figures 2 and 5). Moreover in Appendix B, we demonstrate how our objective function to estimate successor features is beneficial to previous state-of-the-art frameworks such as USFA.
    - Finally, the multi-task Mujoco proved to be a valid and challenging benchmark for current state-of-the-art methods such as DOMiNO and USFA, that failed already in the training set.
Thank you for your follow up. Please, find our answers below:
 
**How does multiplying the reward function by lambda affect the objective function mathematically?**

Lambda is a non-linear continuous function designed to zero-out the diversity reward component of policies not in the objective region. At the same time lamba is not affecting policies that are successful in solving the target task while searching for diversity.

Specifically, let us consider the case in which we set $\beta$ to 0.8 as the desired value. In this case, we are configuring the training objective to disregard policies that get an average reward less than 0.8 times the reward of the target policy, the target policy only optimizes for the extrinsic rewards. Then, Lambda is a vector that modulates the contribution of the intrinsics reward for each policy in the set. In fact, it downweights the intrinsic rewards for policies that get an average reward less than $\beta$ times the rewards of the target policy and converges to 1 when the policy gets rewards above the same threshold. 
Lambda is computed as a sigmoid and, consequently, does not act as a hard limit. That is, Lambda can be different from 0 or 1 with values slightly lower or higher than  $\beta$ times the target policy. Nevertheless, making this limit - and consequently the rewards - continuous, provided the additional stability that we were missing from Lagrange-constrained optimization for DUPLEX to learn competitive diverse behaviors in the GranTurismo.

**Results for the vanilla SAC baseline (with lower rewards) have not been provided.**

We are working towards adding such a baseline to the paper. Nevertheless, even through we realize it cannot replace the requested baseline, as a temporary answer, we provided additional details on our implementation of DOMiNO, and also highlighted that since DUPLEX is built on the foundations of DOMiNO, all the code from DOMiNO is used in DUPLEX. That includes the underlying SAC, the diversity function, the lagrange-constrained optimization and the neural network. 

The differences in code were the novel mechanisms introduced in Section 4.1.1 and 4.2, including the regularization term in the objective function to estimate the SFs.


**(minor) [3] suggests that different networks corresponding to multiple tasks should be as similar as possible, while the goal of DUPLEX is to make these networks perform as differently as possible. These objectives seem contradictory. How do the authors address this?**

We want to highlight that our objectives are not contradictory, and actually they have some resemblance. In [3] authors state that they have N policies learning to solve N tasks, where the tasks have some elements in common. In [3], authors found that encouraging the N policies to behave similarly across different tasks was beneficial. 

By following the notation used in [3] where the authors label a policy that learns a task as a new policy. In DUPLEX, we have M policies that solve N tasks, and according to [3] we are training M*N policies. In our notation, instead, a policy learning a task is not considered a new policy

This distinction is important because it allows us to see that for every policy of the same N group, DUPLEX does not encourage them to be different between them. DUPLEX encourages diversity to the closest variant of policies from another N group.

From here we can extract a first conclusion that DUPLEX and [3] are not contradictory. Moreover, since in our case N is infinite and is provided as input (ie. the context vector) we are encouraging the M policies by design to be similar -this is a property of Universal Estimator frameworks [4]. A similarity which, to a certain extent, can be further incentivised by the diversity rewards, that discourages any policy from the N group to differ from the others in a direction that is already explored in another group of N policies.

[4] Borsa, Diana, et al. "Universal Successor Features Approximators." International Conference on Learning Representations.
Thank you to the authors for the detailed answers to my concerns and questions. I think the paper's clarity has been improved and my major concerns have been answered/mitigated, especially weakness2 (about the features), as the authors actually did not do any feature engineering for the simpler domains. As such, I increase my score to 5: borderline accept. 

I am still not quite buying the argument for entropy: SAC introduces entropy to maximize, i.e., finding policies that achieve high task reward and high diversity (entropy). However, what does it mean to maximize entropy in SF estimation? The key point is that you are trying to calculate/estimate SF, not to maximize them. It is very interesting to see introducing entropy also helps baselines, and I hope to hear more insights from the authors and also in the paper.
Thanks for the response. I changed my score to 5.

We all argree that the "contextual MDP" and "context-conditioned diversity learning" is not a strong claim and novelty of the paper. My major concern is the technical novelty of the paper as well as the research problem. The paper seems like a combination of existing works, some heurisitics (e.g. soft-lower bound), on a less-interested tasks: the diversity in mujoco, especially in 2024 is it really an important problem? 

The paper itself though is complete and sound.
Thank you for the authors' response. However, the following issues remain unresolved during the rebuttal period:

1. How does multiplying the reward function by lambda affect the objective function mathematically?

2. Results for the vanilla SAC baseline (with lower rewards) have not been provided.

3. (minor) [3] suggests that different networks corresponding to multiple tasks should be as similar as possible, while the goal of DUPLEX is to make these networks perform as differently as possible. These objectives seem contradictory. How do the authors address this?