Evolutionary Diversity Optimization with Clustering-based Selection for Reinforcement Learning

1 INTRODUCTION . Reinforcement Learning ( RL ) is an effective method for training agents to make decisions in a given environment , which is often to obtain a policy maximizing the expected cumulative rewards ( Li , 2017 ; Sutton & Barto , 2018 ) . As RL is applied to more real-world scenarios , training a single policy is insufficient to handle complex problems , and we may need to find a set of policies with both high quality ( i.e. , rewards ) and diverse behaviors . For example , when navigating in complex environments ( Zhu et al. , 2017 ; Mirowski et al. , 2017 ) , there are usually deceptive rewards trapping a single policy into the local optima . Maintaining a set of policies allows data to be collected with diverse behaviors and may lead to better exploration in these environments . When controlling robots , it is fragile to have one single policy ( Cully et al. , 2015 ) . In contrast , maintaining a set of policies can improve robustness , e.g. , enable real robots to recover quickly from joint damage ( Cully et al. , 2015 ) , and also be helpful for few-shot adaption when facing unseen tasks ( Kumar et al. , 2020 ) . General deep RL algorithms mostly focus on achieving high rewards , restricting their ability to generate diverse behaviors . Evolutionary Algorithms ( EAs ) are general-purpose heuristic optimization algorithms that maintain a population ( i.e. , a set of solutions ) , and simulate the natural evolution process with iterative reproduction and selection ( Bäck , 1996 ) . Quality-Diversity ( QD ) algorithms ( Pugh et al. , 2016 ; Cully & Demiris , 2018 ) are a specific type of EAs that aim to return a set of high-quality solutions with diverse behaviors in a single run . They have been naturally applied to RL in complex environments , generating the corresponding RL algorithms NSR-ES ( Conti et al. , 2018 ) , ME-ES ( Colas et al. , 2020 ) , DvD-ES ( Parker-Holder et al. , 2020 ) and QD-RL ( Cideron et al. , 2020 ) . Note that classical QD algorithms like NSLC ( Lehman & Stanley , 2011a ) or MAP-Elites ( Mouret & Clune , 2015 ; Cully et al. , 2015 ) often employ a Genetic Algorithm ( GA ) as the underlying optimizer . However , as RL tasks are often high-dimensional , these methods have replaced GA with Evolution Strategies ( ES ) presented by Salimans et al . ( 2017 ) , which has achieved performances comparable to state-of-the-art deep RL algorithms on high-dimensional control tasks . NSR-ES ( Conti et al. , 2018 ) maintains a population of ES agents and introduces Novelty Search ( NS ) ( Lehman & Stanley , 2011b ) to improve the performance on RL tasks with sparse or deceptive rewards . That is , it adds a novelty term ( representing behavior diversity ) to the objective function of ES agents . In each iteration , NSR-ES selects a policy from the population for reproduction , where the probability of selecting a particular policy is proportional to its novelty score . In contrast to NSR-ES ( Conti et al. , 2018 ) , DvD-ES ( Parker-Holder et al. , 2020 ) optimizes all policies of a population simultaneously , by maximizing the sum of their rewards and a population-wide diversity criterion ( i.e. , the volume between the behaviors of the policies ) . Though DvD-ES can maintain a diverse population , it ignores the previously generated policies , leading to “ cycling ” ( i.e. , a phenomenon where the population keeps moving alternatively between two areas ) and thus inefficient performance . It has been shown ( Cully & Demiris , 2018 ) that keeping an additional archive ( i.e. , a subset of policies generated-so-far ) and selecting policies from the archive instead of the population for reproduction can be beneficial . There are different ways to select policies from the archive . ME-ES ( Colas et al. , 2020 ) uses an alternating selection mechanism , which selects several policies with the best quality in one iteration and with the best diversity in the next iteration . QD-RL ( Cideron et al. , 2020 ) uses Pareto-based selection ( Deb et al. , 2002 ) by considering both quality and diversity in one iteration , and achieves better performance . That is , the Pareto front is first calculated with respect to quality and diversity , and then the policies are selected from the Pareto front based on the crowding distance . Although Pareto-based selection can guarantee the selected policies uniformly distributed over the Pareto front , these policies may be still similar in the behavior space , limiting the performance of QD-RL . For example , Figure 1 ( a ) shows the heat map of a synthetic function f ( x , y ) in the behavior space , where the color corresponds to the function value , which we call quality here . The goal is to find the four optimal solutions distributed in the center of four regions of the behavior space . The points represent the solutions in the current archive , from which we need to select some solutions for reproduction . QD-RL first calculates the diversity of each solution ( i.e. , the average Euclidean distance between this solution and its k-nearest neighbors ) in the archive , and then gets the Pareto front with respect to quality and diversity , as shown in Figure 1 ( b ) . By Pareto-based selection , the blue points together with the red star will be selected , which are uniformly distributed over the Pareto front , but concentrated in the lower right corner of the behavior space . In this paper , we propose an Evolutionary Diversity Optimization algorithm with Clustering-based Selection ( EDO-CS ) . In each iteration , the policies in the archive are divided into several clusters based on their behaviors , and a high-quality policy is selected from each cluster for reproduction . This selection mechanism can naturally guarantee the diversity of the selected policies in the behavior space . For example , the red points and red star in Figure 1 ( a ) will be selected by clustering-based selection , which are distributed in the four different areas of the behavioral space . To examine the performance of EDO-CS , we conduct experiments on a variety of continuous control tasks from OpenAI Gym library ( Brockman et al. , 2016 ) . Firstly , we show that EDO-CS can solve navigating problems in environments with deceptive rewards . Then , we test it in multi-modal tasks , showing that EDOCS can efficiently find optimal policies with diverse behaviors . In the standard MuJoCo environments that do not require much diversity , we show that EDO-CS can achieve even better performance than Vanilla ES that focuses on the quality . We also test the influence of the hyper-parameters of EDO-CS in these environments . The remainder of this paper is organized as follows . We first introduce some definitions and notations used in our paper in Section 2 . After that , we propose our method in Section 3 . The results and analyses of experiments are shown in Section 4 . Finally , we conclude this work in Section 5 . 2 BACKGROUND . We consider the setting of a fully observable Markov Decision Process ( MDP ) described as a tuple ( S , A , P , R , γ ) , where S is the state space , A is the action space , P ( st+1|st , at ) gives the probability of transiting to state st+1 after taking action at at state st , rt = R ( st , at , st+1 ) is the reward obtained by transiting to state st+1 after taking action at at state st , and γ is the discount factor . A policy πθ : S → A is a mapping from state space S to action space A , paremeterized by θ , which can be either deterministic or randomized . The goal of general RL algorithms is to optimize the parameters θ such that πθ maximizes the expectation of the cumulative rewards R ( τ ) = ∑T t=0 γ trt , where τ is the trajectory over a horizon T . The objective function can be written as follows : J ( θ ) = Eτ∼πθ [ R ( τ ) ] . ( 1 ) To make better exploration and have more robustness , QD algorithms seek to find a set of policies with both high rewards and diverse behaviors ( Pugh et al. , 2016 ; Cully & Demiris , 2018 ) . The behavior characterization b ( πθ ) of a policy πθ is usually domain-dependent . For example , it can be the coordinates of the final location ( x , y ) of the robot in the robot locomotion problem , and the corresponding behavior space is R2 . It can also be defined as the actions taken by policies , leading to a high-dimensional behavior space . Given an archive A of policies , the diversity of a particular policy πθ is computed as the average Euclidean distance between b ( πθ ) and A′πθ , k , i.e. , the set of k-nearest neighbors of πθ in the archive A : Div ( θ ) = 1 k ∑ π′θ∈A ′ πθ , k ‖b ( πθ ) − b ( π′θ ) ‖2 . ( 2 ) Then , the objective function to be maximized can be expressed as : J ( θ ) = ( 1− λ ) · Eτ∼πθ [ R ( τ ) ] + λ ·Div ( θ ) , ( 3 ) where λ controls the trade-off between the quality ( i.e. , rewards ) and diversity . 3 THE EDO-CS ALGORITHM . In this section , we introduce the proposed algorithm , Evolutionary Diversity Optimization with Clusteringbased Selection ( EDO-CS ) . We first give an overview of EDO-CS in Section 3.1 . Then , we introduce the selection and reproduction mechanisms in Sections 3.2 and 3.3 , respectively . Section 3.4 shows how to balance the quality and diversity . Finally , we compare EDO-CS with other QD algorithms in Section 3.5 . 3.1 OVERVIEW . A general flow of EDO-CS is shown in Algorithm 1 . It first initializes an archiveAwith a maximum capacity of l , which contains the policies generated-so-far together with their corresponding cumulative rewards and behaviors ; and a bandit B , which is used to sample the weight of diversity ( i.e. , λ in Eq . ( 3 ) ) during the reproduction process . Then , it adds l randomly generated policies into the archive A ( lines 1–5 ) , and tries to improve the policies in A iteratively ( lines 7–20 ) . In each iteration , EDO-CS employs K-means ( Lloyd , 1982 ) to partition the policies in the archive into K clusters based on their behaviors ( line 8 ) , and selects one specific policy from each cluster ( line 9 ) . After getting K policies , EDO-CS samples the value of λ in the objective function for each policy from the bandit B , and uses the ES optimizer to update the selected policies in parallel for T ′ iterations ( lines 11–15 ) . The archive and bandit are updated in line 18 after evaluating all the generated offspring policies . To maintain the diversity of the archive , we avoid adding policies with similar behaviors ( Cully & Mouret , 2013 ) . To be specific , whenever we want to add a new policy into the archive , we first select the policy with the most similar behavior from the archive . If their behavior distance is greater than a threshold , we add the new policy into the archive . Otherwise , we compare their rewards and keep the policy with the higher reward . Note that if the archive size exceeds the capacity l after adding a new policy , the oldest policy in the archive will be deleted . EDO-CS repeats the above process until the number of iterations reaches T .

Summary: The authors present EDO-CS an approach to multi-objective optimisation that ensures the high quality and diversity of generated RL policies. Instead of uniformly sampling solutions from the Pareto front, sampling policies from learned clusters is introduced, as a selection mechanism within the ES optimisation. The quality and diversity are further achieved by modifying the objective function of the ES algorithm so it includes both the fitness and behaviour-diversity terms, balanced with a hyperparameter $\lambda$. This hyperparameter is set using a multi-arm bandit approach, which is adapted during the training.  The performance of the proposed approach is evaluated on several MuJoCo continuous control tasks, as well as compared to state-of-the-art QD benchmarks.