Evolutionary Diversity Optimization with Clustering-based Selection for Reinforcement Learning

1 INTRODUCTION . Reinforcement Learning ( RL ) is an effective method for training agents to make decisions in a given environment , which is often to obtain a policy maximizing the expected cumulative rewards ( Li , 2017 ; Sutton & Barto , 2018 ) . As RL is applied to more real-world scenarios , training a single policy is insufficient to handle complex problems , and we may need to find a set of policies with both high quality ( i.e. , rewards ) and diverse behaviors . For example , when navigating in complex environments ( Zhu et al. , 2017 ; Mirowski et al. , 2017 ) , there are usually deceptive rewards trapping a single policy into the local optima . Maintaining a set of policies allows data to be collected with diverse behaviors and may lead to better exploration in these environments . When controlling robots , it is fragile to have one single policy ( Cully et al. , 2015 ) . In contrast , maintaining a set of policies can improve robustness , e.g. , enable real robots to recover quickly from joint damage ( Cully et al. , 2015 ) , and also be helpful for few-shot adaption when facing unseen tasks ( Kumar et al. , 2020 ) . General deep RL algorithms mostly focus on achieving high rewards , restricting their ability to generate diverse behaviors . Evolutionary Algorithms ( EAs ) are general-purpose heuristic optimization algorithms that maintain a population ( i.e. , a set of solutions ) , and simulate the natural evolution process with iterative reproduction and selection ( Bäck , 1996 ) . Quality-Diversity ( QD ) algorithms ( Pugh et al. , 2016 ; Cully & Demiris , 2018 ) are a specific type of EAs that aim to return a set of high-quality solutions with diverse behaviors in a single run . They have been naturally applied to RL in complex environments , generating the corresponding RL algorithms NSR-ES ( Conti et al. , 2018 ) , ME-ES ( Colas et al. , 2020 ) , DvD-ES ( Parker-Holder et al. , 2020 ) and QD-RL ( Cideron et al. , 2020 ) . Note that classical QD algorithms like NSLC ( Lehman & Stanley , 2011a ) or MAP-Elites ( Mouret & Clune , 2015 ; Cully et al. , 2015 ) often employ a Genetic Algorithm ( GA ) as the underlying optimizer . However , as RL tasks are often high-dimensional , these methods have replaced GA with Evolution Strategies ( ES ) presented by Salimans et al . ( 2017 ) , which has achieved performances comparable to state-of-the-art deep RL algorithms on high-dimensional control tasks . NSR-ES ( Conti et al. , 2018 ) maintains a population of ES agents and introduces Novelty Search ( NS ) ( Lehman & Stanley , 2011b ) to improve the performance on RL tasks with sparse or deceptive rewards . That is , it adds a novelty term ( representing behavior diversity ) to the objective function of ES agents . In each iteration , NSR-ES selects a policy from the population for reproduction , where the probability of selecting a particular policy is proportional to its novelty score . In contrast to NSR-ES ( Conti et al. , 2018 ) , DvD-ES ( Parker-Holder et al. , 2020 ) optimizes all policies of a population simultaneously , by maximizing the sum of their rewards and a population-wide diversity criterion ( i.e. , the volume between the behaviors of the policies ) . Though DvD-ES can maintain a diverse population , it ignores the previously generated policies , leading to “ cycling ” ( i.e. , a phenomenon where the population keeps moving alternatively between two areas ) and thus inefficient performance . It has been shown ( Cully & Demiris , 2018 ) that keeping an additional archive ( i.e. , a subset of policies generated-so-far ) and selecting policies from the archive instead of the population for reproduction can be beneficial . There are different ways to select policies from the archive . ME-ES ( Colas et al. , 2020 ) uses an alternating selection mechanism , which selects several policies with the best quality in one iteration and with the best diversity in the next iteration . QD-RL ( Cideron et al. , 2020 ) uses Pareto-based selection ( Deb et al. , 2002 ) by considering both quality and diversity in one iteration , and achieves better performance . That is , the Pareto front is first calculated with respect to quality and diversity , and then the policies are selected from the Pareto front based on the crowding distance . Although Pareto-based selection can guarantee the selected policies uniformly distributed over the Pareto front , these policies may be still similar in the behavior space , limiting the performance of QD-RL . For example , Figure 1 ( a ) shows the heat map of a synthetic function f ( x , y ) in the behavior space , where the color corresponds to the function value , which we call quality here . The goal is to find the four optimal solutions distributed in the center of four regions of the behavior space . The points represent the solutions in the current archive , from which we need to select some solutions for reproduction . QD-RL first calculates the diversity of each solution ( i.e. , the average Euclidean distance between this solution and its k-nearest neighbors ) in the archive , and then gets the Pareto front with respect to quality and diversity , as shown in Figure 1 ( b ) . By Pareto-based selection , the blue points together with the red star will be selected , which are uniformly distributed over the Pareto front , but concentrated in the lower right corner of the behavior space . In this paper , we propose an Evolutionary Diversity Optimization algorithm with Clustering-based Selection ( EDO-CS ) . In each iteration , the policies in the archive are divided into several clusters based on their behaviors , and a high-quality policy is selected from each cluster for reproduction . This selection mechanism can naturally guarantee the diversity of the selected policies in the behavior space . For example , the red points and red star in Figure 1 ( a ) will be selected by clustering-based selection , which are distributed in the four different areas of the behavioral space . To examine the performance of EDO-CS , we conduct experiments on a variety of continuous control tasks from OpenAI Gym library ( Brockman et al. , 2016 ) . Firstly , we show that EDO-CS can solve navigating problems in environments with deceptive rewards . Then , we test it in multi-modal tasks , showing that EDOCS can efficiently find optimal policies with diverse behaviors . In the standard MuJoCo environments that do not require much diversity , we show that EDO-CS can achieve even better performance than Vanilla ES that focuses on the quality . We also test the influence of the hyper-parameters of EDO-CS in these environments . The remainder of this paper is organized as follows . We first introduce some definitions and notations used in our paper in Section 2 . After that , we propose our method in Section 3 . The results and analyses of experiments are shown in Section 4 . Finally , we conclude this work in Section 5 . 2 BACKGROUND . We consider the setting of a fully observable Markov Decision Process ( MDP ) described as a tuple ( S , A , P , R , γ ) , where S is the state space , A is the action space , P ( st+1|st , at ) gives the probability of transiting to state st+1 after taking action at at state st , rt = R ( st , at , st+1 ) is the reward obtained by transiting to state st+1 after taking action at at state st , and γ is the discount factor . A policy πθ : S → A is a mapping from state space S to action space A , paremeterized by θ , which can be either deterministic or randomized . The goal of general RL algorithms is to optimize the parameters θ such that πθ maximizes the expectation of the cumulative rewards R ( τ ) = ∑T t=0 γ trt , where τ is the trajectory over a horizon T . The objective function can be written as follows : J ( θ ) = Eτ∼πθ [ R ( τ ) ] . ( 1 ) To make better exploration and have more robustness , QD algorithms seek to find a set of policies with both high rewards and diverse behaviors ( Pugh et al. , 2016 ; Cully & Demiris , 2018 ) . The behavior characterization b ( πθ ) of a policy πθ is usually domain-dependent . For example , it can be the coordinates of the final location ( x , y ) of the robot in the robot locomotion problem , and the corresponding behavior space is R2 . It can also be defined as the actions taken by policies , leading to a high-dimensional behavior space . Given an archive A of policies , the diversity of a particular policy πθ is computed as the average Euclidean distance between b ( πθ ) and A′πθ , k , i.e. , the set of k-nearest neighbors of πθ in the archive A : Div ( θ ) = 1 k ∑ π′θ∈A ′ πθ , k ‖b ( πθ ) − b ( π′θ ) ‖2 . ( 2 ) Then , the objective function to be maximized can be expressed as : J ( θ ) = ( 1− λ ) · Eτ∼πθ [ R ( τ ) ] + λ ·Div ( θ ) , ( 3 ) where λ controls the trade-off between the quality ( i.e. , rewards ) and diversity . 3 THE EDO-CS ALGORITHM . In this section , we introduce the proposed algorithm , Evolutionary Diversity Optimization with Clusteringbased Selection ( EDO-CS ) . We first give an overview of EDO-CS in Section 3.1 . Then , we introduce the selection and reproduction mechanisms in Sections 3.2 and 3.3 , respectively . Section 3.4 shows how to balance the quality and diversity . Finally , we compare EDO-CS with other QD algorithms in Section 3.5 . 3.1 OVERVIEW . A general flow of EDO-CS is shown in Algorithm 1 . It first initializes an archiveAwith a maximum capacity of l , which contains the policies generated-so-far together with their corresponding cumulative rewards and behaviors ; and a bandit B , which is used to sample the weight of diversity ( i.e. , λ in Eq . ( 3 ) ) during the reproduction process . Then , it adds l randomly generated policies into the archive A ( lines 1–5 ) , and tries to improve the policies in A iteratively ( lines 7–20 ) . In each iteration , EDO-CS employs K-means ( Lloyd , 1982 ) to partition the policies in the archive into K clusters based on their behaviors ( line 8 ) , and selects one specific policy from each cluster ( line 9 ) . After getting K policies , EDO-CS samples the value of λ in the objective function for each policy from the bandit B , and uses the ES optimizer to update the selected policies in parallel for T ′ iterations ( lines 11–15 ) . The archive and bandit are updated in line 18 after evaluating all the generated offspring policies . To maintain the diversity of the archive , we avoid adding policies with similar behaviors ( Cully & Mouret , 2013 ) . To be specific , whenever we want to add a new policy into the archive , we first select the policy with the most similar behavior from the archive . If their behavior distance is greater than a threshold , we add the new policy into the archive . Otherwise , we compare their rewards and keep the policy with the higher reward . Note that if the archive size exceeds the capacity l after adding a new policy , the oldest policy in the archive will be deleted . EDO-CS repeats the above process until the number of iterations reaches T .

Summary: The authors present a new method for finding robust policies in reinforcement learning, those policies which give high rewards but also come from a diversity of behaviours. The authors accomplish this by using a clustering algorithm to first divide policies into relevant clusters and then use evolutionary algorithms to optimise policy choice. The result is an algorithm that performs well over several representative benchmarks, achieving SOTA performance with high convergence rates.