Learning with Social Influence through  Interior Policy Differentiation

1 INTRODUCTION . The paradigm of Reinforcement Learning ( RL ) , inspired by cognition and animal studies ( Thorndike , 2017 ; Schultz et al. , 1997 ) , can be described as learning by interacting with the environment to maximize a cumulative reward ( Sutton et al. , 1998 ) . From the perspective of ecology , biodiversity as well as the development of various skills are crucial to the continuation and evolution of species ( Darwin , 1859 ; Pianka , 1970 ) . Thus the behavioral diversity becomes a rising topic in RL . Previous works have tried to encourage the emergence of behavioral diversity in RL with two approaches : The first approach is to design interactive environments which contain sufficient richness and diversity . For example , Heess et al . ( 2017 ) show that rich environments enable agents to learn different locomotion skills even using the standard RL algorithms . Yet designing a complex environment requires manual efforts , and the diversity is limited by the obstacle classes . The second approach to increase behavioral diversity is to motivate agents to explore beyond just maximizing the reward for the given task . Zhang et al . ( 2019 ) proposed to maximize a heuristically defined novelty metric between policies through task-novelty joint optimization , but the final performance of agents is not guaranteed . In this work , we address the topic of policy differentiation in RL , i.e. , to improve the diversity of RL agents while keeping their ability to solve the primal task . We draw the inspiration from the Social Influence in animal society ( Rogoff , 1990 ; Ryan & Deci , 2000 ; van Schaik & Burkart , 2011 ; Henrich , 2017 ; Harari , 2014 ) and formulate the concept of social influence in the reinforcement learning paradigm . Our learning scheme is illustrated in Fig 1 . The target agent not only learns to interact with the environment to maximize the reward but also differentiate the actions it takes in order to be different from other existing agents . Since the social influence often acts on people passively as a sort of peer pressure , we implement the social influence in terms of social uniqueness motivation ( Chan et al. , 2012 ) and consider it as a constrained optimization problem . In the following of our work , we first define a rigorous policy distance metric in the policy space to compare the similarity of the agents . Then we develop an optimization constraint using the proposed metric , which brings immediate rather than episodic feedback in the learning process . A novel method , namely Interior Policy Differentiation ( IPD ) , is further 1In this work , we use the term “ peer ” to denote a population of RL agents . 2Code will be made available soon Run proposed as a better solution for the constrained policy optimization problem . We benchmark our method on several locomotion tasks and show it can learn various diverse and well-behaved policies for the given tasks based on the standard Proximal Policy Optimization ( PPO ) algorithm ( Schulman et al. , 2017 ) . 2 RELATED WORK . Intrinsic motivation methods . The Variational Information Maximizing Exploration ( VIME ) method is designed by Houthooft et al . ( 2016 ) to tackle the sparse reward problems . In VIME , an intrinsic reward term based on the maximization of information gains is added to contemporary RL algorithms to encourage exploration . The curiosity-driven methods , proposed by Pathak et al . ( 2017 ) and Burda et al . ( 2018a ) define intrinsic rewards according to prediction errors of neural networks . i.e. , when taking previous unseen states as inputs , networks trained with previous states will tend to predict with low accuracy , so that such prediction errors can be viewed as rewards . Burda et al . ( 2018b ) proposed Random Network Distillation ( RND ) to quantify intrinsic reward by prediction differences between a fixed random initialized network and another randomly initialized network trained with previous state information . Liu et al . ( 2019 ) proposed Competitive Experience Replay ( CER ) , in which they use two actors and a centralized critic , and defined an intrinsic reward by the state coincidence of two actors . The values of intrinsic rewards are fixed to be ±1 for the two actors separately . All of those approaches leverage the weighted sum of the external rewards , i.e. , the primal rewards provided by environments , and intrinsic rewards that provided by different heuristics . A challenging problem is the trade-off between external rewards and intrinsic rewards . The Task-Novelty Bisector ( TNB ) learning method introduced by Zhang et al . ( 2019 ) aims to solve such problem by jointly optimize the extrinsic rewards and intrinsic rewards . Specifically , TNB updates the policy in the direction of the angular bisector of the two gradients , i.e. , gradients of the extrinsic and intrinsic objective functions . However , the foundation of such joint optimization is not solid . Besides , creating an extra intrinsic reward function and evaluating the novelty of states or policies always requires additional neural networks such as auto-encoders . Thus extra computation expenses are needed ( Zhang et al. , 2019 ) . Diverse behaviors from rich environments and algorithms . Heess et al . ( 2017 ) introduce the Distributed Proximal Policy Optimization ( DPPO ) method and enable agents with simulated bodies to learn complex locomotion skills in a diverse set of challenging environments . Although the learning reward they utilize is straightforward , the skills their policy learned are quite impressive and effective in traveling terrains and obstacles . Their work shows that rich environments can encourage the emergence of different locomotion behaviors , but extra manual efforts are required in designing such environments . The research of Such et al . ( 2018 ) shows that different RL algorithms may converge to different policies for the same task . The authors find that algorithms based on policy gradient tend to converge to the same local optimum in the game of Pitfall , while off-policy and value-based algorithms are prone to learn sophisticated strategies . On the contrary , in this paper , we are more interested in how to learn different policies through a single learning algorithm and learn the capability of avoiding local optimum . Population-based novelty-seeking methods . Pugh et al . ( 2016 ) establish a standard framework for understanding and comparing different approaches to searching for quality diversity ( QD ) . Conti et al . ( 2018 ) investigate adding novelty search ( NS ) and QD to evolution strategies ( ES ) to avoid local optima as well as achieve higher performance . Lehman & Stanley ( 2011 ; 2008 ) conclude that deriving an open-ended search algorithm that operates without pressure towards the ultimate objective is possible , suggesting ignoring the objective may often benefit the search itself . The work of Wang et al . ( 2019 ) yields a new kind of open-ended algorithm which indicates the solution to one environment might be a stepping stone to a new level of performance in another . Such et al . ( 2017 ) evolve a DNN with a population-based genetic algorithm ( GA ) for challenging RL tasks . By improving the vanilla TRPO algorithm ( Schulman et al. , 2015 ) , Kurutach et al . ( 2018 ) maintains model uncertainty given the data collected from the environment via an ensemble of deep neural networks . 3 QUANTIFYING THE DISTANCE BETWEEN POLICIES . To encourage the emergence of behavioral diversity in RL , we first define a metric to measure the difference between policies , which is the foundation for the later algorithm we propose . We denote the learned policies as { πθi ; θi ∈ Θ , i = 1 , 2 , ... } , wherein θi represents parameters of the i-th policy , Θ denotes the whole parameter space . In the following , we omit π and denote a policy πθi as θi for simplicity unless stated otherwise . 3.1 DEFINITION . Mathematically , a metric should satisfy three important properties , namely the identity , the symmetry as well as the triangle inequality . Definition 1 A metric space is an ordered pair ( M , d ) where M is a set and d is a metric on M , i.e. , a function d : M ×M → R such that for any x , y , z ∈M , the following holds : 1. d ( x , y ) ≥ 0 , d ( x , y ) = 0⇔ x = y , 2. d ( x , y ) = d ( y , x ) , 3. d ( x , z ) ≤ d ( x , y ) + d ( y , z ) . We use the Total Variance Divergence DTV ( Schulman et al. , 2015 ) to measure the distance between policies . Concretely , for discrete probability distributions p and q , this distance is defined as DTV ( p , q ) = ∑ i |pi − qi| . 34 Theorem 1 ( Metric Space ( Θ , DρTV ) ) The expectation of DTV ( · , · ) of two policies over any state distribution ρ ( s ) : D ρ TV ( θi , θj ) : = Es∼ρ ( s ) [ DTV ( θi ( s ) , θj ( s ) ) ] , ( 1 ) is a metric on Θ , thus ( Θ , D ρ TV ) is a metric space . The proof of Theorem 1 is in Appendix A . It is worth mentioning that , although TVD is used in our work , we can easily extend the result to use other distance between distributions as substitutes of TVD ( e.g . Jensen Shannon divergence DJS or Wasserstein metric DW ) ( Endres & Schindelin , 2003 ; Fuglede & Topsoe , 2004 ; Villani , 2008 ) , and similar results can be get Corollary 1 Let DρJS : = Es∼ρ ( s ) [ DJS ( θi ( s ) , θj ( s ) ) ] and D ρ W : = Es∼ρ ( s ) [ DW ( θi ( s ) , θj ( s ) ) ] , ( Θ , D ρ JS ) and ( Θ , D ρ W ) are also metric spaces . On top of the metric space ( Θ , D ρ TV ) , we could then compute the uniqueness of a policy . Definition 2 ( Uniqueness of Policy ) Given a reference policy set Θref such that Θref = { θrefi , i = 1 , 2 , ... } , Θref ⊂ Θ , the uniqueness U ( θ|Θref ) of policy θ is the minimal difference between θ and all policy in the reference policy set , i.e. , U ( θ|Θref ) : = min θj∈Θref D ρ TV ( θ , θj ) . ( 2 ) 3It can be extended to continuous state and action spaces by replacing the sums with integrals . 4The factor 1 2 in Schulman et al . ( 2015 ) is omitted in our work for conciseness . Consequently , to motivate RL with the social uniqueness , we hope our method can maximize the uniqueness of a new policy , i.e. , maxθ U ( θ|Θref ) , where the Θref includes all the existing policies . 3.2 ESTIMATION OF D ρ TV ( θi , θj ) In practice , the calculation of D ρ TV ( θi , θj ) is based on Monte Carlo estimation . i.e. , we need to sample s from ρ ( s ) . Although in finite state space we can get precise estimation after establishing ergodicity , problem arises when we are facing continuous state cases . i.e . it is difficult to efficiently get enough samples . Formally , we denote the domain of ρ ( s ) as S and denote the domain of ρθ ( s ) as Sθ ⊂ S , where ρθ ( s ) : = ρ ( s|s ∼ θ ) and in finite time horizon problems ρ ( s|s ∼ θ ) = P ( s0 = s|θ ) + P ( s1 = s|θ ) + ... +P ( sT = s|θ ) . As we only care about the reachable regions , the domain S can be divided by S = limN→∞ ⋃N i=1 Sθi . In order to improve the sample efficiency , we propose to approximate D ρ TV ( θi , θj ) with D ρθ TV ( θi , θj ) , where θ is a certain fixed behavior policy that irrelevant to θi , θj . Such approximation requires a necessary condition : Condition 1 The domain of possible states are similar between different policies : ∑ s∈S P ( s ∈ ( Sθ ∪ Sθj ) \ ( Sθ ∩ Sθj ) ) ∑ s∈S P ( s ∈ ( Sθ ∩ Sθj ) ) , ∀j . ( 3 ) When such condition holds , we can use ρ ( s|s ∼ θ ) as our choice of ρ ( s ) , and the properties in Definition 1 still holds . In practice , the Condition 1 always holds as we can ensure this by adding sufficiently large noise on θ , while the permitted state space is always limited . And for more general cases , to satisfy the properties in Definition 1 , we must sample s from Sθ ∪ Sθj , accordingly , D ρ TV ( θ , θj ) = Es∼ ( Sθ∪Sj ) [ DTV ( θ ( s ) , θj ( s ) ) ] = Es∼ ( Sθ∩Sθj ) [ DTV ( θ ( s ) , θj ( s ) ) ] + Es∼ ( Sθ∪Sθj ) \Sθj [ DTV ( θ ( s ) , N ) ] + Es∼ ( Sθ∪Sθj ) \Sθ [ DTV ( N , θj ( s ) ) ] ( 4 ) where N represents random action when a policy have never been trained or visited such state domain . Plugging Eq . ( 4 ) into Eq . ( 2 ) , the objective function of policy differentiation is max θ min θj∈Θref D ρ TV ( θ , θj ) = Es∼ ( Sθ∩Sθj ) [ DTV ( θ ( s ) , θj ( s ) ) ] + Es∼ ( Sθ∪Sθj ) \Sθj [ DTV ( θ ( s ) , N ) ] + Es∼ ( Sθ∪Sθj ) \Sθ [ DTV ( N , θj ( s ) ) ] ( 5 ) While the first two terms are related to the policy θ , the last term is only related to the domain Sθ . If we enable sufficient exploration in training as well as in the initialization of θ , the last term will disappear ( i.e . Sθj ⊂ Sθ ) . Hence we can also use D ρθi TV ( θi , θj ) as an approximation of D ρ TV ( θi , θj ) in training of θi as long as sufficient exploration is guaranteed . Proposition 1 ( Unbiased Single Trajectory Estimation ) The estimation of ρθ ( s ) using a single trajectory τ is unbiased . The proof of Proposition 1 is in Appendix B . Given the definition of uniqueness and a practically unbiased sampling method , the next step is to develop an efficient learning algorithm .

Summary: The paper presents a new algorithm for maximizing the diversity of different policies learned for a given task. The diversity is quantified using a metric, where in this case the total variation is used. A policy is different from a set of other policy if its minimum distance to all the other policies is high. The authors formulate a new constraint optimization problem where the diversity to previous policies is lower bounded in order to avoid a tedious search for combining task reward and diversity reward. The algorithm is evaluated on different Mojoco locomotion tasks.