Novel Policy Seeking with Constrained Optimization

1 INTRODUCTION . In the sense of learning through interactions with the environment , the scheme of reinforcement learning ( RL ) is conceptually similar to the emergence of intelligence ( Sutton et al. , 1998 ) : an agent explores and exploits information of a given environment , learns to master some certain skills through trials and errors to gain as much reward as possible . When solving a problem , we humans could be creative to come up with multiple different solutions and gain insights from searching for diverse solutions . e.g. , a self-containing example is the various approaches in RL research . While the state-of-the-art algorithms have achieved superhuman performance in a variety of challenging tasks ( Vinyals et al. , 2019 ; Akkaya et al. , 2019 ; Berner et al. , 2019 ; Badia et al. , 2020 ; Elbarbari et al. , 2021 ) , the task of encouraging individualized diversity 2 of learned agents , on the other hand , is relatively under-explored . Different from conventional RL agents that are only learned through interactions with the external environment , novel policy generation is a task considering the differentiation among individual policies . The differentiation among policies can be explained as the social influence ( Rogoff , 1990 ; Ryan & Deci , 2000 ; van Schaik & Burkart , 2011 ; Henrich , 2017 ; Harari , 2014 ) in social science literature . Although many works have been proposed applying social motivation to Multi-Agent Reinforcement Learning ( MARL ) settings ( Jaques et al. , 2019 ; Hughes et al. , 2018 ; Sequeira et al. , 2011 ; Peysakhovich & Lerer , 2017 ) , how to motivate a single RL agent to perform differently against existing agents is still an open question . In previous attempts for novel policy generation , there are three main challenges : ( 1 ) heuristically defined metric for novelty estimation is computational expensive ( Zhang et al. , 2019 ) , ( 2 ) defining novelty reward for an entire episode yields additional challenge in credit assignment , and ( 3 ) solving 1Code is included in the supplemental material . 2Note that this is different from diversity-driven exploration . While the latter focuses on the inner-diversity of a policy , in our work we focus on the inter-diversity between policies the problem under the formulation of multi-objective optimization leads to the performance decay in the original task . Fig . 1 compares the policy gradients of three cases , namely the one without novel policy seeking , novelty-seeking with multi-objective optimization , and novelty-seeking with constrained optimization methods , respectively . In this work we take into consideration not only the novelty of a set of learned policies but also the performance of those novel policies in the primal task , when addressing the problem of novel-policygeneration . Our contributions can be summarized as follows : 1 . We introduce a lightweight metric to compute the difference between policies with instant feedback at every timestep , to address the first two drawbacks of previous novel policy seeking methods discussed above ; 2 . We propose a constrained optimization formulation for novel policy generation and design a practical algorithm of IPD , resembling the interior point method in constrained optimization literature ; 3 . The proposed method , IPD , is evaluated on several continuous control benchmarks , showing the strengths of our constrained optimization solution for novelty-seeking can generate a series of diverse and well-performing policies , compared to previous multi-objective novel policy generation methods . 2 RELATED WORK . Intrinsic motivation methods . In previous work , different approaches are proposed to provide intrinsic motivation or intrinsic reward as a supplementary to the primal task reward for better exploration ( Houthooft et al. , 2016 ; Pathak et al. , 2017 ; Burda et al. , 2018a ; b ; Liu et al. , 2019 ) . All those approaches use the weighted sum of two rewards , the primal rewards provided by environments and the intrinsic rewards provided by different heuristics . On the other hand , the work of DIAYN and DADS ( Eysenbach et al. , 2018 ; Sharma et al. , 2019 ) learn diverse skills without extrinsic reward . Those approaches focus on decomposing diverse skills of a single policy , while our work focuses on learning diverse behaviors among a batch of policies for the same task . Diverse policy generation methods . The work of Such et al . shows that different RL algorithms may converge to different policies for the same task ( Such et al. , 2018 ) . On the contrary , we are interested in learning different policies through a single algorithm with the capability of avoiding local optimum . The work of Pugh et al . establishes a standard framework for understanding and comparing different approaches to search for quality diversity ( QD ) ( Pugh et al. , 2016 ) . Conti et al . proposes a solution which avoids local optima as well as achieves higher performance by adding novelty search and QD to evolution strategies ( Conti et al. , 2018 ) . The Task-Novelty Bisector ( TNB ) ( Zhang et al. , 2019 ) aims to solve novel policy generation problem by jointly optimize the extrinsic rewards and novelty rewards defined by an auto-encoder . In this work , we first adopt TNB in the constrained optimization framework , resulting in Contrained TNB , to demonstrate the dilemma between the task performance and novelty pursuance . Constrained Markov Decision Process . The Constrained Markov Decision Process ( CMDP ) ( Altman , 1999 ) considers the situation where an agent interacts with the environment under certain constraints . Formally , the CMDP can be defined as a tuple ( S , A , γ , r , c , C , P , s0 ) , where S andA are the state and action space ; γ ∈ [ 0 , 1 ) is a discount factor ; r : S×A×S → R and c : S×A×S → R denote the reward function and cost function ; C ∈ R+ is the upper bound of permitted expected cumulative cost ; P ( ·|s , a ) : S ×A → S denotes the transition dynamics , and s0 is the initial state . Denote the Markovian policy class as Π , where Π = { π : S × A → [ 0 , 1 ] , ∑ a π ( a|π ) = 1 } The learning objective of a policy for CMDP is to find a π∗ ∈ Π , such that π∗ = max π∈Π Eτ∼π , s′∼P [ ∞∑ t=0 γtr ( s , a , s′ ) ] , s.t . Eτ∼π , s′∼P [ ∞∑ t=0 γtc ( s , a , s′ ) ] ≤ C , ( 1 ) where τ indicates a trajectory ( s0 , a0 , s1 , ... ) and τ ∼ π represents the distribution over trajectories following policy π : at ∼ π ( ·|st ) , st+1 ∼ P ( ·|st , at ) ; t = 0 , 1 , 2 , .... Previous literature provide several approaches to solve CMDP ( Achiam et al. , 2017 ; Chow et al. , 2018 ; Ray et al. , 2019 ; Sun et al. , 2021 ) . In Sec.2.1 , we define a metric space that measures the difference between policies , which is the fundamental element for the proposed methods . In Sec.2.2 , we develop a practical estimation method for this metric . Sec.2.3 describes the formulation of constrained optimization on novel policy generation . The implementations of two practical algorithms are further introduced in Sec.2.4 . We denote the policies as { πθi ; θi ∈ Θ , i = 1 , 2 , ... } , wherein θi represents parameters of the i-th policy , Θ denotes the whole parameter space . In this work , we focus on improving the behavioral diversity of policies from PPO ( Schulman et al. , 2017 ) , thus we use Θ to represent ΘPPO in this paper . It is worth noting that the proposed methods can be easily extended to other RL algorithms ( Schulman et al. , 2015 ; Lillicrap et al. , 2015 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) . To simplify the notation , we omit π and denote a policy πθi as θi unless stated otherwise . 2.1 MEASURING THE DIFFERENCE BETWEEN POLICIES . We use the Wasserstein metric Wp ( Rüschendorf , 1985 ; Villani , 2008 ; Arjovsky et al. , 2017 ) to measure the distance between policies . Concretely , we consider the Gaussian-parameterized policies , where the Wp over two policies can be written in the closed form W 22 ( N ( m1 , Σ1 ) , N ( m2 , Σ2 ) ) = ||m1 −m2||2 + tr [ Σ1 + Σ2 − 2 ( Σ1/21 Σ2Σ 1/2 1 ) 1/2 ] as p = 2 , where m1 , Σ1 , m2 , Σ2 are mean and covariance metrics of the two normal distributions . In the following of this paper , we use DW to denote the W2 and it is worth noting that for the deterministic policy class , the trace term disappears and the only term involving the means remains , i.e. , DW = |m1 − m2| for Dirac delta policies located at points m1 and m2 . This diversity metric satisfies the three properties of a metric , namely identity , symmetry as well as triangle inequality . Proposition 1 ( Metric Space ( Θ , DqW ) ) . The expectation of DW ( · , · ) of two policies over any state distribution q ( s ) : D q W ( θi , θj ) : = Es∼q ( s ) [ DW ( θi ( a|s ) , θj ( a|s ) ) ] , ( 2 ) is a metric on Θ , thus ( Θ , D q W ) is a metric space . The proof of Proposition 1 is straightforward . It is worth mentioning that although the KL-divergence is commonly applied in measuring differences between policies ( Schulman et al. , 2015 ; 2017 ; Hong et al. , 2018 ) . It can not be used directly as a metric for novelty quantification as the symmetric property do not hold . In this sense , the Jensen Shannon divergence DJS and Total Variance Distance DTV ( Endres & Schindelin , 2003 ; Fuglede & Topsoe , 2004 ; Schulman et al. , 2015 ) can be also used as alternative choices , we choose DW in our work for that the Wasserstein metric better preserves the continuity ( Arjovsky et al. , 2017 ) . On top of the metric space ( Θ , D q W ) , the novelty of a policy can be computed as follows , Definition 1 ( Novelty of Policy ) . Given a reference policy set Θref such that Θref = { θrefi , i = 1 , 2 , ... } , Θref ⊂ Θ , the novelty U ( θ|Θref ) of policy θ is the minimal difference between θ and all policies in the reference policy set , i.e. , U ( θ|Θref ) : = min θj∈Θref D q W ( θ , θj ) , ( 3 ) Consequently , to encourage the discovery of novel policies discovery , typical novel policy generation methods tend to directly maximize the novelty of a new policy , i.e. , maxθ U ( θ|Θref ) , where Θref includes all existing policies . 2.2 ESTIMATION OF D q W ( θi , θj ) AND THE SELECTION OF q ( s ) In practice , the calculation of D q W ( θi , θj ) is based on Monte Carlo estimation where we need to sample s from q ( s ) . Although in Eq . ( 2 ) q ( s ) can be selected simply as a uniform distribution over the state space , there remains two obstacles : first , in a finite state space we can get precise estimation after establishing ergodicity , but problem arises when facing continuous state spaces due to the difficulty of efficiently obtaining enough samples ; second , when s is sampled from a uniform distribution q , we can only get sparse episodic reward instead of dense online reward which is more useful in learning . Therefore , we make an approximation here based on importance sampling . Formally , we denote the domain of q ( s ) as Sq ⊂ S and assume q ( s ) to be a uniform distribution over Sq , without loss of generality in later analysis . Notice Sq is closely related to the algorithm being used in generating trajectories ( Henderson et al. , 2018 ) . As we only care about the reachable regions of a certain algorithm ( in this work , PPO ) , the domain Sq can be decomposed by Sq = limN→∞ ⋃N i=1 Sθi , where Sθi denotes all the possible states a policy θi can visit given a starting state distribution . In order to get online-reward , we estimate Eq . ( 2 ) with D q W ( θi , θj ) = Es∼q ( s ) [ DW ( θi ( a|s ) , θj ( a|s ) ) ] = Es∼ρθi ( s ) [ q ( s ) ρθi ( s ) DW ( θi ( a|s ) , θj ( a|s ) ) ] , ( 4 ) where we use ρθ ( s ) to denote the stationary state visitation frequency under policy θ , i.e. , ρθ ( s ) = P ( s0 = s|θ ) + P ( s1 = s|θ ) + ... + P ( sT = s|θ ) in finite horizon problems . We propose to use the averaged stationary visitation frequency as q ( s ) , e.g. , for PPO , q ( s ) = ρ ( s ) = Eθ∼ΘPPO [ ρθ ( s ) ] . Clearly , choosing q ( s ) = ρ ( s ) will be much better than choosing a uniform distribution as the importance weight will be closer to 1 . Such an importance sampling process requires a necessary condition that ρθi ( s ) and q ( s ) have the same domain , which can be guaranteed by applying a sufficient exploration noise on θ . Another difficulty lies in the estimation of ρ ( s ) , which is always intractable given a limited number of trajectories . However , during training , θi is a policy to be optimized and θj ∈ Θref is a fixed reference policy . The error introduced by approximating the importance weight as 1 will get larger when θi becomes more distinct from normal policies , at least in terms of the state visitation frequency . We may just regard increasing of the approximation error as the discovery of novel policies . Proposition 2 ( Unbiased Single Trajectory Estimation ) . The estimation of ρθ ( s ) using a single trajectory τ is unbiased . The Proposition 2 follows the usual trick in RL that uses a single trajectory to estimate the stationary state visitation frequency , and indicates that with a sufficiently large number of trajectories , we are able to access the exact value of novelty . Given the definition of novelty and a practically unbiased sampling method , the next step is to develop an efficient learning algorithm .

Summary: This paper proposed a method to leverage the constrained optimization for policy training to learn diverse policies given some references. Based on a diversity metric defined on policy divergences, the paper employs two constrained optimization techniques for this problem with some modifications. Experiments on mujoco environments suggest that the proposed algorithms can beat existing diversity-driven policy optimization methods to learn both better and novel policies. Generally, the paper is well-written and easy to follow. Some concerns/comments: