Novel Policy Seeking with Constrained Optimization

1 INTRODUCTION . In the sense of learning through interactions with the environment , the scheme of reinforcement learning ( RL ) is conceptually similar to the emergence of intelligence ( Sutton et al. , 1998 ) : an agent explores and exploits information of a given environment , learns to master some certain skills through trials and errors to gain as much reward as possible . When solving a problem , we humans could be creative to come up with multiple different solutions and gain insights from searching for diverse solutions . e.g. , a self-containing example is the various approaches in RL research . While the state-of-the-art algorithms have achieved superhuman performance in a variety of challenging tasks ( Vinyals et al. , 2019 ; Akkaya et al. , 2019 ; Berner et al. , 2019 ; Badia et al. , 2020 ; Elbarbari et al. , 2021 ) , the task of encouraging individualized diversity 2 of learned agents , on the other hand , is relatively under-explored . Different from conventional RL agents that are only learned through interactions with the external environment , novel policy generation is a task considering the differentiation among individual policies . The differentiation among policies can be explained as the social influence ( Rogoff , 1990 ; Ryan & Deci , 2000 ; van Schaik & Burkart , 2011 ; Henrich , 2017 ; Harari , 2014 ) in social science literature . Although many works have been proposed applying social motivation to Multi-Agent Reinforcement Learning ( MARL ) settings ( Jaques et al. , 2019 ; Hughes et al. , 2018 ; Sequeira et al. , 2011 ; Peysakhovich & Lerer , 2017 ) , how to motivate a single RL agent to perform differently against existing agents is still an open question . In previous attempts for novel policy generation , there are three main challenges : ( 1 ) heuristically defined metric for novelty estimation is computational expensive ( Zhang et al. , 2019 ) , ( 2 ) defining novelty reward for an entire episode yields additional challenge in credit assignment , and ( 3 ) solving 1Code is included in the supplemental material . 2Note that this is different from diversity-driven exploration . While the latter focuses on the inner-diversity of a policy , in our work we focus on the inter-diversity between policies the problem under the formulation of multi-objective optimization leads to the performance decay in the original task . Fig . 1 compares the policy gradients of three cases , namely the one without novel policy seeking , novelty-seeking with multi-objective optimization , and novelty-seeking with constrained optimization methods , respectively . In this work we take into consideration not only the novelty of a set of learned policies but also the performance of those novel policies in the primal task , when addressing the problem of novel-policygeneration . Our contributions can be summarized as follows : 1 . We introduce a lightweight metric to compute the difference between policies with instant feedback at every timestep , to address the first two drawbacks of previous novel policy seeking methods discussed above ; 2 . We propose a constrained optimization formulation for novel policy generation and design a practical algorithm of IPD , resembling the interior point method in constrained optimization literature ; 3 . The proposed method , IPD , is evaluated on several continuous control benchmarks , showing the strengths of our constrained optimization solution for novelty-seeking can generate a series of diverse and well-performing policies , compared to previous multi-objective novel policy generation methods . 2 RELATED WORK . Intrinsic motivation methods . In previous work , different approaches are proposed to provide intrinsic motivation or intrinsic reward as a supplementary to the primal task reward for better exploration ( Houthooft et al. , 2016 ; Pathak et al. , 2017 ; Burda et al. , 2018a ; b ; Liu et al. , 2019 ) . All those approaches use the weighted sum of two rewards , the primal rewards provided by environments and the intrinsic rewards provided by different heuristics . On the other hand , the work of DIAYN and DADS ( Eysenbach et al. , 2018 ; Sharma et al. , 2019 ) learn diverse skills without extrinsic reward . Those approaches focus on decomposing diverse skills of a single policy , while our work focuses on learning diverse behaviors among a batch of policies for the same task . Diverse policy generation methods . The work of Such et al . shows that different RL algorithms may converge to different policies for the same task ( Such et al. , 2018 ) . On the contrary , we are interested in learning different policies through a single algorithm with the capability of avoiding local optimum . The work of Pugh et al . establishes a standard framework for understanding and comparing different approaches to search for quality diversity ( QD ) ( Pugh et al. , 2016 ) . Conti et al . proposes a solution which avoids local optima as well as achieves higher performance by adding novelty search and QD to evolution strategies ( Conti et al. , 2018 ) . The Task-Novelty Bisector ( TNB ) ( Zhang et al. , 2019 ) aims to solve novel policy generation problem by jointly optimize the extrinsic rewards and novelty rewards defined by an auto-encoder . In this work , we first adopt TNB in the constrained optimization framework , resulting in Contrained TNB , to demonstrate the dilemma between the task performance and novelty pursuance . Constrained Markov Decision Process . The Constrained Markov Decision Process ( CMDP ) ( Altman , 1999 ) considers the situation where an agent interacts with the environment under certain constraints . Formally , the CMDP can be defined as a tuple ( S , A , γ , r , c , C , P , s0 ) , where S andA are the state and action space ; γ ∈ [ 0 , 1 ) is a discount factor ; r : S×A×S → R and c : S×A×S → R denote the reward function and cost function ; C ∈ R+ is the upper bound of permitted expected cumulative cost ; P ( ·|s , a ) : S ×A → S denotes the transition dynamics , and s0 is the initial state . Denote the Markovian policy class as Π , where Π = { π : S × A → [ 0 , 1 ] , ∑ a π ( a|π ) = 1 } The learning objective of a policy for CMDP is to find a π∗ ∈ Π , such that π∗ = max π∈Π Eτ∼π , s′∼P [ ∞∑ t=0 γtr ( s , a , s′ ) ] , s.t . Eτ∼π , s′∼P [ ∞∑ t=0 γtc ( s , a , s′ ) ] ≤ C , ( 1 ) where τ indicates a trajectory ( s0 , a0 , s1 , ... ) and τ ∼ π represents the distribution over trajectories following policy π : at ∼ π ( ·|st ) , st+1 ∼ P ( ·|st , at ) ; t = 0 , 1 , 2 , .... Previous literature provide several approaches to solve CMDP ( Achiam et al. , 2017 ; Chow et al. , 2018 ; Ray et al. , 2019 ; Sun et al. , 2021 ) . In Sec.2.1 , we define a metric space that measures the difference between policies , which is the fundamental element for the proposed methods . In Sec.2.2 , we develop a practical estimation method for this metric . Sec.2.3 describes the formulation of constrained optimization on novel policy generation . The implementations of two practical algorithms are further introduced in Sec.2.4 . We denote the policies as { πθi ; θi ∈ Θ , i = 1 , 2 , ... } , wherein θi represents parameters of the i-th policy , Θ denotes the whole parameter space . In this work , we focus on improving the behavioral diversity of policies from PPO ( Schulman et al. , 2017 ) , thus we use Θ to represent ΘPPO in this paper . It is worth noting that the proposed methods can be easily extended to other RL algorithms ( Schulman et al. , 2015 ; Lillicrap et al. , 2015 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) . To simplify the notation , we omit π and denote a policy πθi as θi unless stated otherwise . 2.1 MEASURING THE DIFFERENCE BETWEEN POLICIES . We use the Wasserstein metric Wp ( Rüschendorf , 1985 ; Villani , 2008 ; Arjovsky et al. , 2017 ) to measure the distance between policies . Concretely , we consider the Gaussian-parameterized policies , where the Wp over two policies can be written in the closed form W 22 ( N ( m1 , Σ1 ) , N ( m2 , Σ2 ) ) = ||m1 −m2||2 + tr [ Σ1 + Σ2 − 2 ( Σ1/21 Σ2Σ 1/2 1 ) 1/2 ] as p = 2 , where m1 , Σ1 , m2 , Σ2 are mean and covariance metrics of the two normal distributions . In the following of this paper , we use DW to denote the W2 and it is worth noting that for the deterministic policy class , the trace term disappears and the only term involving the means remains , i.e. , DW = |m1 − m2| for Dirac delta policies located at points m1 and m2 . This diversity metric satisfies the three properties of a metric , namely identity , symmetry as well as triangle inequality . Proposition 1 ( Metric Space ( Θ , DqW ) ) . The expectation of DW ( · , · ) of two policies over any state distribution q ( s ) : D q W ( θi , θj ) : = Es∼q ( s ) [ DW ( θi ( a|s ) , θj ( a|s ) ) ] , ( 2 ) is a metric on Θ , thus ( Θ , D q W ) is a metric space . The proof of Proposition 1 is straightforward . It is worth mentioning that although the KL-divergence is commonly applied in measuring differences between policies ( Schulman et al. , 2015 ; 2017 ; Hong et al. , 2018 ) . It can not be used directly as a metric for novelty quantification as the symmetric property do not hold . In this sense , the Jensen Shannon divergence DJS and Total Variance Distance DTV ( Endres & Schindelin , 2003 ; Fuglede & Topsoe , 2004 ; Schulman et al. , 2015 ) can be also used as alternative choices , we choose DW in our work for that the Wasserstein metric better preserves the continuity ( Arjovsky et al. , 2017 ) . On top of the metric space ( Θ , D q W ) , the novelty of a policy can be computed as follows , Definition 1 ( Novelty of Policy ) . Given a reference policy set Θref such that Θref = { θrefi , i = 1 , 2 , ... } , Θref ⊂ Θ , the novelty U ( θ|Θref ) of policy θ is the minimal difference between θ and all policies in the reference policy set , i.e. , U ( θ|Θref ) : = min θj∈Θref D q W ( θ , θj ) , ( 3 ) Consequently , to encourage the discovery of novel policies discovery , typical novel policy generation methods tend to directly maximize the novelty of a new policy , i.e. , maxθ U ( θ|Θref ) , where Θref includes all existing policies . 2.2 ESTIMATION OF D q W ( θi , θj ) AND THE SELECTION OF q ( s ) In practice , the calculation of D q W ( θi , θj ) is based on Monte Carlo estimation where we need to sample s from q ( s ) . Although in Eq . ( 2 ) q ( s ) can be selected simply as a uniform distribution over the state space , there remains two obstacles : first , in a finite state space we can get precise estimation after establishing ergodicity , but problem arises when facing continuous state spaces due to the difficulty of efficiently obtaining enough samples ; second , when s is sampled from a uniform distribution q , we can only get sparse episodic reward instead of dense online reward which is more useful in learning . Therefore , we make an approximation here based on importance sampling . Formally , we denote the domain of q ( s ) as Sq ⊂ S and assume q ( s ) to be a uniform distribution over Sq , without loss of generality in later analysis . Notice Sq is closely related to the algorithm being used in generating trajectories ( Henderson et al. , 2018 ) . As we only care about the reachable regions of a certain algorithm ( in this work , PPO ) , the domain Sq can be decomposed by Sq = limN→∞ ⋃N i=1 Sθi , where Sθi denotes all the possible states a policy θi can visit given a starting state distribution . In order to get online-reward , we estimate Eq . ( 2 ) with D q W ( θi , θj ) = Es∼q ( s ) [ DW ( θi ( a|s ) , θj ( a|s ) ) ] = Es∼ρθi ( s ) [ q ( s ) ρθi ( s ) DW ( θi ( a|s ) , θj ( a|s ) ) ] , ( 4 ) where we use ρθ ( s ) to denote the stationary state visitation frequency under policy θ , i.e. , ρθ ( s ) = P ( s0 = s|θ ) + P ( s1 = s|θ ) + ... + P ( sT = s|θ ) in finite horizon problems . We propose to use the averaged stationary visitation frequency as q ( s ) , e.g. , for PPO , q ( s ) = ρ ( s ) = Eθ∼ΘPPO [ ρθ ( s ) ] . Clearly , choosing q ( s ) = ρ ( s ) will be much better than choosing a uniform distribution as the importance weight will be closer to 1 . Such an importance sampling process requires a necessary condition that ρθi ( s ) and q ( s ) have the same domain , which can be guaranteed by applying a sufficient exploration noise on θ . Another difficulty lies in the estimation of ρ ( s ) , which is always intractable given a limited number of trajectories . However , during training , θi is a policy to be optimized and θj ∈ Θref is a fixed reference policy . The error introduced by approximating the importance weight as 1 will get larger when θi becomes more distinct from normal policies , at least in terms of the state visitation frequency . We may just regard increasing of the approximation error as the discovery of novel policies . Proposition 2 ( Unbiased Single Trajectory Estimation ) . The estimation of ρθ ( s ) using a single trajectory τ is unbiased . The Proposition 2 follows the usual trick in RL that uses a single trajectory to estimate the stationary state visitation frequency , and indicates that with a sufficiently large number of trajectories , we are able to access the exact value of novelty . Given the definition of novelty and a practically unbiased sampling method , the next step is to develop an efficient learning algorithm .

Summary: This paper aims at novel policy seeking which incorporates curiosity-driven exploration for better reinforcement learning. This paper first propose to use  a Wasserstein-based metric to calculate the difference between policies, and use it to define  the policy novelty. With these, the authors modeled the novel policy seeking as a constrained markov decision process(CMDP) and solved it using CTNB and IPD.