Regulatory Focus: Promotion and Prevention Inclinations in Policy Search

1 INTRODUCTION . It is not enough to do good ; one must do it the right way . The best-known motivational principle that has been recognized in psychology ( Gray , 1982 ; Atkinson , 1964 ) is the hedonic principle that people approach pleasure and avoid pain . However the principle does not explain how people approach and avoid . Higgins ( 1997 ) proposes the regulatory focus theory to address this question . A promotion focus is concerned with advancement , growth , and accomplishment , and a prevention focus is concerned with security , safety , and responsibility ( Crowe & Higgins , 1997 ) . The theory hypothesized that the promotion focus inclination insures against errors of omission , and the prevention focus inclination insures against errors of commission . Furthermore , the regulatory fit theory ( Higgins , 2000 ; Higgins et al. , 2003 ) proposes that when people pursue a goal in a manner that fit their regulatory orientation , they are stronger motivated during goal pursuit . For example , Lee & Aaker ( 2004 ) showed that ads are more persuasive if there is a regulatory fit : use gain frames ( e.g . “ great looks and exceptional engineering ” ) for individuals who are eager to approach positive outcomes , and use loss frames ( e.g . “ don ’ t be stranded with a disabled vehicle without an emergency safety kit ” ) for individuals who are vigilant to avoid negative outcomes . We study the phenomenon of regulatory fit in the context of reinforcement learning ( RL ) . Be aware that humans own real emotions and are fundamentally more complex than RL agents , thus we borrow terminologies in psychology while trying to retain the resemblance . We hypothesize that different RL environments are more fit with agents of different regulatory focus . For example , in the environments with sparse rewards , it might be advisable to actively exploring those policies that show potentials without being discouraged by a few failed trials ; while in the environments that are fragile , e.g . those where a wrong choice of action can lead to catastrophic consequences down the path , a more conservative , risk-averse stance might be preferable . There are three questions need to answer : ( 1 ) How to automatically determine the suitable regulatory focus for an environment ; ( 2 ) How to train an agent with a specified regulatory focus ; ( 3 ) Is there any benefit when training the agent with the regulatory focus that fits the environment ? In this article , we aim to answer the second and third questions and leave the first question open . The research on deep reinforcement learning is gaining momentum in recent years . A number of learning methods , such as Proximal Policy Optimization algorithm ( PPO ) ( Schulman et al. , 2017 ) , Trust-Region Policy Optimization algorithm ( TRPO ) ( Schulman et al. , 2015a ) , and Advantage ActorCritic algorithm ( A2C ) ( Mnih et al. , 2016 ) , have been developed . These methods and their variants have achieved great success in challenging problems , e.g . continuous control ( Schulman et al. , 2015b ) , locomotion ( Heess et al. , 2017 ) , and video games ( Vinyals et al. , 2017 ) . The core of all these methods is the estimation of the advantage A ( st , at ) , i.e . the gain in the expected cumulative reward relative to the state value V ( st ) if a certain action at is taken . A common practice is to use the n-step estimate over the sampled trajectories . This way has been widely used in actor-critic algorithms such as the A2C . Schulman et al . presents a generalization called Generalized Advantage Estimator ( GAE ) ( Schulman et al. , 2015b ) , which blends 1 to n-step estimators with exponentially decayed weights , thus reducing the variance of policy gradients . For convenience , we refer to the set of 1 to n-step estimators along a trajectory as the path ensemble . We propose to take the maximum over path ensemble to implement the promotion focus inclination , and take the minimum over path ensemble to implement to prevention inclination during training . Intuitively , when taking the maximum or minimum , the agent is comparing a list of counterfactual speculations . When it looks at a k-step estimation where k ≤ n , it is thinking what if it didn ’ t took those actions after step k ? For a promotion focus , in order to insure hits and insure against omission , it would favor the estimation with largest value because it has the possibility of being that good . Similarly , for a prevention focus , it prefers the minimum to insure against errors of commission . We systematically studied different estimation schemes on various environments , including the sparse-reward environments ( Fu et al. , 2017 ) , the Terrain RL Simulator ( Berseth et al. , 2018 ) for biped locomotion , the Atari games ( Bellemare et al. , 2013 ) , and the set of MuJoCo environments for continuous control ( Brockman et al. , 2016 ) . Our study shows that the proposed estimation strategies , when chosen appropriately , can substantially outperform the standard way . In particular , the promotion focus inclination , i.e . taking the maximum over path ensemble , greatly improves the learning efficiency in the environments with sparse rewards . On the other hand , the prevention focus inclination , i.e . taking the minimum , can effectively stabilize the learning by avoiding risky actions . It is noteworthy that the proposed strategic inclinations differ essentially from the recent two lines of work on robustness ( Smirnova et al. , 2019 ; Delage & Mannor , 2007 ) and risk-sensitivity ( Tamar et al. , 2012 ; Chow & Ghavamzadeh , 2014 ) . Robust MDP optimizes for the worst case when there exist uncertainties in the parameters ; risk-sensitive MDP optimizes the value of a risk measure . Our method does not modify the training objective . Instead , it controls the policy gradient towards policies with promotion or prevention inclinations through the alternative ways of estimating the advantage . 2 RELATED WORK . Estimation of Advantage and Action-Value The n-step advantage estimation is derived from the n-step return , which is used in the n-step TD or the n-step Sarsa method ( Sutton & Barto , 2011 ) . Similarly , the generalized advantage estimator ( Schulman et al. , 2015b ) is analogous to the λ-return ( Watkins , 1989 ) in the TD ( λ ) method ( Seijen & Sutton , 2014 ) . They are all linear combinations over the path ensemble . A nonlinear combination scheme is seen in the work of Twin Delayed DDPG ( TD3 ) ( Fujimoto et al. , 2018 ) , which uses the min of two critics to estimate Q . However , it aims to mitigate the overestimation problem in deep Q-learning , and the objects be combined are the network outputs instead of the estimators based on the cumulative rewards . Another nonlinear combination is the Positive Temporal Difference ( PTD ) ( Van Hasselt & Wiering , 2007 ; van Hasselt , 2012 ) , where the advantage is set as 1 when the 1-step estimate of advantage is positive , and is set to 0 otherwise . The benefit of limiting policy updates only toward those actions that have positive advantages is increasing the stability of learning . Robustness and Risk-Sensitivity in RL In the assumption of robust MDPs ( Nilim & El Ghaoui , 2004 ; Xu & Mannor , 2007 ; Delage & Mannor , 2010 ; Mannor et al. , 2012 ; Smirnova et al. , 2019 ) , the parameters of the problem lie in an uncertainty set , and the target is to find a solution that performs best under the worst cases . On the other hand , risk-sensitive MDPs consider the uncertainty of the rewards . The objective is to minimize the risk-measure , which is defined by the exponential utility ( Howard & Matheson , 1972 ) , the Conditional Value-at-Risk ( CVaR ) ( Chow & Ghavamzadeh , 2014 ; Prashanth , 2014 ; Chow et al. , 2015 ; Tamar et al. , 2015 ) , the percentile ( Delage & Mannor , 2007 ) or the variance ( Tamar et al. , 2012 ; Prashanth & Ghavamzadeh , 2013 ) of the cumulative rewards . The statistics over the path ensemble is not the same as those statistics for advantage . However , using a suitable nonlinear combination of the numbers in the path ensemble , the exploration strategy in our algorithms can bias the learning toward risk-averse or risk-seeking policies . Unlike the robust and risk-sensitive approaches , which typically introduce involved computations , our approach is straightforward to implement and requires minimal changes to incorporate the idea into existing algorithms such as A2C , PPO , TRPO and their variants . Distributional RL The value distribution is the object of study in distributional RL . Since the full distribution of return is difficult to estimate , researchers have adopted nonparametric methods ( Morimura et al. , 2010 ) or used simple distributions ( Bellemare et al. , 2017 ; Dabney et al. , 2018 ; Barth-Maron et al. , 2018 ) to approximate it . The RL algorithms can be formulated with any criterion based on the distribution of return , such as the aforementioned risk-sensitivity measures . Again , the distribution formed by elements in the path ensemble does not represent the full distribution of return , but some joint distribution that we can exploit . 3 REGULATORY FOCUS BY ORDER STATISTICS OVER PATH ENSEMBLE . In this section , we first review the preliminary knowledge on RL . Then we introduce the central concept of this article : regulatory focus implemented by estimators over path ensembles . Specifically , we focus on the family of order statistics . Next , we give an illustrative study on the max statistics over the path ensemble , where we show that the induced estimator of action-values would influence the optimization process of RL algorithms . With this specific example in mind , we give a practical algorithm to incorporate the general estimators into existing RL algorithms in the last section . 3.1 PRELIMINARY . We consider the standard formulation of RL , which is typically modeled as an MDP ( S , A , T , γ , R ) , where S is the state space , A is the action space , T = { Psa ( · ) | s ∈ S , a ∈ A } is the transition probabilities , γ ∈ ( 0 , 1 ] is the discount factor , and R is the reward function . At timestep t , the agent in state st interacts with the environment by choosing an action at ∼ π ( st ) following the policy π , and receives a reward Rt from the environment . The environment then transits to the next state st+1 . The discounted return is defined as Gt : = ∑T−1 i=t γ i−tRi , and the goal of RL is to maximize the expected return J = Es0∼S0 [ G0 | s0 ] , where S0 is the distribution of initial states . The action-value function Qπ ( st , at ) : = E [ Gt | st , at ] under a policy π is the expected return of taking action at ∼ π ( st ) in state st . In policy-based model-free deep RL algorithms , policy πθ with parameters θ is optimized via gradient ascent . An estimation of the gradient is the policy gradient ∇θJ ( θ ) = Eτ∼πθ ( τ ) [ T∑ t=0 ∇θπθ ( at | st ) πθ ( at | st ) A ( st , at ) ] , ( 1 ) where τ = { s0 , a0 , s1 , a1 , . . . , sT , aT } is a trajectory following the policy . A ( st , at ) : = Q ( st , at ) − V ( st ) is the advantage function . The n-step look ahead estimation of advantage function is given by Â ( n ) t : = n−1∑ i=0 ( γirt+i ) + γnV ( st+n ) − V ( st ) , n = 1 , 2 , . . . , T − t. ( 2 )

Summary: This paper presents a modification to policy gradient methods that are computed from advantage function estimates. For a given trajectory of n steps, there are n different advantage function estimates: from 1-step to n-step. GAE (Schulman, 2016) proposes to take an exponentially weighted average of these estimates to compute the policy gradients. This paper proposes instead to use order statistics to compute the policy gradient; e.g. the most optimistic estimate, the most pessimistic estimate, or the most extreme estimate.  The paper introduces a regulatory ratio: the probability of using the averaged advantage estimate vs using the order statistic, for computing the policy gradients. This hyper-parameter is justified on the optimistic case (max advantage), as a way to prevent overtly optimistic estimates. The paper conducts experiments on  different domains (sparse and dense rewards, discrete and continuous actions, fully observable and partially observable environments) which show the effect of choosing different order statistics and regulatory ratio on the policy performance.