Regulatory Focus: Promotion and Prevention Inclinations in Policy Search

1 INTRODUCTION . It is not enough to do good ; one must do it the right way . The best-known motivational principle that has been recognized in psychology ( Gray , 1982 ; Atkinson , 1964 ) is the hedonic principle that people approach pleasure and avoid pain . However the principle does not explain how people approach and avoid . Higgins ( 1997 ) proposes the regulatory focus theory to address this question . A promotion focus is concerned with advancement , growth , and accomplishment , and a prevention focus is concerned with security , safety , and responsibility ( Crowe & Higgins , 1997 ) . The theory hypothesized that the promotion focus inclination insures against errors of omission , and the prevention focus inclination insures against errors of commission . Furthermore , the regulatory fit theory ( Higgins , 2000 ; Higgins et al. , 2003 ) proposes that when people pursue a goal in a manner that fit their regulatory orientation , they are stronger motivated during goal pursuit . For example , Lee & Aaker ( 2004 ) showed that ads are more persuasive if there is a regulatory fit : use gain frames ( e.g . “ great looks and exceptional engineering ” ) for individuals who are eager to approach positive outcomes , and use loss frames ( e.g . “ don ’ t be stranded with a disabled vehicle without an emergency safety kit ” ) for individuals who are vigilant to avoid negative outcomes . We study the phenomenon of regulatory fit in the context of reinforcement learning ( RL ) . Be aware that humans own real emotions and are fundamentally more complex than RL agents , thus we borrow terminologies in psychology while trying to retain the resemblance . We hypothesize that different RL environments are more fit with agents of different regulatory focus . For example , in the environments with sparse rewards , it might be advisable to actively exploring those policies that show potentials without being discouraged by a few failed trials ; while in the environments that are fragile , e.g . those where a wrong choice of action can lead to catastrophic consequences down the path , a more conservative , risk-averse stance might be preferable . There are three questions need to answer : ( 1 ) How to automatically determine the suitable regulatory focus for an environment ; ( 2 ) How to train an agent with a specified regulatory focus ; ( 3 ) Is there any benefit when training the agent with the regulatory focus that fits the environment ? In this article , we aim to answer the second and third questions and leave the first question open . The research on deep reinforcement learning is gaining momentum in recent years . A number of learning methods , such as Proximal Policy Optimization algorithm ( PPO ) ( Schulman et al. , 2017 ) , Trust-Region Policy Optimization algorithm ( TRPO ) ( Schulman et al. , 2015a ) , and Advantage ActorCritic algorithm ( A2C ) ( Mnih et al. , 2016 ) , have been developed . These methods and their variants have achieved great success in challenging problems , e.g . continuous control ( Schulman et al. , 2015b ) , locomotion ( Heess et al. , 2017 ) , and video games ( Vinyals et al. , 2017 ) . The core of all these methods is the estimation of the advantage A ( st , at ) , i.e . the gain in the expected cumulative reward relative to the state value V ( st ) if a certain action at is taken . A common practice is to use the n-step estimate over the sampled trajectories . This way has been widely used in actor-critic algorithms such as the A2C . Schulman et al . presents a generalization called Generalized Advantage Estimator ( GAE ) ( Schulman et al. , 2015b ) , which blends 1 to n-step estimators with exponentially decayed weights , thus reducing the variance of policy gradients . For convenience , we refer to the set of 1 to n-step estimators along a trajectory as the path ensemble . We propose to take the maximum over path ensemble to implement the promotion focus inclination , and take the minimum over path ensemble to implement to prevention inclination during training . Intuitively , when taking the maximum or minimum , the agent is comparing a list of counterfactual speculations . When it looks at a k-step estimation where k ≤ n , it is thinking what if it didn ’ t took those actions after step k ? For a promotion focus , in order to insure hits and insure against omission , it would favor the estimation with largest value because it has the possibility of being that good . Similarly , for a prevention focus , it prefers the minimum to insure against errors of commission . We systematically studied different estimation schemes on various environments , including the sparse-reward environments ( Fu et al. , 2017 ) , the Terrain RL Simulator ( Berseth et al. , 2018 ) for biped locomotion , the Atari games ( Bellemare et al. , 2013 ) , and the set of MuJoCo environments for continuous control ( Brockman et al. , 2016 ) . Our study shows that the proposed estimation strategies , when chosen appropriately , can substantially outperform the standard way . In particular , the promotion focus inclination , i.e . taking the maximum over path ensemble , greatly improves the learning efficiency in the environments with sparse rewards . On the other hand , the prevention focus inclination , i.e . taking the minimum , can effectively stabilize the learning by avoiding risky actions . It is noteworthy that the proposed strategic inclinations differ essentially from the recent two lines of work on robustness ( Smirnova et al. , 2019 ; Delage & Mannor , 2007 ) and risk-sensitivity ( Tamar et al. , 2012 ; Chow & Ghavamzadeh , 2014 ) . Robust MDP optimizes for the worst case when there exist uncertainties in the parameters ; risk-sensitive MDP optimizes the value of a risk measure . Our method does not modify the training objective . Instead , it controls the policy gradient towards policies with promotion or prevention inclinations through the alternative ways of estimating the advantage . 2 RELATED WORK . Estimation of Advantage and Action-Value The n-step advantage estimation is derived from the n-step return , which is used in the n-step TD or the n-step Sarsa method ( Sutton & Barto , 2011 ) . Similarly , the generalized advantage estimator ( Schulman et al. , 2015b ) is analogous to the λ-return ( Watkins , 1989 ) in the TD ( λ ) method ( Seijen & Sutton , 2014 ) . They are all linear combinations over the path ensemble . A nonlinear combination scheme is seen in the work of Twin Delayed DDPG ( TD3 ) ( Fujimoto et al. , 2018 ) , which uses the min of two critics to estimate Q . However , it aims to mitigate the overestimation problem in deep Q-learning , and the objects be combined are the network outputs instead of the estimators based on the cumulative rewards . Another nonlinear combination is the Positive Temporal Difference ( PTD ) ( Van Hasselt & Wiering , 2007 ; van Hasselt , 2012 ) , where the advantage is set as 1 when the 1-step estimate of advantage is positive , and is set to 0 otherwise . The benefit of limiting policy updates only toward those actions that have positive advantages is increasing the stability of learning . Robustness and Risk-Sensitivity in RL In the assumption of robust MDPs ( Nilim & El Ghaoui , 2004 ; Xu & Mannor , 2007 ; Delage & Mannor , 2010 ; Mannor et al. , 2012 ; Smirnova et al. , 2019 ) , the parameters of the problem lie in an uncertainty set , and the target is to find a solution that performs best under the worst cases . On the other hand , risk-sensitive MDPs consider the uncertainty of the rewards . The objective is to minimize the risk-measure , which is defined by the exponential utility ( Howard & Matheson , 1972 ) , the Conditional Value-at-Risk ( CVaR ) ( Chow & Ghavamzadeh , 2014 ; Prashanth , 2014 ; Chow et al. , 2015 ; Tamar et al. , 2015 ) , the percentile ( Delage & Mannor , 2007 ) or the variance ( Tamar et al. , 2012 ; Prashanth & Ghavamzadeh , 2013 ) of the cumulative rewards . The statistics over the path ensemble is not the same as those statistics for advantage . However , using a suitable nonlinear combination of the numbers in the path ensemble , the exploration strategy in our algorithms can bias the learning toward risk-averse or risk-seeking policies . Unlike the robust and risk-sensitive approaches , which typically introduce involved computations , our approach is straightforward to implement and requires minimal changes to incorporate the idea into existing algorithms such as A2C , PPO , TRPO and their variants . Distributional RL The value distribution is the object of study in distributional RL . Since the full distribution of return is difficult to estimate , researchers have adopted nonparametric methods ( Morimura et al. , 2010 ) or used simple distributions ( Bellemare et al. , 2017 ; Dabney et al. , 2018 ; Barth-Maron et al. , 2018 ) to approximate it . The RL algorithms can be formulated with any criterion based on the distribution of return , such as the aforementioned risk-sensitivity measures . Again , the distribution formed by elements in the path ensemble does not represent the full distribution of return , but some joint distribution that we can exploit . 3 REGULATORY FOCUS BY ORDER STATISTICS OVER PATH ENSEMBLE . In this section , we first review the preliminary knowledge on RL . Then we introduce the central concept of this article : regulatory focus implemented by estimators over path ensembles . Specifically , we focus on the family of order statistics . Next , we give an illustrative study on the max statistics over the path ensemble , where we show that the induced estimator of action-values would influence the optimization process of RL algorithms . With this specific example in mind , we give a practical algorithm to incorporate the general estimators into existing RL algorithms in the last section . 3.1 PRELIMINARY . We consider the standard formulation of RL , which is typically modeled as an MDP ( S , A , T , γ , R ) , where S is the state space , A is the action space , T = { Psa ( · ) | s ∈ S , a ∈ A } is the transition probabilities , γ ∈ ( 0 , 1 ] is the discount factor , and R is the reward function . At timestep t , the agent in state st interacts with the environment by choosing an action at ∼ π ( st ) following the policy π , and receives a reward Rt from the environment . The environment then transits to the next state st+1 . The discounted return is defined as Gt : = ∑T−1 i=t γ i−tRi , and the goal of RL is to maximize the expected return J = Es0∼S0 [ G0 | s0 ] , where S0 is the distribution of initial states . The action-value function Qπ ( st , at ) : = E [ Gt | st , at ] under a policy π is the expected return of taking action at ∼ π ( st ) in state st . In policy-based model-free deep RL algorithms , policy πθ with parameters θ is optimized via gradient ascent . An estimation of the gradient is the policy gradient ∇θJ ( θ ) = Eτ∼πθ ( τ ) [ T∑ t=0 ∇θπθ ( at | st ) πθ ( at | st ) A ( st , at ) ] , ( 1 ) where τ = { s0 , a0 , s1 , a1 , . . . , sT , aT } is a trajectory following the policy . A ( st , at ) : = Q ( st , at ) − V ( st ) is the advantage function . The n-step look ahead estimation of advantage function is given by Â ( n ) t : = n−1∑ i=0 ( γirt+i ) + γnV ( st+n ) − V ( st ) , n = 1 , 2 , . . . , T − t. ( 2 )

Summary: The paper studies the problem of advantage estimation for actor-critic RL algorithms. The key observation is that the advantage can be computed using 1-step returns, 2-step returns, etc. The paper suggests that, instead of choosing a fixed n, we should aggregate these advantageous together. If the maximum is taken, the resulting policy will be exploratory (i.e., have a "promotion focus"); if the minimum advantage is taken, the resulting policy will be risk sensitive (i.e., have a "regulatory focus").