Success-Rate Targeted Reinforcement Learning by Disorientation Penalty

Current reinforcement learning generally uses discounted return as its learning objective . However , real-world tasks may often demand a high success rate , which can be quite different from optimizing rewards . In this paper , we explicitly formulate the success rate as an undiscounted form of return with { 0 , 1 } -binary reward function . Unfortunately , applying traditional Bellman updates to value function learning can be problematic for learning undiscounted return , and thus not suitable for optimizing success rate . From our theoretical analysis , we discover that values across different states tend to converge to the same value , resulting in the agent wandering around those states without making any actual progress . This further leads to reduced learning efficiency and inability to complete a task in time . To combat the aforementioned issue , we propose a new method , which introduces Loop Penalty ( LP ) into value function learning , to penalize disoriented cycling behaviors in agent ’ s decision-making . We demonstrate the effectiveness of our proposed LP on three environments , including grid-world cliff-walking , Doom first-person navigation and robot arm control , and compare our method with Qlearning , Monte-Carlo and Proximal Policy Optimization ( PPO ) . Empirically , LP improves the convergence of training and achieves a higher success rate . 1 INTRODUCTION . Reinforcement learning usually adopts expected discounted return as objective , and has been applied in many tasks to find the best solution , e.g . finding the shortest path and achieving the highest score ( Sutton & Barto , 2018 ; Mnih et al. , 2015 ; Shao et al. , 2018 ) . However , many real-world tasks , such as robot control or autonomous driving , may demand more in success rate ( i.e . the probability for the agent to fulfill task requirements ) since failures in these tasks may cause severe damage or consequences . Previous works commonly treat optimizing rewards equivalent to maximizing success rate ( Zhu et al. , 2018 ; Peng et al. , 2018 ; Kalashnikov et al. , 2018 ) , but their results can be error-prone when applied to real-world applications . We believe that success rate is different from expected discounted return . The reasons are as follows : 1 ) expected discounted return commonly provides dense reward signals for transitions in an episode , while success or not is a sparse binary signal only obtained at the end of an episode ; 2 ) expected discounted return commonly weights results in the immediate future more than potential rewards in the distant future , whereas success or not does not have such a weighting and is only concerned about the overall or the final result . Policies with high expected discounted returns are often more demanding in short-term performance than those with high success rates and optimizing success rates often leads to multiple solutions . As a result , policies with high success rates tend to be more reliable and risk-averse while policies with high expected discounted returns tend to be risk-seeking . See the cliff-walking example in Fig . 1 where the objective is to walk from the origin state marked with a triangle to the destination state marked with a circle . The “ Slip ” area in light grey winds with a certain probability pfall = 0.1 , making the agent uncontrollably move down ; the dark gray area at the bottom row denotes “ Cliff ” . In Fig . 1 , the blue trajectory shown on the left is shorter but riskier than the green one shown on the right . In commonly-used hyperparameter settings , such as γ = 0.9 , the agent tends to follow the blue trajectory rather than the green one , although the green trajectory has a higher success rate . We acknowledge that for this simple example , optimizing expected discounted return with a careful design of γ that meets ( 1 − pfall ) 4 < γ9−5 can produce a policy with the highest success rate . However , this result relies on task-specific knowledge about the environment , generally not available in more complex tasks . These findings lead us to the following question : can we express success rate in a general form so that it can be directly optimized ? In this paper , we discover a universal way of representing success rate is to 1 ) use a { 0 , 1 } -binary reward indicates whether or not a trajectory is successful , and 2 ) set γ = 1 so that the binary signal back-propagates without any discount . Unfortunately , this expression belongs to undiscounted problems and the convergence of value iteration often can not be guaranteed ( Xu et al. , 2018 ) . Nevertheless , we can still explicitly solve the Bellman equation in a matrix form for the special undiscounted return ( success rate ) . We derive that if the transition dynamics of the environment permit existence of an irreducible ergodic set of states , γ = 1 will lead to an undesirable situation : state or state-action values tend to converge to the same value , which we refer to as uniformity . As shown in Fig . 2 for the contour of state values in our cliff-walking example , uniformity is reflected as a plateau in the right figure , which is caused by non-discounting and does not exist in discounting cases ( left figure ) . Uniformity makes the selection of actions purposeless within the plateau , resulting in disoriented and time-consuming behaviors in the agent ’ s decision-making , and unsatisfactory success rates . Based on the above analysis , we introduce Loop-Penalty ( LP ) into value function learning to penalize disoriented and cycling behaviors in trajectories . We derive that this penalty can be realized by multiplying a special mask function to the original value function . Note that our strategy is general and is applicable to many RL algorithms . We provide concrete loss functions for three popular algorithms in this paper : Monte Carlo , Deep Q-learning and Proximal Policy Optimization ( Schulman et al. , 2017 ) . We verify the effectiveness in three representative environments : grid-world cliffwalking , vision-based robot grasping , and first-person navigation in 3D Vizdoom ( Kempka et al. , 2016 ) , showing that LP can alleviate the uniformity problem and achieve better performance . Finally , we summarize the major contributions of our paper in the following : • We formally introduce the objective of “ success rate ” in reinforcement learning . Our formulation of success rate is general and is applicable for many different RL tasks . • We theoretically analyze the difficulty in optimizing success rate and show that the uniformity among state values and the resulting loops in trajectories are the key challenges . • We propose LP which can be combined with any general RL algorithm . We demonstrate empirically that LP can alleviate the problem of “ uniformity ” among state values and significantly improve success rates in both discrete and continuous control tasks . 2 RELATED WORK . To the best of our knowledge , currently there is no research that adopts success rate directly as the learning objective . The reason is that success rate is usually not the main criterion in tasks investigated by RL , e.g . video games and simulated robot control . Although some studies used success rate to evaluate the performance of the policies ( Andrychowicz et al. , 2017 ; Tobin et al. , 2018 ; Ghosh et al. , 2018 ; Kalashnikov et al. , 2018 ) , they used task-specific reward design and discounted return during training , instead of directly optimizing success rate . The notion of “ success ” may be reflected in constraints considered in the domain of safe RL ( Garcı́a & Fernández , 2015 ) . Geibel & Wysotzki ( 2005 ) considered constraints on the agent ’ s behavior and discouraged the agent from moving to error states . Geibel ( 2006 ) studied constraints on the expected return to ensure acceptable performance . A . & Ghavamzadeh ( 2013 ) proposed constraints on the variance of some measurements to pursue an invariable performance . Previous studies have also considered safety in the exploration process ( Garcı́a & Fernández-Rebollo , 2012 ; Mannucci et al. , 2018 ) . Although these studies deemed success rate as an additional constraint in learning , they either simply assumed that the constraint can be certainly satisfied or penalized constraint violations . The deficiency of expected discounted return as a training objective has been recognized by many studies . Instead of just optimizing expected return , Heger ( 1994 ) ; Tamar et al . ( 2013 ) adopted the minimax criterion that optimizes the worst possible values of the return . By doing so , occasional small returns would not be ignored at test time . Gilbert & Weng ( 2016 ) ; Chow et al . ( 2017 ) extended this idea to arbitrary quantiles of the return . However , all these studies are not optimizing success rate directly since they are based on a quantitative measurement of performance and are unnecessarily sensitive to the worst cases . In contrast , success rate is based on a binary signal which only distinguishes between success and failure . Our work involves optimization of an undiscounted return . The instability in training towards an undiscounted return has been mentioned by Schwartz ( 1993 ) ; Xu et al . ( 2018 ) . However , most studies on undiscounted return focused on continuous settings and considered the average reward as objectives ( Schwartz , 1993 ; Ortner & Ryabko , 2012 ; Zahavy et al. , 2020 ) . There seems to be a general view that the instability in training towards undiscounted return only exists in continuous cases but not in episodic cases ( Pitis , 2019 ) . Contrary to this view , we propose that training instability also exists in episodic cases . For optimizing success rate , we provide a theoretical analysis and show the existence of training instability and propose a practical method that alleviates this problem . 3 SUCCESS RATE IN REINFORCEMENT LEARNING . In this section we provide a formal definition of success rate , explain its relationship with expected discounted sum of rewards , and analyze the problems in optimizing success rate . 3.1 SUCCESS RATE . In RL , given a policy π , success rate specifically refers to the ratio of the successful trajectories to all trajectories . As in a general setting of RL , a trajectory is expressed as τ = { ( s0 , a0 , r0 ) , . . . , ( sT , aT , rT ) , sT+1 } rolled out by following policy π , where st ∈ S is state , at ∈ A denotes action , rt represents immediate reward and T is the length of the trajectory . Because the notion of success should only depend on the visited states in a trajectory , we concisely express “ success ” by defining a set of desired states Sg ⊂ S that denote task completion , e.g . the destination state in our cliff-walking example . At a high level , the goal of the agent is to reach any state in Sg within a given planning horizon T , and the environment terminates either upon arriving at a desired state or reaching a maximum allocated timestep T . Without loss of generality , we say that “ a trajectory τ is successful ” if and only if τ−1 ∈ Sg , where τ−1 is the last state in τ . Formally , we use an indicator function I ( s ∈ Sg ) to denote success , where I ( · ) takes value of 1 when the input statement is true and 0 otherwise . Since this expression is task-independent , our analysis can be widely applicable . Accordingly , we formally define the success rate as follows : Definition 1 . The success rate of a given policy π is defined as βπ ( s0 ) = ∑ τ pπ ( τ |s0 ) I ( τ−1 ∈ Sg ) ( 1 ) where pπ ( τ |s0 ) = ∏T t=0 π ( at|st ) p ( st+1|st , at ) is the probability of observing trajectory τ . In order to find a policy that optimizes success rate , we derive a recursive form of policy evaluation similar to the Bellman equation ( Sutton & Barto , 2018 ) , as shown in Theorem 1 . Theorem 1 . The success rate is a state-value function represented as an expected sum of undiscounted return , with the reward function R ( s ) defined to take the value of 1 if s ∈ Sg , 0 otherwise . Proof sketch : We segment the trajectories and generate sub-trajectories , τ ∈ Γ , τ0 : k ∈ Γ̂ , where k ∈ ( 0 , T ] . Note that Γ = Γ̂ , because 1 ) ∀τ ∈ Γ , we have τ0 : T ∈ Γ̂ = τ , Γ ⊆ Γ̂ , 2 ) τ0 : k is a trajectory , Γ̂ ⊆ Γ . Then the success rate βπ ( st ) can be rewritten as the product sum of the probability of reaching st+k and the indicator I ( τst+k ∈ Sg ) for all st+k : βπ ( st ) = T−t∑ k=1 ∑ st+k pπ ( st+k|st ) I ( st+k ∈ Sg ) ( 2 ) where pπ ( st+k|st ) the probability of reaching st+k from st . Complete proof is in appendix . Therefore , we can optimize success rate through setting the above { 0 , 1 } -binary reward function and adopting an undiscounted form of return . The problem is that this formulation falls into optimizing the undiscounted form of return and may have problems in training stability ( Xu et al. , 2018 ) .

Summary: This paper proposes an alternative surrogate objective for optimizing success rate in episodic tasks with bounded time horizon. Rather than optimize a discounted 0-1 loss (say with discount factor 0.99), the authors suggest to optimize the undiscounted 0-1 loss where reward is counted only for trajectories that do not contain loops. They call this a loop penalty, and show that it can work in 3 appropriate environments.