Faster Reinforcement Learning with Value Target Lower Bounding

1 INTRODUCTION . In temporal difference ( TD ) learning , the value function is adjusted toward its Bellman target , which is the reward of the current step plus the discounted value of the next state . This forms the basis of many state of the art reinforcement learning ( RL ) algorithms such as DQN ( Mnih et al. , 2013 ) , DDPG ( Lillicrap et al. , 2015 ) , TD3 ( Fujimoto et al. , 2018 ) , and SAC ( Haarnoja et al. , 2018 ) . The value of the next state is typically estimated using a “ bootstrapped value ” based on the value function itself , which is being actively learned during training . The bootstrapped values can be random or very inaccurate , especially at the initial stage of training . Consequently , the Bellman value targets as well as the learned value are usually far away from the optimal value . Naturally , this leads to the following idea : If we can make the value target closer to the optimal value , we may speedup TD learning . For example , we know that the optimal value is just the expected discounted return of the optimal policy , which always upper bounds the expected return of any policy . For episodic RL tasks , we could use the observed discounted return up to episode end from the training trajectories to lower bound the value target . This makes the new value target closer to the optimal value , when the empirical return is higher than the Bellman target . Will such a way of lower bounding the value target work : Will it still converge ? Will it converge to the optimal value ? Will it speed up value learning ? 2 THEORETICAL RESULTS FOR THE TABULAR CASE . For the tabular case , value target lower bounding converges to the same optimal value as the original Bellman value learning , and the proof is also straightforward . 2.1 BACKGROUND . In finite MDPs with a limited number of states and actions , a table can be used to keep track of the value of each state . Using dynamic programming algorithms such as value iteration , values are guaranteed to converge to the optimal through Bellman updates ( Chapter 4.4 ( Sutton & Barto , 2018 ) ) . Algorithm 1 : Bellman value iteration with value target lower bounding Data : Finite MDP p ( s′ , r|s , g , a ) , convergence threshold θ Result : State value v ( s ) 1 v ( s ) ← 0 ; 2 repeat 3 ∆← 0 ; 4 for each state s do 5 v ← v ( s ) ; 6 v ( s ) ←max ( f , maxa ∑ s′ , r p ( s ′ , r|s , g , a ) [ r + γv ( s′ ) ] ) ; 7 ∆← max ( ∆ , |v ( s ) − v| ) ; 8 end 9 until ∆ < θ ; The core of the algorithm is the Bellman update of the value function , B ( v ) : B ( v ) ( s ) : = max a ∑ s′ , r p ( s′ , r|s , g , a ) [ r + γv ( s′ ) ] ( 1 ) It is well known that the Bellman operator , B , is a contraction mapping over value functions ( Denardo , 1967 ) . That is , for any two value functions v1 and v2 , |B ( v1 ) − B ( v2 ) | ≤ γ|v1 − v2| for the discount factor γ ∈ [ 0 , 1 ) . This guarantees that any value function under the algorithm converges to the optimal value.1 2.2 VALUE TARGET LOWER BOUNDING CONVERGENCE THEOREM . Theorem 1 . Suppose the optimal value under the Bellman operator is B∞ ( v ) . For any value function f that lower bounds the optimal value , i.e . ∀s , f ( s ) ≤ B∞ ( v ) ( s ) , if we define the lower bounded Bellman operator asMf ◦ B ( v ) : = max ( B ( v ) , f ) , then ( Mf ◦ B ) ∞ ( v ) converges to B∞ ( v ) . A few things to note about the proof ( see Appendix A.1 ) . First , this only proves convergence , not contraction under the original ||v1 − v2||∞ metric . In the case of the Bellman operator , contraction shows that ∀v1 , v2 value functions , ||B ( v1 ) −B ( v2 ) ||∞ ≤ γ||v1− v2||∞ . Here , for value target lower bounding , there can be counter examples whereMf ◦B does not always contract in the original metric space for value functions . Here , convergence relies on the convergence of the Bellman value iteration and the existence of the fixed point v∗ . One difficulty caused by this change is that the stopping criterion in Algorithm 1 ( ∆ < θ ) no longer works , as we do not have access to the converged value during learning . This is perhaps not a serious concern in practice , as people often train algorithms for a fixed number of iterations or time steps . Second , based on the proof , the new algorithm is at least as fast as the original . When the lower bound actually improves the value target , i.e . f ( s ) > B ( v1 ) ( s ) , there is a chance for the convergence to be faster . Convergence is strictly faster when the lower bound f has an impact on the L∞ distance between the current value and the optimal value , i.e . it increases the value target for the states where the differences between the value target and the optimal value are the largest . Third , the lower bound function doesn ’ t have to be static during training . As long as there is a single f during each iteration , convergence property is preserved . Fourth , the theory works even when the underlying MDP is stochastic . Only the lower bounds based on empirical return introduced below require the MDP to be deterministic . 1See , for example , page 8 for the gist of the proof https : //people.eecs.berkeley.edu/ ˜pabbeel/cs287-fa09/lecture-notes/lecture5-2pp.pdf 3 EXAMPLE LOWER BOUND FUNCTIONS . We show a few cases where lower bound functions can be readily obtained from the training experience . Future work may investigate alternative lower bounds . 3.1 EPISODIC TASKS . In episodic tasks , discounted return is only accumulated up to the last step of an episode . In this case , we can wait until an episode ends , and compute future discounted returns of all time steps inside the episode . This discounted return is guaranteed to be a lower bound of the optimal value , if the environment is deterministic , i.e . the reward sequence can be repeated using the exact same sequence of actions . ( The behavior policy need not be deterministic , as long as the policy class contains the deterministic optimal policy . ) To make training efficient , we can compute and store such discounted returns into the replay buffer for each time step , and simply read them out during training . We call this variant lb-DR , short for lower bounding with discounted return . 3.1.1 EPISODIC WITH HINDSIGHT RELABELED GOALS . In goal conditioned tasks , one helpful technique is hindsight goal relabeling ( Andrychowicz et al. , 2017 ) . It takes a future state that is d time steps away from the current state as the hindsight / relabeled goal for the current state . When the goal is reached , a reward of 0 is given , otherwise a -1 reward is given for each time step . In this case , we know it took d steps to reach the hindsight goal , so the discounted future return is : Rd = ∑ i=0 , .. , d−1 −1γi =− 1 ( 1− γd ) / ( 1− γ ) ( 2 ) This calculation can be done on the fly as hindsight relabeling happens , requiring no extra space and very little computation . We call this variant lb-GD , short for lower bounding with goal distance based return . Additionally , we can also apply lb-DR and lb-GD together , with discounted return lower bounding ( lb-DR ) on the original experience and goal distance return lower bounding ( lb-GD ) on the hindsight experience , giving the lb-DR+GD variant , which was used by Fujita et al . ( 2020 ) independently . 3.2 NON-EPISODIC TASKS WITH POSITIVE REWARDS . When the task is continuing , without an episode end , discounted return needs to be accumulated all the way to infinity . This makes it difficult to lower bound the value if rewards can be negative . When rewards are always non-negative , one can still use the discounted return of the future n-steps to lower bound the value . Chapter 3.3 of Sutton & Barto ( 2018 ) has more details on episodic vs continuing tasks . 4 INTEGRATION INTO RL ALGORITHMS . 4.1 BACKGROUND . The value target lower bounds can be readily plugged into RL algorithms that regresses value to a target , e.g . DQN , DDPG or SAC . In these algorithms , the action value q ( s , a ) is learned through a squared loss with the target value y . In one step TD return , for a batch B of experience { s , a→ r , s′ } , the loss is : Lq : = ∑ ( s , a , r , s′ ) ∈B |q ( s , a ) − y|2 ( 3 ) In one step TD return , y is the one step TD return q̂ ( s , a , r , s′ ) : q̂ ( s , a , r , s′ ) : = r ( s , a ) + γq′ ( s′ , µ′ ( s′ ) ) ( 4 ) Here , q′ and µ′ are the bootstrap value and policy functions , typically following the value and policy functions in a delayed schedule during training . ( They are also called “ target value ” and “ target policy ” , and are very different from the “ value target ” y in this paper . ) 4.2 VALUE TARGET LOWER BOUNDING . With lower bounding , we replace the value target y with the lower bounded target : y ← max ( f , q̂ ( s , a , r , s′ ) ) = max ( f , r + γq′ ( s′ , µ′ ( s′ ) ) ) ( 5 ) This is subtly but importantly different from lower bounding the q value directly ( Oh et al. , 2018 ; Tang , 2020 ) : q ( s , a ) ← max ( f , q ( s , a ) ) , which stays overestimated if q ( s , a ) initially overestimates . This is the same as was done by Fujita et al . ( 2020 ) ( confirmed via personal communication ) . This way of simply lower bounding the value target does not require any tuning parameter , but one can always interpolate between these two value targets using a mixing weight α : y ← ( 1− α ) q̂ ( s , a ) + αmax ( f , q̂ ( s , a ) ) ( 6 ) A small α dampens the effect of the new value target , and may be desirable in practice when assumptions of the theorem can be violated , e.g . for non-deterministic tasks . See Appendix A.2 for an illustrative example of how value target lower bounding works in practice .

Summary: This paper proposed value target lower bounding as a simple modification to the Bellman operator that intends to improve convergence speed. It proves that using such a lower bound in the Bellman backup does not change the fixed point in the tabular setting. The paper then proposes two instantiations of particular value target lower bounds: first by using the return in deterministic episodic MDPs and second by hindsight relabeling in goal-directed tasks. Finally, the paper offers some experiments using the proposed lower bounds.