Faster Reinforcement Learning with Value Target Lower Bounding

1 INTRODUCTION . In temporal difference ( TD ) learning , the value function is adjusted toward its Bellman target , which is the reward of the current step plus the discounted value of the next state . This forms the basis of many state of the art reinforcement learning ( RL ) algorithms such as DQN ( Mnih et al. , 2013 ) , DDPG ( Lillicrap et al. , 2015 ) , TD3 ( Fujimoto et al. , 2018 ) , and SAC ( Haarnoja et al. , 2018 ) . The value of the next state is typically estimated using a “ bootstrapped value ” based on the value function itself , which is being actively learned during training . The bootstrapped values can be random or very inaccurate , especially at the initial stage of training . Consequently , the Bellman value targets as well as the learned value are usually far away from the optimal value . Naturally , this leads to the following idea : If we can make the value target closer to the optimal value , we may speedup TD learning . For example , we know that the optimal value is just the expected discounted return of the optimal policy , which always upper bounds the expected return of any policy . For episodic RL tasks , we could use the observed discounted return up to episode end from the training trajectories to lower bound the value target . This makes the new value target closer to the optimal value , when the empirical return is higher than the Bellman target . Will such a way of lower bounding the value target work : Will it still converge ? Will it converge to the optimal value ? Will it speed up value learning ? 2 THEORETICAL RESULTS FOR THE TABULAR CASE . For the tabular case , value target lower bounding converges to the same optimal value as the original Bellman value learning , and the proof is also straightforward . 2.1 BACKGROUND . In finite MDPs with a limited number of states and actions , a table can be used to keep track of the value of each state . Using dynamic programming algorithms such as value iteration , values are guaranteed to converge to the optimal through Bellman updates ( Chapter 4.4 ( Sutton & Barto , 2018 ) ) . Algorithm 1 : Bellman value iteration with value target lower bounding Data : Finite MDP p ( s′ , r|s , g , a ) , convergence threshold θ Result : State value v ( s ) 1 v ( s ) ← 0 ; 2 repeat 3 ∆← 0 ; 4 for each state s do 5 v ← v ( s ) ; 6 v ( s ) ←max ( f , maxa ∑ s′ , r p ( s ′ , r|s , g , a ) [ r + γv ( s′ ) ] ) ; 7 ∆← max ( ∆ , |v ( s ) − v| ) ; 8 end 9 until ∆ < θ ; The core of the algorithm is the Bellman update of the value function , B ( v ) : B ( v ) ( s ) : = max a ∑ s′ , r p ( s′ , r|s , g , a ) [ r + γv ( s′ ) ] ( 1 ) It is well known that the Bellman operator , B , is a contraction mapping over value functions ( Denardo , 1967 ) . That is , for any two value functions v1 and v2 , |B ( v1 ) − B ( v2 ) | ≤ γ|v1 − v2| for the discount factor γ ∈ [ 0 , 1 ) . This guarantees that any value function under the algorithm converges to the optimal value.1 2.2 VALUE TARGET LOWER BOUNDING CONVERGENCE THEOREM . Theorem 1 . Suppose the optimal value under the Bellman operator is B∞ ( v ) . For any value function f that lower bounds the optimal value , i.e . ∀s , f ( s ) ≤ B∞ ( v ) ( s ) , if we define the lower bounded Bellman operator asMf ◦ B ( v ) : = max ( B ( v ) , f ) , then ( Mf ◦ B ) ∞ ( v ) converges to B∞ ( v ) . A few things to note about the proof ( see Appendix A.1 ) . First , this only proves convergence , not contraction under the original ||v1 − v2||∞ metric . In the case of the Bellman operator , contraction shows that ∀v1 , v2 value functions , ||B ( v1 ) −B ( v2 ) ||∞ ≤ γ||v1− v2||∞ . Here , for value target lower bounding , there can be counter examples whereMf ◦B does not always contract in the original metric space for value functions . Here , convergence relies on the convergence of the Bellman value iteration and the existence of the fixed point v∗ . One difficulty caused by this change is that the stopping criterion in Algorithm 1 ( ∆ < θ ) no longer works , as we do not have access to the converged value during learning . This is perhaps not a serious concern in practice , as people often train algorithms for a fixed number of iterations or time steps . Second , based on the proof , the new algorithm is at least as fast as the original . When the lower bound actually improves the value target , i.e . f ( s ) > B ( v1 ) ( s ) , there is a chance for the convergence to be faster . Convergence is strictly faster when the lower bound f has an impact on the L∞ distance between the current value and the optimal value , i.e . it increases the value target for the states where the differences between the value target and the optimal value are the largest . Third , the lower bound function doesn ’ t have to be static during training . As long as there is a single f during each iteration , convergence property is preserved . Fourth , the theory works even when the underlying MDP is stochastic . Only the lower bounds based on empirical return introduced below require the MDP to be deterministic . 1See , for example , page 8 for the gist of the proof https : //people.eecs.berkeley.edu/ ˜pabbeel/cs287-fa09/lecture-notes/lecture5-2pp.pdf 3 EXAMPLE LOWER BOUND FUNCTIONS . We show a few cases where lower bound functions can be readily obtained from the training experience . Future work may investigate alternative lower bounds . 3.1 EPISODIC TASKS . In episodic tasks , discounted return is only accumulated up to the last step of an episode . In this case , we can wait until an episode ends , and compute future discounted returns of all time steps inside the episode . This discounted return is guaranteed to be a lower bound of the optimal value , if the environment is deterministic , i.e . the reward sequence can be repeated using the exact same sequence of actions . ( The behavior policy need not be deterministic , as long as the policy class contains the deterministic optimal policy . ) To make training efficient , we can compute and store such discounted returns into the replay buffer for each time step , and simply read them out during training . We call this variant lb-DR , short for lower bounding with discounted return . 3.1.1 EPISODIC WITH HINDSIGHT RELABELED GOALS . In goal conditioned tasks , one helpful technique is hindsight goal relabeling ( Andrychowicz et al. , 2017 ) . It takes a future state that is d time steps away from the current state as the hindsight / relabeled goal for the current state . When the goal is reached , a reward of 0 is given , otherwise a -1 reward is given for each time step . In this case , we know it took d steps to reach the hindsight goal , so the discounted future return is : Rd = ∑ i=0 , .. , d−1 −1γi =− 1 ( 1− γd ) / ( 1− γ ) ( 2 ) This calculation can be done on the fly as hindsight relabeling happens , requiring no extra space and very little computation . We call this variant lb-GD , short for lower bounding with goal distance based return . Additionally , we can also apply lb-DR and lb-GD together , with discounted return lower bounding ( lb-DR ) on the original experience and goal distance return lower bounding ( lb-GD ) on the hindsight experience , giving the lb-DR+GD variant , which was used by Fujita et al . ( 2020 ) independently . 3.2 NON-EPISODIC TASKS WITH POSITIVE REWARDS . When the task is continuing , without an episode end , discounted return needs to be accumulated all the way to infinity . This makes it difficult to lower bound the value if rewards can be negative . When rewards are always non-negative , one can still use the discounted return of the future n-steps to lower bound the value . Chapter 3.3 of Sutton & Barto ( 2018 ) has more details on episodic vs continuing tasks . 4 INTEGRATION INTO RL ALGORITHMS . 4.1 BACKGROUND . The value target lower bounds can be readily plugged into RL algorithms that regresses value to a target , e.g . DQN , DDPG or SAC . In these algorithms , the action value q ( s , a ) is learned through a squared loss with the target value y . In one step TD return , for a batch B of experience { s , a→ r , s′ } , the loss is : Lq : = ∑ ( s , a , r , s′ ) ∈B |q ( s , a ) − y|2 ( 3 ) In one step TD return , y is the one step TD return q̂ ( s , a , r , s′ ) : q̂ ( s , a , r , s′ ) : = r ( s , a ) + γq′ ( s′ , µ′ ( s′ ) ) ( 4 ) Here , q′ and µ′ are the bootstrap value and policy functions , typically following the value and policy functions in a delayed schedule during training . ( They are also called “ target value ” and “ target policy ” , and are very different from the “ value target ” y in this paper . ) 4.2 VALUE TARGET LOWER BOUNDING . With lower bounding , we replace the value target y with the lower bounded target : y ← max ( f , q̂ ( s , a , r , s′ ) ) = max ( f , r + γq′ ( s′ , µ′ ( s′ ) ) ) ( 5 ) This is subtly but importantly different from lower bounding the q value directly ( Oh et al. , 2018 ; Tang , 2020 ) : q ( s , a ) ← max ( f , q ( s , a ) ) , which stays overestimated if q ( s , a ) initially overestimates . This is the same as was done by Fujita et al . ( 2020 ) ( confirmed via personal communication ) . This way of simply lower bounding the value target does not require any tuning parameter , but one can always interpolate between these two value targets using a mixing weight α : y ← ( 1− α ) q̂ ( s , a ) + αmax ( f , q̂ ( s , a ) ) ( 6 ) A small α dampens the effect of the new value target , and may be desirable in practice when assumptions of the theorem can be violated , e.g . for non-deterministic tasks . See Appendix A.2 for an illustrative example of how value target lower bounding works in practice .

Summary: This paper proposes a new RL algorithm based on a modified Bellman backup equation. The main idea is to estimate the value of a state in multiple ways (using a Q function and using Monte Carlo returns) and then to take the maximum over these estimates. The paper shows that, if all estimators are lower bounds on the true value, then the proposed method converges to the optimal policy. Experiments confirm that the proposed method outperforms standard Q learning (i.e., with regular Bellman backups) on some tasks.