Title: A Long $N$-step Surrogate Stage Reward for Deep Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep reinforcement learning

Reward Estimation
Abstract: We introduce a new stage reward estimator  named the long $N$-step surrogate stage (LNSS) reward for deep reinforcement learning (RL). It aims at mitigating the high variance problem, which has shown impeding successful convergence of learning, hurting task performance, and hindering applications of deep RL in continuous control problems. In this paper we show that LNSS, which utilizes a long reward trajectory of  rewards of future steps, provides consistent performance improvement measured by average reward, convergence speed, learning success rate,and variance reduction in $Q$ values and rewards.  Our evaluations are based on a variety of environments in DeepMind Control Suite and OpenAI Gym  by using  LNSS in baseline deep RL algorithms such as DDPG, D4PG, and TD3. We show  that LNSS reward has enabled good results that have been challenging to obtain by deep RL previously. Our analysis also shows that  LNSS exponentially reduces the upper bound on the variances of $Q$ values from respective single-step methods.

A Long N -step Surrogate Stage Reward for Deep Reinforcement Learning

Abstract

We introduce a new stage reward estimator named the long N -step surrogate stage (LNSS) reward for deep reinforcement learning (RL). It aims at mitigating the high variance problem, which has shown impeding successful convergence of learning, hurting task performance, and hindering applications of deep RL in continuous control problems. In this paper we show that LNSS, which utilizes a long reward trajectory of future steps, provides consistent performance improvement measured by average reward, convergence speed, learning success rate, and variance reduction in Q values and rewards. Our evaluations are based on a variety of environments in DeepMind Control Suite and OpenAI Gym by using LNSS piggybacked on baseline deep RL algorithms such as DDPG, D4PG, and TD3. We show that LNSS reward has enabled good results that have been challenging to obtain by deep RL previously. Our analysis also shows that LNSS exponentially reduces the upper bound on the variances of Q values from respective single-step methods.

1 Introduction

Reinforcement learning (RL) shows great promise as a theory of learning in complex dynamic tasks [1, 2]. For continuous control problems involving high-dimensional states and control spaces, several deep reinforcement learning (DRL) algorithms, such as deep deterministic policy gradient (DDPG) [3], proximal policy optimization (PPO) [4], Soft actor critic (SAC) [5], D4PG [6] and twin delayed DDPG (TD3) [7], have demonstrated their great potential.

However, the high variance problem [8, 9] in DRL still poses great challenges for it to be effective in solving complex contentious control problems. On one hand, this problem is intrinsic to the trial and error nature of reinforcement learning, which causes low data efficiency, slow learning, or even instability of learning and poor task performance [10, 11]. On the other hand, high variances are often caused by several factors, such as variances in value estimation [12], variances resulting from different random seeds [10], and variances due to noisy stochastic reward or corrupted sparse reward [13]).

Different approaches have been proposed to address the high variance problem, from the original TD( λ ) algorithm to Q ( σ ) [14] as a generalization of TD( λ ) [15]. Rollout methods [16, 17, 18] have enabled the success of some of the most important milestone cases such as Backgammon [17] and AlphaGo [19] in machine learning history. Although these algorithms has been demonstrated in discrete state and control problems, variants of rollout are believed to be effective also in continuous control tasks. Modern DRL algorithms such as Rainbow [20] integrate n -step learning on top of DQN and results in a speed-up in learning and an improvement of the final performance on Atari 2600 games. D4PG [6] uses the same n -step methods to update target values, and have been shown effective in continuous control tasks. PPO [4] uses a truncated version of generalized advantage estimation (GAE) to update policy which reduces policy variance [21]. However these methods do

not explicitly account for noisy rewards where such uncertainties have been reported to degrade the model/controller performance [22].

In this paper, we propose a new algorithm, namely the long N -step surrogate stage (LNSS) reward estimator. LNSS aims at providing variance reduction for DRL methods in continuous control problems, even under noisy reward signals or corrupted sparse reward. LNSS is simple, and can be easily piggybacked on DRL algorithms. It is based on a surrogate reward r ′ obtained from a trajectory of rewards of future " N -step"s. In turn, r ′ is used in Q value update where the target function is then of the same form as that in single-step TD methods. Using a long " N -step"s of the reward sequence containing environment information, it is expected to be more advantageous to learning than using a short trajectory of n -step of the reward sequence.

We show theoretically that the LNSS method exponentially reduces the upper bound on the variance of Q value from a respective single-step method. We then conduct extensive experiments in selected environments in DMC and GYM where most current DRL algorithms such as DDPG, D4PG, and TD3 that typically struggle to obtain good results [23]. We consider uniform random noises of different magnitudes to show how LNSS can benefit a variety of DRL algorithms while the performances of other methods degrade significantly. We further show how LNSS can still be effective in environments where only sparse rewards are available.

Contributions . 1) We introduce a new, simple yet effective surrogate reward r ′ that interpolates between the single-step TD methods and Monte-Carlo (MC) methods [15], to trade off the strengths from the two extremes. 2) We theoretically show the proposed method helps reduce the upper bound on the variance of Q value with a long trajectory of rewards of future " N -step"s. 3) Extensive experiments on DMC and GYM benchmarks show that LNSS robustly benefits the learning performance of a variety of DRL algorithms even under reward noise or sparse reward. Mainly, the LNSS consistently reduces variances in the learned value function and achieves robust results from different random seeds in several different types of continuous control problems. Please check our code

2 Related Work

The " n -step" methods . Here, as in the literature, n -step methods refer to those involving a trajectory of rewards of multiple future steps in DRL algorithms for speeding up learning and reducing variance [24, 25, 6]. They usually proceed as follows. First, collect n -step returns to form a discounted sum of rewards [26, 27]. Then, perform n -step return backup to update the value function [15]. It was empirically suggested that n -step methods perform better than single-step methods since n -step returns propagate rewards to relevant state-action pairs more efficiently [24, 25].

Using longer trajectories with a larger n is expected to improve learning performance [28]. However, to date, the n -step DRL methods have only been demonstrated by using relatively small n . Rainbow [20] uses n = 1 , 3 , 5 to evaluate performance of 57 Atari 2600 games and reported n = 3 as the best-performing reward trajectory length. D4PG [6] uses n = 1 , 5 to evaluate a variety of continuous control tasks in DMC and results show that n = 5 performs uniformly better than others. However, 3 and 5 are relative small for complex tasks which usually have a maximum of 1000 steps in an episode.

There are potentially two reasons that prevent n -step methods from using larger length n . One is that the longer the n -step return, the larger reward scale that can cause saturation and inefficiency in learning for most DRL algorithms [10]. The second reason is that the longer the n -step return, the smaller discount factor ( γ n ) in n -step backup [15]. Based on previous studies [29], variations in the scale of the discount factor directly causes uncertainty to the RL performance. New algorithmic ideas are needed for DRL algorithms to benefit from longer reward bootstrapping.

Reward estimation . Reward estimation aims at reducing sample complexity and learning variance as reward variances may be introduced from using random seeds in training [9], from sensor noises in real-world applications [8] and from corrupted sparse rewards [13]. Several approaches have been proposed for reward estimation. Small backups [30], instead of directly using an actual reward, use multi-step imaginary rollouts as a predicted reward. Model-based policy search with Bayesian neural network [31] finds expected cost by averaging over multiple virtual rollout. The λ -prediction [32] uses an extra network to predict reward and the training of which may introduce additional variance. Model-based value expansion [33] uses an imaginary reward from a model to estimate the values of state value functions. To reduce learning variance resulted from noisy rewards, regression reward

estimator [13] expects the estimated reward to reduce the variance propagated to the value function. Unbiased reward estimator [22] estimates a surrogate reward by learning a reward confusion matrix which estimates true reward from noisy reward. However, all these estimation methods require extra models (either in the form of neural networks, environment transitions, or reward noise model), which imposes overhead on learning and introduces estimation error. Additionally, how to effectively and efficiently integrate these reward estimates into DRL algorithms is not clear, or at least not straightforward.

LNSS . In contrast, our method does not introduce any new hyper parameters into learning as we still use the same form of reward signals but only to a longer horizon of " N -step"s. The LNSS method can be piggybacked on any DRL algorithms directly. Comparing to n -step methods, by using our surrogate reward r ′ to perform single-step updates, we can benefit from n -step returns without changing the reward scale. This frees LNSS from a limited reward horizon to a much longer length N .

Selecting DRL algorithms . First, note that few existing n -step methods have been successfully applied to different DRL algorithms with consistent learning variance reduction. We selected DDPG, D4PG, and TD3 as base DRL algorithms, with which our LNSS is to be piggybacked on for the following considerations. Among all DMC based benchmark results [23], distributional method D4PG [6] outperforms other DRL algorithms such as DDPG [3], and TD3 [5]. On the other hand, TD3, DDPG have relatively good performance in simple tasks such as walker but struggled to achieve good results in harder DMC tasks such as humanoid-walk and fish-swim. We thus use the three chosen DRL algorithms to systematically test the idea of LNSS piggybacked on promising DRL algorithms in complex tasks, even in noisy reward environments.

3 Background

Reinforcement Learning . We consider a reinforcement learning agent interacts with its environment in discrete time. At each time step k , the agent observes a state s k ∈ S and select an action a k ∈ A based on its policy π : S → A , namely, a k = π ( s k ) , and receives a scalar reward r ( s k , a k ) ∈ R (use r k as short hand notation).

Evaluation of a policy π is performed using the expected return after taking an action a k based on state s k following the policy π :

with 0 < γ < 1 . The two common approaches (single-step and n -step) to update the Q value for a policy π are as follows.

Single-step RL Algorithms . RL algorithms rely on using TD error or residual of the Bellman equation to update the state-action Q value for a policy π , as described below,

This equation provides the basis for single-step RL methods. It is also known as a backup operation since it transfers information from one step ahead back to the current step.

" n -step" RL algorithms . Using n -step rewards for faster reward propagation in RL has long been investigated [26, 14, 6, 20]. In n -step methods, the value function Q π ( s k , a k ) update using an n -step return is based on the following,

The n -step return is expected to help agents learn more efficiently by affecting multiple state action pairs within one update and gain information over the n -step horizon.

More details on specific modern DRL algorithms such as the single-step methods (DDPG and TD3) and the n -step methods (D4PG and PPO) can be found in Appendix D.

4 Long N -step Surrogate Stage (LNSS) Reward

In this section, we introduce LNSS based on infinite horizon discounted reward formulation of reinforcement learning. Given a reward trajectory of N steps from time step k , let G ( s k : k + N -1 , a k : k + N -1 ) ∈ R (use G k as short hand notation) denote the discounted N -step reward, i.e.,

where r t is the t th stage reward and t is from k to k + N -1 . In LNSS, we introduce r ′ k as a surrogate stage reward in place of r k in Equation (2). To determine r ′ k , we treat it as a weighted average of the N -step reward sequence, namely

We then propose the surrogate stage reward r ′ k to be

This surrogate stage reward r ′ k as formulated in Equation (6) relies on a discounted reward of an N -step horizon, from time step k to step ( k + N -1) , from the stored experiences into a temporary replay buffer D ′ . For a training episode of T steps [0 , 1 , 2 ..., T ] , the D ′ is a moving window of size N from the initial state s 0 until the terminal state s T . As a result, when there is a sufficient number (i.e., N ) of reward samples, LNSS computes the surrogate reward r ′ k from below,

Note that, when the reward is estimated at the beginning or toward the end of a long trial, less than N reward samples are available for estimating r ′ k . A simple adjustment is given in Appendix C.

Once r ′ k is obtained, r ′ k and state action pairs ( s k , a k , s k +1 ) will append as a new transition ( s k , a k , r ′ k , s k +1 ) stored into the memory buffer D .

Note that many DRL algorithms [4, 20, 6] use distributed learning procedure to accelerate experience sample collection. We use the same technique to speed up sampling experiences. Then a DRL algorithm is ready to update the Q value and the respective policy based on mini-batch data from the memory buffer. In general form, we have

where i is iteration number. Putting the above two equations together, we have

Remark 1 . 1) Different from n -step methods, in our LNSS, N is the number of steps for accumulating rewards in Equation (6. We still perform a single-step update as in Equation (8). This allows us to prevent steep discounts of γ n in n -step backup when using longer n -step returns. As a result, LNSS can effectively use large N at a scale of 100 while D4PG (Barth-Maron et al., 2018) uses n at a scale of 5 steps, and the same for Rainbow (Hessel et al., 2018).

2) LNSS can also be effective in a sparse reward environment. With a longer reward trajectory of N future steps, LNSS continuously provides a reward estimate as feedback to the agent by assigning credits progressively backward from the time of achieving the desired states. That is to say that LNSS

has turned a sparse reward into a dense reward. Thus, this helps the agent effectively and efficiently learn the task. However, the single-step method does not have the ability to provide any guidance or feedback to the learning agent until reaching a goal.

3) The Q value for policy π due to LNSS differ from that in Equation 1. The former aims to maximize the sum of a sequence of surrogate rewards which are the weighted averages of the sequence of N -steps of the original rewards. As such, goal-related information can be progressively propagated back from the goal state to the current state. This prevents the agent from getting trapped from reaching its intended goal and thus being stuck in an undesirable set of states due to a lack of feedback.

5 Variance Analysis

We now analyze the behavior of an actor-critic RL and our LNSS actor-critic RL. We consider the infinite horizon discounted reward formulation of RL (with 0 < γ < 1 ). Specifically, we show that the upper bound on the variance in Q value due to LNSS differs by an exponential factor from that of a single step actor-critic RL (AC). As this upper bound reduces exponentially as N increases, it suggests significant variance reduction by using LNSS from using single step reward.

We first represent the Q values using single step reward r k in Equation (2) and using surrogate reward r ′ k from LNSS in Equation (8), respectively as follows,

Lemma 1 . Assume { r k } is IID and drawn from the memory buffer D . Let Q i ( s k +1 , a k +1 ) in Equation (8) be the i -th approximated return to solve Equation (1). We then have the following,

̸

Theorem 1 . Consider the variances of two Q value sequences, denoted as Q i and Q i , in Equation (10) and Equation (11), which are obtained respectively from a single step method and an LNSS method. Additionally, assume that Q 0 = Var [ Q 0 ] = 0 and Q 0 = Var [ Q 0 ] = 0 . Let the IID reward { r k } and { r ′ k } be drawn from the memory buffer D . Assume the variance of { r k } is upper bounded by a finite positive number B , i.e., Var [ r k ] ≤ B . Further define a constant ψ as,

Then the upper bounds of the variances of the two Q value sequences, Var [ Q i +1 ] and Var [ Q i +1 ] , are respectively described below,

Proof. Both proofs of Lemma 1 and Theorem 1 are given in Appendix A.

Remark 2 . We now provide some insights based on the variance analysis above.

1) Given ψ in Equation (14), i.e., ψ = ( γ -1 γ N -1 ) 2 ( γ 2 N -1 γ 2 -1 ) , it follows that for large N , ψ = ( γ -1) 2 ( -1 2 ) = 1 -γ .

2) Furthermore, by the following identifies, γ 2 -1 = ( γ -1)( γ +1) and γ 2 N -1 = ( γ N -1)( γ N +1) , we have that ψ = ( γ -1 γ +1 )(1 + 2 γ N -1 ) . Therefore, ψ decreases exponentially (refer to Figure 1). 3) From inspecting Equations (15) and (16), we can see a clear advantage of using long " N -step" in LNSS over the typical reward r k .

6 Experiments and Results

In this section, we first show how LNSS can benefit learning by using a simple Maze environment. To provide insight on the effect of using LNSS, we compare the performance of the original Q -learning with the one that the stage reward is replaced by LNSS reward.

We then provide a comprehensive evaluation of our proposed LNSS piggybacked on three promising DRL algorithms (DDPG, D4PG, TD3) by measuring their performance on several benchmarks in DMCand GYM (for results on GYM, please refer to Appendix E.5). Details of the implementation, training, and evaluation procedures are provided in Appendix B. In reporting evaluation results below, we use the following short-form descriptions.

'Base': the original DRL algorithms (TD3, D4PG, DDPG).

'LNSS": LNSS piggybacked on the respective DRL algorithms where N =100 unless otherwise specified.

'n5": the DRL algorithms with an " n -step" implementation as in Equation (3) with n = 5 .

Our evaluations aim to quantitatively address the following questions:

Q 1 . How is LNSS different from original reward? and How it helps learning?

Q 2 . How does LNSS improve Base method?

Q 3 . How does LNSS compare with previous n -step method ( n 5 )?

Q 4 . Is LNSS robust enough to compensate for stochastic reward?

Q 5 . Does LNSS improve Base methods under a sparse reward setting?

Q 6 . How does LNSS reduce the variance of estimated Q values?

Q 7 . How different N in LNSS affects performance?

Q-learning Single Step

Q-learning LNSS

WithPenalty

WithOutPenalty

Epois Reward

0

Epois Reward

1.00

-2

0.75

So

S1

S2

-4

0.50

0

200

400

600

0

200

400

600

S3

Episode

Episode

Stage Reward

1.0

Stage Reward

1.0

0.5

0.5

S5

0.0

0.0

0

2

4

0

2

4

State

State

6.1 The Simple Maze Problem

The right panel of Figure 2 shows the Maze environment where S 0 is the initial state, S 5 is the goal state, and black blocks are walls. The unique optimal policy is {right,right,down,down,left}. We consider two problem formulations:

Without Penalty: the agent only receives a reward 1 upon reaching goal state S 5 . For all other states, the agent receives 0's.

With Penalty: the agent only receives a reward 1 upon reaching goal state S 5 . For all other states, the agent receives -0.1 per stage as a penalty.

Q 1 LNSS takes future rewards into formulating the surrogate stage reward to guide a RL agent in reaching the goal state. As Figure 2 lower-left panels show, the stage reward from LNSS is more

informative as LNSS provides sequences of gradual and smoother rewards to be used in Q -learning. As a result, from the upper-left panels, this informative guidance from LNSS significantly enhances the performance in terms of episode rewards. The effect is even more pronounced when dealing with penalties.

From Table 1, with the original reward, the Q value is the sum of discounted expected rewards as in Equation 4. The farther away the goal state, the smaller the Q values are as a result of discounts over the horizon. When learning under the environment of using penalty rewards, the Q values in States S 0 , S 1 , S 2 show that they are even farther away from the goal state value. The Q values can not be effectively updated due to a lack of informative feedback signal, and thus, causing the agent to get stuck and cease learning. In comparison, with the LNSS reward, goal-related information can be propagated back from the goal state to the initial state. This guidance plays a crucial role in assisting agents to overcome situations where they might otherwise get stuck.

6.2 Main Results

Evaluation results in Table 2 are performed using default setup of Acrobot SwingUp, Humanoid Walk, Fish Swim, and Finger TurnHard in DMC. In the Table 2, "Success" is shorthand for success rate, "Avg. Rwd" stands for average reward, and "Rank" (%) is the "percent of reward difference",

which is ( the average reward of the evaluated algorithm over that of the top performing algorithm 1), the closer it is to 0 the better. Note that, in computing the success rate, only those trials that have achieved a reward of at least 50 are accounted for as successes.

The results are obtained at different noise levels, based on the last 50 evaluations of 10 different random seeds (same for all compared algorithms). Best performances are boldfaced for average reward (Avg. Rwd).

Q 2 LNSS improves Base methods. The learning curves for the four continuous dense reward benchmark environments are shown in Figure 3. Quantitative comparisons to the baselines are summarized in the first section of Table 2 (noise = 0). Overall, LNSS (solid lines) outperforms their respective base methods (dash lines) in terms of averaged reward (Awg.Rwd), learning speed, and success rate for all three DRL algorithms. Among the measures, the success (learning) rate is to address the random initialization challenge caused by random seeds [10]. LNSS method helps its Base method to achieve a 100% learning success rates whereas DDPG and TD3 Base method struggle with Humanoid Walk and Finger TurnHard. Besides, according to the "Rank" measure in Table 2, LNSS helps enhance the performances of almost all Base methods up to the top-performing algorithm in each category in all the tested environments. Detailed plots of pair-wise comparison for without noise condition are provided in appendix E.1.

D4PG-BASE

D4PG-LNSS

I III

DDPG-BASE

DDPG-LNSS

DMCAcrobotSwingUp

DMCHumanoidWalk

DMCFishSwim

Episode Reward

500

1000

400

750

250

200

500

250

0

0

0

DMCFinger TurnHard

DMCCheetahRunSparse

DMCCartpoleSwingupSparse

1000

1000

1000

Episode Reward

750

750

750

500

500

500

250

250

250

0

0

0

0

2000004000006

600000

800000

0

200000400000

6000008

800000

0

200000400000

600000

800000

Q 3 LNSS outperforms previous " n -step" methods ( n 5 ). The main improvement of LNSS over n 5 are reflected in improved success rate, averaged reward, and "Rank". This improvement can be quantitatively viewed in the first section of Table 2 (noise = 0). LNSS consistently enables the Base methods to achieve 100% learning success rate. However, the n 5 method may even cause some learning to fail such as TD3 in Humanoid Walk and D4PG in Finger TurnHard. Additionally, according to "Rank" in Table 2, LNSS methods enable all cases to reach performances near those from the best algorithms. However, only TD3n 5 has resulted in slightly better "Rank" in Fish Swim only for without noise condition while all other n 5 methods are below LNSS or even below its Base such as TD3n 5 for humanoid walk and DDPGn 5 for Fish swim. We also observe that LNSS has improved the average reward and lowered the standard deviation for all Base algorithms over those by the n 5 method.

Q 4 LNSS is robust under noisy rewards. We performed the same set of evaluations in Table 2 in DMC experiments under random uniform reward noise settings with magnitudes at 1% and 10% of the maximum DMC step reward (1). As results in the appendix E.2 (noise level 1%), Appendix E.3 (Noise level 10%), and the last two sections in Table 2 (noise = 1% and 10%) show, at larger noise level, all algorithms have degraded performance with lower mean averaged reward, larger standard division, and more negative "Rank". However, LNSS consistently outperformed Base and n 5 methods in terms of all measures for all three DRL algorithms. It is important to point out that we observe LNSS methods robustly improve the success rate with 100% for all environments. However, we observe that both Base and n 5 methods suffer greatly due to the increased noise level such as

in Humanoid Walk and Finger Turn Harde environments. As Equation (3) shows the performance degradation for n 5 methods are due to accumulated noise when collecting the discounted sum of returns. This accumulated noise will induce a much larger approximation variance and result in degraded performance.

Q 5 LNSS is effective in environments with sparse rewards. For setting up sparse rewards in Cheetah environments, refer to appendix B. As Cheetah Sparse and Cartpole Sparse result in Figure 3. One can observe that LNSS enables all DRL algorithms successfully learn all three sparse reward tasks with converged episode rewards greater than 800. However, due to the limited feedback from sparse rewards, all base methods struggle to learn and even fail to learn (e.g, Cheetah Run). Moreover, it is important to point out that LNSS helps improve the learning speed over the base methods. This is expected as a long future reward trajectory of N -step is able to provide continuous reward feedback to the agent by assigning credits progressively backward from the time of achieving the desired states. In contrast, a single-step method does not provide any guidance or feedback to the learning agent until reaching the goal.

Q 6 LNSS helps reduce the variance of Q value estimation . For this evaluation, we let coefficient of variation of Q value (in percentage) as cv ( Q ) = std ( Q ) Q % . In Figure 4, we show cv ( Q ) every 2e5 steps where as an example, we show TD3-LNSS outperforms TD3-Base and TD3-n5 under 1% noise in rewards. Similar results are obtained for DDPG and D4PG classes of algorithms and shown in appendix E.4. Also shown in the same Appendix are respective results at different noise levels. Results (Figure 12 to Figure 20 in appendix E.4) also show that for each LNSS augmented DRL algorithms, it has resulted in improved performance in cv ( Q ) to near the best performance among all DRL algorithms. Figure 4 provides an empirical validation for our theoretical result on bounds of variances of the Q value, namely, with large LNSS parameter N or at the late stage of learning, Q value variances become smaller as the upper bound of Q value variance decreases significantly according to Equation (14) in Theorem 1.

TD3-n5

DMCAcrobotSwingUp

DMCHumanoidWalk

DMCFishSwim

DMCFinger TurnHard

20

10

20

20

CV(Q) (%)

5

10

10

10

0

0

200000400000600000800000

0

200000400000600000800000

0

200000400000600000800000

0

200000400000600000800000

Q 7 . Effects of different length N in LNSS. In Figure 5, we show the effect of different choices of N ( N = 5 , 10 , 20 , 50 , 100) in our proposed LNSS on top of TD3. (Similar results hold for D4PG and DDPG which can be found in appendix E.7.) Comparisons are performed using the same set of hyperparameters except a varying N . Figure 5 depicts performance curves for DMC tasks. The LNSS with N = 50 and N = 100 outperform LNSS with N = 5 , 10 in terms of final reward and learning speed. The results corroborate what we expect from our theoretical analysis (please refer to Figure 1 in the paper). We can see a clear benefit of using large N ( N = 50, and 100 ) over small( N = 5, and 10).

Additionally, in Figure 6, we show how cv ( Q ) (coefficient of variation of Q ) varies with different N . Among the three N values, N = 100 outperforms others. Also, we observe that, in more complex environment, a longer N is favorable. Notice that, for cartpole sparse environment, there is little difference among different N values. However, in Finger TurnHard and Humanoid Walk, the cv ( Q ) values of Base cases and N = 5 cases increase dramatically. However, the cv ( Q ) values for N = 50 and N = 100 change only slightly. Figure 6 provides an empirical validation for our theoretical analysis on variance discount factor ψ in Figure 1 that the longer the N , the more reduction in variance of Q value estimation.

N=5

N=10

N=100

DMCHumanoidWalk

DMC CartpoleSwingUpSparse

1000

800

300

800

Epois reward

600

200

600

400

400

100

200

200

0

0

0

0

200000

400000

600000

800000

0

200000

400000

600000

800000

0

125000

250000

375000

500000

DMCHumanoidWalk

DMCFingerTurnHard

DMCCartpoleSwingupSparse

50

12

6.0

40

9

4.5

ariance

30

6

3.0

20

Q

10

3

1.5

0

0

0.0

BASE

N5

N50

N100

BASE

N5

N50

N100

BASE

N5

N50

N100

Figure 6: cv ( Q ) (coefficient of variaiton of Q value, the lower the better) for the 3 tasks in DMC by LNSS with different N ( N = 5 , 50 , 100 ) in Equation (6).

6.3 Limitation of This Study.

Here we introduce a new multi-step method, the LNSS, to effectively utilize longer reward trajectories of future steps than those used in existing methods to estimate the Q value with an aim of reducing variances in learning. By piggybacking LNSS on top of TD3, DDPG, and D4PG, we demonstrate improved performance in terms of final reward, learning speed, and variance reduction in Q value. However, there exist two limitations. 1) The LNSS requires positive semi-definite reward values in a long trajectory. Large negative reward may wash out the significance of good positive rewards. However, this limitation can be easily fixed by elevated reward stored in the temporary buffer D ′ by a positive constant, and lower bounded it by 0. 2) Early termination is another limitation during implementation, which mostly affects LNSS in GYM environments (Hopper, Humanoid, Walker2d). Different from DMC tasks, GYM allows an environment to terminate before reaching the maximum time steps. As such, in early training stage, only 10 or 20 steps may be used to compute LNSS which diminishes the power of large N (such as 50 or 100 ). To resolve this limitation, re-programming task settings in the original GYM environments is required.

7 Discussion and Conclusion

1) In this work, we introduce a novel " N -step" method, the LNSS which utilize longer reward trajectories of future steps than those used in existing methods to estimate the Q value with an aim of reducing variances in learning. It is easy to implement, as shown in the paper, and it can be easily piggybacked on policy gradient DRL algorithms. It has been shown consistently outperform respective Base and short n -step algorithms in solving benchmark tasks in terms of performance score, learning success rate, and convergence speed. 2) We provide a theoretical analysis to show that LNSS reduces the upper bound of variance in Q value exponentially from respective single step methods. 3) We empirically demonstrate the performance of LNSS piggybacked on TD3, DDPG, and D4PG in a variety of benchmark environments that have been challenging for existing methods to obtain good results. 4) We show LNSS can provide consistent performance improvement under various reward noise settings, and for sparse rewards. Our results suggest that LNSS is a promising tool for improving learning speed, learning performance score, and reducing learning variance. Further investigation on how to maximize the benefit of selecting an optimal reward length N , and how to take advantage of a different n in Q value update are exciting questions to be addressed in future work.

8 Acknowledgment

This research was supported in part under NSF grants #1808752 and #2211740.

References

Wentao Liu, Junmin Zhong, Ruofan Wu, Bretta L Fylstra, Jennie Si, and He Helen Huang. Inferring human-robot performance objectives during locomotion using inverse reinforcement learning and inverse optimal control. IEEE Robotics and Automation Letters , 7(2):2549-2556, 2022.

Ruofan Wu, Junmin Zhong, Brent Wallace, Xiang Gao, He Huang, and Jennie Si. Human-robotic prosthesis as collaborating agents for symmetrical walking. Advances in Neural Information Processing Systems , 35:27306-27320, 2022.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.08617 , 2018.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning , 110(9):2419-2468, 2021.

Johan Bjorck, Carla P Gomes, and Kilian Q Weinberger. Is High Variance Unavoidable in RL? A Case Study in Continuous Control. arXiv preprint arXiv:2110.11222 , 2021.

Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning , pages 1329-1338. PMLR, 2016.

Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research , 5(9), 2004.

Joshua Romoff, Peter Henderson, Alexandre Piché, Vincent Francois-Lavet, and Joelle Pineau. Reward estimation for variance reduction in deep reinforcement learning. arXiv preprint arXiv:1805.03359 , 2018.

Kristopher De Asis, J Hernandez-Garcia, G Holland, and Richard Sutton. Multi-step reinforcement learning: A unifying algorithm. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Dimitri P Bertsekas. Rollout algorithms for discrete optimization: A survey. Handbook of Combinatorial Optimization, D. Zu and P. Pardalos, Eds. Springer , 2010.

Gerald Tesauro. TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural computation , 6(2):215-219, 1994.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140-1144, 2018.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence , 2018.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015.

Jingkang Wang, Yang Liu, and Bo Li. Reinforcement learning with perturbed rewards. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 6202-6209, 2020.

Fabio Pardo. Tonic: A deep reinforcement learning library for fast prototyping and benchmarking. arXiv preprint arXiv:2011.07537 , 2020.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. PMLR, 2016.

Yonathan Efroni, Gal Dalal, Bruno Scherrer, and Shie Mannor. Beyond the one-step greedy approach in reinforcement learning. In International Conference on Machine Learning , pages 1387-1396. PMLR, 2018.

Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.

Jing Peng and Ronald J Williams. Incremental multi-step q-learning. In Machine Learning Proceedings 1994 , pages 226-232. Elsevier, 1994.

J Fernando Hernandez-Garcia and Richard S Sutton. Understanding multi-step deep reinforcement learning: a systematic study of the dqn target. arXiv preprint arXiv:1901.07510 , 2019.

Harm Van Seijen, Mehdi Fatemi, and Arash Tavakoli. Using a logarithmic mapping to enable lower discount factors in reinforcement learning. Advances in Neural Information Processing Systems , 32, 2019.

Harm Van Seijen and Richard S Sutton. Efficient planning in MDPs by small backups. In Proc. 30th Int. Conf. Mach. Learn , volume 28. Citeseer, 2013.

Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi-Velez, and Steffen Udluft. Learning and policy search in stochastic dynamical systems with bayesian neural networks. arXiv preprint arXiv:1605.07127 , 2016.

David Silver, Hado Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David Reichert, Neil Rabinowitz, Andre Barreto, et al. The predictron: End-toend learning and planning. In International Conference on Machine Learning , pages 3191-3199. PMLR, 2017.

Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101 , 2018.

Alfredo V Clemente, Humberto N Castejón, and Arjun Chandra. Efficient parallel methods for deep reinforcement learning. arXiv preprint arXiv:1705.04862 , 2017.

Yinlong Yuan, Zhu Liang Yu, Zhenghui Gu, Yao Yeboah, Wu Wei, Xiaoyan Deng, Jingcong Li, and Yuanqing Li. A novel multi-step Q-learning method to improve data efficiency for deep reinforcement learning. Knowledge-Based Systems , 175:107-117, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article considers a new way to formulate the n-step rewards in deep reinforcement learning. Specifically, while the normal n-step reward discount the Q-value by \\gamma^n, the authors propose to only discount the Q-value by \\gamma while using the normal n-step reward (times a constant) for r. For a simplified model, the authors mathematically prove that this decreases an upper bound of the variance of Q-values. Empirically, the authors add the proposed method to three standard RL algorithms (DDPG, TD3, D4PG) and evaluate them on DMC and GYM. For four tasks in DMC, the proposed method improves the performance of the baseline algorithms. For GYM, the proposed methods improve TD3 on a task.\n\nEDIT: I have read the rebuttal and will keep my score but decrease my certainty.\n\nSTRENGTHS:\nThe paper proposes a natural and simple method. It should be simple to implement and could conceivably improve performance.\n\nThe authors show strong empirical performance for the proposed method.\n\nWEAKNESSES:\nThe authors show improvements on a few tasks for a few algorithms. It is unclear how the environments are selected, and there is some risk of cherry-picking. Furthermore, it is not clear if the baseline algorithms and their implementations are SOTA. So it’s not really clear if the proposed methods would improve SOTA RL, they might just improve some mediocre baselines.\n\nThe assumptions of theorem 1 are not clear. Furthermore, I think the interpretation of the results is misleading. The authors show that an upper bound on the variance of the Q-values decreases with the proposed method. However, a difference in an upper bound does NOT imply that the actual variance decreases. That is only true if the upper bound is reasonably tight.\n\nQUESTIONS:\nAre the experiments all state-based? If so, could you add experiments when learning from pixels?  \n\nHow were the environments selected?\n\nWill the upper bound in Theorem 1 be tight?\n\nCan you clarify the assumptions in Theorem 1?\n\nAre the implementations SOTA? If not, can you try adding your method to a SOTA implementation?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-21"}

comments: SUMMARY:
This paper introduces a new stage reward estimator called the long N-step surrogate stage (LNSS) reward for deep reinforcement learning (RL). The LNSS aims to mitigate the high variance problem in deep RL, which has been shown to impede successful convergence of learning, hinder task performance, and limit the application of deep RL in continuous control problems. The paper shows that LNSS, which uses a long reward trajectory of future steps, consistently improves performance in terms of average reward, convergence speed, learning success rate, and variance reduction in Q-values and rewards. The evaluations are based on various environments in DeepMind Control Suite and OpenAI Gym using LNSS in baseline deep RL algorithms such as DDPG, D4PG, and TD3. The paper also demonstrates that LNSS significantly reduces the upper bound on the variances of Q-values compared to single-step methods.

STRENGTHS:
The paper proposes a new stage reward estimator, LNSS, which addresses the high-variance problem in RL. This can be useful in improving performance in continuous control tasks.

WEAKNESSES:
1. Compared with the common n-step bootstrap method, it seems that the difference is only the larger n?
2. The paper uses off-policy RL baselines like TD3 and such a large N in LNSS reward, but didn't pay any attention to the off-policy issue.
3. The paper is not well written, making it difficult to identify the originality.

QUESTIONS:
1. It is obvious that increasing the length of bootstrap can reduce variance, but how do you handle bias (off policy issue)?

LIMITATIONS:
The paper seems to lack originality.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper propose a new reward estimator for off-policy RL to reduce the variance of the Q-value target estimate in bellman backup. Specifically, the method works by computing the discounted return of a trajectory segment and then augment such reward along with the original transition (state, action, next state) in the replay buffer. On a range of simulated environments, the proposed method is able to enhance existing off-policy RL algorithm such as DDPG, D4PG and TD3 on a range of environments in DeepMind Control Suite and OpenAI Gym. The authors also provide variance analysis of the Q-value target estimate and demonstrate an improved variance upper-bound.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. The proposed method is very simple and can be directly added on top of any TD-style RL algorithms. In addition, the paper contains a set of thorough empirical evaluations that demonstrate that the effectiveness and the robustness of the proposed method (LNSS) under a range of reward functions (e.g., noisy, sparse), different hyperparameters as well as insights on the variance reduction effect that LNSS has on the Q value estimation.\n\nWEAKNESSES:\nIf I understand correctly, the proposed method can lead to a biased value estimate, which conceptually could hurt performance. The bias comes from the fact that the LNSS reward estimate assumes the trajectory is filled with a constant reward. For example, for sparse reward task where the reward is only non-zero, positive in the end of the trajectory, the Q-value would get overestimated. It would be great if the authors could discuss more the implication of the bias and whether there are any pathological examples that could break the method.\n\nQUESTIONS:\n- Have you experimented with using the full trajectory to compute the LNSS reward? I imagine that this could actually help the sparse reward tasks even more (e.g., Cartpole SwingUp Sparse).\n- It would be also very interesting to study the interaction of LNSS with more gradient steps per environment step (UTD ratio). Intuitively, having more gradient updates per data collection step could also greatly help with propagating the signal back in time fast (in terms of the number of environment steps).\n\nLIMITATIONS:\nThe author has adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors proposed a long N-step surrogate stage (LNSS) reward for DRL which mitigates the high variance problem in continuous control problems. It is shown that the proposed LNSS can provide consistent performance improvement measured by average reward, convergence speed, learning success rate, and variance reduction in Q values and rewards.\n\nSTRENGTHS:\nWell written and well organised.\nProposed a novel and easy-to-implement n-step method LNSS which reduces variance in training. The method can be easily applied on existing algorithms such as DDPG, TD3 for enhanced performance. \nPromising experimental results with large N-steps and theoretical analysis provided to support the conclusions. Nice ablation studies provided.\nMain experimental results are presented in a clean table which is easy to interpret.\n\nWEAKNESSES:\nI think some interesting experimental results lacks explanation. Some figures can be improved for better readability.\n\nQUESTIONS:\nOne question is that for baseline comparison, the authors chose original algorithms and a simple n-step variation of loss function with n=5. LNSS reported in the table is using N=100. I'm wondering can this be considered as a fair comparison? From ablation studies, smaller N such as N=5, and N=10 does not perform as good as N=100 in training and variance reduction. Can authors justify this? Would it be possible to also include LNSS N=5 in the main results table for comparison. \n\nI noticed that in some environments, a simple n-step variation with n=5 is even much worse than the baseline algorithm, while LNSS with N=100 provide promising improvements. This is very interesting. Any explanations for this phenomenon? Will LNSS with N=5 have the same improvements? \n\nToo many colors in figure 2 and 3, maybe use the same color for one baseline algorithm, for example, TD3-base TD3-LNSS both use red but one solid line and one dash line as you already use in the figures for an easy comparison. \n\nA final question out of curiosity, to my knowledge, TD3 itself would fail in some complex environments , and it is nice to see how LNSS helped to improve the situation, however, some algorithms such as max-entropy algorithm SAC would succeed. Will LNSS be combined with max-entropy algorithm also provide similar improvements in performance and variance reduction?\n\nLIMITATIONS:\nI think for me some phenomenon in experimental results may need some further explanation. A large N is required for powerful performance.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a simple but effective surrogate reward, which uses the future n-step cumulative rewards (with some coefficient). \nThe theoretical results show that the proposed method has a smaller variance than the classical one-step method.\nThe experimental results show the effectiveness of the proposed method.\n\nSTRENGTHS:\nGenerally, this paper writes well.\nThis paper proposes an interesting surrogate reward, which is very simple but has shown to be very effective.\nThe paper also provides a theoretical analysis of the variance compared to that of existing one-step method.\nThe experimental results are complete and sufficient to show the effectiveness of the proposed method.\n\nWEAKNESSES:\nI agree that this paper tries to solve an important problem and proposes a very simple but empirically effective method. \n\nHowever, the proposed method does not make sense to me. I give my reason below.\n\n- My major concern is whether the proposed method with the surrogate reward learns exactly the same solution as the one using the original reward. The paper didn't provide any discussion or proof of this critical problem.\n\n- In my view, my answer to this problem will be negative. Based on eq.7, the surrogate loss is, in fact, a multi-step cumulative reward times a coefficient. Compared to the existing n-step method (eq. 3), the difference is the coefficient and the next state (the proposed method use $s_t$, and the n-step method use $s_{t+N}$). The solution of the proposed method should be different from the original $Q^\\pi$.\n\n- I believe that eq. 5 is very suspicious. I don't understand why the $G'_k$ should be in that form (and $r_k'$ is a constant). In addition, in this way, the final computed $r'_k$ will be different over different k.\n- (I spent a lot of hours on this paper, trying to understand, but I failed to convince myself. I'm happy to increase my score if the author can prove I'm wrong)\n\nThe paper didn't make a comparison of the variance with n-step methods.\n\nIn addition,  how does the paper fit into the off-policy setting? It is no problem for the one-step method because there is no bias. However,   when utilizing multi-step data, this could be one problem for the proposed method (DDPG and several baseline methods used in the paper are off-policy methods).\n\nMinor Issues:\n- Line 115: It's not very general to assume the policy to be deterministic.\n- eq. 1: It is better to use $s_{k'+1} \\sim p( | s_{k'}, a_{k'} )$. Just differentiate $k'$ and $k$\n- eq. 2: missing an expectation over $s_{k+1}$.\n\nQUESTIONS:\nFor my major concern, please see the weakness above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-09"}