SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A Long  $N$ -step Surrogate Stage Reward for Deep Reinforcement Learning
Junmin Zhong
Ruofan Wu
Jennie Si *
We introduce a new stage reward estimator named the long  $N$ -step surrogate stage (LNSS) reward for deep reinforcement learning (RL). It aims at mitigating the high variance problem, which has shown impeding successful convergence of learning, hurting task performance, and hindering applications of deep RL in continuous control problems. In this paper we show that LNSS, which utilizes a long reward trajectory of future steps, provides consistent performance improvement measured by average reward, convergence speed, learning success rate, and variance reduction in  $Q$  values and rewards. Our evaluations are based on a variety of environments in DeepMind Control Suite and OpenAI Gym by using LNSS piggybacked on baseline deep RL algorithms such as DDPG, D4PG, and TD3. We show that LNSS reward has enabled good results that have been challenging to obtain by deep RL previously. Our analysis also shows that LNSS exponentially reduces the upper bound on the variances of  $Q$  values from respective single-step methods.
Reinforcement learning (RL) shows great promise as a theory of learning in complex dynamic tasks [1, 2]. For continuous control problems involving high-dimensional states and control spaces, several deep reinforcement learning (DRL) algorithms, such as deep deterministic policy gradient (DDPG) [3], proximal policy optimization (PPO) [4], Soft actor critic (SAC) [5], D4PG [6] and twin delayed DDPG (TD3) [7], have demonstrated their great potential.
However, the high variance problem [8, 9] in DRL still poses great challenges for it to be effective in solving complex contentious control problems. On one hand, this problem is intrinsic to the trial and error nature of reinforcement learning, which causes low data efficiency, slow learning, or even instability of learning and poor task performance [10, 11]. On the other hand, high variances are often caused by several factors, such as variances in value estimation [12], variances resulting from different random seeds [10], and variances due to noisy stochastic reward or corrupted sparse reward [13]).
Different approaches have been proposed to address the high variance problem, from the original  $\mathrm{TD}(\lambda)$  algorithm to  $Q(\sigma)$  [14] as a generalization of  $\mathrm{TD}(\lambda)$  [15]. Rollout methods [16, 17, 18] have enabled the success of some of the most important milestone cases such as Backgammon [17] and AlphaGo [19] in machine learning history. Although these algorithms has been demonstrated in discrete state and control problems, variants of rollout are believed to be effective also in continuous control tasks. Modern DRL algorithms such as Rainbow [20] integrate  $n$ -step learning on top of DQN and results in a speed-up in learning and an improvement of the final performance on Atari 2600 games. D4PG [6] uses the same  $n$ -step methods to update target values, and have been shown effective in continuous control tasks. PPO [4] uses a truncated version of generalized advantage estimation (GAE) to update policy which reduces policy variance [21]. However these methods do
*Corresponding author: si@asu.edu
not explicitly account for noisy rewards where such uncertainties have been reported to degrade the model/controller performance [22].
In this paper, we propose a new algorithm, namely the long  $N$ -step surrogate stage (LNSS) reward estimator. LNSS aims at providing variance reduction for DRL methods in continuous control problems, even under noisy reward signals or corrupted sparse reward. LNSS is simple, and can be easily piggybacked on DRL algorithms. It is based on a surrogate reward  $r'$  obtained from a trajectory of rewards of future "  $N$ -step"s. In turn,  $r'$  is used in  $Q$  value update where the target function is then of the same form as that in single-step TD methods. Using a long "  $N$ -step"s of the reward sequence containing environment information, it is expected to be more advantageous to learning than using a short trajectory of  $n$ -step of the reward sequence.
We show theoretically that the LNSS method exponentially reduces the upper bound on the variance of  $Q$  value from a respective single-step method. We then conduct extensive experiments in selected environments in DMC and GYM where most current DRL algorithms such as DDPG, D4PG, and TD3 that typically struggle to obtain good results [23]. We consider uniform random noises of different magnitudes to show how LNSS can benefit a variety of DRL algorithms while the performances of other methods degrade significantly. We further show how LNSS can still be effective in environments where only sparse rewards are available.
Contributions. 1) We introduce a new, simple yet effective surrogate reward  $r'$  that interpolates between the single-step TD methods and Monte-Carlo (MC) methods [15], to trade off the strengths from the two extremes. 2) We theoretically show the proposed method helps reduce the upper bound on the variance of  $Q$  value with a long trajectory of rewards of future "N-step"s. 3) Extensive experiments on DMC and GYM benchmarks show that LNSS robustly benefits the learning performance of a variety of DRL algorithms even under reward noise or sparse reward. Mainly, the LNSS consistently reduces variances in the learned value function and achieves robust results from different random seeds in several different types of continuous control problems. Please check our code
The "  $n$ -step" methods. Here, as in the literature,  $n$ -step methods refer to those involving a trajectory of rewards of multiple future steps in DRL algorithms for speeding up learning and reducing variance [24, 25, 6]. They usually proceed as follows. First, collect  $n$ -step returns to form a discounted sum of rewards [26, 27]. Then, perform  $n$ -step return backup to update the value function [15]. It was empirically suggested that  $n$ -step methods perform better than single-step methods since  $n$ -step returns propagate rewards to relevant state-action pairs more efficiently [24, 25].
Using longer trajectories with a larger  $n$  is expected to improve learning performance [28]. However, to date, the  $n$ -step DRL methods have only been demonstrated by using relatively small  $n$ . Rainbow [20] uses  $n = 1,3,5$  to evaluate performance of 57 Atari 2600 games and reported  $n = 3$  as the best-performing reward trajectory length. D4PG [6] uses  $n = 1,5$  to evaluate a variety of continuous control tasks in DMC and results show that  $n = 5$  performs uniformly better than others. However, 3 and 5 are relative small for complex tasks which usually have a maximum of 1000 steps in an episode.
There are potentially two reasons that prevent  $n$ -step methods from using larger length  $n$ . One is that the longer the  $n$ -step return, the larger reward scale that can cause saturation and inefficiency in learning for most DRL algorithms [10]. The second reason is that the longer the  $n$ -step return, the smaller discount factor  $(\gamma^n)$  in  $n$ -step backup [15]. Based on previous studies [29], variations in the scale of the discount factor directly causes uncertainty to the RL performance. New algorithmic ideas are needed for DRL algorithms to benefit from longer reward bootstrapping.
Reward estimation. Reward estimation aims at reducing sample complexity and learning variance as reward variances may be introduced from using random seeds in training [9], from sensor noises in real-world applications [8] and from corrupted sparse rewards [13]. Several approaches have been proposed for reward estimation. Small backups [30], instead of directly using an actual reward, use multi-step imaginary rollouts as a predicted reward. Model-based policy search with Bayesian neural network [31] finds expected cost by averaging over multiple virtual rollout. The  $\lambda$ -prediction [32] uses an extra network to predict reward and the training of which may introduce additional variance. Model-based value expansion [33] uses an imaginary reward from a model to estimate the values of state value functions. To reduce learning variance resulted from noisy rewards, regression reward
estimator [13] expects the estimated reward to reduce the variance propagated to the value function. Unbiased reward estimator [22] estimates a surrogate reward by learning a reward confusion matrix which estimates true reward from noisy reward. However, all these estimation methods require extra models (either in the form of neural networks, environment transitions, or reward noise model), which imposes overhead on learning and introduces estimation error. Additionally, how to effectively and efficiently integrate these reward estimates into DRL algorithms is not clear, or at least not straightforward.
LNSS. In contrast, our method does not introduce any new hyper parameters into learning as we still use the same form of reward signals but only to a longer horizon of "  $N$ -step"s. The LNSS method can be piggybacked on any DRL algorithms directly. Comparing to  $n$ -step methods, by using our surrogate reward  $r'$  to perform single-step updates, we can benefit from  $n$ -step returns without changing the reward scale. This frees LNSS from a limited reward horizon to a much longer length  $N$ .
Selecting DRL algorithms. First, note that few existing  $n$ -step methods have been successfully applied to different DRL algorithms with consistent learning variance reduction. We selected DDPG, D4PG, and TD3 as base DRL algorithms, with which our LNSS is to be piggybacked on for the following considerations. Among all DMC based benchmark results [23], distributional method D4PG [6] outperforms other DRL algorithms such as DDPG [3], and TD3 [5]. On the other hand, TD3, DDPG have relatively good performance in simple tasks such as walker but struggled to achieve good results in harder DMC tasks such as humanoid-walk and fish-swim. We thus use the three chosen DRL algorithms to systematically test the idea of LNSS piggybacked on promising DRL algorithms in complex tasks, even in noisy reward environments.
Reinforcement Learning. We consider a reinforcement learning agent interacts with its environment in discrete time. At each time step  $k$ , the agent observes a state  $s_k \in S$  and select an action  $a_k \in \mathcal{A}$  based on its policy  $\pi : S \to \mathcal{A}$ , namely,  $a_k = \pi(s_k)$ , and receives a scalar reward  $r(s_k, a_k) \in \mathcal{R}$  (use  $r_k$  as short hand notation).
Evaluation of a policy  $\pi$  is performed using the expected return after taking an action  $a_{k}$  based on state  $s_k$  following the policy  $\pi$ :
$$
Q ^ {\pi} \left(s _ {k}, a _ {k}\right) = \mathbb {E} \left[ R _ {k} \mid s _ {k}, a _ {k} \right]
$$
$$
\text {w h e r e} R _ {k} = \sum_ {t = k} ^ {\infty} \gamma^ {t - k} r _ {t}, \tag {1}
$$
$$
s _ {k} \sim p (\cdot | s _ {k - 1}, a _ {k - 1}),
$$
$$
a _ {k} = \pi (s _ {k}),
$$
with  $0 < \gamma < 1$ . The two common approaches (single-step and  $n$ -step) to update the  $Q$  value for a policy  $\pi$  are as follows.
Single-step RL Algorithms. RL algorithms rely on using TD error or residual of the Bellman equation to update the state-action  $Q$  value for a policy  $\pi$ , as described below,
$$
Q ^ {\pi} \left(s _ {k}, a _ {k}\right) = r _ {k} + \gamma Q ^ {\pi} \left(s _ {k + 1}, a _ {k + 1}\right). \tag {2}
$$
This equation provides the basis for single-step RL methods. It is also known as a backup operation since it transfers information from one step ahead back to the current step.
"  $n$ -step" RL algorithms. Using  $n$ -step rewards for faster reward propagation in RL has long been investigated [26, 14, 6, 20]. In  $n$ -step methods, the value function  $Q^{\pi}(s_k,a_k)$  update using an  $n$ -step return is based on the following,
$$
Q ^ {\pi} \left(s _ {k}, a _ {k}\right) = \sum_ {t = k} ^ {k + n - 1} \gamma^ {t - k} r _ {t} + \gamma^ {n} Q ^ {\pi} \left(s _ {k + n}, a _ {k + n}\right). \tag {3}
$$
The  $n$ -step return is expected to help agents learn more efficiently by affecting multiple state action pairs within one update and gain information over the  $n$ -step horizon.
More details on specific modern DRL algorithms such as the single-step methods (DDPG and TD3) and the  $n$ -step methods (D4PG and PPO) can be found in Appendix D.
4 Long  $N$ -step Surrogate Stage (LNSS) Reward
In this section, we introduce LNSS based on infinite horizon discounted reward formulation of reinforcement learning. Given a reward trajectory of  $N$  steps from time step  $k$ , let  $G(s_{k:k+N-1}, a_{k:k+N-1}) \in \mathbf{R}$  (use  $G_k$  as short hand notation) denote the discounted  $N$ -step reward, i.e.,
$$
G _ {k} = \sum_ {t = k} ^ {k + N - 1} \gamma^ {t - k} r _ {t}, \tag {4}
$$
where  $r_t$  is the  $t$ th stage reward and  $t$  is from  $k$  to  $k + N - 1$ . In LNSS, we introduce  $r_k'$  as a surrogate stage reward in place of  $r_k$  in Equation (2). To determine  $r_k'$ , we treat it as a weighted average of the  $N$ -step reward sequence, namely
$$
r _ {k} ^ {\prime} = \frac {\sum_ {t = k} ^ {k + N - 1} \gamma^ {t - k} r _ {t}}{\sum_ {n = 0} ^ {N - 1} \gamma^ {n}}. \tag {5}
$$
We then propose the surrogate stage reward  $r_k'$  to be
$$
r _ {k} ^ {\prime} = G _ {k} \frac {\gamma - 1}{\gamma^ {N} - 1}. \tag {6}
$$
This surrogate stage reward  $r_k'$  as formulated in Equation (6) relies on a discounted reward of an  $N$ -step horizon, from time step  $k$  to step  $(k + N - 1)$ , from the stored experiences into a temporary replay buffer  $\mathbb{D}'$ . For a training episode of  $T$  steps  $[0, 1, 2\ldots, T]$ , the  $\mathbb{D}'$  is a moving window of size  $N$  from the initial state  $s_0$  until the terminal state  $s_T$ . As a result, when there is a sufficient number (i.e.,  $N$ ) of reward samples, LNSS computes the surrogate reward  $r_k'$  from below,
$$
r _ {k} ^ {\prime} = \frac {\gamma - 1}{\gamma^ {N} - 1} \sum_ {t = k} ^ {k + N - 1} \gamma^ {t - k} r _ {t}. \tag {7}
$$
Note that, when the reward is estimated at the beginning or toward the end of a long trial, less than  $N$  reward samples are available for estimating  $r_{k}^{\prime}$ . A simple adjustment is given in Appendix C.
Once  $r_k'$  is obtained,  $r_k'$  and state action pairs  $(s_k, a_k, s_{k+1})$  will append as a new transition  $(s_k, a_k, r_k', s_{k+1})$  stored into the memory buffer  $\mathbb{D}$ .
Note that many DRL algorithms [4, 20, 6] use distributed learning procedure to accelerate experience sample collection. We use the same technique to speed up sampling experiences. Then a DRL algorithm is ready to update the  $Q$  value and the respective policy based on mini-batch data from the memory buffer. In general form, we have
$$
Q _ {i + 1} \left(s _ {k}, a _ {k}\right) = r _ {k} ^ {\prime} + \gamma Q _ {i} \left(s _ {k + 1}, \pi_ {i} \left(s _ {k + 1}\right)\right),
$$
$$
\pi_ {i} \left(s _ {k}\right) = \arg \max  _ {a _ {k}} Q _ {i} \left(s _ {k}, a _ {k}\right), \tag {8}
$$
where  $i$  is iteration number. Putting the above two equations together, we have
$$
Q _ {i + 1} \left(s _ {k}, a _ {k}\right) = r _ {k} ^ {\prime} + \gamma \max  _ {a _ {k + 1}} Q _ {i} \left(s _ {k + 1}, a _ {k + 1}\right). \tag {9}
$$
Remark 1. 1) Different from  $n$ -step methods, in our LNSS,  $N$  is the number of steps for accumulating rewards in Equation (6). We still perform a single-step update as in Equation (8). This allows us to prevent steep discounts of  $\gamma^n$  in  $n$ -step backup when using longer  $n$ -step returns. As a result, LNSS can effectively use large  $N$  at a scale of 100 while D4PG (Barth-Maron et al., 2018) uses  $n$  at a scale of 5 steps, and the same for Rainbow (Hessel et al., 2018).
2) LNSS can also be effective in a sparse reward environment. With a longer reward trajectory of  $N$  future steps, LNSS continuously provides a reward estimate as feedback to the agent by assigning credits progressively backward from the time of achieving the desired states. That is to say that LNSS
has turned a sparse reward into a dense reward. Thus, this helps the agent effectively and efficiently learn the task. However, the single-step method does not have the ability to provide any guidance or feedback to the learning agent until reaching a goal.
3) The  $Q$  value for policy  $\pi$  due to LNSS differ from that in Equation 1. The former aims to maximize the sum of a sequence of surrogate rewards which are the weighted averages of the sequence of  $N$ -steps of the original rewards. As such, goal-related information can be progressively propagated back from the goal state to the current state. This prevents the agent from getting trapped from reaching its intended goal and thus being stuck in an undesirable set of states due to a lack of feedback.
5 Variance Analysis
We now analyze the behavior of an actor-critic RL and our LNSS actor-critic RL. We consider the infinite horizon discounted reward formulation of RL (with  $0 < \gamma < 1$ ). Specifically, we show that the upper bound on the variance in  $Q$  value due to LNSS differs by an exponential factor from that of a single step actor-critic RL (AC). As this upper bound reduces exponentially as  $N$  increases, it suggests significant variance reduction by using LNSS from using single step reward.
We first represent the  $Q$  values using single step reward  $r_k$  in Equation (2) and using surrogate reward  $r_k'$  from LNSS in Equation (8), respectively as follows,
$$
\operatorname {V a r} \left[ Q _ {i + 1} \left(s _ {k}, a _ {k}\right) \right] = \operatorname {V a r} \left[ r _ {k} \right] + \operatorname {V a r} \left[ \gamma Q _ {i} \left(s _ {k + 1}, a _ {k + 1}\right) \right] + 2 \operatorname {C o v} \left[ r _ {k}, \gamma Q _ {i} \left(s _ {k + 1}, a _ {k + 1}\right) \right]. \tag {10}
$$
$$
\operatorname {V a r} \left[ \mathbb {Q} _ {i + 1} \left(s _ {k}, a _ {k}\right) \right] = \operatorname {V a r} \left[ r _ {k} ^ {\prime} \right] + \operatorname {V a r} \left[ \gamma \mathbb {Q} _ {i} \left(s _ {k + 1}, a _ {k + 1}\right) \right] + 2 \operatorname {C o v} \left[ r _ {k} ^ {\prime}, \gamma \mathbb {Q} _ {i} \left(s _ {k + 1}, a _ {k + 1}\right) \right]. \tag {11}
$$
Lemma 1. Assume  $\{r_k\}$  is IID and drawn from the memory buffer  $\mathbb{D}$ . Let  $Q_{i}(s_{k + 1},a_{k + 1})$  in Equation (8) be the  $i$ -th approximated return to solve Equation (1). We then have the following,
$$
\operatorname {C o v} \left[ r _ {k}, r _ {j \neq k} \right] = 0, \tag {12}
$$
$$
\operatorname {C o v} \left[ r _ {k}, Q _ {i} \left(s _ {k + 1}, a _ {k + 1}\right) \right] = 0. \tag {13}
$$
Theorem 1. Consider the variances of two  $Q$  value sequences, denoted as  $Q_{i}$  and  $\mathbb{Q}_i$ , in Equation (10) and Equation (11), which are obtained respectively from a single step method and an LNSS method.
Additionally, assume that  $Q_0 = \operatorname{Var}[Q_0] = 0$  and  $\mathbb{Q}_0 = \operatorname{Var}[\mathbb{Q}_0] = 0$ . Let the IID reward  $\{r_k\}$  and  $\{r_k'\}$  be drawn from the memory buffer  $\mathbb{D}$ . Assume the variance of  $\{r_k\}$  is upper bounded by a finite positive number  $\mathbb{B}$ , i.e.,  $\operatorname{Var}[r_k] \leq \mathbb{B}$ . Further define a constant  $\psi$  as,
$$
\psi = \left(\frac {\gamma - 1}{\gamma^ {N} - 1}\right) ^ {2} \left(\frac {\gamma^ {2 N} - 1}{\gamma^ {2} - 1}\right). \tag {14}
$$
Then the upper bounds of the variances of the two  $Q$  value sequences,  $\operatorname{Var}[Q_{i+1}]$  and  $\operatorname{Var}[\mathbb{Q}_{i+1}]$ , are respectively described below,
Figure 1: Variance discount factor  $\psi$  in Equation (14).
$$
\operatorname {V a r} \left[ Q _ {i + 1} \left(s _ {k}, a _ {k}\right) \right] \leq \sum_ {t = 1} ^ {i + 1} \left(\gamma^ {t - 1}\right) ^ {2} \mathbb {B}, \tag {15}
$$
$$
\operatorname {V a r} \left[ \mathbb {Q} _ {i + 1} \left(s _ {k}, a _ {k}\right) \right] \leq \psi \sum_ {t = 1} ^ {i + 1} \left(\gamma^ {t - 1}\right) ^ {2} \mathbb {B}. \tag {16}
$$
Proof. Both proofs of Lemma 1 and Theorem 1 are given in Appendix A.
Remark 2. We now provide some insights based on the variance analysis above.
1) Given  $\psi$  in Equation (14), i.e.,  $\psi = (\frac{\gamma - 1}{\gamma^N - 1})^2 (\frac{\gamma^{2N} - 1}{\gamma^2 - 1})$ , it follows that for large  $N$ ,  $\psi = (\gamma - 1)^2 (\frac{-1}{\gamma^2 - 1}) = \frac{1 - \gamma}{1 + \gamma}$ .
2) Furthermore, by the following identifies,  $\gamma^2 - 1 = (\gamma - 1)(\gamma + 1)$  and  $\gamma^{2N} - 1 = (\gamma^N - 1)(\gamma^N + 1)$ , we have that  $\psi = \left(\frac{\gamma - 1}{\gamma + 1}\right)\left(1 + \frac{2}{\gamma^N - 1}\right)$ . Therefore,  $\psi$  decreases exponentially (refer to Figure 1).
3) From inspecting Equations (15) and (16), we can see a clear advantage of using long "  $N$ -step" in LNSS over the typical reward  $r_k$ .
6 Experiments and Results
In this section, we first show how LNSS can benefit learning by using a simple Maze environment. To provide insight on the effect of using LNSS, we compare the performance of the original  $Q$ -learning with the one that the stage reward is replaced by LNSS reward.
We then provide a comprehensive evaluation of our proposed LNSS piggybacked on three promising DRL algorithms (DDPG, D4PG, TD3) by measuring their performance on several benchmarks in DMC and GYM (for results on GYM, please refer to Appendix E.5). Details of the implementation, training, and evaluation procedures are provided in Appendix B. In reporting evaluation results below, we use the following short-form descriptions.
1) "Base": the original DRL algorithms (TD3, D4PG, DDPG).
2) "LNSS": LNSS piggybacked on the respective DRL algorithms where  $N = 100$  unless otherwise specified.
3) "n5": the DRL algorithms with an "  $n$ -step" implementation as in Equation (3) with  $n = 5$ .
Our evaluations aim to quantitatively address the following questions:
Q1. How is LNSS different from original reward? and How it helps learning?
Q2. How does LNSS improve Base method?
Q3. How does LNSS compare with previous  $n$ -step method ( $n5$ )?
Q4. Is LNSS robust enough to compensate for stochastic reward?
Q5. Does LNSS improve Base methods under a sparse reward setting?
Q6. How does LNSS reduce the variance of estimated  $Q$  values?
Q7. How different  $N$  in LNSS affects performance?
Figure 2: Evaluation of LNSS piggybacked on  $Q$ -learning in simple Maze (right panel). The evaluations are averaged over 10 seeds. The x-axis is the number of episodes (top-left panel) and the state number (bottom-left panel) in the plots for episode reward and stage reward, respectively. Note that the stage reward for LNSS is calculated based on Equation 6
6.1 The Simple Maze Problem
The right panel of Figure 2 shows the Maze environment where  $S_0$  is the initial state,  $S_5$  is the goal state, and black blocks are walls. The unique optimal policy is {right,right,down,down,left}. We consider two problem formulations:
1) Without Penalty: the agent only receives a reward 1 upon reaching goal state  $S_{5}$ . For all other states, the agent receives 0's.
2) With Penalty: the agent only receives a reward 1 upon reaching goal state  $S_{5}$ . For all other states, the agent receives -0.1 per stage as a penalty.
Q1 LNSS takes future rewards into formulating the surrogate stage reward to guide a RL agent in reaching the goal state. As Figure 2 lower-left panels show, the stage reward from LNSS is more
No Penalty Case
Up	Down	Right	Left
State	Original Reward	LNSS Reward	Original Reward	LNSS Reward	Original Reward	LNSS Reward	Original Reward	LNSS Reward
0	0	0	0	0	0.47	1.33	0	0
1	0	0	0	0	0.62	1.45	0	0
2	0	0	0.76	1.48	0	0	0	0
3	0	0	0.89	1.36	0	0	0	0
4	0	0	0	0	0	0	0.99	0.99
With Penalty Case
Up	Down	Right	Left
State	Original Reward	LNSS Reward	Original Reward	LNSS Reward	Original Reward	LNSS Reward	Original Reward	LNSS Reward
0	-0.09	-0.05	-0.09	-0.05	-0.09	1.1	-0.09	-0.05
1	-0.07	-0.02	0	-0.01	-0.1	1.26	-0.13	-0.03
2	-0.055	0	-0.055	1.36	-0.054	0	-0.055	-0.01
3	-0.03	0	0.04	1.3	-0.03	0.03	0	0
4	-0.01	0.03	-0	0.06	0	0.07	0.38	0.997