SlowGuess/NeurIPS_2023_OCR huggingface dataset:
informative as LNSS provides sequences of gradual and smoother rewards to be used in  $Q$ -learning. As a result, from the upper-left panels, this informative guidance from LNSS significantly enhances the performance in terms of episode rewards. The effect is even more pronounced when dealing with penalties.
From Table 1, with the original reward, the  $Q$  value is the sum of discounted expected rewards as in Equation 4. The farther away the goal state, the smaller the  $Q$  values are as a result of discounts over the horizon. When learning under the environment of using penalty rewards, the  $Q$  values in States  $S_0, S_1, S_2$  show that they are even farther away from the goal state value. The  $Q$  values can not be effectively updated due to a lack of informative feedback signal, and thus, causing the agent to get stuck and cease learning. In comparison, with the LNSS reward, goal-related information can be propagated back from the goal state to the initial state. This guidance plays a crucial role in assisting agents to overcome situations where they might otherwise get stuck.
Table 1:  $Q$  tables of  $Q$ -learning using original reward and LNSS reward. The  $Q$  values reported are averages across 10 different random seeds.
Acrobot SwingUp	Humanoid Walk	Fish Swim	Finger TurnHard
Algorithm	Success [%]	Avg. Rwd [μ±2σ]	Rank [%]	Success [%]	Avg. Rwd [μ±2σ]	Rank [%]	Success [%]	Avg. Rwd [μ±2σ]	Rank [%]	Success [%]	Avg. Rwd [μ±2σ]	Rank [%]
Noise Level 0% (0.0)
DDPG-LNSS	100	304.6 ± 82.9	-25.1	100	250.9 ± 16.5	-34.6	100	645.1 ± 86.9	-12.7	100	943.3 ± 32.9	0
DDPG-Base	100	187.9 ± 68.5	-53.8	0	1.3 ± 0.4	-99.6	100	272.9 ± 27.3	-63.1	80	301.8 ± 160.3	-68.0
DDPG-n5	100	195.2 ± 67.6	-51.9	0	2.2 ± 1.5	-99.4	100	150.5 ± 33.7	-79.6	100	459.8 ± 126.3	-51.3
TD3-LNSS	100	114.8 ± 33.8	-71.8	100	307.9 ± 16.9	-19.8	100	676.8 ± 44.2	-8.4	100	940.1 ± 29.6	-0.3
TD3-Base	0	5.2 ± 4.1	-98.7	60	184.2 ± 178.8	-52.1	100	483.1 ± 127.6	-34.6	90	251.8 ± 101.4	-73.3
TD3-n5	70	61.7 ± 45.7	-84.8	0	1.01 ± 0.41	-99.6	100	731.7 ± 132.3	-1.1	90	394.2 ± 164.4	-58.2
D4PG-LNSS	100	406.3 ± 13.1	0	100	383.9 ± 37.1	0	100	738.7 ± 59.8	0	100	811.5 ± 59.9	-14.0
D4PG-Base	100	133.5 ± 16.2	-67.1	100	277.7 ± 79.8	-27.7	100	585.7 ± 36.1	-20.7	100	336.9 ± 107.8	-64.3
D4PG-n5	100	310.1 ± 49.5	-23.7	100	360.3 ± 50.4	-6.2	100	683.5 ± 95.6	-7.5	90	413.7 ± 243.8	-56.1
Noise Level 1% (0.01)
DDPG-LNSS	100	243.5 ± 62.3	-40.1	100	255.1 ± 22.6	-33.6	100	700.4 ± 107.6	-5.2	100	934.3 ± 30.58	-0.1
DDPG-Base	100	188.2 ± 58.7	-53.7	0	0.99 ± 0.25	-99.8	100	340.1 ± 135.4	-53.9	100	302.5 ± 130.4	-67.9
DDPG-n5	100	159.6 ± 25.8	-60.7	0	1.15 ± 0.43	-99.7	100	149.4 ± 59.1	-79.8	100	416.9 ± 168.4	-55.8
TD3-LNSS	100	109.3 ± 20.8	-73.1	100	197.6 ± 45.1	-48.5	100	700.1 ± 84.5	-5.2	100	912.8 ± 92.2	-3.2
TD3-Base	0	2.4 ± 1.53	-99.4	0	2.05 ± 1.43	-99.5	100	506.9 ± 111.5	-31.4	60	193.4 ± 142.2	-79.5
TD3-n5	70	69.6 ± 23.5	-82.9	0	1.55 ± 0.8	-99.6	100	688.7 ± 132.3	-6.8	90	384.1 ± 232.4	-59.3
D4PG-LNSS	100	330.3 ± 60.6	-18.7	100	247.9 ± 42.3	-35.4	100	688.3 ± 55.3	-6.8	100	801.9 ± 80.2	-15.0
D4PG-Base	100	136.5 ± 30.2	-66.4	100	210.5 ± 44	-45.2	100	650.2 ± 89.3	-11.9	100	300.5 ± 143.2	-68.1
D4PG-n5	100	262.9 ± 67.1	-35.3	0	10.9 ± 2.3	-97.2	100	578.7 ± 78.9	-21.7	90	383.7 ± 293.8	-59.3
Noise Level 10% (0.1)
DDPG-LNSS	100	232.6 ± 61.5	-42.8	100	235.4 ± 25.7	-38.7	100	699.5 ± 95.3	-5.3	100	940.2 ± 54.1	-0.3
DDPG-Base	100	170.7 ± 42.1	-58.0	0	0.76 ± 0.42	-99.8	100	330.42 ± 144.2	-55.3	100	315.2 ± 205.4	-66.6
DDPG-n5	100	139.1 ± 28.7	-65.8	0	1.3 ± 0.9	-99.7	100	116.5 ± 16.8	-84.2	100	456.5 ± 219.7	-51.6
TD3-LNSS	100	80.7 ± 15.73	-80.1	100	200.2 ± 117.6	-47.3	100	696.5 ± 174.1	-5.7	100	888.54 ± 92	-5.8
TD3-Base	0	2.19 ± 1.35	-99.5	0	1.5 ± 0.33	-99.7	100	454.9 ± 272.9	-38.4	60	190.1 ± 215.7	-79.9
TD3-n5	0	24.9 ± 22.7	-93.9	0	1.1 ± 0.28	-99.7	100	654.7 ± 130.2	-11.4	80	338.1 ± 216.4	-64.2
D4PG-LNSS	100	331.3 ± 49.5	-18.5	100	240.8 ± 46.3	-37.3	100	681.4 ± 212.6	-7.8	100	803.9 ± 83.3	-14.8
D4PG-Base	100	130.1 ± 28.9	-67.9	0	1.42 ± 0.7	-99.7	100	645.3 ± 245.2	-12.6	100	302.5 ± 153.2	-67.9
D4PG-n5	100	251.4 ± 56.4	-38.1	0	0.89 ± 0.6	-99.8	100	563.4 ± 214.7	-23.7	90	355.1 ± 285.5	-62.4
Table 2: Systematic evaluations of LNSS respectively augmented Base algorithms, and comparisons to the Base and  $n5$ -augmented Base algorithms. "Rank" (\%) is the "percent of reward difference", the closer it is to 0 the better.
6.2 Main Results
Evaluation results in Table 2 are performed using default setup of Acrobot SwingUp, Humanoid Walk, Fish Swim, and Finger TurnHard in DMC. In the Table 2, "Success" is shorthand for success rate, "Avg. Rwd" stands for average reward, and "Rank" (\%) is the "percent of reward difference",
which is ( the average reward of the evaluated algorithm over that of the top performing algorithm - 1), the closer it is to 0 the better. Note that, in computing the success rate, only those trials that have achieved a reward of at least 50 are accounted for as successes.
The results are obtained at different noise levels, based on the last 50 evaluations of 10 different random seeds (same for all compared algorithms). Best performances are boldfaced for average reward (Avg. Rwd).
Q2 LNSS improves Base methods. The learning curves for the four continuous dense reward benchmark environments are shown in Figure 3. Quantitative comparisons to the baselines are summarized in the first section of Table 2 (noise = 0). Overall, LNSS (solid lines) outperforms their respective base methods (dash lines) in terms of averaged reward (Awg.Rwd), learning speed, and success rate for all three DRL algorithms. Among the measures, the success (learning) rate is to address the random initialization challenge caused by random seeds [10]. LNSS method helps its Base method to achieve a  $100\%$  learning success rates whereas DDPG and TD3 Base method struggle with Humanoid Walk and Finger TurnHard. Besides, according to the "Rank" measure in Table 2, LNSS helps enhance the performances of almost all Base methods up to the top-performing algorithm in each category in all the tested environments. Detailed plots of pair-wise comparison for without noise condition are provided in appendix E.1.
Figure 3: Systematic evaluation of LNSS piggybacked on three DRL algorithms (DDPG, D4PG, TD3) in DMC environments without reward noise. The shaded regions represent the  $95\%$  confidence range of the evaluations over 10 seeds. The x-axis is the number of steps. Detailed pair-wise comparison plots are provided in appendix E.1.
Q3 LNSS outperforms previous "n-step" methods  $(n5)$ . The main improvement of LNSS over  $n5$  are reflected in improved success rate, averaged reward, and "Rank". This improvement can be quantitatively viewed in the first section of Table 2 (noise = 0). LNSS consistently enables the Base methods to achieve  $100\%$  learning success rate. However, the  $n5$  method may even cause some learning to fail such as TD3 in Humanoid Walk and D4PG in Finger TurnHard. Additionally, according to "Rank" in Table 2, LNSS methods enable all cases to reach performances near those from the best algorithms. However, only TD3- $n5$  has resulted in slightly better "Rank" in Fish Swim only for without noise condition while all other  $n5$  methods are below LNSS or even below its Base such as TD3- $n5$  for humanoid walk and DDPG- $n5$  for Fish swim. We also observe that LNSS has improved the average reward and lowered the standard deviation for all Base algorithms over those by the  $n5$  method.
Q4 LNSS is robust under noisy rewards. We performed the same set of evaluations in Table 2 in DMC experiments under random uniform reward noise settings with magnitudes at  $1\%$  and  $10\%$  of the maximum DMC step reward (1). As results in the appendix E.2 (noise level  $1\%$ ), Appendix E.3 (Noise level  $10\%$ ), and the last two sections in Table 2 (noise  $= 1\%$  and  $10\%$ ) show, at larger noise level, all algorithms have degraded performance with lower mean averaged reward, larger standard division, and more negative "Rank". However, LNSS consistently outperformed Base and  $n5$  methods in terms of all measures for all three DRL algorithms. It is important to point out that we observe LNSS methods robustly improve the success rate with  $100\%$  for all environments. However, we observe that both Base and  $n5$  methods suffer greatly due to the increased noise level such as
in Humanoid Walk and Finger Turn Harde environments. As Equation (3) shows the performance degradation for  $n5$  methods are due to accumulated noise when collecting the discounted sum of returns. This accumulated noise will induce a much larger approximation variance and result in degraded performance.
Q5 LNSS is effective in environments with sparse rewards. For setting up sparse rewards in Cheetah environments, refer to appendix B. As Cheetah Sparse and Cartpole Sparse result in Figure 3. One can observe that LNSS enables all DRL algorithms successfully learn all three sparse reward tasks with converged episode rewards greater than 800. However, due to the limited feedback from sparse rewards, all base methods struggle to learn and even fail to learn (e.g., Cheetah Run). Moreover, it is important to point out that LNSS helps improve the learning speed over the base methods. This is expected as a long future reward trajectory of  $N$ -step is able to provide continuous reward feedback to the agent by assigning credits progressively backward from the time of achieving the desired states. In contrast, a single-step method does not provide any guidance or feedback to the learning agent until reaching the goal.
Q6 LNSS helps reduce the variance of  $Q$  value estimation. For this evaluation, we let coefficient of variation of  $Q$  value (in percentage) as  $cv(Q) = \frac{std(Q)}{Q}\%$ . In Figure 4, we show  $cv(Q)$  every 2e5 steps where as an example, we show TD3-LNSS outperforms TD3-Base and TD3-n5 under  $1\%$  noise in rewards. Similar results are obtained for DDPG and D4PG classes of algorithms and shown in appendix E.4. Also shown in the same Appendix are respective results at different noise levels. Results (Figure 12 to Figure 20 in appendix E.4) also show that for each LNSS augmented DRL algorithms, it has resulted in improved performance in  $cv(Q)$  to near the best performance among all DRL algorithms. Figure 4 provides an empirical validation for our theoretical result on bounds of variances of the  $Q$  value, namely, with large LNSS parameter  $N$  or at the late stage of learning,  $Q$  value variances become smaller as the upper bound of  $Q$  value variance decreases significantly according to Equation (14) in Theorem 1.
Figure 4: The  $cv(Q)$  (coefficient of variation of  $Q$  value, the lower the better) of TD3 in DMC with  $1\%$  noise. Results are averaged over 10 seeds. The x-axis is the number of steps. Additional and respective results for DDPG and D4PG can be found in appendix E.4
Q7. Effects of different length  $N$  in LNSS. In Figure 5, we show the effect of different choices of  $N$  ( $N = 5, 10, 20, 50, 100$ ) in our proposed LNSS on top of TD3. (Similar results hold for D4PG and DDPG which can be found in appendix E.7.) Comparisons are performed using the same set of hyperparameters except a varying  $N$ . Figure 5 depicts performance curves for DMC tasks. The LNSS with  $N = 50$  and  $N = 100$  outperform LNSS with  $N = 5, 10$  in terms of final reward and learning speed. The results corroborate what we expect from our theoretical analysis (please refer to Figure 1 in the paper). We can see a clear benefit of using large  $N$  ( $N = 50$ , and 100) over small ( $N = 5$ , and 10).
Additionally, in Figure 6, we show how  $cv(Q)$  (coefficient of variation of  $Q$ ) varies with different  $N$ . Among the three  $N$  values,  $N = 100$  outperforms others. Also, we observe that, in more complex environment, a longer  $N$  is favorable. Notice that, for cartpole sparse environment, there is little difference among different  $N$  values. However, in Finger TurnHard and Humanoid Walk, the  $cv(Q)$  values of Base cases and  $N = 5$  cases increase dramatically. However, the  $cv(Q)$  values for  $N = 50$  and  $N = 100$  change only slightly. Figure 6 provides an empirical validation for our theoretical analysis on variance discount factor  $\psi$  in Figure 1 that the longer the  $N$ , the more reduction in variance of  $Q$  value estimation.
Figure 5: Episode rewards for the 3 tasks in DMC by TD3-LNSS with different  $N$  ( $N = 5,10,20,50,100$ ) in Equation (6). The results are averaged from 10 seeds. The x-axis is the number of steps. Respective results for DDPG and D4PG are provided in appendix E.7
Figure 6:  $cv(Q)$  (coefficient of variation of  $Q$  value, the lower the better) for the 3 tasks in DMC by LNSS with different  $N$  ( $N = 5, 50, 100$ ) in Equation (6).
6.3 Limitation of This Study.
Here we introduce a new multi-step method, the LNSS, to effectively utilize longer reward trajectories of future steps than those used in existing methods to estimate the  $Q$  value with an aim of reducing variances in learning. By piggybacking LNSS on top of TD3, DDPG, and D4PG, we demonstrate improved performance in terms of final reward, learning speed, and variance reduction in  $Q$  value. However, there exist two limitations. 1) The LNSS requires positive semi-definite reward values in a long trajectory. Large negative reward may wash out the significance of good positive rewards. However, this limitation can be easily fixed by elevated reward stored in the temporary buffer  $\mathbb{D}'$  by a positive constant, and lower bounded it by 0. 2) Early termination is another limitation during implementation, which mostly affects LNSS in GYM environments (Hopper, Humanoid, Walker2d). Different from DMC tasks, GYM allows an environment to terminate before reaching the maximum time steps. As such, in early training stage, only 10 or 20 steps may be used to compute LNSS which diminishes the power of large  $N$  (such as 50 or 100). To resolve this limitation, re-programming task settings in the original GYM environments is required.
7 Discussion and Conclusion
1) In this work, we introduce a novel "  $N$ -step" method, the LNSS which utilize longer reward trajectories of future steps than those used in existing methods to estimate the  $Q$  value with an aim of reducing variances in learning. It is easy to implement, as shown in the paper, and it can be easily piggybacked on policy gradient DRL algorithms. It has been shown consistently outperform respective Base and short  $n$ -step algorithms in solving benchmark tasks in terms of performance score, learning success rate, and convergence speed. 2) We provide a theoretical analysis to show that LNSS reduces the upper bound of variance in  $Q$  value exponentially from respective single step methods. 3) We empirically demonstrate the performance of LNSS piggybacked on TD3, DDPG, and D4PG in a variety of benchmark environments that have been challenging for existing methods to obtain good results. 4) We show LNSS can provide consistent performance improvement under various reward noise settings, and for sparse rewards. Our results suggest that LNSS is a promising tool for improving learning speed, learning performance score, and reducing learning variance. Further investigation on how to maximize the benefit of selecting an optimal reward length  $N$ , and how to take advantage of a different  $n$  in  $Q$  value update are exciting questions to be addressed in future work.
This research was supported in part under NSF grants #1808752 and #2211740.
[1] Wentao Liu, Junmin Zhong, Ruofan Wu, Bretta L Fylstra, Jennie Si, and He Helen Huang. Inferring human-robot performance objectives during locomotion using inverse reinforcement learning and inverse optimal control. IEEE Robotics and Automation Letters, 7(2):2549–2556, 2022.
[2] Ruofan Wu, Junmin Zhong, Brent Wallace, Xiang Gao, He Huang, and Jennie Si. Human-robotic prosthesis as collaborating agents for symmetrical walking. Advances in Neural Information Processing Systems, 35:27306-27320, 2022.
[3] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
[4] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[5] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[6] Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.08617, 2018.
[7] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[8] Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 110(9):2419-2468, 2021.
[9] Johan Bjorck, Carla P Gomes, and Kilian Q Weinberger. Is High Variance Unavoidable in RL? A Case Study in Continuous Control. arXiv preprint arXiv:2110.11222, 2021.
[10] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[11] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pages 1329-1338. PMLR, 2016.
[12] Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(9), 2004.
[13] Joshua Romoff, Peter Henderson, Alexandre Piché, Vincent Francois-Lavet, and Joelle Pineau. Reward estimation for variance reduction in deep reinforcement learning. arXiv preprint arXiv:1805.03359, 2018.
[14] Christopher De Asis, J Hernandez-Garcia, G Holland, and Richard Sutton. Multi-step reinforcement learning: A unifying algorithm. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[15] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[33] Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101, 2018.
[34] Alfredo V Clemente, Humberto N Castejón, and Arjun Chandra. Efficient parallel methods for deep reinforcement learning. arXiv preprint arXiv:1705.04862, 2017.
[35] Yinlong Yuan, Zhu Liang Yu, Zhenghui Gu, Yao Yeboah, Wu Wei, Xiaoyan Deng, Jingcong Li, and Yuanqing Li. A novel multi-step Q-learning method to improve data efficiency for deep reinforcement learning. Knowledge-Based Systems, 175:107-117, 2019.