Weighted Bellman Backups for Improved Signal-to-Noise in Q-Updates
Authors: Kimin Lee
Michael Laskin
Aravind Srinivas
Pieter Abbeel
Venue:  (2021)
Keywords: Deep reinforcement learning
ensemble learning
Q-learning

Abstract: Off-policy deep reinforcement learning (RL) has been successful in a range of challenging domains. However, standard off-policy RL algorithms can suffer from low signal and even instability in Q-learning because target values are derived from current Q-estimates, which are often noisy. To mitigate the issue, we propose ensemble-based weighted Bellman backups, which re-weight target Q-values based on uncertainty estimates from a Q-ensemble. We empirically observe that the proposed method stabilizes and improves learning on both continuous and discrete control benchmarks. We also specifically investigate the signal-to-noise aspect by studying environments with noisy rewards, and find that weighted Bellman backups significantly outperform standard Bellman backups. Furthermore, since our weighted Bellman backups rely on maintaining an ensemble, we investigate how weighted Bellman backups interact with UCB Exploration. By enforcing the diversity between agents using Bootstrap, we show that these different ideas are largely orthogonal and can be fruitfully integrated, together further improving the performance of existing off-policy RL algorithms, such as Soft Actor-Critic and Rainbow DQN, for both continuous and discrete control tasks on both low-dimensional and high-dimensional environments.

Under review as a conference paper at ICLR 2021
WEIGHTED BELLMAN BACKUPS FOR IMPROVED SIGNAL-TO-NOISE IN Q-UPDATES
Anonymous authors
Paper under double-blind review
ABSTRACT
Off-policy deep reinforcement learning (RL) has been successful in a range of challenging domains. However, standard off- alg can suffer from low signal and even instability in Q-learning because target values are derived from current Q-estimates, which are often noisy. To mitigate the issue, pro- pose ensemble- based weighted Bellman backups, which re- re-weight target Q-values based on uncertainty estimates from Q-ensemble. We empirically observe that the proposed method stabilizes and improves learning on both continuous and dis- crete control benchmarks. We also specifically investigate the signal-to-noise as- pect by studying environments with noisy rewards, and find that weighted Bellman backups significantly outperform standard Bellman backups. Furthermore, since our weighted Bellman backups rely on maintaining an ensemble, we investigate how weighted Bellman backups interact with UCB Exploration. By enforeing the diversity between agents using Bootstrap, we show that these different ideas are largely orthogonal and can be fruitfully integrated, together further improving the performance of existing off-policy RL algorithms, such as Soft Actor-Critic and Rainbow DQN, for both continuous and discrete control tasks on both low- dimensional and high- dinenssiaa environments.
INTRODUCTION
Model-free reinforcement learning (RL), with high-capacity function approximators, such as deep neural networks (DNNs), has been used to solve variety of sequential decision-making problems, including board games (Silver et al., 2017; 2018), video games (Mnih et al., 2015; Vinyals et al., 2019), and robotic manipulation (Kalashnikov et al., 2018). It has been well established that the above successes are highly sample inefficient (Kaiser et al., 2020). Recently, lot of progress has been made in more sample-efficient model-free RL algorithms through improvements off-policy learning both in discrete and continuous domains (Fujimoto et al., 2018; Haarnoja et al., 2018; Hessel et al., 2018; Amos et al., 2020). However, standard off- policol RL algorithms can suffer from instability in Q-learning due to error propagation in the Bellman backup, i.e., the errors induced in the target value can lead an increase in overall error in the Q-function (Kumar al., 2019; 2020).
One way to address the error propagation issue is to use ensemble methods, which combine multiple models of the value function (Hasselt, 2010; Van Hasselt al., 2016; Fujimoto et al., 2018). For dis- crete control tasks, double Q-learning (Hasselt, 2010; Van Hasselt et al., 2016) addressed the value overestimation by maintaining two independent estimators of the action values and later extended to continuous control tasks in TD3 (Fujimoto et al., 2018) 8). While most prior work has improved the stability by taking the minimum over Q-functioss, this also needlessly loses signal, and we propose an alternative way that utilizes ensembles to estimate uncertainty and provide more stable backups.
In this paper, we propose ensemble-bart based weighted Bellman backups that can be applied to most modern ooflpy RL. algoritmss, such as Q-learning and actor-critic algorithms. Our main idea is to reweight sample transitions based on uncertainty estimates from a Q-ensemble. Because prediction emors can be characterized by uncertainty estimates from ensembles (i.e., variance of predictions) as shown in Figure ((b), we find that the proposed method significantly improves the signal-to-noise in the Q-updates and stabilizes the learning process. Finally, we present unified framework, coined SUNRISE, that combines our weighted Bellman backups with an inference method that selects actions using highest upper-confidence bounds (UCB) for efficient exploration (Chen et al., 2017).

---

Under review as a conference paper at ICLR 2021
We find that these different ideas can be fruitfully integrated, and they largely complementary (see Figure I(a)..
We demonstrate the effectiveness of the proposed method using Soft Actor-Critic (SAC; Haarnoja et al. 2018) for continuous control benchmarks (specifically. OpenAI Gym (Brockman et al., 2016) and DeepMind Control Suite (Tassa al., 2018)) and Rainbow DQN (Hessel al., 2018) for dis- crete control benchmarks (specifically, Atari games (Bellemare et al., 2013)). In our experiments, SUNRISE consistently improves the performance of existing off-policy RL methods. Furthermore, we find that the proposed weighted Bellman backups yield improvements in environments with noisy reward, which have a low signal-to-noise ratio.
2 RELATED WORK
Off-poolicy RL algorithms. Recently, various off-policy RL algorithms have provided large gains in sample-effiiincy by reusing past experiences (Fujimoto et al., 2018; Haarnoja et al., 2018; Hessel et al., 2018). Rainbow DQN (Hessel et al., 2018) achieved lfate-of-the-ar performance on the Atari games (Bellemare et al., 2013) by combining several techniques, such as double Q- learning (Van Hasselt et al., 201 6) and distributional DQN (Bellemare et al., 2017). For continuous control tasks, SAC (Haarnoja et al., 2018) achieved state-of-the-art sample-efficiens results by in- corporating the maximum entropy framework. Our ensemble method brings orthogonal benefits and is complementary and compatible with these existing state-of-the art algorithms.
Stabilizing Q-learning. has been empirically observed that instability in Q-learning can caused by applying the Bellman backup on the learned value function (Hasselt, 2010; Van Hasselt et al., 2016; Fujimoto al., 2018; Song et al., 2019; Kim et al., 2019; Kumar et al., 2019; 2020). By fol- lowing the principle of double Q-learning (Hasselt, 2010; Van Hasselt et al., 2016), twin-Q trick (Fu- jimoto et al., 2018) was proposed to handle the overestimation of value functions for continuous control tasks. Song et al. (2019) and Kim et al. (2019) proposed to replace the max operator with Softmax and Mellowmax, respectively, to reduce the overestimation error. Recently, Kumar et al. (2020) handled the error propagation issue by reweighting the Bellman backup based on cumulative Bellman errors. However, our method is different in that we propose an alternative way that also utilizes ensembles to estimate uncertainty and provide more stable, higher-signal-to-noie backups.
Ensemble methods in RL. Ensemble methods have been studied for different purposes in RL. (Wier- ing & Van Hasselt, 2008; Osband et al., 201 6a; Anschel al., 2017; Agarwal et al., 2020; Lan al., 2020). Chua et al. (2018) showed that modeling errors in model-based RL can be reduced using an ensemble of dynamics models, and Kurutach et al. (2018) accelerated policy learning by generating imagined experiences from the ensemble of dynamics models. For efficient exploration, Osband al. (2016a) and Chen al. (2017) also leveraged the ensemble of Q-functions. However, most prior works have studied the various axes of improvements from ensemble methods in isolation, while we propose a unified framework that handles various issues in off-policy RL algorithms.
Exploration in RL. To balance exploration and exploitation, several methods, such as the maximum entropy frameworks (Ziehart, 2010; Haarnoja et al., 2018), exploration bonus rewards (Bellemare et al., 2016; Houthooft et al., 2016; Pathak et al., 2017: Choi et al., 2019) and randomization (Osband et al., 20;6,; have been proposed. Despite the success of these exploration methods, a potential drawback is that agents can focus irrelevant aspects of the environment because these methods do not depend on the rewards. To handle this issue, Chen et al. (2017) proposed an exploration strategy that considers both best estimates (i.e., mean) and uncertainty (i.e., variance) of Q-functions for discrete control tasks. We further extend this strategy to continuous control tasks and show that it can be combined with other techniques.
3 BACKGROUND
Reinforcement learning. We consider standard RL framework where agent interacts with an environment in discrete time. Formally, at each timestep t, the agent receives a state st from the environment and chooses an action a, based on its policy The environment returns reward r; and the agent transitions to the next state $1+1- The return R, = E <rt4 is the total accumulated rewards from timestep & with discount factor 7 E [0, 1). RI then maximizes the expected return.
2

---

Formally, consider ensemble of N SAC agents, i..., (Qe, Te. 1i=1, where 0 and Φ: denote the parameters of the i-th soft Q-function and policy.' Since conventional Q-learning is based on the Bellman backup in (2), it can affected by error propagation. Lec, error the target Q-function Qø(8441, ((+1) gets propagated into the Q-function Qo(st, 04) at the current state. In other words, errors in the previous Q-function induce the "noise" to the learning "signal" (i.e., true Q-value) of the current Q-function. Recently, Kumar al. (2020) showed that this error propagation can cause Under review as a conference paper at ICLR 2021
Bootstrap with random initialization
Actor 1 Critic
Ground fruth Predicted mean
Weighted Bellman backup
Traiting samples
Actor 2
Critic 2
UCB exploration
Action
Actor N
Replay buffer
Critic N
(a) SUNRISE: actor-critic version
(b) Uncertainty estimates
Figure (a) Illustration of our framework. We consider independent agents (i.e., no shared parameters between agents) with one replay buffer. (b) Uncertainty estimates from an ensemble of neural networks on toy regression task (see Appendix C for more experimental details). The black line is the ground truth curve, and the red dots are training samples. The blue lines show the mean and variance of predictions over ten ensemble models. The ensemble can produce well-calibrated uncertainty estimates (i.e., variance) unseen samples.
Soft Actor-Critic. SAC (Haarnoja et al., 2018) is an off-policy actor-critic method based on the maximum entropy RL framework (Ziebart, 2010), which encourages the robustness noise and ex- ploration by maximizing weighted objective of the reward and the policy entropy (see Appendix A for further details). To update the parameters, SAC alternates between a soft policy evaluation and soft policy improvement. At the soft policy evaluation step, a soft Q-function, which is modeled as a neural network with parameters updated by minimizing the following soft Bellman residual:
Certtic (0) ET,=s(Co (T,,
[SAC
(T 0) (Qo(st, a.) -7 [Qo(s++1, (141)) log ग. Ta(a (
(2)
where Ti = (St, at, Tt, 8t+1) transition, a replay buffer, are the delayed parameters, and a temperature parameter. At the soft policy improvement step, the policy with its parameter is updated by minimizing the following objective:
Lactor (Ø) = Es,~B [C+ (st, o)], where (s Φ) = Ea,~T [a log To (s)) (o(st. a.]] (3)
Here, the policy modeled as a Gaussian with mean and covariance given by neural networks to handle continuous action spaces.
4 SUNRISE
In this section, propose the ensemble-basee weighted Bellman backups, and then introduce SUN- RISE:
Simple UNified framework for ReInforcement learning using enSEmbles, which combines various ensemble methods. In principle, our method can be used in conjunction with most modern off-policy RL algorithms, such as SAC (Haarnoja et al., 2018) and Rainbow DQN (Hessel et al., 2018). For the exposition, we describe only the SAC version in the main body. The Rainbow DQN version follows the same principles and is fully described in Appendix B.
4.1 WEIGHTED BELLMAN BACKUPS TO IMPROVE SIGNAL- TO-NL-TO--NISE IN Q-UPDATES
inconsistency and unstable convergence. To mitigate this issue, for each agent i, we consider a weighted Bellman backup as follows:
LwQ (Tx, 0:)
w (8t-1: 0++1) (Qo, (8t, az) 7 (Q0, (st+1, 44+1) - a log n (ete (4)
We remark that each Q-function Qo, (s, .a) has unique target Q-function Qo, (8, a).

---

Under review as a conference paper at ICLR 2021
where Ti (st, ag: (t, St+1) 1 is transition, a++1 TT(aass). and w(s, a) is confidence weight based on ensemble of target Q-functions:
w(s, a) a (-Qata(s, a) * T) + 0..,
where T > O is temperature, is the sigmoid function, and Qata(s, a) is the empirical standard deviation of all target Q Q-functions {Qa, 1 Note that the confidence weight is bounded in [0.5, 1.0] because standard deviation is always positive. The proposed objective LwQ down-weights the sample transitions with high variance across target Q-functions, resulting loss function for the Q-updates that has a better signal-to-noise ratio.
4.2 COMBINATION WITH ADDITIONAL TECHNIQUES THAT LEVERAGE ENSEMBLES
We integrate the proposed weighted Bellman backup with UCB exploration into single framework by utilizing the bootstrap with random initialization.
Bootstrap with random initialization. To train the ensemble of agents, we use the bootstrap with random initialization (Efron, 1982; Osband et al., 2016a), which enforces the diversity between agents through two simple ideas: First, we initialize the model parameters of all agents with random parameter values for inducing an initial diversity in the models. Second, we apply different samples to train each agent. Specifically, for each SAC agent i each timestep 4, we draw the binary masks We, from the Bernoulli distribution with parameter B E (0. 1], and store them in the replay buffer. Then, when updating the model parameters of agents, we multiply the bootstrap mask to each objec- tive function, such as: m2 (st, Ø.) and m,,CCwQ(TT 0,) in (3) and (4), respectively. We remark that Osband et al. (2016a) applicd this simple technique to train an ensemble of DQN (Mnih et al., 2015) only for discrete control tasks, while apply SAC (Haarnoja et al., 2018) and Rainbow DQN (Hessel et al., 2018) for both continuous and discrete tasks with additional techniques.
UCB exploration. The ensemble can also be leveraged for efficient exploration (Chen et al., 2017; Osband et al., 201 6a) because it can express higher uncertainty on unseen samples. Motivated by this, by following the idea of Chen et al. (2017), consider optimism-based exploration that chooses the action that maximizes
ax max (Qnean (sc, a) + (8t,
where Gnoan (s, a) and Quta(s, a) the empirical mean and standard deviation of all Q-functions (Qo, J. and the > is a hyperparameter. This inference method can encourage exploration by adding an exploration bonus (i.e., standard deviation Qata) for visiting unseen state-action pairs similar to the UCB algorithm (Auer et al., 2002). We remark that this inference method was origi- nally proposed in Chen al. (2017) for efficient exploration in discrete action spaces. However, in continuous action spaces, finding the action that maximizes the UCB is not straightforward To han- dle this issue, we propose a simple approximation scheme, which first generates N candidate action set from ensemble policies (π J.M and then chooses the action that maximizes the UCB (Line 4 in Algorithm 1). For evaluation, we approximate the maximum a posterior action by averaging the mean of Gaussian distributions modeled by each ensemble policy.
The full procedure of our unified framework, coined SUNRISE, summarized in Algorithm 1.
EXPERIMENTAL RESULTS
We designed our experiments answer the following questions:
Can SUNRISE improve off-policy RL algorithms, such SAC (Haarnoja et al., 2018) and Rainbow DQN (Hessel et al., 2018), for both continuous (see Table 1 and Table 2) and discrete (see Table 3) control tasks?
How crucial is the proposed weighted Bellman backups in (4) for improving the signal-to toisee in Q-updates (see Figure 2)?
Can UCB exploration be useful for solving tasks with sparse rewards (see Figure 3(b))?
Is SUNRISE better than single agent with more updates and parameters (see Figure 3(c))?
How does ensemble size affect the performance (see Figure 3(d))?
"We find that it is empirically stable to minimum value of weight w(ss, a) as 0.5.
4

---

Under review as a conference paper at ICLR 2021
Algorithm SUNRISE: SAC version
1: for each iteration do
2:
for each timestep do
3:
// UCB EXPLORATION
4:
Collect N action samples: A, {ar, To. (als;)] Di E (l....,N},
5:
Choose the action that maximizes UCB: a arg max Quesa (s,, az,s) +eQeaa(sea
e,,
6:
Collect state 8 and reward r; from the environment by taking action dz
7:
Sample bootstrap masks M, = {me, ~ Bernoulli (3) - E {1,.... N} Store transitions Tt (st, at St-1, re) and masks in replay buffer B B ((T,. end for
8:
9:
// UPDATE AGENTS VIA BOOTSTRAP AND WEIGHTED BELLMAN BACKUP
11:
for each gradient step do
12:
Sample random minibatch M for each agent do
13:
14:
Update the Q-function by minimizing Eド=ı 1 mijiaiaaaa (T₁, in (4)
15:
Update the policy by minimizing 1/ E m (s 61) in (3)
16:
end for
17:
end for
18:
end for
5.1 SETUPS
Continuous control tasks. We evaluate SUNRISE on several continuous control tasks using simu- lated robots from OpenAl Gym (Brockman et al., 2016) and DeepMind Control Suite (Tassa et al., 2018). For OpenAl Gym experiments with proprioceptive inputs (e.g.. positions and velocities), compare to PETS (Chua et al., 2018), a state-of-the-ar model-based RL method based on ensem- bles of dynamics models; POPLIN-P (Wang & Ba, 2020), state-of- ffate-of-the-art model-
RL method which uses a policy network to generate actions for planning; POPLIN-A (Wang & Ba, 2020), vari- of POPLIN-P which adds noise in the action space: METRPO (Kurutach al., 2018), hybrid RL method which augments TRPO (Schulman et al.. 2015) using ensembles of dynamics models; and ofate-of-the-art model-free RL methods, TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018). For our method, we consider combination of SAC and SUNRISE, as described in Algorithm 1. Following the setup Wang & Ba (2020) and Wang et al. (2019), we report the mean and standard deviation across ten runs after 200K timesteps five complex environments: Cheetah, Walker, Hopper, Ant and SlimHumanoid with early termination (ET). More experimental details and learning curves with 1M timesteps are in Appendix D.
For DeepMind Control Suite with image inputs, we compare to PlaNet (Hafner et al., 2019), a model-based RL method which learns a latent dynamics model and uses it for planning; Dreamer (Hafner et al., 2020), a hybrid RL method which utilizes the latent dynamics model to generate syn- thetic roll-outs; SI.AC (Lee et al., 2020), hybrid RL method which combines the latent dynamics model with SAC; and three state-of-the-arr model-free RL methods which apply contrastive learning (CURL: Srinivas et al. 2020) data augmentation (RAD (Laskin al., 2020) and DrQ (Kostrikov et al., 2020)) to SAC. For our method, we consider a combination of RAD (i.e., SAC with ran- dom crop) and SUNRISE. Following the setup in RAD, report the mean and standard deviation across five runs after 100k (i.e., low sample regime) and 500k (i.e., asymptotically optimal regime) environment steps on six environments: Finger-spin, Cartpole-swing, Reacher-easy, Cheetah-run, Walker-walk, and Cup-catch. More experimental details and learning curves are in Appendix F.
Discrete control benchmarks. For discrete control tasks, we demonstrate the effectiveness of SUN- RISE on several Atari games (Bellemare et al., 2013). We compare to SimPLe (Kaiser et al., 2020), a hybrid RL method which updates the policy only using samples generated by learned dynamics model; Rainbow DQN (Hessel et al., 2018) with modified hyperparameters for sample-efficiency (van Hasselt et al., 2019); Random agent (Kaiser et al., 2020); two sfate-of-the-aee model-free RL methods which apply the contrastive learning (CURL; Srinivas al. 2020) and data augmentation (DrQ; Kostrikov et al. 2020) to Rainbow DQN; and Human performances reported Kaiser et al. (2020) and van Hasselt et al. (2019). Following the setups in SimPLe, we report the mean across
5

---

Under review as a conference paper at ICLR 2021
Cheetah
Walker
Hopper
Ant
SlimHumanoid-EFT
PETS
2288.4
1019.0
282.5
501.6
114.9 621.0
1165.5 + 226.9
2055.1 771.5
POPLIN-A
1562.8
1136.7
-105.0 249.8
202.5 962.5
1148.4 + 438.3
POPLIN-P
4235.0
1133.0
478.8
2055.2 = 613.8
2330.1 + 320.9
METRPO
2283.7 + 900.4
-1609.3 657.5
1272.5 = 500.9
282.2 + 18.0
76.1
TD3
3015.7 +: 969.8
-516.4 812.2
1816.6 = 994.8
870.1 + 283.8
1070.0 168.3
SAC
4474.4 + 700.9
299.5 + 921.9
1781.3 + 737.2
979.5 + 253.2
1371.8 + 473.4
SUNRISE
4501.8 + 443.8
1236.5 + 1123.9
2643.2 + 472.3
1502.4 t 483.5
1926.6 375.0
Table 1: Performance on OpenAI Gym at 200K timesteps. The results show the mean and standard deviation averaged over ten runs, and the best results are indicated in bold. For baseline methods, we report the best number in prior works (Wang Ba, 2020; Wang et al., 2019).
500K step
PlaNet
Dreamer
SLAC
CURL
DrQ
RAD
SUNRISE
Finger-spin
561 +
796 +
673
926 +
938 +
975 +
983 +1
Caripole-swing
475 - 71
762 + 27
845 45
868 - 10
873 =
876 +
Reacher-easy
210 +
793 + 164
929 +
942 +
916 +
982 +
Cheetah-run
305
570 +
640
518
660 96
624 10
678
Walker-walk
351 + s8
897 + 49
842 + 51
902 + 43
921 + 45
938 +
953 + 13
Cup-catch
460 +
879
852
959 +
963 9
966 =
969 +
100K step
Finger-spin
136 1 216
341 + 70
693 + 141
767 1 56
901 + 101
811 + 146
905 1 57
Cartpole-swing
297 +
326 +
582 + 146
759 +
373 +
591 +
Reacher-easy
1 50
314 1
538 1 233
+ 213
567 54
722
Cheetah-run
138 1: 88
235 1 137
319 1 56
299 1 48
344
381 1 79
413 1: 35
Walker-walk
224 +
277
+
361 +
403 +
612 164
641 + 89
667 +
Cup-catch
0.:00
246
+
512 - 110
769 43
9131 53
666 181
633 1: 241
Table 2: Performance on DeepMind Control Suite at 100K and 500K environment steps. The results
show the mean and standard deviation averaged five runs, and the best results are indicated in bold. For baseline methods, we report the best numbers reported in prior works (Kostrikov et al., 2020).
three runs after 100K interactions (i.e., 400K frames with action repeat 4). For our method, we consider combination of sample-efficient versions of Rainbow DQN and SUNRISE (see Algo- rithm 3 in Appendix B). More experimental details and learning curves are in Appendix G.
For our method, we do not alter any hyperparameters of the original RL algorithms and train five en- semble agents. There are only three additional hyperparameters 3, T. and À for bootstrap, weighted Bellman backup, and UCB exploration, where provide details in Appendix D, F, and G.
5.2 COMPARATIVE EVALUATION
OpenAl Gym. Table shows the average returns evaluation roll-outs for all methods. SUN- RISE consistently improves the performance of SAC across all environments and outperforms the model-based RL methods, such as POPLIN-P and PETS, on all environments except Ant and SlimHumanoid-ETT. Even though cent emphasis on making RL more sample efficient, we find that the gain from SUNRISE becomes even more significant when training longer (see Figure 3(c) and Appendix D). We remark that SUN- RISE is more compute-efficient than modern model- RI. methods, such as POPLIN and PETS, because they also utilize ensembles (of dynamies models) and perform planning to select actions. Namely, SUNRISE is simple to implement, computationally efficient, and readily parallelizable.
focus on performance after small samples because of the
DeepMind Control Suite. As shown Table 2, SUNRISE also consistently improves the perfor- mance of RAD (i.e., SAC with random crop) all environments from DeepMind Control Suite. This implies that the proposed method can be useful for high-dimensional and complex input obser- method outperforms existing pixel-based RL methods in almost all environ- ments. We remark that SUNRISE can also be combined with DrQ, and expect that can achieve better performances Cartpole-swing and Cup-catch 100K environment steps.
vations. Moreover,

---

Under review as a conference paper at ICLR 2021
Game
Human
Random
SimPLe
CURL
DrQ
Rainbow
SUNRISE
Alien
7127.7
227.8
616.9
558.2
761.4
789.0
872.0
Amidar
1719.5
5.8
88.0
142.1
97.3
118.5
122.6
Assault
742.0
222.4
527.2
600.6
489.1
413.0
594.8
Asterix
8503.3
210.0
1128.3
734.5
637.5
533.3
735.0
BankHeist
753.1
14.2
34.2
131.6
196.6
97.7
266.7
BattlZZone
37187.5
2360.0
5184.4
14870.0
13520.6
7833.3
15700.0
Boxing
12.1
0.1
9.1
1.2
6.9
0.6
6.7
Breakout
30.5
1.7
16.4
4,9
14.5
2.3
1.8
ChopperCommand
7387.8
811.0
1246.9
1058.5
646.6
590.0
1040.0
CrazyClimber
35829.4
10780.5
62583.6
12146.5
19694.1
25426.7
22230.0
DemonAttack
1971.0
152.1
208.1
817.6
1222.2
688.2
919.8
Freeway
29.6
0.0
20.3
26.7
15.4
28.7
30.2
Frostbite
4334.7
65.2
254.7
1181.3
449.7
1478.3
2026.7
Gopher
2412.5
257.6
771.0
669.3
598.4
348.7
654.7
Hero
30826.4
1027.0
2656.6
6279.3
4001.6
3675.7
8072.5
Jameshond
302.8
29.0
125.3
471.0
272.3
300.0
390.0
Kangaroo
3035.0
52.0
323.1
872.5
1052.4
1060.0
2000.0
Krull
2665.5
1598.0
4539.9
4229.6
4002.3
2592.1
3087.2
KungFuMaster
22736.3
258.5
17257.2
14307.8
7106.4
8600.0
10306.7
MsPacman
6951.6
307.3
1480.0
1465.5
1065.6
1118.7
1482.3
Pong
14.6
-20.7
12.8
-16.5
-11.4
-1..0
-19.3
PrivateEye
69571.3
24.9
58.3
218.4
49.2
97.8
100.0
Qbert
13455.0
163.9
1288.8
1042.4
1100.9
646.7
1830.8
RoadRunner
7845.0
11.5
5640.6
5661.0
8069.8
9923.3
11913.3
Seaquest
42054.7
68.4
683.3
384.5
321.8
396.0
570.7
UpNDown
11693.2
533.4
3350.3
2955.2
3924.9
3816.0
5074.0
Table 3: Performance Atari games at 100K interactions. The results show the scores averaged three runs, and the best results are indicated in bold. For baseline methods, we report the best numbers reported in prior works (Kaiser et al., 2020; van Hasselt et al., 2019).
SAC
DisCor
SUNRISE (wilhout WBB)
SUNRISE (with W8B)
10.000
1000
1000
5000
4000
1000
50m10h
-2000
1+107
Peigi
-3000
1-107
Swip
2--04
2010
Thaokgo
Tenoolapa
Throaapo
Sraoogo
Tamoulopp
Figure 2: Learning curves on OpenAl Gym with noisy rewards. To verify the effects of the weighted Bellman backups (WBB), consider SUNRISE with WBB and without WBB. The solid line and shaded regions represent the mean and standard deviation, respectively, across four runs.
Atari games. We also evaluate SUNRISE on discrete control tasks from the Atari benchmark using Rainbow DQN. Table shows that SUNRISE improves the performance of Rainbow in almost all environments, and outperforms the state-of-the CURL and SimPLe Here, we remark that SUNRISE is also compatible with CURL, which could enable even better performance. These results demonstrate that SUNRISE general approach.
out of 26 Atari games.
5.3 ABLATION STUDY
Effeets of weighted Bellman backups. To verify the effectiveness of the proposed weighted Bell- man backup (4) in improving signal-to-noise in Q-updates, we evaluate on modified OpenAl Gym environments with noisy rewards. Following Kumar et al. (2019), we add Gaussian noise to the reward function: a)) r(s,a) a) where NN 1) only during training, and report the deterministic ground-truth reward during evaluation. For our method, we also consider variant of SUNRISE, which updates Q-functions without the proposed weighted Bellman backup isolate its effect. We compare to DisCor (Kumar et al., 2020), which improves SAC by reweighting the Bellman backup based on estimated cumulative Bellman errors (see Appendix E for more details).

---

Under review as a conference paper at ICLR 2021
14,00
2JARGE Leth UCB
30,003
CeC perCE opdoards
1500
Bt Votul 2010
10005
4/K0
SUNVIGL 1irh (993)
600
20.00
1100
4000
200
2005
2+10" 2xte"
4: 10
13-10" 15-10" 2941
Dw sement sias
Tredsps
Tremnpee
(a) Large noise
(b) Sparse reward
(c) Gradient update
(d) Ensemble size
Figure 3: (a) Learning curves of SUNRISE with random weight (RW) and the proposed weighted Bellman backups (WBB) on the SlimHumanoid environment with noisy rewards.
(b) Effects of UCB exploration on the Cartpole environment with sparse reward. (c) Learning curves of SUN- RISE and single agent with h hidden units and five gradient updates per each timestep the SlimHumanoid-FF environment. (d) Learning curves of SUNRISE with varying values of ensemble size N on the Ant environment.
Figure 2 shows the learning curves of all methods OpenAl Gym with noisy rewards. The pro- posed weighted Bellman backup significantly improves both sample-cfficiency and asymptotic per- formance of SUNRISE, and outperforms baselines such SAC and DisCor. One can note the performance gain due to our weighted Bellman backup becomes more significant in complex envi- ronments, such SlimHumanoid-ET. We remark that DisCor still suffers from error propagation issues in complex environments like SlimHHmanoid-ET and Ant because there are some approxima- tion errors in estimating cumulative Bellman errors (see Section 6.1 for more detailed discussion). These results imply that errors in the target Q-function can be characterized by the proposed confi- dent weight in equation effectively.
We also consider another variant of SUNRISE, which updates Q-functions with random weights sampled from [0.5, 1.0] uniformly at random. In order to evaluate the performance of SUNRISE, we increase the noise rate by adding Gaussian noise with a large standard deviation to the reward function: r'(s, a) = r(s,a) + z, where ~ (O,0 5). Figure 3(a) shows the learning curves of all methods on the SlimHumanoid-ET environment over 10 random seeds. First, one can not that SUNRISE with random weights (red curve) is worse than SUNRISE with the proposed weighted Bellman backups (blue curve). Additionally, even without UCB exploration, SUNRISE with the proposed weighted Bellman backups (purple curve) outperforms all baselines. This implies that the proposed weighted Bellman backups can handle the error propagation effectively even though there is a large noise in reward function.
Effects of UCB exploration. To verify the advantage of UCB exploration in (6), we evaluate on Cartpole-swing with sparse-reward from DeepMind Control Suite. For our method, we con- sider variant of SUNRISE, which selects action without UCB exploration. As shown in Fig 3(b), SUNRISE with UCB exploration (blue curve) significantly improves the sample-efficiency on the environment with sparse rewards.
Comparison with a single agent with more updates/parameters. One concern in utilizing the ensemble method is that its gains may come from more gradient updates and parameters. To clarify this concern, we compare SUNRISE (5 ensembles using 2-layer MLPs with 256 hidden units each) single agent, which consists of 2-layer MLPs with 1024 (and 256) hidden units with 5 updates using different random minibatches. Figure 3(c) shows that the learning curves on SlimHumanoid- ET, where SUNRISE outperforms all baselines. This implies that the gains from SUNRISE can not be achieved by simply increasing the number of updates/parameters. More experimental results on other environments are also available in Appendix D.
Effects of ensemble size. We analyze the effects of ensemble size N on the Ant environment from OpenAl Gym. Figure 3(d) shows that the performance can be improved by increasing the ensemble size. but the improvement is saturated around N 5. Thus, we use five ensemble agents for all experiments. More experimental results on other environments are also available in Appendix D, where the overall trend is similar.
8

---

Under review as a conference paper at ICLR 2021
6 DISCUSSION
6.1
CONNECTION WITH DISCOR
Kumar et al. (2020) show that naive Bellman backups can suffer from slow learning in certain envi- ronments, requiring exponentially many updates. To handle this problem, they propose the weighted Bellman backups, which make steady learning progress by inducing some optimal data distribution (see (Kumar et al., 2020) for more details). Specifically, addition to a standard Q-learning. DisCor trains an error model A a), which approximates the cumulative sum of discounted Bellman rors over the past iterations of training. Then, using the error model, DisCor reweights the Bellman
backups based on confidence weight defined as follows: wss, a) x exp A..(a. where y
a discount factor and T is a temperature.
However, we remark that DisCor can still suffer from the eπror propagation issues because there is also approximation error in estimating cumulative Bellman errors. Therefore, we consider an alternative approach that utilizes the uncertainty from ensembles. Because it has been observed that the ensemble can produce well-calibrated uncertainty estimates (i.e., variance) on unseen sam- ples (Lakshminarayanan et al., 2017), we expect that the weighted Bellman backups based on en- sembles can handle error propagation more effectively. Indeed, in experiments, find that ensemble- based weighted Bellman backups can give rise to more stable training and improve the data-efficiency of various off- off-policy RL algorithms.
6.2 COMPUTATION OVERHEAD
One can expect that there is an additional computation overhead by introducing ensembles. When have N ensemble agents, our method requires Nx inferences for weighted Bellman backups and 2N x inferences (N for actors and N for critics). However, we remark that our method can be more computationally efficient because it is parallelizable. Also, as shown in Figure 3(c), the gains from SUNRISE can not be achieved by simply increasing the number of updates/parameters.
CONCLUSION
In this paper, we present the ensemble-based weighted Bellman backups, which compatible with various off- off-policy RL algorithms. By re-weighting target Q-values based uncertainty estimates, we stabilize and improve the learning process on both continuous and discrete control benchmarks. Additionally, we introduce SUNRISE, simple unified ensemble method, which integrates the pro- posed weighted Bellman backups with bootstrap with random initialization, and UCB exploration handle various issues in off-policy RL algorithms. Our experiments show that SUNRISE con- sistently improves the performances of existing off off-policy RL algorithms, such as Soft Actor Critic and Rainbow DQN, and outperforms state-of-the-at RL algorithms for both continuous and dis- crete control tasks on both low-dimensional and high-dimensional environments. We hope that SUNRISE could be useful to other relevant topics such as sim-to-real transfer (Tobin et al., 2017) imitation learning (Torabi al., 2018), understanding the connection between on-policy and off- policy RL (Schulman et al., 2017), offline RL (Agarwal et al., 2020), and planning (Srinivas et al., 2018; Tamar et al., 2016).
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. International Conference on Machine Learning, 2020.
Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson. On the model- based stochastic value gradient for continuous reinforcement learning. arXiv preprint arXiv:2008. 12775, 2020.
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dgnt Variance reduction and stabilization for deep reinforcement learning. In International Conference Machine Learning, 2017.

---

Under review as a conference paper at ICLR 2021
Peter Auer, Nicolo Cesa-Bianchi and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235 256,,,255666622 2002.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, 2016.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal Artificial Intelligence Research, 47: 253-279, 2013.
Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective reinforcement learning. In International Conference on Machine Learning, 2017.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 2016.
Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q- ensembles. arXiv preprint 17X6v.1706.0/555 2017.
Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and Honglak Lee. Contingency-ave exploration in reinforcement learning. In International Con- ference on Learning Representations, 2019.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. In Advances Neural Information Processing Systems, 2018.
Bradley Efron. The jackknife, the bootstrap, and other resampling plans, volume 38. Siam, 1982.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International Conference on Machine Learning, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- on Machine Learning, 2018
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, 2019.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
Hado V Hasselt. Double g-learning. Advances in Neural Information Processing Systems, 2010.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bital Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, 2016.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model-based reinforcement learning for atari. In International Conference on Learning Representations, 2020.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-basee robotic manipulation. Conference on Robot Learning, 2018.
10

---

Under review as a conference paper at ICLR 2021
Seungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris. Deepmellow: removing the need for a target network in deep q-learning. In International Joint Conference on Artificial Intelligence, 2019.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Sys- tems, 2019.
Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. In Advances in Neural Information Processing Systems, 2020.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations, 2018.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estimation bias of q-learning. In International Conference on Learning Representations, 2020.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. In Advances in Neural Information Processing Systems, 2020.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. In Advances in Neural Information Processing Systems, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems, 2016a.
Ian Osband, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via randomized value functions. In International Conference on Machine Learning, 2016b.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driver exploration by self-supervised prediction. In International Conference on Machine Learning, 2017.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference Learning Representations, 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, 2015.
John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q- 2017.
learning. arXiv preprint
David Silver. Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550,7676)335,, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Artfhur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 140- 1144, 2018.
Zhao Song, Ron Parr, and Lawrence Carin. Revisiting the softmax bellman operator: New benefits and new perspective. In International Conference on Machine Learning, 2019.
11

---

Under review as a conference paper at ICLR 2021
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks. In International Conference on Machine Learning, 2018.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa-
tions for reinforcement learning. In International Conference on Machine Learning, 2020.
Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks. In Advances in Neural Information Processing Systems, 2016.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: /8100000 2018.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In International Conference Intelligent Robots and Systems, 2017.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. Interna- lional Joint Conferences Artificial Intelligence Organizution, 2018.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. AAAI Conference on Artificial Intelligence, 2016.
Hado P van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in rein-
forcement learning? In Advances in Neural Information Processing Systems, 2019.
Oriol Vinyals, Igor Babuschkin, Wojeiech Czarnekki, Michael Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3500-354 2019.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. Interna- fional Conference on Learning Representations, 2020.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforce- ment learning. arXiv preprint arXiv: 1ttp::90000005555 2019.
Marco A Wiering and Hado Van Hasselt. Ensemble algorithms reinforcement learning. JEEE Transactions Systems, Mun, and Cybernetics, 38(4):930-936, 2008
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint 1010019100011411 2019.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
12

---

Under review as a conference paper at ICLR 2021
Appendix
A SUNRISE: SOFT ACTOR-CRITIC
Background. SAC (Haarnoja al., 2018) is a state-of-the-an off-policy algorithm for continuous control problems. SAC learns policy, Ts (a]s), and critic, Qp(s, a). and aims to maximize weighted objective of the reward and the policy entropy, Es [2, 7 + a To update the parameters, SAC alternates between soft policy evaluation and soft policy improve- ment. At the soft policy evaluation step. soft Q-function, which is modeled as neural network with parameters 0, is updated by minimizing the following soft Bellman residual:
Ceritic (0) En,- (C 0)],
Lo(TE, 0) = (Qa(st, az) Tt -B (0 ae+1) a log To To((4+1|$441))
where Ti (st, Tt, Tt, st+1) transition, is a replay buffer, 0 are the delayed parameters, and is temperature parameter. At the soft policy improvement step, the policy π with its parameter updated by minimizing the following objective:
CSAC (Ø) = E,,~B (st, Ø)], where L=(se, Eu (a log क (a(())))) (o 4z)]-
We remark that this corresponds to minimizing the Kullback- divergence between the policy and Boltzmann distribution induced by the current soft Q-function.
SUNRISE without UCB exploration. For SUNRISE without UCB exploration, we use random inference proposed in Bootstrapped DQN (Osband al., 6a), which randomly selects an index of policy uniformly at random and generates the action from the selected actor for the duration of that episode (see Line 3 in Algorithm 2).
Algorithm 2 SUNRISE: SAC version (random inference)
1: for each iteration do
2: // RANDOM INFERENCE
3: Select an index of policy using 7 (1, ,N
4: for each timestep do
5:
Get the action from selected policy: a, Téz (als:)
6:
Collect state 8+41 and reward r; from the environment by taking action aç
Sample bootstrap masks M, {m,,, Bernoulli (3) -- (1..... N)} 8: Store transitions Tę = (St, Gt, $1-1, re) and masks in replay buffer B t BU Mc)
7:
9:
end for
10:
// UPDATE AGENTS VIA BOOTSTRAP AND WEIGHTED BELLMAN BACKUP
11:
for each gradient step do
12:
Sample random minibatch My) B for each agent do
14:
Update the Q-function by minimizing 1/ ΣØ myaLwq (T)
Update the policy by minimizing 1/ ش m3, C end for
17: end for
end for
B EXTENSION TO RAINBOW DQN
B. PRELIMINARIES: RAINBOW DQN
Background. DQN algorithm (Mnih et al., 2015) learns Q-function, which is modeled as neural network with parameters 6, by minimizing the following Bellman residual:
DON (0) E,>B (Qo(st) re max Q7 Q5(5(41,
13

---

Under review as a conference paper at ICLR 2021
where (st, as,, ". $1+1) is transition, B is replay buffer, and 0 are the delayed parameters. Even though Rainbow DQN integrates several techniques, such as double Q-learning (Van Hasselt et al., 2016) and distributional DQN (Bellemare et al., 2017), applying SUNRISE to Rainbow DQN can be described based on the standard DQN algorithm. For exposition, we refer the reader to Hessel al. (2018) for more detailed explanations of Rainbow DQN.
Algorithm 3 SUNRISE: Rainbow version
1: for each iteration do for each timestep do
2:
3:
// UCB EXPLORATION
4:
Choose the action that maximizes UCB: a₁ = arg max Quean at,5) + XQuta at,,
ar,
5: Collect state $:+1 and reward from the environment by taking action de
6:
Sample bootstrap masks Mt {m,,, Bernoulli (3) - i {L..... N}}
7: Store transitions = (8c, Ge, 8,+1, rz) and masks in replay buffer B <- BU {(T), M))
8:
end for
9:
// UPDATE Q-FUNCTIONS VIA BOOTSTRAP AND WEIGHTED BELLMAN BACKUP
10:
for each gradient step do
11:
Sample random minibatch {(T,) M,)) B for each agent do
12:
13:
DON Update the Q-function by minimizing 1/3 ΣØ= mj WQ (T), 0.) 14: end for
end for
16:
end for
B.2 SUNRISE: RAINBOW DQN
Bootstrap with random initialization. Formally, we consider an ensemble Q-functions, ie.. {Qo: 1. where Og denotes the parameters of the i-th Q-function.' To train the ensemble of Q-functions, we use the bootstrap with random initialization (Efron, 1982; Osband et al., 201 2016 which enforces the diversity between Q-functions through two simple ideas: First, we initialize the model parameters of all Q-functions with random parameter values for inducing an imitial diversity in the models. Second, we apply different samples to train each Q-function. Specifically, for each Q-function i in each timestep i, we draw the binary masks m4, from the Bernoulli distribution with parameter 3 E (0, 1], and store them in the replay buffer. Then, when updating the model parameters of Q-functions, we multiply the bootstrap mask to each objective function.
Weighted Bellman backup. Since conventional Q-learning is based on the Bellman backup in equation 7, it can be affected by error propagation. Le., error in the target Q-function Q8(5 a(+1) gets propagated into the Q-function (s( az) at the current state. Recently, Kumar al. (2020) showed that this error propagation cause inconsistency and unstable convergence. To mitigate this issue, for each Q-function i, we consider a weighted Bellman backup as follows:
LWQ (Tr, 0c) w (+)+1) (Qn,(H,, (st, (z) - Tt - max Qø, (t)+
where = (st, Pt. 8141) is transition, and w(s) is confidence weight based on ensemble of target Q-functions:
w(s) a (-Quta()) * 7) + 0.5,
where T >>> 0 is temperature, is the sigmoid function, and Quea(s) is the empirical standard de- viation of all target Q-functions {max, Qa, (s, a) Note that the confidence weight bounded in [0.5, 1.0] because standard deviation is always positive.* The proposed objective C down- weights the sample transitions with high variance across target Q-functions, resulting in a loss func- tion for the Q-updates that has better signal-to-noise ratio. Note that combine the proposed
Herre, we remark that each Q-function has unique target Q-function.
We find that it is empirically stable to minimum value of weight w(s, a) as 0.5.
14

---

Under review as a conference paper at ICLR 2021
weighted Bellman backup with prioritized replay (Schaul al., 2016) by multiplying both weights to Bellman backups.
UCB exploration. The ensemble can also be leveraged for efficient exploration (Chen et al., 2017; Osband et al., 2016a) because can express higher uncertainty on unseen samples. Motivated by this, by following the idea of Chen et al. (2017), consider optimism- baasdd exploration that chooses the action that maximizes
a2 max (Quean (st, a) + (seta(tt,a))
where Quean (s, a) and (8 a) are the empirical mean and standard deviation of all Q-functions {Qe,), J.V V and the > 0 a hyperparameter. This inference method can encourage exploration by adding an exploration bonus (i.e., standard deviation Quua) for visiting unseen state-action pairs similar the UCB algorithm (Auer et al., 2002). This inference method was originally proposed in Chen et al. (2017) for efficient exploration in DQN, but we further extend it to Rainbow DQN. For evaluation, we approximate the maximum a posterior action by choosing the action maximizes the mean of Q-functions, i.e., = max, @nean (St, (st,a)). The full procedure is summarized in Algorithm 3.
C IMPLEMENTATION DETAILS FOR TOY REGRESSION TASKS
We evaluate the quality uncertainty estimates from an ensemble neural networks on a toy regression task. To this end, we generate twenty training samples drawn y x where ~ -N(0,,3)) 32), and train ten ensembles of regression networks using bootstrap with random ini- tialization. The regression network is fully-connected neural networks with 2 hidden layers and 50 rectified linear units in each layer. For bootstrap, we draw the binary masks from the Bernoulli distribution with mean 33 0.3. As uncertainty estimates, measure the empirical variance of the networks' predictions. As shown in Figure ((b), the ensemble can produce well-calibrated uncer- tainty estimates (i.e., variance) on unseen samples.
D EXPERIMENTAL SETUPS AND RESULTS: OPENAI GYM
Environments. We evaluate the performance of SUNRISE on four complex environments based on the standard bench-marking environments" from OpenAl Gym (Brockman et al., 2016). Note that we do not use a modified Cheetah environments from PETS (Chua al., 2018) (dented Cheetah in POPLIN (Wang& & Ba, 2020)) because it includes additional information in observations.
Training details. We consider a combination of SAC and SUNRISE using the publicly released implementation repository (https: gitphb.thhhhh.ogggg com/ wwwtchrrrllkkit without any modifica- tions on hyperparameters and architectures. For our method, the temperature for weighted Bellman backups is chosen from T E {10, 20, 50), the mean of the Bernoulli distribution is chosen from (0.5, 1.0). the penalty parameter chosen from A E [1, 5, 10), and we train five ensemble agents. The optimal parameters are chosen to achieve the best performance on training environ- ments. Here, we remark that training ensemble agents using same training samples but with dif- ferent initialization (i.e., 3 1) usually achieves the best performance in most cases similar to Osband et al. (2016a) and Chen et al. (2017). We expect that this is because splitting samples can reduce the sample-efficiency Also, initial diversity from random initialization be enough be- cause each Q-function has a unique target Q-function, i.e., target value is also different according to initialization.
Learning curves. Figure 4 shows the learning curves on all environments. One can note that SUNRISE consistently improves the performance of SAC by large margin.
Effects of ensembles. Figure 5 shows the learning curves of SUNRISE with varying values of ensemble size on all environments. The performance can be improved by increasing the ensemble size, but the improvement is saturated around N 5.
We used the reference implementation https: //gi thub. com com/wi 1 sonWang/HH//mbb (Wang al., 2019).
15

---

Under review as a conference paper at ICLR 2021
SAC (hr256, updaten1)
SAC (hm256, update
SAC (trm 1024, update
SUNRISE (hm256, models)
00.003
Happo
5000
Walke
6000
Chroraah
5000
7900
4000
Aarqe nive
Jucrge 14
1000
0000
3000
00.03-
2000
1000
1900
10-10"
5-10"
10430"
D4T2"
10-12"
10K10
6-10"
Figure 4: Learning curves of SUNRISE and single agent with hidden units and five gradient updates per each timestep OpenAl Gym. The solid line and shaded regions represent the mean and standard deviation, respectively, across four runs.
SAC
SUNRISE (Nm2)
SUNRISE (N=b)
SUNRISE (N=10)
30,003-
AMy
20,02-
2000
1020
2033
1500
500
10-10
-2000
S-lof
10-10-110 ithorig
-300
5-107 1CH1PHD-1100211
5e16f 10- 10-10000-00000000--00000
15-1050-100 Tenodao
0
SriF 10-18-15H ohowig"
Trassggo
Tenedapa
Tirexaso
Figure 5: Learning curves of SUNRISE with varying values of ensemble size N. The solid line and shaded regions represent the mean and standard deviation, respectively, across four runs.
E EXPERIMENTAL SETUPS AND RESULTS: NOISY REWARD
DisCor. DisCor (Kumar et al., 2020) was proposed to prevent the error propagation issue in Q- learning. In addition to standard Q-learning, DisCor trains error model (s, which ap- proximates the cumulative sum of discounted Bellman errors over the past iterations of training. Then, using the error model, DisCor reweights the Bellman backups based on a confidence weight defined as follows:
w(s, a) exp (_rAp(s,a)
where y is discount factor and T is a temperature. By following the setups in Kumar et al. (2020), we take a network with 1 extra hidden layer than the corresponding Q-network as an error model, and chose T 10 for all experiments. We update the temperature via a moving average and use the learning rate of 0.0003. We use the SAC algorithm as the RL objective coupled with DisCor and build top of the publicly released implementation repository (https: //tt:/ggittt //ggtbbbb...... com/ vi tttchr//rkkit).
16

---

Under review as a conference paper at ICLR 2021
Hyperparameter
Value
Hyperparameter
Value
Random crop
True
Initial temperature
0.1
Observation rendering
(100, 100)
Learning rate (fs, Re. Q+)
2c 4 cheetah, run
Observation downsampling
(84, 84)
le 3 otherwise
Replay buffer size
100000
Learning rate (a)
le 4
Initial steps
1000
Batch Size
512 (cheetah), 256 (rest)
Stacked frames
3
Q function EMA T
0.01
Action repeat
finger. spin: walker, walk
Critic target update freq
2
caripole, swingup
Convolutional layers
4
otherwise
Number of filters
Hidden units (MLP)
1024
Non-linearity Encoder FMA
ReLU
Evaluation episodes
10
0.05
Optimizer
Adam
Latent dimension
($,, 82) (o Qo) (91, (2) (a)
(9,.999)
Discount ?
.99
(99)
Table 4: Hyperparameters used for DeepMind Control Suite experiments. Most hyperparameters values are unchanged across environments with the exception for action repeat, learning rate, and batch size.
F EXPERIMENTAL SETUPS AND RESULTS: DEEPMIND CONTROL SUITE
Training details. We consider a combination of RAD and SUNRISE using the publicly released implementation repository (https: /ttt:///gitttt
com hom/MMiee shaLask /ik/ii// /ad) with a full list of hyperparameters in Table 4. Similar to Laskin et al. (2020), we use the same encoder architecture as in (Yarats et al., 2019), and the actor and critic share the same encoder to embed image observations. For our method, the temperature for weighted Bellman backups is chosen from T E {10, 100]. the mean of the Bernoulli distribution is chosen from B E {0.5, 1.0], the penalty parameter is chosen from À {1,5, 10). and train five ensemble agents. The optimal parameters are chosen to achieve the best performance on training environments. Here, we remark that training ensemble agents using same training samples but with different initialization (i.e., B 1) usually achieves the best performance in most cases similar to Osband et al. (2)16a) and Chen et al. (2017). We expect that this is because training samples can reduce the sample-efficiency. Also, initial diversity from random initialization can be enough because each Q-function has a unique target Q-function, i.e., target value is also different according to initialization.
Learning curves, Figure 6(g). 6(h), 6(). 6(),, 6(k), and 6(1) show the learning curves on all environ- ments. Since RAD already achieves the near optimal performances and the room for improvement small, we can see small but consistent gains from SUNRISE. To verify the effectiveness of SUN- RISE more clearly, we consider combination of SAC and SUNRISE in Figure 6(a), 6(b), 6(c), 6(d), 6(e), and 6(1), where the gain from SUNRISE is more significant.
G EXPERIMENTAL SETUPS AND RESULTS: ATARI GAMES
Training details. We consider combination of sample-efficient versions of Rainbow DQN and SUNRISE using the publicly released implementation repository (https:/ //t:///ttt h//ggtttt /ggithub..... com/ Ka schimi n/ Ra nnbow) without any modifications on hyperparameters and architectures. For our method, the temperature for weighted Bellman backups is chosen from T {10, 40). the mean of the Bernoulli distribution is chosen from 3 E {0.5, 1.0]. the penalty parameter is chosen from chosen to achieve the
A E (1, 10), and train five ensemble agents. The optimal parameters best performance on training environments. Here, we remark that training ensemble agents using same training samples but with different initialization (i.e., 3 1) usually achieves the best perfor- mance in most cases similar to Osband et al. (201 6a) and Chen et al. (2017). We expect that this is because splitting samples can reduce the sample-effiiiency. Also, initial diversity from random ini- tialization can be enough because each Q-function has unique target Q-function, i.c., target value also different according to initialization.
'oweeerr we remark that each agent does not share the encoders unlike Bootstrapped DQN (Oshand et al., 2016a).
17

---

Under review as a conference paper at ICLR 2021
900
800
600
SAC
SAC SUNRISE
700
SUNRISE
500
600
600
400-
Score
Score
Score
500
300
300
400
200
SAC
300
100
SUNRISE
200
100
200 Environment steps
400
100
300
400
200
500
Environment steps (K)
Environment steps (K)
(a) Finger-spin
(b) Cartpole-swing
(c) Reacher-casy
400
600
800
SAC
SUNRISE
500
SUNRISE
700
SUNRISE
300
600
Score
Score
400
Score
500
200
300
400
200
300
100
200
100
100
0-
100
200
300
400 500
0
100
200
300
400 500
100
200
300
400
500
Environment steps (K)
Environment steps (K)
Environment steps (K)
(d) Cheetah-run
Walker-walk
Cup-catch
1000
900
1000
800
900
900
700
Score
Score
Score
800
500
700
800
400
RAD SUNRISE
300
RAD SUNRISE
600
RAD SUNRISE
700
200
500
100
200 Environment steps
300
400
500
0
100
200
300
400 500
100
200
300 400
500
Environment steps (K)
Environment staps (K)
(g) Finger-spin
(h) Cartpole-swing
(i) Reacher-easy
800
1000
1000
700
SUNRISE
900
900
Soore
600
Score
Score
700
800
500
600
400
700
500
400
SUNRISE
SUNRISE
300
600
300
100
200
300
400
500
0
100 200
300 400
500
100
200
300
400 500
Environment steps (K)
Environment steps (K)
Environment steps (K)
(j) Cheetah-run
(k) Walker-walk
Cup-catch
Figure 6: Learning curves of (a-f) SAC and (g-I) RAD on DeepMind Control Suite. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
Learning curves. Figure 7, Figure 8 and Figure 9 show the learning curves on all environments.
18

---

Under review as a conference paper at ICLR 2021
1000
1200
700
Rainbow SUNRISE
Rainbow SUNRISE
Rainbow SUNRISE
800
900
Soore
Scong
Score
600
600
400
300
400
200
300
100
200
300
400
500
100
200
300
400 500
100
200
300
400 500
Environment steps (K)
Environment steps (K)
Environment steps (K)
Seaquest
(b) BankHeist
(c) Assualt
50,000
2500
1500
w
Rainbow SUNRISE
40,000
2000
1000
Score
30,000
Score
1500
Score
20,000
1000
500
10,000
Rainbow SUNRISE
500
Rainbow SUNRISE
500
100
500
200
300
Environment steps (K)
Environment steps (K)
Environment steps (K)
(d) CrazyClimber
(c) DemonAttack
(f) ChopperCommand
25,000
10,000
10,000
20,000
8,000
8,000
gorer
15,000
e6,000
e6,000
11,,000
654,000
004,000
5,000
Rainbow
2,000
Rainbow SUNRISE
2,000
Rainbow SUNRISE
SUNRISE
0
100 200 300
400 500
100 200 300
400
500
100
200
300 400 500
Environment steps (K)
Environment steps (K)
Environment steps (K)
(g) KunteeMaster
(h) Kangaroo
UpNDown
Figure 7: Learning curves Atari games. The solid line and shaded regions represent the mean and standard deviation, respectively, across three runs.
19

---

Under review as a conference paper at ICLR 2021
500
2000
20
Rainbow SUNRISE
Rainbow SUNRISE
Rainbow SUNRISE
400
1600
10
Score
300
Score
Score
1200
200
100
0
400
-20
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
Environment steps (K)
Environment steps (K)
Environment steps (K)
(a) Amidar
(b) Alien
(c) Pong
2000
3000
2500
1800
Spore
Score
Score
5
1200
0
1500
1000
800
-10
500
Rainbow SUNRISE
Rainbow SUNRISE
-15
Rainbow SUNRISE
400
-20
100
200
300
100
200 300 400 500
100
200
300
Environment steps (K)
Environment steps (K)
Environment steps (K)
(d) Frostbite
(c) MsPacman
(f) Boxing
800
4000
25,000
Rainbow SUNRISE
600
20,000
3000
Score
Score
Score
15,000
10,000
2000
200
Rainbow SUNRISE
Rainbow SUNRISE
1000
0
100
200 300
400 500
100
200
300
20
40
60
100
Environment steps (K)
Environment steps (K)
Environment steps (K)
(g) Jameshond
(h) Krull
(i) BattleZone
15,000
12,000
1000
Rainbow SUNRISE
Rainbow SUNRISE
12,000
10,000
800
8,000
Score
9,000
Score
Score
6,000
6,000
400
4,000
3,000
2,000
200
Rainbow
SUNRISE
20
80
100
20
40
60
80
100
40
80
100
Environment steps
Environment steps (K)
Environment steps (K)
(i) RoadRunner
(k) Hero
(1) Asterix
Figure 8: Learning curves on Atari games. The solid line and shaded regions represent the mean and standard deviation, respectively, across three runs.
20

---

Under review as a conference paper at ICLR 2021
Rainbow
Rainbow
Rainbow
6000
4
2000
Score
4000
Score
Score
3
2000
1000
1
-000
40
100
20
40
60
80
100
20
40
80
80
100
Environment steps (K)
Environment steps (K)
Environment steps
(a) PrivateEye
(b) Qbert
(c) Breakout
30
SUNRISE
600
Score
400
200
Rainbow
20
40
80
80
100
20
40
60
80
100
Environment steps (K)
Environment steps (K)
(d) Freeway
(e) Gopher
Figure 9: Learning curves on Atari games. The solid line and shaded regions represent the mean and standard deviation, respectively, across three runs.
21


Reviews:
[{"review_id": "kUmOpgOCP35", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary   \n\nThis paper proposes to weight the Bellman backups according to the empirical std of Q-functions estimated by ensemble method. The paper claims that this proposed idea stabilizes and improves the learning in both continuous and discrete control tasks. It then integrates this proposed weighted Bellman backups with two of the existing advantages of ensembles: bootstrap and ucb exploration to form a unifying framework namely SUNRISE. SUNRISE is then compared with Actor-Critic and Rainbow in both discrete and continuous control tasks.  \n\n### Strong points \n-\tClarity: The paper is well written and organized \n-\tEmpirical significance: The integrated framework SUNRISE appears to have promising (yet seemly not fully convincing) performance as compared to the prior frameworks in both discrete and continuous control tasks.\n\n### Weak points\n-\tNovelty: This work seems to have low novelty and low technicality. It combines several known results to integrate into a unifying framework using ensembles. The only new idea here is perhaps a specific way of reweighting the Bellman backups though the idea of reweighting the Bellman backups to stabilize the learning is already known e.g., Kumar et al. 2020. \n-\tSignificance: In addition, at the present form I find it hard to be convinced both empirically and theoretically (or at least more elaborate explanation or intuition) why the proposed weighted Bellman backups using empirical std of Q-functions improve the signal-to-noise in Q-updates as claimed in the paper (see Questions for the authors).   \n\n###  Questions for the authors \n-\tIn the last sentence of Section 4.1, the paper claims that “the proposed objective … has a better signal-to-noise ratio”. I would like the authors to elaborate in this claim. What is exactly considered signal and what is exactly consider noise in this context? Why down weighting the sample transitions with high variance across target Q-functions result in a better signal-to-noise ratio? Why does the weighting proposed in this paper have better signal-to-noise ratio than the weighting in Discor (Kumar et al. 2020)? \n-\tThe paper claims that the proposed weighted Bellman updates improve signal-to-noise in Q-updates but appears to show only one experimental setting (presented in Fig. 2) where the reward is perturbed with *a standard Gaussian noise*.   For simplicity for the moment let’s call by “reward-to-noise ratio” the ratio of the magnitude of the original reward signal r(s,a) to the magnitude of the added noise. Since Section 5.3, I assume that the “reward-to-noise ratio” has something to do with the signal-to-noise ratio mentioned in the paper. Then, how the performance of the proposed weighted Bellman updates when the “reward-to-noise ratio” varies?    \n\n###  My initial recommendation\nOverall, I vote for weak rejecting for the weak points mentioned above. \n\n### My final recommendation \nThe authors did not fully address my points. I remain my initial score and recommend for rejection. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "djRsDQUh2Yh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use uncertainty estimates from an ensemble of action-values, to provide a weighting on the updates in Q-learning. The main idea is to use the sigmoid of the negative of this uncertainty in the next state, to produce a weighting between 0.5 and 1 to downweight updates with high uncertainty targets. This uncertainty estimate from the ensemble is also used to improve exploration, in a combined algorithm called Sunrise that leverages learning an ensemble in these two ways. \n\nThe idea of using weighted Bellman updates is, as far as I know, novel. The evidence for the idea, however, needs more work. First, the weighted update in Eq (4) is not motivated from first principles. Second, the empirical evidence is weak because the experiments highlighting the role of the weighting do not demonstrate significant differences.  \n\nThe first issue is the justification for the approach. The ensemble of Q-learning agents is trained using the weighting, derived from that ensemble. There are natural questions as to the interaction between the ensemble uncertainty estimates and the ensemble estimates. Does it result in any instability? What is the final point of convergence? Does it change the solution?\n\nBut, one could argue that that is not much of a problem, since the weighting w(s,a) is always between 0.5 and 1, so it is not that skewed. Then the question arises how much it is helping, and why this small reduction in weight helps. This is particularly important to ask, considering the algorithm requires an ensemble to be learned, with subsets of data used for each action-value. There is a lot of effort expended for that weighting.   \n\nThe experiments then do include ablations, to examine the effect of these weightings. Unfortunately, the results are inconclusive. The experimental time spent must have been large to get all the results in this paper, across so many environments and algorithms. But, the ablations themselves are not sufficiently in-depth to provide insight into the idea and algorithm. The results in Figure 2 are key, since that figure examines Sunrise with and without the weighting. Due to the variance across runs, with only 4 runs, there are large standard errors (and so even larger 95% confidence intervals); it is hard to conclude that weighting is helping. The additional results in Figure 5 in the appendix have a similar issue.\n\nThe results in Figure 3, which motivate the exploration utility, are more clear in Cartpole. This provides some motivation for learning ensembles, so they can be used for exploration. But, this exploration approach with ensembles is an existing method. The main novelty in this work is the weighting. \n\nI highly recommend taking a few domains and carefully studying the impact of the weight. More runs would help for significance, as well as parameter sensitivity analysis to gain insight into the generality of the improvement. Sometimes performance gains are from hyperparameter tuning, rather than from the utility of an idea; here, you really want to know if and why this weighting improves performance. \n\nAs a more minor comment, Sunrise is pitched as combining three ideas for using ensembles: your weighting, bootstrapping and UCB exploration. However, I see Sunrise as combining two ideas: weighting and UCB exploration. The Bootstrap DQN approach gives you a way to learn your ensemble of bootstrap models, so that it provides a useful uncertainty estimate. Given that ensemble, you can then use it to compute a weighting and optimistic action. It would be more clear to separate it out that way, rather than saying \"Furthermore, since our weighted Bellman backups rely on maintaining an ensemble, we investigate how weighted Bellman backups interact with other benefits previously derived from ensembles: (a) Bootstrap; (b) UCB Exploration.\" The bootstrap is arguably not a benefit, but an approach to obtain confidence (uncertainty estimates). \n \nMinor comments:\n1. Bootstrap DQN is listed under \"Ensemble Methods in RL\", rather than under \"Exploration in RL\", but is it an exploration approach.\n2. \"Recently, Kumar et al. (2020) showed that this error propagation can cause inconsistency and unstable convergence.\" The terms inconsistency and unstable convergence should be explained, since they seem like technical terms. \n3. Bellman backup seems to be used to describe the squared error to the expectation over next action, in Equation (2), and then to a stochastic sample of the action in (4). Which is it?\n4. What is meant by the signal-to-noise in Q-updates? \n5. A natural baseline to include is to tune an agent that uses random weights between 0.5 and 1 in the update, but keeping other parts of Sunrise the same. The ablation removes the weighting all together, which is also important to include. But, it's worthwhile observing if random weights performs similarly, especially if that agent is tuned. \n\n------------ Update\nThank you for the clear reply. Unfortunately, I remain concerned about the significance of experiments. I mentioned above that 4 or 5 runs is typically not enough, and because the standard errors are overlapping, the differences could be due to chance. The addition of a result with 10 runs is a good step. But, as part of the reply, the authors state: \"Figure 3(a) shows the learning curves of all methods on the SlimHumanoid-ET environment over 10 random seeds. First, one can not that SUNRISE with random weights (red curve) is worse than SUNRISE with the proposed weighted Bellman backups (blue curve). Additionally, even without UCB exploration, SUNRISE with the proposed weighted Bellman backups (purple curve) outperforms all baselines. This implies that the proposed weighted Bellman backups can handle the error propagation effectively even though there is a large noise in reward function.\" However, if you look at this figure, the error bars all still overlap. 10 random seeds is still not enough. \n\nI am also not confident that the issue will be remedied, as the authors additionally state in the rebuttal: \"we believe that SUNRISE is evaluated in a broad collection of domains in the RL literature and the performance gap is also noticeable.\" An insignificant gap across many domains does not tell us anything. Actually, if you take the runs and tried to do significance tests by pooling all the runs across environments, then maybe the result might actually be significant. But, of course, there will be higher variance due to differences in the environments, so it is not obvious this would be true. Nonetheless, this could be a natural next step.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_26Mc5QHPg", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "= Overview = \n\nThe paper proposes SUNRISE, an approach to reinforcement learning that leverages ensembles of agents to build more robust RL updates. SUNRISE comprises a number of similar agents  (in the paper, SAC agents) that perform parallel updates. Sample transitions for which there is larger variability (across the ensemble) in the estimates of the next-step Q-values are down-weighted in the computation of the loss, thus potentially rendering the learned Q-function more robust to noise.\n\nThe proposed approach is then combined with bootstrapping masks and UCB exploration, and is shown to outperform a number of state-of-the-art approaches in several benchmark domains from the RL literature.\n \n= Positive points =\n\nThe paper is clearly written. Additionally, the proposed approach is sensible and the empirical evaluation is, in my perspective, quite comprehensive: SUNRISE is evaluated in a broad collection of domains in the RL literature. \n\n= Negative points =\n\nThe paper would benefit, in my opinion, from additional discussion regarding: (a) the impact of the use of bootstrap with random initialization; and (b) the computational complexity of SUNRISE (even if the paper does briefly discuss the latter in Section 5.2)\n\n= Comments = \n\nI quite enjoyed reading the paper. The problem addressed is a relevant problem in RL, and the approach proposed in the paper is, in my opinion, simultaneously simple and sensible. The paper provides a solid empirical evaluation, covering a broad range of domains and comparing with multiple state of the art approaches from the literature. The results show that SUNRISE compares favorably -- in terms of performance -- with several of these other methods in multiple domains.\n\nThere are, however, two aspects that I would like to see discussed at greater length. On one hand, the paper proposes the use of bootstrapping masks and random initialization to induce variety in the ensemble. While the paper introduces both bootstrapping and UCB exploration as a \"useful complement\", it seems to me that this is quite central to the performance of the algorithm. Is this correct? In fact, without this device, the agents in the ensemble would essentially train from the same replay buffer, so variability would only come from the initialization. It is a pity that this particular element isn't included in the ablation study, for I would like to gain a clearer understanding on how critical this device is for the performance of the algorithm.\n\nOne other aspect that I would like to see discussed is regarding the computational complexity of the proposed approach. The paper remarks that SUNRISE is more computationally efficient than competing methods such as POPLIN and PETS, and being an ensemble method, I expect it to be naturally heavier than non-ensemble approaches such as standard SAC. However, I would like to understand how much more computation such a method involves. In particular the computation of the Bellman weights requires multiple passes through the critic network, as does the UCB exploration policy, and I was wondering how much more computation this entails.\n\nIn spite of the above aspects, I again remark that I quite enjoyed the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cnjf0GYLbiw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission developed an ensemble based approach to weight the Bellman backups from different agents. As claimed by the authors, the proposed method can improve the signal-to-noise ratio. To further boost the performance, the authors combined the weighted backups with a few other techniques, including UCB exploration and Bootstrap. The authors finally tested the proposed method on both continuous and discrete reinforcement learning tasks, and showed the improved or competitive performance, compared with baselines.\n\nThe proposed ensemble-based approach is interesting and the authors conducted an extensive experiments to verify its empirical performance, which I really appreciate. Compared with other baselines, the performance gap is also noticeable. \n\nOn the other hand, I am a bit concerned about whether the improvement is indeed because of the weighted backups. For example, Figure 3(a) showed that if removing UCB, the performance for SUNRISE dropped a lot. I tried to see if there are any other ablation studies w.r.t. UCB on the results in tables (removing UCB and keeping other steps the same in Algorithm 1), but did not find them. Use of UCB seems orthogonal with the weighted backup, as one is focused on exploration and the other for Q updates. Therefore, it's a bit questionable whether UCB or the proposed weighted backups is the main factor for performance improvement.\n\nThe authors claimed a few times \"signal-to-noise ratio\". I hope there could be more rigor here. What exactly is the definition for this term? What are the signal and noise here?\n\nFurthermore, I also doubt about the fairness in Table 3: The results there are only for 100K interactions; however, when comparing with Figure 8, Rainbow has not become stable at 100K and the scores for some games are just too low (e.g., Breakout), compared with results in the Rainbow paper.\n\nCould you comment on the increased complexity, when employing multiple agents? There are a few recent papers on the weighted Q updates as well, e.g.,\n\nSong, Z., Parr, R. and Carin, L., 2019, May. Revisiting the softmax bellman operator: New benefits and new perspective. In International Conference on Machine Learning (pp. 5916-5925).\n\nKim, S., Asadi, K., Littman, M. and Konidaris, G., 2019, August. Deepmellow: removing the need for a target network in deep Q-learning. In Proceedings of the Twenty Eighth International Joint Conference on Artificial Intelligence.\n\nThese papers avoid the need of multiple agents and show the benefits of weighted updates, which the authors need to discuss.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is acknowledged by all the reviewers as making a novel contribution -- the proposal to reweight state-action pairs depending on the variation in their Q-value estimates during learning. However, despite its extensive reporting of numerical experiments, its arguments in favor of the proposed approach are found to be wanting on both empirical and theoretical fronts. Reviewer 3 points out (correctly, in my opinion) that 5 (or even 10 in the updated version) independent trials are not sufficient to establish the validity of the approach up to statistical significance, and that even a well-reasoned heuristic explanation of why reweighting is expected to work in terms of reducing Q-value estimation error is missing. I agree with this point, which also struck me when reading the submission myself, that at the very least, the submission ought to contain a basic (and not necessarily rigorous) argument as to why the variance reduction ostensibly achieved due to reweighting should lead the estimation algorithm to the right Q-function in a general function approximation setting. For instance, even in the simplest multi-armed bandit setting, it is of interest to ask why this procedure should perform consistently without introducing unwanted bias in an unforeseen sense, and a clear explanation offered for this would be interesting. Another important concern that most reviewers are left with is about the lack of sufficient insight into the action of the UCB mechanism against the backdrop of the reweighting procedure (reviewers 1, 2, 3). I hope that the author(s) assimilate the feedback to strengthen the paper's main pitch further and make a strong case in the near future. 
Decision: Reject