Tactical Optimism and Pessimism for Deep Reinforcement Learning
Authors: Ted Moskovitz
Jack Parker-Holder
Aldo Pacchiano
Michael Arbel
Michael Jordan
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
deep reinforcement learning
optimism

Abstract: In recent years, deep off-policy actor-critic algorithms have become a dominant approach to reinforcement learning for continuous control. One of the primary drivers of this improved performance is the use of pessimistic value updates to address function approximation errors, which previously led to disappointing performance. However, a direct consequence of pessimism is reduced exploration, running counter to theoretical support for the efficacy of optimism in the face of uncertainty. So which approach is best? In this work, we show that the most effective degree of optimism can vary both across tasks and over the course of learning. Inspired by this insight, we introduce a novel deep actor-critic framework, Tactical Optimistic and Pessimistic (TOP) estimation, which switches between optimistic and pessimistic value learning online.  This is achieved by formulating the selection as a multi-arm bandit problem. We show in a series of continuous control tasks that TOP outperforms existing methods which rely on a fixed degree of optimism, setting a new state of the art in challenging pixel-based environments. Since our changes are simple to implement, we believe these insights can easily be incorporated into a multitude of off-policy algorithms. 

Tactical Optimism and Pessimism for Deep Reinforcement Learning
Ted Moskovitz
Jack Parker-Holder University of Oxford j jackph@robots. ox. ac.
Gatsby Unit, UCL
ted0gatsby. ucl ac uk
Aldo Pacchiano
Michael Arbel
Microsoft Research
Université Grenoble Alpes, Inria, CNRS*
opacchiano@miicroooft..oo
michael. n. arbel@gmai..comm com
Michael I. Jordan
University of California, Berkeley jordanêcs Aardan@ss.berkeleey edu
Abstract
In recent years, deep off-policy actor-critic algorithms have become a dominant approach reinforcement learning for continuous control. One of the primary drivers of this improved performance the use of pessimistic value updates to address function approximation errors, which previously led disappointing performance. However, a direct consequence of pessimism is reduced exploration, running counter to theoretical support for the efficacy of optimism in the face of uncertainty. So which approach is best? In this work, we show that the most effective degree of optimism can vary both across tasks and over the course of learning. Inspired by this insight, we introduce novel deep actor-critic framework, Tactical Optimistic and Pessimistic (TOP) estimation, which switches between optimistic and pessimistic value learning online. This is achieved by formulating the selection as a multi-arm bandit problem. We show in series of continuous control tasks that TOP outperforms existing methods which rely fixed degree of optimism, setting a new state the art challenging pixel-based based environments. Since our changes are simple to implement, we believe these insights can easily be incorporated into multitude of off-policy algorithms.
Introduction
Reinforcement learning (RI.) has begun to show significant empirical success in recent years, with value function approximation via deep neural networks playing fundamental role in this success [37
49 55 However, this success has been achieved in a relatively narrow set of problem domains, and
an emerging of challenges arises when one considers placing RL systems in larger systems. In particular, the use of function approximators can lead to positive bias in value computation [53], and therefore systems that surround the learner do receive an honest assessment of that value. One can attempt to turn this vice into virtue, by appealing to a general form of the optimism-unde under- uncertainty principle- principle-overestmmaaee overestimation actions that would otherwise not be explored. Such exploration can be dangerous, however, there is not a clear understanding of the nature of the overestimation.
the expected reward can trigger exploration of states and
Work mostly completed at the Gatsby Unit.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

This tension has not been resolved in the recent literature on RL approaches to continuous-com problems. On the one hand, some authors seek correct the overestimation, for example by using the minimum of two value estimates as form of approximate lower bound [20]. This approach can seen as a form of pessimism with respect to the current value function. On the other hand, [14] have argued that the inherent optimism of approximate value estimates is actually useful for encouraging exploration of the environment and/or action space. Interestingly, both sides have used their respective positions to derive
algorithms. How can this be, if their views are seemingly opposed? Our key hypothesis the following:
The degree of estimation bias, and subsequent efficacy of an optimistic strategy, varies as a function of the environment, the stage of optimization, and the overall context which learner is embedded.
This hypothesis motivates us view optimisn/pessimism as a spectrum and investigate procedures that actively move along that spectrum during the learning process. We operationalize this idea by measuring two forms of uncertainty that arise during learning: aleatoric uncertainty and epistemic uncertainty. These notions of uncertainty, and their measurement, are discussed in detail in Section 5.1 We then further aim to control the effects of these two kinds of uncertainty, making the following learning-the theorreicc assertion:
When the level of bias unknown, an adaptive strategy can be highly effective.
In this work, we investigate these hypotheses via the development of a new framework for value estimation in deep RL that we refer to Tactical Optimism and Pessimism (TOP). This approach acknowledges the inherent uncertainty in the level estimation bias present, and rather than adopt a blanket optimistic pessimistic strategy, estimates the optimal approach the fly, by formulating the optimism/pessimisst dilemma as a multi aelt--armed bandit problem. Furthermore, TOP explicitly isolates the aleatoric and epistemic uncertainty by representing the environmental return using distributional critic and model uncertainty with an ensemble. The overall concept is summarized in Figure
We show series of experiments that not only does the efficacy of optimism indeed vary as we suggest, but TOP is able to capture the best of both worlds, achieving a state of the art for challenging continuous control problems.
Our main contributions are as follows:
Our work shows that the efficacy of optimism for a fixed function approximator varies across environments and during training for reinforcement learning with function approximation.
We propose novel framework for value estimation, Tactical Optimism and Pessimism (TOP), which learns to balance optimistic and pessimistic value estimation online. TOP frames the choice of the degree of optimism or pessimism as multi-armed bandit problem.
Our experiments demonstrate that these insights, which require only simple changes to popular algorithms, lead state-of-the-art results on both state- and pixel-based control.
2
Related Work
episudl 1,2,...
Bandit Alganthm
6m
Much of the recent success of off-policy actor- critic algorithms build on DDPG [34] which extended the deterministic policy gradient approach to off-policy learning with deep net- works, using insights from DQN [37] Like D4PG H combine DPG with distributional value estimation. However, unlike D4PG, we use two critics, quantile representation rather than a categorical distribution [10), and, criti- cally, we actively manage the tradeoff between optimism and pessimism. We also note sev- eral other success stories in the actor-critic vein, including TD3, SAC, DrQ, and PI-SAC
timestep
49
Actar T t
Critics
Env
Replay Buffer
(S Pts 8641)
Figure 1: Visualization of the TOP framework. Blue arrows denote stochastic variables.

---

20 21 58 33
these represent the state-of-the-
art for continuous control and will serve as a baseline for our experiments.
The principle of optimism in the face of uncertainty S RO 3
provides a design tool for algorithms that trade off exploitation (maximization of the reward) against the need explore state-action pairs with high epistemic uncertainty. The theoretical tool for evaluating the success of such designs is the notion of regret, which captures the loss incurred by failing to explore. Regret bounds have long been used in research on multi-armed bandits, and they have begun to become more prominent in RL as well, both in the tabular setting 126 I8
g 55 and in the setting of function approximation (28 / However, optimistic approaches have had limited empirical success when combined with deep neural networks in RL [4. To be successful, these approaches need to be optimistic enough to upper bound the true value function while maintaining low estimation error [AI). This becomes challenging when using function approximation, and the result is often an uncontrolled, undesirable overestimation bias.
Recently, there has been increasing evidence in support of the efficacy of adaptive algorithms An example is Agent57 [5), the first agent to outperform the human baseline for all 57 games in the Arcade Learning Environment M. Agent57 adaptively switches among different exploration strategies. Our approach differs in that it aims to achieve a similar goal by actively varying the level of optimism in its value estimates.
6 48 46 E.
Finally, our work is also related to automated RL (AutoRL), as we can consider TOP to be an example of on-the-fly learning procedure IS
An exciting area of future work will be consider the interplay between the degree of optimism and model hyperparameters such as architecture and learning rate, and whether they can be adapted simultaneously.
39 29
3 Preliminaries
Reinforcement learning considers the problem of training an agent to interact with its environment so as to maximize its cumulative reward. Typically, task and environment are cast as Markov decision process (MDP), formally defined as a tuple (S, A. 7), where S is the state space, A is the space of possible actions, p : 8 A P(S) is transition kernel, r : S X A - IR is the reward function, and E [0, 1) is discounting factor. For a given policy the return 2 = E. 7'T,, is random variable representing the sum of discounted rewards observed along one trajectory of states obtained from following until some time horizon T, potentially infinite. Given a parameterization of the set of policies, {To E }). the goal is to update so as to maximize the expected return, or discounted cumulative reward, (0) = E,, [ rrr = R[Z*].
Actor-critic algorithms are framework for solving this problem in which the policy π, here known as the actor, is trained to maximize expected return, while making use of a critic that evaluates the actions of the policy. Typically, the critic takes the form of value function which predicts the expected return under the current policy, Q= a) == Ex [z] 8, a, = a). When the state space is large. Q" may be parameterized by model with parameters Φb The deterministic policy gradient (DPG) theorem 500 shows that gradient ascent on J can be performed via
VeJ(0) = E. a' (s, a))==>(s) Ve
(1)
The critic is updated separately, usually via SARSA [5II which, given a transition st, as 81+1, forms a learning signal via gemi-gradient descent on the squared temporal difference (TD) error, 57, where
+
- v=- -- Q= ae) = r2+1 + 7Q (S141, (S141, - (xt,
(2)
and where Yt is the Bellman target. Rather than simply predicting the mean of the return 2 under the current policy, it can be advantageous to learn a full distribution of 7" given the current state and action, 2d (St, (4) [10
47 In this framework, the return distribution is typically parameterized via a set of K functionals of the distribution (e.g., quantiles or expectiles) which are learned via minimization of an appropriate loss function. For example, the ktth quantile of the distribution at state s and associated with action a, qu (s, a). can be learned via gradient descent on the
16
Huber loss 255 of the distributional Bellman error, ōg = 2 q1<(8, a), for 2 ~ 2n (-1s, a). While Ż is formally defined as sample from the return distribution, &% is typically computed in practice as K 1 >x + 795(s, (s, a) [ a) 17
3

---

4 Optimism versus Pessimism
Reducing overestimation bias with pessimism It was observed by [ that Q-learning [56 with function approximation is biased towards overestimation. Noting that this overestimation bias can introduce instability in training, [20] introduced the Twin Delayed Deep Deterministic (TD3) policy gradient algorithm to correct for the bias. TD3 can be viewed as a pessimistic heuristic in which values are estimated via a SARSA-like variant of double Q-learning [24] and the Bellman target is constructed by taking the minimum of two critics:
yt T++1 + 7 min 07, (s, (s) + e). (E().2)
(3)
Here <~ (lippNV(..s))) c) is drawn from clipped Gaussian distribution (c a constant). This added noise is used for smoothing in order to prevent the actor from overfitting to narrow peaks in the value function. Secondly, TD3 delays policy updates, updating value estimates several times between each policy gradient step. By taking the minimum of two separate critics and increasing the number of critic updates for each policy update, this approach takes a pessimistic view on the policy's 's value in order to reduce overestimation bias. These ideas have become ubiquitous in sfate-of-the-an continuous control algorithms 7, such as SAC, RAD, (PI)-SAC 21
Optimism in the face of uncertainty While it is valuable to attempt to correct for overestimation of the value function, is also important to recall that overestimation can be viewed as a form of optimism, and as such can provide guide for exploration, a necessary ingredient in theoretical
treatments of RL in terms of regret 27 26 4 In essence, the effect of optimistic value estimation is to induce the agent to explore regions of the state space with high epistemic uncertainty, encouraging
further data collection unexplored regions. Moreover, [14 found that reducing value estimates, as done in pessimistic algorithms, can lead to pessimistic underexploration. in which actions that could lead to experience that gives the agent a better long-term reward. To address this problem, [4] introduced the Optimistic Actor-Critic (OAC) algorithm, which trains exploration policy using an optimistic upper bound on the value function while constructing targets for learning using the lower bound of [201. OAC demonstrated improved performance compared to SAC, hinting at complex interplay between optimism and pessimism in deep RL. algorithms.
Trading off optimism and pes- simism As we have discussed, there arguments for both optimism and pessimism in RL. Optimism can aid exploration, but if there is significant estimation error, then a more pes- simistic approach may be needed to stabilize learning. Moreover, both ap- proaches have led algorithms that are supported by strong empirical ev- idence. We aim to reconcile these seemingly contradictory perspectives by hypothesizing that the relative contributions of these two ingredients can vary depending on the nature of the task, with
HallCheetah-v2
Hopper-22
10000
7500
2000
1000
1500
Pasimiatic
1000
Optimisik
122
6.4
66
30
C4
0.4
08
10
Time stape (1o6)
Time soape (1e6)
Figure 2: Optimistic and Pessimistic algorithms averaged over 10 seeds. Shading is one half std.
relatively simple settings revealing predominantly one aspect. As an illustrative example, we trained "Optimistic" and "Pessimistic" versions of the same deep actor- ciitor-ciiticaaa algorithm (details in Section K for two different tasks and compared their performance in Figure 2 As we can see, in the HalfCheetah task, the Optimistic agent outperforms the Pessimistic agent, while in the Hopper task, the opposite is true. This result suggests that the overall phenomenon is multi-faceted and active management of the overall optimism-pessimee use of an adaptive approach which the degree of optimism or pessimism is adjusted dynamically during training. As consequence of this approach, the optimal degree of optimism can vary across tasks and over the course of a single training run as the model improves. Not only does this approach reconcile the scemingly contradictory perspectives in the literature, but it also can outperform each individual framework in a wider range of tasks.
trade-off is necessary. Accordingly, in the current paper we propose the
5 Tactical Optimistic and Pessimistic Value Estimation
TOP is based on the idea of adaptive optimism in the face of uncertainty. We begin by discussing how TOP represents uncertainty and then turn to a description of the mechanism by which TOP dynamically adapts during learning.

---

5.1 Representing uncertainty in TOP estimation
TOP distinguishes between two types uncertainty-aleatoo uncertainty and epistemic uncer- tainty-and represents them using two separate mechanisms.
Aleatoric uncertainty reflects the noise that inherent to the enviromment regardless of the agent's understanding of the task. Following [T 16 47). TOP represents this uncertainty by learning the full return distribution, 2 (s,,a,, for given policy and state-action pair (s, a) rather than only the expected return, Q (s, a) = |E|2* (s, a)}, 2 (s, a) ~ (-(-|s, a). Note here we are using Z"(s, a) denote the return random variable and z" (s, a) to denote the return distribution of policy π for state-action pair (s, a). Depending on the stochasticity of the environment, the distribution 2" a) more or less spread out, thereby acting as a measure of aleatoric uncertainty.
Epistemic uncertainty reflects lack of knowledge about the environment and expected to decrease the agent gains experience. TOP uses this uncertainty quantify how much an optimistic belief about the return differs from pessimistic one. Following [14), we model epistemic uncertainty via Gaussian distribution with mean Z(x a) and standard deviation (s, a) of the quantile estimates across multiple critics as follows:
(s, 2 a(),a) (s)(s,)),
(4)
where indicates equality in distribution. However, unlike in [14], where the parameters of the Gaussian are deterministic, we treat both Z(s, a) and a(s, a) as random variables underlying Bayesian representation aleatoric uncertainty. As describe next, only (=(s,a) is modeled
(via a quantile representation), hence ((s, a) and ass, a) are unknown. Proposition shows how to recover them from (s(s, a) and is proven Appendix E
Proposition 1. The quantile function (Z(),a) for the distribution of 7 is given by:
(Z(a,a) = Ee [9z=(sa))
(5)
where (z=(s,a) the quantile function of 2 (s, a) knowing e and ofs, a) and E, denotes the expectation w.r.t c NNO0,) 1). Moreover, o² (s, a) satisfies:
0² = E
11
(6)
Quantile approximation Following [17.. TOP represents the return distribution Z Z= a) using a
quantile approximation, meaning that it forms K statistics, gik) a), serve an approximation of the quantiles of 2 (" a). The quantiles (s, a) can be learned as the outputs of parametric function- in our case, a deep neural network-wil with parameter vector ø. To measure epistemic uncertainty. TOP stores two estimates, 2 (T a) and 2 (s, a). with respective quantile functions (k) (*) (s, a) and 92 (s, a) and parameters Φ1 and dz. This representation allows for straightforward
estimation the mean ((s, a) and variance (ss, a) in using Proposition |
Indeed, applying ( and O and treating Z (s, a) and Z, (s, a) as exchangeable draws from we approximate the quantiles (2(s,a) and (e,a)) of the distribution of (8, a) and (s,, a) as follows:
(s. a) (1 ( (s, (1)
(1)
(K)
(8
Σ
(k) di
( (7)
(s.
Next, we will show these approximations can be used to define an exploration strategy for the agent.
5.2 An uncertainty-based strategy for exploration
We use the quantile estimates defined in (7) to construct belief distribution (s, over the expected return whose quantiles are defined by
(z=(,.a) (2(+.a) (uo(s.))
This belief distribution 2nH (s,,0) a) is said be optimistic when 3 2 O and pessimistic when 3 < 0. The amplitude of optimism or pessimism is measured by (s, a), which quantifies epistemic uncertainty.
5

---

The degree of optimism depends on 9 and is adjusted dynamically during training, we will see in Section 5.3 Note that replaces ~ (0, 1). making the belief distribution non-Gaussian.
Learning the critics. TOP uses the belief distribution in s to form a target for both estimates of the distribution, (s, a) and 2 (s, a). To achieve this, TOP computes an approximation of 2' (s. a) using K quantiles g(6) = q" + (a(()) The temporal difference error for each 2 (s, a) is given by و(عا)] + )タ()) g.(b) with i {1,2} and where (j. k) ranges over all possible combinations of quantiles. Finally, following the quantile regression approach in [77 we minimize the Huber loss CHuber evaluated at each distributional error 8 o((ih,, which provides a gradient signal to learn the distributional critics as given by 4
Adi O
Vol Cuberr (8(.)))..
1SKjSK
The overall process is summarized in Algorithm
Learning the actor, The actor is trained to maximize the expected value Q(s, a) under the belief distribution 2d (s. a). Using the quantile approximation, Q(s,, a) simply given as an average over G()); ((s.a) a) = K Em= q(a) (s.a). The update of the actor follows via the DPG gradient:
AO x VaQ(s (ass,a)aaaa a))a==a(<) Vors(s).
(10)
This process is summarized in Algorithm 3 To reduce variance and leverage past experience, the critic and actor updates in 9 and 100 are both averaged over N transitions, (s, a, r, s') sampled from a replay buffer B B5
In the special case of 3 = - the average of (g) reduces to min; (7 a) and 10 recovers a distributional version of TD3, a pessimistic algorithm. On the other hand, when 3 > 0. the learning target is optimistic with respect to the current value estimates, recovering procedure that can be viewed as a distributional version of the optimistic algorithm of [4. However, in our case, when 3 2 0 the learning target is also optimistic. Hence, e and (0) can be seen as a generalization of the existing literature to a distributional framework that can recover both optimistic and pessimistic value estimation depending on the sign of B. In the next section we propose principled way to adapt B during training to benefit from both the pessimistic and optimistic facets of our approach.
5.3 Optimism and pessimism multi-arm bandit problem
As we have seen (see Figure 2 the optimal degree of optimism or pessimism for given algorithm may vary across environments. As we shall see, it also be beneficial to be more or less optimistic over the course of a single training run. It is therefore sensible for an agent to adapt its degree of optimism dynamically in response to feedback from the environment. In our framework, the problem can be cast in terms of the choice of 3. Note that the evaluation of the effect of /3 is form of bandit feedback, where learning episodes tell us about the absolute level of performance associated with particular value of 3, but do not tell us about relative levels. We accordingly frame the problem as a multi-armed bandit problem, using the Exponentially Weighted Average Forecasting algorithm [K] In our setting, each bandit arm represents particular value of 3, and we consider D experts making recommendations from a discrete set of values { After sampling a decision dm E {l,.... D} at episode m, we form a distribution Pm. E AD of the form (m) x exp (w, (d)). The learner receives feedback signal, fm E IR, based on this choice. The parameter wm updated as follows:
fm Wm (d) + pm]
if dm
Wmn+ (d)
(11)
Wmm (d)
otherwise,
for step size parameter 17 O. Intuitively, if the feedback signal obtained is high and the current probability of selecting a given arm is low, the likelihood of selecting that arm again will increase. For the feedback signal fm. we use improvement in performance. Concretely, we set fm = Rm R Rm-1- where R., is the cumulative reward obtained in episode m. Henceforth, we denote by p the exponential weights distribution over 3 values at episode
Our approach can be thought of as implementing a form of model selection similar to that of (4). where instead of maintaining distinct critics for each optimism choice, we simply update the same pair of critics using the choice 3 proposed by the bandit algorithm. For more thorough discussion of TOP's connection to model selection, see Appendix D

---

Algorithm TOP-TD3
Initialize critic networks Qø Qø and actor we
Initialize target networks 0f + Φ1, 0 - Φ2 0 -
Initialize replay buffer and bandit probabilities B 0, pi 2((00111))
2: for episode in m = 1, ....... do
3: Initialize episode reward Rm <- 0
4: Sample optimism 3m ~ p
5: for time step 1. 1, - 7 do
6: Select noisy action a, = Te (8,) +c, + N(O s²), obtain To+1; 8t+1
7:
Add to total reward R'
Rm +r+1
8:
Store transition B + B U (8t, 0e, 1141
9:
Sample N transitions T (s, a,, UpdateCritics (T, mm, 0, 04 02)
B.
10:
11:
if mod 6 then UpdateActor(T,, (T, Update 5/: d, T0;
12:
Bm 0, 01, 02)
13:
7 o i (1,2)
14:
Update 8¹: + 78 + (1 - ()-
end for
16:
Update bandit pa weights using
end for
5.4 The TOP framework
The general TOP framework can be applied any off-policy actor-critic architecture. As an example, integration of the procedure with TD3 (TOP-TD3) is shown in Algorithm [ with key differences from TD3 highlighted in purple. Like TD3, we apply target networks, which use slow-varying averages of the current parameters, 01 62. to provide stable updates for the critic functions. The target parameters 8', &1, Ø2 are updated every time steps along with the policy. We use two critics, which has been shown to be sufficient for capturing epistemic uncertainty [14]. However, it is likely that the ensemble would be more effective with more value estimates, as demonstrated in 40
6 Experiments
The key question we seek to address with our experiments is whether augmenting state-of-the-arr off-policy actor-critic methods with TOP can increase their performance on challenging continuous- control benchmarks. We also test our assumption that the relative performance of optimistic and pessimistic strategies should vary across environments and across training regimes. We perform abla- tions to ascertain the relative contributions of different components of the framework to performance. Our code is available //ttts::/gttiiiii
com/ tedmoskovitz. TOP
State-based control To address our first question, we augmented TD3 20 with TOP (TOP-TD3) and evaluated its performance on seven state-based continuous-control tasks from the MuJoCo framework
ball via OpenAl Gym 12 As baselines, we also trained standard TD3 200 SAC 21), OAC [4], as well as two ablations of TOP. The first, QR-TD3, simply TD3 with distributional critics. and the second, non-distributional (ND) TOP-TD3, is our bandit framework applied to TD3 without distributional value estimation. TD3, SAC, and OAC use their default hyperparameter settings, with TOP and its ablations using the same settings as TD3. For tactical optimism, we set the possible corresponds
values to be { {1,,0). such that corresponds to simply using the average of the critic. It's important to note that 3 = 0 is optimistic setting, as the mean biased towards optimism. We also tested the effects different settings for (Appendix, Figure 5). Hyperparameters were kept constant across all environments. Further details can be found in Appendix B We trained all algorithms for one million time steps and repeated each experiment with ten random seeds. To determine statistical significance, used two-sided t-test.
pessimistic lower bound, and 8
7

---

Table 1: Average reward over ten trials on Mujoco tasks, trained for 1M time steps. + values denote one standard deviation trials. Values within standard deviation of the highest performance listed in bold. indicates that gains over base TD3 are statistically significant (p < 0.05).
Task
TOP-TD3
ND TOP-TD3
QR-TD3
TD3
OAC
SAC
Humanoid
58991142
5445
5003
5386
5349
5315
HalfCheetah
13144 701*
12477
11170
9566
11723
10815
Hopper
3688 33"
3458
3392
3390
2896
2237
Walker2d
5111 + 220"
4832
4560
4412
4786
4984
Ant
6336 + 181*
6096
5642
4242
4761
3421
DoDuleelednnd
9337 20
9330
9299
8582
9356
9348
Reacher
-3.85 + 0.96
-3.91
-3.95
-4.22
-4.15
-4.14
Our results, displayed in Figure 3 and Table demonstrate that TOP-TD3 is able to outperform or match baselines across all environments, with State-of-the-ard performance in the 1M time step regime for the challenging Humanoid task. In addition, we see that TOP-TD3 matches the best optimistic and pessimistic performance for HalfCheetah and Hopper in Fig. Without access raw scores for all environments we cannot make strong claims of statistical significance. However, it is worth noting that the mean minus one standard deviation of TOP-RAD outperforms the mean performance all baselines in five out of the seven environments considered.
HalfCheetah-v2
Hopper-v2
Walker2d-v2
Humanoid-v2
12500
6003
10000
4002
4000
7500
1000
1002
5000
2000
2500
1.0
0.5
0.0
0.0
1.0
Ant-v2
Reacher-v2
Humanoid-v2 (zoom)
10005
6000
6000
Reward
-12
2000
-16
5000
0.0
0.5
1.0
0.5
10
0.0
0.5
0.0
075
1.00
steps (le6)
lok
steps (1e6)
Time stops (1e6)
Time steps (Ic6)
TD3
SAC
OAC
TOP-TD3
Figure 3: Reward curves for Mujaco tasks. The shaded region represents one half of a standard deviation over ten runs. Curves uniformly smoothed. The lower right plot zooms in on the second half of the learning curve Humanoid, omitting the shading for clarity.
Table 2:
Final average reward over ten trials on DMControl tasks for 100k and 500k time steps. + values denote one standard deviation across trials. Values within one standard deviation of the highest performance are listed in bold. indicates that gains over base RAD are statistically significant (p < 0.05).
(100k)
TOP-RAD TOP-RAD
QR-RAD
DrQ
Cheetah, Run
674 + 31'
600
546
.99
344
460
299
307
235
Finger. Spin
560
Walker, Walk
862 1: 43'
799
753
644
612
514
403
221
277
Cartpole, Swing
887 1: 13"
870
863
864
759
B16
SH2
563
326
Reacher, Easy
991
991
772
518
Cup. Catch
970 1: 12"
959
962
950
913
933
769
710
246
(500k)
TOP-RAD
QR-RAD
Dr(
Cheetah, Run
910:1:4 4"
828
805
774
660
H01
518
568
570
Finger, Spin
928. ::
920
907
938
957'
926
718
796
Walker, Walk
988 4
977
917
921
946
9U2
478
897
Carpole, Swing
890 + 28'
873
866
858
868
81.6"
845
787
762
Reacher, Easy
588
793
Cap. Catch
972 + 53'
970
971
970
963
933
959
939
879
Pixel-based control We next consider a suite of challenging pixel-based environments, to test the scalability of TOP to high-dimensional regimes. We introduce TOP-RAD, new algorithm that

---

1.0
HaltCheecah
Cheetal, Run
Optimism
Hopper
0.8
Finger, Spin
Walker2d
Walker, Walk
Humanoid
0.6
Campole, Swing
Ant
Reacher, Easy
ImDoublePand
0.4
Cup. Catch
Reacher
0.2
0.0
0.2
0.4
0.6
0.8
1.0
200000
400000
Time scaps (1o6)
Time steps
Figure Mean optimism plotted across ten seeds. The shaded areas represent one half standard deviation.
dynamically switches between optimism and pessimism while using SAC with data augmentation (as in (BI). We evaluate TOP-RAD on both the 100k and 500k benchmarks on six tasks from the
DeepMind (DM) Control Suite [52
In addition to the original RAD, we also report performance
from DrQ (58] PI-SAC
33 CURL
32
PlaNet 23 and Dreamer [22]. representing state-of-the-ar
methods. All algorithms use their standard hyperparameter settings, with TOP using the same settings as in the state-based tasks, with no further tuning. We report results for both settings averaged over ten seeds (Table 2 We see that TOP-RAD sets a new state of the art in every task except one (Finger, Spin), and in that case there is still significant improvement compared to standard RAD. Note that this is a very simple method, requiring only the a few lines of change versus vanilla RAD- and yet the gains over the baseline method are sizeable.
Does the efficacy of optimism vary across environments? To provide insight into how TOP's degree of optimism changes across tasks and over the course of learning, we plotted the average arm choice made by the bandit algorithm over time for each environment in Figure 4 Optimistic choices were given a value of 1 and and pessimistic selections were assigned O. A mean of 0.5 indicates that (optimism) and
1 (pessimism) were equally likely. From the plot, we can see that in some environments (e.g.. Humanoid and Walker, Walk), TOP learned to be more optimistic over time, while in others (e.g., Hopper and Finger, Spin), the agent became more pessimistic. Importantly, these changes were not always monotonic. On Ant, for example, TOP becomes steadily more pessimistic until around halfway through training, at which point it switches and grows more optimistic over time. The key question, then, is whether this flexibility contributes to improved performance.
To investigate this, we compared TOP to two baselines, a "Pessimistic" ver- sion in which B = -1 for every episode, and an "Optimistic" version in which 3 fixed to O. If TOP is able to accurately gauge the degree of optimism that's effective for given task, then it should match the best per- forming baseline in each task even these vary. We tested this hypothesis in the HalfCheetah and Hopper environments, and obtained the results shown in Figure 5 We see TOP matches the Optimistic performance for HalfCheetah and the Pessimistic performance in Hopper. This aligns with Figure 4 where we see that TOP does indeed favor a more Optimistic strategy for HalfCheetah, with a more Pessimistic one for Hopper. This result can be seen connected to the bandit regret guarantees referenced in Section 5.3] in which an adaptive algorithm is able to perform least well the best fixed optimism choice in hindsight.
HaliCheetah-v2
Hopper-22
1006
750
1000
2000
Pessmistic
1500
Optimietic
1000
TOP-TD3
Time
Time (1e6)
Figure 5: Mcan performance of Pessimistic, Optimistic, and TOP actoss ten seeds. Shaded regions are one half standard deviation.
7 Conclusion
We demonstrated empirically that differing levels of optimism are useful across tasks and over the course of learning. As previous deep actor-critic algorithms rely on a fixed degree of optimism, we introduce TOP, which is able to dynamically adapt its value estimation strategy, accounting for both aleatoric and epistemic uncertainty to optimize performance. We then demonstrate that TOP is able to outperform state-of-the-an approaches on challenging continuous control tasks while appropriately modulating its degree of optimism.

---

One limitation of TOP is that the available settings for 8 are pre-specified. It would be interesting learn 3. either through meta-learning or Bayesian framework. Nevertheless, we believe that the bandit framework provides a useful, simple-to-implemeet
template for adaptive optimism that could be easily be applied other settings in RI. Other future avenues could involve adapting other parameters online, such as regularization [3 using natural gradient methods A8 2
constructing
the belief distribution from more than two critics, and learning a weighting over quantiles rather than simply taking the mean. This would induce form of optimism and/or pessimism specifically with
respect to aleatoric uncertainty and has connections to risk-sensitive RL, as described by []6] 36].
Acknowledgements
We'd d like thank Rishabh Agarwal and Maneesh Sahani for their useful comments and suggestions on earlier drafts.
10

---

References
[1] A. Agarwal, H. Luo, B. Neyshabur, and R. E. Schapire. Corralling a band of bandit algorithms. In Conference on Learning Theory, pages 12-38. PMLR, 2017.
[2] M. Arbel, A. Gretton, W. Li. and G. Montúfar. Kernliized wasserstein natural gradient. arXiv preprint arXiv:1910.09655. 2019.
[3] J. Audibert, R. Munos, and C. Szepesvári. Tuning bandit algorithms in stochastic environments. In Algorithmic Learning Theory, 18th International Conference, ALT 2007, Sendai, Japan, October 1-4, 2007, Proceedings, volume 4754 of Lecture Notes in Computer Science, pages 150-165, Springer, 2007.
[4] M. G. Azar, I. Osband, and R. Munos. Minimax regret bounds for reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning. ICML, volume 70 of Proceedings Machine Learning Research, pages 263 262, 2017.
[5] A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, D. Guo, and C. Blundell. Agent57: Outperforming the atari human benchmark. In Proceedings of the 37th International Conference Machine Learning. ICML, 2020.
[6] P. Ball, J. Parker-Holder, A. Pacchiano, K. Choromanski, and S. Roberts. Ready policy one: World building through active learning. In Proceedings of the 37th International Conference on Machine Learning, volume 1 119, pages 591-601, 13-18 Jul 2020.
[7] J. Ball and S. J. Roberts. Offcon What is state of the art anyway?, 2021.
[8] G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. TB, A. Muldal,
N. Heess, and T. Lillicrap. Distributional policy gradients. In International Conference on Learning Representations, 2018.
[9] L. Bartlett and A. Tewari. REGAL: A regularization based algorithm for reinforcement learning in weakly communicating MDPs. CoRR, abs/1205.2661, 2012.
[10] M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, page 449-458. JMLR.org, 2017.
[11] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. CoRR, abs/ (bs//20074478, 2012.
[12] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba. Openai gym. CoRR, 2016.
[13] N. Besa-Bianchi and G. Lugosi. Prediction, learning, and games. Cambridge University Press, 2006.
[14] K. Ciosek. Q. Vuong, Loftin, and K. Hofmann. Better exploration with optimistic actor-critic. In Advances in Neural Information Processing Systems. NeurIPS, 2019.
[15] J. D. Co-Reyes, Y. Miao, D. Peng, Q. V. Le, S. Levine, H. Lee, and A. Faust Evolving reinforcement learning algorithms. In International Conference on Learning Representations, 2021.
[16] W. Dabney, G. Ostrovski, D. Silver, and R. Munos. Implicit quantile networks for distributional reinforcement learning. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1096-1105, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
[17] W. Dabney, M. Rowland, with quantile regression. In AAAI AAAI, 2018.
Bellemare, and R. Munos. Distributional reinforcement learning
[18] S. Filippi, O. Cappé, and A. Garivier. Optimism in reinforcement learning and kullback- leibler divergence. 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 115-122, 2010.

---

[19] R. Fruit, M. Pirotta, A. Lazaric, and R. Ortner. Efficient bias-span stas-ssa--oonttaanee exploration- exploitation reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, ICMI,, volume 80, pages 1573-1581, 2018.
[20] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. ICML, 2018.
[21] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR, abs//8811001299 2018.
[22] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
[23] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages
PMLR, 09-15 Jun 2019.
[24] H. Hasselt. Double q-learning. In Lafferty, C. Williams, Shawe-Taylor, R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23, pages 2613-2621, 2010.
[25] P. J. Huber. Robust estimation of a location parameter. Annals of Mathematical Statistics, 35(1):73-101, 1964.
[26] T. Jaksch, R. Ortner, and P. Auer. Near-optimal regret bounds for reinforcement learning. J. Mach. Learn, Res., 11:1563-1600, Aug. 2010.
[27] C. Jin, Z. Allen-Zhu, Bubeck, and M. I. Jordan. Is Q-learning provably efficient? In S. Vishwanathan, H. Wallach, S. Larochelle, K. Grauman, and N. Cesa-Bianchi, editors, Advances Neural Information Processing, volume 31. Curran Associates, 2018
[28] C. Jin, Z. Yang, Z. Wang, and M. Jordan. Provably efficient reinforcement learnin with linear function approximation. In Conference Learning Theory, pages 2137-2143. PMLR, 2020.
[29] I.. Kirsch, S. van Steenkiste, and Schmidhuber. Improving generalization in meta reinforce- ment learning using learned objectives. In International Conference Learning Representa- tions, 2020.
[30] L. Kocsis and C. Szepesvári. Bandit based Monte-Carle planning. In Machine Learning: ECML 2006, 17th European Conference on Machine Learning. Berlin, Germany, September 18-22, 2006, Proceedings, volume 4212 of Lecture Notes in Computer Science, pages 282-293. Springer, 2006.
[31] M. Laskin, K. Lee, A. Stooke, I.. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 19884-19895. Curran Associates, Inc., 2020.
[32] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. Proceedings of the 37th International Conference Machine Learning, Vienna, Austria, PMLR 119, 2020. arXiw:2004.04136.
[33] K.-H. Lee, I. Fischer, A. Liu, Y. Guo, H. Lee, J. Canny, and S. Guadarrama. Predictive information accelerates learning in rl. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1 11890-19911. Curran Associates, Inc., 2020.
[34] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Frez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
[35] L.-J. Lin. Self-improving reactive agents based on reinforcement leannii planning and teaching. Mach. Learn., 8(3-4):2933321. May 1992.
12

---

[36] S. Ma, J. Y. Yu, and A. Satir. A scheme for dynamic risk-sensitive sequential decision making, 2019.
[37] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning, Nature, 518(7540):529-533 Feb. 2015.
[38] T. Moskovitz, M. Arbel, F. Huszar, and A. Gretton. Efficient wasserstein natural gradients for reinforcement learning. In International Conference on Learning Representations. ICLR, 2021.
[39] Oh, M. Hessel, W. M. Czarnecki, Z. Xu, H. van Hasselt, S. Singh, and D. Silver. Discovering reinforcement learning algorithms. In Advances in Neural Information Processing Systems 33. NeurIPS, 2020.
[40] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, pages 4026-4034, 2016.
[41] A. Pacchiano, P. Ball, J. Parker-Holder, K. Choromanski, and S. Roberts. On optimism in model-based reinforcement learning. CoRR, 2020.
[42] A. Pacchiano, C. Dann, C. Gentile, and P. Bartlett. Regret bound balancing and climination for model selection in bandits and rl arXiv preprint 200::2002.13443 2020.
[43] A. Pacchiano, J. Parker-Holder, Y. Tang, K. Choromanski, A. Choromanska, and M. Jordan. Learning to score behaviors for guided policy optimization. Proceedings of the 37th Interna- tional Conference on Machine Learning, volume 119, pages 7445-7454 13-18 Jul 2020.
[44] A. Pacchiano, M. Phan, Y. Abbasi- Yadkori, A. Rao, J. Zimmert, T. Lattimore, and C. Szepesvari. Model selection contextual stochastic bandit problems, arXiv preprint arXiv:2003. 01704, 2020.
[45] J. Parker-Holder, A. Pacchiano, K. Choromanski, and S. Roberts. Effective diversity in population-bast reinforcement learning. In Advances in Neural Information Processing Systems 34. NeurIPS, 2020.
[46] H. Penedones, C. Riquelme, D. Vincent, H. Maennel, T. A. Mann, A. Barreto, S. Gelly, and G. Neu. Adaptive temporal-differene learning for policy evaluation with per-state uncertainty estimates. In Advances in Neural Information Processing Systems. NeurIPS, 2019.
[47] M. Rowland, R. Dadashi, S. Kumar, R. Munos, M. G. Bellemare, and W. Dabney, Statistics and samples in distributional reinforcement learning. 2019.
[48] T. Schaul, D. Borsa, D. Ding, D. Szepesvari. G. Ostrovski, W. Dabney, and S. Osindero. Adapting behaviour for learning progress. CoRR, abs/1912.066900, 2019.
[49] D. Silver, A. Huang, C. J. Maddison, A. Guez, I.. Sifre, G. van den Driessche, J. Schrittwieser,
I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalch- brenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukeuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484- 489, 2016.
[50] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learning, pages 387-395, 2014.
[51] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
[52] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki.
J. Merel, A. Lefrancq. T. Lillicrap, and M. Riedmiller. Deepmind control suite, 2018
[53] S. Thrun and A. Schwartz. Issues in using function approximation for reinforcement learning. In In Proceedings of the Fourth Connectionist Models Summer School, Erlbaum, 1993.
13

---

[54] E. Todorov, Т. Erez, and Y. Tassa. Mujoco: A physics engine for moddel-based control. In IROS, pages 0023-5033. IEEE, 2012.
[55] A. C. Y. Tossou, D. Basu, and C. Dimitrakakis. Near-optimal optimistic reinforcement learning using empirical bernstein inequalities. CoRR, abs/1905, 12425, 2019.
[56] C. C. H. Watkins and P. Dayan. Q-learning. Machine Learning, 8:279-292, 1992.
[57] I.. Yang and M. Wang. Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning, pages 10746-10756. PMLR, 2020.
[58] D. Yarats, I. Kostrikov, and Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021.
[59] Zhang and H. Yao. Quota: The quantile option architecture for reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 33001)55777-5844, 2019.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section f and Section 5.4
(c) Did you discuss any potential negative societal impacts of your work? [No] We present a generic framework for improving deep actor-critic algorithms, and so the societal impacts, if any, are difficult to predict.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Our work has no evident ethical ramifications for the categories listed in the guidelines, nor does it contain
any private information about individuals.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 5.1
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix E
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Section g
Appendix B and attached code.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they
were chosen)? [Yes] See Section 0 and Appendix B
(c) Did you report error bars (e.g., with respect to random seed after running experi-
ments multiple times)? [Yes] See Section and Appendix A
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix B
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See citations
throughout the paper.
(b) Did you mention the license of the assets? [Yes] See Appendix
(c) Did you include any assets either in the supplemental material URL? [Yes] We included the code for our experiments in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "sH-nekw9OO", "reviewer": "Reviewer_b4Gb", "summary": "The paper considers the problem of approximate value estimation in continuous-action actor-critic architectures, raising the argument that the overestimation bias is not always (in all environments) bad. In particular, they accord to the *optimism in the face of uncertainty* principle to argue that overestimation helps in some domains due to promoting exploration, while in others the negative impacts of such bias outweigh the exploratory benefits (which is why pessimistic approaches such as TD3 have improved performance in many domains). Based on this argument the direction of the paper is a natural one: create a method that adapts the agent's degree of optimism in a single training lifetime. The authors propose such an approach where (at least) two action-value critics are trained, based on which a measure of *epistemic uncertainty* can be obtained. Based on this epistemic uncertainty, the agent's value estimation can be adjusted to be more optimistic or pessimistic. To isolate this measure from *aleatoric uncertainty*, they use distributional critics. There is only a single scalar parameter, $\\beta$, that determines the degree of optimism, and the proposed method applies a multi-armed bandit formulation across episodes to optimize this parameter (over discretized values of the parameter in the range of $[-1, 0]$).", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I salute the authors for this fine paper. The paper packs a lot of information but so concisely and clearly. \n\nThe paper presents a fresh take on optimism/pessimism in approximate value estimation. Most successful methods in continuous control address the overestimation of values with pessimism, whereas Ref. [13] proposes an optimistic actor-critic approach that achieves better performance. As such, the present work addresses a natural gap by maintaining the best of both worlds adaptively in the context of the environment. \n\nThe paper is written beautifully. The ideas are clearly presented. The extensions needed to implement the method are minimalistic. I like the fact that the paper doesn't try to use meta-learning, or other more complicated settings, to showcase the base ideas. This makes the paper easy to follow, without confounding effects that would make validating the fundamental ideas difficult.   \n\nThe arguments are well motivated with experiments in Hopper and HalfCheetah domains (see Fig. 5). The experiments are thorough (the number of seeds is sufficient, baseline set is reasonable and up-to-date), showing that improvements across all domains can be obtained with a single algorithm. Also, one interesting finding is that (Fig. 4 left) the degree of optimism need not be constant or monotonic during the training course; this further illustrates the utility of using adaptive optimism during training.\n\n**Minor:**\n- L116: This is really Q-learning rather than SARSA because $\\pi(s_{t+1})$ is not generally the action that the agent has executed in the environment; it's critical to off-policy learning in DPG-style methods to use a Q-learning-style Bellman update on the critic.\n- Why focus only on continuous-action algorithms? Doesn't the same ideas hold in discrete-action methods? E.g. Q-learning vs Double Q-learning?\n- Why not use $\\beta > 0$ in experiments? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "WaqQ2mCdHY", "reviewer": "Reviewer_o4fR", "summary": "This paper is motivated by the two (seemingly contradicting) views of learning efficiently in the RL setup, that of pessimism and optimism. It touches upon why pessimism has been useful in computing better value estimates when deploying function approximation. It then argues that pessimism also leads to reduced exploration and contradicts the optimism favouring view predominant in RL theory. Finally, it claims that the right levels of pessimism/optimism are problem dependent and thus proposes adding a bandit algorithm to adaptively choose either optimism or pessimism for individual problems. This algorithm (called TOP) is tested on both state-based and pixel-based MuJoCo benchmarks. There is a decent amount of performance improvement seen in both cases when comparing against popular baseline methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The motivation of the work is solid in my opinion. The main question the authors ask is indeed important and I do not know of any other works that empirically test for the trade-off between pessimism and optimism (though I do not follow this exact topic in too much depth). In terms of the solution proposed, I do not rate it too highly in terms of originality. \n\nQuality: The overall quality of the work is quite high. The experiments are well executed and observations are nicely reported. \n\nClarity: The paper reads quite clearly overall. The flow of content is pretty nice as well.\n\nSignificance: Since the main contribution of the paper is empirical, there could be two ways to evaluate the significance in my opinion: 1) how much insights do the experiments shed on the trade-off between pessimism and optimism depending on the problem in hand, and 2) how much empirical gains can one get from the proposed solution + how simple would the method be to implement. I strongly feel that this paper makes contributions in the second category. In that regard, I do not completely get why the distributional framework is necessary here. It seems that the pessimism/optimism trade-off can be modelled without bringing in the distributional framework, as is done in the ND TOP-TD3 method. Therefore, to me it seems as if it was added just to boost the performance further and add more content for the paper. That is fine in general but I would argue that it then hampers the simplicity component, and does not highlight as well how much difference the bandit algorithm can provide in and of itself.\n\n\nQuestions:\n\n1. Is there any special reason to choose to work with TD3 instead of SAC? I believe both use the pessimistic form for updating the value functions.\n\n2. The authors say that the code is attached but I could not find any. Can the authors please share the code (with a read-me)?\n\n3. Was there a similar bifurcation done for the pixel-based case like in the state-based case? i.e. ND TOP-TD3 and QR-TD3 for the pixel-based method?\n\n4. I would really like to see error statistics for all methods, instead of just for the TOP method in the tables.\n\n5. Based on the above comments, I am inclined to look at the ND TOP-TD3 column as the main results. Is that a fair way to interpret the results? Can the authors explain more about the inclusion of the distributional framework in this respect?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Suggestions: I believe making contributions in the first category (as described above) would be really valuable. In particular, if we can have an experiment with a simple task such as Mountain Car, and show that the amount of exploration directly varies with the tradeoff in optimism/pessimism. The effect pessimism and optimism have on the exploration seems to be the key to better performance (maybe there is something else as well? maybe they directly affect the representations learnt?) and this does not come forth in the experiments with more complex domains. Furthermore, going deeper into why/how the representations get better when we apply the proposed adaptive strategy can give even more insights.\n\n\nI do not think there any explicit negative societal impacts of this work. ", "review_text": "", "extra_scores": {"ethical_concerns": "I do not think there any explicit ethical concerns raised by this work. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hrs"}, "extra_text": {}}, {"review_id": "R4BIYwHX6CM", "reviewer": "Reviewer_Xse6", "summary": "The paper introduces a bandit algorithm to adaptively change optimisim/pesimism in the actor critic algorithms\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In many ways, it looks like the paper builds on top of “K. Ciosek, Q. Vuong, R. Loftin, and K. Hofmann, Better exploration with optimistic actor-critic, In Advances in Neural Information Processing Systems”. While that work considers only optimistic setup (where beta is positive), this work allows for both positive and negative beta.\nThe idea is to use (discretized) beta in a multi-arm bandit problem to adapt the optimism/pesimism per task (and also during training) \n \nThe paper is well written and clear. Previous work seems to be well cited. The motivation is well explained, and further supported by the experiments in the last (sub)section.\nAuthors also include quite a few Mujoco tasks to evaluate the method and compare it to previous ones. The evaluation is also done very well, \n \nI like the paper, and I see only two negative points here. First, the resulting comparison to previous methods is not that amazing - it mostly does better (to be fair, that is usually the case for many algorithms, it’s very rare to substantially do better in all domains). Second, the work is very much incremental to prior methods.\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Sufficiently addressed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "LFIWxrQ8Hx", "reviewer": "Reviewer_fVYV", "summary": "**Summary and contributions**\n\nThis paper studies an interesting contradiction in the reinforcement learning literature, which suggests that both optimistic and pessimistic attitudes toward exploration can lead to high performance. The paper argues that no single attitude applies equally-well to all domains and to all stages of learning; for example, it can sometimes be beneficial to explore pessimistically in the beginning and then slowly change to adopt an optimistic attitude after some amount of experience.\n\n To resolve the apparent contradiction, the paper proposes the algorithm called TOP. TOP estimates epistemic uncertainty and uses it to bias sample estimates of a distributional reinforcement learning algorithm. The magnitude of bias is adapted online using one of the EXP bandit algorithms, with meta-like feedback in the form of differential episodic returns. Thus, the system will choose the exploration strategy to improve the utility of its learning process.     \n\nThe paper applies TOP to algorithms based on the actor-critic architecture, and it evaluates the algorithm extensively in continuous control domains, showing positive results to suggest the extra adaptation is beneficial. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**General impressions**\n\nI quite like this paper. I think that it contains many interesting ideas, and it presents solid empirical evidence that support its main claims. But with that said, it currently feels like a stretch to call the paper complete. I have a few issues with the writing and presentation, and there are some technical details that I’d like to clarify with the authors before committing to accept the paper.\n\n**Further details**\n\nIn Section 1, the paper does a good job of motivating its research problem by presenting an apparent contradiction. But it follows this up with a vague hypothesis (Lines 36-37). This can be fixed by making the statement more specific and using terms that have been previously introduced. Remove loose language such as “function of the environment”, and “overall context in which a learner is embedded”. The content of a hypothesis needs to be specific and unambiguous enough to be falsifiable.  These comments also apply to the claim on line 44.\n\nIn general the language needs to be improved. \n* “estimates the optimal approach” (Line 48). This algorithm can’t claim to be optimal, but it can claim to be good or useful.\n* “Learning-theoretic” (Line 43): This is a misuse of the term. \n* I think it’s wrong to say pessimism and optimism are in balance, because choosing one doesn’t come at the expense of the other. \n* “A Bayesian representation of aleatoric uncertainty” is a contradiction in terms. As more information is collected, a Bayesian representation of uncertainty will eventually vanish. But aleatoric uncertainty will persist despite the amount of experience the system has seen.\n* “Degree of optimism” is vague. The system has an attitude toward exploration, which can be pessimistic or optimistic. Choosing an attitude gives the system an exploration strategy. \n* The main contributions need to be more clearly stated, without ambiguous language. \n\nThe way epistemic uncertainty is discussed and estimated is confusing (Lines 189--196). It took me several passes to realize that epistemic uncertainty referred to the estimation error of the return distribution’s quantile. The notation and text do not make this clear. \n\nThe random return does not have to be modeled as a Gaussian. One can decompose a random variable into its mean and zero-mean residual. I believe this is what is done in Equation 8. Saying this to start with and removing the language around Gaussians will simplify the exposition.\n\nProposition 1 seems obvious and potentially not helpful. Unless the authors can justify its purpose, I suggest it be removed. \n\nThe notation is a bit of a disaster. Random variables and distributions are denoted the same way. Equation 5 uses a random variable as the subscript of the quantile function, but Equation 8 uses the belief distribution as the subscript. Equation 4 should reflect equality in distribution. Indices are used without defining their range. I strongly recommend taking another editing pass to simplify and clarify notation. \n\nHow important are having multiple estimates of the quantile function for epistemic uncertainty? Right now you have two.\n\nCan you justify the choice for the baselines and describe what the differences between the proposed algorithm and these represent?\n\n**Other feedback / questions**\n\n* Why is some text of your algorithm red? \n* Aleatoric and Epistemic uncertainty are introduced as terms without definition. It is better to describe the idea of a concept before the term, if it’s not going to be defined.\n* How was significance determined in Table 1?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a solid piece of work and the reviewers agreed in the end. The paper is well written, it unifies two views on exploration, the experiments are well done, and the authors did a good job both (a) responding to the reviews and (b) improving on the previous submission. There were some concerns about the working being incremental, but consensus emerged that the work exhibited a high degree of correctness (which is critical and not as common as should be) that the paper provides useful insights and foundations of future work. Another reviewer was generally favourable after discussion but did not update their score to reflect that. Given the relevance and difficulty of efficient exploration this work will make a welcome addition to the neurips program.

There were some important missing details and notational issues flagged by one reviewer. I am confident these can be addressed easily based on the author response.
Decision: Accept (Poster)