Faster Reinforcement Learning with Value Target Lower Bounding
Authors: Le Zhao
Wei Xu
Venue: NeurIPS 2022 Submitted (2022)

Abstract: We show that an arbitrary lower bound of the maximum achievable value can be used to improve the Bellman value target during value learning.  In the tabular case, value learning using the lower bounded Bellman operator converges to the same optimal value as using the original Bellman operator, at a potentially faster speed.  In practice, discounted episodic return in episodic tasks and n-step bootstrapped return in continuing tasks can serve as lower bounds to improve the value target.  We experiment on Atari games, FetchEnv tasks and a challenging physically simulated car push and reach task.  We see large gains in sample efficiency as well as converged performance over common baselines such as TD3, SAC and Hindsight Experience Replay (HER) in most tasks, and observe a reliable and competitive performance against the stronger n-step methods such as td-lambda, Retrace and optimality tightening.  Prior works have already successfully applied a special case of lower bounding (using episodic return), but are limited to a small number of episodic tasks.  To the best of our knowledge, we are the first to propose the general method of value target lower bounding (with possibly bootstrapped return), to demonstrate its optimality in theory, and effectiveness in a wide range of tasks over many strong baselines.

Faster Reinforcement Learning with Value Target Lower Bounding
Anonymous Author(s)
Affiliation Address email
Abstract
We show that an arbitrary lower bound of the maximum achievable value can be used to improve the Bellman value target during value learning. In the tabular case, value learning using the lower bounded Bellman operator converges to the same optimal value as using the original Bellman operator, a potentially faster speed. In practice, discounted episodic return in episodic tasks and n-step bootstrapped return in continuing tasks can serve as lower bounds improve the value target. We experiment on Atari games, FetchEnv tasks and a challenging physically simulated car push and reach task. We see large gains in sample efficiency as well as converged performance over common baselines such as TD3, SAC and Hindsight Experience Replay (HER) most tasks, and observe reliable and competitive performance against the stronger n-step methods such as td- Id-lamnda, Retrace and optimality tightening. Prior works have already successfully applied a special case of lower bounding (using episodic return), but are limited to small number of episodic tasks. To the best our knowledge, we are the first to propose the general method of value target lower bounding (with possibly bootstrapped return), to demonstrate its optimality in theory, and effectiveness in a wide range of tasks over many strong baselines.
10
11
12
14
16
Introduction
18
The value function is key concept in dynamic programming approaches Reinforcement Learning (RL) (Bellman 1957). It estimates the sum of all future rewards (usually time-discounted) of a given state. In temporal difference (TD) learning, the value function is adjusted toward its Bellman target which adds the reward of the current step with the (discounted) value of the next state (Sutton &
20
21
22
23
Barto 2018). This forms the basis of many state of the art RL algorithms such as DQN (Mnih et
24
2013]. DDPG (Lillicrap et al. 2016), TD3 (Fujimoto et al. 2018), and SAC (Haarnoja et al. 2018)).
The value of the next state is typically estimated using "bootstrapped value" based on the value function itself, which is being actively learned during training. The bootstrapped values can be random and far from the optimal value, especially at the initial stage of training, or with sparse reward tasks where rewards can only be achieved through long sequence of actions. Consequently. the Bellman value targets well the learned values usually far away from the optimal value (the value of the optimal policy).
26
28
29
30
31 Naturally, this leads to the following idea: If we can make the value target closer to the optimal value,
32 we may speedup TD learning. For example, we know that the optimal value is just the expected discounted return of the optimal policy, which always upper bounds the expected return of any policy,
34 For episodic RL tasks, we could use the observed discounted return up to episode end from the training trajectories to lower bound the value target. This makes the new value target closer to the
36 optimal value, when the empirical return higher than the Bellman target.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

Algorithm Value iteration with value target lower bounding
Input: Finite MDP p(s', r/ss convergence threshold 0, a lower bound f(s) of the maximum achievable value (*((s)
Output: State value v(s)
v(s) 6-0
repeat
A -
Ep(s) vss)
for each state s do
ö(s) & max, Σ Ex' fis',rls,oo rop(s')
(s(s) f max f(s),
((s) + (s(s)
A max( A, (u(s) (s)))
end for until A
37 The case for continuing non-episodic tasks is less clear though. When continuing task can 38 return negative rewards, any safe lower bound the optimal value can too low be useful. One could take the risk and use n-step bootstrapped return lower bound, which is unsafe because bootstrapped return can overestimate and be greater than the optimal value. Can we still use them as lower bounds to improve TD value targets?
40
Theoretical Results for the Tabular Case
42
43 Our results show that for the tabular case, arbitrary functions below a certain bootstrap bound can be used to lower bound the value target to still converge to the same optimal value.
44
2.1
Background
46
In finite MDPs with limited number states and actions, a table can keep track of the value of each state. Using dynamic programming algorithms such value iteration, values guaranteed to 48 converge to the optimum through Bellman updates (Chapter 4.4 (Sutton & Barto 2018)).
The core of the value iteration algorithm (Algorithm is the Bellman update of the value function,
50
B(v). where v(8') bootstrapped value:
(())(s) = max Σ p(s', a)s, for++00 + yo('s) 20
is well known that the Bellman operator, 13, is contraction mapping over value functions (Denardo 1967). That is, for any two value functions in and U2, [[B(m) (((2)) |mx S 771/11 r2 for the discount factor 7 E [0, 1) and ||>||| max; Jas (the T norm). This guarantees that any value function under the algorithm converges to the optimal value B (v) o*
52
54
55
2.2
Convergence of value target lower bounding
56
Definition 2.1.
The expected n-step bootstrapped return for a given policy æ and value function v(s) is defined as the expected bootstrapped return of taking steps according to policy
57
G (su) ET (r1 ... + n-1 rn + vvns(s
58 Here, the step rewards r; and the resulting n-th step state 8n are random variables, with the expectation 59 Ed taken over all possible n-step trajectories under the policy и and the given MDP.
For the gist of the proof. see for example page 8 of otfhttps://poople.coo eecs. www.eley.edu/eeeeeeeeeeooo.o..ee pabbeel/ 2pp.pdff

---

60
Definition 2.2. Given the current learned value function e(s), policy class II, the maximum achievable value of state is defined as:
61
(s) = max TEl,,NE
G
(3)
62 This is more relaxed definition of maximum because for each state s, a different policy its) and
63 different number of steps n(s) can be used to achieve the maximum (* And the theorem below
64 says any function not exceeding the maximum achievable value can be used lower bound the value target, and still achieve the optimal value in convergence.
65
66
Theorem 2.3. Under the same assumptions for Bellman value contraction, for any function f that lower bounds the maximum uchievable value, i.e. V8, [(s) S (s)), if we define the lower bounded Bellman operator as B5(v) == max(B((), J), then B (v) = 8 (v).
67
69 Note, the value v(s) and the bootstrapped value can be inaccurate, and even above the optimal value.
70 As consequence, when n is finite, the maximum achievable value (*(s) (and f) can be above the
71 maximum expected return (i.e. the optimal value). On the other hand, when n is sufficiently large,
72 the effect of the bootstrap value v(s,) diminishes (see Equation 2). and the maximum achievable
73 value becomes the maximum expected return (i.e. the optimal value). Therefore, Vs, G' (s) is no smaller than the optimal value (s)... As special case of the theorem, as long as is below the
75 optimal value, value target lower bounding converges correctly:
Corollary 2.4. If function ʃ lower bounds the optimal value, i.e. Vs, S B (v), then B (v) = B
76
few things to note about the proof of Theorem 2.3 (included in Appendix 1.1
79 First, this only proves convergence, not contraction under the original ||01 - - metrie. In the Blo 80 case of the Bellman operator, contraction shows that Ve,, 02 value functions, [[B(e,) -B(2)l)) S
81 yll - 22 12 Here, for value target lower bounding, what's proved convergence to v at a rate of y. not contraction. There can be counter examples where the distance between V1 and 02 under By can increase in the original Los metric space, even though Vi and v2 are both getting closer to v² at rate of 7. One difficulty caused by convergence instead of contraction is that the stopping criterion in Algorithm [ (A < 8) no longer works, due to the inaccessible v* during learning. In practice, this may not be serious concern, people often train algorithms for fixed number of iterations or time steps.
85 one application
85
87
88 Second, based on the proof, the new algorithm is at least as fast as the original. When the lower bound actually improves the value target, i.e. f(s) > B(o;)(s) there is chance for the convergence to be faster. Convergence is strictly faster when the lower bound f has an impact on the Loo distance between the current value and the optimal value, i.e. it increases the value target for the states where the differences between the current value and the optimal value are the largest.
89
91
93 Third, the lower bound function doesn't have to be static during training. As long as there single during each training update, convergence is preserved.
94
The following sections detail how to compute lower bounds of the maximum achievable value (Section 33). how to integrate the lower bounds into state of the art RL algorithms (Section 4 and provide illustration of how this method may benefit value learning in practice (Section 4.3)
96
Example Lower Bound Functions
98
99 We show a few cases where lower bound functions can be readily obtained from the training experience. Future work may investigate alternatives.
100
101
3.1
Episodic tasks
102
In episodic tasks, discounted return is accumulated up to the last step of an episode. In this case, we 103 can wait until an episode ends, and compute future discounted returns of all time steps up to the end
of the episode. This episodic return is a lower bound of the optimal value when the environment is
3

---

105
deterministic, because the reward sequence can be repeated using the same sequence of actions To
106
make training efficient, we can compute and store such discounted returns into the replay buffer for each time step, and simply read them out during training, which adds very little computation to the baseline one-step TD computation.
108
[(80) = Σ
rr((s;,
(4)
i=
109
We call this variant "Ib-DR", short for lower bounding with discounted return.
3.1.1
Episodic with hindsight relabeled goals
In goal conditioned tasks, one helpful technique is hindsight goal relabeling (Andrychowicz et al.
112 2017). takes a future state that is d time steps away from the current state as the hindsight / relabeled goal for the current state. When the goal is reached, reward of O is given, otherwise -1 reward is
114 given for each time step.
115 In this case, we know it took d steps to reach the hindsight goal, so the discounted future return is:
f(so) = Σ -1y
1=0,.,0-1
(5)
1(1 - 7 /
This calculation can be done on the fly as hindsight relabeling happens, requiring extra space and 117 very little computation.
118
We call this variant "Ib-GD", short for lower bounding with goal distance based return.
Additionally, we can also apply Ib-DR and Ib-GD together, with discounted episodic return (Ib-DR) on the original experience and goal distance based return (Ib-G) (lb-GD) on the hindsight experience, giving 121 the "Ib-DR+GD" variant, which was used in Fujita et al. (2020).
120
3.2 In general (including non-episodic tasks)
If the task is continuing, without episode end discounted return needs to be accumulated all the way to infinity. When rewards are always non-negative, one can still use the accumulated discounted reward of the future n-steps to lower bound the value. But accumulated n-step discounted reward is no longer a lower bound when rewards can be negative, in which case, the more general lower bounding with bootstrapped value can be used: given trajectory of training experience T =< S0s .... Sn
124
126
G. (T) == r1 722 + >- In + ((8()
6
128
Assuming the rewards and the state sn can be repeated with the same action sequence, G (T) lower bounds the maximum achievable value (*o(s) (Equation 3
130
Two variations are possible: Given trajectory of length n,
1. compute v(s;) for all i E []. and take the maximum of all G)(r) to obtain tighter lower bound. We call this variant "Ib-b-nstep":
132
f(so) max (T)
(E
133
2. only evaluate y on the last (nth) step and use the nath-tte bootstrapped return as the lower bound, which involves less compute but results in looser bound. (When n large enough, this becomes the lb-DR variant.) We call this variant "lb-b-nstep-onlyy"
134
135
f(x0) (T)
Note that behavior policy can stochastic, as long as the policy class contains the optimal policy, value learning will converge to the optimal.
Chapter 3.3 Sutton & Barto 2018 has details episodic continuing tasks.

---

Integration into RL algorithms
136
137
4.1
Background
138
The value target lower bounds can be readily plugged into RL algorithms that regresses value to target, e.g. DQN, DDPG or SAC.
In these algorithms, the action value q(s, a) is learned through squared loss with the target value y. In one step TD return, for a batch of experience {8, a - r, 8'). the loss is:
141
Σ (s,a,r,"))
lg(s, a) y/²
(9)
142
In one step TD return, y the one step TD return ((,,
r((,,))) ('',
(10)
Here, o' and n' are the bootstrap value and policy functions, typically following the value and policy functions in a delayed schedule during training. (They are also called "target value" and "target policy", and are very different from the "value target" y in this paper.)
144
148
4.2
Value target lower bounding
147
With lower bounding. we replace the value target y with the lower bounded target:
max(J,
max(S,r (8' ( (8)))
(11)
148
This way of lower bounding the value target is the same as was done by Fujita et al. (020)) (confirmed via personal communication), but is subtly and importantly different from lower bounding the
149
value directly (Oh et al. 2018 Tang 2020 q(s, a) <- max(f, a)), which stays overestimated if
151
(s, a) initially overestimates.
152
To the best of our knowledge, value target lower bounding with bootstrapped values is a novel
153
contribution of this work.
4.3
An Illustrative Example
Figure Souureiinclueee Lincludes a fairly general example showing how value target lower bounding would improve value learning. Suppose enhance off policy algorithm such as DDPG with value target lower bounding (Ib-DR), when there is no training experience hitting the target state, no meaningful training happens for the baseline lb-DR. However, when there is trajectory hitting the target state, all states along the trajectory will soon propagated with meaningful return, and nearby states will also enjoy faster learning. As the state space becomes larger and the time horizon longer, successful trajectory will speed up learning quite a bit.
155
157
159
160
161
Experiments
182
183
The goal is to demonstrate the sample efficiency of lower bounding the value target over baseline such DDPG, TD3, SAC and HER. Because the lower bounded value target now look potentially many steps into the future, we suspect it to be best suited for long horizon, sparse reward tasks. Hence, we choose experiment
165
sampled subset of Atari games, the goal conditioned FetchEnv tasks and the harder goal conditioned Pioneer Push and Reach tasks. See details of the experiment setup in Appendix 1.2
167
5.1 Baselines
170
Baselines include DDPG (Lilicerap al. 2016). TD3 (Fujimoto et al. 2018). SAC (Haarnoja et al.,
2018) and HER (Andrychowicz et al. 2017 Plappert al. 2018). Implementations are based on
5

---

Baselinc: 0
Baseline:
Baseline: 2
Baseline: 3
Lowerbound 0
Lowerbound
Lowerbound
Lowerbound
Figure I: Illustration of value target lower bounding speeding up value learning training progresses from stages O to 3. The task is to navigate in the state space from start state S to end state T, with sparse reward the target. The shaded areas denote roughly states whose value has been significantly improved during training up to that stage.
and elsewhere. The curve from S denotes training experience that reaches
172 open sourced repositories, and baseline performance is verified against published results under similar settings. The Appendix [T.] 1.3 [.3]include results more basclines such DDQN (van Hasselt 174 et al. 2015), td-labmda (Sutton & BBrrto 2018) and Retrace (Munos et al. 2016).
175
5.2
Hyperparameters
Value target lower bounding is applied top of these baselines without any additional hyperparameter (Section 4). The only hyperparameters come from the baselines. These hyperparameters follow published work much as possible. When baseline hyperparameters need to be tuned for environment, e.g. Atari games or Pioneer tasks, we search for the best performance in total episode reward averaged across all tasks for that environment on one set of random seeds, then the optimal hyperparameters are fixed and evaluated on separate set of random seeds never seen during development. Value target lower bounding simply uses the the parameter values optimal for the baselines. Details are in Appendix 1.3
177
179
180
182
183
184
5.3
Results
185
We report results on both episodic and comtinuing/non-cpisodri tasks. We report evaluation per- formance averaged across several runs of the algorithms (five for the less stable Atari games and three for the others). Each run uses a random seed never seen during development. Due to space constraints, the main paper only reports performance aggregated across all tasks for each environment. During each run, we take one task and one random seed, run baseline and treatment algorithms, and record whether treatment agent evaluates strictly above the baseline agent as training progresses. We average across all the runs of the same environment, and plot the fraction times where treatment above baseline and the standard deviation of that fraction in Figure 2 evaluation curves.
186
187
189
191
Appendix [.4 contains per task
193
194
Overall, value target lower bounding is a simple, effective, efficient, carefree (no hyperparameter) and theoretically justified approach. Although the example lower bounds are limited deterministic environments, the theory is generally applicable to stochastic environments. A similar prior work
195
196
197 to compare would be Hindsight Experience Replay (HER) (Andrychowicz et al., (017), which is simple, effective, efficient, and also limited to deterministic environments (Blier & Ollivier (021). However, unlike our work, HER relies on the task being goal conditioned with full knowledge the reward function, has one hyperparameter to tune (the proportion of hindsight experience). and is not justified in theory for stochastic environments, Our work shows further significant gains on top of
199
201
HER on hard continuous control tasks.

---

10
10
04
1.3
044
1.5
Linooer
(a) Atari (lb-DR)
(h) Atari (hootstrapped)
(c) FetchEnn
(d) Pioneer
Figure 2: Aggregated evaluation performance: The fraction of times where treatment performs strictly above baseline, plotted along the number of time steps used for training. The solid curve is the sample mean of the fraction across all runs, and the shaded area is +/- one standard deviation. We use, for Atari (Ib-DR), 85 runs - 17 games each with seeds, for Atari (bootstrapped), 20 runs - games seeds, for FetchEnv, 9 runs 3 tasks 3 seeds, and for Pioneer, 6 runs 2 tasks 3 seeds.
$1.00
BGH jurt belaara/e
S-OHnAOO Puiti araaee
624
3:00
0.04
00
84
00
000
20
(a) Atari (Ib-DR)
(b) Atari (bootstrapped)
(c) FetchEny
(d) Pioneer
Figure 3: Average fraction of training experience where lower bounding improves Bellman value target, plotted along the number of iterations of training. The solid curve is the average of the fraction across all runs number of tasks times number of seeds, and the shaded area is +/- one standard deviation. Other setups are the same as Figure]
5.3.1 Lower bounding vs baselines SAC/DDP//HER
Figure 2 compares the lower bounding treatment with SAC, DDPG or HER baseline 17 sampled Atari games, the FetchEny tasks and the Pioneer tasks. For all the environments, value target lower bounding is not only more sample efficient, but also enjoys higher converged performance. After training starts, it quickly gains ground and outperforms the baseline for 70% to 100% of the runs. It keeps that advantage even at the end of the training, outperforming baseline in converged performance. These plots show how frequently treatment is above baseline, but is insensitive to the magnitude of change.
205
207
208
209
210
211
Appendix I.4] shows the magnitude of change with total episode return plotted for each task, often with large gains in sample efficiency and sometimes much higher converged performance. Among all the 22 tasks, only one task (Atari Breakout) shows lb-DR underperforming the baseline.
213
214
Investigations show that the loss on Atari Breakout is likely due to the mismatch between training objective and evaluation metric. During training, raw rewards are clipped to [-1, and step discounted at 7 0.99 to compute value, while in evaluation, total reward is the unclipped and undiscounted cumulative sum of episode rewards. The discount and clipping together severely penalizes large rewards earned later in the episode, which is what's 's happening for Breakout, because hitting top layer block produces a reward of 7 while hitting a bottom layer block produces 1. When we use non-clipped rewards a higher in training, the lower bounding method performs much better in total reward. Note, this train-tes discrepancy as well as an additional training bias (Thomas 2014] is
215
217
219
221
likely present in all of the prior works using policy gradient methods on Atari games.
223
5.3.2
Value target improvement
224
The lb-DR method is mostly effective, but is it really due to improvements to the value targets? looks at the fraction of training experience where lower bounding actually improves the
Figure 3
7

---

228
Bellman value target over the course of training. Overall, improved value target roughly coincides with performance gains, Appendix I.4 1.4 shows the plots per each task.
227
228
The Appendix also has more results, comparing with baselines such as n-step methods, DDQN and optimality tightening, and more analyses such as ablations and robustness to hyperparameter choices.
229
Related Work
230
231
Prior works (Fujita et al., 2020 Hoppe& & Toussaint, 2020 He et al., 2017 Oh et al. 2018 Tang 2020) employed several different ways of computing future returns and using that as a lower bound to improve value learning. It is quite easy to introduce biases and inefficiencies into the process and end up with suboptimal inefficient algorithm. Our work is the first to propose the general form of value target lower bounding (possibly with bootstrapping), to show its convergence to the optimal
233
235
236
value in the tabular case, and to demonstrate its effectiveness in illustrative examples and extensive experiments on a wide range of tasks.
237
238
Fujita et al. (2020)'s is method is similar to a special case (the Ib-DR+GD variant) of the general method. They used it as a part of a large system and showed that it improved sample efficiency for a robotic grasping task. Hoppe& & Toussaint (2020) also bounded the value target. But instead of using empirical return, they used a simplified MDP with subset of actions. Although without theoretical
239
241
proof and only experimented on a limited set of robotic manipulation tasks, both works show that value target lower bounding increased sample efficiency. This work, in addition to the theory and the more general method, shows that lower bounding improves both sample efficiency and converged performance in a wide range of tasks.
243
245
246
He et al. (017)) used empirical return with bootstrap to improve value learning. They formulated value learning as a constrained optimization problem with the empirical bootstrapped value being the lower (and upper) constraints of the value function. In their experiments, the Lagrangian multiplier was fixed, which would likely lead to suboptimal solutions. Our lb-b-nistp method also uses bootstrapped value. But we lower bound the value target directly, which is simpler, more efficient, and likely more optimal. Our work points out that for episodic tasks, even more efficient and effective methods like lb- DR exist. Appendix offers more discussion and results related to this.
247
249
251
252
253
Our work subtly but importantly different from the prior works on lower bound Q learning or Self Imitation Learning (SIL) (Oh et al., 2018 Tang 2020). SIL uses empirical return R to lower bound the value function itself (instead of the value target). This is achieved by adding an off policy value loss during on on-policy (AC or PPO) training (Leauee = ž e(s) max(v(s) (s)) 12). When the value function overestimates, the SIL value loss becomes zero, and keeps overestimating. Mixing the SIL
254
255
256
257
loss with the loss from the baseline algorithms probably helped to correct the overestimation, but no theoretical guarantee was given. In evaluation, SIL was often compared to on-policy Actor Critic or
259
PPO baselines, so it not clear how much of the gain was due to lower bounding and how much due to off-policy value learning. In this work, we bound the Bellman value target (Equation [T], so overestimates are automatically corrected via Bellman updates, and convergence is guaranteed in the tabular case. We also use off-policy algorithms as baselines for a cleaner comparison.
261
262
263
264 N- step return methods such as td-lambda (Sutton & Barto 2018) and Retrace (Munos et al. 2016) also look a few steps ahead, but obtain more accurate value of the behavior policy. Traditionally, this requires careful off-policy correction, and the value can still be far from the optimal value due to the often suboptimal behavior. This work shows that value target lower bounding efficiently and effectively looks ahead much further without the need for off off-policy correction, due to aiming at the optimal value. Appendix 1.5 has more detailed observations and discussions.
265
257
269
270
Plamning methods can look into the future to achieve higher value targets and better control. Examples include Monte Carlo Tree Search (MCTS) (Schrittwieser et al., 2019 Ye al. (02T] and Model Predictive Control (MPC) or receding horizon planning with raw actions (Chua et al. [2018] Hafner
271
272
273
et al. 2019; Zhang et al., (022), options (Silver & Ciosek, 2012), or subgoals (Nasiriany
al.
274
Nair & Finn |020]] Chane-Sane et al. (021I). Planning methods use either a dynamics model together with the learned value or just the learned value (in the case of goal conditioned tasks)
2019
275
Nasiiiaay et al. (201)) to improve policy or value estimates. Planning typically happens during roll
277
out (Nasiriany et al. 2019), but can also be used to improve the value target, as in Reanalyze of During value improvement, if planning takes
MuZero (Schrittwieser al. 2019 Ye et al. 202I
8

---

279
the maximum over set of possible future values (e.g. from different trajectories as in the case of MPC), and if this set includes the one step Bellman value target, then the planner essentially using alternative trajectories and their values to lower bound the Bellman value target. In this sense, the theory developed here can potentially justify and improve Reanalyze. In general, planning is orthogonal to value target lower bounding, and typically requires additional components and a lot more compute than the basic TD learning does. Therefore, we leave it to future work to explore the synergy between the two.
280
282
284
286
Interestingly, it is common practice to lower and upper bound the returns to the possible region. e.g. Andrychowicz et al. (2017) bounds value between |- T- 0j. Similar to lower bounding with episodic return (Section 3.1 such strict bounds of the actual value can be thought of as admissible heuristics (bounds) used during search of the optimal solution (Russell & Norvig, 2020). What's new in this work is that lower bounding with bootstrapped values (which can overestimate the value) still converges to the optimal value.
287
289
290
291
292
Kumar et al. (2020) (DisCor) also recognized that bootstrapped value targets can be inaccurate. This bias impacts learning adversely under function approximation. DisCor uses distribution correction to sample experience with accurate bootstrap targets more frequently, while value target lower bounding aims directly reduce the bias.
283
294
295
296 W While in theory using empirical return to lower bound the value target is only correct for deterministic environments, in practice, it seems as long as the environment is not heavily impacted by random fluc- tuations, they still perform well. In fact, with function approximation, the agents cannot distinguish
298
between two slightly different states, making the problem partially observable (Sutton & Barto 2018)
300
and appear slightly random. Prior methods such as SIL (Oh et al., 2018), Optimality Tightening (He
301
ct al. 2017). and even Hindsight relabeling (Andrychowicz al. 2017 and MuZero (Schrittwieser
302
et al.
2019) require the environment to be deterministic. Despite this theoretical limitation, the lower
303
bounding methods and the prior methods can still be very useful, outperforming baselines often by
304
large margins and when deploying to the real world (Fujita et al. 20220).
305
7
Conclusions
308
We propose a general form of lower bounding the value target using possibly bootstrapped return. In theory, value target lower bounding converges to the same optimal solution as the original Bellman operator. In practice, several ways of finding value lower bounds are examined.
307
308
309
For episodic tasks, discounted episodic return is an efficient and effective method involving very little extra computation. Precomputing the episodic return and storing it into the replay buffer allows efficient lower bound computation. It achieves much higher sample efficiency and converged performance than one-step baselines such SAC, DDPG TD3 in most tasks, and is competitive
310
311
312
313
among n-step baselines. Simple goal distance based return uses even less compute and achieves large gains in certain long horizon tasks over Hindsight relabeling (HER).
314
315
For non-episodic tasks or in general, lower bounding with n-step bootstrapped return outperforms step baselines and is strong competitor to the n-step methods such (truncated) td- td-lambda and Retrace.
317
7.1
Future Work
319
There are probably better ways of finding value lower bounds that improve training even more. One direction may be to use planning (e.g. Monte Carlo Tree Search, the Cross Entropy Method or using subgoals) to achieve tighter lower bounds given model of the task.
320
321
Estimating value lower bound for stochastic tasks may be possible, e.g. by learning reward function and a dynamics model and using imagined rollouts to obtain bootstrapped returns without overestimation.
323
324
325
Other ways of bounding the value target, e.g. upper bounding (He et al. 2017). may be worth investigating as well, e.g. to reduce overestimation in regions of poor reward.

---

References
327
328
Andrychowicz, M., Crow, D., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay, In Guyon, I., von Luxburg, U. Bengio, S., Wallach, H. M.. Fergus, R., Vishwanathan, S. V. N., and Gar- nett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017. Long Beach, CA. www.:/ddinnnnnn.nnssuoeeee cttpripps..cppppp/211111111
330
332
USA, pp. 0048-5058, 2017. URL Absss533377--Atrttctt.... html
334
335
Bellman, R. Dynamic Programming. Princeton Univ. Press, Princeton, NJ, USA, 1957. ISBN 0-691-07951-X.
336
Blier, L. and Ollivier, Y. Unbiased methods for multi-goal reinforcement learning. CoRR, abs/2106.08863, 2021. URL http:::///xxii..oorrr abtp///g///////.....// 08863
338
339
Chane-Samm E., Schmid, C., and Laptev, I. Goal-conditioned reinforcement learning with imagined
340
subgoals. In Meila, M. and Zhang. T. (eds.), Proceedings of the 38th International Conference on
341
Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1430-1440.
342
PMLR, 18-24 Jul 2021. URL https://
mlr. colrspres////e............. v139/ chane-
343
html
344
Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 4759-4477777 4770, Red Hook, NY, USA, 2018. Curran Associates Inc.
345
346
347
348
Denardo, E. V. Contraction mappings in the theory underlying dynamic programming. SIAM Review, 9(2)1155177,, 1967. ISSN 00361445. URL http::/ww//// jstor org/ sttg://taale//2200 /0227440
349
350
Fujimoto, S., van Hoof, H., and Meger. D. Addressing function approximation error in actor-critic methods. CORR, 2018. http:://///axiiooortr http://aaav..rrr/aa///////0//0//
351
352
Fujita, Y., Uenishi, K., Ummadisingu, A., Nagarajan, P., Masuda, S., and Castro, M. Distributed rein-
353
forcement learning of targeted grasping with active vision for mobile manipulators. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 9712-9719, Oct 2020.
354
355
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018. URL httttt::/ppoceeeedeoteeo Fress/v80/haarie aaw.:j.a1........ html
356
357
358
mlr
359
380
Hafner, D.,, Lillicrap. T., Fischer, I., Villegas, R., Ha, D.., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. html
381
382
PMLR, 09-15 Jun 2019. URL htt.h/tps//prroceenggss mlr
363
364
press/v91 http///99/aaaafffaaae
385 He, F. S., Liu, Y.. Schwing, A. G., and Peng. J. Learning to play in a day: Faster deep reinforcement learning by optimality tightening. In 5th International Conference on Learning Representations, ICIR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net,
386
367
368
2017. URL http://///ppeeeeeeeeeee/erraeaee
389
Hoppe, S. and Toussaint, M. Qgraph-bounded q-learning: Stabilizing model-free off off-policy deep reinforcement learning. CoRR, abs/2007.07582 2020. URI. htt.h:tt/://aaart http///xxvvv.//// wwwxiv..or//aee abs/2007
370
07582
372 Kumar, A., Gupta, A., and Levine, S. Discor: Corrective feedback in reinforeement learning via
373
distribution correction. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F,, and Lin,
H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 18560-18572. Curran Associates, Inc., 2020. URL https:: /tatttss://proceee / http://rrrrree.ppppp//////////// FT4456cbbcaaee23355552222.....d
375
.dff
10

---

377
Lillicrap. T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver. D., and Wierstra, D.
378
Continuous control with deep reinforcement learning. In Bengio. Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May org. org//bs//////o/oeo abs/ 1509. 1
380
2-4, 2016, Conference Track Proceedings, 2016.
381
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller,
382
M. A. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. URL
383
http://arx:s /ttp:///axiv.../ org/ httg///////// 111//3112560
Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. G. Safe and efficient off-policy reinforcement learning. In Lee, D. D., Sugiyama, M., von Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 29: Annual Con- ference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 1046-1054, 2016. URL |ttp::tts:/p C3992e9a68c5ae
385
387
388
Aate::dddd-bttttt..hhhh www.seiiinnnnnuuris...mom wwwrripsscc/pppee//000
389
Nair, S. and Finn, C. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. In International Conference on Learning Representations, 2020. URL
391
https://openrevieet www.:nneef//nnmndemmmmmmm http:///www...eetooooooooooa/a//
393
Nasiriany, S., Pong. V., Lin, S., and Levine, S. Planning with goal-conditioned policies. Advances in Neural Information Processing Systems, 2019.
385
Oh, J. Guo, Y., Singh, S., and Lee, H. Self- learning. CoRR, abs/1806.006655, 2018. URL http://arxiv..rr/abbee w1wpg///b/////.....o. 05635
396
397
Plappert, M., Andrychowicz, M., Ray, A., McGrew, B., Baker, B., Powell, G., Schneider, J., Tobin,
398
J.. Chociej. M., Welinder, P., Kumar, V., and Zaremba, W. Multi-goal reinforcement learning: Challenging robotics environments and request for research. CoRR, abs/1802.09464, 2018. URL
399
400
http://arxiv.co org/ abs / 09464
Russell, J. and Norvig, P. Artificial Intelligence: Modern Approach (4th Edition). Pearson, 2020.
402
ISBN 7780134610999 URL http: aima. cs berkeley. edu/
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K.,, Sifree, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap. T. P., and Silver, D. Mastcring atari, go, chess and shogi by planning with learned model. CoRR, abs/191 1.08265, 2019. URL http:://axxxi org abs/
404
406
1911. 08265
Silver, D. and Ciosek, K. Compositional Planning Using Optimal Option Models. In Proceedings of the 29th International Conference on Machine Learning, pp. 165. icml.ce Omnipress, 2012.
408
409
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction.
A Bradford Book,
410
Cambridge, MA, USA, 2018. ISBN 0262039249.
411
Tang, Y. Self-imitation learning via generalized lower bound q-learning. CORR, abs/2006.07442, 2020. URL/https://///// httttttss///xxit>/ org/ abs/
412
413
Thomas, P. Bias in natural actor-critic algorithms. In Xing, E. P. and Jebara, T. (eds.), Proceedings of the 31st International Conference Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 441 448, Bejing, China, 22-24 Jun 2014. PMLR. URL. https:// proceedings.co mlm..rr... www.r..rrss//... www.srressssee vvteass///2//// wttp://hhoaas...hham
415
417 van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double g-learning. CoRR, 150.0066664,1, 2015, URL http:://axxx..or///bb///a 166400
419 Ye, W., Liu, S.. Kurutach, T., Abbeel. P., and Gao, Y. Mastering atari games with limited data. CoRR, abs/2111.00210, 2021.
00210
Zhang, H., Xu, W. and Yu, H. Generative planning for temporally coordinated exploration in reinforcement learning. In Internationaal Conference on Learning Representations, 2022. URL https://openrevier: wwt.://w.ww..o.rrr.reeettttaaaa.
422
423

---

Checklist
424
425
1. For all authors...
426
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
427
(b) Did you describe the limitations of your work? [Yes] Regarding the implementations being theoretically limited to deterministic MDPs, see Section 3.1 and the last paragraph of Related Works (Section 6 for details, and Appendix [.7 for example. The theorems, however, do not assume a deterministic MDP, and follow standard Bellman value contraction assumptions. Experiments use function approximation on continuous observations or action spaces, which already violate the theoretical assumptions, but the lower bounding methods still show large gains over the baselines.
429
430
431
432
439
434
435
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Appendix
436
1.9
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
438
439
2. If you are including theoretical results...
440
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
441
(b) Did you include complete proofs of all theoretical results? [Yes] Appendix
442
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (cither in the supplemental material or as a URL)? [No] Omitted due to anonymity. Code is already in a github repository, and will be released upon publication.
444
446
447
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Appendix 1.2 and 1.3
448
449
(c) Did you report error bars (e-g.. with respect to the random seed after running experi-
ments multiple times)? [Yes] Done for all plots
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] Treatments only cost a bit more compute than the baselines.
452
453
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
455
(a) If your work uses existing assets, did you cite the creators? [Yes] Except for the open source code repository, to keep anonymity.
456
457
(b) Did you mention the license of the assets? [No]
458
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
459
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [N/A]
460
481
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
462
463
5. If you used crowdsourcing or conducted research with human subjects...
464
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
465
466
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
467
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
469
12


Reviews:
[{"review_id": "kXDvF6Cum8V", "reviewer": "Reviewer_5ybC", "summary": "This paper proposes a form of target value lower bounding, which induces a new Bellman operator. The authors show the operator is a contraction, and propose several lower bounds. Extensive experiments are conducted (provided in the supplementary material) to show the performance of the proposed approach.", "questions": "- For the proposed lower bounds - are the theoretical gaps between the deterministic and non-deterministic settings?\n- When is $\\bar{G}$ more beneficial than $G_\\infty$\n- Why does $B_f$ improve convergence?\n- How does $B_f$ affect the overestimation problem?", "limitations": "The authors discuss limitations of their work. Particularly, deterministic requirements.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is written clearly and the overall concept is interesting. There are several aspects of this paper that are problematic. \n\nFirst, theoretically, the paper is not strong. Theorem 2.3 does not present any interesting new theory, and it is unclear why this result is important. The authors do not show or prove any benefit of using the bounded Bellman operator. When would this operator be useful for cases that don't bound $G_\\infty$? It would seem that due to over-optimism of the Bellman operator, $G_0$ will usually maximize $\\bar{G}$. \n\nSecond, the examples for lower bounds in the paper require deterministic environments. The lower bounds themselves feel somewhat trivial. It is unclear why these lower bounds are good. \n\nFinally, the experiments do not show any significant improvement using the proposed approach. Also, the experiments themselves are shown only in the appendix, which should not be used to place major contributions. The authors chose to do this due to space constraints, which is not fair to other reviewed papers.\n\nStrengths:\n\n- Paper is clearly written\n- Proposed Bellman operator may be having interesting characteristics to be researched\n- Authors conducted extensive experiments\n\nWeaknesses:\n\n- Lack of theory, and unsure about novelty\n- Setting is limited. Examples require deterministic environments. Unsure why $\\bar{G}$ is useful.\n- Experiments don't show a clear benefit of approach. Also, experiments are only provided in appendix which does not need to be evaluated for review.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TzJxMUce-gu", "reviewer": "Reviewer_41F1", "summary": "This paper proposes value target lower bounding to improve the Bellman value target during value function learning. The proposed method computes and utilizes a lower bound for the value target during Bellman backup, and the resulting lower-bounded Bellman operator is claimed to converge to the same optimal value as using the vanilla Bellman operator in the tabular case. Additionally, the paper presents several ways of estimating the lower bound function in practice, such as using the episodic return in episodic tasks and n-step bootstrapped return in non-episodic tasks, and shows the effectiveness of value target lower bounding on several RL environments. ", "questions": "Questions & Suggestions:\n\n\n1. [Technical Part]\n\n\n- 1.a. Please check Theorem 2.3. If $f$ is updated during training, please include the update rule/equation for $f$ in the main paper and in Algorithm 1.          \n\n\n- 1.b. Value iteration (VI) results do not directly generalize to RL settings and are not sufficient to rigorously justify performance in RL in my opinion. The authors may refer to Rmax [1] or randomized value function [2] for examples of rigorous analysis in the RL setting. \n\n\n- 1.c. Exploration is crucial for efficient/faster RL. In the example of Section 4.3, it would be more interesting to discuss for instance how value target lower bounding may help the agent reach the target state faster (with fewer number of episodes of training). Propagating the value back is a less interesting problem once the target state has been reached.                            \n\n\n2. [Clarity]\n\n\n- 2.a. In the actual implementation, is the lower bound function $f(s)$ fixed? or being constantly updated? It seems that the observed episodic return is written into the buffer and read out during training as the lower bound. So, $f(s)$ is updated by the newly-observed episodic return? Please try to clarify this in the paper revision.    \n\n\n- 2.b. Is a larger value always a better Bellman target? Breakout in Figure 2 and 3 could be a good counter-example. In Figure 3, lb­-DR on Breakout shows consistently 25-30% larger value targets than the baseline, which is also the highest percentage among all the 20 environments. However, the performance of lb-DR is much worse than the baseline (and I believe Breakout is a deterministic environment where the proposed lower-bounding should apply based on the claims in the paper). Further investigation is needed to better understand when lower bounding the target is helpful.         \n\n\n[1] Brafman, Ronen I., and Moshe Tennenholtz. \"R-max-a general polynomial time algorithm for near-optimal reinforcement learning.\" Journal of Machine Learning Research 3.Oct (2002): 213-231.\n\n[2] Osband, Ian, et al. \"Deep Exploration via Randomized Value Functions.\" J. Mach. Learn. Res. 20.124 (2019): 1-62.     \n\n\n\n", "limitations": "The authors have mentioned the limitation of their work in Section 6, and I would encourage the authors to discuss a bit more on scenarios where the proposed lower-bounding method may fail, e.g., why the performance is poor on Breakout? The potential negative societal impact has not been discussed in the paper. As an illustrative example, what if the algorithm was used by a malicious user to build a robot? Targeting tasks with (sparse) rewards that lead to some harmful behaviors for a society? I believe such kind of scenario could be discussed. In addition, what can and should be done to prevent it from happening?   ", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The related work is well cited and discussed in the paper, and it is clear how this work differs from previous contributions. \n\n- The proposed method is simple, reproducible, and compatible with most state-of-the-art RL algorithms. \n\n\nWeaknesses:\n- There are several technical concerns in the paper. Specifically, \n\n    - I believe the main theoretical result, Theorem 2.3, does not hold if the lower bound function $f$ is fixed (or static). In the proof, for any $s$, where $f(s) \\geq v^*(s)$ and $f(s) \\geq B(v)(s)$, we have\n    $$ |B_f(v)(s) – v^*(s)| = |max(B(v)(s), f(s)) – v^*(s)| = |f(s) – v^*(s)| = f(s) – v^*(s) = Constant$$\n    So, the distance to the optimal value would not shrink under the new lower-bounded Bellman operator. If $f$ is updated together with the value function during learning, please do make this clear and clearly show how $f$ is updated together with $v$ in the main paper. Based on the current presentation, Theorem 2.3 does not hold. \n\n\n    - Results from value iteration (VI) do not directly generalize to RL settings where the transition dynamics and the reward function are unknown to the RL agent. This leads to a fundamental problem in RL that does not exist in VI, exploration vs. exploitation. However, the paper did not mention or address/discuss the exploration problem when transitioning from VI to RL, and thus the proposed method is not well justified in the RL setting.              \n\n    \n    - In general, the episodic return (or Monte Carlo estimate of the return) is not guaranteed to be a lower bound of the optimal value (or the expected return following an optimal policy). Although the authors mentioned this in Section 6 (Related Works) as the limitation of their work, it is still concerning to me in terms of the soundness and the algorithmic design of the proposed method.                \n\n\n- The clarity of the paper could be further improved, and the paper needs to be better organized in my opinion. I encourage the authors to move key results and plots to the main paper. Putting all experimental figures in the appendix would jeopardize the readability of the paper.     \n\n\nOverall, the paper feels more like a work in progress to me, and certain technical details may need a bit rework and the paper could be better organized. Unfortunately, I don’t think the current version can be accepted and vote for rejection. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SbOZSuVtmOg", "reviewer": "Reviewer_3Knv", "summary": "This paper suggests that learning a lower bound of the value function can facilitate dynamic programming such enhance RL performance. The author(s) provided theoretical insights about the optimality of the value function learning with their lower bounding method. It was claimed the proposed methods showed effectiveness in a wide range of tasks.\n", "questions": "How to implement your method in deep RL? Because Algorithm 1 is for tabular case.\n", "limitations": "See Weaknesses.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed method seems simple but somehow effective.\n2. Novel theoretical insights about lowerbounding the value function.\n3. The author(s) carefully explained the novelty of the proposed method in Sec.6.\n\n\nHowever, I could not evaluate the effectiveness of the proposed methods because all the empirical results are in the appendix. Meanwhile, as I have shortly scanned the appendix, the are also other things to notice, which I will details below.\n\nI think the manuscript in its current form is not acceptable as a NeurIPS conference paper. Because the important methods (how to implement in deep RL) and experimental results are all in the appendix, as well as some other problems exist. If the authors(s) cannot include all the essential methods and results in a conference paper, I recommand to submit to a journal.\n\nWeaknesses:\n\n1. As I said, the paper should be self-contained with essential practical algorithms and experimental results in its main texts.\n2. Estimating value function has always been an vital problem in deep RL. Although the paper discussed related work about lower bounding the value function, there are other studies pointed out that Q-learning like algorithms, e.g., SAC, tend to over-estimate the value targets , which is bad and should be regularized. For example, [1][2][3] used different methods to control the over-estimating bias of SAC and observed very large performance gain. The paper should discuss the relation to them and conduct comparison to at least one of them.\n\n\nminor issue I found:\n\nline 140, \"action value\" --> \"state-action value\"\n\n\n[1] Kuznetsov, Arsenii, et al. \"Controlling overestimation bias with truncated mixture of continuous distributional quantile critics.\" International Conference on Machine Learning. PMLR, 2020.\n\n[2] Chen, Xinyue, et al. \"Randomized Ensembled Double Q-Learning: Learning Fast Without a Model.\" International Conference on Learning Representations. 2021.\n\n[3] Hiraoka, Takuya, et al. \"Dropout Q-Functions for Doubly Efficient Reinforcement Learning.\" International Conference on Learning Representations. 2022.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Krh5NkjX25_", "reviewer": "Reviewer_zNRT", "summary": "The paper proposes value target lower bounding, where a function lower bounds the maximum achievable value is used as the target value.  Such a function admits the same convergence with the original Bellman operator but converges faster. The proposed value target lower bounding is claimed to be general and can be combined with any off-policy RL algorithms. The experiments verify this on SAC and DDPG. ", "questions": "Refer to above", "limitations": "Limitations are discussed in Section 7.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The proposed method appears to be simple and effective empirically.  \n- The proposed method is evaluated in several configurations. \n- The writing is clear and easy to follow. \n\n**Weaknesses**\n\n- The main concern is why the proposed method is not evaluated on DQN. The proposed method is claimed to be general (\"The value target lower bounds can be readily plugged into RL algorithms that regress value to a target, e.g. DQN, DDPG or SAC.\"), but only evaluated on off-policy actor-critic style methods, like SAC and DDPG. Is this because it does not work well on DQN?\n- Currently, the empirical return (or n-step return) is taken as the lower bounding function, which limits the proposed method to only valid when the environment is deterministic. This is one limitation of this work. \n- *Faster learning* is only shown empirically and intuitively. Are there theoretical results to support this claim?\n- It will be better to show the difference between the lower bounding and upper bounding (He et al., 2017).\n\n**originality and significance**\n- It is not totally new to use the bounding of target value. \n\n---\nAFTER REBUTTAL\n\nI thank the authors for the response. After reading the response and other reviews, I think the paper still has a few weaknesses/limitations: (1) limited to deterministic environments. It is better to show how to extend it to stochastic environments; (2) better to compare it with other methods, like upper bounding. \n\nNevertheless, I still enjoy seeing such a simple method works well empirically. So, I keep my score.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject