Batch Reinforcement Learning Through Continuation Method
Authors: Yijie Guo
Shengyu Feng
Nicolas Le Roux
Ed Chi
Honglak Lee
Minmin Chen
Venue: ICLR 2021 Poster (2021)
Keywords: batch reinforcement learning
continuation method
relaxed regularization

Abstract: Many real-world applications of reinforcement learning (RL) require the agent to learn from a fixed set of trajectories, without collecting new interactions.  Policy optimization under this setting is extremely challenging as: 1) the geometry of the objective function is hard to optimize efficiently; 2) the shift of data distributions causes high noise in the value estimation. In this work, we propose a simple yet effective policy iteration approach to batch RL using global optimization techniques known as continuation.  By constraining the difference between the learned policy and the behavior policy that generates the fixed trajectories, and continuously relaxing the constraint, our method 1) helps the agent escape local optima; 2) reduces the error in policy evaluation in the optimization procedure.   We present results on a variety of control tasks, game environments, and a recommendation task to empirically demonstrate the efficacy of our proposed method.

Published as a conference paper at ICLR 2021
BATCH REINFORCEMENT LEARNING THROUGH CONTINUATION METHOD
Vijie Guo- Shengyu Fengb Nicolas Le Roux²
Ed Chi Honglak Leel.2 Minmin Chen
1 University of Michigan Google AI
{guoyijie, shengyuf) @ @umich. edu {nlr, edchi, honggli,, http:/ak...mmmmme/pooooo......
ABSTRACT
Many real-world applications of reinforcement learning (RL) require the agent to learn from a fixed set of trajectories, without collecting new interactions. Policy optimization under this setting is extremely challenging as: 1) the geometry of the objective function is hard to optimize efficiently; 2) the shift of data distributions causes high noise in the value estimation. In this work, we propose a simple yet ef- fective policy iteration approach to batch RL using global optimization techniques known as continuation. By constraining the difference between the learned pol- icy and the behavior policy that generates the fixed trajectories, and continuously relaxing the constraint, our method 1) helps the agent escape local optima: 2) re- duces the error in policy evaluation in the optimization procedure. We present results on a variety of control tasks, game environments and a recommendation task to empirically demonstrate the efficacy of our proposed method.
INTRODUCTION
While RL is fundamentally an online learning paradigm, many practical applications of RL al- gorithms, e.g., recommender systems [5, 7] or autonomous driving [36], fall under the batch RL setup. Under this setting, the agent is asked to learn its policy from a fixed set of interactions collected by a different (and possibly unknown) policy commonly referred to as the behavior policy, without the flexibility to gather new interactions. Realizing the interactive nature of on- line RL has been hindering its wider adoptions, researchers strive to bring these techniques of- fline [24, 11, 20, 23, 31, 12, 21, 2, 32, 8J. We focus on policy optimization under batch RL. setup- As pointed out in [3. 26]. even with access to the exact gradient, the loss surface of the objective function maximizing the expected return is difficult to optimize, leading to slow convergence. Chen et al. [8] show that the objective function of expected return exhibits sub-optimal plateaus and expo- nentially many local optima in the worst case. Batch setup makes the learning even harder as it adds large variance to the gradient estimate, especially when the learned policy differs from the behavior policy used to generate the fixed trajectories. Recent works propose to constrain the size of the policy update [27, 28] or the distance between the learned policy and the behavior policy [14, 21]. The strength of that constraint is critical hyperparameter that can be hard to tune [28], as a loose constraint does not alleviate the distribution shift while a strict one results in conservative updates.
Here we propose to address the challenges using continuation methods [35, 6, 17). Continuation methods attempt to solve the global optimization problem by progressively solving a sequence of new objectives that can be optimized more efficiently and then trace back the solutions to the orig- inal one. We change the objective function of policy optimization by including an additional term penalizing the KI. divergence between the parameterized policy To and the behavior policy. We then gradually decrease the weight of that penalty, eventually converging to optimizing the expected return. With this additional constraint, we benefit from more accurate policy evaluation in the early stage of training as the target policy is constrained to be close to the behavior policy. As training continues, we relax the constraint and allow for more aggressive improvement over the behavior policy long the policy evaluation is still stable and relatively reliable, i.c. with small enough variance. By doing so, the proposed method exhaustively exploits the information in the collected trajectories while avoiding the overestimation of state-action pairs that lack support.
The contributions of this paper are as follows: (1) We propose a soft policy iteration approach to batch RL through the continuation method. (2) We theoretically verify that in the tabular setting with exact gradients, maximizing KL regularized expected return leads to faster convergence than optimizing the expected return alone. Also, our method converges the globally optimal policy if there are sufficient data samples for accurate value estimation.
(3) We demonstrate the effectivenesss of our method in reducing errors in value estimation using visualization; (4) We empirically verify the advantages of our method over existing batch RL. methods on various complex tasks.

---

Published as a conference paper at ICLR 2021
2 RELATED WORK
Batch Reinforcement Learning. Off-policy reinforcement learning has been extensively studied [11, 20, 30, 23, 311 11]. with many works [12, 21, 2] focusing on variants of Q-learning. Fujimoto et al. [12], Kumar et al. [21] investigated the extrapolation eπror in batch RL resulting from the mismatch of state-action visitation distribution between the fixed dataset and the current policy, and proposed to address it by constraining the action distribution of the current policy from deviating much from the training dataset distribution. Recent works [29, 33] studied policy iteration under batch RL. The Q function is estimated in the policy evaluation step without special treatment while the policy updates are regularized to remain close to the prior policy with a fixed constraint. To further reduce uncertainty in Q learning, an ensemble of Q networks [21, 29] and distributional Q-function [2, 33] are introduced for the value estimation. [34, 18] use the KL divergence between the the target policy and the hehavior policy as regularization term in the policy update and/or value estimation. The constraint is controlled by a fixed weight of the KI. regularization KL divergence. While all of these works apply a fixed constraint determined by a sensitive hyper- parameter control the distance between the behavior/prior policy and the target policy, we focus on gradually relaxed constraints.
fixed threshold for the
Constrained Policy Updates. Several works [27, 1, 15] studied constrained policy updates in online settings. Kakade & Langford [19] show that large policy updates can be destructive, and propose a conservative policy iteration algorithm to find an approximately optimal policy. Schulman et al. [27] constrain the KI. divergence between the old policy and new policy to guarantee policy improvement in each update. Grau- Prrau-Myya&e et al. [15] force the policy to stay close to a learned prior distribution over actions, deriving inntualinfforattinn regularization between state and action. Cheng et al. [9] propose to regularize in the function space. Again these methods focused on a fixed constraint while are interested in continuing relaxing the constraint to maximize the expected return eventually. Also none of these methods have been extensively tested for batch RL with fixed training data.
Continuation Method. Continuation method [35] is global optimization technique. The main idea is to transform a nonlinear and highly non-convex objective function to a series of smoother and easier optimize objective functions. The optimization procedure is successively applied the new functions that are progressively more complex and closer to the original non-contexi problem, to trace their solutions back the original objective function. Chapelle al. [6] use the continua- tion method to optimize the objective function of semi-supervised SVMs and reach lower test error compared with algorithms directly minimizing the original objective. Hale et al. [17] apply the con- tinuation method to ll I1-regularized problems and demonstrate better performance for compressed sensing problems. Inspired by prior works, we employ the continuation method to transform the objective of batch RI. problems by adding regularization. We gradually decrease the regularization weight to trace the solution back to the original problem.
METHOD
In classical RL, an agent interacts with the environment while updating its policy. At each step t, the agent observes state st S, selects an action a₂ E A according to its policy to receive a reward r: r(s,, 4z) S x A - RR and transitions to the next state St-1 ~ P(.lss, oz). The state value of a policy π at a state s is V V*(s)
E ( (1 [--((- EE 7 7'r(s,, a.)]. y E [o. 1] is the discounting factor. At each step, the agent updates the policy so that the expected return (p) = EB-ppV [V"(s)] (where p is the initial state distribution) is maximized.
In batch RI. the agent is not allowed to interact with the environment during policy learning. In- stead it has access to a fixed set of trajectories sampled from the environment according to a behav- ior policy'. A trajectory {so, do, ro), (31, a1,
, (ST, ar, [T)} is generated by sampling so from the initial state distribution p. sampling the action a, ~ (((ss:) at the state s; and moving to 8141 P( st, a.) for each step t [0, 1, .............................. .). The length T can vary among trajectories. We then convert the generated trajectories to a dataset D = {(s,, ag Fi, s 1 where s is the next state after 8; in trajectory.
(1,r)), rT),...
The goal of batch RL is to learn a parameterized policy ny with the provided dataset to maximize the expected return V*(p). In Sec. 3. will first introduce a new objective function 57 (p), i.e. the expected return of policy π with KL regularization term and the regularization weight 7. With exact gradients, V (p) can be optimized more efficiently than the original objective (²) With the
If the behavior policy is not known in advance, it can be fitted from the data [30, 71.
2

---

Published as a conference paper at ICLR 2021
continuation method, solving sequence of optimization problems for (p) with decaying value of T converges toward optimizing (*(p) and makes the optimization easier. In Sec. 3.2, we derive soft policy iteration with KL regularization to optimize V*.T (p), without the assumption of exact gradients. Finally, in Sec. 3.3, we propose practical batch RI. algorithm with value estimation for target policy based on this theory.
3.1 OPTIMIZING EXPECTED RETURN WITH KL REGULARIZATION
In batch RL, the distribution of the trajectories generated by the behavior policy can be very different from that of the learned policy. We thus restrict the learned policy to stay close to the behavior policy via the regularization of KL divergence. Define the soft state value of a policy x at a state s as
V (s) E =0 (1))
Σ
(((s).a) T log
x(a st)
(1)
B(ar se)
where the temperature parameter T controls the deviation from B. The new objective function be- comes V< (p) "E This KL. regularized objective differs from the original objective (p). which however can be recovered as r 0.
As pointed out in [3], even with exact gradients, the objective function (T(p) is still difficult to optimize due to its highly non-smooth landscape. Mei et al. [26] further prove that, in tabular set- ting with softmax parameterized policy and exact gradients, the vanilla policy gradient method (i.e. directly updating the parameters of policy π to maximize (p(p) with gradient descent) converges to the global optimal policy at convergence rate O(L/t), while the regulpy-regulrrized policy gradi- ent enjoys a significantly faster linear convergence rate O Motivated by this line of work, we investigate the convergence rate of optimizing V (p) with the exact gradient descent and compare it with the vanilla policy gradient method. We study the smoothness and Lojasiewicz inequality for the function V (p) to prove the convergence rate, similar to [26]. The detailed proofs of all following theorems are provided in the appendix.
Theorem 1. In the tabular setting with softmax parameterized policy Te, maximizing V. (p) using (1-7)²
policy gradient with the learning rate 1) (8M++ log for- all > 1, we have
p (p) - Vao, (p) S C. C-ltt-). M + r log
(1 7)2
where πテ is the optimal policy maximizing V (p). M is the bound of the absolute value of +)+ (1-y)* T log ([ass), A is the size of action space, S is the size of state space, Cr x 3LM/7+4+8Mm log A).S. and C is a constant independent with t and T.
Theorem states that KL regularized expected return 5ि, (p) can be optimized with a convergence rate O(e=t) rather than the O(1/t), the convergence rate of vanilla policy gradient for expected return alone. The faster convergence inspires us to optimize V (p) to reach policy n;, then use n. as initialization, gradually decrease the temperature towards O, and eventually move from o, to T = arg max; (''(p.. With a reasonable value of r, we enjoy a linear convergence rate toward n; from the randomly initialized policy Ty. As decreases, π gets closer x². The final optimization of VT (p) from x. can be much faster than from a randomly initialized n..
We construct toy example to illustrate this motiva- tion. In the grid world (Fig. la), the start state, anno- tated with 'S', is in the center and the terminal states are marked in yellow. There are only two states with positive rewards (0.9 and 1). There are four actions {up, down, left, right). A badly initialized policy Te, is shown as arrows in Fig. la). The initializa- tion results in poor policy, having high tendency to go right toward a terminal state with zero reward. The vanilla policy gradient method (i.e. maximiz- ing (p(p) with true gradient) starting from this ini- tial point takes more than 7000 iterations to escape a sub-optimal solution (Fig. 1b). In contrast, we es- cape the sub-optimal solution much faster when ap- plying the continuation method to update the policy with the gradients of V (p), where the be- 1,2,3, 3, randomly sampled from 2/(0, 1] and
0.9
32-
3000 4200
6000
300
1200
(a)
(b)
Figure 1: (a) grid world with sparse rewards. (b) Learning curve of the value of learned pol- icy тa, We conduct a hyper-parameter search for the learning rate 5, 1,0. 1.0.0.00000000......... 10.00.00,,0000 and report the best performance for each method.
havior policy B(-1s)
[ui, 02, Ug us) with ue,
3

---

Published as a conference paper at ICLR 2021
Algorithm Soft Policy Iteration through Continuation Method
Initialize: actor network лT. ensemble critic network (Quis, Qater... PacKy) J. behavior pol-
icy network Sg. penalty coefficient т, decay rate X, number of iterations for each 7
2: Input: training dataset D = {(s;, Gz Pt. kmm
3: for update j 0, 1, do
4: Sample batch of data {(ss, as r:)] from D
5: # Learn the behavior policy with behavior cloning objective
6: Update & to maximize Di=1 log 3 (a,)s;)
7:
# Train the critic network
Update (66) to minimize the temporal difference BB Σn a))
=i (r, +V - (scss))
8:
9:
where V(s)
Σx=1 Euwsro--ss) (Q (), (
10:
# Train the actor network
Update 0 maximize B Di Σ Ba-wa(-les) (Qacs (st, a)) ((d)))-K(((nn (1)
12:
# Decay the weight of KL regularization T for every I updates
13:
j mod = then
14:
15: end if
end for
normalized for each state s. In Fig. 1b, as we decrease т, the value of learned policy Te, for each it- eration i quickly converges to the optimal value. In other words, optimizing sequence of objective functions V. (p) can reach the optimal solution for V (*(p) significantly faster.
3.2 SOFT POLICY ITERATION WITH KL REGULARIZATION
As explained in the previous section, we focus on the new objective function 5. (p). which can be optimized more efficiently, and use continuation method to relax toward optimizing V (p)). Batch RL adds the complexity of estimating the gradient of V=. with respect to from fixed set of trajectories. We propose adapt soft actor-critic[ a general algorithm learn optimal max- imum entropy policies in batch RL for our use case. We change the entropy regularization to KL regularization and derive the soft policy iteration to learn KL regularized optimal policy. For a policy and temperature т, the soft state value is defined in Eq. and soft Q function is defined as:
Qn (s. = r(s, -B.s-(()).a))) (T"
(2)
In the step of soft policy evaluation, we aim to compute the value of policy according to the minimum KL divergence objective V Esspp (VT. (s)]. According to Lemma in Appendix, the soft Q value can be computed by repeatedly applying the soft bellman backup operator.
x (a
T Q(s, a) = r(s, a) (as5,a++70 +7 (V (s')), where V(s) = Ea-r(1s)
Q(s, a) - T log B(a
In the step of policy improvement, maximize the expected return based on Q-value evaluation with the KL divergence regularization. The following policy update can be guarantced to result in an improved policy in terms of its soft value (Lemma 2 in Appendix).
Tnew (+|s) arg max Eassr(lss)
(Qmotss" (s, - - (
(3)
HEN
log (als))
where K (L(T(1s)))((1s)) -E (a)
3(a
The soft policy iteration algorithm alternates between the soft policy evaluation and soft policy improvement, and will provably converge to the optimal policy maximizing the objective / (p).
Theorem 2. Repeated application of soft policy evaluation and soft policy improvement converges to policy गテ such that QT): > (T.((s, a)) for any E Il and (s, a) E S A.
The soft policy iteration finds a policy π.. with optimal soft Q value for each state-action pair and hence gets the optimal value of V= (p). Here propose objectives (p= with decreasing value of r and move back to the objective V*(() as = 0. The method is guaranteed to asymptotically converge the optimal policy n² for the objective VT()).
the soft policy iteration solve
4

---

Published as a conference paper at ICLR 2021
Error in Q at iteration 0
Error in Q
Error in Q
Error Q
Learned policy '
at iteration 250
at iteration 500
at iteration 750
at iteration 1000
Constant 0.001
AAA AAA
Constant T=1
Continuation 1=1
Figure 2: Visualization of the error in soft Q value estimation and quality of the learned policy. In the first four columns, triangles represent the error for actions that move in different directions. Darker color indicates higher error. To investigate the performance of the learned policy Tono the length of arrows represents the probability taking each actions in states. run Re the grid world and visualize the visitation count in the last column (heatmap). Darker color means more visitation.
Theorem 3. Let n. (a(s) ) the optimal policy from soft policy iteration with fixed temperature т. We have क. (als) x exp (*Ti"((aa) B(als). As t + 0, aT(a)s) will take the optimal action a² with optimal Q value for state s.
3.3 ERROR IN VALUE ESTIMATE
In the previous section, we show that the soft policy iteration with the continuation method provably converges the global optimal policy maximizing expected return. However, in batch RL with fixed dataset and limited samples, we cannot perform the soft policy iteration with KL regularization in its exact form. Specifically, in the policy evaluation step, when the learned policy deviates for the behavior policy 3, and chooses the state-action pair (s, a) rarely visited by 3, the estimation of target r(s, a) + +-,.,=p((ss... (s')) can be very noisy. The error in the value estimate Q(s, a) will be further propagated to other state-action pairs through the bellman update. Finally, inaccurate value estimation will cause errors the policy improvement step, resulting in a worse policy. On the other hand, if we constrain the learned policy to be very close to the behavior policy 8, we can expect the policy evaluation be reliable and safely update the learned policy. The tight constraint however prevents to be much better than B due the conservative update.
On the grid world, we study this problem value estimation with different values of r. Figure vi- sualizes the propagation of Q value estimation errors and the learned policies. We assume a mediocre behavior policy tending to move left and down. For the rarely visited states in the upper right part of the grid, there are eπrors in the value estimation of Q (s, a). i.c. IQ(s, a) (s.T(s.a) where Q(s, a) is the Q value we learn during training and Qu. (s,a) a) is the ground truth soft Q value. Because the bad initial policy (Fig. la) tends move towards the right part, without strong KL regularization, the policy evaluation can be problematic due to the errors of value estimation in the right part of the grid world. In Fig. 2, with a small KL regularization weight row shows that errors even propagate to the frequently visited states by the behavior policy. On the other hand, when we set large value of smaller. Yet the performance of the learned policy is not much better than the behavior policy. Our continuation method gradually moves the policy update between these two spectra. The value esti- mation benefits from the gradually relaxed KL. regularization and the errors remain small. The last column of Fig. 2 visualizes the learned policy in these methods. With constant r 0.001, the wrong value estimates in some states mislead the agent. It fails to visit any terminal state and gets stuck at the state in dark orange. With constant T = 1, the tight constraint of KL. divergence makes the learned policy close the behavior policy, mostly visiting the left bottom part of the environment. With continuation method, the agent learns to always take the optimal path moving left directly and obtains the highest expected return. More details of this example provided in the appendix.
0
0.001, the first
1 (second row), the error [Q(s, a) QT. (s. a) is
5

---

Published as a conference paper at ICLR 2021
In the toy example, gradually relaxing KL regularization towards zero alleviates the prop- agation of errors in the soft Q estimate and helps the agent converge to the optimal pol- icy. In more complicated domains, we find that as decays close to 0, the policy evalua- tion is still erroneous. To mitigate this issue, we introduce an ensemble of critic networks (Qga, Quees,.... , Qotar) to approximate the soft Q value, and monitor the variance of value timation in different critic networks to measure the uncertainty. Given a batch of data samples {s: C CD, vQr(Q*)= = D> Bart( (arttl:eearraaatt (s;, a), Qacar (5i; a),... Qate: (Si,a)) in- dicates whether the current policy и tends to take actions with highly noisy value estimation.
Our method is summarized Algorithm 1. Instead of running the soft policy evaluation and policy improvement until convergence, we alternate between optimizing the critic network and actor net- work with stochastic gradient descent. We set 7 to large value initially and let the KI. divergence term dominate the objective, thus performing behavior cloning. We record a moving average of the Q value estimation variance var(Q" var(Q"..o) over 1000 updates at the end of the phase. After that, we decay the temperature gradually with X 0.9 every / steps. When the moving average of the Q value estimation variance var(Q".") is large compared with the initial value var(Q".77) ) (i.e. at the end of behavior cloning), we no longer trust the value estimate under the current temperature T and take the policy checkpointed before the temperature decays to this J as our solution.
4
EXPERIMENTS
4.1 MUJOCO
We evaluate method with several baselines continuous control tasks. We train Proximal Policy Optimization agent [28] with entropy regularization for 1000 million steps in the environ- ments. We parameterize the policy using Gaussian policies where the mean is a linear function of the agent's state 0 8 and variance is an identity matrix to keep the policy simple as introduced in [3]. To generate training datasets D with varying quality, we construct the behavior policy by mixing the well-trained policy A( opt 0.5I), i.c. checkpoint with the highest score during training, and a poor policy N/O 0.51) i.e. checkpoint at the beginning of the training, with the weight a. Then the behavior policy ((-(s) is AN((1 - a 0 + atp)" s, 0.5I). We generate trajectories and store a total of one million data samples from the mixed behavior for different values of the coefficient a.
The architecture of the target policy is the same as the behavior policy. We consider six baseline approaches: BCQ [14], BEAR [21), ABM+SVG [29], CRR [33], CQL [22], BRAC [34]. For fair comparison, the architectures of the ensemble critic network and the policy network are the same in the baselines and our method, except BCQ which has no policy network. To evaluate and compare the methods, we run the learned policy in the environments for 100 episodes and report the average episode reward in Fig 3. As for the continuation method, we report the score of policy checkpointed last with reasonable value estimation variance, as explained in Section 3.3. For the baselines, we report the score of the final policy when we terminate the training at 1.5M updates.
BCQ
BEAR
ABM
CRR
CQL
BRAC
Ours
0.2 1908.0 +327.0
1661.2 + 163.3
1814.2 176.99
1861.5 112.88
1962.6 + 194.1
2375.4 + 181.3
2085.8 204.8
0.4 876.0 1462.2
+ 1028.9:199
1113.0 4201.3
1281.0 4218.2
1262.0+116.7
991.0 39.5
1463.7 - 195.0
Hopper
0.6 429.9 1
898.6 28.2
1402.6 +350.5
791.1
609.1 135.1
573.7 773.7441.6 41.6
1524.3 511.4
0.8 450.3 =174.2
3.3
3.6 #0.0
#36.3
295.7
#41.9
83.6 113.1
234.8 #26.4
0.2 1765.8 4-1.2
2143.4 + 18.3
2168.3 1621
2154.1 46.6
2105.7 +37.2
1649.3 + 64.3
2149.3 +32.9
Half
0.4 1336.4 #35.6
1809.3 1 18.1
1914.7 113.2
1860.
138.2
1811.9 160.7
1744.5 15.1
1839.9 166.9
Cheetah
0.6 513.6 #335.6
1149.9 39.3
1457.9
136.4
1161.7111.
143.3 1156.6
1245.1 = 67.2
1524.8137.6
0.8 179.0 #14.1
700.6
+ 11.4
600.4
+8.6
572.8 114.3
605.8 #:4.5
499.1#11
595,4# 19.1
0.2 1400.2 1:30.9
1414.6 19.4
1405.2
+57.3
1442.9
119.
1385.2 #73.4
437.5 + 638.9
1467.3
445.2
0.4 963.9 169.0
117.334211
1259.3
#27.7
1233.5 E35.8
979.6
+83.5
1311.5 - 29.7
1223.5+15.7
Walker
0.6 266.3 #56.9
486.3 = 43.3
664.9 #37.5
529.9
#46.8
218.8 #16.9
559.1 $59.1 37.6
872.9
+228.7
0.8 3.5 +5.0
10.9 + 0.9
5.9
+0.6
10.7
+0.3
3.3 46.55
5.6 0.9
7.0 6.3
Table I: Results on Mujoco. We show the average and standard deviation of the scores in independent runs.
Tab. 1 shows that method outperforms all the baselines on 5 settings. On the dataset with relatively reasonable quality (i.e. a = 0.2, 0.4, 0.6), ours performs comparable or better than the baselines. With a 0.2, i.e., close to optimal behavior policy, all the methods perform similarly and
6

---

Published as a conference paper at ICLR 2021
Hopper-v2 alpha0.6
HalfCheetahh-2 alpha0.6 6
Walker2d-v2 alpha0. 6
BCD
BEAR
1400
500
Aecroqe Revard
25.00
ADN
Aeerage Reward
Ineeage tarcord
600
2000
BMAC
1000
500
Our
Corsaatt
400
600
200
OCN B1MM 1.2M
1PMM 1.5M
update
update
update
Figure 3: Learning curves of average reward over runs on Mujoco tasks with a = 0.6. The shaded area with light color indicates the standard deviation of the reward. The gray vertical lines on the yellow curves indicate
where we take the checkpointed policy, according to the measure of Q value variance, as our final solution.
Amidar
Asterix
Breakout
Enduro
MsPacman
Qbert
Seaquest
SpaceInvaders
154.4
2466.7
203.8
604.7
2299.7
4088.3
4420.0
726.5
BCQ
+11.0
273.4
+ +19.6
+33.7
+150.2
+332.8
+548.7
+58.77
32.0
741.4
3.1
244.2
1997.4
2062.9
474.0
678.4
REM
+3.55
=236.7
+0.0
119.8
+6.4
#551..
+61.9
+:41.3
145.0
2618.7
253.7
206.6
2234.6
4094.7
4652.6
493.5
CQL
+16
=102.1
114.4
+
+203.2
+74.1
22017.1
#11.4
174.5
3476.7
199.0
922.9
2494.0
4732.5
9935.0
1070.3
Ours
77.1
229.0
+32.0
#31.7
+301.3
1172.5
11175.9
#137.1
Table 2: Results on Atari, the mean and standard deviation of scores achieved in 3 indenpendent runs.
one can achieve a good return by simply cloning the behavior policy. With a 0.8, i.e., low-quality behavior policy, there are few good trajectories in the dataset for any methods to learn. The advan- tage of our method is most obvious when a 0.6 (Fig. 3), as the dataset contains trajectories of both high and low cumulative rewards. Our method can learn from the relatively large number of good trajectories and at the same time deviate from the behavior policy to avoid those bad trajectories and achieve higher rewards. In Fig. 3, 'Constant' is the method of optimizing KL regularized expected reward with constant value of т. We search several values of 7 and report the best result. We can see that gradually relaxing constraint performs better than the fixed constraint. In Fig. 3(left), decays to close to O, the learned policy can degrade due to errors in the Q estimation, the stopping condition explained in Section 3.3 is however able to identify good policy before the degenerating point. More experimental details are in the Appendix.
4.2 ATARI
We further study our method on several Atari games from the Arcade Learning Environment (ALE) [4]. The rich observation space requires more complicated policies and makes policy optimization even more challenging. We focus on eight games and generate the datasets as discussed in Fu- jimoto et al. [13]. We use mediocre DQN agent, trained online for 10 million timesteps (40 million frames). The performance of the DQN agent is shown 'Online DQN' in Fig. 4. We add exploratory noise on the DQN agent (at 10 million timesteps) to gather a new set of million transitions, similar to [13]. The line "Behavior" in Fig. 4 shows the average of trajectory reward in the dataset D. The dataset D is used to train each offline RL agent. We compare with BCQ [13], REM [2] and CQL [22] because they are recently proposed offline RI. algorithms and work well on Atari domain. For evaluation, we run 10 episodes on the Atari games with the learned policies and record the average episode reward (Fig. 4). Tab. 2 summarizes the performance of BCQ, REM and CQL after 6M updates. For our method, we report the score before the variance of Q estimate becomes too high.
Our approach achieves higher scores than the baselines on 7 out of 8 games, and perform compara- bly on the other one. Agarwal al. [2] reports that REM performs well in the dataset consisting of the entire replay experiences collected in the online training of the DQN agent for 50M timesteps (200M frames). We hypothesize that learning on the entire replay experience makes the setup easier as the training dataset contains more exploratory and higher quality trajectories. With the dataset of much smaller size and worse quality, REM performs poorly in this single behavioral policy setting. We use the same architecture of the critic network for both our method and BCQ with ensemble of 4 Q networks. As mentioned in [13], BCQ only matches the performance of the online DQN on most games. In contrast, ours is able to outperform online DQN significantly on several games. As pre- sented in [22], on the Atari dataset, CQL performs better than REM, while our method outperforms CQL in 7 out of 8 datasets.
7

---

Published as a conference paper at ICLR 2021
AmidarkoFrrmsskto vo
AgserixidFramcsiiboo vo
Breakiouthorrameskpoo vv
Endurotooramcssip vb
000
1200
1.50
Aue Aue Rceart
2000
fuertge Rrward
600
100
Aif
2002-
220
150
500
50-
2002
100
10-
200
09
2.3H
2
4.5M
OM
DM
1,2
2.4H
3.5M
4.MM
051
1.3M
2
3
4.3M
CN
CM
1.2M
2,41
3.0M
4.00
McPhermentofr ameesip--ll
Obert rameskiipua
Spacetrvaderston wame
3000
5002
25:00
Avergge Hoaard
4002
2000
100
#000
2000
Jvengy
1000
500-
1000
200
400
Mym
3
1.2M 2.4M Updiskea
Updilees
Lydates
Updates
Figure 4: Learning curves of average reward over 3 runs on Atari games.
data with avg. reward 0.49
data with avg. reward 0.63
data avg. reward 0.81
sample size
30,000
60,000
30,000
60,000
30,000
60,000
Cross Cooss-Entropy
74.3+0001
73.5+0.01
82.0=+0.01
81.3=1001
86.7=0.00
85.6+t0.00
IPS
80.5+0..33
82.1=0.02
86.4
88.0=
87.2 = 0.02
90.0 + 0.01
Ours
83.0
85.8=0.00
88.7+00001
90.1 0.00
89.4 0.00
90.5
Table 3: Results on Moviel.ens dataset. The number is precision at 10, i.c. the percent of recommended movies getting positive feedhack when we use the well-trained policy to recommend top- movies to the test users. We report the mean and standard deviation of the precision over 20 independent runs, runs,
4.3 RECOMMENDER
We also showcase our proposed method for building a softmax recommender agent. We use a publicly available dataset MovieLens-IM, popular benchmark for recommender system. There are million ratings of 3,900 movies (with the title and genre features) from 6,040 users (with demographic features). The problem of recommending movies for each user can be converted to contextual bandit problem, where we aim to learn a target policy wo (a(as) (a]s) selecting the proper action (movie) for each state (user) s to get high reward (rating) in a single step. The ratings of 5-score are converted to binary rewards using a cutoff of 4. To evaluate whether a learned target policy works well, ideally we should run the learned policy in real recommendation environments. However, such environments for online test are rarely publicly available. Thus, we use the online simulation method. We train a simulator to predict the immediate binary feedback from user and movie features, and the well-trained simulator can serve as proxy of the real online environments, because it outputs the feedback for any user-movie pair. Similar to [25], we train the simulator with all records of logged feedback in MovieLens- IM dataset. The behavior policy is trained with partial data in MovieLens-
We then construct the bandit datasets D = {s), a, Tt, 3(a, [(a.ss)))> N of different size and quality, by using different behavior policies / to select movies 0; for users 8; and getting the binary feedback r; from the well-trained simulator. We train offline RL agents on the generated dataset D and use the simulator to evaluate the learned policies on held-out test set of users.
We compare our method with two baselines, as they are commonly used in current industrial recom- mender systems [10, 7J. (1) Cross-Entropy: a supervised learning method for the softmax recom- mender where the learning objective is the cross-entropy loss (C) - N Σn I's log Ta (a. gradient method introduced in [7] with the learning objective N Σ B(s,.,)) r, log To(a,) (s)) where sg indiçates a stop-gradient operation. Jirs(P) (0) produces the same gradient as that of the function -N Σi= r; B[a.)e,) Thus, minimizing the loss (0) is to maximizing the expected return with importance sampling. (3) Ours: in the bandit setting, we simply perform IPS with gradually decaying KL regularization since estimating the soft Q from bellman update is not needed. Tab. 3 clearly demonstrates the advantage of our proposed method over the baselines. IPS can be viewed as vanilla policy gradient with importance sampling to correct the distribution shift. Our method clearly outperforms it across datasets collected using different behavior policies.
(2) IPS: the off- off-policy (g) Jirs(()
1
To(a.]s.).
8

---

Published as a conference paper at ICLR 2021
5 CONCLUSION
We propose a simple yet effective approach. soft policy iteration algorithm through continuation method to alleviate two challenges in policy optimization under batch reinforcement learning: (1) highly non-smooth objective function which is difficult to optimize (2) high variance in value esti- mates. We provide theoretical ground and visualization tools to help understand this technique. We demonstrate its efficacy on multiple complex tasks.
REFERENCES
[1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning coerning-Volumee 70, pp- 22-31. JMLR. org, 2017.
[2] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. Striving for simplicity in off- policy deep reinforcement learning. arXiv preprint arXiv:1907.04543 2019.
[3] Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understand- ing the impact of entropy on policy optimization. In International Conference on Machine Learning, pp. 151-160. PMLR, 2019.
[4] Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[5] James Bennett, Stan Lanning, et al. The netflix prize. Citeseer, 2007.
[6] Olivier Chapelle, Mingmin Chi, and Alexander Zien. A continuation method for semi- supervised svms. In Proceedings of the 23rd international conference on Machine learning, pp. 185-192. 2006.
[7] Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. Top- off off-policy correction for a reinforce recommender system. Proceedings of the Twelfih ACM International Conference Web Search and Data Mining, pp.
2019.
[8] Minmin Chen, Ramki Gummadi, Chris Harris, and Dale Schuurmans. Surrogate objectives for batch policy optimization in one-step decision making. In Advances in Neural Information Processing Systems, pp. 8825-8835, 2019.
[9] Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel W Burdick. Control regularization for reduced variance reinforcement learning. arXiv preprint 1rXiv:1905.05330, 2019.
[10] Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommen- dations. Proceedings of the 10th ACM conference on recommender systems, pp. 191-198, 2016.
[11] Damien Emstt Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6(Apr):503-556, 2005.
[12] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning 2018.
without exploration. arXiv preprint
[13] Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint arXiv:1910.01708, 2019.
[14] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2052-2062. 2019.
[15] Jordi Grau-Moya, Felix Leibfried, and Peter Vrancx. Soft q-learning with mutual-informann information regularization. 2018.

---

Published as a conference paper at ICLR 2021
[16] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018.
[17] Elaine T Hale, Wotao Yin, and Yin Zhang. A fixed-point continuation method for 11- regularized minimization with applications to compressed sensing. CAAM TRO7-07, Rice University, 43:44, 2007.
[18) Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00453 2019.
[19] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learn- ing. In ICML, volume 2, pp. 267-274, 2002.
[20] Shivaram Kalyanakrishnan and Peter Stone. Batch reinforcement learning in complex do- main. In Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems, pp. 94, ACM, 2007.
[21] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. In Advances in Neural Information Pro- cessing Systems, pp. 11784-13 19794, 2019.
[22] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
[23] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Rein- forcement learning, pp. 45-73. Springer, 2012.
[24] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[25] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Ji Yang, Minmin Chen, Jiaxi Tang. Lichan Hong, and Ed H Chi. Off-policy learning in two-stage recommender systems. In Proceedings The Web Conference 2020, pp. 463-473, 2020.
[26] Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global conver- gence rates of softmax policy gradient methods. arXiv preprint arXiv:2005.06392 2020.
[27] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
[28] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
[29] Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Neunert, Thomas Lampe. Roland Hafner, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint trXiv:2002.08390 2020.
[30] Alex Strehl, John Langford, Lihong Li, and Sham M Kakade. Learning from logged implicit exploration data. In Advances in Neural Information Processing Systems, pp. 2217-2225, 2010.
[31] Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. Journal of Machine Learning Research, 16(1):1731- 1755, 2015.
[32] Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforce- ment learning. In International Conference Machine Learning, pp. 2139-2148, 2016.
[33] Ziyu Wang, Alexander Novikov, Konrad Zomn, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. arXiv preprint arXiv:2006. 15/44, 2020.
10

---

Published as a conference paper at ICLR 2021
[34] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learn- 2019.
ing. arXiv preprint
[35] Zhijun Wu. The effective energy transformation scheme as a special continuation approach to global optimization with application molecular conformation. SIAM Journal on Optimiza- lion, 6(3):748-768, 1996.
[36] Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. Bdd100k: diverse driving video database with scalable annotation tooling. arXiv preprint 1rXiv:1805.04688 2018.


Reviews:
[{"review_id": "jiMEpu9fbd", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a KL regularized approach for batch RL, where the importance of the KL term is reduced during learning. Theoretical guarantees are provided in the tabular domain. The algorithm is tested in several domains (MuJoCo, Atari, and a recommender task). \n\nStrengths:\n- Although the theoretical results are mostly straightforward extensions of existing results, they provide a solid backing to the method. In particular, Theorem 1 is an interesting result which motivates KL regularization in this setting. \n- The experimental section is very thorough. I would prefer more seeds but the coverage over many domains and behavior policies is convincing evidence for the empirical success of the method.\n- The appendix is very comprehensive. Although code is missing, the method and experiments are reproducible with the provided descriptions.\n- The writing was clear, and the method was well-motivated. Overall, the paper feels fairly complete and well polished. \n\nWeaknesses:\n- There is one very glaring weakness to this paper- the proposed KL regularized approach for offline RL already exists (Wu et al., 2019) & related (Jacques, et al., 2019). This is additionally problematic as both methods are not cited or discussed in the paper. To the best of my knowledge, the continuation aspect is still novel as well as the theoretical contributions. However, a discussion on (Wu et al., 2019) is necessary.\n- My first impression was that the variance solution to checkpointing $\\tau$ in Section 3.3 was somewhat \"hacky\". On second thought, however, as suggested by the authors in the introduction, gradually reducing $\\tau$ provides a mechanism for searching for the \"optimal\" value which trades between the constraint and learning on top of the proposed benefits of continuation for optimization. I think this is an interesting component of the method. On the plus side I think the checkpointing solves an important problem for batch RL, but on the downside, I think measuring the variance of the ensemble is not as well motivated as the rest of the method. I think the paper could benefit from additional discussion, or experiments which examine this aspect further. \n- While the method is the \"best\" over a wide range of domains, the performance benefit seems fairly incremental. Consequently, for most users it's unclear if the benefits are sufficiently significant to warrant the additional complexity.\n\nRecommendation:\n\nSo firstly, the novelty concerns absolutely need to be addressed and the mentioned papers cited/discussed in the paper. Regardless, I do feel like there is a meaningful contribution that builds on this prior work at both a theoretical and empirical level. As a result, I'm leaning on the side of accept.\n\nReferences\n- Wu, Yifan, et al. \"Behavior regularized offline reinforcement learning.\" 2019.\n- Jaques, Natasha, et al. \"Way off-policy batch deep reinforcement learning of implicit human preferences in dialog.\" 2019.\n\n**Post-Rebuttal\n\nThe authors have addressed most of my concerns. I have increased my score. Although the additional experiments on the variance/checkpointing are helpful I would still like to see more discussion in the paper itself. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Wg1INsIFxPy", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:\nThis paper proposed a new batch RL algorithm based on the continuation method in numerical optimization. The proposed method makes use of KL regularization in the offline policy optimization process, with a decreasing temperature parameter -- which comes from the continuation method of optimization. The paper shows that the policy learning with the KL regularized value yields a faster convergence rate, as the motivation of using KL regularization, and a decreasing temperature will ensure the final convergence to the optimal policy. Experiments on Mojoco, atari, and a recommender data-set shows that the proposed algorithm is effective.\n\nJustification for the score:\nThe proposed algorithm is a natural improvement over the current behavior regularized policy optimization methods. The intuition from the continuation method provides a justification for using a decreasing temperature. I think the main merit of this paper is the solid experiment, in simulation tasks with discrete actions and continuous actions, and in a real data-set. This contribution seems solid, but I have some concerns about Theorem 1 and important related work that is missed.\n\nDetailed comments:\nPro:\n1. The intuition behind the algorithmic change is clear enough. Section 3.1 gives the motivation of using a KL regularization and section 3.3 gives an illustrative example of why the continuation method can be better than a constant threshold.\n2. The experiment section covers the standard RL benchmark in both continuous and discrete action settings. It additionally studied the performance of the proposed algorithm in a real dataset. Since while most batch RL work only run experiments in simulation tasks, this is a good step to bring the algorithm close to the motivation of doing batch RL.\n\nCons:\n1. Theorem 1 seems to be a bit disconnected from the main contribution of the paper and it's hard to understand its role for me. Later theoretical analysis and practical approximation are all based on the policy iteration algorithm, but Theorem 1 seems to be based on policy gradient. Could the convergence rate improvement be shown in the policy iteration case? Additionally, Theorem 1 says \"maximizing $\\widetilde{V}^{\\pi,\\tau}$\", does that mean theorem 1 consider the on-policy setting? In such a case what does $\\beta$ mean?\n2. The proposed algorithm seems to be very related to the BRAC framework and the algorithm BRAC with the KL value penalty. Why is it not mentioned at all? The BRAC paper may not directly use such a decreasing temperature or link it to the continuation method, but as prior work in batch RL, it also considered KL regularization and even studied using an adaptive regularization coefficient. \n3. More recent batch RL algorithms like BRAC, CQL, etc also reported their result in mujoco and atari (CQL did) domains and seems to be better than BCQ, BEAR, and REM. An open dataset D4RL has the reported performance of those more recent baselines. I think in general it will be better to compare with these more recent baselines (especially BRAC since it's very related.).\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3BH34ALeFD-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "While the title of the paper suggests that it leverages techniques from the vast literature on numerical continuation, the proposed approach is much more specific. The main idea consists in annealing the temperature parameter of the soft Bellman operator, and to warm start the the corresponding series of problems across time. In the language of numerical continuation, this approach fits under the umbrella of \"natural parameter continuation\", which I would describe succintly as \"warm starting\". \n\nFrom the perspective of addressing the challenging optimization landscape presupposed in many problems, the combination of \"continuation\" + smooth approximation to the optimal equations makes sense. However, for me the narrative doesn't hold when the authors motivate their method in the context of offline batch policy gradient methods. As the authors point out, the main challenge associated with offline data is the inability to sample new data. This is problematic in the policy gradient setting because our derivative estimator only holds under the distribution under which the samples have been collected. As soon as the policy parameters are updated, the distributional shift should be addressed via an appropriate change of measure (or via a model). To me, this is the main challenge in the off-policy setting and the proposed continuation-based solution does not address this issue.  I view this as a Monte Carlo estimation problem first; not one pertaining to the optimization landscape. \n\nPerhaps the paper should have been named differently because the remaining theoretical contributions in the paper do not pertain to \"continuation\" per se. Theorem 1 provides a bound on the policy gradient methods with a softmax policy (this is different from the \"soft\" optimality equations). Theorem 2 and 3 follow from the results in Rust (1994, 1996) and in econometrics where the smooth (soft) Bellman operator has been widely used. Theorem 2 follows from the perspective of policy iteration as an application Newton-Kantorovich to the smooth Bellman optimality equations. Theorem 3 follows from Dini's theorem where $\\lim_{\\tau \\to 0} T^\\star_\\tau v = T^\\star v$ where $T_\\tau$ would be the smooth (soft) Bellman operator and $T^\\star$ the usual Bellman operator. See Rust 1996 \"Numerical Dynamic Programming in Economics\", section 4, more specifically equations 4.2 and 4.4\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0STSDnEhEAb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\nThe paper extends soft actor-critic (SAC) to the batch RL setting, replacing the policy entropy in the objective function with the KL divergence from the behavioral policy. The temperature parameter tau weighting the reward agains the KL term is annealed towards zero during the optimization process, which corresponds to starting with behavioral cloning for high values of tau and ending up with the standard reward maximization RL objective for tau=0. Theoretical analysis and experiments confirm the advantages of the proposed method.\n\nDecision\n-----------\nI vote for accepting the paper. The idea of annealing the KL constraint is simple and elegant. Although it is very similar to other constrained policy update methods discussed in the Related Work section, the evaluation in the batch RL setting and demonstration of the improved convergence properties is novel. The execution is of high quality, with evaluations on tabular problems, MuJoCo, Atari, and a contextual bandit problem for movie recommendation.\n\nQuestions\n--------------\n1. As pointed out in Sec. 3.3, when the policy deviates too much from the behavioral policy, the value estimate becomes erroneous. Therefore, a criterion based on the ensemble variance of the Q-function estimates is proposed. Is there a way to derive such a criterion from first principles?\n2. Can you relate your work to [1]?\n3. Does your method work when the behavior policy is not known but only the dataset is available?\n4. Can you quantify how far the optimal policy is allowed to be from the behavioral policy? For example, on a pendulum swing-up task, if the behavioral policy is taking random actions and the pendulum always jitters at the bottom, inferring the optimal policy from this data appears quite challenging. Can one give some criteria when the method is expected to work well?\n\nReferences\n---------------\n[1] Nachum, O., & Dai, B. (2020). Reinforcement learning via fenchel-rockafellar duality. arXiv preprint arXiv:2001.01866.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper got a quite high disagreement in the scores from the reviewers. R2 voted for rejecting the paper as he did not see the connection of the algorithm to the continuation method and also that the continuation method does not address the distributional shift, which is one of the main problems for offlline RL. Yet, these concerns have been properly answered in the rebuttal of the authors and the distributional shift is also addressed by the continuation method by reducing the error in policy evaluation. Further concerns from the reviewers were raised in terms of related work to a similar algorithm (BRAC), which is also addressed in the revision of the paper. 

The reviewers also identified the following strong points of the paper:
- The algorithm is a simple and very effective adaptation to SAC 
- The presented results are exhaustive and convincing
- The paper provides strong theoretical results for the presented algorithm
- The authors did a very good job with their revision, adding more comparisons and ablation studies.

I agree that this paper very interesting and recommend acceptance.
Decision: Accept (Poster)