BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
Authors: Xinyue Chen
Zijian Zhou
Zheng Wang
Che Wang
Yanqiu Wu
Qing Deng
Keith Ross
Venue:  (2020)
Keywords: Deep Reinforcement Learning
Batch Reinforcement Learning
Sample Efficiency

Abstract: The field of Deep Reinforcement Learning (DRL) has recently seen a surge in research in batch reinforcement learning, which aims for sample-efficient learning from a given data set without additional interactions with the environment. In the batch DRL setting, commonly employed off-policy DRL algorithms can perform poorly and sometimes even fail to learn altogether.  In this paper we propose anew algorithm, Best-Action Imitation Learning (BAIL), which unlike many off-policy DRL algorithms does not involve maximizing Q functions over the action space. Striving for simplicity as well as performance, BAIL first selects from the batch the actions it believes to be high-performing actions for their corresponding states; it then uses those state-action pairs to train a policy network using imitation learning.  Although BAIL is simple, we demonstrate that BAIL achieves state of the art performance on the Mujoco benchmark, typically outperforming BatchConstrained deep Q-Learning (BCQ) by a wide margin.

Under review as a conference paper at ICLR 2020
BAIL: BEST-ACTION IMITATION LEARNING FOR BATCH DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The field of Deep Reinforcement Learning (DRL) has recently seen surge in research in batch reinforcement learning, which aims for sample-efficiet learning from a given data set without additional interactions with the environment. In the batch DRL setting, commonly employed off-policy DRL algorithms can perform poorly and sometimes even fail to learn altogether. In this paper we propose a new algorithm, Best-Action Imitation Learning (BAIL., which unlike many off- policy DRL algorithms does not involve maximizing Q functions over the action space. Striving for simplicity as well as performance, BAIL first selects from the batch the actions believes be high-performing actions for their corresponding states; it then uses those state-action pairs to train policy network using imitation learning. Although BAIL is simple, we demonstrate that BAIL achieves state of the art performance on the Mujoco benchmark.
INTRODUCTION
The field of Deep Reinforcement Learning (DRL) has recently seen a surge in research in batch rein- forcement learning, which is the problem of sample-efficient learning from given data set without additional interactions with the environment. Batch reinforcement learning is appealing because dis-entangles policy optimization (exploitation) from data collection (exploration). This enables reusing the data collected by a policy to possibly improve the policy without further interactions with the environment. Furthermore, batch learning reinforcement learning algorithm can potentially be deployed part of growing-batch algorithm. where the batch algorithm seeks nigh-performing exploitation policy using the data in an experience replay buffer, combines this policy with explo- ration to add fresh data to the buffer, and then repeats the whole process (Lange et al. 2012).
Fujimoto et al. (2018a) recently made the critical observation that commonly employed off-policy algorithms based on Deep Q-Learning (DQL) often perform poorly and sometimes even fail to learn altogether. Indeed, off-policy DRL algorithms typically involve maximizing an approximate Q- function over the action space (Lillicrap al. 2015 Fujimoto et al., 2018b) [Haarnoja et al., [20 8a), leading to an extrapolation error, particularly for state-action pairs that are not in the batch distribu- tion.
deep Q-learning (BCQ). which obtains good performance for many of the Mujoco environments (Todorov et al., 2012), avoids the extrapolation error problem by constraining the set of actions over which the approximate Q-function is optimized (Fujimoto et al. 2018a).
We propose new algorithm. Best-Action Imitation Learning (BAIL), which strives for both sim- plicity and performance. BAIL does not suffer from the extrapolation error problem since does not maximize over the action space in any step of the algorithm. BAII. is simple, thereby satisfying the principle of Occam's razor.
The BAIL algorithm has two steps. In the first step, it selects from the batch subset of state-action pairs for which the actions are believed to be good actions for their corresponding states. In the second step, it simply trains policy network with imitation learning using the selected actions from the first step. To find the best actions, we train a neural network to obtain the "upper envelope" of the Monte Carlo returns in the batch data, and then select from the batch the state-action pairs that are near the upper envelope. We believe the concept of the upper-envelope of a data set is also novel and interesting in its own right.

---

Under review as a conference paper at ICLR 2020
Because the BCQ code is publicly available, are able make careful comparison of the per- formance of BAIL and BCQ. We do this for batches generated by training DDPG (Liliccapp et al.) 2013) for the Half-Cheetah, Walker, and Hopper environments, and for batches generated by train- ing Soft Actor Critic (SAC) for the Ant environment (Haarnoja et al., 2018a Although BAII. is simple, demonstrate that BAIL. achieves state of the art performance the Mujoco benchmark, often outperforming Batch Constrained deep Q-Learning (BCQ) by a wide-margin. We also provide anonymized code for reproducibility
2 RELATED WORK
Batch reinforcement learning in both the tabular and functional approximator settings has long been studied (Lange et al. [2012; Strehl ct al.] 2010) and continues to be highly active area of research
(Swaminathan & Joachims 2015] & Li
2015] Thomas & Brunskill 2016 Farajtabar et al. Imitation learning is also well-studied problem
2018 IIpaan et al. [2019] Jaques al.] 2019).
(Schaal 1999| Argall et al. 2009 Hussein
Hussenn et al.
2017) and also continues to be a highly active area
of research (Kim et al. [013 Piot al.) 2014
Chemali & Lazaric 2015) Hester et al. 2018 Ho
ct al. 2016 Sun et al. 2017
2018 Cheng et al.
2018: Gao et al. 2018))
This paper relates most closely (Fujimoto al. 2018a); which made the critical observation that when conventional DQL-based algorithms are employed for batch reinforcement learning, per- formance can be very poor, with the algorithm possibly not learning at all. Off-policy DRL al- gorithms involve maximizing an approximate action-value functioo Q(s, a) over all actions in the action space. (Or over the actions in the manifold of the parameterized policy.) The approximate action-value function can be very inaccurate, particularly for state-action pairs that are not in the state-action distribution of the batch (Fujimoto et al., 2018a). Due to this extrapolation error, poor- performing actions can be chosen when optimizing Q(s, a) over all actions. With traditional off- policy DRL algorithms (such as DDPG (Lillicrap et al.) 2015). TD3 (Fujimoto et al. 2018b) and SAC (Haarnoja et al.) 2018a)). if the action-value function over-estimates state-action pair, the policy will subsequently collect new data in the over-estimated region, and the estimate will get corrected. In the batch setting, however, where there is further interaction with the environment, the extrapolation error is not corrected, and the poor choice of action persists in the policy (Fujimoto al. 2018a).
Bacch-oonttraieed deep Q-learning (BCQ) avoids the extrapolation error problem by constraining the set of actions over which the approximate Q-function optimized (Fujimoto et al., 201 8a). More specifically, BCQ first trains state-dependent Variational Auto Encoder (VAE) using the state action pairs in the batch data. When optimizing the approximate Q-function over actions, instead of optimizing over all actions, it optimizes over subset of actions generated by the VAE. The BCQ algorithm is further complicated by introducing perturbation model, which employs an additional neural network that outputs an adjustment to an action. BCQ additionally employs a modified version of clipped-Double Q-Learning to obtain satisfactory performance. We show experimentally that our much simpler BAIL algorithm typically performs better than BCQ by wide margin.
Kumar al. (009 recently proposed BEAR for batch DRL. BEAR is also complex, employing
Maximum Mean Discrepancy (Gretton et al. 2012), kernel selection, a parametric model that fits a Lanh-Gaussian distribution, and test policy that is different from the learned actor policy. In this paper we do not experimentally compare BAIL with BEAR since the code for BEAR is not publicly available at the time of writing.
Agarwal et al. (2019) recently proposed another algorithm for batch DRL called Random Ensemble Mixture (REM), ensembling scheme which enforces optimal Bellman consistency on random convex combinations of the Q-heads of a multi-headed Q-network. For the Atari 2600 games, batch REM can out-perform the policies used to collect the data. REM and BAIL are orthogonal, and may be possible to combine them in the future to achieve even higher performance. No experimental results are provided for REM applied the Mujoco benchmark (Agarwal et al., 2019).
https://ancnymous. 4open sci wecieenceeeeeee -awt5-ffbee7733222-4553-33 a32d-
2

---

Under review as a conference paper at ICLR 2020
3 BATCH DEEP REINFORCEMENT LEARNING
We represent the environment with a Markov Decision Process (MDP) defined by tuple (S, A, 9, P, where S is the state space, A is the action space, is the initial state distribu- tion, and y the discount factor. The functions g(s, a) and r(s, a) represent the dynamics and reward function, respectively. In this paper we assume that the dynamics of the environment are deterministic, that is, there are real-valued functions g(s, a) and r(s, a) such that when in state and action a is chosen, then the next state is s' = g(8, a) and the reward received is r(s, a). We note that all the simulated robotic locomotion environments in the Mujoco benchmark are deterministic, and many robotic tasks are expected to be deterministic environments. Furthermore, many of the Atari game environments are deterministic (Bellemare et al., 2013). Thus, from an applications perspec- tive, the class of deterministic environments is large and important class. Although we assume that the environment is deterministic, as is typically the case with reinforcement learning, we do not assume the functions g(s, a) and r(s, a) are known.
In batch reinforcement learning, we are provided a batch of m data points B = {(8), 0i Fi, 8(), i = 1,..,m), We assume B is fixed and given, and there is further interaction with the environment. Often the batch B is training data, generated in some episodic fashion. However, in the batch reinforcement learning problem, we do not have knowledge of the algorithm, models, or seeds that were used to generate the episodes in the batch B.
Typically the batch data is generated during training with a non-stationary DRL policy. After train- ing, the original DRL algorithm produces a final-DRL policy, with exploration turned off. In our numerical experiments, we will compare the performance of policies obtained by batch algorithms with the performance of the final-DRL. policy. Ideally, we would like the performance of the batch- derived policy to be as good or better than the final-DRL. policy.
The case where batch data is generated from a non non-stationary training policy is of particular interest because it is typically a rich data set from which it may be possible to derive high-perforning policies. Furthermore, a batch learning algorithm can potentially be deployed as part of a growing- batch algorithm, where the batch algorithm seeks a high- high--erhonmee performing exploitation policy using the current data in experience replay buffer, combines this policy with exploration to add fresh data to the buffer, and then repeats the whole process (Lange et al. 2012).
4 BEST-ACTION IMITATION LEARNING (BAIL)
In this paper we present BAIL., an algorithm that not only performs well on simulated robotic loco- motion tasks, but is also conceptually and algorithmically simple. BAIL has two steps. In the first step, it selects from the batch data B the state-action pairs for which the actions are believed to be good actions for their corresponding states. In the second step, we simply train a policy network with imitation learning using the selected actions from the first step.
Many approaches could be employed to select the best actions. In this paper we propose training a single neural network to create an upper envelope of the Monte Carlo returns, and then selecting the state-action pairs in the batch B that have returns near the upper envelope.
4.1 UPPER ENVELOPE
We first define a X-smooth upper envelope, and then provide an algorithm for finding it. To the best of our knowledge, the notion of the upper envelope of data set is novel.
Recall that we have a batch of data B = {(s,, a4, rs, s(), = L...., m}. Although we do not assume we know what algorithm was used to generate the batch, make the natural assumption that the data in the batch was generated in an episodic fashion, and that the data in the batch is ordered accordingly. For each data point i E {1...., m), we caleulate an approximate Monte Carlo return G as the sum of the discounted returns from state 8; to the end of the episode:
T.
G =
rt
(1)
1=
3

---

Under review as a conference paper at ICLR 2020
where T; denotes the time which the episode ends for the data point 8,. The Mujoco environments environments (Suttonn & Barto 2018). During training. however, researchers typically create artificial episodes of length 1000 time steps; after 1000 time steps, a random initial state is chosen and a new episode begins. Because the Mujoco environments continuing tasks, it is desirable to approximate the return the infinite horizon, particularly for i values that are close to the (artificial) end of an episode. To do this, we note that the data-generation policy from one episode the next typically changes slowly. We therefore apply simple augmentation heuristic of concatenating the subsequent episode to the current episode, and running the sum in II to infinity. (In practice, we end the sum when the discounting reduces the contribution of the rewards to negligible amount.) Our ablation study in the Appendix shows that this simple heuristic can significantly improve performance. Note this approach also obviates the need for knowing when new episodes begin in the data set B.
are naturally anfinit-hhorizon non-episodic continuing-tan
Having defined the return for each data point in the batch, we now seek an upper-envelope V(s) for the data G := {ss, G.), = I..., mj. Let (s)s) be neural network with parameters & that takes as input a state s and outputs a real number. We say that Va- (s) is a X-smooth upper envelope for G if it has the following properties:
1. (s:) 2 G, for all i I,..., m.
2. Among all the parameterized functions Vo(s) satisfying condition 1, it minimizes:
L(d) Σ (v(s.) G. + \||0||
where À is non-negative constant. An upper-envelope thus smooth function that lies above all the data points, but is nevertheless close to the data points. Theorem 4.1. Suppose that Vg- (s) is X-smooth upper envelope for G. Then,
(1) Va-(s) max : .... m) as X co.
(2) If there is sufficient capacity in the network and X = O, then the Va- interpolates the data in G. For example, if x = and (8) is a neural network with ReLU activation functions with least 2m + weights and two layers, where is the dimension of the state space S, then Va. (s:) G₁ for all i 2....... m.
From the above theorem, we see that when À is very small, the upper envelope aims to interpolate the data, and when X is large, the upper envelope approaches a constant going through the highest data point. Just in classical regression, there is a sweet-spot for X, the one that provides the best generalization.
We note that there are other natural approaches for defining upper-envelope, some based on alter- native loss functions, others based on data clustering without making use function approximators. Also, it may be possible to combine episodes generate improved upper envelopes. These are all questions for future research.
To obtain approximate upper envelope of the data G, we employ classic regression with modified loss function, namely,
L(Ø)= = Gi) + K1 (Vals))a.)) + +|
-
2=1
where K >> 1 and (()) is the indicator function.
For a finite K value, the above loss function will only produce an approximate upper envelope, since it is possible V(s;) may be slightly less than G; for few data points. In practice, find K = 10, 000 works well for all environments tested. When K + oc, the approximation becomes exact, as stated in the following:
Theorem 4.2. Let o² be an optimal solution that minimizes I(Φ). Then, when K oo, Vo-(s) will be exact X-smooth upper envelope.
Also, instead of L2 regularization. in practice we employ the simpler stoppigg regularization, thereby obviating a search for the parameter X. We also clip the upper envelope at values near max; G,, described in the appendix, which can potentially provide further gains in performance.
4

---

Under review as a conference paper at ICLR 2020
4.2 SELECTING THE BEST ACTIONS
The BAIL algorithm employs the upper envelope to select the best (s, a) pairs from the batch data B. It then uses ordinary imitation learning (behavioral cloning) to train policy network using the selected actions. Let (s) denote the upper envelope obtained from minimizing the loss function 3 for fixed value of K.
We consider two approaches for selecting the best actions. In the first approach, which we call BAIL-border, we choose all (8,, a,) pairs from the batch data set such that
G: <V())
We set r such that p% of the data points are selected, where is 25 for all environments and batches. Thus BAIL-border chooses state-action pairs whose returns are near the upper envelope. The pseudo-code for the Bail-border algorithm is given in the appendix.
In this paper we
In the second approach, which we refer to as BAIL-TD, we select pair (s;, 0z) if
V(4) (s:)
where x is a hyper-parameter close to 1. Thus BAIL-TD chooses state-action pairs for which backed- up estimated return r; + +V((ss) is close the upper envelope value V(s;).
In summary, BAIL employs two neural networks. The first neural network is used to approximate value function based on the data in the batch B. The second neural network is the policy network, which is trained with imitation learning. This simple approach does not suffer from extrapolation error since it does not perform any optimization over the action space. algorithmic description of BAIL is given in Algorithm
Algorithm BAIL
1: Initialize upper envelope parameters é, policy parameter 8, obtain batch data B.
2: Compute return for each data point è: G, >i -
3: Obtain upper envelope by minimizing the loss:
4: for = 1, do
5: sample minibatch of data from the batch B
6: minimize over Φo: [(Ø) = (V (1 + K1 K1 (Va()))<))) + A||
7: Select data i for G; vVa select so that p% data are selected, let be set of selected
data
8: for l= do
9: sample minibatch U of data from U
minimize over 0: I.(0) (To())((s az)²
5 EXPERIMENTAL RESULTS
We carried out experiments with four of the most challenging environments in the Mujoco bench- mark (Todorov al., 2012) of OpenAl Gym. For the environments Hopper-v2, Walker-v2 and HalfCheetah-v2, we used the "Final Buffer" batch exactly as described in Fujimmto et al. (2018a) to allow for a fair comparison with BCQ. Specifically, we trained DDPG for one million time steps with 0.5 to generate batch. For the environment Ant-v2, we trained adaptive SAC (Haarnoja ct al. 2018b) again for one million time steps to generate a batch.
found that different batches generated with different seeds but with the same algorithm in the same environment can lead to surprisingly different results for batch DRL algorithms. To address this, for each of the environments generated four batches, giving total of 16 data sets.
In experiments,
Figure provides visualizations of 4 of the 16 upper envelopes, for each of the environments. In each visualization, the data points in the corresponding batch are ordered according to their upper- envelope values V(s.). With this new ordering, the figure plots (si, G:) for each of the one million
5

---

Under review as a conference paper at ICLR 2020
data points. The monotonically increasing blue line is the the upper envelope obtained by minimiz- ing (s)). Note that in all of the figures, a small fraction the data points are above upper envelopes due to the finite value of K 10, 000. But also note that the upper envelope mostly hugs the data. The constant black line is the clipping value. The final upper envelope is the minimum of the blue and black lines. All 16 upper envelopes are shown in the appendix.
Visi Vis1 Comparision
V(S) Compariiion
Ves) Comparision
800
VISI Com Comparision
200
432
302
330
Cheing Deiiey
11.999
Di SOWIA
100000 Jeste te
state
state
state
stabe
(a) H2pper-v2
(b) W
(c) HalfCheetah--2
(d) Ant-v2
Figure 1: Five illustrative upper envelopes trained from data with adaptive clipping
Figure compares the performance BAIL, BCQ, Behavioral Cloning (BC), and the final DDPG/SAC policy for the four environments. When training with BCQ, used the code pro- vided by the authors (Fujimoto et al. 2018a). Because at the time of writing the code for BEAR was not available, do compare presented in this section are for BAlL-border, which we simply refer to as BATL. In the appendix we provide results for BAIL-TD. The x-axis is the number of parameter updates and the y-axis the test return averaged over 10 episodes. BAIL, BCQ, and BC are each trained with five seeds. The figure shows the mean and standard deviation confidence intervals for these values. The figure also shows test result of the Maal-DDPG/SAC policy. This value is obtained by averaging test results from the last 100,000 timesteps (of one million time steps). During this period, test performance of SAC and particularly DDPG can greatly fluctuate with relatively small improvement on average. We calculate the mean and standard deviation of the test results over this period, plot the mean as a straight line, and use the transparent green background to show the confidence intervals. This enables us fairly compare the performance of the final test policy obtained with the behavioral algorithm with the test policies from the batch algorithms.
results with BEAR (Kumar et al., 2019). Also, all the results
We make the following observations. For Hopper. Walker and Ant, BAIL always beats BCQ usually by wide margin. For HalfCheetah, BAIL does better than BCQ for half of the batches. In almost all of the curves, BAIL has a much lower confidence interval than BCQ. Perhaps more importantly, BAIL's performance is stable over training, whereas BCQ vary dramatically. (This is serious issue for batch reinforcement learning, since cannot interact with the environment to find the best stopping point.) Importantly, BAIL also performs as well better than the -DDG/SDD policy in all but of the 16 batches. This gives promise that BAIL, or future variation of BAIL, could also be employed within growing-bard batch algorithm.
Table 1: Performance comparison at one million samples (mean and std over batches and random seeds). Last column shows percentage improvement of BAIL over BCQ.
Environment
Final-DDPG/SAC
BCQ
BAIL
Improvement
Hopper-v2
2547.7 750.4
1468.9 552.6
2437.5 + 489.7
65.9%
Walker2d-v2
1742.1 +:2 656.3
2020.2 699.3
2496.7 + 409.9
23.6%
HalfCheetah-v2
2612.4 + 342.2
2449.7 + 267.7
2660.0 + 77.7
8.6%
Ant-v2
4506.0 + 483.6
4315.6 + 416.4
4630.7 + 310.9
7.3%
We also summarize the results in Table For this table, average the performance of each algorithm over four batches, using the performance values at one million updates. Table T shows that BAIL's performance is better than that of BCQ for all four environments, with 66% and 23% average improvement for Hopper and Walker, respectively. BAIL also beats the Final-DDPG/SAC policies in three of the environments, and has significantly lower variance.

---

Under review as a conference paper at ICLR 2020
3300
2500
1500
2000
110
400
1000
05
0.2
04
06
08
0.2
O8
05
0.2
0.3
36
08
044
(a) Hopper-v2 1st
(b) Hopper-v2 2nd
(c) Hopper-v2 3rd
(d) Hopper-v2 v2 4th
3000
3000
2300
2000
1000
1000
1030
1000
500
Jacamedder spdtries
aatamaaa apoinas
pacaeaare apdiras
(e) Walker2d-v2 1st
(1) Walker2d-v2 2nd
(g) Walker2d-v2 3rd
(h) Walker2d-v2 4th
1400
0200
000
1000
4000
05
044
05
13
08
0.2
0.4
1.5
08
05
0.2
36
08
04
08
19
(i) HalfCheetah--2 1st
(i) HalfCheetah- 2nd
(k) HalfCheetahh 3rd
(1) HallCheetah -v2 4th
2300
800
4000
380
4000
2000
2000
00
3,2
2.4
5.
68
06
02
02
044
0.5
0.0
0.4
(m) Ant-v2 1st
(n) Ant-v2 2nd
(0) Ant-v2 3st
(p) Ant-v2 4th
Figure 2: Performance comparison of BAIL, BCQ, BC and FinAS-DDPGSSACC
In the appendix we also provide experimental results for DDPG batches generated with 0.1. which is similar to the "Concurrent" dataset in Fujimoto et al. (2018)). For this low noise level 0.1, BAIL continues to beat BCQ by a wide margin for Hopper and Walker, and continues to beat Final-DDPG for half of the batches. However, in the low noise case for HalfCheetah, BCQ beats BAIL for 3 of the batches.
5.1 ABLATION STUDY
BAIL uses an upper envelope to select the "best" data points for training a policy network with imitation learning. We have shown that BAIL typcially beats ordinary behavioral cloning and BCQ by a wide margin, and often performs better than the Final-DDPG and Final-SAC policies. But it is natural to ask how BAIL performs when using more naive approaches for selecting the best actions. We consider two naive approaches. The first approach, "Highest Returns," is to select from the batch the 25% of data points that have the highest G, values. The second approach, "Recent Data." " is to select the last 25% data points from the batch. Figure (3) shows the results for all four environments. We see that for each environment, the upper envelope approach the winner for most of the batches: for Hopper, the upper envelope wins for all four batches by wide margin; for Walker the upper-envelope approach wins by a wide margin for two batches, and ties Highest Returns for two batches; for HalfCheetah, the upper-envelope approach wins for three batches and ties Highest Returns for one batch; and for Ant, the upper-ence upper-envelope approach wins for three batches and ties the other two approaches for the other batch. We conclude, for the environments and batch generation mechanisms considered in this paper, the upper envelope approach performs significantly better and is more robust than both naive approaches.
7

---

Under review as a conference paper at ICLR 2020
3000
2500
Rove Date
1300
2000
1000
05
04
06
08
0.2
08
05
0.2
0.3
36
08
044
5.6
(a) Hopper-v2 1st
(b) Hopper-v2 v2 2nd
(c) Hopper-v2 3rd
(d) Hopper-v2 4th
2980
2000
2.400
3mo
2.00
1,000
2000
2.30
Jacamederr spdtries
pacareeee spdiras
(e) Walker-v2 1st
(1) Walker-v2 v2 2nd
(g) Walker-v2 v2 3rd
(h) Walker-v2 4th
7100
1000
120
900
300
C730
1750
0.
0.2
044
05
13
08
0.2
1.5
08
05
0.2
03
26
0.
-44
5.5
08
(i) HallCheetah-v2 1st
(j) HalfCheetahh-e 2nd
(k) HallCheetahh-v/ v2. 3rd
(l) HallCheetah 4th
$130
4200
4300
su:0
4200
ext0
.300
4150
4150
4000
4100
Jacameder spdras
peramaar apOrns
(m) Ant-v2 1st
(n) Ant-v2 2nd
(0) Ant-v2 3rd
(p) Ant-v2 4th
Figure 3: Comparison of BAIL scheme with Highest Returns and with Recent Samples schemes. All schemes use 25% of the data in the batch.
In the Appendix we provide additional ablation studies. Our experimental results show that modify- ing the returns to approximate infinite horizon returns is often useful for BAIL's performance, and that clipping the upper envelope also provides gains although much more modest.
In summary, our experimental results show that BAIL achieves state-of-thh-ar performance, and often beats BCQ by wide margin. Moreover, BAIL's performance stable over training. whereas BCQ typically varies dramatically over training. Finally, BAIL achieves this superior performance with an algorithm that is much simpler than BCQ.
6 CONCLUSION
Although BAIL as described in this paper simple and gives state-of-the-art performance, there are several directions that could be explored in the future for extending BAIL. One avenue is generating multiple upper envelopes from the same batch, and then ensembling or using a heuristic to pick the upper envelope which we believe would give the best performance. A second avenue is to optimize the policy by modifying the best actions. third avenue is to assign weights to the state-action pairs when training with imitation learning. And fourth avenue is to explore designing growing batch algorithm which uses BAIL as a subroutine for finding a high- gegh-perforing exploitation policy for each batch iteration.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Josh Achiam. Openai spinning up documentation.
.omm
en/latest/ http:/iindx...mmmmm Accessed: 2018-12-20.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. Striving for simplicity in off off-policy deep reinforcement learning. arXiv preprint 1XXiv:99770045000 2019.
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning, A survey of robot learning from demonstration. Roboties and autonomous systems, 57(5):469-483, 2009.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Jessica Chemali and Alessandro Lazaric. Direct policy iteration with demonstrations. In Twenty- Fourth International Joint Conference on Artificial Intelligence, 2015.
Ching-An Cheng, Xinyan Yan, Nolan Wagener, and Byron Boots. Fast policy learning through imitation and reinforcement. arXiv preprint arXiv: 10413, 2018.
Mchrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. arXiv preprint arXiv: /80200349, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv: 2018a.
Scott Fujimoto, Herke van Hoof, and Dave Meger. Addressing function approximation emro in actor-critic methods. arXiv preprint arXiv:1802.09477 20100.
Yang Gao, Ji Lin, Fisher Yu, Sergey Levine, Trevor Darrell, et al. Reinforcement learning from imperfect demonstrations. arXiv preprint 1rXiv:1802.005 2018.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. Journal of Muchine Learning Research, 13(Mar):723-773, 2012.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint 1rXivv/801.012000 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishck Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv: htt.:///2.00000 2018b.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et Deep q-learning from demonstrations, In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Jonathan Ho, Jayesh Gupta, and Stefano Ermon. Model-free imitation learning with policy opti-
mization. In International Conference on Machine Learning, pp. 2760-2769, 2016.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):21, 2017.
Alex Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine. Off-policy evaluation via off-policy classification. arXiv preprint arXivv/9066.01 2019.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint
2019.
Nan Jiang and Lihong Li. Doubly robust off- off-policy value evaluation for reinforcement learning. arXiv preprint /5/1:5//11037 2015.

---

Under review as a conference paper at ICLR 2020
Beomjoon Kim, Amir- massoud Farahmand, Joelle Pineau, and Doina Precup. Learning from limited demonstrations. In Advances in Neural Information Processing Systems, pp. 2859-2888888 22699 2013.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 6980, 2014.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv: /906, 00949, 2019.
Sascha Lange. Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0299 52977,9 2015.
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted bellman residual minimization handling expert demonstrations. Joint European Conference on Muchine Learning and Knowledge Dis- covery in Databases, pp. 549-564. Springer, 2014.
Stefan Schaal. Is imitation learning the route humanoid robots? Trends in cognitive sciences, (6):233- 242, 1999.
Alex Strehl, John Langford, Lihong Li, and Sham M Kakade. Learning from logged implicit explo- ration data. Advances in Neural Information Processing Systems, pp. 2217-2225, 2010.
Wen Sun, Arun Venkatraman, Geoffrey Gordon, Byron Boots, and J Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 3309-3318. JMLR. org. 2017.
Wen Sun, Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining rein- forcement learning & imitation learning. arXiv preprint arXiv:/805. /240, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. Journal of Machine Learning Research, 16(1):1331-1755,
2015.
Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference Machine Learning, pp.
2016.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, 5026- 5033. IEEE, 2012.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017.
10

---

Under review as a conference paper at ICLR 2020
A PROOFS
A. Proof of Theorem 4.1
Proof. First, let consider the when À - +00. We can re-write the definition of the upper envelope as a constrained optimization problem:
min Vo(s))) [.,] A|
(6)
8.1. G; - Vo(s)) S 0,
= 1,2, 1,2....
where Va- is the optimal solution to the above optimization problem. We write the Lagrangian function:
1t [iv((s) GiJo ||
Σ 14 Va(s:)
As the optimal solution, Va- must satisfy the KKT conditions specified below:
dL(o.u) b= 0
M.G. Vo- (s:) 0. i = ................................ Hi
(8)
220 0,
i 1,2 772
Suppose Φ (0i,..., Pa), ), by the first KKT condition, we have
a
0Φ, Vo(s:) - Gil = Go]:1 + 220 +
Mi - Va(s))lo=))) =
j=1,2.....
Oo;
So we have:
1 0 m
1 2 m
0f 2A dΦj
-
b,
j= 1,2......
2x
When À + 0c, we have o² O. In this case, it follows that Vg. C. for some constant C. As Va- (s,) 2 G., in order to minimize (3) we must have = max (G,, i= 1,2....... m).
For the case of À 0, notice that only have finitely many input s; to be the input of the neural network. Therefore, this is a typical problem regarding the finite-sample expressivity of the neural networks, and the proof directly follows from the work done by Zhang al. (Zhang et al. 2017). D
B. Proof of Theorem 4.2
Proof. Let be the optimal value that minimizes L(Ø). Let's proceed by contradiction and assume that there exists some k such that
(a-(s)) G2 Let o' be an arbitrary given value such that Vor (s,) 2 G, for all i .1.,....,mm Then we have
m
Σ (Vo-(s.) 1c)) (.:)).)) K(V (sk) G2)² + 1 0 L(d')
I(#*) < T(00) = Σ (s) G.) + 1|00112
1=1
This implies that
m
m
(Vvo - (1)())))G)))MM))))))))) K(Vo- i=1
(xV Gx) 1110*112
Σ (Vr(s)) - -.)² + 110'112 -=1
which is impossible when + co. Therefore, we must have Va- (g 2 G, forall E {1, ...,, m} as K oo. In this way, when K + 6² actually minimizes
L( = 2 (V((s.) G.) ||
which completes the proof.
11

---

Under review as a conference paper at ICLR 2020
B
IMPLEMENTATION DETAILS AND HYPERPARAMETERS
B.1
IMPLEMENTATION OF BAIL ALGORITHM
Table 2: Upper Envelope Hyperparameters
Parameter
Value
optimizer learning rate discount (r)
Adam Kingma & Ba 2014
3
0.99
regularization constant À
2 2-10-2
K
10,000
number of hidden units
128 x 128
Table 3: BAIL Hyperparameters
Parameter
Value
data batch optimizer learning rate
106
Adam
Kingma & Ba 2014
10-3
regularization constant À
0
mini-batch size
100
BAIL-borderr p%
25%
BAIL-TD number of hidden units
0.96
400 300
B.2 IMPLEMENTATION OF COMPETING ALGORITHMS
For the behavioral DDPG algorithm, we used the implementation of Fujimoto et al. 2018b). For the behavioral SAC algorithm, implemented it in Pytorch, mainly following the pseudocode provided by (Achiam), and used hyperparameters in Haarnoja et al. (2018b). For the BCQ algorithm, used the authors" implementation (Fujimoto al. 2018a). For behavioral cloning and its variants in the ablation study section, the network structure, learning rate, mini-batch size, and so on are identical those in Table for BAIL.
For the upper envelope network, our network has two hidden layers as does the Q network in BCQ and SAC. However, the number of hidden units in our network is less than those used in BCQ (400* x 300) and SAC (256 x 256). In future work we will see if we can obtain further improvements with BAIL using larger networks for the upper envelope.
B.3 CLIPPING HEURISTICS
As is shown in Figure [ in practice the trained upper envelope does always fit well the data points on the right side of the plots, where the upper envelope can become very large. In that region, the data points with the highest returns will not be selected "best actions" and therefore for imitation learning step. We observe that if we plot the upper envelope values for all the states in the buffer in ascending order as is shown Figure I the upper envelope value V(s;) starts to deviate from the Monte Carlo return G3 at point where V(s,) max(G;). We therefore use the following heuristic. We say that the upper envelope value begins to deviate from the Monte Carlo return at state if Vs,)) > Gj for i < < i 10000. We set the clipping value C' = V(s') where is the starting point of this deviation. Then the actual UE values used to select data is min (s), C). In practice, the clipping heuristic gives small boost in performance is shown in Figure
used
12

---

Under review as a conference paper at ICLR 2020
C ADDITIONAL EXPERIMENTAL RESULTS
C.I EXPERIMENTS ON BAIL-TD
We present some of the experimental results for BAIL-TD in this subsection. Figure shows that the performance of BAIL-TD is similar to the performance of naive Behavioral Cloning in Hopper- v2 and Walker2d-v2 environments. The result implies that the BAIL-TD approach limited in the ability to distinguish between good data points and the bad ones. This is possibly due to the inaceuracy the trained upper envelope. Recall that in BAIL-TD algorithm, we select state action pairs based on the difference between the values of + (s{) and (ss)). Since only one-step Monte Carlo return, and the value of r is very small compared with the value of u u(s;), the selection very sensitive to the accuracy of V (s;).
3000
BC0
300
1000
Relf SCPM
3300
BAL-TD
190
300 BASIOO
00.00 K00
(a) Hopper-v2 1st
(h) Walker2d-v2 1st
(c) Hopper-v2 2nd
(d) Walker2d-v2 2nd
Figure 4: Performance comparison of BAIL-border, BAIL-TD, BCQ, BC and Behavioral Policy (DDPG)
13

---

Under review as a conference paper at ICLR 2020
C.2 ABLATION STUDY FOR BAIL
We do additional ablation studies for BAIL, where we focus on two heuristic features in the BAIL algorithm: clipping of upper envelope, and return augmentation approximate non-episodic contin- uous tasks. We removed each of these features one at time from the BAIL algorithm and compare with the original BAIL algorithm. Each performance curve again averaged over 5 independent runs.
2009
2903
3-000
2000
2200
5400
2130
1300
2000
1000
02
0.1
(a) Hopper-v2 1st
(b) Hopper-v2 2nd
(c) Hopper-v2 3rd
(d) Hopper-v2 4th
2000
2300
3000
3:00
2000
2200
100
2000
aponts
(c) Walker2d-v2 1st
(f) Walker2d-v2 2nd
g Walker2d-v2 3rd
(h) Walker2d-v2 4th
1390
1200
5
1130
1.00
1300
8000
114
0.1
spdtraee
(i) HalfChectah-v2 1st
(j) HalfCheetah-Ve 2nd
(k) HalfCheetah-v2 3rd
(l) HalfCheetah-v2 4th
110
400
4700
4500
410
<200
4000
parmmttrr
(m) Ant-v2
(n) Ant-v2 2nd
(0) Ant-v2
(p) Ant-v2 4th
//
Figure 5: Ablation of BAIL without Upper Envelope clipping
We found that clipping often, but not always, gives a small boost in performance. As for return aug- mentation, found does not harm the performance of BAIL,, and sometimes gives considerable improvement, particularly for Hopper.
14

---

Under review as a conference paper at ICLR 2020
2.00
2000
1770
2130
2400
2130
2230
250
1300
2000
1000
(a) Hopper-v2 1st
(b) Hopper-v2 2nd
(c) Hopper-v2 3rd
(d) Hopper-v2 4th
2430
3000
2000
2 2/0
2400
2000
100
2200
2000
1.4
0.2
02
222
01
114
2.8
(e) Walker2d-v2 1st
(f) Walker2d-v2 2nd
Walker2d-v2 3rd
(h) Walker2d-v2 4th
1340
1320
4000
4200
4100
Jonatmeeer
(i) HalfCheetah-v2 1st
ii HalfChectah--2? 2nd
(k) HalfChectah--2
(I) HalfCheetah-r2 4th
4700
2v0n
4000
400
2400
144
62
08
C3
(m) Ant-v2 1st
(n) Ant-v2 2nd
(o) Ant-v2 3rd
(p) Ant-v2 4th
Figure 6: Ablation of BAIL without return augmentation
15

---

Under review as a conference paper at ICLR 2020
C.3 BAIL FOR Low NOISE-LEVEL DATA
In the main body of the paper we present BAIL in the "Final Buffer" case as described in Fujimoto al. (2018a), where the exploration noise In this section we examine the performance of BAIL in a low-noise scenario. To this end, we set 0.1 and do similar experiments as were done 0.5 for the Hopper, Walker and HalfCheetah environments. (Recall that for Ant we used adaptive SAC, which does not have an explicit noise parameter.)
0.5 added to behavior policy is relatively large.
The results are shown in Figure 7 We see that even in this low-noise scenario, BAIL out- out-performs BCQ by wide margin for Hopper and Walker, and BAIL continues to out-perform the Final-DDPG policy in most batches. For HalfCheetah, where the Final-DDPG policy gives greatly different results depending on the batch, BAIL's performance stable and typically near that of the of Final- DDPG policy. After sufficient training, however, BCQ can often do better than the Final-DDPG policy in this environment.
3000
3000
2300
3:00
1500
0.2
05
(a) Hopper-v2 1st
(b) Hopper-v2 v2 2nd
(c) Hopper-v2 3rd
(d) Hopper-v2 v2 4th
5500
2000
2000
1000
1399
1:30
000
600
1300
800
patamear apcinss
parmtseer Ladian
(e) Walker-v2 1st
(I) Walker-v2 v2 2nd
(g) Walker-v2 3rd
(h) Walker-v2 4th
7000
1300
500
4000
4001
50
0.2
1.5
05
04
(i) HallCheetah--22 1st
(i HalfCheetah--2
(k) HalfCheetah-
(1) HallCheetah- 4th
Figure 7: Performance comparison of BAIL, BCQ, BC and Final-DDPG with noise level = 0.1
16

---

Under review as a conference paper at ICLR 2020
C.4 VISUALIZATION OF UPPER ENVELOPES USED
In this section we present the upper envelopes used in the training of BAIL in Figure 2
V(a) Compantsion
200
Vie Comparision
Visl Comparision
1000
Visa Comparision
400
300
200
100
100
2003 wotte
yent yott Lo00000 state
state
state
state
(a) Hopper-v2 1st
(b) H2pper-v2 2nd
(c) Hopper-v2 3rd
(d) Hopper-v2 4th
Viai Compansion
800
Compantioon
Ulged
Via. Comparision
Visa Comparision
600
600
80
400
20
200
100
VisI
20830 222l Agstll
2003 Lysth byott
20000 200 Booth
state
state
state
state
(e) Walker2d-v2 1st
(f) Walker2d-v2 2nd
Walker2d-v2 3rd
(h) Walker2d-v2 4th
Via) Comparision
Vis) Comparision
200
Visi Comparision
Comparision
800
100
350
600
50
-100
yefff Watt gartl Loesbea
state
state
state
state
(i) HalfChectah-v2 1st
(j) HalfChectah-v2 2nd
(k) HalfChectah-v2
(l) HalfCheetah-v2 4th
Figure 8: Upper Envelopes of BAIL (DDPG data)
Comparision
Vis) Comparision
Compartsion on
Liger Iecage Rocres
V(G)
V(s)
V(G)
-200
Fallans
-200
200008 Seent Bent Roconts
Sart Tounnoo
(a) Ant-y2 1st
(b) Ant-v2 2nd
(c) Ant-v2 3rd
Ant-v2 4nd
Figure 9: Upper Envelopes of BAIL (SAC data)
17


Reviews:
[{"review_id": "SJlbdX86tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper tries to solve a batch reinforcement learning problem with a very simple but efficient algorithm. It first learns a smooth upper bound of Monte Carlo returns in the batch data (called the \"upper envelope\"). Then, the algorithm chooses state action pairs of the batch data that have returns larger than constant times the upper envelope. It lowers the constant until the algorithm gets 25% of the data. Then the algorithm trains the policy on chosen state-action pairs. The algorithm is shown to outperform BCQ in experiments.\n\nAlthough I like the idea of the paper, I vote for rejection. While there is no theoretical guarantee on the performance of the algorithm, the design of the algorithm does not follow the usual design the other researchers follow. The way of reporting the experiment results does not seem very professional. I recommend the authors to consult with some other researchers who have publication experience. In the current form, the paper is very poor in detail that makes readers hard to be convinced with the results.\n\nThese are some points that I could not understand:\n\n1. Why do you fix K=10000 on modified loss instead of dual gradient descent for constrained optimization?\n2. How do you guarantee that choosing (s,a) such that G>xV gives you good samples? Since mean returns are not zero, it won't pick the top 25% actions for all states. States with the high mean return will have all of its samples included, while states with the low mean return will have all of its samples excluded. Although the authors concatenated all the experiences to compute returns (which is ad-hoc as well), the initial states will have a lower return than other states. This means that most of the actions of the initial states will be excluded in the training set while more actions of the other states will be included, which does not seem desirable. (e.g. in Figure 1 Ant. If we set x=0 (extreme case), states of timestep >600000 will be all included where t<600000 will be partially excluded. )\n3. In the explanation of Figure 2, it is written as \"standard deviation confidence interval\". Is it standard deviation, or confidence interval? Also, why are the standard deviation in the Figure 2 and the Table 1 so different? How do you compute Improvement in the Table 1? What happens if the environment gives negative returns only (i.e. Pendulum), such that BCQ gives you -1000 and BAIL gives you -500?\n4. As claimed in theorems, V=max(G) if lambda->infinity. This means that the \"Highest Returns\" in figure 3 is also one specific hyperparameter choice of the suggested algorithm. There might be a better choice of regularization that outperforms both BAIL and Highest Returns as early-stopping done in the paper is just one random amount of regularization. What was the early-stopping criterion and how is it chosen? How do we know it is the best regularization option?\n5. Is the final DDPG or final SAC evaluated with a deterministic policy? According to the paper, I assume that it was not. Those algorithms usually add large noise while training for exploration, and such noise is removed while in evaluation. In Bear Q learning, better action selection technic is used, which chooses the action sample that maximizes critic Q. Is the evaluations really fair for all algorithms? As far as I know, Mujoco environments are deterministic except the initial state sampling, and there should only be very small variance.\n\nAlso, I believe the paper should be compared to Bear Q learning as well, as it is very easy to implement and outperforms BCQ by a large margin. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJelB0Pj5H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper studies the problem of learning a policy from a fixed dataset. The authors propose to estimate a smooth upper envelope of the episodic returns from the dataset as a state-value function. The policy is then learned by imitating the state action pairs from the dataset whose actual episodic return is close to the estimated envelope.\n\nRecommended decision:\nThe direction of imitating \"good\" actions from the dataset is interesting. The intuition of estimating an upper envelope of the value function seems reasonable. However, I feel like this paper is not ready to be published in terms of its overall quality, mainly due to the lack of correctness, rigorousness and justification in statements and approaches.\n\nMajor comments:\n\n- On the top of page 4:  \"Because the Mujoco environments are continuing tasks, it is desirable to approximate the return over the infinite horizon, particularly for i values that are close to the (artificial) end of an episode. To do this, we note that the data-generation policy from one episode to the next typically changes slowly. We therefore apply a simple augmentation heuristic of concatenating the subsequent episode to the current episode, and running the sum in (1) to infinity.\" I cannot see how this approach is validated. The reset of initial state makes cross-episode cumulative reward from a state s not an approximation to the real return from state s. Estimating the infinite horizon return from finite horizon data is indeed a challenge here and simply cut the return at the end of an episode is be problematic. But the solution proposed by the authors is wrong in principle and cannot be simply justified by \"good empirical performance\". I feel hard to regard this choice a valid part of an algorithm unless further justification can be provided.\n\n- Statements of theorems (4.1 and 4.2) are non-rigorous and contain irrelevant information: \"lambda-smooth\" is not an appropriate terminology when lambda is the weight of the regularizer. The actual \"smoothness\" also depends on the other term in the loss (same lambda does not indicate same smoothness in different objectives). For the same reason, Theorem 4.2 is wrong as changing K also changes the smoothness of the learned function. Proof of Theorem 4.2 in appendix is wrong as the authors ignore the coefficients in the last equation. Theorem 4.1-(1) cannot be true unless how V_\\phi is parameterized is given: e.g. if there is no bias term or the regularization is applies to the bias term V will always output 0 as lambda 0-> \\infty. The \"2m+d\" in Theorem 4.1-(2) is irrelevant to this work and cannot be justified without more detailed statements about how the network is parameterized. I appreciate the motivation that the authors try to validate the use of their objective to learn a \"smooth upper envelope\" but most of these statements are somewhat trivial and/or wrong section 4.1 does not actually deliver a valid justification.\n\n- The use of \"smooth upper envelope\" itself can bring both over-estimation and under-estimation. For example, if one can concatenate different parts from different episodes to get a trajectory with higher return, the episodic return for the states along this trajectory is an under-estimate. Although it is fine to use a conservative estimate it would be better to be explicit about this and explain why this may not be a concern. On the other hand, it can bring over estimation to the state-values due to the smoothness enhanced to the fitted V. It would be better to see e.g. when these concerns do not matter (theoretically) or they are not real concerns in practice (by further inspecting the experiments). \n\n- Regarding Experiments: Why Hopper, Walker, HalfCheetah are trained with DDPG while Ant is trained by SAC? The performance of Final-DDPG/SAC after training for 1m steps looks way below what SAC and TD3 can get. Is it because they are just partially trained or noise is added to them? The baseline online-trained policy should not contain noise for a fair comparison. That said, in batch RL setting it is not necessary to compare to online-trained policy because it is a different setting. But if the authors want to compare to those, choice of baseline should be careful. An important baseline which is missing is to run vanilla DDPG/TD3/SAC as a batch-mode algorithm.  \n\n\nMinor comments:\n\n- Section 3, first paragraph: It is not very meaningful to say \"simulators are deterministic so deterministic environments are important\". Simulators are made by humans so they can be either deterministic or stochastic. \"many robotic tasks are expected to be deterministic environments\" is probably not true. I do not view \"assuming deterministic envs\" as a major limitation but I do not find these statements convincing as well. Similarly, the argument for studying non-stationary policy seems unsupportive: if the dataset comes from training a policy online then why do we care about learning another offline policy rather than just use or continue training the online policy. One argument I can see is that the online policy is worse. But the fact that these policies are worst than running e.g. SAC for a million steps makes the motivation questionable. Again, I do not view \"choice of setting\" as a limitation but I just find these statements a bit unsupportive. \n\n\nPotential directions for improvement:\n\nTo me the main part of the paper that looks problematic is Section 4.1 (both the approximation of infinite horizon returns and the theorems). It would be better to see a more rigorous and coherent justification of this approach (or some improved version), e.g. by either presenting analysis that is rigorous, correct and actually relevant or leave the space for more detailed empirical justification (e.g. whether potential over/under-estimating happens or not, comparing the estimated V to real episodic return of the learned policy).\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1gZB2I2YB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of Claims:\n\nThe paper proposes a batch RL method that they claim is simpler than most existing methods that try to avoid the extrapolation error that is prevalent among batch RL methods. They do this by completely avoiding the minimization/maximization (cost/reward) of the approximate value function that is fit to the batch transitions. Instead, they train an approximation for the state value function's tight upper bound (which they refer to as the upper-envelope) by using their monte-carlo returns. By fitting such an approximator, they sample the state-action pairs that are close to the envelope (thus have high values/returns), and use behavioral cloning to fit a parameterized policy to those state-action pairs.\n\nDecision:\n\nWeak Reject.\nMy decision is influenced by two main reasons:\n\n(1) Although the simplicity of the method is apparent and a very desirable feature, the authors don't highlight situations where this can lead to bad policies. For example, consider that there are two pairs (s, a_1, s') and (s, a_2, s') in the batch that are close to the upper-envelope, and hence will both be used for training the policy. Using Behavioral cloning, the policy would regress to the mean of a_1 and a_2, which could be a terrible action altogether. The issue here is that only one of these two pairs has higher return and our policy needs to only predict that action (or in the case of tie, either one.) This can be really bad in situations where two very different actions can lead to same returns (e.g. in a reacher-like task the arm can reach a goal in two different rotations.) Even though I pointed out a very specific case, one could think of many other cases where the proposed approach might result in a bad policy. \n\nHaving said all of this, it might be true that such cases do not appear in practice (which I highly doubt) but its the authors job to raise and clarify that. The current set of experimental setups (mujoco locomotion problems) are not good enough evidence for that and they need experiments where optimal-policies can be multi-modal or have diverse experimental setups (manipulation etc.)\n\n(2) Experimental results are a little unsettling. The primary reason is that in all of the plots, BCQ, BAIL, BC aren't starting from the same test return at 0 parameter updates! In most plots BAIL starts off way higher in return than BCQ, BC with no parameter updates yet, which suggests that the experiments were not setup well. Maybe, they didn't initialize the policy in the same way for all the approaches, maybe the random seeds were not the same for all approaches, or maybe BAIL had some sort of pretraining for the policy that was not accounted for in the parameter updates. In any way, this needs to be addressed. This is also highlighted by the fact that the learning curves for BAIL are almost always flat across a million parameter updates! If you are starting off with a random initialization, there should be an upwards slope for the learning curve. Also, as raised in the previous point I think using these Mujoco locomotion environments is not convincing enough to claim that BAIL is a viable competitive batch RL approach.\n\nComments and Questions:\n\n(1) I like the simplicity of the approach and the fact that it is much more easier to understand than existing works like BCQ\n\n(2) Paper is well-written. It was clear, lucid and descriptive.\n\n(3) Why is the deterministic dynamics assumption needed? I am curious\n\n(4) The paper makes some subjective statements such as \"BEAR is also complex\", which is not substantiated well enough. Refrain from making such statements\n\n(5) Not comparing to BEAR because their code is not publicly available is a contentious reason. I personally feel that the authors could have reimplemented it and compared but I am not sure what the community feels about that\n\n(6) Is there any reason why REM cannot be applied to mujoco environments? If it can be, then why did the authors not compare to REM as well?\n\n(7) Another subjective statement (that is clearly wrong) \"many robotic tasks are expected to be deterministic environments\" - although this is slightly true, the reason we model environments to be stochastic is not because there is inherent randomness in them but because our state descriptions are never complete. The state descriptors are always partial and we account for them by assuming stochasticity in the dynamics. For example, consider a robotic manipulation task where if you know all the environmental factors as part of your state space(such as the friction coefficients) you can assume deterministic dynamics, else you are better off assuming stochastic dynamics because the same actuation might not result in the same motion every time (because of varying friction)\n\n(8) Concatenating subsequent episodes in a batch only makes sense (as the authors point out) if the policy doesn't change much across episodes. But this is not true of current off-policy RL methods like DDPG, SAC. You either need very small learning rate or a trust-region constraint to ensure that the policy doesn't change much across episodes. \n\n(9) Why do different batches with different seeds and the same algorithm lead to widely different results for batch RL? There is clearly something fishy here. Is it because of the off-policy RL methods used to collect the data, is it due to the batch RL method used? More investigation needed", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a novel algorithm for batch RL with offline data. The method is simple and outperforms a recently proposed algorithm, BCQ, on Mujoco benchmark tasks.

The main points that have not been addressed after the author rebuttal are:
* Lack of rigor and incorrectness of theoretical statements. Furthermore, there is little analysis of the method beyond the performance results.
* Non-standard assumptions/choices in the algorithm without justification (e.g., concatenating episodes).
* Numerous sloppy statements / assumptions that are not justified.
* No comparison to BEAR, making it challenging to evaluate their state-of-the-art claims.
The reviewers also point out several limitations of the proposed method. Adding a brief discussion of these limitations would strengthen the paper.

The method is interesting and simple, so I believe that the paper has the potential to be a strong submission if the authors incorporate the reviewers suggestions in a future submission. However, at this time, the paper falls below the acceptance bar.
Decision: Reject