Title: BAIL: BEST-ACTION IMITATION LEARNING FOR BATCH DEEP REINFORCEMENT LEARNING

Abstract: The field of Deep Reinforcement Learning (DRL) has recently seen a surge in research in batch reinforcement learning, which aims for sample-efficient learning from a given data set without additional interactions with the environment. In the batch DRL setting, commonly employed off-policy DRL algorithms can perform poorly and sometimes even fail to learn altogether. In this paper we propose a new algorithm, Best-Action Imitation Learning (BAIL), which unlike many offpolicy DRL algorithms does not involve maximizing Q functions over the action space. Striving for simplicity as well as performance, BAIL first selects from the batch the actions it believes to be high-performing actions for their corresponding states; it then uses those state-action pairs to train a policy network using imitation learning. Although BAIL is simple, we demonstrate that BAIL achieves state of the art performance on the Mujoco benchmark.

1. INTRODUCTION:

The field of Deep Reinforcement Learning (DRL) has recently seen a surge in research in batch reinforcement learning, which is the problem of sample-efficient learning from a given data set without additional interactions with the environment. Batch reinforcement learning is appealing because it dis-entangles policy optimization (exploitation) from data collection (exploration). This enables reusing the data collected by a policy to possibly improve the policy without further interactions with the environment. Furthermore, a batch learning reinforcement learning algorithm can potentially be deployed as part of a growing-batch algorithm, where the batch algorithm seeks a high-performing exploitation policy using the data in an experience replay buffer, combines this policy with exploration to add fresh data to the buffer, and then repeats the whole process (Lange et al., 2012) . Fujimoto et al. (2018a) recently made the critical observation that commonly employed off-policy algorithms based on Deep Q-Learning (DQL) often perform poorly and sometimes even fail to learn altogether. Indeed, off-policy DRL algorithms typically involve maximizing an approximate Qfunction over the action space (Lillicrap et al., 2015; Fujimoto et al., 2018b; Haarnoja et al., 2018a) , leading to an extrapolation error, particularly for state-action pairs that are not in the batch distribution. Batch-Constrained deep Q-learning (BCQ), which obtains good performance for many of the Mujoco environments (Todorov et al., 2012) , avoids the extrapolation error problem by constraining the set of actions over which the approximate Q-function is optimized (Fujimoto et al., 2018a) .

We propose a new algorithm, Best-Action Imitation Learning (BAIL), which strives for both simplicity and performance. BAIL does not suffer from the extrapolation error problem since it does not maximize over the action space in any step of the algorithm. BAIL is simple, thereby satisfying the principle of Occam's razor.

The BAIL algorithm has two steps. In the first step, it selects from the batch a subset of state-action pairs for which the actions are believed to be good actions for their corresponding states. In the second step, it simply trains a policy network with imitation learning using the selected actions from the first step. To find the best actions, we train a neural network to obtain the "upper envelope" of the Monte Carlo returns in the batch data, and then we select from the batch the state-action pairs that are near the upper envelope. We believe the concept of the upper-envelope of a data set is also novel and interesting in its own right.

Because the BCQ code is publicly available, we are able to make a careful comparison of the performance of BAIL and BCQ. We do this for batches generated by training DDPG (Lillicrap et al., 2015) for the Half-Cheetah, Walker, and Hopper environments, and for batches generated by training Soft Actor Critic (SAC) for the Ant environment (Haarnoja et al., 2018a; b) . Although BAIL is simple, we demonstrate that BAIL achieves state of the art performance on the Mujoco benchmark, often outperforming Batch Constrained deep Q-Learning (BCQ) by a wide-margin. We also provide anonymized code for reproducibility1 .

2. RELATED WORK:

Batch reinforcement learning in both the tabular and functional approximator settings has long been studied (Lange et al., 2012; Strehl et al., 2010) and continues to be a highly active area of research (Swaminathan & Joachims, 2015; Jiang & Li, 2015; Thomas & Brunskill, 2016; Farajtabar et al., 2018; Irpan et al., 2019; Jaques et al., 2019) . Imitation learning is also a well-studied problem (Schaal, 1999; Argall et al., 2009; Hussein et al., 2017) and also continues to be a highly active area of research (Kim et al., 2013; Piot et al., 2014; Chemali & Lazaric, 2015; Hester et al., 2018; Ho et al., 2016; Sun et al., 2017; 2018; Cheng et al., 2018; Gao et al., 2018) .

This paper relates most closely to (Fujimoto et al., 2018a) , which made the critical observation that when conventional DQL-based algorithms are employed for batch reinforcement learning, performance can be very poor, with the algorithm possibly not learning at all. Off-policy DRL algorithms involve maximizing an approximate action-value function Q(s, a) over all actions in the action space. (Or over the actions in the manifold of the parameterized policy.) The approximate action-value function can be very inaccurate, particularly for state-action pairs that are not in the state-action distribution of the batch (Fujimoto et al., 2018a) . Due to this extrapolation error, poorperforming actions can be chosen when optimizing Q(s, a) over all actions. With traditional offpolicy DRL algorithms (such as DDPG (Lillicrap et al., 2015) , TD3 (Fujimoto et al., 2018b) and SAC (Haarnoja et al., 2018a) ), if the action-value function over-estimates a state-action pair, the policy will subsequently collect new data in the over-estimated region, and the estimate will get corrected. In the batch setting, however, where there is no further interaction with the environment, the extrapolation error is not corrected, and the poor choice of action persists in the policy (Fujimoto et al., 2018a) .

Batch-Constrained deep Q-learning (BCQ) avoids the extrapolation error problem by constraining the set of actions over which the approximate Q-function is optimized (Fujimoto et al., 2018a) . More specifically, BCQ first trains a state-dependent Variational Auto Encoder (VAE) using the state action pairs in the batch data. When optimizing the approximate Q-function over actions, instead of optimizing over all actions, it optimizes over a subset of actions generated by the VAE. The BCQ algorithm is further complicated by introducing a perturbation model, which employs an additional neural network that outputs an adjustment to an action. BCQ additionally employs a modified version of clipped-Double Q-Learning to obtain satisfactory performance. We show experimentally that our much simpler BAIL algorithm typically performs better than BCQ by a wide margin. Kumar et al. (2019) recently proposed BEAR for batch DRL. BEAR is also complex, employing Maximum Mean Discrepancy (Gretton et al., 2012) , kernel selection, a parametric model that fits a tanh-Gaussian distribution, and a test policy that is different from the learned actor policy. In this paper we do not experimentally compare BAIL with BEAR since the code for BEAR is not publicly available at the time of writing. Agarwal et al. (2019) recently proposed another algorithm for batch DRL called Random Ensemble Mixture (REM), an ensembling scheme which enforces optimal Bellman consistency on random convex combinations of the Q-heads of a multi-headed Q-network. For the Atari 2600 games, batch REM can out-perform the policies used to collect the data. REM and BAIL are orthogonal, and it may be possible to combine them in the future to achieve even higher performance. No experimental results are provided for REM applied to the Mujoco benchmark (Agarwal et al., 2019) .

3. BATCH DEEP REINFORCEMENT LEARNING:

We represent the environment with a Markov Decision Process (MDP) defined by a tuple (S, A, g, r, ρ, γ), where S is the state space, A is the action space, ρ is the initial state distribution, and γ is the discount factor. The functions g(s, a) and r(s, a) represent the dynamics and reward function, respectively. In this paper we assume that the dynamics of the environment are deterministic, that is, there are real-valued functions g(s, a) and r(s, a) such that when in state s and action a is chosen, then the next state is s = g(s, a) and the reward received is r(s, a). We note that all the simulated robotic locomotion environments in the Mujoco benchmark are deterministic, and many robotic tasks are expected to be deterministic environments. Furthermore, many of the Atari game environments are deterministic (Bellemare et al., 2013) . Thus, from an applications perspective, the class of deterministic environments is a large and important class. Although we assume that the environment is deterministic, as is typically the case with reinforcement learning, we do not assume the functions g(s, a) and r(s, a) are known.

In batch reinforcement learning, we are provided a batch of m data points B = {(s i , a i , r i , s i ), i = 1, ..., m}. We assume B is fixed and given, and there is no further interaction with the environment. Often the batch B is training data, generated in some episodic fashion. However, in the batch reinforcement learning problem, we do not have knowledge of the algorithm, models, or seeds that were used to generate the episodes in the batch B.

Typically the batch data is generated during training with a non-stationary DRL policy. After training, the original DRL algorithm produces a final-DRL policy, with exploration turned off. In our numerical experiments, we will compare the performance of policies obtained by batch algorithms with the performance of the final-DRL policy. Ideally, we would like the performance of the batchderived policy to be as good or better than the final-DRL policy.

The case where batch data is generated from a non-stationary training policy is of particular interest because it is typically a rich data set from which it may be possible to derive high-performing policies. Furthermore, a batch learning algorithm can potentially be deployed as part of a growingbatch algorithm, where the batch algorithm seeks a high-performing exploitation policy using the current data in an experience replay buffer, combines this policy with exploration to add fresh data to the buffer, and then repeats the whole process (Lange et al., 2012) .

4. BEST-ACTION IMITATION LEARNING (BAIL):

In this paper we present BAIL, an algorithm that not only performs well on simulated robotic locomotion tasks, but is also conceptually and algorithmically simple. BAIL has two steps. In the first step, it selects from the batch data B the state-action pairs for which the actions are believed to be good actions for their corresponding states. In the second step, we simply train a policy network with imitation learning using the selected actions from the first step.

Many approaches could be employed to select the best actions. In this paper we propose training a single neural network to create an upper envelope of the Monte Carlo returns, and then selecting the state-action pairs in the batch B that have returns near the upper envelope.

4.1. UPPER ENVELOPE:

We first define a λ-smooth upper envelope, and then provide an algorithm for finding it. To the best of our knowledge, the notion of the upper envelope of a data set is novel.

Recall that we have a batch of data B = {(s i , a i , r i , s i ), i = 1, ..., m}. Although we do not assume we know what algorithm was used to generate the batch, we make the natural assumption that the data in the batch was generated in an episodic fashion, and that the data in the batch is ordered accordingly. For each data point i ∈ {1, . . . , m}, we calculate an approximate Monte Carlo return G i as the sum of the discounted returns from state s i to the end of the episode:

G i = Ti t=i γ t-i r t (1)

where T i denotes the time at which the episode ends for the data point s i . The Mujoco environments are naturally infinite-horizon non-episodic continuing-task environments (Sutton & Barto, 2018) .

During training, however, researchers typically create artificial episodes of length 1000 time steps; after 1000 time steps, a random initial state is chosen and a new episode begins. Because the Mujoco environments are continuing tasks, it is desirable to approximate the return over the infinite horizon, particularly for i values that are close to the (artificial) end of an episode. To do this, we note that the data-generation policy from one episode to the next typically changes slowly. We therefore apply a simple augmentation heuristic of concatenating the subsequent episode to the current episode, and running the sum in (1) to infinity. (In practice, we end the sum when the discounting reduces the contribution of the rewards to a negligible amount.) Our ablation study in the Appendix shows that this simple heuristic can significantly improve performance. Note this approach also obviates the need for knowing when new episodes begin in the data set B.

Having defined the return for each data point in the batch, we now seek an upper-envelope V (s) for the data G := {(s i , G i ), i = 1, ..., m}. Let V φ (s) be a neural network with parameters φ that takes as input a state s and outputs a real number. We say that V φ * (s) is a λ-smooth upper envelope for G if it has the following properties:

1. V φ * (s i ) ≥ G i for all i = 1, . . . , m.

2. Among all the parameterized functions V φ (s) satisfying condition 1, it minimizes:

L(φ) = m i=1 [V φ (s i ) -G i ] 2 + λ||φ|| 2 (2)

where λ is a non-negative constant. An upper-envelope is thus a smooth function that lies above all the data points, but is nevertheless close to the data points. Theorem 4.1. Suppose that

V φ * (s) is a λ-smooth upper envelope for G. Then, (1) V φ * (s) = max{G i : i = 1, 2, . . . , m} as λ → ∞.

(2) If there is sufficient capacity in the network and λ = 0, then the V φ * interpolates the data in G. For example, if λ = 0 and V φ (s) is a neural network with ReLU activation functions with at least 2m + d weights and two layers, where d is the dimension of the state space S, then V φ * (s i ) = G i for all i = 1, 2, . . . , m.

From the above theorem, we see that when λ is very small, the upper envelope aims to interpolate the data, and when λ is large, the upper envelope approaches a constant going through the highest data point. Just as in classical regression, there is a sweet-spot for λ, the one that provides the best generalization.

We note that there are other natural approaches for defining an upper-envelope, some based on alternative loss functions, others based on data clustering without making use of function approximators. Also, it may be possible to combine episodes to generate improved upper envelopes. These are all questions for future research.

To obtain an approximate upper envelope of the data G, we employ classic regression with a modified loss function, namely,

EQUATION

where K >> 1 and 1 (•) is the indicator function.

For a finite K value, the above loss function will only produce an approximate upper envelope, since it is possible V (s i ) may be slightly less than G i for a few data points. In practice, we find K = 10, 000 works well for all environments tested. When K → ∞, the approximation becomes exact, as stated in the following: Theorem 4.2. Let φ * be an optimal solution that minimizes L(φ). Then, when K → ∞, V φ * (s) will be an exact λ-smooth upper envelope.

Also, instead of L2 regularization, in practice we employ the simpler early-stopping regularization, thereby obviating a search for the parameter λ. We also clip the upper envelope at values near max i G i , as described in the appendix, which can potentially provide further gains in performance.

4.2. SELECTING THE BEST ACTIONS:

The BAIL algorithm employs the upper envelope to select the best (s, a) pairs from the batch data B. It then uses ordinary imitation learning (behavioral cloning) to train a policy network using the selected actions. Let V (s) denote the upper envelope obtained from minimizing the loss function (3) for a fixed value of K.

We consider two approaches for selecting the best actions. In the first approach, which we call BAIL-border, we choose all (s i , a i ) pairs from the batch data set B such that

EQUATION

We set x such that p% of the data points are selected, where p is a hyper-parameter. In this paper we use p = 25 for all environments and batches. Thus BAIL-border chooses state-action pairs whose returns are near the upper envelope. The pseudo-code for the Bail-border algorithm is given in the appendix.

In the second approach, which we refer to as BAIL-TD, we select a pair (s i , a i ) if

r i + γV (s i ) > xV (s i ) (5)

where x is a hyper-parameter close to 1. Thus BAIL-TD chooses state-action pairs for which backedup estimated return r i + γV (s i ) is close to the upper envelope value V (s i ).

In summary, BAIL employs two neural networks. The first neural network is used to approximate a value function based on the data in the batch B. The second neural network is the policy network, which is trained with imitation learning. This simple approach does not suffer from extrapolation error since it does not perform any optimization over the action space. An algorithmic description of BAIL is given in Algorithm 1. 

(φ) = |B| i=1 (V φ (s i ) -G i ) 2 {1 (V φ (si)>Gi) + K1 (V φ (si)<Gi) } + λ φ 2 7: Select data i for G i > xV φ (s i ), select

x so that p% data in B are selected, let U be set of selected data 8: for l = 1, . . . , L do 9: sample a minibatch U of data from U 10: minimize over θ: L

(θ) = |U | i=1 (π θ (s i ) -a i ) 2

5. EXPERIMENTAL RESULTS:

We carried out experiments with four of the most challenging environments in the Mujoco benchmark (Todorov et al., 2012) of OpenAI Gym. For the environments Hopper-v2, Walker-v2 and HalfCheetah-v2, we used the "Final Buffer" batch exactly as described in Fujimoto et al. (2018a) to allow for a fair comparison with BCQ. Specifically, we trained DDPG for one million time steps with σ = 0.5 to generate a batch. For the environment Ant-v2, we trained adaptive SAC (Haarnoja et al., 2018b) again for one million time steps to generate a batch.

In our experiments, we found that different batches generated with different seeds but with the same algorithm in the same environment can lead to surprisingly different results for batch DRL algorithms. To address this, for each of the environments we generated four batches, giving a total of 16 data sets.

Figure 1 provides visualizations of 4 of the 16 upper envelopes, one for each of the 4 environments. In each visualization, the data points in the corresponding batch are ordered according to their upperenvelope values V (s i ). With this new ordering, the figure plots (s i , G i ) for each of the one million data points. The monotonically increasing blue line is the the upper envelope obtained by minimizing V (s). Note that in all of the figures, a small fraction of the data points are above upper envelopes due to the finite value of K = 10, 000. But also note that the upper envelope mostly hugs the data. The constant black line is the clipping value. The final upper envelope is the minimum of the blue and black lines. All 16 upper envelopes are shown in the appendix. When training with BCQ, we used the code provided by the authors (Fujimoto et al., 2018a) . Because at the time of writing the code for BEAR was not available, we do not compare our results with BEAR (Kumar et al., 2019) . Also, all the results presented in this section are for BAIL-border, which we simply refer to as BAIL. In the appendix we provide results for BAIL-TD. The x-axis is the number of parameter updates and the y-axis is the test return averaged over 10 episodes. BAIL, BCQ, and BC are each trained with five seeds. The figure shows the mean and standard deviation confidence intervals for these values. The figure also shows test result of the final-DDPG/SAC policy. This value is obtained by averaging test results from the last 100,000 timesteps (of one million time steps). During this period, test performance of SAC and particularly DDPG can greatly fluctuate with relatively small improvement on average. We calculate the mean and standard deviation of the test results over this period, plot the mean as a straight line, and use the transparent green background to show the confidence intervals. This enables us to fairly compare the performance of the final test policy obtained with the behavioral algorithm with the test policies from the batch algorithms.

We make the following observations. For Hopper, Walker and Ant, BAIL always beats BCQ usually by a wide margin. For HalfCheetah, BAIL does better than BCQ for half of the batches. In almost all of the curves, BAIL has a much lower confidence interval than BCQ. Perhaps more importantly, BAIL's performance is stable over training, whereas BCQ can vary dramatically. (This is a serious issue for batch reinforcement learning, since it cannot interact with the environment to find the best stopping point.) Importantly, BAIL also performs as well or better than the Final-DDPG/SAC policy in all but of the 16 batches. This gives promise that BAIL, or a future variation of BAIL, could also be employed within a growing-batch algorithm. We also summarize the results in Table 1 . For this table, we average the performance of each algorithm over four batches, using the performance values at one million updates. Table 1 shows that BAIL's performance is better than that of BCQ for all four environments, with a 66% and 23% average improvement for Hopper and Walker, respectively. BAIL also beats the Final-DDPG/SAC policies in three of the environments, and has significantly lower variance. In the appendix we also provide experimental results for DDPG batches generated with σ = 0.1, which is similar to the "Concurrent" dataset in Fujimoto et al. (2018a) . For this low noise level 0.1, BAIL continues to beat BCQ by a wide margin for Hopper and Walker, and continues to beat Final-DDPG for half of the batches. However, in the low noise case for HalfCheetah, BCQ beats BAIL for 3 of the 4 batches.

5.1. ABLATION STUDY:

BAIL uses an upper envelope to select the "best" data points for training a policy network with imitation learning. We have shown that BAIL typcially beats ordinary behavioral cloning and BCQ by a wide margin, and often performs better than the Final-DDPG and Final-SAC policies. But it is natural to ask how BAIL performs when using more naive approaches for selecting the best actions. We consider two naive approaches. The first approach, "Highest Returns," is to select from the batch the 25% of data points that have the highest G i values. The second approach, "Recent Data," is to select the last 25% data points from the batch. Figure (3) shows the results for all four environments. We see that for each environment, the upper envelope approach is the winner for most of the batches: for Hopper, the upper envelope wins for all four batches by a wide margin; for Walker the upper-envelope approach wins by a wide margin for two batches, and ties Highest Returns for two batches; for HalfCheetah, the upper-envelope approach wins for three batches and ties Highest Returns for one batch; and for Ant, the upper-envelope approach wins for three batches and ties the other two approaches for the other batch. We conclude, for the environments and batch generation mechanisms considered in this paper, the upper envelope approach performs significantly better and is more robust than both naive approaches. In the Appendix we provide additional ablation studies. Our experimental results show that modifying the returns to approximate infinite horizon returns is often useful for BAIL's performance, and that clipping the upper envelope also provides gains although much more modest.

In summary, our experimental results show that BAIL achieves state-of-the-art performance, and often beats BCQ by a wide margin. Moreover, BAIL's performance is stable over training, whereas BCQ typically varies dramatically over training. Finally, BAIL achieves this superior performance with an algorithm that is much simpler than BCQ.

6. CONCLUSION:

Although BAIL as described in this paper is simple and gives state-of-the-art performance, there are several directions that could be explored in the future for extending BAIL. One avenue is generating multiple upper envelopes from the same batch, and then ensembling or using a heuristic to pick the upper envelope which we believe would give the best performance. A second avenue is to optimize the policy by modifying the best actions. A third avenue is to assign weights to the state-action pairs when training with imitation learning. And a fourth avenue is to explore designing a growing batch algorithm which uses BAIL as a subroutine for finding a high-performing exploitation policy for each batch iteration.

A PROOFS:

A. Proof of Theorem 4.1

Proof. First, let us consider the case when λ → +∞. We can re-write the definition of the upper envelope as a constrained optimization problem:

min φ m i=1 [V φ (s i ) -G i ] 2 + λ φ 2 (6) s.t. G i -V φ (s i ) ≤ 0, i = 1, 2 .

. . , m where V φ * is the optimal solution to the above optimization problem. We write the Lagrangian function:

EQUATION

As the optimal solution, V φ * must satisfy the KKT conditions specified below:

EQUATION

Suppose φ = (φ 1 , . . . , φ n ), by the first KKT condition, we have

∂ ∂φ j m i=1 [V φ (s i ) -G i ] 2 | φj =φ * j + 2λφ * j + ∂ ∂φ j m i=1 µ i [G i -V φ (s i )]| φj =φ * j = 0, j = 1, 2, . . . , n.

So we have:

φ * j = - 1 2λ ∂ ∂φ j m i=1 [V φ (s i )-G i ] 2 | φj =φ * j - 1 2λ ∂ ∂φ j m i=1 µ i [G i -V φ (s i )]| φj =φ * j , j = 1, 2, . . . , n.

When λ → ∞, we have φ * = 0. In this case, it follows that V φ * = C for some constant C. As V φ * (s i ) ≥ G i , in order to minimize (3) we must have C = max{G i , i = 1, 2, . . . , m}.

For the case of λ = 0, notice that we only have finitely many input s i to be the input of the neural network. Therefore, this is a typical problem regarding the finite-sample expressivity of the neural networks, and the proof directly follows from the work done by Zhang et al. (Zhang et al., 2017) .

B. Proof of Theorem 4.2:

Proof. Let φ * be the optimal value that minimizes L(φ). Let's proceed by contradiction and assume that there exists some k such that

V φ * (s k ) < G k Let φ be an arbitrary given value such that V φ (s i ) ≥ G i for all i ∈ {1, 2, . . . , m}. Then we have m i=1 (V φ * (s i ) -G i ) 2 1 (V φ * (si)>Gi) + K(V φ * (s k ) -G k ) 2 + λ φ * 2 ≤ L(φ * ) L(φ * ) ≤ L(φ ) = m i=1 (V φ (s i ) -G i ) 2 + φ 2 This implies that m i=1 (V φ * (s i ) -G i ) 2 1 (V φ * (si)>Gi) + K(V φ * (s k ) -G k ) 2 + λ φ * 2 ≤ m i=1 (V φ (s i ) -G i ) 2 + φ 2

which is impossible when K → ∞. Therefore, we must have V φ * (s i ) ≥ G i for all i ∈ {1, 2, . . . , m} as K → ∞. In this way, when K → ∞, φ * actually minimizes

L(φ) = m i=1 (V φ (s i ) -G i ) 2 + φ 2

which completes the proof. For the behavioral DDPG algorithm, we used the implementation of Fujimoto et al. (2018b) . For the behavioral SAC algorithm, we implemented it in Pytorch, mainly following the pseudocode provided by (Achiam) , and used hyperparameters in Haarnoja et al. (2018b) . For the BCQ algorithm, we used the authors' implementation (Fujimoto et al., 2018a) . For behavioral cloning and its variants in the ablation study section, the network structure, learning rate, mini-batch size, and so on are identical to those in Table 3 for BAIL.

B IMPLEMENTATION DETAILS AND HYPERPARAMETERS B.1 IMPLEMENTATION OF BAIL ALGORITHM:

For the upper envelope network, our network has two hidden layers as does the Q network in BCQ and SAC. However, the number of hidden units in our network is less than those used in BCQ (400 × 300) and SAC (256 × 256). In future work we will see if we can obtain further improvements with BAIL using larger networks for the upper envelope.

B.3 CLIPPING HEURISTICS:

As is shown in Figure 1 , in practice the trained upper envelope does not always fit well the data points on the right side of the plots, where the upper envelope can become very large. In that region, the data points with the highest returns will not be selected as "best actions" and therefore not used for imitation learning step. We observe that if we plot the upper envelope values for all the states in the buffer in ascending order as is shown in Figure 1 , the upper envelope value V (s i ) starts to deviate from the Monte Carlo return G i at a point where V (s i ) ≈ max{G i }. We therefore use the following heuristic. We say that the upper envelope value begins to deviate from the Monte Carlo return at state s i if V (s j ) > G j for i ≤ j < i + 10000. We set the clipping value C = V (s ) where s is the starting point of this deviation. Then the actual UE values used to select data is min{V (s), C}. In practice, the clipping heuristic gives a small boost in performance as is shown in Figure 5 .

C ADDITIONAL EXPERIMENTAL RESULTS:



C.1 EXPERIMENTS ON BAIL-TD:

We present some of the experimental results for BAIL-TD in this subsection. Figure 4 shows that the performance of BAIL-TD is similar to the performance of naive Behavioral Cloning in Hopper-v2 and Walker2d-v2 environments. The result implies that the BAIL-TD approach is limited in the ability to distinguish between good data points and the bad ones. This is possibly due to the inaccuracy of the trained upper envelope. Recall that in BAIL-TD algorithm, we select state action pairs based on the difference between the values of r +V (s i ) and V (s i ). Since we only use one-step Monte Carlo return, and the value of r is very small compared with the value of v(s i ), the selection is very sensitive to the accuracy of V (s i ). We do additional ablation studies for BAIL, where we focus on two heuristic features in the BAIL algorithm: clipping of upper envelope, and return augmentation to approximate non-episodic continuous tasks. We removed each of these features one at a time from the BAIL algorithm and compare with the original BAIL algorithm. Each performance curve is again averaged over 5 independent runs. We found that clipping often, but not always, gives a small boost in performance. As for return augmentation, we found it does not harm the performance of BAIL, and sometimes gives a considerable improvement, particularly for Hopper. 

:

https://anonymous.4open.science/r/e5fbe703-a32d-4679-a2a8-095e74b96e85

Figures and Tables:

upper envelope parameters φ, policy parameter θ, obtain batch data B. 2: Compute return for each data point i: G i = Ti t=i γ t-i r t 3: Obtain upper envelope by minimizing the loss: 4: for j = 1, . . . , J do 5: sample a minibatch of data B from the batch B 6: minimize over φ: L

Figure 1: Five illustrative upper envelopes trained from data with adaptive clipping

Figure 2: Performance comparison of BAIL, BCQ, BC and Final-DDPG/SAC

Figure 3: Comparison of BAIL scheme with Highest Returns and with Recent Samples schemes. All schemes use 25% of the data in the batch.

Figure 4: Performance comparison of BAIL-border, BAIL-TD, BCQ, BC and Behavioral Policy (DDPG)

Figure 5: Ablation of BAIL without Upper Envelope clipping

Figure 6: Ablation of BAIL without return augmentation

Performance comparison at one million samples (mean and std over batches and random seeds). Last column shows percentage improvement of BAIL over BCQ.

Table content: <table><tr><td>Environment</td><td colspan="2">Final-DDPG/SAC BCQ</td><td>BAIL</td><td>Improvement</td></tr><tr><td>Hopper-v2</td><td>2547.7 ± 750.4</td><td colspan="2">1468.9 ± 552.6 2437.5 ± 489.7</td><td>65.9%</td></tr><tr><td>Walker2d-v2</td><td>1742.1 ± 656.3</td><td colspan="3">2020.2 ± 699.3 2496.7 ± 409.9 23.6%</td></tr><tr><td colspan="2">HalfCheetah-v2 2612.4 ± 342.2</td><td colspan="2">2449.7 ± 267.7 2660.0 ± 77.7</td><td>8.6%</td></tr><tr><td>Ant-v2</td><td>4506.0 ± 483.6</td><td colspan="3">4315.6 ± 416.4 4630.7 ± 310.9 7.3%</td></tr></table>

Upper Envelope Hyperparameters

Table content: <table><tr><td>Parameter</td><td>Value</td></tr><tr><td>optimizer</td><td>Adam (Kingma &amp; Ba, 2014)</td></tr><tr><td>learning rate</td><td>3 • 10 -3</td></tr><tr><td>discount (γ)</td><td>0.99</td></tr><tr><td colspan="2">regularization constant λ 2 • 10 -2</td></tr><tr><td>K</td><td>10,000</td></tr><tr><td>number of hidden units</td><td>128 × 128</td></tr><tr><td colspan="2">Table 3: BAIL Hyperparameters</td></tr><tr><td>Parameter</td><td>Value</td></tr><tr><td>data in batch</td><td>10 6</td></tr><tr><td>optimizer</td><td>Adam (Kingma &amp; Ba, 2014)</td></tr><tr><td>learning rate</td><td>10 -3</td></tr><tr><td colspan="2">regularization λ 0</td></tr><tr><td>mini-batch size</td><td>100</td></tr><tr><td>BAIL-border p%</td><td>25%</td></tr><tr><td>BAIL-TD x</td><td>0.96</td></tr><tr><td>number of hidden units</td><td>400 × 300</td></tr><tr><td colspan="2">B.2 IMPLEMENTATION OF COMPETING ALGORITHMS</td></tr></table>

Peer review (iclr2020):
Summary of Claims: The paper proposes a batch RL method that they claim is simpler than most existing methods that try to avoid the extrapolation error that is prevalent among batch RL methods. They do this by completely avoiding the minimization/maximization (cost/reward) of the approximate value function that is fit to the batch transitions. Instead, they train an approximation for the state value function*s tight upper bound (which they refer to as the upper-envelope) by using their monte-carlo returns. By fitting such an approximator, they sample the state-action pairs that are close to the envelope (thus have high values/returns), and use behavioral cloning to fit a parameterized policy to those state-action pairs. Decision: Weak Reject. My decision is influenced by two main reasons: (1) Although the simplicity of the method is apparent and a very desirable feature, the authors don*t highlight situations where this can lead to bad policies. For example, consider that there are two pairs (s, a_1, s*) and (s, a_2, s*) in the batch that are close to the upper-envelope, and hence will both be used for training the policy. Using Behavioral cloning, the policy would regress to the mean of a_1 and a_2, which could be a terrible action altogether. The issue here is that only one of these two pairs has higher return and our policy needs to only predict that action (or in the case of tie, either one.) This can be really bad in situations where two very different actions can lead to same returns (e.g. in a reacher-like task the arm can reach a goal in two different rotations.) Even though I pointed out a very specific case, one could think of many other cases where the proposed approach might result in a bad policy. Having said all of this, it might be true that such cases do not appear in practice (which I highly doubt) but its the authors job to raise and clarify that. The current set of experimental setups (mujoco locomotion problems) are not good enough evidence for that and they need experiments where optimal-policies can be multi-modal or have diverse experimental setups (manipulation etc.) (2) Experimental results are a little unsettling. The primary reason is that in all of the plots, BCQ, BAIL, BC aren*t starting from the same test return at 0 parameter updates! In most plots BAIL starts off way higher in return than BCQ, BC with no parameter updates yet, which suggests that the experiments were not setup well. Maybe, they didn*t initialize the policy in the same way for all the approaches, maybe the random seeds were not the same for all approaches, or maybe BAIL had some sort of pretraining for the policy that was not accounted for in the parameter updates. In any way, this needs to be addressed. This is also highlighted by the fact that the learning curves for BAIL are almost always flat across a million parameter updates! If you are starting off with a random initialization, there should be an upwards slope for the learning curve. Also, as raised in the previous point I think using these Mujoco locomotion environments is not convincing enough to claim that BAIL is a viable competitive batch RL approach. Comments and Questions: (1) I like the simplicity of the approach and the fact that it is much more easier to understand than existing works like BCQ (2) Paper is well-written. It was clear, lucid and descriptive. (3) Why is the deterministic dynamics assumption needed? I am curious (4) The paper makes some subjective statements such as *BEAR is also complex*, which is not substantiated well enough. Refrain from making such statements (5) Not comparing to BEAR because their code is not publicly available is a contentious reason. I personally feel that the authors could have reimplemented it and compared but I am not sure what the community feels about that (6) Is there any reason why REM cannot be applied to mujoco environments? If it can be, then why did the authors not compare to REM as well? (7) Another subjective statement (that is clearly wrong) *many robotic tasks are expected to be deterministic environments* - although this is slightly true, the reason we model environments to be stochastic is not because there is inherent randomness in them but because our state descriptions are never complete. The state descriptors are always partial and we account for them by assuming stochasticity in the dynamics. For example, consider a robotic manipulation task where if you know all the environmental factors as part of your state space(such as the friction coefficients) you can assume deterministic dynamics, else you are better off assuming stochastic dynamics because the same actuation might not result in the same motion every time (because of varying friction) (8) Concatenating subsequent episodes in a batch only makes sense (as the authors point out) if the policy doesn*t change much across episodes. But this is not true of current off-policy RL methods like DDPG, SAC. You either need very small learning rate or a trust-region constraint to ensure that the policy doesn*t change much across episodes. (9) Why do different batches with different seeds and the same algorithm lead to widely different results for batch RL? There is clearly something fishy here. Is it because of the off-policy RL methods used to collect the data, is it due to the batch RL method used? More investigation needed