Multi-batch Reinforcement Learning via Sample Transfer and Imitation Learning
Authors: Di Wu
Tianyu Li
David Meger
Michael Jenkin
Xue Liu
Gregory Dudek
Venue: ICLR 2022 Submitted (2022)
Keywords: Multi-bacth
batch reinfrocement learning
sample transfer

Abstract: Reinforcement learning (RL), especially deep reinforcement learning, has achieved impressive performance on different control tasks. Unfortunately, most online reinforcement learning algorithms require a large number of interactions with the environment to learn a reliable control policy. This assumption of the availability of repeated interactions with the environment does not hold for many real-world applications due to safety concerns, the cost/inconvenience related to interactions, or the lack of an accurate simulator to enable effective sim2real training. As a consequence, there has been a surge in research addressing this issue, including batch reinforcement learning. Batch RL aims to learn a good control policy from a previously collected dataset. Most existing batch RL algorithms are designed for a single batch setting and assume that we have a large number of interaction samples in fixed data sets. These assumptions limit the use of batch RL algorithms in the real world. We use transfer learning to address this data efficiency challenge. This approach is evaluated on multiple continuous control tasks against several robust baselines. Compared with other batch RL algorithms, the methods described here can be used to deal with more general real-world scenarios. 

Under review as a conference paper at ICLR 2022
MULTI-BATCH REINFORCEMENT LEARNING VIA SAM- PLE TRANSFER AND IMITATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning (RL), especially deep reinforcement learning, has achieved impressive performance on different control tasks. Unfor- tunately, most online reinforcement learning algorithms require large number of interactions with the environment to learn a reliable control policy. This assumption the availability of repeated interactions with the environment does not hold for many real-world applications due to safety concerns, the cost/inconvemience related to interactions, or the lack of an accurate simulator to enable effective sim2real training. As a consequence, there has been a surge in research addressing this issue, including batch reinforcement learning. Batch RL aims to learn good control policy from a previously collected dataset. Most existing batch RL algorithms designed for single batch setting and assume that we have large number of interaction samples in fixed data sets. These assumptions limit the use of batch RL algorithms in the real world. We use transfer learning to address this data efficiency challenge. This ap- proach is evaluated on multiple continuous control tasks against several robust baselines. Compared with other batch RL algorithms, the meth- ods described here can be used to deal with more general real-world scenarios.
INTRODUCTION
Reinforcement learning aims to learn an optimal control policy through interactions with the envi- ronment (Sutton Barto, 2018). Deep reinforcement learning (Deep RL or DRL) combines neural networks with reinforcement learning and further enables RL. agents to deal with more complex environments. DRL has achieved impressive successes in different areas including the game of Go (Silver et al., 2017), Atari games (Mnih et al., 2015), and continuous control tasks (Lillicrap et al., 2015). However, deploying RL algorithms for weal-world problems can be very challenging. Com- pared with supervised learning algorithms (e.g., classification regression), most reinforcement learning algorithms need to interact with the environment many times to learn a reliable control policy. This process be very costly even dangerous for some real-world applications, e.g., safety-critical applications. The current success of deep RI. algorithms heavily depends on a large number interactions with the environment. Thus the practical application of reinforcement learn- ing algorithms in the real world is critically limited by its poor data efficiency and its inflexibility of learning in an offline fashion.
Batch reinforcement learning (also known as offline reinforcement learning) algorithms have been developed to solve this issue. Batch RL aims learn a control policy from a previously collected dataset without further interactions with the environment Batch reinforcement learning has attracted a considerable amount of attention due to its potential in dealing with real-world problems. There have been many efforts in developing batch RL methods: early approaches include fitted Q iteration method (Ernst et al., 2005) which uses a tree-based model to estimate the state-action value function (Q function) in iterative way, the neural fitted Q method (Riedmiller, 2005), which leverages multilayer perceptron (MLP) to approximate the Q function. Since these earlier efforts, a number of batch reinforcement learning algorithms have been developed that further improve the learning per- formance. These approaches can be generally categorized as Q function based methods (Fujimoto al., 2019; Kumar et al., 2020) and imitation learning based methods (Wang et al., 2018; Peng et al.,

---

Under review as a conference paper at ICLR 2022
2019). Examples of Q function based methods include (Fujimoto et al., 2019), in which the authors constrain updates of Q values to deal with distribution drift, and (Kumar et al., 2020), where an additional penalty term was added to constrain the update for the Q function. Best Action Imitation Learning (BAIL) (Chen et al., 2019). on the other hand, leveraging the so-called upper envelope, directly select sub-batch of the dataset and then execute imitation learning on the selected data to learn the control policy.
Even with these advances, most existing batch RL algorithms assume that we have a large number of data points in the batch. In the real world, this may be an unrealistic assumption. For example, when learning energy management control policies, we may only have a very limited amount of collected data for newly built houses and buildings. It is hard for most current batch RL algorithms to learn a reliable policy with a limited amount of data points. In this work, we use transfer learning (Torrey & Shavlik, 2010) to address this issue. Transfer learning aims to use the knowledge from source domains (domains for which we have a large amount of data) to improve the learning performance in the target domain (the domain we are interested in, but for which we only have a limited amount of data). Depending on the manner in which knowledge is reused from the source domains, there three main categories of transfer learning: sample transfer, model transfer, and representation transfer. In this work, we use sample transfer, i.e., we transfer some related data points from the source tasks to improve the learning performance in the target control task. The proposed algorithm is referred to as BAIL+ as it is a direct extension of the BAIL. algorithm proposed by (Chen et al., 2019).
Multitask learning (MTL) (Caruana, 1997) aims to learn a set of tasks jointly instead of learning them separately to achieve a better overall learning performance. In general, multitask learning can help to improve the data efficiency via reusing the shared representations and training on data from multiple sources. Multitask learning has shown its effectiveness on different applications such as image classification (Li et al., 2015; 2018). nature language processing (Collobert & Weston, 2008; Liu et al., 2019), and speech recognition (Siohan Rybach, 2015). It has shown to be useful in deep reinforcement learning field in recent years (Li et al., 2019). The main objective is to learn one (or more) policy that can perform well on multiple tasks. In (Rusu et al., 2015), the authors proposed to learn multitask policy from set of DQN experts that are pre-trained on set of tasks separately. Lately, (Hessel et al., 2019) investigated how to balance the learning over multiple tasks with one deep neural network based policy network. The benefit of representation sharing is also investigated in (D' D'Eramo et al., 2019) for learning deep RL based control agent. Most of existing multitask RL works are based on online RL. There are some preliminary investigations on batch RL. As shown in (Li et al., 2019), the authors proposed to learn distill pretarined BCQ models into one model. However, despite few attempts, how to further improve the task-level generalization for batch RL is still an open question. In this work, different from previous works, we propose to first improve the learning performance on single tasks and then utilize the policy distillation to combine the learned policies into one single policy.
Batch RL algorithms are typically designed to deal with single task scenario (a single batch setting). In the real world, it is more common to have batches collected from a set of tasks that have similar Markov Decision Process (MDP) settings. For example, we may have collected datasets from a set of houses/buildings in the same area. Thus, it will very helpful if one general policy that can be learned from different bacthes that perform well on these different tasks even including unseen tasks without further adaption. In (Li et al., 2019), the authors implemented preliminary investigations on multitask batch reinforcement learning. In our work, to improve the task-level generalization of the policy learned with batch RL, further extend BAIL+ for multi-batch settings via policy distillation. The resulting algorithm is referred to as MBAIL. Specifically, we aims to learn general policy without the need to inferring the task identity which can make the batch RL more applicable in real world.
The remainder of this paper is organized as follows. The preliminaries of this work including batch reinforcement learning, multi-batch reinforcement learning, and BAIL algorithm are presented in Section II. The details for BAIL+ and MBAIL are presented in Section III. The effectiveness of the proposed methods are showcased in Section IV. Finally, conclusions and future work are presented in Section V.
2

---

Under review as a conference paper at ICLR 2022
2 PRELIMINARIES
This section reviews the main technical background and notation used in this paper including rein- forcement learning, batch reinforcement learning, BAIL, and multi-batch reinforcement learning.
Reinforcement Learning Reinforcement learning (Sutton Barto, 2018) a learning paradigm in which a learning agent aims to learn an optimal control policy by interacting with the environment. Reinforcement learning has been successfully applied in many areas including transportation (Hay- dari & Yilmaz, 2020), smart grids (Yang et al., 2020), and recommendation systems (Zou et al., 2019). Formally, an RL problem is typically formulated as a Markov Decision Process (MDP), i.e., tuple (S,A, A. p. J4. 7), where S is the state space, A is the action space, p S @ A S is the state transition function, r S A - I is the reward function, l is the initial state distribution, and y is the discount factor. The solution an RL problem (control policy) a function: : S + A. To obtain this solution, the agent needs to achieve the maximum expected cumulative reward, that and a policy π. the state value function (a.k...,
is, the so-called value function. Given state value function) under policy π is defined by (*(s) = 12 (Ri + yR2 + ...7"" (R,)), where R; denotes the reward obtained time step i.
Batch Reinforcement Learning In batch reinforcement learning (batch RL), the goal is to learn a high performance control policy using an offline dataset without further interactions with the en- vironment. The dataset consists of N data points B = {(s,, q₁, Tr, s,) = 1, N.. In general. we have no prior requests on the policy used collect the batch B. B can be obtained while training an RL policy in episodic fashion or running some other control policy (e.g.. rule-based methods) in the same way. The distribution of states and actions in the dataset can be far from those induced by the current policy under consideration. This complicates computations related to evaluating and optimizing behaviors, and this has been the primary consideration of several Batch RL solution methods. To counter this extrapolation problem, (Fujimoto et al., 2019) proposed an algorithm to learn policies with soft constrain lie near the batch, which alleviate the extrapolation problem.
BAIL: Best Action Imitation Learning Best Action Imitation Learning (BAIL) (Chen et al., 2019) is a simple and computationally efficient imitation learning based batch RL algorithm. The core concept of BAIL is very simple: finding actions that can achieve high return for each state and then learning a control policy based on these selection state-action pairs. To be more specific, for a particular state-action pair (s, a), let G(s, a) denote the return starting in state s and action a, under the policy л. Denote the optimal value function by (s(s). Then if the action a* satisfies G(s, a²) (*(s), a* is an optimal action for state s. The problem now becomes how to obtain V in batch setting. Since there is no further interaction with the environment, it is impossible to find V*, Therefore we seek to eliminate many useless state-action pairs in the batch as possible, to avoid the algorithm inferring bad actions. To do this, we estimate supremum of the optimal value function V*, which referred to as the upper envelope. Given & = (ur, b), a neural network parameterized V S - R. a regularization weight À and dataset D of size m, where D; (st, G&) and G, is the accumulated return of the state s; computed within the given batch, then the upper envelope function V* S + IR is estimated by minimizing the following loss function:
m
min
(V(+)) - -,,2 + A/100
s.l. Volss) G3 where i = 1,2,................. ............ m
Once the upper envelope function Vg is estimated, the best state-action pairs can be selected from the batch data B based on the estimated Vg. One way of selecting such pair that for a fixed / > 0. we choose all (8i, az) pairs from the batch data set B such that:
G > (Vo
Typically, set B such that p% of the data points are selected, where p is a hyper-parameee In this work, we follow the same setting as (Chen et al., 2019), in which is set to ensure that approximately 25% of all the data points selected for each batch.
3

---

Under review as a conference paper at ICLR 2022
Algorithm BAIL+: Best Action Imitation Learning with Multi-source Sample Transfer
Input: A target task batch B; and N source task batches B ......... .B and the pre-defined sample selection ratio threshold ā
1: Learn the upper envelope function Vi and the reward function i for batch B..
2: for j = 1...... .N do
3: for d d= ,, do
4: Denote the current state action pair by (5 a²))
5: Following equation 5, estimate return of sample (5 az)) denote by G2.
6: Compute the sample selection ratio ass a²) via equation 4.
7:
if al'ss o²)) ā then
8: Append (s', a²)) to dataset B,
9: end if
end for
11: Learn the final policy ne on B, via imitation learning
12: end for
Output: the final policy of the target task n..
Multi- Batth Reinforcement Learning To make batch RL more suitable for real-world applica- tions, it is desirable that the learned control policy performs well in multiple situations. In this work, aim to learn one RL agent from set of batches sampled from set of tasks (T,, TN... Then the multi-task (multi-batch) batch reinforcement learning can be formulated as:
arg max J(Ø) = ET) (Jr: (To)]
(3)
6
where (TO) ) is referred to the performance of control policy To task i. Here p(T) defines the task distribution and for each task i, we have a corresponding dataset consisting of tuples B = ((0), af, ri, of )== =
K}. k,..k.
3 METHODOLOGY
In this work, aim to improve the RL agent's performance over mutiple tasks given set datasets collected from multiple task. We tackle this problem in two stages. We first improve the BAIL algorithm via sample transfer, which results in the BAIL+ algorithm, in which we have one target task with set of related source tasks. Furthermore, leveraging policy distillation, we extend BAIL+ MBAIL (Multi-batch Best Action based Imitation learning with sample transfer) to achieve better generalization over multiple tasks. Below we will illustrate both of our methods in more details.
3.1 BAIL+
When dealing with set of related tasks, it is natural to think about how to leverage data across all the tasks. To achieve this, one straight forward approach is through sample transfer. The core idea of sample transfer is utilize samples from numerous source tasks construct comprehensive dataset to improve the learning on the target task. In terms of BAIL, we employ an effective ap- proach, that is, given target task Tr, state action pair (s, a) from any source task and its trajectory Ms,a ((8, a), (81, ai)..... ..., (Sk; d& )), we define the sample selection ratio of the state action pair (s, a) similar the definition of BAIL's sample selection ratio 3, i.e.
(3(,,))) a(s,a) a) V*(s)
(4)
Note (*(r)) is the estimated return of the source task samples evaluated on the target task and the same is for (s). Then given a selection threshold å, if any state action pair (s, a) has a<s, a) > a, will incorporate this pair into newly selected batch. The motivation for this is that, by assuming the correct estimation of G, and V', we just need to follow the original BAIL routine to pick the state action pair that induces the best action.
4

---

Under review as a conference paper at ICLR 2022
13
Transfer
H ad
13
Transfer
1
Distil
Bv
Transfar
Figure I: The overview for multi-batch BAIL.
Then, the imminent problem is to obtain an estimate of the return GH evaluated on the target task. To solve this, we first learn reward function the target task fz S x A - IR. Then given a discount factor y and a trajectory of state action pairs 7 = (81,01), (82, a2),.'.. (sM, ax) from any source task batch, we can obtain its return estimate on the target task, that is:
(n
Σ ft
(5)
Note that we assume all tasks share the same transition function, and the batch is collected via the same policy, therefore Equation is a reasonable estimation of the return on the target task. Once the return estimation is complete, we just need to select the samples based on the selection ratio function a and some threshold ā. Similarly to BAIL, we select such that the top p% of all datasets from the source tasks. Once the data is selected, then use standard supervised learning based imitation learning method to obtain the final BAIL+ policy.
The pseudo code for BAIL+ is given in Algorithm 1. Assume have target task T. and N source tasks Tı, .T.,..... ... TN and their batch dataset Br and B1, B2,... .e.,.,BBB. By of state action pairs. These tasks share the same state space, action space and transition functions, but have different |Bxl M. For the sake of simplicity, we also assume the
reward functions. Assume = length of all trajectories is L. Note this assumption can be easily lifted.
3.2 MULTI-BATCH BAIL+
In multitask reinforcement learning, we are often faced with set of similar tasks and it is desirable to learn policy that able leverage knowledge from all tasks and obtain a policy that has similar or better performance across all tasks. Policy distillation Rusu et al. (2015) is a classic multitask reinforcement learning approach, where the distillation agent aggregates knowledge from all of the policies and distill them into one consistent policy (Rusu et al., 2015). This distillation process leverages knowledge from all tasks and thus can potentially further improve policy performance. In this section, will introduce our approach for multitask batch reinforcement learning based on the previous BAIL+ method, we will refer to this method as MBAIL.
Given set of policies II { = [1,2, 1,.,,...,,,,. ,N] and corresponding tasks Ti,..., Tv,, want to learn policy π -> A such that Σi=1 E.ER. d((((s), (s)) minimized, where d is distance measure, and is chosen to be L2 distance. In addition, help the task identification, we incorporate a task inference module q S A IR x + k² The distilled policy and the task inference module are all parameterized by a neural network. Denote the context tuple = (s, a, R, s ), our algorithm aims to minimize the following loss function:
E 8.0.8.
[[T((() - T(s, 2.)) SKL(G(c))IW g(c:)
(6)
However, this approach is not directly deployable under the batch setting. In (Li et al., 2019), the authors observed that the task inference module has learned to model the posterior over task identity conditionally dependent only the state-action pairs, but omit the effect of reward, which is
5

---

Under review as a conference paper at ICLR 2022
Algorithm 2 MBAAL: Best Action Imitation Learning for Multiple Batches
Input: Batches B,, ",,.. .B
N tasks, maximum number of epochs
1: for 1, .. , do
2: Following Algorithm 1, train policy ne.
3: end for
4: for , do
5: Compute the distillation loss L via Equation
6: Compute the triplet loss Ctriplet via Equation 7.
7: Do gradient descent w.r.t. æ and q for the loss function: Ctriplet + L
8: end for
Output: The distilled policy æ and the task inference module q-
crucial in the multitask setting. The reason for this behavior lies in the fact that there is overlap (or little) between each batch. Therefore in this case, minimizing Equation 6 only leads to the algorithm learning the trivial correlations.
To avoid this problem, in (Li et al., 2019), the authors propose to add an additional loss function, namely the triplet loss. The motivation behind this loss is to enforce reward information take part in the task inference. The authors achieve this by introducing a relabeling process. Given context tuple c; (8,, (3; R2, s)) from batch B, and reward estimation of task j. f 8 x A R, the relabelling of c; task j. denoted by c/, is defined as: of (sis as, (5 at), s)). Then the triplet loss function is defined by:
N
N
Ctriplet
N(N Σ
Σ
- -((())))))))(())()))) + +a++
1=1,8
where is the triplet margin, H+ is the ReLU function, .q outputs the posterior over task identity and is a divergence measure which is chosen to be the KL diverge. Through minimizing Equation 7, we essentially encourage q to infer similar task representations when given either c; or c.. Moreover, it helps enforcing to infer different task identities for cf and cj. which forces g to account for the reward information instead of only relying on the state-action pairs.
Now, to add with the previous distillation loss, our multi-bate policy distillation loss is:
C = criplet + Lr.
(8)
By minimizing the loss function, will be able obtain the final distilled policy æ, well the inference module The pseudo-code of MBAIL is summarized in Algorithm 2. There are two main stages for MBAIL. In the first stage, BAIL+ used train policies for each task each group of tasks with identical properties for BAIL+. In the second stage, the policies learned from each of the individual policies are distilled into one single multitask policy.
EXPERIMENTS
In the experiment, evaluate both BAIL+ and MBAIL set of datasets introduced in (Li al., 2019). As baseline comparisons, we have also included the results for single task BAIL, as well contextual BCQ, where the networks in BCQ modified to accept the inferred task identity as input as illustrated in (Li et al., 2019).
4.1 DATASET DESCRIPTION
The datasets consist three challenging task distributions, namely Ant-Dir, Ant-Goal and HalfChee- tah Vel, from MuJoCo (Todorov al., 2012). For the multi-task setup, we utilize the exact data and evaluation protocols from (Li et al., 2019). In Ant-Dir, a target direction defines a task, where the agent maximizes returns by running with maximal speed in the target direction. In Ant-Goal, task is defined by a goal location, to which the agent should navigate. In HalfCheetah Vel, task is defined as constant velocity the agent should achieve. As specified by the dataset authors (Li al.,

---

Under review as a conference paper at ICLR 2022
1.5
1.0
0.6
-0.75
0.0
-0.5
(a) The directions of all tasks for Ant- Dir. The dashed black line indicates the an- gle border of the task distribution. The sim- ulated robot ant starts at (0, 0) and needs to quickly possible in each of the designated directions.
(b) Goal distribution of all 10 tasks for Ant- Goal. The simulated robot ant starts at (0, 0) and needs to navigate to each designated goal represented by dots in the figure.
Eplsodh
(C) example simulated ant robol. This robot is used for both Ant-Dir and Ant- Goal environments.
(d) An example the simulated half cheetah robot. This robot is used for the Halfchee- tah tah environment. The task for this envi- ronment is for the half cheetah robot to reach and maintain specific speed.
Figure 2: Illustrations of the experiment environment and their task distribution.
2019), there are total 200,000 samples for Ant-Dir, 300,000 samples for Ant-Goal and 60,000 trajectories for HalfCheetah Vel. The state measurements for each task follows exactly the OpenAl gym state. No task-specific information, for example goal location or the target velocity, added to the state. For Ant-Dir the target directions and goals are sampled from 120" circular arc. Figure illustrates these environments in detail.
4.2 EXPERIMENT SETUP
In the experiments, illustrate the performance of approaches for both Algorithm and Al- gorithm 2. For single task BAIL, we use a neural network of two layers with 128 neurons for each layer to first approximate the upper envelope function. As mentioned in the previous section, select such that the top 25% data for each batch will be selected. Afterwards, we use a three-layers MLP with 128 neurons on each layer to approximate the policy via simple behavior cloning method based on supervised learning. Both networks use ReLu activation function (Nair & Hinton, 2010) and are optimized via Adam (Kingma & Ba, 2014) with learning rate 0.001.
For BAIL+, the upper envelope function is approximated by an MLP of the same structure as for BAIL. To estimate the returns G", directly use the trained reward models from (Li et al., 2019), which is an ensemble of 10 neural networks. We then follow Equation 5 to estimate the return with the discount factor set to 0.99. Then the transfer selection ratio threshold â is determined 2.5% all the source tasks' data is selected. The imitation learning procedure follows the same as for vanilla BAIL.
that
For our MBAIL method, the distilled policy x follows the same structure as before, i.e. three layers of 128 neurons with ReLu activation function. The inference module is parameterized by two-

---

Under review as a conference paper at ICLR 2022
Training samples
Improvement (%)
5
10
20
40
60
80
100
ratio (%)
BAIL (Single)
-0.12
3.18
5.79
14.82
19.87
22.33
23.21
BAIL+
0.37
3.38
6.30
14.95
20.20
21.45
22.91
Ant-Dir
MBAIL
5.70
5.65
5,90
16.47
18.72
23,43
23.84
Contextual BCQ
0.74
2.40
4.08
11.87
14.46
18.59
19.49
BAIL (Single)
-76.81
-51.04
-50.31
-50.08
-49.91
-50.40
-49.00
BAIL+
-62.87
-51.90
-50.30
-49.74
-50.61
-50.19
-49.74
Ant-Goal
MBAII.
50.994
-49.86
-50.24
-47.07
-47.27
-6.883
-47.44
Contextual BCQ
-72.06
-44.09
-34.49
-30.25
-29.30
-26.18
-27.29
BAIL (Single)
-22.15
-22.46
-22.80
-22.08
-20.08
-17.41
-13.35
BAIL+
-22.09
-22.22
-22.52
-16.05
-13.33
12.87
-11.22
HalfChcetah
MBAIL
116995
16.885
-16.76
-15.12
-12.43
-12.04
-11.34
Contextual BCQ
-20.50
-20.77
-20.44
-16.03
-12.69
-00.69
-9.05
Table 1: Improvement percentage over baseline single task BAIL.
layered ReLu network with 200 neurons each layer. We also use Adam optimize both of these networks with 0.0005 learning rate.
Contextual BCQ have the same components the vanilla BCQ (Fujimoto et al., 2019), including
Q function, a perturbation network and variational autoencoder (Kingma & Welling, 2013). For contextual BCQ, we use exact same model structure as (Li al., 2019), where the Q function is estimated via 9 layers neural network with 1024 neurons for each layer, the perturbation network is parameterized via layers neural network with 1024 neurons and the VAE has 7 layers of the same number of neurons.
4.3 EXPERIMENT RESULTS
To evaluate our models, we run the learned policies on each of the training environments for 1,000 steps and compute their returns. We then normalize all the returns for each the tasks by the average of the Monte Carlo estimated in-batch returns G for task t. The reason for this normalization is that each task, although in the same environment, may differ in reward scaling (e.g., may be intrinsically easier to walk slowly or reach nearby goal compared with fast running or distant goals)- note we inherit this property from the public dataset and have not modified environment code or raw data. Next, we average all the normalized returns to form evaluation on the current task. general, for a policy म, let r. denote reward collected at time step when following policy evaluate all the learned policies by Equation:
(9)
To illustrate the benefit of sample transfer as well as multitask learning, we show the results using different sizes of the training batches. Because the batch for each environment varies in sample size, for the simplicity in the result presentation, we evaluate the models with different percentages the training batches. We take the first x% of the batch the training batch, where æ takes the value of 5, 10, 20, 40, 60, 80, 100, respectively. The averaged normalized returns for different training sample ratios listed in Table and the learning curves for all environments on training sample ratio 0.1 and 0.8 are presented in Figure 3.
4.4 RESULTS ANALYSIS
From Table 1, we can first clearly observe that our BAIL+ and MBAIL outperform the single trained BAIL method for nearly all of the environments. This is especially the when encountering smaller training sample sizes. This shows that our algorithms, by leveraging information from other tasks, whether it's from sample transfer or policy distillation, have on average improved the policy performance on each task.
More specifically, from Table This result shows that BAIL+, by transferring sample from source tasks to the target task, has im- proved the performance the original single task BAII policy. In addition,
that BAIL+ consistently outperforms single task BAIL.
from the
8

---

Under review as a conference paper at ICLR 2022
Ant-Gool
Ant-Dir
HaifCheetanVel
-35.0
Avg. Return 10% Data
-37.5
-40.0
-18
-19
-47.5
-50.0
-21
-.225
-22
-55.0
Aug Return 80% Data
-20
MM
-25
-10
20-
-35
13
-40
MBAIL
BAIL+
Single, BAIL
-16
-30
Contextual BCO
100 125 Number epechs
50
75
150
175
0
49
60
100
50
125 150 175
Number of epachs
Number of epochs
Figure 3: Learning curves for our methods (MBAIL and BAIL+) and the baseline single task BAIL and contextual BCQ. The top row is trained on batch of sample ratio 0.1, that 30,000 training examples for Ant-Goal, 20,000 for Ant- Dir and 6,000 for HalfCheetah Vel. The bottom row is trained on batch of sample ratio 0.8, that is 240,000 training examples for Ant-Goal, 160,000 for Ant-Dir and 48,000 for HalfCheetah
learning curve figures 3, when the sample size is too small, namely 20,000 for Ant-Goal and 6,000 for HalfCheetah Vel, there is some model degeneration happening when training for longer epochs. We can see that although model degeneration does happen for BAIL+, it can often converge to better solution. Moreover, HalfCheetah Vel, even with 48,000 examples, this degeneration still happens for BAIL, while for BAIL+ it does not affect its learning. These results further showcase the benefits BAIL+ compared to BAIL by simply enriching the target batch's 's dataset.
Another observation is that our MBAIL method consistently outperforms BAIL+ across almost all different sample sizes, as shown Table This indicates that the policy distillation process of our MBAIL method does further improve the policy's performance Unlike single BAIL or BAIL+, even with low amount of data, as shown in the Figure 3, the distilled policy does not seem have model degeneration observed for other methods, indicating stable performance under small data regime.
top of the BAIL+ algorithm.
One interesting result is that it seems Ant-Goal environment, none of the BAIL-based methods achieves the desired level of return. This behavior persists across all sample sizes. It appears that BAIL is struggling to obtain meaningful policy for this particular environment. One potential rea- son is that the upper envelope function approximated for this environment is not accurate enough, thus resulting in sub-optimal batches be selected. Further analysis of the failure cases and relative performance of BAIL across conditions is an important question, but one left for future work. De- spite the weakness in all BAIL-driven methods for this one environment, MBAIL still outperforms single task BAIL and BAIL+, further indicating that our approach for multi-batch BAII. works as expected.
CONCLUSION AND FUTURE WORK
Sample efficiency still remains a main obstacle for most reinforcement learning (RL) algorithms be applicable for real-world applications. Batch reinforcement learning has shown to be promising to deal with real-world sequential decision making problems. To further improve the batch RL model's performance with limited amount of training data points and its performance over multiple tasks, in this work, we propose to tackle these problems with sample transfer and policy distillation. Ex- periment results show that the proposed methods (BAIL+ and MBAIL) other baselines when the training data is limited. In the future, we plan to do more evaluation of the proposed methods and investigate how improve the policy distillation method used in this work.
significantly outperform

---

Under review as a conference paper at ICLR 2022
REFERENCES
Rich Caruana. Multitask learning, Machine learning, 28(1):41-75, 1997.
Xinyue Chen, Zijian Zhou, Zheng Wang. Che Wang. Yanqiu Wu, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. arXiv preprint arXi::910.12179, 2019.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160-167, 2008
Carlo D' Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowl- edge in multi-task deep reinforcement learning. In International Conference Learning Repre- sentations, 2019.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. Proceedings of the International Conference on Machine Learning (ICMI), 2019.
Ammar Haydari and Yasin Yilmaz. Deep reinforcement learning for intelligent transportation sys- tems: survey. IEEE Transactions on Intelligent Transportation Systems, 2020.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi Masti-task deep reinforcement learning with popart. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 33, pp. 3796-3803, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6982 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv: 14, 2013.
arXiv preprint
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Proc. Neural Information Processing Systems NeurIPS, Vancouver, Canada, 2020.
Ao Li, Zhiqiang Wu, Huaiyin Lu, Deyun Chen, and Guanglu Sun. Collaborative self-regression method with nonlinear feature based multi-task learning for image classification. IEEE Access, 6:43513-43525, 2018.
Jiachen Li, Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Keith Ross, Henrik Iskov Chris- tensen, and Hao Su. Multi-task batch reinforcement learning with metric learning. arXiv preprint 19XY119909.1133... 2019.
Jayy Li, Hongyan Zhang, and Liangpei Zhang. Efficient superpixel-level multitask joint sparse rep- resentation for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 53(10):5338-535) 2015.
Timothy P Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (509.0050090977.7. 2015.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11500 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning, nature. 518(7540):599-533 2015.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
10

---

Under review as a conference paper at ICLR 2022
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Aivantage-weighted regression: Simple and scalable ooflpo reinforcement learning. arXiv preprint 1rXiv:1910.00017,,222 2019.
Martin Riedmiller. Neural fitted q iteration-first experiences with data efficient neural reinforce- ment learning method. In European conference on machine learning, pp. 317-328. Springer, 2005.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distil- lation. arXiv preprint arXiv:1511.0629. 2015.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676)3354 359, 2017.
Olivier Siohan and David Rybach. Multitask learning and system combination for automatic speech recognition. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 389-595. IEEE, 2015.
R. Sutton and G. Barto. Reinforcement learning: An Introduction. MIT press, Boston, MA, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 JEEF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10.1
Lisa Torrey and Jude Shavlik. Transfer learning. Handbook of research on machine learning applications and trends: algorithms, methods, and techniques, pp.
IGI global, 2010.
Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imita- tion learning for batched historical data. In Neural Information Processing Systems (NeurIPS), Montreal, Canada, 2018.
Ting Yang, Liyuan Zhao, Wei Li, and Albert Y Zomaya. Reinforcement learning in sustainable energy and electric systems: A survey, Annual Reviews in Control, 19:145-1 2020.
Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, and Dawei Yin. Reinforcement learning to optimize long-term user engagement in recommender systems. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- 2810-2818, 2019.


Reviews:
[{"review_id": "RL4vav7NGn2", "reviewer": "Reviewer_u4cA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n\nthe method proposes a good new baseline method for multitask batch reinforcement learning, which is the main justification for my score\n\nThings to improve:\n\n(1) there is a room for improvement of the experimental analysis (see below)\n\n(2) on the novelty, the method heavily relies upon existing ideas (especially MBAIL which combines policy distillation of (Rusu et al, 2015) and triplet loss (Li et al, 2019)), which, in the reviewer’s opinion, there is nothing wrong about and good methods are often a good combination of existing ideas; however it would be helpful if the authors could state explicitly (for example in the introduction), which new contributions have been made in this paper. This would help readers see the particular novelties of this approach and build upon them. \n\nImprovement of the experimental analysis: \n\n(1)While it is shown that the BAIL+ method improves upon BAIL on a set of proposed tasks, it still would not answer whether it would also improve the distilled policy of MBAIL. To answer this question, is it possible to evaluate an ablation of MBAIL with BAIL instead of BAIL+ as the backbone for distillation?\n\n(2)More insight should be given into why and when it works better. In (Li et al, 2019), for example, they additionally experiment with UMazeGoal-M, HumanoidDir-M and HalfCheetahVel, as well as with WalkerParam, exhibiting different transition functions for  each task.The latter would be extremely interesting in the light of the ablation study proposed in point (1), so it might be possible to see the behaviour of different methods with different transition functions.  Is it possible to give results on any of these tasks? \n\nMinor comments:\n(1) Page 3, section BAIL:Best Action Imitation Learning: A minor comment on clarity of writing. For someone browsing through some parts of the section it might look like some parts of BAIL are proposed by the authors (especially given the algorithm below):  \"To do this, we estimate a supremum of the optimal value function V ∗, which is referred to as the upper envelope.”\n\n(2) So it might be a good idea to somehow rephrase the parts of the text in this section referencing to ‘we' to improve clarity (the reviewer understands that it is the background section so it describes the existing work, but nevertheless). \n\n==\nUPDATE: as the comments (mine and other reviewers) have not been addressed yet, there are still open questions which unfortunately do not allow recommending acceptance as things stand at the moment. ", "summary_of_the_paper": "The paper describes multiple task batch reinforcement learning method, building upon a well-known single task batch reinforcement method BAIL. The method proposes two separate improvements, namely BAIL+, introducing the idea of leveraging data across tasks with the same transition function, and MBAIL, policy distillation approach for multiple batch learning.", "summary_of_the_review": "Recommending acceptance subject to convincing experimental analysis improvement and improved description of the novelty of hte approach", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LS-55yT5Iv", "reviewer": "Reviewer_ZN2n", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "$\\textbf{Strengths}$\nThe problem studied by this paper is interesting, and the motivation is clearly expressed. Overall, the method makes sense to me and achieves improvements when there are very few training samples for each batch.\n\n$\\textbf{Weaknesses}$\nThis paper needs lots of polishment in writing, especially for the second half. Some explanations are needed for the KL term in Equation (6), Equation (7) contains many errors and there are many typos in the paper, e.g., for section 3.2, I think your method is named as multi-batch BAIL not multi-batch BAIL+. More descriptions are needed for each figure. \n\nIt looks like most of the components in the proposed algorithms are from previous works, I think the authors should clarify their contributions.\n\nThe experimental results are not convincing, with contextual BCQ almost dominating the Ant-Goal game. I also recommend the authors to add more baselines, e.g., CDS [1] (just for example since this one is quite recent, but authors should consider some SOTA batch RL algorithms).\n\n[1] Conservative Data Sharing for Multi-Task Offline Reinforcement Learning. Yu et al.", "summary_of_the_paper": "This work makes use of transfer learning to deal with the multi-batch reinforcement learning problem. It improves the BAIL algorithm on the multi-batch setting and further extends it to a multi-task model, it empirically demonstrates the advantage in three simulated environments.", "summary_of_the_review": "This paper proposes two algorithms for the multi-batch RL problem, and experimental results demonstrate their advantage to some extent. However, the writing of this paper needs lots of improvements, the technical contribution is limited, and the final results are not solid enough to support the claims, I feel this work is incomplete and recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IERLEiqWj2", "reviewer": "Reviewer_A9qi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper rightfully pointed out that single-batch setting and a large number of interaction samples limited the use cases of offline RL in the real world. The paper provided a viable solution to the multi-task setting with some assumptions to the environment and the data collected. The evolution from BAIL to BAIL+ to MBAIL is clear and the experiments showed clearly that MBAIL is the best out of the three.\n\nThere are several aspects that make me hesitant to recommend an acceptance. \n* Assuming that there will be new SOTA batch RL algorithms in the future, do the authors have plans to make this work more applicable to non-BAIL algorithms? It is unclear from the paper how easy or hard that would be. If it is hard, then the community will benefit less from this piece of work.\n* Section 3.2 is heavily inspired by MBML (Li et al 2019), where the authors made some modifications to make it applicable to an imitation learning based policy. Is it intentional that you do not compare your algorithm with Li et al 2019 in the experiments section?\n* The major technical contribution in BAIL+ is that it proposed a way to use the inductive bias that the transition function remains constant for all tasks and the target task appeared in the batch dataset. But with such assumptions, it also limits its use case to a narrow set of dataset and environments. Clear and strong performance can sometimes justify having a narrow focus, but as the experiment result shows, MBAIL does not consistently perform better than contextual BCQ (albeit the latter has privileged info -- namely the identity of the task).\n\n\nClarification questions: \n* In Figure 3, does the number of epochs mean the same thing for BAIL+ and MBAIL? e.g. for MBAIL I think it is the constant E in algorithm 2. What about BAIL+? It would be great to further clarify.\n* in 3.1, it states “Note that we assume all tasks share the same transition function, and the batch is collected via the same policy”. Is “the batch” here referring to all data or the batch of data per task?\n\nSuggestions:\n* Would it be possible to list out the assumptions more explicitly? I can see several important assumptions spread throughout the paper and it’s hard to keep track of them. e.g. 1. data batches are labeled with their respective tasks; 2. The tasks only differ by their reward function and they all share the same transition function. 3. The target task is known for BAIL+ and is a part of the batch dataset for both BAIL+ and MBAIL. In the meantime, the introduction made it sound like the algorithm is quite general.\n* In the introduction on the second page, you have several sentences that describe how your work differs from previous works and I think the writing there can be improved. E.g. I first read “ In this work, we use sample transfer, ... The proposed algorithm is referred to as BAIL+” and then I read “In this work, ..., we propose to first improve the learning performance on single tasks and then utilize the policy distillation to combine the learned policies into one single policy.” it is confusing because I wasn’t sure whether you use sample transfer, or you use policy distillation, or both. Only later did I realize one refers to BAIL+ and the other refers to MBAIL. \n\nTypos: \n* Section 2 “... (Fujimoto et al., 2019) proposed an algorithm to learn policies with soft constrain to lie near the batch, which alleviate the extrapolation problem.” -> “... (Fujimoto et al., 2019) proposed an algorithm to learn policies with soft **constraints** to lie near the batch, which **alleviates** the extrapolation problem.”\n* Section 2 “and then learning a control policy based on these selection state-action pairs.”-> “these **selected** state-action pairs.”\n", "summary_of_the_paper": "(Note that the rest of the review will use Offline RL and Batch RL interchangeably) \n\nBuilding on top of BAIL (Chen et al. 2019), this paper provides two algorithms for the multi-task offline RL setting. The BAIL+ algorithm assumes that we know the identity of the target task and that all tasks share the same transition function. It uses sample transfer from non-target task data to boost its performance on the target task. The MBAIL algorithm builds on top of BAIL+ and MBML (Li et al 2019), where it distills the BAIL+ policies for each individual task into one master policy that performs decently on all tasks.\nThe authors evaluated the proposed methods on the Ant-Dir, Ant-Goal, and HalfCheetahVel environments from Mujoco. The authors compared MBAIL, BAIL+, BAIL, and contextual BCQ. MBAIL consistently performed the best out of the first three, while contextual BCQ performed better on the Ant-Goal env.\n\n", "summary_of_the_review": "Although the paper showed that MBAIL consistently outperforms BAIL under the multi-task setting, I struggle to recommend acceptance because 1. The algorithm is closely tied to one specific offline RL algorithm, which limits its use cases 2. There could be more baseline comparisons in the experiments section 3. The main technical contribution is specific to tasks where they all share the same transition function, which further limits its use cases. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6r6EM1497Z", "reviewer": "Reviewer_CMTJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "=======strengths=======\n- The first stage of improving on each task before distilling seems to be an interesting novelty. \n- The problem of multi-task batch RL is a direction that is not adequately studied. \n\n=======weaknesses=======\n- the contribution of the paper seems a bit lacking, a large portion of BAIL+ and MBAIL are based on the multi-task batch RL methods described in Li et al., 2019, the first stage in the proposed 2-stage method is probably novel, but I'm not sure if the rest of the technical content are significantly novel. \n- lack of comparison to previous methods, in particularly, while a majority of the work is based on the MBML algorithm and results from Li et al., 2019, the proposed method is not compared to MBML, and the figures use a \"normalized\" yaxis, making them look entirely different from what is in the Li et al. paper, thus it becomes impossible to compare the two methods. \n- lack of reasoning on why the proposed method would work better than other previous methods. This is partly due to lack of comparison to previous methods, but the authors also provided very little theoretical/empirical analysis on why the proposed method would work well. Very little insight is provided. At one point the authors mention the problem of performance drop after training BAIL for a while. This is a very interesting problem but it is not investigated further in the paper. \n- Why is BAIL a good base algorithm? What are the advantages of using it together with your 2-stage scheme, authors should consider refine their writing and emphasize the unique advantages of such a design and give better insight on this design choice. For example, perhaps BAIL is simpler and faster than other methods? If that's the case, provide a comparison on computation complexity and provide discussions.\n- the paper is lacking in clarity of writing, a number of things are unclear, a large num of typos, things are unclear. Authors should go over the paper carefully and fix all the small issues. \n- page 5 top \"we assume all tasks share the same transition function, and the batch is collected via the same policy\", really unclear what \"the same policy\" means? You use the same policy to collect data in different tasks?\n- page 8 Table 1, really not sure what is the difference between \"BAIL (single)\" and the \"baseline single task BAIL\", I assume these are two different things? What is more confusing is that the numbers in the table are mostly negative. \n- Table 1, the difference between some of the reported performance numbers are quite small, but I don't see where you report the number of seed used to get these performance and there is no std reported. How statistically significant are these comparisons? \n- page 4 bottom \"by assuming the correct estimation of...\" why is it that such an assumption is realistic? \n", "summary_of_the_paper": "This paper studies the problem of multi-task offline RL and propose 2 methods, BAIL+ and MBAIL, each is an algorithmic variant based on the BAIL algorithm. The overall goal is to improve performance on multiple tasks. This setting concerns with a number of tasks coming from a MDP distribution where transition function is the same and reward function is different.\n\nThe authors propose a 2-stage method: first use the BAIL+ algorithm to boost the performance of a particular task with data from other tasks. Then, distillation is used to distill policies from these tasks into one policy. ", "summary_of_the_review": "- A major issue here is lack of consistency and adequate comparison with prior work, especially for Li et al., 2019, where the authors proposed and studied reward relabeling, multi-task distillation and use the triplet loss. It becomes a bit unclear how significant the technical contributions. This also affects the contribution on empirical results: it is hard to evaluate their significance without a proper comparison to prior methods. I recommend the authors try to design other analysis or ablations to showcase why the proposed design is a good choice. And to further investigate some of the interesting observations such as the performance drop issue after training BAIL for a long time. \n- Another major issue is the clarify of the paper. Authors should go through the paper slowly and refine the writing, this is more than just minor issues on grammar and typos, the writing in the paper should be very clear and make sense, any claims should be made with supportive evidence, etc. \n- The paper proposes some interesting ideas, and the multi-task batch RL setting is relatively not well explored, but due to the large number of issues, it's hard to believe the paper is ready for publication in its current state.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main identified issues were the limited contribution and use cases, poor writing and missing baseline comparisons and more needed experiments. These issues were not addressed satisfactorily by the rebuttal and hence, I believe the paper should be revised by the authors and undergo another review process at another conference. I therefore recommend rejection.
Decision: Reject