Curriculum Offline Imitating Learning
Authors: Minghuan Liu
Hanye Zhao
Zhengyu Yang
Jian Shen
Weinan Zhang
Li Zhao
Tie-Yan Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
offline reinforcement learning

Abstract: Offline reinforcement learning (RL) tasks require the agent to learn from a pre-collected dataset with no further interactions with the environment. 
Despite the potential to surpass the behavioral policies, RL-based methods are generally impractical due to the training instability and bootstrapping the extrapolation errors, which always require careful hyperparameter tuning via online evaluation. 
In contrast, offline imitation learning (IL) has no such issues since it learns the policy directly without estimating the value function by bootstrapping. 
However, IL is usually limited in the capability of the behavioral policy and tends to learn a mediocre behavior from the dataset collected by the mixture of policies. 
In this paper, we aim to take advantage of IL but mitigate such a drawback. 
Observing that behavior cloning is able to imitate neighboring policies with less data, we propose \textit{Curriculum Offline Imitation Learning (COIL)}, which utilizes an experience picking strategy to make the agent imitate from adaptive neighboring policies with a higher return, and improves the current policy along curriculum stages. 
On continuous control benchmarks, we compare COIL against both imitation-based methods and RL-based methods, showing that COIL not only avoids just learning a mediocre behavior on mixed datasets but is also even competitive with state-of-the-art offline RL methods.

Curriculum Offline Imitating Learning
Minghuan Liu' Hanye Zhao Zhengyu Yang' Jian Shen'
Weinan Zhang"t Li Zhao Tan Liu
Shanghai Jiao Tong University, 2 Microsoft Research
(minghuanliu, fineartz, zyyang, rockyshen, ynzhang]@s1 @sjtu. besjttu.du.cco cn, [lizo,tyliu] www.fff....c.comm
Abstract
Offline reinforcement learning (RL) tasks require the agent learn from pre- collected dataset with no further interactions with the environment. Despite the potential surpass the behavioral policies, RL-based methods are generally im- practical due to the training instability and bootstrapping the extrapolation errors, which always require careful hyperparameter tuning via online evaluation. In contrast, offline imitation learning (IL) has no such issues since it learns the policy directly without estimating the value function by bootstrapping. However, is usually limited in the capability of the behavioral policy and tends to learn a mediocre behavior from the dataset collected by the mixture of policies. In this paper, we aim to take advantage of IL but mitigate such drawback. Observing that behavior cloning able to imitate neighboring policies with less data, we propose Curriculum Offline Imitation Learning (COIL), which utilizes experience pick- ing strategy for imitating from adaptive neighboring policies with a higher return, and improves the current policy along curriculum stages. On continuous control benchmarks, we compare COIL against both imitation- basdd and RL-based meth- ods, showing that it not only avoids just learning mediocre behavior mixed datasets but is also even competitive with taateoffthe-ae offline RL methods.
Introduction
Offline reinforcement learning (RL), or batch RI, aims to learn a well-behaved policy from arbitrary datasets without interacting with the environment. This setting is generally more practical paradigm than online RL since it is expensive or dangerous to interact with the environment in most real-world applications. Typically, two main kinds of offline datasets are considered in previous offline RL works 18]: one contains transitions sampled by a single behavioral policy; the other includes buffer collected by a mixture of policies.
Two main approaches have been deeply investigated for Offline RL. First, RL-based methods, in particular, Q-learning and policy gradient-brr based algorithms [B 7 [2], have the potential to outperform the behavioral policy. However, they always suffer from serious bootstrapping errors and training instability. This shortcoming makes such algorithms impractical to be utilized since too many hyperparameters need to be tuned to achieve a good performance, and it hard to evaluate a suitable model in offline manner, revealed in (18). In contrast, offline imitation learning [1 17 2 specifically, behavior cloning (BC), can always stably learn to perform as the behavioral policy, which may be helpful under behavio datasets. However, BC may fail in learning good behavior under diverse dataset containing mixture of policies (both goods and bads).
Quantity-quality dilemma mixed dataset. As supervised learning technique, BC is not casy to fulfill a desired result, especially on a mixed dataset. Specifically, requires both quantity and quality
"quual contribution. #Correspondigg author. Codes are available htttts:://gtaaiii com/ aporr1/COIIII
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

of the demonstration data, which can hardly be satisfied in offline RL tasks. Directly mimicking the policy from a mixed dataset that contains bad-to-good demonstrations can be regarded as imitation learning from a mediocre behavior policy. To achieve the best performance of the dataset, a naive idea is to imitate the top trajectories ordered by its return. However, such simple strategy will reach the suantity-quality dilemma on the mixed dataset. For example, Fig. 1a illustrates the ordered trajectories on the Walker-finae iiinal-dataset, which contains the whole training experience sampled by an online training agent. Different BC agents are trained by the top 10%, 25%, 50%, and 100% trajectories, but none of them gets rid a mediocre performance, as shown in Fig. lb. Typically, on such dataset, less data owns higher quality but less quantity, and thus cause serious compounding error problems [21, 10 6 on the other hand, more data provides a larger quantity, yet its mean quality becomes worse. In this work. aim to solve such dilemma and exploit the most potential of II. to derive a stable and practical algorithm reaching the best performance of a given dataset.
Our intuition comes from the observation that under RI. scenarios, the agent can imitate neighboring policy with much fewer samples. This observation promotes curriculum solution for the above challenge. Specifically, for mixed datasets, the agent can adaptively imitate the better neighboring policies step by step and finally reach the optimal behavior policy of the dataset.
Our work. We propose Curriculum Offline Imitation Learning (COIL), a simple yet effective offline imitation learning framework for offline RL. At each training iteration, COIL. im- proves the current policy with the data sampled by neighboring policies. To achieve that, COIL utilizes an adap- tive experience picking strategy and a return filter to select proper trajec- tories from the offline dataset for the current level of the agent and thus pro- duces stages of the curriculum. No- tably, COIL stops with a close-to- data-optimal policy without finding the best model under online evaluation. This feature allows to deploy the algorithm in practi- cal problems. In experiments, we show the effectiveness of COIL various kinds of offline datasets. Fig. I offers a quick review of our results: depending solely on BC, COIL can learn from scratch to reach the best performance of the given dataset.
39%
Coll c'inn
25%
4000-
DC 18%
39%
100%
Avente Return
Retum
3000
3000
2000
2000
1000
1000
0 400 800 1200 1600 2000 Trajectory Number
20
40
80
100
Truining Iteration
(a) Ordered trajectories.
(b) Returns of BC v.s. COIL.
Figure 1: Examples of the quality-quantis dilemma for BC. (a) Tra- jectories of the Walker2d-fina dataset ordered by their accumulated return. (b) Performances of behavior cloning (BC) for learning the top 10%, 25%, 50%, and 100% trajectories of the datasct.
Contributions. To summarize, the main technical contributions of this paper are as follows.
We highlight how the discrepancy between the target policy and the initialized policy affects the number of samples required by BC (Section B);
Depending on BC, we propose a practical and effective offline RL algorithm with practical
neighboring experience picking strategy that ends with a good policy (Section 4
We present promising comparison results with comprehensive analysis for our algorithm, which
is competitive to the state-of-ther methods (Section 6
2 Preliminaries
Notations. We consider standard Markov Decision Process (MDP) as tuple M (S, A,, T.Po y). where S is the state space, AA represents the action space, T : S x A x + [0, is the state transition probability distribution, Po S + [0, 1] is the initial state distribution. E [o. 1] is the discounted factor, and r =: S x A -> is the reward function. The goal of [0 that maximizes the expected cumulative dis- counted rewards (or called return) along a trajectory 7: R(T) Σi 7'r,. The dataset D consists of trajectories (TN² that are the collected by a mixture of bad-to-good policies, where a trajectory 7, = ((g) oh. s0 rf). (sf, aj, sft rf).... (sh. a "hy,rhh)) }. and h; is the horizon of Ty. For any dataset, we assume a behavior policy F6 that collects such data and its empir- ical estimation tn can be induced from D as (als) = Eie 1[s "/o'=aa=u)
RL is to find policy T(als) S x A
Eved [|==a]] where I is the indicator function. We further introduce a common used term, occupancy measure, which is

---

defined as the discounted occurrence probability of states or state-action pairs under policy л: P a) = Ei= 7¹ P(st s, a a π)
(( With x x(lss) Z P(s, s/m) (..))--.111
such a definition we can write down that =
= P..
Definition 1. The partial order of different policies is defined as the relative return quantity that policy can achieve when deployed in the environment. Formally, given two policies Ti and T22
T 3 m2 R1 1 R2
Therefore, by definition, in a mixed dataset D that is collected by K different policies T1, the optimal behavior policy IT" can be determined such that for vi E K), ग.
TTK
Curriculum learning. Curriculum learning design and construct a curriculum automatically as sequence of tasks G1y..., Gy train on, such that the efficiency or performance on target task Ge is improved. The expected loss on the jth task is denoted Lj.
3 Empirical Observations and Theoretic Analysis
In this section, we begin with empirical observations that motivate the core idea of our method, followed by the theoretical analysis to support our motivation. Generally, we aim to investigate how the asymptotic performance of BC is affected by the discrepancy between the demonstrated policy and the initialized imitating policy. Previous research shows that BC tends to fail with small number of high-quality demonstrations but can learn well from large-quantity and high-quality data [10, 8]. On the contrary, we find that the requirement of quantity can be highly relaxed as the similarities between the demonstrated policy and the initialized imitating policy increase.
4000
Tejasory
3000
Trjestrics
Average Return
3000
Log Discrepancy
256 Triectoris
2500
Final Average
2000
1500
Randon Truined
1000
500
x
1/3
Return
2/3 Trained
-1.0
-1.2
Random
1/3
2/3
Demo
Initial
256
1024
Random
1/3 Trained 2/3 Trained
Tested Policies
Number Trajectories
Tested Policies
(a) Online training curve.
(b) Final performance of BC.
(c) Empirical discrepancy.
and dashed lines indicate the stage selected policies. (b) Final performances achieved by imitating the demo policy using BC, initialized with different stages of policies. The curves depict the fact that dlos--todeemonstattiin policy can easily imitate the demonstrated policy with fewer samples. (c) Empirical estimation on the discrepancy between the initialized policy and the trained policy outside the support of the demonstrations. Initialized with closcr-to-demo policy always enjoys more minor discrepancy.
Figure 2: (a) Online training curves of SAC agent trained on the Hopper environment, where the
3.1 BC with Different Initialization
To construct the motivating example, we choose Hopper as the testbed, and train an SAC agent until convergence to sample various counts of trajectories the demonstration data. We then take the online-trained policy checkpoints at different training iterations as the initiated policy to train an IL agent. Particularly, the first agent adopts the Random policy imitate the demonstration by BC; the second uses the policy of 1/3 Return and the other two agents start with 1/3 Trained and 2/3 Trained policy separately, terms of training iterations (see Fig. 2ap.
The results are shown in Fig. 26 where we illustrate the average return of each agent given the different number of demonstrations (the exact quantitative results can be found in Appendix E.I). The results show that initialized with a Random policy, the agent can only learn to imitate the demonstrated policy well with large number of samples? in addition, both 1/3 Return and 1/3 Trained policies can achieve sub-optimal performance with fewer samples, where the 1/3
We note that the Random agent can only work well on large datasets with normalized state space; however the other agents learn well upon states.
3

---

Trained one is more efficient. In comparison. the 2/3 Trained agent, which is the closest to the optimal policy, can stably achieve the best performance with all amounts of trajectories.
3.2 Theoretical Analysis
Beyond these observations, we are inquisitive to find a theoretical explanation to support our claim. Standing on the primary result of existing works, we obtain a performance bound of BC with possible solution to handle the quantity-qualing dilemma.
Theorem 1 (Performance bound of BC). Let II be the set of all deterministic policy and [I| |A||S]
Assume that there does not exist policy & II such that (Ts')= = a', Vi E fi,-.., , [PI]. Let To be the empirical behavior policy as well as the corresponding state marginal occupancy is P&c. Suppose BC begins from initial policy no, and define pro similarly. Then, for any 6 >> O, with probability at least 1 8, the following inequality holds:
Drv (s, P)l (s, a)) S c(TO: Tb. (P1)
where c(TO, No. [D] = 2
1 Σ
Pro (s) +
1
Σ
(p=(s)
Pro p+)(5)] +
Σ .ED
P (8) - (+)
2
aED
.ED
intrialization gup
1 Σ (s) - (s) + + 2
1 D
D
log |S| (2/))
log III (2/5)
I
# a'
+
2|D|
2|D)
#ED
data gap
BC pop
The proof can be found in Appendix B.1 Theorem I shows the upper bound of the state-action distribution between the imitating policy т and the behavior policy Th. which consists of three important terms: the initialization gap, the BC gap and the data gap. Specifically, the BC gap arises from the empirical error and the difference between the imitating policy and the empirical behavior policy, which is corresponding to the training procedure. The data gap, however, depends on the number of samples and complexity of the state space, acting as an intrinsic gap due to the dataset and the environment. As for the initialization gap, it is in the form of distance between the state marginal distribution of the initial policy To and behavior policy T out of the dataset. Notice that the second term in Eq. (2) relates the distance between the state marginal of the initial policy wo and the learned policy Ty outside the data support, which is hard to measure theoretically due to the Markov property of the environment dynamics. Therefore, we estimate the empirical discrepancy outside the dataset E Ip (s) Pa., (s)l for this term The results shown in Fig, 2c as expected, suggests that the second term in Eq. ( fact decreases as the initialized policy gets close to the demonstrated policy because of the poor generalization on unseen states, and the error can be further reduced with a larger dataset.
Such analysis brings a possible theoretical explanation to our empirical intuition Section 3.1 Gen- erally, given the same discrepancy c(TO, To. (D)) = C, if the initialized policy narrows down the initialization gap as is close to the demonstrated policy, then the requirement for more samples to minimize the data gap be relaxed. This may seem unreasonable in the learning theory in the traditional supervised learning domain. However, under the RL scenario, the performance of a policy depends on the accumulated reward along the rollout trajectories, which will lead to serious compounding error problems (2 20). Therefore, as the distance between the initialized policy and the demonstrated policy gets closer, the generalization errors of the learned policy can be reduced.
Brief conclusion. Both the experimental and the theoretical results indicate an interesting fact that the asymptotic performance of BC is highly related to the discrepancy between the initialized policy and the demonstrated policy. Specifically, a close-to-demonstration policy can easily imitate the demonstrated policy with fewer samples. On the contrary, when the distance between the initialized policy and the demonstrated policy is far, then successfully mimicking the policy will require much more samples. Such an observation motivates the intuition for proposing our Curriculum Offline Imitation Learning (COIL) in the following literature. The key insight enabling COIL is adaptively imitating the close policies with small number of samples and finally terminates with the optimal behavior policy of the dataset.
For implementation details. see Appendix C.I
4

---

Adaptive experience picking
Interact and sample data
Offline
Dataset
Network
Policy Network
Replay Buffer
Policy Network
Candidate
Trajs
Off-policy training
Imilation learning
(a) Online off-policy training.
(b) Curriculum offline imitation lcarning.
Figure 3: Comparison between online off-policy training and curriculum offline imitation learning.
4
Curriculum Offline Imitation Learning
4.1
Online RL as Imitating Optimal Policies
Before starting to formulate our methodology, we first introduce the formulation of the online RL training. Typically, if we treat an optimization step of the policy as a training stage, an RL online learning algorithm can be realized as on-policy / off-policy depending on whether the agent is trained using the data collected by policies in the previous training stage. Taking off- off-policy RL as an example: beginning with a randomized policy π², at every training stage the agent uses its policy x' to interact with the environment to collect trajectory r' and save it to replay buffer B. The agent then samples several state-action pairs from B and take an optimization step to get policy m++1 towards obtaining the most accumulated rewards:
maximize En [R(T)]
(3)
Under the principle of maximum entropy, we can model the distribution of trajectories sampled by the optimal policy as a Boltzmann distribution 27 as:
P (T) x exp (I((T)
(4)
With such model, trajectories with higher rewards exponentially more preferred. And finding the optimal policy through RL is equivalent to imitating the optimal policy modeled by Eq. 4 3]:
minimize D (P<
where (T( = p(so) Et T(s+ 18s, a, (T (s))s)) the distribution of generating a trajectory т according to policy æ. Thus, n r++1 is updated follows the direction of minimizing the KL divergence:
= x" - (x
4.2 Offline RL as Adaptive Imitation
Compared with online policy training, the offline agent can only have pre-collected dataset for policy training. Such dataset could be generated by a single policy, or collected by kinds of policies. In analogy to online RL, similar solution offline tasks can be imitating the optimal behavior policy from the dataset in an offline way. However, as we show before, offline IL methods, like BC, are only capable of matching the performance of the behavior policy, but hard to reach good performance on the mixed dataset due to the quantity-quali dilemma.
4.2.1 Leverage Behavior Cloning with Curriculum Learning
In Section we have evidence that possible solution to the quantity- dilemma could be adaptive imitation through the offline dataset. An overview of such an adaptive imitation learning diagram compared with online RL is shown in Fig. 3 Formally, with dataset D {T) every training stage i, the agent updates its policy x by adaptively selecting 7 ~ म² from D as the imitating target such that:
n .+1
T² E, [f se
R3 26
where and 8 are small positive numbers that limit the difference between the demonstrated policy # and the learner n, and prevent π from learning poorly behaved policies. Correspondingly, each training
5

---

iteration i creates curriculum automatically such that a task G2 is to imitating the closet demonstrated policy n; with L (RI(P=. (r))(a(())) (T))) and the target task G2 is to imitate the optimal policy o.. Specifically, we aims construct finite sequence T # x1, 7 T², ,.... F T such that n' 3 x where 7' is characterized by its trajectory, and is picked from D based on the current policy ,--,; n' is the imitation result taken 7' as the target policy.
In the following sections, we explain how our algorithm is designed to achieve Eq. 7 that leads the target policy á to finally collapse into the optimal behavior policy #², while solving the quality- quantity dilemma of BC to avoid getting a mediocre result.
4.2.2 Adaptive Experience Picking by Neighboring Policy Assessment
We first provide practical solution evaluate whether E, [Dxz (+[(|s)][((][s)] c. In other words, we want to know whether a trajectory т; E 'D is sampled by a neighboring policy. This can be regarded as finding a policy whose importance sampling ratio is near to 1, and thus a lot of density ratio estimation works can be referred to [16] 26. However, such estimation requires extra costs on training neural networks, and the estimation is inaccurate with fewer data points. Therefore, in this paper, we design a simple yet efficient neighboring policy assessment principle instead that brings the algorithm into practice.
We assume that each trajectory is sampled by single policy. Beyond such a slight and practical as- sumption, let x be the current policy and trajectory Tx {(so, au, Sy. ro). (5%, ag & ra)] collected by an unknown deterministic behavior policy क with exploration noise such that (( [log से [a)]s:)] > log (1 3), where 3 denote the portion of exploration. In this way, we find a practical solution that relaxes the KL--divergence constraint through an observation:
Observation Under the assumption that each trajectory Ts in the dataset 'D collected by an unknown deterministic behavior policy it with an exploration ratio B. The requirement of the KI. divergence constraint E.g (#xx( (*++]))[[TT)) [[((ss))] S e suffices to finding a trajectory that at least B state-action pairs are sampled by the current policy ı with a probability of more than e. such that 6c 2 1/ exp e, i.e.:
(s..))e (((((als) c.)] 11
(8)
The corresponding deviation is shown in Appendix B.2 Therefore, to find whether Ta is sampled by neighboring policy, we calculate the probability of sampling a state s; by π in T for every timestep Ta (T) = (T(aul (o).... ..... Tt (an]s))); where is the horizon of the trajectory. In practice, and 3. we heuristically set / 0.05 an intuitive ratio of exploration. As
instead of fine-tuning for cc, we let the agent choose the value through finding N nearest policies that matches Eq. (8).
4.2.3 Return Filter
We now present how to ensure the second constraint R₂ - R, 2 8, which is designed to refrain the performance from getting worse by imitating to a poorer target than the current level of the imitating policy. In practical offline scenario, it is impossible get the accurate return of the current policy, but we can evaluate its performance based on the current curriculum. To this end, we adopt a return filtering mechanism that filtrates the useless, poor- poha-be trajectories.
In practice, we initialize the return filter V with O, and update the value at each curriculum. Specifi- cally, if we choose {T] from D at iteration k, then V is updated by moving average:
V = (1 - a) V 1 + a min [(())]
where {² is the accumulated reward set of trajectories {r) and a is the moving window determining the filtering rate. Then, the dataset is updated as D = (r E D R(T) 2 V7.
4.2.4 Overall Algorithm
Combining the adaptive experience picking strategy and the return filter, we finally get the simple and practical curriculum offline imitation learning (COIL) algorithm. To be specific, COIL holds an experience pool that contains the candidate trajectories to be selected. Every training time creates a stage of the curriculum where the agent selects appropriate trajectories as the imitation target from the pool and learns them via direct BC. After training, the used experience will be cleaned from the pool, and the return filter also filtrates a set of trajectories. An attractive property of COIL is that it

---

has terminating condition that stops the algorithm automatically with good policy when there is no candidate trajectory to be selected. This makes it easier be applied in real-world applications without further finding the best-learned policy checkpoints under online evaluation as the previous algorithms do. The step-by-step algorithm is shown in Algo. T
It is worth noting that COIL has only two critical hyperparameters, namely, the number of selected trajectories N and the moving window of the return filter a, both of which can be determined by the property of the dataset. Specifically, V is related to the average discrepancy between the sampling policies in the dataset; dataset. In the ablation study Section 6.3 and Appendix E.2 we demonstrate how we select different hyperparameters for different datasets.
influenced by the changes of the return of the trajectories contained in the
5 Related Work
As long-studied branch of RL, numerous solutions have been developed for offline RL with both model-free 77 12 and model-based [25] [ algorithms. We briefly discuss the former from two categories, including RL-based methods and imitation-bant methods.
RL-based methods. A naive thought is to apply off-policy RL algorithms such as SAC [9] and DDPG [1] directly. However, as previous researchers [2] reveal, those online algorithms fail to work due to the severe extrapolation error out-of-distribun
problem. Thence, kinds of specifically designed algorithms have been proposed for offline RL. For instance, BCQ 7 adopts a perturbation model to disturb the action sampled by a BC module to conduct Q-learning on the offline data. actions, over which
BEAR [1] augments constraints on the policy to avoid maximizes the approximate Q function. The current state-of-the-ant algorithm CQL [13] performs strict constraints on the Q-function to learn an expectation lower bound of the true value, avoiding overestimation duitof-distribution data. Typical drawbacks for these RL-based algorithms serious bootstrapping errors and training instability that requires careful hyperparameter tuning with online evaluation. In comparison, imitation- based methods offer candidate mimic the demonstration.
Imitation-based methods. Another inspiration grows from the technique of offline imitation learning on how to learn from the demonstration. Most of these methods take the idea of behavior cloning (BC) [ that utilizes supervised learning to learn act. However, due to the quamtity-qualin dilemma, BC limits mediocre performance on many datasets with mixed samples. Therefore, BAIL (2) constructs the upper-envelope
the value of the data and selects the best actions imitate each state and learn the policy based on BC. The main problem in BAIL is the requirement of regressing the value function with tricks to compute value in the infinite horizon and the hyperparameter threshold on determining the best action. ABM [2), MARWIL 23 and AWR [7 all take the idea of using exponentially weighted version of BC, where the weights determined by different forms of advantage function. These methods also require estimating the value function based on the offline data, which can be unstable and need many hyperparameters to control the learning procedure. Besides, it also hard to tune an appropriate scale for the advantage for imitating the behavior policy. Compared with these methods, our COIL only has few important hyperparameters be tuned without regressing any value function to achieve a stable performance.
6 Experiments
Alongside the simple algorithm, we surprisingly find that COIL not only alleviates the quantity- quality dilemma but also achieves efficient and stable performance against competitive offline methods. Furthermore, we carry out comprehensive analysis of the algorithm behind the phenomenon.
6.1 Offline Learning from Online Learning Experience
We are curious about the performance of COIL on the final buffer datasets containing a complete training experience an online agent 77. because such dataset is mixed with various levels of policies. Therefore direct BC will easily fail as shown in Section [ In our case, we first train an SAC agent from scratch to convergence and save all the training experience from its interactions with the environment, and therefore including exploration actions. Specifically, we conduct our experiments
7

---

5000
10000
3000
4000
8000
Return
2000
Roturn
3000
Return
6000
2000
1000
2000
100 200
300
400
500
400
800 1200 1600 2000
300 600 900 120015001800
Trajectory Index
Trajectory Index
Irajectory Index
(a) Hopper-final.
(b) Walker2d-final.
(c) HalfCheetah-fimal
Figure Trajectories in final datasets, sorted in the online training order.
on three continuous control tasks:
Hopper. Walker2d, and HalfCheetah. Fig. 4 fillustrates trajectories in each dataset, sorted in the collected order.
To show the effectiveness of COIL, we compare it with several strong baselines, including the state- of-the-art offline RL algorithm CQL [3]. and imitation based methods AWR [77] and BAIL 22. For CQL. AWR and BAIL we use their open-source implementation with their default hyperparameter, and for BC we test the same implementation as COIL. The quantitative converged results are listed in Tab. from which we observe that COIL substantially outperforms the other baselines for the final buffer dataset. Also, BAIL and AWR can not always find the optimal behavior due to the difficulty of its hyperparameters tuning and value regression. Specifically, compared with BC that learns mediocre policy, COIL reaches the performance close to the optimal policy.
To further illustrate how our algorithm works, we also record the oracle online order of the trajectory
sampled by the offline agent, as shown in Fig. 5 Notably, COIL keeps a similar training path as the online agent, thanks to the experience picking strategy and the return filter. The corresponding learning curves shown in Fig. [5] are stable and following similar shapes as the ordered datasets. Notably, COIL finally terminates with near-data-ope data--ptimall policy, suggesting nice property that the last offline model can be a great model for deployment, unlike previous algorithms that relies on online evaluation to select the best checkpoint.
Table Average performances on final datasets, the means and standard deviations are calculated over random seeds. Behavior shows the average performance of the behavior policy that collects the data.
Expert (SAC)
Behavior
AWR
BAIL
COIL (Ours)
happar-final
3163.3 (44.4)
974.5
1480 (80012)
1609.7 (489.7)
2296 9 (915.9)
501 5 (227.5)
2872.5 (133.9)
valleor2d-r1mm rinal
4866.03 (68.6)
2684.9
2099.6 (2101.3)
3213.8 (1682.9)
4236. (1531.)
2604.3 (1937.6)
4391.3 (697.8)
(113.)
712244
(3910.9)
(1153.4)
9745.000800.3
10882.0 (1042.7)
9328.5 (1940.6)
2500
2400
4000
9000
Averuge Ketum
Averuge Renum
2000
3200
8000
1600
Atrengee
2400
7000
1200
800
1600
5000
60
12
24
Training Iseration
Training Iteration
Triining Eeoration
Selected Trajectory Orders
500
Selected Iijector Orders
2000
Selected Trajectory Orders
ScakO
1500
Soad-1
8500
Seak-
Sead-
300
1000
1000
200
500
0
60
20
60
100
12
24
48
Training Temation
Training Ttenttion
Trininng
Dictation
(a) Hopper-final.
(b) Walker2d-final
(c) HalfCheetah-fima
Figure 5: Training curves and orders of selected trajectories.
6.2 D4RL Benchmark
To further show the power of COIL, we conduct comparison experiments on a common-used D4RL
benchmark [5] Tab. 2 For BC we include the BC results reported Fu et [5] (denoted as BC
(D4RL)) and our implementation of BC (denoted BC (Ours)). Besides, we compare BAIL (2).
8

---

MOPO [25]] and the ofte--of-the-rr results reported in Fu et al. [3] (denoted as SoTA). As our expected, BC is able to approach or outperform the performance of the behavior policy on the datasets generated from single policy, marked as "random", and "medium", but still remains a gap between the optimal behavior policy (Best 1% column). As a comparison, COIL achieves the performance of the optimal behavior policy on most datasets. As is noticed that doing so will allow COIL to beat or compete with the state-of-the-art results. To our surprise, on the halfcheetah domain, previous RL-based offline algorithms have the potential to surpass the optimal behavior policy (although with careful hyperparameter tuning), showing the advantage of RL-based learning. worth noting that for model-based algorithm like MOPO, it behaves well the medium-rege replaay datasets due the sufficient data to learn a good environment model; but can hardly outperform SoTA model-free results other datasets.
Table 2: Average performance on D4RL datasets. Results in gray columns is our implementation that are tested
among 5 random seeds. The other results are based on numbers reported in D4RL among random seeds without standard deviations. Best 1% shows the average return of the top 1% best trajectories, representing the performance of the optimal behavior policy; Behavior shows the average performance of the dataset.
Expert (BeRI.)
Behasior
Best 1%
(C4R))
BC (Oamo
con (Dam))
BATI.
MOPO
SoTA (MMRI)
heopes candoo
3234.3
330.1 (3.5)
215
hogger-aediin
3234,3
466.9
853.6 (397.5)
1353.7 (271.))
(19.5
3009.6
1227.3
hopper acdius
3234,3
1846.3
3621.2
3527.4
(165,5
(1265.)
1682.0
3558.5
walkerze randon walker2d zedium
4592.3
1.1
25.0
73.0
(5933
320.5 (70.7)
130.5(88.2)
997.1
336,3
4592.3
4964
3515,5
304.8
1521.9 (1351.3)
2184,5(1279 2)
1242 (1545.7)
643.0
3725
un weikkee----dd------ reegllyy
4591.
156.6
1593.7
S1866
715.0 (408 S)
1490-01347.0 20.0
532. (380.))
1961.1
1227.3
4592.5
1059.7
5133.4
207.0
3488 (815.8)
4012.3:1463.0)
36399 (183.7)
2526.0
3099.3
hat fehwetah-rrndde sediun
12135.0
-101.5
-17.9
-134 900069
-0
(40.7)
411488
12135.0
4196.4
315007)
(243.)
(177:6615499)
5473.8
chhechot mediun roplay
12138.0
4812,0 (1-77))
(966.3)
uo-espert
9470.3
7184.7
7750.8
6.3 Ablation Study
hopper- medium-replay
hopper-mediie
The ablation study aims to illustrate how hyper- parameters can be determined according to the property of the offline datasets without online evaluation. In particular, COIL has two critical hyperparameters: a and N. Since a determines the rate of filtering out the bad trajectories the agent imitates better policy where small a high filtering rate, it is highly influenced by the changes in returns of the trajec- tories in the dataset. Taking hopper-mede replay and hopper- for examples, the former consists of online training experience, and the return of trajectories changes rapidly; on the contrary, most of the trajectories in the latter
1020
250-
500
1000
250-
400
800
1200
0
1000
2000
3000
corresponds
Trajectrr Index
Trueetty Iodex
alpha-o..u 40
30700
alpba
alpha-3.50
Averuge
2000
are at the same return level. Therefore, intu- itively, small value of a should be assigned to hopper-med wtm---mmmmdmr-eepeel----- replay while large value to hopper- mm, um, as confirmed by the exper- imental results shown in Fig. [6] Ablation ex- are illustrated in Appendix saying that it can also be tuned according to the characterization of the dataset.
bogest-oddiun-eplay
Leeper-sclllu
Danse
Figure 6: Returss of trajectories in Sopper-mediie replay and hopper-medium, and final performances of COIL. with different
periments on the other hyperparameter
6.4 Compared with Naive Curriculum Strategies
In this section, we propose comparing the curriculum strategy of COIL with naive curriculum strategies to determine whether COIL is better choice. The first strategy is called return-ordered BC, which conducts curriculum imitation learning based only on returns the trajectories. Similar to COIL, it picks N NABC trajectories with the lowest returns for each curriculum to perform behavioral cloning, and then removes them from the dataset. When the dataset empty, the algorithm stops. The other strategy is called buffer-shrinking BC. As its name suggests, it conducts curriculum imitation learning by shrinking the dataset ordered by the return after training at each curriculum. detail, nuffer-shrinkie BC begin its training with the entire datasct in the buffer; after a fixed number of

---

gradient steps, it shrinks the buffer by discarding p% of trajectories with the lowest returns; then the training is continued on the remaining trajectories. our experiment, we choose p 20 that the algorithm will stop after 5 times of shrinkage.
The learning results final datasets shown in Fig. 7 As expected, retur-ordered BC leads to mediocre behaviors since similar returns do not always account for similar policies due to the exploration noise. addition, buffer-shrinkknng BC not usually stable achieve the optimal behavior. Since the shrink strategy is totally hand-crafted, the quantity-qulli dilemma is not eliminated. The mediocre trajectories in the last curriculum will lead to the failure. On the contrary. COIL succeeds in the optimal behavior policy with the highest training efficiency (the least gradient steps), indicating the advantage of the policy-distee siiioy-dittanee-iiie -aseedbadddeeee curriculum.
2800
9600
2400
4000
8400
Avenge Ketum
2000
BUC.
3200
Retum
7200
6000
1600
Avenge
2400
4800
1200
1600
3600
800
2400
400
800
1200
0-
6000
12000 18000 24000
50000 100000 1000015000 500000000000
0000000000030000000000
Gradicat Steps
Gradicnt Steps
Steps
(a) Hopper-final.
(b) Walker2d-final.
(c) HalfCheetah-final
Figure 7: Comparison of training curves between COIL and Return-ordered BC (R BC) and Buffer-shrinking
BC (B BC) on final datasets with the same batch size. Different strategies terminate with diffcrent gradient steps.
7 Conclusion
In this paper, analyze the quantity-quality dilemma of behavior cloning (BC) from both an experimental and theoretical point of view, which motivates us to propose the curriculum offline imitation learning (COIL). COIL takes advantage of imitation learning by improving the current policy with adaptive neighboring policies. Experiments show good properties of COIL with competitive evaluation results against state-of-the-art offline RI. algorithms. COIL. may provide practical way for bringing offline RI. into practice due to its simplicity and effectiveness, but it also limits into the performance of the dataset.
Acknowledgements
The corresponding author Weinan Zhang is supported by "New Generation of AI 2030" Ma- jor Project (201 AAAA01009),), Shanghai Municipal Science and Technology Major Project (201ISHZDZZX0102)) and National Natural Science Foundation of China (62076161, 61632017), The author Minghuan Liu is supported by Wu Wen Jun Honorary Doctoral Scholarship. Al Institute, Shanghai Jiao Tong University. We sincerely thank the reviewers for helpful feedback.
References
[1] Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelli- gence 15, pages 103-129, 1995.
[2] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. BAIL: Best-action imitation learning for batch deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 18353-18363, 2020.
[3] Chelsea Finn. Paul Christiano, Pieter Abbeel, and Sergey Levine. connection between generative adversarial networks, inverse reinforcement learning, and energy-based models, In NeurlPS 20/6 Workshop on Adversarial Training, 2016.
[4] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. Proceedings of The 33rd International Conference on Machine Learning, volume 48, pages 49-58. PMLR, 2016.
10

---

[5] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint PrXiv:2004.07219, 2020.
[6] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
[7] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference Machine Learning, volume 97, pages 2052-2062. PMLR, 2019.
[8] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimiza- tion perspective on imitation learning methods. In Proceedings of the Conference on Robot Learning, volume 100, pages 1259-1277. PMIR, 2020.
[9] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 1861-1870. PMIR, 2018.
[10] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, volume 29, pages 4565-4573, 2016.
[11] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model-based offline reinforcement learning. In Advances Neural Information Processing Systems, volume 33, pages 10-0--812, 2020.
[12] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, volume 32, pages 11744-1179, 2019.
[13] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 1179-1191, 2020.
[14] Timothy P. Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[15] Leland McInnes, John Healy, and James Melville. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint
2018.
[16] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DuaIDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems, volume 32, pages 2318-2328, 2019.
[17] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. ddvantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 2019.
[18] Rongjun Qin, Songyi Gao, Xingyuan Zhang, Zhen Xu, Shengkai Huang, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: near real-world benchmark for offline reinforcement learning. arXiv preprini 102.0000,,, 2021.
[19] Murray Rosenblatt. Remarks on some nonparametric estimates of a density function. The Annals of Mathematical Statisties, 27(3):832 837, 1956.
[20] Stephane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9, pages 661-668. PMLR, 2010.
[21] Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15, pages 627-635. PMLR, 2011.
[22] Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference Learning Representations, 2020.
[23] Qing Wang, Jiechao Xiong, Lei Han, peng sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. In Advances Neural Information Processing Systems, volume 31, pages 629 2291-60 1-6000, 2018.

---

[24] Tian Xu, Ziniu Li, and Yang Yu. On value discrepancy of imitation learning. arXiv preprint vrXiv:1911.0702% 2019.
[25] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, volume 33, pages 14129-14142, 2020.
[26] Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. GenDICE: Generalized offline estimation of stationary values. In International Conference on Learning Representations, 2020.
[27] Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence, volume 3, page 1433-1438. AAAI Press, 2008
12


Reviews:
[{"review_id": "Nm5dAygtZB7", "reviewer": "Reviewer_8Yop", "summary": "This paper proposed and interesting algorithm to do imitation learning with an automatic curriculum learning technique. In particular an automatic filtering of better trajectories under the current policy  and a selection of the best ones in that set.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\n\nOriginality:  The idea is original, even though it sounds like BAIL for trajectories. But the math has been done well and it is well supported.\n\nQuality: Claims are supported. I believe some sections / paragraphs would benefit for more clarity. Also it would greatly benefit from more extensive experimentation. Adding more environments.\n\nClarity: The submission is well organized but incomplete. The notation seems porr and writing need clarity, The figures would benefit from extra information some\n\nSignificance: The results are better than SOTA. Event thought in D4RL is not compara with CQL and BAIL, and more environments would benefit the claims.\n\nNotes:\nConfused by notation in line 91, unclear how the sum of samples belonged to the BC policy corresponds to a well formed policy.\nLine 95, notation is not clear, D was defined as having trajectories but here it is used as having policies.\nLine 97, re-used D for indication of tasks.\nLine 159, Clarity \"the close policy\".\nLine 181. \"kinds of policies\" grammar.\nLine 195. Poor notation. \"The target Dt\" .\nFigure 5 ( show comparison) needs to improve the graph, the seeds graphs as points are confusing.\nWould benefit from adding D4RL don't compare with BAIL and CQL [ or even model based offline rl)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Its seems not readily applicable, but the authors could have written about the benefits of doing Offline RL in the sense of the impact of training. Or a computational analysis could have interesting, in the sense of cost of training, CO2 impact, cost, etc.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Djz_0_TUuMC", "reviewer": "Reviewer_PdUi", "summary": "This paper applied Online Imitation Learning to offline RL. The central idea is, at each iteration of training, the proposed approach will sample the data points from the neighboring policies based on the experience picking. The intuition is built based on the observation/analysis of the discrepancy between demo policy and initialized imitating policy.  I like how authors scrutinized their observations with theoretical analysis. The neighboring policy picking targets at solving the \"dilemma\" of quantity-quality. However, the main technical contribution is not super very in-depth, authors tackle the KL-divergence constraints through the assumption that each trajectory is sampled by a single policy, which I am not sure whether it is very practical as claimed by authors (yield good bound and performance as illustrated in experimental studies).", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- Paper is very well-written and easy to follow. The authors moved almost all proof to the appendix and made the paper accessible.\n- The central ideas are motivated by real-world observations and theoretical analysis\n\nWeakness:\n- The experimental evaluation could be done with more complex datasets\n- The technical depth: The evaluation of the two policies is based on relaxing the trajectories sampled by a single policy, not sure whether the assumption is validated.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "CTQiwJjQNSR", "reviewer": "Reviewer_FWCF", "summary": "The method proposes an algorithm for Offline Reinforcement Learning. The work starts from the idea that behavioral cloning struggles at learning from data coming from several policies but, on the contrary, easily learns a behavior policy if initialized with a “close” policy.\nA curriculum is thus built for behavioral cloning to efficiently learn the “best policy shown in the dataset”. For this, a behavioral cloning is trained on a sequence of trajectories defined the following way: the next trajectory selected for training is the most probable according to current policy. The trajectories-dataset is then filtered by saying that the return should be above a threshold given by a running average of the trajectories seen so far.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " \nI appreciate the simplicity of the method, although some explanations could, in my opinion, be simplified. I have questions regarding the experiments although the results are quite convincing.\n\nRemarks:\n- 3.1: It would have been more informative to take the ⅓ (resp. ⅔ ) policy at ⅓ of the final performance rather than ⅓ of the training steps. The initializations are quite close in terms of return so I find it quite normal that such a good inits lead to great final performance.\n\nDerivation of the algorithm:\n\n- I find the derivation of the algorithm a bit confusing. First, there are some mistakes in the mathematical notations that make reading harder (e.g. l.213, no consistency between $(s_t, a_t)$ and $(s, a)$) or in the Eq.7 where there are confusion between $\\pi_i$ and $\\pi$, as well as between $\\hat{\\pi}_i$ and  $\\hat{\\pi}$.\n\n- Proposition 1) is not rigorous. “Corresponds to” does not mean anything mathematically. The proof in B.2 uses implications rather than equivalences. So you proved that a trajectory satisfying the \"KL condition\" also satisfies the “probability condition”. However you are using the converse property.\n\n- However, the final way of defining a “close policy” is simple and makes sense. I would advise either cutting off the KL based argumentation and go directly with the heuristic or refine the mathematical derivations.\n\n- Even though the return filter is a very simple idea, I very much appreciate the fact that it brings a natural termination condition to the algorithm. I would like to see an ablation without the return filter at least in the appendix.\n\nExperiments\n- I argue that results of Table 1 should not appear as is. Indeed, when introducing a new dataset, it is unfair for baselines to use “default hyperparameters”. If you introduce a new dataset, you need to sweep the hyperparameters of the baselines at least as much as the ones of your own algorithm (or at the very least the learning rate...). You also need to specify how hyperparameter selection was performed. For your information, [1] discussed how to select HPs in offline RL and [2] discussed other ways to perform HP selection for imitation.\n\n- Otherwise, the results are quite well presented and convincing.\nBaselines in appendix E.2 are very natural and I would recommend adding it to the core of the paper if you have the space to do so. Can you however explain the weird x-axis with algorithms not all starting at step 0?\n\n-Ablation E.3 is also very insightful. It is very helpful to assist potential users in finding good hyperparameters.\n\nIt is overall a good paper. I would be ready to increase my score if authors clarify the algorithm mathematical derivation and either solve the experimental issues I mentioned or debunk my arguments.\n\n[1] Paine, Tom Le, et al. \"Hyperparameter selection for offline reinforcement learning.\" arXiv preprint arXiv:2007.09055 (2020).\n[2] Hussenot, Leonard, et al. \"Hyperparameter Selection for Imitation Learning.\" arXiv preprint arXiv:2105.12034 (2021).\n\n\n\n\n-----------    UPDATE AFTER AUTHOR RESPONSE  -------------\n\nThanks for the very detailed answers!\n\nQ1: Thanks for adding this additional datapoint. I am glad to see that this 1/3 performance point also confirms your hypothesis.\n\nQ3: It is a shame we cannot update the pdf, but I am happy the authors will now present the KL derivations as a relaxation rather than an equivalence. This will make the paper more rigorous and will not diminish the impact of its results.\n\nQ4: Thank you for adding this ablation. It looks confusing in the authors response (typo in the reported numbers?) but the figure that was shared is clear and supporting the importance of the return filter.\n\nQ5: Thanks for the additional efforts tuning the baselines. Please include these new results in the main paper.\n\nQ7: I would recommend aligning the beginning of the training rather than the end... otherwise the x-axis is just.... wrong.\n\nGiven the additional work, including results and clarifications, provided by the authors, I increased my score to 7.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No direct societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper generated an interesting discussion between the reviewers and the authors. Especially, the authors made a very good job at addressing reviewers' concerns. They ran new experiments, finetuned the baselines and conducted a thorough ablation study. The paper is now considered as presenting a SotA algorithm for Offline imitation learning.
Decision: Accept (Poster)