Curriculum Offline Imitating Learning

Abstract:
Offline reinforcement learning (RL) tasks require the agent to learn from a precollected dataset with no further interactions with the environment. Despite the potential to surpass the behavioral policies, RL-based methods are generally impractical due to the training instability and bootstrapping the extrapolation errors, which always require careful hyperparameter tuning via online evaluation. In contrast, offline imitation learning (IL) has no such issues since it learns the policy directly without estimating the value function by bootstrapping. However, IL is usually limited in the capability of the behavioral policy and tends to learn a mediocre behavior from the dataset collected by the mixture of policies. In this paper, we aim to take advantage of IL but mitigate such a drawback. Observing that behavior cloning is able to imitate neighboring policies with less data, we propose Curriculum Offline Imitation Learning (COIL), which utilizes an experience picking strategy for imitating from adaptive neighboring policies with a higher return, and improves the current policy along curriculum stages. On continuous control benchmarks, we compare COIL against both imitation-based and RL-based methods, showing that it not only avoids just learning a mediocre behavior on mixed datasets but is also even competitive with state-of-the-art offline RL methods.

heading: 1 Introduction
text: Offline reinforcement learning (RL), or batch RL, aims to learn a well-behaved policy from arbitrary datasets without interacting with the environment. This setting is generally a more practical paradigm than online RL since it is expensive or dangerous to interact with the environment in most real-world applications. Typically, two main kinds of offline datasets are considered in previous offline RL works [5, 18]: one contains transitions sampled by a single behavioral policy; the other includes a buffer collected by a mixture of policies.
Two main approaches have been deeply investigated for Offline RL. First, RL-based methods, in particular, Q-learning and policy gradient-based algorithms [13, 7, 12], have the potential to outperform the behavioral policy. However, they always suffer from serious bootstrapping errors and training instability. This shortcoming makes such algorithms impractical to be utilized since too many hyperparameters need to be tuned to achieve a good performance, and it is hard to evaluate a suitable model in an offline manner, as revealed in [18]. In contrast, offline imitation learning [1, 17, 2], specifically, behavior cloning (BC), can always stably learn to perform as the behavioral policy, which may be helpful under single-behavior datasets. However, BC may fail in learning a good behavior under a diverse dataset containing a mixture of policies (both goods and bads).
Quantity-quality dilemma on mixed dataset. As a supervised learning technique, BC is not easy to fulfill a desired result, especially on a mixed dataset. Specifically, it requires both quantity and quality ∗Equal contribution. †Corresponding author. Codes are available at https://github.com/apexrl/COIL.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
of the demonstration data, which can hardly be satisfied in offline RL tasks. Directly mimicking the policy from a mixed dataset that contains bad-to-good demonstrations can be regarded as imitation learning from a mediocre behavior policy. To achieve the best performance of the dataset, a naive idea is to imitate the top trajectories ordered by its return. However, such a simple strategy will reach the quantity-quality dilemma on the mixed dataset. For example, Fig. 1a illustrates the ordered trajectories on the Walker-final-dataset, which contains the whole training experience sampled by an online training agent. Different BC agents are trained by the top 10%, 25%, 50%, and 100% trajectories, but none of them gets rid of a mediocre performance, as shown in Fig. 1b. Typically, on such dataset, less data owns higher quality but less quantity, and thus cause serious compounding error problems [21, 10, 6]; on the other hand, more data provides a larger quantity, yet its mean quality becomes worse. In this work, we aim to solve such a dilemma and exploit the most potential of IL to derive a stable and practical algorithm reaching the best performance of a given dataset.
Our intuition comes from the observation that under RL scenarios, the agent can imitate a neighboring policy with much fewer samples. This observation promotes a curriculum solution for the above challenge. Specifically, for mixed datasets, the agent can adaptively imitate the better neighboring policies step by step and finally reach the optimal behavior policy of the dataset.
Our work. We propose Curriculum Offline Imitation Learning (COIL), a simple yet effective offline imitation learning framework for offline RL. At each training iteration, COIL improves the current policy with the data sampled by neighboring policies. To achieve that, COIL utilizes an adaptive experience picking strategy and a return filter to select proper trajectories from the offline dataset for the current level of the agent and thus produces stages of the curriculum. Notably, COIL stops with a close-todata-optimal policy without finding
the best model under online evaluation. This feature allows to deploy the algorithm in practical problems. In experiments, we show the effectiveness of COIL on various kinds of offline datasets. Fig. 1 offers a quick review of our results: depending solely on BC, COIL can learn from scratch to reach the best performance of the given dataset.
Contributions. To summarize, the main technical contributions of this paper are as follows. • We highlight how the discrepancy between the target policy and the initialized policy affects the
number of samples required by BC (Section 3); • Depending on BC, we propose a practical and effective offline RL algorithm with a practical
neighboring experience picking strategy that ends with a good policy (Section 4); • We present promising comparison results with comprehensive analysis for our algorithm, which
is competitive to the state-of-the-art methods (Section 6).

heading: 2 Preliminaries
text: Notations. We consider a standard Markov Decision Process (MDP) as a tuple M = 〈S,A, T , ρ0, r, γ〉, where S is the state space,A represents the action space, T : S ×A×S → [0, 1] is the state transition probability distribution, ρ0 : S → [0, 1] is the initial state distribution, γ ∈ [0, 1] is the discounted factor, and r : S × A → R is the reward function. The goal of RL is to find a policy π(a|s) : S × A → [0, 1] that maximizes the expected cumulative discounted rewards (or called return) along a trajectory τ : R(τ) = ∑T t=0 γ
trt. The dataset D consists of trajectories {τN1 } that are the collected by a mixture of bad-to-good policies, where a trajectory τi = {(si0, ai0, s′i0 , ri0), (si1, ai1, s′i1 , ri1), · · · , (sihi , a i hi , s′ihi , r i hi
)}, and hi is the horizon of τi. For any dataset, we assume a behavior policy πb that collects such data and its empir-
ical estimation π̂b can be induced from D as π̂b(a|s) = ∑ (s′,a′)∈D I[s ′=s,a′=a]∑
s′∈D I[s′=s] , where I is the
indicator function. We further introduce a common used term, occupancy measure, which is
defined as the discounted occurrence probability of states or state-action pairs under policy π: ρπ(s, a) = ∑∞ t=0 γ tP (st = s, at = a|π) = π(a|s) ∑∞ t=0 γ
tP (st = s|π) = π(a|s)ρπ(s). With such a definition we can write down that Eπ[·] = ∑ s,a ρπ(s, a)[·] = E(s,a)∼ρπ [·].
Definition 1. The partial order of different policies is defined as the relative return quantity that a policy can achieve when deployed in the environment. Formally, given two policies π1 and π2:
π1 π2 ⇔ R1 R2 (1)
Therefore, by definition, in a mixed dataset D that is collected by K different policies π1, · · · , πK , the optimal behavior policy π∗ can be determined such that for ∀i ∈ [1,K], πi π∗.
Curriculum learning. Curriculum learning design and construct a curriculum automatically as a sequence of tasks G1, . . . , GN to train on, such that the efficiency or performance on a target task Gt is improved. The expected loss on the jth task is denoted Lj .

heading: 3 Empirical Observations and Theoretic Analysis
text: In this section, we begin with empirical observations that motivate the core idea of our method, followed by the theoretical analysis to support our motivation. Generally, we aim to investigate how the asymptotic performance of BC is affected by the discrepancy between the demonstrated policy and the initialized imitating policy. Previous research shows that BC tends to fail with a small number of high-quality demonstrations but can learn well from large-quantity and high-quality data [10, 8]. On the contrary, we find that the requirement of quantity can be highly relaxed as the similarities between the demonstrated policy and the initialized imitating policy increase.

heading: 3.1 BC with Different Initialization
text: To construct the motivating example, we choose Hopper as the testbed, and train an SAC agent until convergence to sample various counts of trajectories as the demonstration data. We then take the online-trained policy checkpoints at different training iterations as the initiated policy to train an IL agent. Particularly, the first agent adopts the Random policy to imitate the demonstration by BC; the second uses the policy of 1/3 Return and the other two agents start with 1/3 Trained and 2/3 Trained policy separately, in terms of training iterations (see Fig. 2a).
The results are shown in Fig. 2b, where we illustrate the average return of each agent given the different number of demonstrations (the exact quantitative results can be found in Appendix E.1). The results show that initialized with a Random policy, the agent can only learn to imitate the demonstrated policy well with a large number of samples2; in addition, both 1/3 Return and 1/3 Trained policies can achieve a sub-optimal performance with fewer samples, where the 1/3
2We note that the Random agent can only work well on large datasets with normalized state space; however the other agents learn well upon raw states.
Trained one is more efficient. In comparison, the 2/3 Trained agent, which is the closest to the optimal policy, can stably achieve the best performance with all amounts of trajectories.

heading: 3.2 Theoretical Analysis
text: Beyond these observations, we are inquisitive to find a theoretical explanation to support our claim. Standing on the primary result of existing works, we obtain a performance bound of BC with a possible solution to handle the quantity-quality dilemma.
Theorem 1 (Performance bound of BC). Let Π be the set of all deterministic policy and |Π| = |A||S|. Assume that there does not exist a policy π ∈ Π such that π(si) = ai,∀i ∈ {1, · · · , |D|}. Let π̂b be the empirical behavior policy as well as the corresponding state marginal occupancy is ρπ̂b . Suppose BC begins from initial policy π0, and define ρπ0 similarly. Then, for any δ > 0, with probability at least 1− δ, the following inequality holds:
DTV(ρπ(s, a)‖ρπb(s, a)) ≤ c(π0, πb, |D|)
where c(π0, πb, |D|) = 1
2 ∑ s/∈D ρπb(s) + 1 2 ∑ s/∈D |ρπ(s)− ρπ0(s)|+ 1 2 ∑ s/∈D
|ρπ0(s)− ρπb(s)|︸ ︷︷ ︸ initialization gap
+ 1
2 ∑ s∈D |ρπ(s)− ρπ̂b(s)|+ 1 |D| |D|∑ i=1 I [ π(si) 6= ai ] ︸ ︷︷ ︸
BC gap
+
[ log |S|+ log(2/δ)
2|D|
] 1 2
+
[ log |Π|+ log(2/δ)
2|D|
] 1 2
︸ ︷︷ ︸ data gap
(2)
The proof can be found in Appendix B.1. Theorem 1 shows the upper bound of the state-action distribution between the imitating policy π and the behavior policy πb, which consists of three important terms: the initialization gap, the BC gap and the data gap. Specifically, the BC gap arises from the empirical error and the difference between the imitating policy and the empirical behavior policy, which is corresponding to the training procedure. The data gap, however, depends on the number of samples and complexity of the state space, acting as an intrinsic gap due to the dataset and the environment. As for the initialization gap, it is in the form of distance between the state marginal distribution of the initial policy π0 and behavior policy πb out of the dataset. Notice that the second term in Eq. (2) relates to the distance between the state marginal of the initial policy π0 and the learned policy πb outside the data support, which is hard to measure theoretically due to the Markov property of the environment dynamics. Therefore, we estimate the empirical discrepancy outside the dataset 12 ∑ s/∈D |ρ̂π(s)− ρ̂π0(s)| for this term3. The results shown in Fig. 2c, as expected, suggests that the second term in Eq. (2) in fact decreases as the initialized policy gets close to the demonstrated policy because of the poor generalization on unseen states, and the error can be further reduced with a larger dataset.
Such analysis brings a possible theoretical explanation to our empirical intuition Section 3.1. Generally, given the same discrepancy c(π0, πb, |D|) = C, if the initialized policy narrows down the initialization gap as is close to the demonstrated policy, then the requirement for more samples to minimize the data gap can be relaxed. This may seem unreasonable in the learning theory in the traditional supervised learning domain. However, under the RL scenario, the performance of a policy depends on the accumulated reward along the rollout trajectories, which will lead to serious compounding error problems [21, 20]. Therefore, as the distance between the initialized policy and the demonstrated policy gets closer, the generalization errors of the learned policy can be reduced.
Brief conclusion. Both the experimental and the theoretical results indicate an interesting fact that the asymptotic performance of BC is highly related to the discrepancy between the initialized policy and the demonstrated policy. Specifically, a close-to-demonstration policy can easily imitate the demonstrated policy with fewer samples. On the contrary, when the distance between the initialized policy and the demonstrated policy is far, then successfully mimicking the policy will require much more samples. Such an observation motivates the intuition for proposing our Curriculum Offline Imitation Learning (COIL) in the following literature. The key insight enabling COIL is adaptively imitating the close policies with a small number of samples and finally terminates with the optimal behavior policy of the dataset.
3For implementation details, see Appendix C.1.

heading: 4 Curriculum Offline Imitation Learning

heading: 4.1 Online RL as Imitating Optimal Policies
text: Before starting to formulate our methodology, we first introduce the formulation of the online RL training. Typically, if we treat an optimization step of the policy as a training stage, an RL online learning algorithm can be realized as on-policy / off-policy depending on whether the agent is trained using the data collected by policies in the previous training stage. Taking off-policy RL as an example: beginning with a randomized policy π0, at every training stage i, the agent uses its policy πi to interact with the environment to collect trajectory τ i and save it to replay buffer B. The agent then samples several state-action pairs from B and take an optimization step to get policy πi+1, towards obtaining the most accumulated rewards:
maximize π Eτ∼π[R(τ)] . (3)
Under the principle of maximum entropy, we can model the distribution of trajectories sampled by the optimal policy as a Boltzmann distribution [27, 4] as:
P ∗(τ) ∝ exp (R(τ)) (4) With such a model, trajectories with higher rewards are exponentially more preferred. And finding the optimal policy through RL is equivalent to imitating the optimal policy modeled by Eq. (4) [4, 3]:
minimize π
DKL(Pπ(τ)‖P ∗(τ)) , (5)
where Pπ(τ) = ρ(s0) ∑T t=0 T (st+1|st, at)π(at|st) is the distribution of generating a trajectory τ according to policy π. Thus, πi+1 is updated follows the direction of minimizing the KL divergence:
πi+1 = πi −∇πDKL(Pπ(τ)‖P ∗(τ)) (6)

heading: 4.2 Offline RL as Adaptive Imitation
text: Compared with online policy training, the offline agent can only have a pre-collected dataset for policy training. Such dataset could be generated by a single policy, or collected by kinds of policies. In analogy to online RL, a similar solution on offline tasks can be imitating the optimal behavior policy from the dataset in an offline way. However, as we show before, offline IL methods, like BC, are only capable of matching the performance of the behavior policy, but hard to reach a good performance on the mixed dataset due to the quantity-quality dilemma.

heading: 4.2.1 Leverage Behavior Cloning with Curriculum Learning
text: In Section 3 we have seen evidence that a possible solution to the quantity-quality dilemma could be adaptive imitation through the offline dataset. An overview of such an adaptive imitation learning diagram compared with online RL is shown in Fig. 3. Formally, with dataset D = {τ}N1 , at every training stage i, the agent updates its policy πi by adaptively selecting τ ∼ π̃i fromD as the imitating target such that:
πi+1 = πi −∇πDKL(Pπ̃i(τ)‖Pπ(τ)) s.t. Eπ̃ [ DKL(π̃ i(·|s)‖πi(·|s)) ] ≤
Riπ̃ −Riπ ≥ δ (7)
where and δ are small positive numbers that limit the difference between the demonstrated policy π̃i and the learner π, and prevent π from learning poorly behaved policies. Correspondingly, each training
iteration i creates a curriculum automatically such that a taskGi is to imitating the closet demonstrated policy π̃i with Lπi = DKL(Pπ̃i(τ)‖Pπ(τ)), and the target task Gt is to imitate the optimal policy π̃∗. Specifically, we aims to construct a finite sequence π0, π̃1, π1, π̃2, π2, · · · , π̃N , πN such that πi πi+1, where π̃i is characterized by its trajectory, and is picked from D based on the current policy πi−1; πi is the imitation result taken π̃i as the target policy.
In the following sections, we explain how our algorithm is designed to achieve Eq. (7) that leads the target policy π̃ to finally collapse into the optimal behavior policy π̃∗, while solving the qualityquantity dilemma of BC to avoid getting a mediocre result.

heading: 4.2.2 Adaptive Experience Picking by Neighboring Policy Assessment
text: We first provide a practical solution to evaluate whether Eπ̃ [DKL(π̃(·|s)‖π(·|s))] ≤ . In other words, we want to know whether a trajectory τi ∈ D is sampled by a neighboring policy. This can be regarded as finding a policy whose importance sampling ratio is near to 1, and thus a lot of density ratio estimation works can be referred to [16, 26]. However, such estimation requires extra costs on training neural networks, and the estimation is inaccurate with fewer data points. Therefore, in this paper, we design a simple yet efficient neighboring policy assessment principle instead that brings the algorithm into practice.
We assume that each trajectory is sampled by a single policy. Beyond such a slight and practical assumption, let π be the current policy and trajectory τπ̃ = {(s0, a0, s′0, r0), · · · , (sh, ah, s′h, rh)} is collected by an unknown deterministic behavior policy π̃ with exploration noise such that E(s,a)∈τπ̃ [log π̃(at|st)] ≥ log (1− β), where β denote the portion of exploration. In this way, we find a practical solution that relaxes the KL-divergence constraint through an observation: Observation 1. Under the assumption that each trajectory τπ̃ in the dataset D is collected by an unknown deterministic behavior policy π̃ with an exploration ratio β. The requirement of the KL divergence constraint Eπ̃ [DKL(π̃(·|s)‖π(·|s))] ≤ suffices to finding a trajectory that at least 1− β state-action pairs are sampled by the current policy π with a probability of more than c such that c ≥ 1/ exp , i.e.:
E(s,a)∈τπ̃ [I(π(a|s) ≥ c)] ≥ 1− β , (8)
The corresponding deviation is shown in Appendix B.2. Therefore, to find whether τπ̃ is sampled by a neighboring policy, we calculate the probability of sampling at at state st by π in τ for every timestep τπ̃(π) = {π(a0|s0), · · · , π(ah|sh)}, where h is the horizon of the trajectory. In practice, instead of fine-tuning and β, we heuristically set β = 0.05 as an intuitive ratio of exploration. As for c, we let the agent choose the value through finding N nearest policies that matches Eq. (8).

heading: 4.2.3 Return Filter
text: We now present how to ensure the second constraint Rπ̃ −Rπ ≥ δ, which is designed to refrain the performance from getting worse by imitating to a poorer target than the current level of the imitating policy. In a practical offline scenario, it is impossible to get the accurate return of the current policy, but we can evaluate its performance based on the current curriculum. To this end, we adopt a return filtering mechanism that filtrates the useless, poor-behaved trajectories.
In practice, we initialize the return filter V with 0, and update the value at each curriculum. Specifically, if we choose {τ}n1 from D at iteration k, then V is updated by moving average:
Vk = (1− α) · Vk−1 + α ·min{R(τ)}n1 (9)
where {R(τ)}n1 is the accumulated reward set of trajectories {τ}n1 , and α is the moving window determining the filtering rate. Then, the dataset is updated as D = {τ ∈ D | R(τ) ≥ V }.

heading: 4.2.4 Overall Algorithm
text: Combining the adaptive experience picking strategy and the return filter, we finally get the simple and practical curriculum offline imitation learning (COIL) algorithm. To be specific, COIL holds an experience pool that contains the candidate trajectories to be selected. Every training time creates a stage of the curriculum where the agent selects appropriate trajectories as the imitation target from the pool and learns them via direct BC. After training, the used experience will be cleaned from the pool, and the return filter also filtrates a set of trajectories. An attractive property of COIL is that it
has a terminating condition that stops the algorithm automatically with a good policy when there is no candidate trajectory to be selected. This makes it easier to be applied in real-world applications without further finding the best-learned policy checkpoints under online evaluation as the previous algorithms do. The step-by-step algorithm is shown in Algo. 1.
It is worth noting that COIL has only two critical hyperparameters, namely, the number of selected trajectories N and the moving window of the return filter α, both of which can be determined by the property of the dataset. Specifically, N is related to the average discrepancy between the sampling policies in the dataset; α is influenced by the changes of the return of the trajectories contained in the dataset. In the ablation study Section 6.3 and Appendix E.2, we demonstrate how we select different hyperparameters for different datasets.

heading: 5 Related Work
text: As a long-studied branch of RL, numerous solutions have been developed for offline RL with both model-free [7, 12] and model-based [25, 11] algorithms. We briefly discuss the former from two categories, including RL-based methods and imitation-based methods.
RL-based methods. A naive thought is to apply off-policy RL algorithms such as SAC [9] and DDPG [14] directly. However, as previous researchers [7, 12] reveal, those online algorithms fail to work due to the severe extrapolation error or out-of-distribution problem. Thence, kinds of specifically designed algorithms have been proposed for offline RL. For instance, BCQ [7] adopts a perturbation model to disturb the action sampled by a BC module to conduct Q-learning on the offline data. BEAR [12] augments constraints on the policy to avoid out-of-distribution actions, over which it maximizes the approximate Q function. The current state-of-the-art algorithm CQL [13] performs strict constraints on the Q-function to learn an expectation lower bound of the true value, avoiding overestimation on out-of-distribution data. Typical drawbacks for these RL-based algorithms are serious bootstrapping errors and training instability that requires careful hyperparameter tuning with online evaluation. In comparison, imitation-based methods offer a candidate to mimic the demonstration.
Imitation-based methods. Another inspiration grows from the technique of offline imitation learning on how to learn from the demonstration. Most of these methods take the idea of behavior cloning (BC) [1] that utilizes supervised learning to learn to act. However, due to the quantity-quality dilemma, BC limits mediocre performance on many datasets with mixed samples. Therefore, BAIL [2] constructs the upper-envelope on the value of the data and selects the best actions to imitate at each state and learn the policy based on BC. The main problem in BAIL is the requirement of regressing the value function with tricks to compute a value in the infinite horizon and the hyperparameter threshold on determining the best action. ABM [22], MARWIL [23] and AWR [17] all take the idea of using an exponentially weighted version of BC, where the weights are determined by different forms of advantage function. These methods also require estimating the value function based on the offline data, which can be unstable and need many hyperparameters to control the learning procedure. Besides, it is also hard to tune an appropriate scale for the advantage for imitating the behavior policy. Compared with these methods, our COIL only has few important hyperparameters to be tuned without regressing any value function to achieve a stable performance.

heading: 6 Experiments
text: Alongside the simple algorithm, we surprisingly find that COIL not only alleviates the quantityquality dilemma but also achieves efficient and stable performance against competitive offline methods. Furthermore, we carry out a comprehensive analysis of the algorithm behind the phenomenon.

heading: 6.1 Offline Learning from Online Learning Experience
text: We are curious about the performance of COIL on the final buffer datasets containing a complete training experience of an online agent [7], because such dataset is mixed with various levels of policies. Therefore direct BC will easily fail as shown in Section 1. In our case, we first train an SAC agent from scratch to convergence and save all the training experience from its interactions with the environment, and therefore including exploration actions. Specifically, we conduct our experiments
on three continuous control tasks: Hopper, Walker2d, and HalfCheetah. Fig. 4 illustrates trajectories in each dataset, sorted in the collected order.
To show the effectiveness of COIL, we compare it with several strong baselines, including the stateof-the-art offline RL algorithm CQL [13], and imitation-based methods AWR [17] and BAIL [2]. For CQL, AWR and BAIL we use their open-source implementation with their default hyperparameter, and for BC we test the same implementation as COIL. The quantitative converged results are listed in Tab. 1, from which we observe that COIL substantially outperforms the other baselines for the final buffer dataset. Also, BAIL and AWR can not always find the optimal behavior due to the difficulty of its hyperparameters tuning and value regression. Specifically, compared with BC that learns a mediocre policy, COIL reaches the performance close to the optimal policy.
To further illustrate how our algorithm works, we also record the oracle online order of the trajectory sampled by the offline agent, as shown in Fig. 5. Notably, COIL keeps a similar training path as the online agent, thanks to the experience picking strategy and the return filter. The corresponding learning curves shown in Fig. 5 are stable and following similar shapes as the ordered datasets. Notably, COIL finally terminates with a near-data-optimal policy, suggesting a nice property that the last offline model can be a great model for deployment, unlike previous algorithms that relies on online evaluation to select the best checkpoint.

heading: 6.2 D4RL Benchmark
text: To further show the power of COIL, we conduct comparison experiments on a common-used D4RL benchmark [5] in Tab. 2. For BC we include the BC results reported in Fu et al. [5] (denoted as BC (D4RL)) and our implementation of BC (denoted as BC (Ours)). Besides, we compare BAIL [2],
MOPO [25], and the state-of-the-art results reported in Fu et al. [5] (denoted as SoTA). As our expected, BC is able to approach or outperform the performance of the behavior policy on the datasets generated from a single policy, marked as “random”, and “medium”, but still remains a gap between the optimal behavior policy (Best 1% column). As a comparison, COIL achieves the performance of the optimal behavior policy on most datasets. As is noticed that doing so will allow COIL to beat or compete with the state-of-the-art results. To our surprise, on the halfcheetah domain, previous RL-based offline algorithms have the potential to surpass the optimal behavior policy (although with careful hyperparameter tuning), showing the advantage of RL-based learning. It is worth noting that for model-based algorithm like MOPO, it behaves well on the medium-replay datasets due to the sufficient data to learn a good environment model; but it can hardly outperform SoTA model-free results on other datasets.
6.3 Ablation Study
The ablation study aims to illustrate how hyperparameters can be determined according to the property of the offline datasets without online evaluation. In particular, COIL has two critical hyperparameters: α and N . Since α determines the rate of filtering out the bad trajectories as the agent imitates a better policy where small α corresponds to a high filtering rate, it is highly influenced by the changes in returns of the trajectories in the dataset. Taking hopper-mediumreplay and hopper-medium for examples, the former consists of online training experience, and the return of trajectories changes rapidly; on the contrary, most of the trajectories in the latter are at the same return level. Therefore, intuitively, a small value of α should be assigned to hopper-medium-replay while a large value to hopper-medium, as confirmed by the experimental results shown in Fig. 6. Ablation experiments on the other hyperparameter N are
illustrated in Appendix E, saying that it can also be tuned according to the characterization of the dataset.

heading: 6.4 Compared with Naive Curriculum Strategies
text: In this section, we propose comparing the curriculum strategy of COIL with naive curriculum strategies to determine whether COIL is a better choice. The first strategy is called return-ordered BC, which conducts curriculum imitation learning based only on returns of the trajectories. Similar to COIL, it picks NRBC trajectories with the lowest returns for each curriculum to perform behavioral cloning, and then removes them from the dataset. When the dataset is empty, the algorithm stops. The other strategy is called buffer-shrinking BC. As its name suggests, it conducts curriculum imitation learning by shrinking the dataset ordered by the return after training at each curriculum. In detail, buffer-shrinking BC begin its training with the entire dataset in the buffer; after a fixed number of
gradient steps, it shrinks the buffer by discarding p% of trajectories with the lowest returns; then the training is continued on the remaining trajectories. In our experiment, we choose p = 20 so that the algorithm will stop after 5 times of shrinkage.
The learning results on final datasets are shown in Fig. 7. As expected, return-ordered BC leads to mediocre behaviors since similar returns do not always account for similar policies due to the exploration noise. In addition, buffer-shrinking BC is not usually stable to achieve the optimal behavior. Since the shrink strategy is totally hand-crafted, the quantity-quality dilemma is not eliminated. The mediocre trajectories in the last curriculum will lead to the failure. On the contrary, COIL succeeds in the optimal behavior policy with the highest training efficiency (the least gradient steps), indicating the advantage of the policy-distance-based curriculum.

heading: 7 Conclusion
text: In this paper, we analyze the quantity-quality dilemma of behavior cloning (BC) from both an experimental and a theoretical point of view, which motivates us to propose the curriculum offline imitation learning (COIL). COIL takes advantage of imitation learning by improving the current policy with adaptive neighboring policies. Experiments show good properties of COIL with competitive evaluation results against state-of-the-art offline RL algorithms. COIL may provide a practical way for bringing offline RL into practice due to its simplicity and effectiveness, but it also limits into the performance of the dataset.

heading: Acknowledgements
text: The corresponding author Weinan Zhang is supported by “New Generation of AI 2030” Major Project (2018AAA0100900), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102) and National Natural Science Foundation of China (62076161, 61632017). The author Minghuan Liu is supported by Wu Wen Jun Honorary Doctoral Scholarship, AI Institute, Shanghai Jiao Tong University. We sincerely thank the reviewers for helpful feedback.

Review summaries:
This paper applied Online Imitation Learning to offline RL. The central idea is, at each iteration of training, the proposed approach will sample the data points from the neighboring policies based on the experience picking. The intuition is built based on the observation/analysis of the discrepancy between demo policy and initialized imitating policy.  I like how authors scrutinized their observations with theoretical analysis. The neighboring policy picking targets at solving the "dilemma" of quantity-quality. However, the main technical contribution is not super very in-depth, authors tackle the KL-divergence constraints through the assumption that each trajectory is sampled by a single policy, which I am not sure whether it is very practical as claimed by authors (yield good bound and performance as illustrated in experimental studies).

The method proposes an algorithm for Offline Reinforcement Learning. The work starts from the idea that behavioral cloning struggles at learning from data coming from several policies but, on the contrary, easily learns a behavior policy if initialized with a “close” policy. A curriculum is thus built for behavioral cloning to efficiently learn the “best policy shown in the dataset”. For this, a behavioral cloning is trained on a sequence of trajectories defined the following way: the next trajectory selected for training is the most probable according to current policy. The trajectories-dataset is then filtered by saying that the return should be above a threshold given by a running average of the trajectories seen so far.