Curriculum Offline Imitating Learning

1 Introduction . Offline reinforcement learning ( RL ) , or batch RL , aims to learn a well-behaved policy from arbitrary datasets without interacting with the environment . This setting is generally a more practical paradigm than online RL since it is expensive or dangerous to interact with the environment in most real-world applications . Typically , two main kinds of offline datasets are considered in previous offline RL works [ 5 , 18 ] : one contains transitions sampled by a single behavioral policy ; the other includes a buffer collected by a mixture of policies . Two main approaches have been deeply investigated for Offline RL . First , RL-based methods , in particular , Q-learning and policy gradient-based algorithms [ 13 , 7 , 12 ] , have the potential to outperform the behavioral policy . However , they always suffer from serious bootstrapping errors and training instability . This shortcoming makes such algorithms impractical to be utilized since too many hyperparameters need to be tuned to achieve a good performance , and it is hard to evaluate a suitable model in an offline manner , as revealed in [ 18 ] . In contrast , offline imitation learning [ 1 , 17 , 2 ] , specifically , behavior cloning ( BC ) , can always stably learn to perform as the behavioral policy , which may be helpful under single-behavior datasets . However , BC may fail in learning a good behavior under a diverse dataset containing a mixture of policies ( both goods and bads ) . Quantity-quality dilemma on mixed dataset . As a supervised learning technique , BC is not easy to fulfill a desired result , especially on a mixed dataset . Specifically , it requires both quantity and quality ∗Equal contribution . †Corresponding author . Codes are available at https : //github.com/apexrl/COIL . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of the demonstration data , which can hardly be satisfied in offline RL tasks . Directly mimicking the policy from a mixed dataset that contains bad-to-good demonstrations can be regarded as imitation learning from a mediocre behavior policy . To achieve the best performance of the dataset , a naive idea is to imitate the top trajectories ordered by its return . However , such a simple strategy will reach the quantity-quality dilemma on the mixed dataset . For example , Fig . 1a illustrates the ordered trajectories on the Walker-final-dataset , which contains the whole training experience sampled by an online training agent . Different BC agents are trained by the top 10 % , 25 % , 50 % , and 100 % trajectories , but none of them gets rid of a mediocre performance , as shown in Fig . 1b . Typically , on such dataset , less data owns higher quality but less quantity , and thus cause serious compounding error problems [ 21 , 10 , 6 ] ; on the other hand , more data provides a larger quantity , yet its mean quality becomes worse . In this work , we aim to solve such a dilemma and exploit the most potential of IL to derive a stable and practical algorithm reaching the best performance of a given dataset . Our intuition comes from the observation that under RL scenarios , the agent can imitate a neighboring policy with much fewer samples . This observation promotes a curriculum solution for the above challenge . Specifically , for mixed datasets , the agent can adaptively imitate the better neighboring policies step by step and finally reach the optimal behavior policy of the dataset . Our work . We propose Curriculum Offline Imitation Learning ( COIL ) , a simple yet effective offline imitation learning framework for offline RL . At each training iteration , COIL improves the current policy with the data sampled by neighboring policies . To achieve that , COIL utilizes an adaptive experience picking strategy and a return filter to select proper trajectories from the offline dataset for the current level of the agent and thus produces stages of the curriculum . Notably , COIL stops with a close-todata-optimal policy without finding the best model under online evaluation . This feature allows to deploy the algorithm in practical problems . In experiments , we show the effectiveness of COIL on various kinds of offline datasets . Fig . 1 offers a quick review of our results : depending solely on BC , COIL can learn from scratch to reach the best performance of the given dataset . Contributions . To summarize , the main technical contributions of this paper are as follows . • We highlight how the discrepancy between the target policy and the initialized policy affects the number of samples required by BC ( Section 3 ) ; • Depending on BC , we propose a practical and effective offline RL algorithm with a practical neighboring experience picking strategy that ends with a good policy ( Section 4 ) ; • We present promising comparison results with comprehensive analysis for our algorithm , which is competitive to the state-of-the-art methods ( Section 6 ) . 2 Preliminaries . Notations . We consider a standard Markov Decision Process ( MDP ) as a tuple M = 〈S , A , T , ρ0 , r , γ〉 , where S is the state space , A represents the action space , T : S ×A×S → [ 0 , 1 ] is the state transition probability distribution , ρ0 : S → [ 0 , 1 ] is the initial state distribution , γ ∈ [ 0 , 1 ] is the discounted factor , and r : S × A → R is the reward function . The goal of RL is to find a policy π ( a|s ) : S × A → [ 0 , 1 ] that maximizes the expected cumulative discounted rewards ( or called return ) along a trajectory τ : R ( τ ) = ∑T t=0 γ trt . The dataset D consists of trajectories { τN1 } that are the collected by a mixture of bad-to-good policies , where a trajectory τi = { ( si0 , ai0 , s′i0 , ri0 ) , ( si1 , ai1 , s′i1 , ri1 ) , · · · , ( sihi , a i hi , s′ihi , r i hi ) } , and hi is the horizon of τi . For any dataset , we assume a behavior policy πb that collects such data and its empir- ical estimation π̂b can be induced from D as π̂b ( a|s ) = ∑ ( s′ , a′ ) ∈D I [ s ′=s , a′=a ] ∑ s′∈D I [ s′=s ] , where I is the indicator function . We further introduce a common used term , occupancy measure , which is defined as the discounted occurrence probability of states or state-action pairs under policy π : ρπ ( s , a ) = ∑∞ t=0 γ tP ( st = s , at = a|π ) = π ( a|s ) ∑∞ t=0 γ tP ( st = s|π ) = π ( a|s ) ρπ ( s ) . With such a definition we can write down that Eπ [ · ] = ∑ s , a ρπ ( s , a ) [ · ] = E ( s , a ) ∼ρπ [ · ] . Definition 1 . The partial order of different policies is defined as the relative return quantity that a policy can achieve when deployed in the environment . Formally , given two policies π1 and π2 : π1 π2 ⇔ R1 R2 ( 1 ) Therefore , by definition , in a mixed dataset D that is collected by K different policies π1 , · · · , πK , the optimal behavior policy π∗ can be determined such that for ∀i ∈ [ 1 , K ] , πi π∗ . Curriculum learning . Curriculum learning design and construct a curriculum automatically as a sequence of tasks G1 , . . . , GN to train on , such that the efficiency or performance on a target task Gt is improved . The expected loss on the jth task is denoted Lj . 3 Empirical Observations and Theoretic Analysis . In this section , we begin with empirical observations that motivate the core idea of our method , followed by the theoretical analysis to support our motivation . Generally , we aim to investigate how the asymptotic performance of BC is affected by the discrepancy between the demonstrated policy and the initialized imitating policy . Previous research shows that BC tends to fail with a small number of high-quality demonstrations but can learn well from large-quantity and high-quality data [ 10 , 8 ] . On the contrary , we find that the requirement of quantity can be highly relaxed as the similarities between the demonstrated policy and the initialized imitating policy increase . 3.1 BC with Different Initialization . To construct the motivating example , we choose Hopper as the testbed , and train an SAC agent until convergence to sample various counts of trajectories as the demonstration data . We then take the online-trained policy checkpoints at different training iterations as the initiated policy to train an IL agent . Particularly , the first agent adopts the Random policy to imitate the demonstration by BC ; the second uses the policy of 1/3 Return and the other two agents start with 1/3 Trained and 2/3 Trained policy separately , in terms of training iterations ( see Fig . 2a ) . The results are shown in Fig . 2b , where we illustrate the average return of each agent given the different number of demonstrations ( the exact quantitative results can be found in Appendix E.1 ) . The results show that initialized with a Random policy , the agent can only learn to imitate the demonstrated policy well with a large number of samples2 ; in addition , both 1/3 Return and 1/3 Trained policies can achieve a sub-optimal performance with fewer samples , where the 1/3 2We note that the Random agent can only work well on large datasets with normalized state space ; however the other agents learn well upon raw states . Trained one is more efficient . In comparison , the 2/3 Trained agent , which is the closest to the optimal policy , can stably achieve the best performance with all amounts of trajectories . 3.2 Theoretical Analysis . Beyond these observations , we are inquisitive to find a theoretical explanation to support our claim . Standing on the primary result of existing works , we obtain a performance bound of BC with a possible solution to handle the quantity-quality dilemma . Theorem 1 ( Performance bound of BC ) . Let Π be the set of all deterministic policy and |Π| = |A||S| . Assume that there does not exist a policy π ∈ Π such that π ( si ) = ai , ∀i ∈ { 1 , · · · , |D| } . Let π̂b be the empirical behavior policy as well as the corresponding state marginal occupancy is ρπ̂b . Suppose BC begins from initial policy π0 , and define ρπ0 similarly . Then , for any δ > 0 , with probability at least 1− δ , the following inequality holds : DTV ( ρπ ( s , a ) ‖ρπb ( s , a ) ) ≤ c ( π0 , πb , |D| ) where c ( π0 , πb , |D| ) = 1 2 ∑ s/∈D ρπb ( s ) + 1 2 ∑ s/∈D |ρπ ( s ) − ρπ0 ( s ) |+ 1 2 ∑ s/∈D |ρπ0 ( s ) − ρπb ( s ) |︸ ︷︷ ︸ initialization gap + 1 2 ∑ s∈D |ρπ ( s ) − ρπ̂b ( s ) |+ 1 |D| |D|∑ i=1 I [ π ( si ) 6= ai ] ︸ ︷︷ ︸ BC gap + [ log |S|+ log ( 2/δ ) 2|D| ] 1 2 + [ log |Π|+ log ( 2/δ ) 2|D| ] 1 2 ︸ ︷︷ ︸ data gap ( 2 ) The proof can be found in Appendix B.1 . Theorem 1 shows the upper bound of the state-action distribution between the imitating policy π and the behavior policy πb , which consists of three important terms : the initialization gap , the BC gap and the data gap . Specifically , the BC gap arises from the empirical error and the difference between the imitating policy and the empirical behavior policy , which is corresponding to the training procedure . The data gap , however , depends on the number of samples and complexity of the state space , acting as an intrinsic gap due to the dataset and the environment . As for the initialization gap , it is in the form of distance between the state marginal distribution of the initial policy π0 and behavior policy πb out of the dataset . Notice that the second term in Eq . ( 2 ) relates to the distance between the state marginal of the initial policy π0 and the learned policy πb outside the data support , which is hard to measure theoretically due to the Markov property of the environment dynamics . Therefore , we estimate the empirical discrepancy outside the dataset 12 ∑ s/∈D |ρ̂π ( s ) − ρ̂π0 ( s ) | for this term3 . The results shown in Fig . 2c , as expected , suggests that the second term in Eq . ( 2 ) in fact decreases as the initialized policy gets close to the demonstrated policy because of the poor generalization on unseen states , and the error can be further reduced with a larger dataset . Such analysis brings a possible theoretical explanation to our empirical intuition Section 3.1 . Generally , given the same discrepancy c ( π0 , πb , |D| ) = C , if the initialized policy narrows down the initialization gap as is close to the demonstrated policy , then the requirement for more samples to minimize the data gap can be relaxed . This may seem unreasonable in the learning theory in the traditional supervised learning domain . However , under the RL scenario , the performance of a policy depends on the accumulated reward along the rollout trajectories , which will lead to serious compounding error problems [ 21 , 20 ] . Therefore , as the distance between the initialized policy and the demonstrated policy gets closer , the generalization errors of the learned policy can be reduced . Brief conclusion . Both the experimental and the theoretical results indicate an interesting fact that the asymptotic performance of BC is highly related to the discrepancy between the initialized policy and the demonstrated policy . Specifically , a close-to-demonstration policy can easily imitate the demonstrated policy with fewer samples . On the contrary , when the distance between the initialized policy and the demonstrated policy is far , then successfully mimicking the policy will require much more samples . Such an observation motivates the intuition for proposing our Curriculum Offline Imitation Learning ( COIL ) in the following literature . The key insight enabling COIL is adaptively imitating the close policies with a small number of samples and finally terminates with the optimal behavior policy of the dataset . 3For implementation details , see Appendix C.1 .

Summary: This paper applied Online Imitation Learning to offline RL. The central idea is, at each iteration of training, the proposed approach will sample the data points from the neighboring policies based on the experience picking. The intuition is built based on the observation/analysis of the discrepancy between demo policy and initialized imitating policy.  I like how authors scrutinized their observations with theoretical analysis. The neighboring policy picking targets at solving the "dilemma" of quantity-quality. However, the main technical contribution is not super very in-depth, authors tackle the KL-divergence constraints through the assumption that each trajectory is sampled by a single policy, which I am not sure whether it is very practical as claimed by authors (yield good bound and performance as illustrated in experimental studies).