Year: 2021

Abstract:
We study the problem of offline Imitation Learning (IL) where an agent aims to learn an optimal expert behavior policy without additional online environment interactions. Instead, the agent is provided with a static offline dataset of stateaction-next state transition triples from both optimal and non-optimal behaviors. This strictly offline imitation learning problem arises in many real-world problems, where environment interactions and expert annotations are costly. Prior works that address the problem either require that expert data occupies the majority proportion of the offline dataset, or need to learn a reward function and perform offline reinforcement learning (RL) based on the learned reward function. In this paper, we propose an imitation learning algorithm to address the problem without additional steps of reward learning and offline RL training for the case when demonstrations containing large proportion of suboptimal data. Built upon behavioral cloning (BC), we introduce an additional discriminator to distinguish expert and non-expert data, we propose a cooperation strategy to boost the performance of both tasks, this will result in a new policy learning objective and surprisingly, we find its equivalence to a generalized BC objective, where the outputs of discriminator serve as the weights of the BC loss function. Experimental results show that our proposed algorithm achieves higher returns and faster training speed compared to baseline algorithms.

heading: 1 INTRODUCTION
text: The recent success of reinforcement learning (RL) in many domains showcases the great potential of applying this family of learning methods to real-world applications. A key prerequisite for RL is to design a reward function that specifies what kind of agent behavior is preferred. However, in many real-world applications, designing a reward function is prohibitively difficult (Ng et al., 1999; Irpan, 2018). By contrast, imitation learning (IL) provides a much easier way to leverage the reward function implicitly from the collected demonstrations and has achieved great success in many sequential decision making problems (Pomerleau, 1989; Ng et al., 2000; Ho & Ermon, 2016).
However, popular IL methods such as behavioral cloning (BC) and generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016), assume the expert demonstration is optimal. Unfortunately, it is often difficult to obtain optimal demonstrations for many real-world tasks, because human experts often make mistakes due to various reasons, such as the difficulty of the task, partial observability of the environment, or the presence of distraction. Given such noisy expert demonstrations, which contain records of both optimal and non-optimal behaviors, BC and GAIL all fail to imitate the optimal policy (Wu et al., 2019b; Ma, 2020). Current methods that deal with suboptimal demonstrations either require additional labels, which can be done explicitly by annotating each demonstration with confidence scores by human experts (Wu et al., 2019b), or implicitly by ranking noisy demonstrations according to their relative performance through interacting with the environment (Brown et al., 2019; 2020). However, human annotation and environment interaction are laborious and expensive in real-world settings, such as in medicine, healthcare, and industrial processes.
In this work, we investigate a pure offline learning setting where the agent has access to neither the expert nor the environment for additional information. The agent, instead, has only access to a small pre-collected dataset of state-action-next state transition triples sampled from the expert and a large batch offline dataset sampled from one or multiple behavior policies that could be highly sub-optimal. Prior works that address the problem are based on variants of BC or inverse RL, Sasaki
& Yamashina (2021) reuse another policy learned by BC as the weight of original BC objective, however, this requires that expert data occupy the majority proportion of the offline dataset, otherwise the policy will be misguided to imitate the suboptimal data. Zolna et al. (2020a) first learns a reward function that prioritizes expert data over others and then performs offline RL based on this reward function. This algorithm is extremely expensive to run, requiring solving offline RL in an inner loop, which itself is a challenging problem and prone to training instability (Kumar et al., 2019) and hyperparameter sensitivity (Wu et al., 2019a).
In this paper, we propose an offline imitation learning algorithm to learn from demonstrations that (perhaps) contain a large proportion of suboptimal data without additional steps of reward learning and offline RL training. Built upon the task of behavioral cloning (BC), we introduce an additional task to learn a discriminator to distinguish expert and non-expert data, we propose a cooperation strategy to boost the performance of both tasks. This results in a new policy learning objective and surprisingly, we find its equivalence to a generalized BC objective, where the outputs of the discriminator serve as the weights of the BC loss function. We thus term our resulting algorithm Discriminator-Weighted Behavioral Cloning (DWBC). Experimental results show that DWBC achieves higher returns and faster training speed compared to baseline algorithms, under different scenarios.

heading: 2 PRELIMINARY

heading: 2.1 PROBLEM SETTING
text: We consider the standard fully observed Markov Decision Process (MDP) setting (Sutton et al., 1998), M = {S,A, P, r, γ, d0}, where S is the state space, A is the action space, P : S × A → ∆(S) is the MDP’s transition probability, r is the reward function, γ ∈ [0, 1) is the discount factor for future reward and d0 is the initial distribution. A policy π : S → ∆(A) maps from state to distribution over actions. We denote dπ ∈ ∆(S × A) as the discounted state-action distribution of π under transition kernel P , that is, dπ = (1− γ) ∑∞ t=0 γ tdπt , where d π t ∈ ∆(S × A) is the distribution of(
s(t), a(t) )
under π at step t. Following the standard IL setting, the ground truth reward function r is unknown. Instead, we have the demonstrations by the expert specified by πe : S → ∆(A) (potentially stochastic and not necessarily optimal). Concretely, we have an expert dataset in the form of i.i.d tuples De = {si, ai, s′i} ne i=1 where (s, a) is sampled from distribution d
πe and s′ is sampled from P (s, a).
In our problem setting, we also have an offline static dataset consisting of i.i.d tuples Do = {si, ai, s′i} no i=1 s.t. (s, a) ∼ ρ(s, a), s′ ∼ P (s, a), where ρ ∈ ∆(S × A) is an offline state-action distribution resulting from some other behavior policies. Note that these behavior policies could be much worse than the expert πe. Our goal is to only leverage the offline batch data Db = De ∪ Do to learn an optimal policy π with regard to optimizing the ground truth reward r, without any interaction with the environment or expert.

heading: 2.2 A GENERALIZED BEHAVIORAL CLONING OBJECTIVE
text: In order to discard low-quality demonstrations and only clone the best behavior available, previous works (Sasaki & Yamashina, 2021; Zolna et al., 2020a) consider a generalized behavioral cloning objective to imitate demonstrations unequally, that is,
min π E (s,a)∼Db [− log π(a|s) · f(s, a)] (1)
where f : S ×A → [0, 1] denotes an arbitrary weight function.
• If f(s, a) = 1 for ∀(s, a) ∈ S ×A, the objective (1) corresponds to the vanilla BC objective. • If f(s, a) = π′(a|s), where π′ is an old policy which was previously optimized with Db, the
objective (1) corresponds to the objective of Behavioral Cloning from Noisy Demonstrations (Sasaki & Yamashina, 2021). Since ∑ a π ′(a|s) = 1 for ∀s ∈ S is satisfied, π′(a|s) can be interpreted as
the weights for the weighted action sampling. • If f(s, a) = 1 [Aπ(s, a)], where 1 is the indicator function which creates a boolean mask that elim-
inates samples which are thought to be worse than the current policy, the objective (1) corresponds to the objective of Offline Reinforced Imitation Learning (Zolna et al., 2020a).
The objective (1) can also be deemed as the objective of Soft Q Imitation Learning (Reddy et al., 2019) with f(s, a) = 1 for (s, a) ∈ De and f(s, a) = 0 for (s, a) ∈ Do in online IL literature; or the objective of off-policy actor-critic (Off-PAC) algorithm (Degris et al., 2012) with f(s, a) = Qπ(s, a) · π(a|s)/πb(a|s) in online RL literature.

heading: 3 DISCRIMINATOR-WEIGHTED BEHAVIORAL CLONING
text: We now continue to describe our approach for offline imitation learning from demonstrations that (perhaps) contain large-proportional suboptimal data without additional steps of reward learning and offline RL training. Built upon the task of BC, we introduce an additional task to learn a discriminator to distinguish expert and non-expert data, we propose a cooperation strategy to boost the performance of both tasks, this will result in a new generalized BC objective. We then provide the interpretation of weights in our generalized BC objective, this gives the intuition about why our method can work.

heading: 3.1 LEARN THE POLICY AND DISCRIMINATOR SEPARATELY
text: It is obvious that we can avoid the negative impact of suboptimal demonstrations presented in Do by only imitating De, which can be written as
min π E (s,a)∼De [− log π(a|s)] (2)
We call the task of learning a policy using objective (2) as the BC task. The drawback of BC task is that the learned policy may not be able to generalize due to the potential limited size and state coverage of De. It does not fully utilize the information from Do. If we can select those high-reward transitions from Do and combine them with De, we are expected to get a better policy. Now let’s consider another different task, which aims to learn a discriminator by contrasting expert and non-expert transitions, given by
min d E (s,a)∼De [− log d(s, a)] + E (s,a)∼Do [− log(1− d(s, a))] (3)
We call this the discriminating task, which is similar to how the discriminator is trained in GAIL (Ho & Ermon, 2016) and GAN (Goodfellow et al., 2014a), except that the second term is sampled from a fixed dataset instead of new samples obtained from interacting with the environment.
However, optimizing objective (3) will make the discriminator learned to assign 1 to all transitions from De and 0 to all transitions from Do. This limiting behavior is unsatisfactory because Do can contain some successful (high-reward) transitions. This bears similarity to the positive-unlabeled classification problem (Elkan & Noto, 2008), where both positive and negative samples exist in the unlabeled data.
To solve this problem, we adopt the approach from positive-unlabeled (PU) learning (Du Plessis et al., 2015; Xu & Denil, 2019; Zolna et al., 2020b). The main idea is to re-weight different losses for positive and unlabeled data, in order to obtain an estimate of model loss on negative samples that is not directly available. Applying PU learning to objective (3) yields the following objective:
min d η E (s,a)∼De [− log d(s, a)] + E (s,a)∼Do [− log(1− d(s, a))]− η E (s,a)∼De [− log(1− d(s, a))] (4)
where η is a hyperparameter, corresponds to the proportion of positive samples to unlabeled samples. Intuitively, the second term in (4) could make d(s, a) of state-action pairs from De become 0 if similar state-action pairs are included in Do, and the third term in (4) balances the impact of the second term, i.e., avoids d(s, a) of state-action pairs from De becoming 0. To summarize, the BC task aims to imitate the expert behavior from De, but ignores the valuable information in Do; the discriminating task aims to contrast expert and non-expert transitions from De and Do, by only using state-action information as input. Both tasks lack enough information to improve their own performance, which, however, can be obtained from the other task. It then seems natural to find a scheme to incorporate the policy into the training of the discriminator and effectively use the discriminator to help the training of the policy.

heading: 3.2 LEARN THE POLICY AND DISCRIMINATOR COOPERATIVELY
text: We propose a cooperation strategy to boost the performance of both tasks. To boost the performance of the discriminating task, we add the imitation information from π to the input of the discriminator, yielding the following discriminator learning objective Ld to be minimized as:
min d Ld = min d η E (s,a)∼De [− log d(s, a, log π(a|s))] +
E (s,a)∼Do [− log(1− d(s, a, log π(a|s)))]− η E (s,a)∼De
[− log(1− d(s, a, log π(a|s)))] (5)
Supposed π is learned to be optimal, i.e., assigns large probabilities to expert actions in expert states, the discriminator will receive additional learning signal. It will be easier for the discriminator to contrast expert and non-expert transitions in Do, as log π(a|s) will be large if (s, a) are from expert behaviors and small if (s, a) are from non-expert behaviors. Without this information from π, the discriminator is much harder to learn by only using information from (s, a).
Now let’s dive deeper into objective (5). As π now appears in the input of the discriminator d, hence both d and its loss Ld become functionals of π. We are interested to see how does the imitation information from log π affect Ld, and further impact d. In other words, given the current d, we want to explore how to change the behavior of π such that d can be better learned. To achieve this goal, we define a functional J(π) for Ld with the function π as the variable, and fix the parameter θd of d to exclude its own influence. Note that in the functional J(π), d is now fixed and its parameter θd will no longer be considered as a variable. J(π) can be formally define as the following integral form to further eliminate the effect of changes in s, a on d:
J(π) = ∫ ∫ ∂Ld(s, a, d, log π) ∂d(s, a, log π) ds da = ∫ ∫ F (s, a, π, π′) ds da, (6)
where we denote F = ∂Ld/∂d and F is assumed to be continuously differentiable with respect to s, a, π and π′ (derivative of π). To more robustly learn the discriminator d, inspired by the idea of adversarial learning (Lowd & Meek, 2005), we make the policy π challenge the discriminator d by doing the opposite to the task of d (i.e., minimize Ld), in other words, we want π to maximize Ld under current d. By doing so, the policy will provide as little information in log π as possible such that minimizing Ld becomes harder for the discriminator, this can be seen as minimizing the worst-case error (Carlini et al., 2019; Fawzi et al., 2016; Goodfellow et al., 2014b), which makes the robustness of the discriminator significantly improved.
To maximize Ld for π under current d, we let the functional J(π) attains its maxima with respect to π. According to the calculus of variations (Gelfand et al., 2000), the extrema (maxima or minima) of functional J(π) can be obtained by finding a function π such that the functional derivative of J(π) is equal to zero. We can show with theoretical derivation (see Appendix A.2) that ensuring the functional derivative of J(π) equal to zero requires ∂F/∂θπ = 0, where θπ is the parameters of π. By the chain rule, we can have
∂F ∂θπ = ∂F ∂ log π ·∇θπ log π = − E (s,a)∼De [η d · ∇θπ log π(a|s) ] +
E (s,a)∼Do
[ 1
1− d · ∇θπ log π(a|s)
] − E
(s,a)∼De
[ η
1− d · ∇θπ log π(a|s)
] , (7)
where we write the output value of d(s, a, log π(a|s)) as d for simplicity by slightly abusing notations. Notice that above derviative can be equivalently perceived as the gradient of a new loss term Lw of the policy π (∂Lw/∂θπ = −∂F/∂θπ) with following form:
Lw = E (s,a)∼De
[ log π(a|s) · η
d
] − E
(s,a)∼Do
[ log π(a|s) · 1
1− d
] + E
(s,a)∼De
[ log π(a|s) · η
1− d
] .
Hence minimizing Lw with respect to π (make ∂Lw/∂θπ = 0) will drive the functional derivative of J(π) to zero, which leads to the maxima of J(π) (update with gradient direction−∂F/∂θπ). Adding this loss term Lw to the BC task, we get the following new learning objective of π as:
min π α E (s,a)∼De [− log π(a|s)]− E (s,a)∼De
[ − log π(a|s) · η
d (1− d)
] + E
(s,a)∼Do
[ − log π(a|s) · 1
1− d
] ,
(8)
where α is the weight factor (α ≥ 1). This new objective essentially transforms the original BC task into a cost-sensitive learning problem (Ling & Sheng, 2008) by imposing the following weight on imitating each state-action transition as
Behavioral cloning weights = { α− η/d(1− d), (s, a) ∈ De 1/ (1− d) , (s, a) ∈ Do . (9)
Note that the derivation of the above behavioral cloning weights requires uniform continuity to be satisfied in F and its derivative (details see Appendix A.2). The involvement of 1/d and 1/(1− d) may violate the continuity assumption. We thus clip the value d to the range of [0.1, 0.9].
Above behavioral cloning weights induce different behaviors on the imitation of transitions from De andDo. Supposed d is learned to be optimal, i.e., assigns large values (close to 1) to expert transitions and small values to non-expert transitions (close to 0). The weight of those expert transitions in Do will become large while the weight of those non-expert transitions will become small. For transitions in De, their weights can be adjusted by tuning the parameter α. Note that even if the discriminator is learned to be totally wrong (i.e., assign small values to expert transitions and large values to non-expert transitions), which may occur at the very beginning of training, the behavior cloning weights α− η/d(1− d) (α ≥ 1, η < 1) will not be drastically changed under value clipping. This means that the policy can still learn from the expert dataset De. Even though the weight for De is temporarily incorrect, it will be corrected as the discriminator becomes better and better.
Eq. (8) implies that our approach is also a variant of generalized behavioral cloning objective, as discussed in section 2.2, but uses a different form of weights. Unlike Offline Reinforced Imitation Learning (Zolna et al., 2020a), which uses the discriminator as the reward and learns a value function as the weight, our approach uses the discriminator outputs directly as the weight. This can greatly reduce the training time and avoid the overestimation issue in estimating the value function offline (Kumar et al., 2019). We thus term our algorithm Discriminator-Weighted Behavioral Cloning (DWBC).
A keen reader may find the similarity of our approach to the adversarial learning in GAN (Goodfellow et al., 2014a), which makes the generator (in our case is the policy π) and the discriminator d learn against each other, thus improves the performance of both tasks. However, the learning strategy of DWBC has several differences compared with GAN and is also much easier to learn. In constrast to the fully adversarial setting in GAN, DWBC adopts a semi-cooperative strategy that shares information between the policy π and the discriminator d to improve the performance of both parties. Moreover, GAN needs to solve a min-max optimization problem and is known to suffer from training instability and issues such as mode collapse (Arjovsky et al., 2017). In DWBC, we solve the min-max optimization problem implicitly by transforming the max optimization problem to a new learning loss imposed on the policy. We present an illustration of the learning mechanism of DWBC in Figure 1.

heading: 4 RELATED WORK

heading: 4.1 OFFLINE IMITATION LEARNING
text: Offline IL, which has not received considerable attention, is a promising area because it makes IL more practical to satisfy critical safety desiderata. Offline IL methods can be typically folded into two paradigms: Behavioral Cloning (BC) and Offline Inverse Reinforcement Learning (Offline IRL).
BC (Pomerleau, 1989) is the simplest IL method that can be used in the offline setting, it considers the policy as a conditional distribution π(·|s) over actions, recent work (Florence et al., 2021)
enhances BC by using energy-based models (LeCun et al., 2006). BC has shown to have no inferior performance compared to popular IL algorithms such as GAIL (Ho & Ermon, 2016) when clean expert demonstrations are available (Ma, 2020). Unlike BC, offline IRL considers matching the state-action distribution induced by the expert policy, this can be achieved implicitly by adversarial training or explicitly by learning a reward function. Offline IRL algorithms based on adversarial training (Kostrikov et al., 2019; Sun et al., 2021; Swamy et al., 2021; Jarboui & Perchet, 2021) use Intergral Probability Metrics (IPMs) (Sriperumbudur et al., 2009) as a distance measure to solve the dual problem. They introduce a discriminator and aim to find the saddle point of a min-max optimization problem, like GAN (Goodfellow et al., 2014a). Jarrett et al. (2020) avoids the need of min-max problem by fixing the policy to be energy-based models, in such case the KL divergence from the demonstrator’s state-action distribution to that of the policy can be computed in closed form. However, recent work finds several fundamental mathematical misconceptions in their proposed approach and we refer the reader to Swamy et al. (2021) for more details.
The common problem of these works is that they imitate equally to all demonstrations, this will hinder the performance if the demonstrations contain suboptimal data. To solve this, Sasaki & Yamashina (2021) reuses another policy learned by BC as the weight of original BC objective, however, this requires that expert data occupies the majority proportion of the offline dataset, otherwise the policy will be misguided to imitate the suboptimal data. Zolna et al. (2020a) and Konyushkova et al. (2020) first construct a reward function that discriminates expert and exploratory trajectories, then use it to solve an offline RL problem. Instead of the adversarial learning scheme, the reward function can also be learned by cascading to two supervised learning steps (Klein et al., 2013). However, offline IRL based on reward learning is extremely expensive to run, requiring solving offline RL in an inner loop, which itself is a challenging problem and prone to training instability (Kumar et al., 2019) and hyperparameter sensitivity (Wu et al., 2019a). Our algorithm can be seen as a combination of these two algorithms in that we use a generalized BC objective to imitate demonstrations selectively and we train a discriminator to distinguish expert and non-expert data and use the output of the discriminator as the weight of the generalized BC objective. There is also one recent work (Chang et al., 2021) that tries to solve the offline IL problem by adopting techniques from pessimistic model-based offline policy learning (Yu et al., 2020; 2021).

heading: 4.2 OFFLINE REINFORCEMENT LEARNING
text: One research area highly related to offline IL is offline RL (Lange et al., 2012; Levine et al., 2020), which considers performing effective RL by utilizing arbitrary given, static offline datasets, without any further environment interactions. Note that in offline RL, the training dataset is allowed to have non-optimal trajectories and the reward for each state-action-next state transition triple is known.
Our algorithm draws connection to a branch of methods in offline RL literature that performs "filtered" behavioral cloning explicitly or implicitly. More specifically, Peng et al. (2019), Nair et al. (2020) and Wang et al. (2020) estimate an advantage function, which represents the change in expected return when taking action a instead of following the current policy, and perform weighted regression based on the advantage function, defined as Lπ = E(s,a)∼Db [− log π(a|s) · f (Aπ(s, a))]. The advantage Aπ can be estimated by Monte-Carlo methods (Schulman et al., 2017; Peng et al., 2019) or Q-value based methods (Schulman et al., 2015; Nair et al., 2020). The filter function f can be a binary filter (Wang et al., 2020) or an exponential filter (Peng et al., 2019; Nair et al., 2020).
While Chen et al. (2021) and Janner et al. (2021) perform filtered behavioral cloning more implicitly. They abstract offline RL as a sequence modeling problem and use Transformer architecture (Vaswani et al., 2017) to perform credit assignments directly via self-attention. Owing to the memorization power of Transformer in capturing long-term dependencies across timesteps, these methods are able to discard low-quality transitions, do behavior cloning only on high-reward transitions across different trajectories and stitch them together to obtain an optimal trajectory.

heading: 5 EXPERIMENTS
text: We present empirical evaluations of DWBC in a variety of settings. We start with describing our experimental setup, considered datasets and baseline algorithms. Then we evaluate DWBC against other baseline algorithms on a range of robotic locomotion tasks with different types of datasets.
Finally, we analyze the property of the discriminator. We show that a well-trained discriminator can be used to perform offline policy selection (Fu et al., 2020b), which is of independent interest.

heading: 5.1 SETTINGS
text: We construct experiments on both widely-used D4RL MuJoCo datasets and more complex Adroit hand manipulation environment (Fu et al., 2020a). To verify the effectiveness of our methods, we use three setting to generate De and Do. Note that we use ground truth rewards only to perform this data split step and we discard the rewards afterward.
• In setting 1, we use mixed datasets in Mujoco environments. We sort from high to low of all trajectories based on the total reward summed over the entire trajectory. We define a trajectory as well-performing if it is among the top 20% of all trajectories. We then sample every X th trajectory from the well-performing trajectories to constitute De and use the remaining trajectories in the dataset to constitute Do. Note that with X becomes larger, Do will contain more proportion of well-performing data.
• In setting 2, we use expert and random datasets in Mujoco environments. We first sample 10 trajectories from expert datasets and 1000 trajectories from random datasets. We then random sample X trajectories from those 10 expert trajectories and combine them with those 1000 random trajectories to constitute Do, we use the remaining 10−X trajectories to constitute De.
• In setting 3, we use human datasets in Adroit environments. We use the same procedure to constitute De and Do as in setting 1.
We list all datasets used in this paper and the number of trajectories and transitions in De and Do in Appendix C, different X is labeled after the dataset name.

heading: 5.2 BASELINE AND ABLATED ALGORITHMS
text: We compare DWBC with the following baseline algorithms:
BC-exp & BC-all: Behavioral cloning on expert data or on all data. BC-exp is trained only on De, hence it is not exposed to low-performing transitions in Do. On the other hand, it may not be able to generalize due to the limited size of De. BC-all may generalize better than BC-pos due to access to a much larger dataset, but its performance may be negatively impacted by the low-quality data in Do. BCND: BCND is trained on all data, it reuses another policy learned by BC as the weight of BC, its performance will be worse if the suboptimal data occupies the major part of the offline dataset.
ORIL: ORIL learns a reward function and uses it to solve an offline RL problem. It suffers from large computational costs and the difficulty of performing offline RL under distributional shift.
DWBC-noads: We include one ablation of DWBC that trains the discriminator without the adversarial learning machinery (i.e., no log π as input), with all others remaining the same as DWBC.

heading: 5.3 COMPARATIVE EVALUATIONS
text: We show the comparative results in Table 1 and include the learning curves in Appendix C. It can be shown from Table 1 that DWBC outperforms baseline algorithms on most tasks (25 out of 32 tasks), especially in Mujoco datasets (18 out of 20 tasks), showing that DWBC is well suited to make effective use of the expert dataset De and the mixed quality dataset Do. As expected, the performance of BC-exp declines as X becomes larger. This is because that a larger X means the number of well-performing transitions is smaller. In some datasets (e.g., Halfcheetah_exp-rand-6 and Ant_mixed-10), there is no clear winner between BC-exp and BC-all, which suggests that the quality of Do for the considered tasks varies. BCND performs poorly compared to other methods due to the majority of low-quality data in the mixed datasets. It usually scores below BC-all. ORIL struggles to learn in some tasks (especially in the Ant datasets), which suggests their learned reward function does not accurately contrast expert and
non-expert data. We also find that the performance of ORIL tends to decrease in some tasks (e.g., Halfcheetah_mixed-5 and Ant_exp-rand-6), this "overfitting" phenomenon also occurs in the experiments of offline RL papers (Kumar et al., 2019; Wu et al., 2019a). This is perhaps due to the limited data size and model generalization bottleneck (Neyshabur, 2017).
We also find that DWBC-noads performs worse than DWBC, especially when Do contains more expert data, under which circumstance it is hard for the discriminator to distinguish between expert and non-expert data without the help of our proposed adversarial learning machinery.

heading: 5.4 ADDITIONAL EXPERIMENTS
text: Offline policy selection by the discriminator. Offline policy selection (OPS) (Paine et al., 2020; Yang et al., 2020; Dereventsov et al., 2021) considers the problem of choosing the best policy from a set of policies given only offline data. This problem is critical in the offline settings (i.e., offline RL and offline IL) because the online execution is often very costly and safety-aware, deploying a problematic policy may damage the real-world systems (Tang & Wiens, 2021). Note that existing offline RL/IL methods break the offline assumption by evaluating different policies corresponding to their rewards in online environment interactions. However, this online evaluation is often infeasible and hence undermines the initial assumption of offline RL/IL.
In this paper, we propose a novel offline policy selection method, simply by using our learned discriminator d. More specifically, as the discriminator gets state s, action a and the log probability log π(a|s) as input, we can use expert state-action pairs from De and different policy π as input. The discriminator learned by DWBC will assign large values (close to 1) when the evaluated policy is close to the policy learned by DWBC, this means that the policy is more optimal.
To validate our proposed idea, we conduct experiments in Hopper, Walker2d and Pen environment. In Hopper and Walker2d, we use mixed-2, mixed-5 and mixed-10 datasets, in Pen, we use mixed-2, 3 and mixed-5 datasets. We compare three algorithms (BC-exp, ORIL and DWBC) trained in these datasets, total of 9 policies in each environment. We first train DWBC, then we use the learned discriminator along with De to compute the value d(s, a, log πi(a|s)) of each policy πi. We plot average d(s, a, log πi(a|s)) versus the policy’s true return in Figure 2. As shown, the discriminator’s values can well reflect the rank between almost every two policies. This means that we can first train a DWBC policy and then use the trained discriminator d to select the best policy among our candidate policy sets, without executing them in the environment to get the actual returns.
Comparision of run time. We also evaluate the run time of training DWBC and other baseline algorithms for 250 thousand training steps (does not include evaluation run time cost). All run time
experiments were executed on NVIDIA V100 GPUs. For a fair comparison, we use the same policy network size in BC, BCND, ORIL and DWBC. The discriminator network size is also kept the same in ORIL and DWBC. The results are reported in Figure 3. Unsurprisingly, we find the run time of our approach is only slightly more than BC, while other baselines (ORIL, BCND) are over 7 times more costly than BC. The reason that ORIL is costly to run is due to the additional effort to solve an offline RL problem. The high computation cost of BCND is due to its inner iterations of training K policy ensembles (K = 5 in our experiment), which is also mentioned in their paper (Sasaki & Yamashina, 2021). This demonstrates the effectiveness of DWBC by only adds a limited cost to the original BC algorithm while providing substantially improved performance.

heading: 6 CONCLUSION AND LIMITATION
text: In this paper, we propose a new offline imitation learning algorithm that can learn from suboptimal demonstrations without environment interactions or expert annotations. Experimental results show that our algorithm achieves higher returns and faster training speed compared to baseline algorithms, under different scenarios. One limitation of our work is that since our algorithm is based on weighted BC, the covariate shift problem of BC (Ross et al., 2011) will be inherited. That is, there is no way for the policy to learn how to recover if it deviates from the behavior policy to a new state not seen in the demonstrations. In future work, we will consider modifying the main task from action matching to state-action distribution matching, which is known to be more robust to distributional shift (Kostrikov et al., 2019).

heading: A DERIVATION DETAILS
text: In this section, we provide the detailed design intuition and theorectical derivation of DWBC.
A.1 DECOMPOSITION AND REFORMULATION OF LEARNING TASKS
As discussed in Section 3.1, learning from both the expert dataset De and suboptimal dataset Do implies the need of jointly solving two tasks: a BC task and a discriminator task. A straightfoward solution is to learn the two tasks separately, which solves the BC task by imitating the expert demonstrations in De and learn the discriminator via PU learning using data from both De and Do:
BC task: π(a|s)← arg min π LBC
Discriminating task: d(s, a)← arg min d Ld
where LBC and Ld are discussed and given in objectives (2) and (4) in the main article as follows: LBC = E
(s,a)∼De [− log π(a|s)]
Ld = η E (s,a)∼De [− log d(s, a)] + E (s,a)∼Do [− log(1− d(s, a))]− η E (s,a)∼De [− log(1− d(s, a))]
Naïvely solving above two tasks separately is insufficient. First, the BC task only learn from the expert dataset De, fails to utilize the potential valuable information in the supoptimal dataset Do. Second, both tasks lack sufficient information to improve their own performance. For example, a good discriminator could provide important information to distinguish the potential expert samples in the suboptimal dataset De, which are valuable for the learning of policy π; a well-performed policy π(a|s) will assing large probabilities to expert actions under expert states, which could provide additional learning signal for the discriminator d to more easily contrast expert and non-expert transitions in Do.
There are two existing approaches can be used to jointly solve above two tasks, however, both of them have some drawbacks. One approach is to cast the problem into a multi-task style multi-objective optimziation problem, by optimizing an augmented loss βLBC + (1− β)Ld, β ∈ (0, 1) for both π and d. The problem is that the BC task and the discriminating task are different tasks, the potential contradiction of the two tasks in certain settings may impede both tasks from achieving the best perforamnce. Moreover, properly selecting the hyperparameter β is very tricky. Another approach is to adopt a GAN-style model (Goodfellow et al., 2014a) which treats the policy as the generator and optimize it implicitly through solving a min-max optimization problem with the discriminator loss Ld. However, this is very costly and is known to suffer from training instability and issues such as mode collapse (Arjovsky et al., 2017). Moreover, although we have an explict loss function LBC for π, it is not used in such a GAN-style model, which results in potential loss of information.
In this paper, we design an new cooperative learning mechansim to address above issues, which also results in a computationally efficient practical algorithm. It includes three key ingredients: 1) sharing information between the BC task and the discriminating task to achieve cooperative learning; 2) enabling the BC task to learn on both expert and suboptimal data by introduce an additional corrective loss Lw impacted by the discriminator outputs; 3) solving both tasks in fully supervised learning manner to maintain computational efficiency. In our approach, we consider following alternative formulation to establish information sharing across the two tasks and enable cooperative learning:
BC task: π(a|s)← arg min π αLBC + Lw, α > 1
Discriminating task: d(s, a, log π(a|s))← arg min d Ld
(10)
with the new Ld given in objective (5) as follows: Ld =η E
(s,a)∼De [− log d(s, a, log π(a|s))] + E (s,a)∼Do [− log(1− d(s, a, log π(a|s)))]
− η E (s,a)∼De [− log(1− d(s, a, log π(a|s)))]
In above reformulation, we design the information provided by the policy to discriminator as the element-wise imitation loss value log π(a|s), and the information provided to the policy as the
addional corrective loss term Lw computed using output values of the discriminator d on samples from both De and Do. With the involvement of Lw, we can allow the policy π learning from sub-optimal data under the guidance of the discriminator. Moreover, to more robustly learn the discriminator d, we borrow the idea of adversairal learning (Lowd & Meek, 2005) and make the policy π challenge d by providing as little useful information in log π(a|s) as possible. We will show in the next section that with the choice of element-wise imitation loss log π(a|s) as the form of information and the advasarial behavior of policy π, an exact form of Lw can be derived, and eventually, transforms the original BC task into a cost sensitive learning problem.
A.2 DRIVATION OF THE CORRECTIVE LOSS TERM Lw
In this section, we resort to functional analysis and calculus of variation to derive the exact form of Lw. Under the reformulated problem (10), both the discriminator d and its loss Ld are impacted by the information provided by policy π (log π(a|s)). Hence they are now functional of π. We are interested to see how the variation of π impacts Ld, and further influence d. Moreover, we can use a specific form of Lw to alter the behavior of the learned π to achieve the desired adversarial behavior. Denote θπ and θd as the parameters of the policy π and discriminator d respectively. As discussed in Section 3.2, to analyze the impact of π on d, we fix the parameters θd of d to exclude its own influence. The function of the discriminator d(θd, s, a, log π(a|s)) can thus be replaced by d(s, a, log π(a|s)). Similarly, with θd fixed, we can define a functional J(π) for Ld which depends on function π and further eliminate the effect of changes in s, a on d by integrating over s and a as in Eq.(6):
J(π) = ∫ ∫ ∂Ld(s, a, d, log π(a|s)) ∂d(s, a, log π(a|s)) ds da = ∫ ∫ F (s, a, π, π′) ds da,
where F = ∂Ld/∂d is assumed to be a continuously differentiable function with respect to s, a, π and π′ (derivative of π). Note that as θd is fixed and not considered as a variable in our analysis, θd no longer contributes gradients in the partial derivative ∂Ld/∂d, and functional F is given as follows:
F (s, a, π, π′) =− E (s,a)∼De
[ η
d(s, a, log π(a|s))
] + E
(s,a)∼Do
[ 1
1− d(s, a, log π(a|s)) ] − E
(s,a)∼De
[ η
1− d(s, a, log π(a|s))
] ,
(11)
To enforce the adversarial behavior of π, we want to make π challenge the discriminator d by maximizing Ld under current d (doing the opposite to d). By doing so, the policy will provide as little information in log π as possible such that minimizing Ld becomes harder for the discriminator. This can be seen as minimizing the worst-case error, which makes the robustness of the discriminator significantly improved.
Maximizing Ld for π under current d is equivalent to finding the maxima of functional J(π) (Ld with θd and d fixed). According to the calculus of variations (Gelfand et al., 2000), the extrema (maxima or minima) of functional J(π) can be obtained by finding a function π where the functional derivative is equal to zero. We can show with following proposition that ensuring the functional derivative of J(π) equal to zero requires ∂F/∂θπ = 0. Proposition 1. With the parameters θd of the discriminator d fixed, the functional J(π) and F (s, a, π, π′) defined as in Eq.(6) and (11), if uniform continuity of both F and its derivative is satisfied, the necessary condition for J(π) attaining its extrema is ∂F (s, a, π, π′)/∂θπ = 0.
Proof. We derive the finite difference approximation of the functional derivative of J(π) by Taylor expansion,
J(π) = J(πθ0π ) + J ′(πθ0π )δπ + o(δπ), (12)
where δπ is the variation of π. As o(δπ) is a higher order infinitesmall term when θπ → θ0π, which can be ignored. Plug Eq.(12) into the form of J(π), we obtain the finite difference approximation of J(π) as follows:
∆J(π) = J(π)− J(πθ0π ) = J ′(πθ0π )δπ =
d ∫ ∫ F (s, a, π, π′) ds da
dθπ δπ. (13)
If the uniform continuity of both F and its derivative is satisfied, we can swap the order of integration and differentiation as well as ds da and δπ. The above equation can be simplified as:
∆J(π) =
∫ ∫ ∂F (s, a, π, π′) ds da
∂θπ δπ =
∫ ∫ ∂F (s, a, π, π′)
∂θπ δπ ds da. (14)
As π can be any function in some function space F . Ensuring the functional derivative of J(π) equal to 0 also requires ∂F (s, a, π, π′)/∂θπ = 0.
Given the form of functional F in Eq.(11), we can derive the detailed form of ∂F/∂θπ as presented in Eq.(7) of the main article:
∂F (s, a, π, π′) ∂θπ = ∂F (s, a, π, π′) ∂ log π · ∂ log π ∂π · ∂π ∂θπ = ∂F (s, a, π, π′) ∂ log π · ∇θπ log π
=− E (s,a)∼De [η d · ∇θπ log π(a|s) ] + E (s,a)∼Do [ 1 1− d · ∇θπ log π(a|s) ] − E
(s,a)∼De
[ η
1− d · ∇θπ log π(a|s)
] ,
where in the last equation, we slightly abuse the notations and write the output value of d(s, a, log π(a|s)) as d for simplicity. As discussed in Section 3.2, above derivative can be equivalently perceived as the graident of a new loss term for π, which is exactly the corrective loss term Lw that we are looking for. If we set ∂Lw/∂θπ = −∂F/∂θπ , we recover Lw in following form:
Lw = E (s,a)∼De
[ log π(a|s) · η
d
] − E
(s,a)∼Do
[ log π(a|s) · 1
1− d
] + E
(s,a)∼De
[ log π(a|s) · η
1− d ] The introduction of the minus sign on ∂F/∂θπ is to ensure we find the maxima of J(π) rather than its minima. Hence minimizing Lw with respect to π (make ∂Lw/∂θπ = −∂F/∂θπ = 0) will drive the functional derivative of J(π) to zero, which also lead to the maxima of J(π) (update with gradient direction −∂F/∂θπ). Adding the new corrective loss term Lw back to our reformulated problem (10), we obtain the final learning objective of π for our BC task (Eq.(8) in the main article):
min π α E (s,a)∼De [− log π(a|s)]− E (s,a)∼De
[ − log π(a|s) · η
d (1− d)
] + E
(s,a)∼Do
[ − log π(a|s) · 1
1− d ] Note that the derivation of Lw requires uniform continuity to be satisfied in F and its derivative. The involvement of discriminator output values 1/d(s, a, log π(a|s)) and 1/(1− d(s, a, log π(a|s)) may violate the continuity assumption. We thus clip the discriminator output values to the range of [0.1, 0.9] in our practical algorithm.

heading: B TRAINING PROCEDURE DETAILS
text: B.1 ALGORITHM DETAILS
In this section, we present the pseudocode of DWBC in Algorithm 1.
B.2 IMPLEMENTATION DETAILS
In this paper, all experiments are implemented with Tensorflow and executed on NVIDIA V100 GPUs. For all function approximators, we use fully connected neural networks with RELU activations. For policy networks, we use tanh (Gaussian) on outputs. We use Adam for all optimizers. The batch size is 256 and γ is 0.99. We rescale the reward to [0, 1] as r′ = (r − rmin) / (rmax − rmin), where rmax and rmin is the maximum and the minimum reward in the dataset. Note that any affine transformation of the reward function does not change the optimal policy of the MDP. The policy and discriminator network are all 3-layer MLP with 256 hidden units in each layer. The learning rate for the policy is 1e− 5 and the learning rate for the discriminator network is 1e− 4. We search α in {1, 2, 5, 10} for best model performance. We clip the output of d to [0.1, 0.9]. We set η to 0.5 across all tasks, which is the same as the ORIL paper (Zolna et al., 2020a).
Algorithm 1 Discriminator-Weighted Behavior Cloning (DWBC) Require: Dataset De and Do, hyperparameter η, α 1: Initialize the imitation policy π and the discriminator d 2: while training do 3: Sample (se, ae) ∼ De and (so, ao) ∼ Do to form a training batch B 4: Compute log π(a|s) values for samples in B using the learned policy π 5: Compute discriminator output values d(s, a, log π(a|s)) using sampled (s, a) and computed log π(a|s) 6: Update d by minimizing the learning objective in Eq.(5) 7: Update π by minimizing the learning objective in Eq.(8) 8: end while

heading: C ADDITIONAL RESULTS
text: C.1 DATASETS DETAILS
In this section, we list all datasets used in our paper and the number of trajectories and transitions in De and Do, different X is labeled after the dataset name.
C.2 LEARNING CURVES
In this section, we provide the learning curves of listed algorithms in Section 5. As the learning procedure of BC is quite fast and stable, for more clearly presentation, we plot the results of BC-exp and BC-all as a horizon bar with the shaded area as the standard deviation across different seeds. The average return in De is plotted as the red dashed line in all plots.

Review summaries:
The authors consider the problem of offline imitation learning in the presence of suboptimal datasets. In the presence of suboptimal data, classical baselines like behavior cloning suffer performance hits, the drop in performance often correlates positively with increase in number of suboptimal trajectories. In this work, the authors propose a novel learning objective inspired by the min-max formulation in GANs. Particularly the agent learns a discriminator to distinguish between samples from the expert and the suboptimal demonstrator. Building on prior work in cost-sensitive learning, this discriminator is used to reweight loss per sample in the offline buffer.   The proposed algorithm is evaluated on standard offline RL benchmarks, across multiple environments. In many environments, e.g Hopper-v2 style environments, the policy improves against strong imitation learning benchmarks. The discriminator (which takes action probabilities as input) is also evaluated in the context of offline policy evaluation. On Hopper-v2 datasets, the discriminator output is compared with true reward accumulated by multiple policies.

The paper proposes a new offline imitation learning algorithm, DWBC, for datasets that combine both optimal and suboptimal demonstrations. The approach is based on a modified behavioral cloning loss that weighs expert and non-expert data based on a learned discriminator. DWBC is compared against prior methods in OpenAI Gym tasks, and it is shown to yield better policies compared to the prior work. As a by-product, the method learns a discriminator that can be used to estimate the relative performance of any policies without rolling them out in the environment.

This paper proposes an offline imitation learning framework that incorporates both optimal and suboptimal datasets to learn decision-making tasks, without requiring any reward annotations. To leverage high reward transitions from the suboptimal dataset, the authors formulate a discriminator that optimizes a positive-unlabeled learning objective, where positive samples come from the optimal dataset and unlabeled samples come from the suboptimal dataset. This discriminator is trained in an adversarial fashion along with the policy, resulting in a behavior cloning objective where samples from the optimal and suboptimal datasets are weighted differently according to the discriminator’s predictions. Experiments demonstrate that on a set of simulated locomotion domains, the proposed algorithm can leverage the suboptimal dataset to learn more performant policies compared to vanilla behavior cloning objectives and prior offline IL/RL baselines.

The paper deals with the following setup: offline imitation learning in the presence of both an expert dataset and a non-expert dataset. More precisely, the goal is to learn a policy as close as possible to the one(s) that generated the samples in a dataset $D_e$, while making the most of samples in a non-expert dataset $D_o$. The “reward” information is not present/used in the dataset. The authors draw inspiration from the positive-unlabeled classification as well as the adversarial imitation learning literature to propose a new algorithm to tackle this problem. They interleave the training of a discriminator and a policy. The discriminator is trained to discriminate between expert and non-expert dataset (using a positive/unlabeled loss) and takes as input the state, the action and the logit of the policy $\pi(a | s)$. The policy is trained to imitate the expert on $D_e$ and to “fool” the discriminator.   The authors present results on four environments from the Gym Mujoco suite with datasets extracted from the D4RL datasets.