Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations
Authors: Haoran Xu
Xianyuan Zhan
Honglei Yin
Huiling Qin
Venue: ICLR 2022 Submitted (2022)
Keywords: imitation learning
offline imitation learning

Abstract: We study the problem of offline Imitation Learning (IL) where an agent aims to learn an optimal expert behavior policy without additional online environment interactions. Instead, the agent is provided with a static offline dataset of state-action-next state transition triples from both optimal and non-optimal expert behaviors. This strictly offline imitation learning problem arises in many real-world problems, where environment interactions and expert annotations are costly. Prior works that address the problem either require that expert data occupies the majority proportion of the offline dataset, or need to learn a reward function and perform offline reinforcement learning (RL) based on the learned reward function. In this paper, we propose an imitation learning algorithm to address the problem without additional steps of reward learning and offline RL training for the case when demonstrations containing large-proportion of suboptimal data. Built upon behavioral cloning (BC), we introduce an additional discriminator to distinguish expert and non-expert data, we propose a cooperation strategy to boost the performance of both tasks, this will result in a new policy learning objective and surprisingly, we find its equivalence to a generalized BC objective, where the outputs of discriminator serve as the weights of the BC loss function. Experimental results show that the proposed algorithm can learn behavior policies that are much closer to the optimal policies than policies learned by baseline algorithms.

Under review as a conference paper at ICLR 2022
DISCRIMINATOR-WEIIO LEARNING FROM SUBOPTIMAL DEMONSTRATIONS
OFFLINE IMITATION
Anonymous authors Paper under double-blind review
ABSTRACT
We study the problem of offline Imitation Learning (IL) where an agent aims to learn an optimal expert behavior policy without additional online environment interactions. Instead, the agent is provided with static offline dataset of state- action-next state transition triples from both optimal and non-optimal behaviors. This strictly offlin imitation learning problem arises in many real-world problems, where environment interactions and expert annotations are costly. Prior works that address the problem either require that expert data occupies the majority proportion the offline dataset, or need to learn a reward function and perform offline reinforcement learning (RL) based on the learned reward function. In this paper, we propose an imitation learning algorithm to address the problem without additional steps of reward learning and offline RL training for the case when demonstrations containing large proportion of suboptimal data. Built upon behavioral cloning (BC), we introduce an additional discriminator distinguish expert and non-expert data, we propose a cooperation strategy to boost the performance of both tasks, this will result in a new policy learning objective and surprisingly, we find its equivalence a generalized BC objective, where the outputs of discriminator serve as the weights the BC loss function. Experimental results show that our proposed algorithm achieves higher returns and faster training speed compared to baseline algorithms.
INTRODUCTION
The recent success of reinforcement learning (RL) in many domains showcases the great potential of applying this family of learning methods to real-world applications. key prerequisite for RL is to design reward function that specifies what kind of agent behavior is preferred. However, in many real-world applications, designing a reward function is prohibitively difficult (Ng et al.) 1999 Irpan 2018). By contrast, imitation learning (IL) provides much easier way to leverage the reward function implicitly from the collected demonstrations and has achieved great success in many sequential decision making problems (Pomerleau 1989 Ng al. 2000 Ho & Ermon 2016).
However, popular IL methods such as behavioral cloning (BC) and generative adversarial imitation learning (GAIL) (Ho & Ermon 2016), assume the expert demonstration is optimal. Unfortunately, it often difficult to obtain optimal demonstrations for many real-world tasks, because human experts often make mistakes due to various reasons, such as the difficulty of the task, partial observability of the environment, or the presence of distraction. Given such noisy expert demonstrations which contain records of both optimal and non-optimal behaviors, BC and GAIL all fail to imitate the optimal policy (Wu et al. 2019b Ma 2020). Current methods that deal with suboptimal demonstrations either require additional labels, which can be done explicitly by annotating each demonstration with confidence scores by human experts (Wu et al. 2019b). or implicitly by ranking noisy demonstrations according to their relative performance through interacting with the environment (Brown et al.) 20199 2020). However, human annotation and environment interaction are laborious and expensive in real-world settings, such as medicine, healthcare, and industrial processes.
In this work, we investigate pure offline learning setting where the agent has access to neither the expert nor the environment for additional information. The agent, instead, has only access to a small pre- collected dataset of state-action-ner state transition triples sampled from the expert and a large batch offline dataset sampled from one or multiple behavior policies that could be highly sub-optimal. Prior works that address the problem are based variants of BC inverse RI., Sasaki

---

Under review as a conference paper at ICLR 2022
& Yamashina (2021) reuse another policy learned by BC the weight of original BC objective, however, this requires that expert data occupy the majority proportion of the offline dataset, otherwise the policy will be misguided to imitate the suboptimal data. [Zolna et al. (2020a) first learns reward function that prioritizes expert data over others and then performs offline RL based on this reward function. This algorithm extremely expensive to run, requiring solving offline RL in an inner loop, which itself is a challenging problem and prone to training instability (Kumar et al. 2019) and hyperparameter sensitivity (Wu et al. 2019a).
In this paper, we propose an offline imitation learning algorithm to learn from demonstrations that (perhaps) contain large proportion of suboptimal data without additional steps of reward learning and offline RL training. Built upon the task of behavioral cloning (BC), we introduce an additional task to learn discriminator to distinguish expert and non-expert data, we propose a cooperation strategy to boost the performance of both tasks. This results in new policy learning objective and surprisingly. we find its equivalence to a generalized BC objective, where the outputs of the discriminator serve as the weights of the BC loss function. We thus term resulting algorithm Discriminator- Weighted Behavioral Cloning (DWBC), Experimental results show that DWBC achieves higher returns and faster training speed compared to baseline algorithms, under different scenarios.
2 PRELIMINARY
2.1 PROBLEM SETTING
We consider the standard fully observed Markov Decision Process (MDP) setting (Sutton et al.
1998).
M ((S.A. A. P.r.7 7 do), where S is the state space, A is the action space, P S x A (
the MDP's transition probability, r is the reward function, y E [0, 1) is the discount factor for future reward and do is the initial distribution. A policy
S + (.A) maps from state to distribution over actions. We denote d= E ASS x AA) as the discounted state-action distribution of π under transition kernel P, that is, d= = (1 - 7) Ei= y'd,,, where df A(S A) is the distribution of (s(",, al under ı at step t. Following the standard IL setting, the ground truth reward function is unknown. Instead, (potentially stochastic and not necessarily optimal). Concretely, we have an expert dataset in the form of i.d tuples D. {s;, a, " where (s, a) sampled from distribution d and s sampled from P(s.a).
have the demonstrations by the expert specified by me S - A(A)
In our problem setting, we also have an offline static dataset consisting of i..d tuples D, {s; 04, &1 s.t. (s,a) p(s, a). s' ~ P(s. a), where p E ASS x A) is an offline state-action distribution resulting from some other behavior policies. Note that these behavior policies could be much worse than the expert Te. Our goal is to only leverage the offline batch data D, DD_ U D, to learn an optimal policy æ with regard to optimizing the ground truth reward r, without any interaction with the environment or expert.
2.2 A GENERALIZED BEHAVIORAL CLONING OBJECTIVE
In order to discard luw-quality demonstrations and only clone the best behavior available, previous works (Sasaki & Yamashina 2021 Zolna et al. 2020a) consider generalized behavioral cloning objective to imitate demonstrations unequally. that is,
min E [- log (als) (s)
(s
where /:S x A [0, denotes an arbitrary weight function.
If f(s,a) for a)) S A, the objective | corresponds to the vanilla BC objective. If f(s,a) n' (a)ss, where T' is an old policy which was previously optimized with D., the objective (1) corresponds the objective Behavioral Cloning from Noisy Demonstrations (Sasaki) & Yamashina 2021). Since Σ. T (a]s) = for V's E S is satisfied, n' r'(als) can be interpreted as the weights for the weighted action sampling.
If as) 1 (s, a)), where 1 is the indicator function which creates boolean mask that elim- inates samples which are thought to be worse than the current policy, the objective (T) corresponds the objective of Offline Reinforced Imitation Learning (Zolna et al. 2020a).
2

---

Under review as a conference paper at ICLR 2022
The objective [ can also be deemed as the objective of Soft Q Imitation Learning (Reddy et al. 2019) with fs..a) = 1 for (s.a) E De and f(s. a) = for (s.a) E D, in online IL literature; or the objective of off off-policy actor-critic (Off-PAC) algorithm (Degris et al. 2012) with f(s,, a) Q (s, a) ग (als)/ (aals/////(aa))) in online RI. literature.
3 DISCRIMINATOR- WEIRRTEEEE BEHAVIORAL CLONING
We now continue to describe our approach for offlin imitation learning from demonstrations that (perhaps) contain large-proportionl suboptimal data without additional steps of reward learning and offline RL training. Built upon the task of BC, we introduce an additional task to learn a discriminator to distinguish expert and non-expert data, we propose a cooperation strategy to boost the performance of both tasks, this will result in new generalized BC objective. We then provide the interpretation of weights in our generalized BC objective, this gives the intuition about why our method can work.
3.1 LEARN THE POLICY AND DISCRIMINATOR SEPARATELY
is obvious that we can avoid the negative impact of suboptimal demonstrations presented Do by only imitating De, which can be written as
min E (s,a)~D.
x lllgg((ass))
We call the task of learning a policy using objective as the BC task. The drawback of BC task that the learned policy may not be able to generalize due to the potential limited size and state coverage of De. It does not fully utilize the information from D.. If we select those high-reward transitions from DD, and combine them with De., we are expected to get a better policy.
Now let's consider another different task, which aims learn discriminator by contrasting expert and non-expert transitions, given by
min E d' (s,0)~DD
[-log d(ss, a)l +
E (s.a)~Da
[- - d(s, a))
(3)
We call this the discriminating task, which is similar to how the discriminator is trained in GAIL (Ho & Ermon 2016) and GAN (Goodfellow et al. (0144)), except that the second term is sampled from fixed dataset instead of new samples obtained from interacting with the environment.
However, optimizing objective (3) will make the discriminator learned to assign 1 to all transitions from D. and 0 to all transitions from D,, This limiting behavior is unsatisfactory because D, can contain some successful (high-reward) transitions. This bears similarity the positive-unlabeled classification problem (Elkan & Noto 2008), where both positive and negative samples exist in the unlabeled data.
To solve this problem, we adopt the approach from positive-unlabele (PU) learning (Du Plessis al. 2013] Xu & Denil 2019 Zolna et al. 2020b). The main idea is to re-weight different losses for positive and unlabeled data, in order to obtain an estimate of model loss on negative samples that is not directly available. Applying PU learning to objective (3) yields the following objective:
min 17 (s.a)~D,
|- log d)s
E (s.a)~D,
[- log(1 d(s,a))) 7
E (s.a)>DD.
|- log(1 a)s) (4)
where 17 is a hyperparameter, corresponds the proportion of positive samples to unlabeled samples. Intuitively, the second term in (4) could make d(s a) of state-action pairs from D. become 0 if similar state-action pairs are included D... and the third term (4) balances the impact of the second term, i.c., avoids d(s, a) of state-action pairs from D. becoming O.
To summarize, the BC task aims to imitate the expert behavior from D., but ignores the valuable information in D.; the discriminating task aims to contrast expert and non-expert transitions from De and D.. by only using state-action information as input. Both tasks lack enough information to improve their own performance, which. however, natural to find a scheme to incorporate the policy into the training of the discriminator and effectively the discriminator help the training of the policy.
be obtained from the other task. It then seems
3

---

Under review as a conference paper at ICLR 2022
3.2 LEARN THE POLICY AND DISCRIMINATOR COOPERATIVELY
We propose a cooperation strategy to boost the performance of both tasks. To boost the performance of the discriminating task, add the imitation information from the input of the discriminator, yielding the following discriminator learning objective La to be minimized as:
min La = min 1 d
E |- log d(s,, a, log (a]b)) +
d
(s,a)~DD
I- - d(s, logs)()s)))) (5)
E [- log(11 - d(ss, a, log - (s.a)~>,
(s.a)~-o
Supposed it is learned to be optimal, i.e., assigns large probabilities to expert actions in expert states, the discriminator will receive additional learning signal. It will be easier for the discriminator to contrast expert and non-expert transitions in Do. as log Taass) will be large if (s, a) are from expert behaviors and small if (s, a) are from non-expert behaviors. Without this information from x, the discriminator is much harder to learn by only using information from (s, a).
Now let's dive deeper into objective ( As iT now appears in the imput of the discriminator d, hence both d and its loss La become functionals of и. We are interested to see how does the imitation information from log æ affect L., and further impact d. In other words, given the current d, we want to explore how to change the behavior of such that d can be better learned. To achieve this goal, we define a functional (TT) for La with the function π as the variable, and fix the parameter Og of d to exclude its own influence. Note that in the functional J (rr, d is now fixed and its parameter 0a will no longer be considered as a variable. J(T) can be formally define as the following integral form to further eliminate the effect of changes in 8, d:
J(x) J(x)= ʃ @Ld(s, d, log TT) Öd(s, a, log
F(s, n,, n') ds da = F
ds da,
where we denote F= aLa/ /Od and F is assumed to be continuously differentiable with respect to 8, a, and T' (derivative of 1). To more robustly learn the discriminator d, inspired by the idea of adversarial learning (Lowd Meek 2005), we make the policy π challenge the discriminator by doing the opposite to the task of (i.e., minimize Ca), in other words, want La under current d. By doing so, the policy will provide as little information in log π as possible such that minimizing La becomes harder for the discriminator, this can be seen as minimizing the worst-case error (Carlini et al., 2019 Fawzi et al. 2016 Goodfellow et al. 2014b), which makes the robustness of the discriminator significantly improved.
maximize
To maximize Ca for æ under current d, we let the functional ((T) attains its maxima with respect to т. According to the calculus of variations (Gelfand al., 2000), the extrema (maxima or minima) of functional (Tz) can be obtained by finding a function ग such that the functional derivative of J(x) is equal zero. We can show with theoretical derivation (see Appendix A.2 that ensuring the functional derivative of (TT) equal to zero requires OF /00 = O, where 0, Is the parameters of π.
By the chain rule, we can have
OF
OF
Ve- log
E (e.0)~De
? l
Ve, log logs(a|s) +
00,
log π
E
1- Vor log (als) (s,0)~D,
1
" 1-d
Vo_ log (((s)
(s,a)~D,
where we write the output value of d(ss, a, log (als)) as d for simplicity by slightly abusing notations. Notice that above derviative can be equivalently perceived as the gradient of new loss term C. of the policy π (dCw/ 000 = =0FF/00, with following form:
L
E (s,a)-De
log (cls) 21
log Tals) (s,a)~D.
E
+ E (s.a)<D.
log logm(als) i-d
1 i-
Hence minimizing Lu with respect to ा (make /., = 0) will drive the functional derivative of J(x) (r) to zero, which leads to the maxima of J(x) (update with gradient direction - -DF/a0=). Adding this loss term fy to the BC task, we get the following new learning objective of π as:
min a E |- log [[ass)] (s.a)~D. (s,a)>D.
log ((ass)
"
+
log π (als)
d(1 - d)
(a.a)DD
1
(8)
4

---

Under review as a conference paper at ICLR 2022
BC policy ((ass)
Epc
Luy
Imitation loss
Corrective loss
Discriminator d(s, a, log
Ld
De/Do
Figure 1: Illustraiion of DWBC learning mechanism
where is the weight factor (a 1). This new objective essentially transforms the original BC task into a cost-sensitive learning problem (Ling & Sheng 2008) by imposing the following weight on imitating each state-action transition
aa- 7/d(1 -d), 11 d),
(s,a) E De (8, () E D.
Behavioral cloning weights
Note that the derivation of the above behavioral cloning weights requires uniform continuity to be satisfied in F and its derivative (details see Appendix A.2). The involvement 1/d and 1/(1 - -d) may violate the continuity assumption. We thus clip the value the range of [0.1, 0.9].
Above behavioral cloning weights induce different behaviors on the imitation of transitions from De and D., Supposed d is learned to be optimal, i.e., assigns large values (close to expert transitions and small values to non-expert transitions (close to O). The weight of those expert transitions in Do will become large while the weight of those non-expert transitions will become small. For transitions in De, their weights can be adjusted by tuning the parameter a. Note that even if the discriminator learned to be totally wrong (i.e., assign small values expert transitions and large values to non-expert transitions), which may occur at the very beginning of training, the behavior cloning weights a - 1//d(1 - d) (a 2 1, 1. < 1) will not be drastically changed under value clipping. This means that the policy can still learn from the expert dataset D.. Even though the weight for D. is temporarily incorrect, it will be corrected as the discriminator becomes better and better.
Eq. (8) implies that our approach is also variant of generalized behavioral cloning objective, as discussed in section 2.2 but uses a different form of weights. Unlike Offline Reinforced Imitation Learning (Zolna et al. 2020a), which uses the discriminator the reward and learns value function as the weight, our approach uses the discriminator outputs directly as the weight. This can greatly reduce the training time and avoid the overestimation issue in estimating the value function offline (Kumar et al. 2019). We thus term our algorithm Discriminator- Weighted Behavioral Cloning (DWBC).
keen reader may find the similarity of our approach to the adversarial learning in GAN (Godffelow al. (2014)), which makes the generator (in our case is the policy т) and the discriminator d learn against each other, thus improves the performance of both tasks. However, the learning strategy of DWBC has several differences compared with GAN and is also much easier learn. constrast to the fully adversarial setting in GAN, DWBC adopts a semi-cooperative strategy that shares information between the policy π and the discriminator to improve the performance of both parties. Moreover, GAN needs to solve min-max optimization problem and is known to suffer from training instability and issues such as mode collapse (Arjovsky et al. 2017). 7). In DWBC, we solve the min-max optimization problem implicitly by transforming the max optimization problem to new learning loss imposed on the policy. We present an illustration the learning mechanism of DWBC in Figure
RELATED WORK
4.1 OFFLINE IMITATION LEARNING
Offline II, which has received considerable attention, is promising area because it makes IL more practical to satisfy critical safety desiderata. Offline IL methods can be typically folded into two paradigms: Behavioral Cloning (BC) and Offline Inverse Reinforcement Learning (Offline IRL).
BC (Pomerleau) 1989) is the simplest method that can be used in the offline setting, considers the policy conditional distribution лt (-|s)) over actions, recent work (Florence al. 2021)
5

---

Under review as a conference paper at ICLR 2022
enhances BC by using energy- baaeg models (LeCun al. 2006). BC has shown to have inferior performance compared to popular IL. algorithms such as GAIL (Ho & Ermoo (016) when clean expert demonstrations are available (Mal 2020). Unlike BC, offline IRL considers matching the state-action distribution induced by the expert policy, this can be achieved implicitly by adversarial training or explicitly by learning reward function. Offline IRL. algorithms based adversarial training (Kostrikov et al., 2019 [Snn ct al. [021]; Swamy et al. 2202 Jarboui & Perchet, (202I) use Intergral Probability Metries (IPMs) (Sriperumbudur
al.] 2009)
distance measure to solve
the dual problem. They introduce a discriminator and aim to find the saddle point of a min-max optimization problem, like GAN (Goodfellow et al. 2014a). [Jarrttt al. (2020) avoids the need of min-max problem by fixing the policy to be energy- -ased models, in such case the KL divergence from the demonstrator's 's state-action distribution that of the policy can be computed in closed form. However, recent work finds several fundamental mathematical misconceptions in their proposed approach and we refer the reader to Swamy et al. (202I) for more details.
The common problem of these works is that they imitate equally to all demonstrations, this will hinder the performance the demonstrations contain suboptimal data. To solve (Saaai & Yamashina (2021) reuses another policy learned by BC the weight of original BC objective, however, this requires that expert data occupies the majority proportion of the offline dataset, otherwise the policy will be misguided to imitate the suboptimal data. Zoolaa et al. (2020a) and KndKuussskoova et al. (2020) first construct a reward function that discriminates expert and exploratory trajectories, then use it to solve an offline RL problem. Instead of the adversarial learning scheme, the reward function can also be learned by cascading to two supervised learning steps (Klein et al. (013). However, offline IRL based on reward learning extremely expensive to run, requiring solving offline RL in inner loop, which itself is a challenging problem and prone to training instability (Kumar et al. (019) and hyperparameter sensitivity (Wu et al. 2019a). Our algorithm can be seen as a combination of these two algorithms in that we use generalized BC objective to imitate demonstrations selectively and we train a discriminator to distinguish expert and non-expert data and use the output of the discriminator as the weight of the generalized BC objective. There is also one recent work (Chang et al. [021] that tries to solve the offline IL problem by adopting techniques from pessimistic model- baseed offline
policy learning Yu et al. 2020 2021
4.2 OFFLINE REINFORCHMENT LEARNING
One research area highly related to offline IL is offline RL (Lange et al. 2012; Levine et al., (020). which considers performing effective RL by utilizing arbitrary given, static offline datasets, without any further environment interactions. Note that in offline RL, the training dataset is allowed to have non-optimal trajectories and the reward for each state-action-nexx state transition triple is known.
Our algorithm draws connection to branch of methods in offline RL literature that performs "filtered" behavioral cloning explicitly or implicitly. More specifically, Peng et al. 2011), Nair et al. (2020) and Wang et al. (2020) estimate advantage function, which represents the change in expected return when taking action a instead of following the current policy, and perform weighted regression based on the advantage function, defined as L. (s,))=,, [- log (a) s) (AT (s. a))]. The advantage
AT can be estimated by Monte-Carle methods (Schulman et al. 2017 Peng et al. 2019) or Q-value
based methods (Schulman et al. 2015 Nair et al., 2020). The filter function can be binary
filter (Wang et al. (020) or an exponential filter (Peng et al. 2019 Nair et al. 2020).
While Chen al. (202 and and Janner et al. (2021) perform filtered behavioral cloning more implicitly. They abstract offline RL as a sequence modeling problem and use Transformer architecture (Vaswani al. (017) to perform credit assignments directly via self-attentio.. Owing to the memorization power of Transformer in capturing long-term dependencies across timesteps, these methods are able to discard low-quality transitions, do behavior cloning only on high-reward transitions across different trajectories and stitch them together to obtain an optimal trajectory.
5 EXPERIMENTS
We present empirical evaluations of DWBC in variety of settings. We start with describing our experimental setup. considered datasets and bascline algorithms. Then we evaluate DWBC against other baseline algorithms range of robotic locomotion tasks with different types of datasets.

---

Under review as a conference paper at ICLR 2022
Finally, we analyze the property of the discriminator. We show that well-trained discriminator can be used to perform offline policy selection (Fu et al. 2020b), which is independent interest.
5.1
SETTINGS
We construct experiments on both widely- widely-used used D4RL MuJoCo datasets and more complex Adroit hand manipulation environment (Fu et al. 2020a). To verify the effectiveness of methods, three setting to generate De. and D... Note that we use ground truth rewards only to perform this data split step and we discard the rewards afterward.
In setting all trajectories based on the total reward summed over the entire trajectory. We define a trajectory well-performing it is among the top 20% of all trajectories. We then sample every Xth trajectory from the well-performing trajectories to constitute Dr and use the remaining trajectories in the dataset to constitute D., Note that with X becomes larger, Do will contain more proportion of well-performing data.
mixed datasets in Mujoco environments. We sort from high to low of
In setting 2, we use expert and random datasets in Mujoco environments. We first sample trajectories from expert datasets and 1000 trajectories from random datasets. We then random sample X trajectories from those 10 expert trajectories and combine them with those 1000 random trajectories to constitute Do, we use the remaining 10 trajectories to constitute De.
In setting 3, we use human datasets in Adroit environments. We use the same procedure to constitute De and DD as in setting 1.
We list all datasets used in this paper and the number of trajectories and transitions in D. and D, in Appendix C, different X is labeled after the dataset name.
5.2 BASELINE AND ABLATED ALGORITTM
We compare DWBC with the following baseline algorithms:
Behavioral cloning on expert data or on all data. BC-exp is trained only on De. hence it is not exposed low-performing transitions in D,, On the other hand, it may not be able to generalize due to the limited size of De-- BC-all may generalize hetter than BC-pos due to access to a much larger dataset, but its performance may be negatively impacted by the low-quality data in D,-.
BC-exp & BC-all:
BCND: BCND is trained on all data, it reuses another policy learned by BC as the weight of BC, its performance will be worse if the suboptimal data occupies the major part of the offline dataset.
ORIL: ORII. learns a reward function and uses it to solve an offline RL problem. It suffers from large computational costs and the difficulty of performing offline RL under distributional shift.
DWBC-noads: We include one ablation of DWBC that trains the discriminator without the adversarial learning machinery (i.e., log input), with all others remaining the same DWBC.
5.3 COMPARATIVE EVALUATIONS
We show the comparative results in Table I and include the learning curves in Appendix C. It can be shown from Table D that DWBC outperforms baseline algorithms on most tasks (25 of 32 tasks), especially in Mujoco datasets (18 out of 20 tasks), showing that DWBC is well suited to make effective use of the expert dataset D. and the mixed quality dataset D..
As expected, the performance of BC-exp declines as X becomes larger. This is because that a larger X means the number of well-performing transitions is smaller. In some datasets (e.g., Hal wwwheehhhheee.e sarehheeep-nndd--ee rand-6 and ment_mmixed111 10), there is no clear winner between BC-exp and BC-all, which suggests that the quality of D. for the considered tasks varies. BCND performs poorly compared to other methods due to the majority of low- low-quality data in the mixed datasets. usually scores below BC-all. ORIL struggles to leam in some tasks (especially in the Ant datasets), which suggests their learned reward function does not accurately contrast expert and
7

---

Under review as a conference paper at ICLR 2022
Table 1: Results for Mujoco and Adroit datasets. Scores undiscounted average returns of the policy at the last iteration of training, averaged over random seeds. We bold the highest values.
Task name
BC-exp
BC-all
BCND
ORII.
DWBC-noads
DWBC
mixed-2
1547
811
437
1345
2450
2531
mixed-5
1263
811
437
998
2271
2451
Hopper
mixed-I1 10
1458
811
437
1489
1798
2231
exp-rand-3
1200
52
1531
2231
exp-rand-6
1070
314
52
51
1604
1610
mixed-2
4451
4210
4456
4980
5011
mixed-5
4553
4210
4456
44
5011
5018
HalfCheetah
mixed-10 10
4358
4210
4456
989
5022
5017
exp-rand-3
6072
5753
6007
6013
6021
6107
exp-rand-6
5875
5753
6007
6110
5803
5955
mixed-2
2031
784
760
2208
2355
2436
mixed-5
2014
760
2481
3112
3111
Walker2d
mixed-10
1611
784
760
2384
3219
3258
exp-rand-3
3078
211
6
955
4547
4666
exp-rand-6
2871
211
6
3673
4250
mixed-2
2682
2255
917
1050
2000
mixed-5
2381
2255
917
1982
3111
Ant
mixed-10
2285
2255
917
2310
3417
exp-rand-3
1071
151
1045
710
875
1230
exp-rand-6
870
151
1045
639
626
1127
human-2 1888
806
1684
2262
2571
2486
Pen
human-3
1780
806
1684
2487
2362
2617
human-5
1531
806
1684
2111
2271
2487
human-2
45
31
-4
-12
51
53
Door
human-3
40
-4
human-5
38
31
-4
-3
0
0
human-2
-187
-230
-163
-222
-88
Hammer
human-3
-191
-230
-163
-237
-97
-96
human-S
-213
-230
-163
-159
-60
24
human-2
4
2
7
-4
3
3
Relocate
human-3
3
2
7
-8
0
1
human-5
3
7
non-expert data. We also find that the performance of ORIL tends to decrease in tasks (c.g.. salfchhetaah mi me.mmiiettt mex-- and Ant_exp rand-6 6), this "overfitting" phenomenon also occurs in the experiments of offline RL papers (Kumar al. 2019 Wu al. 2019a). This is perhaps due to the limited data size and model generalization bottleneck (Neyshabur 2017).
We also find that DWBC-noads performs expert data, under which circumstance it is hard for the discriminator to distinguish between expert and non-expert data without the help of our proposed adversarial learning machinery.
than DWBC, especially when D. contains more
5.4 ADDITIONAL EXPERIMENTS
Offline policy selection by the discriminator. Offline policy selection (OPS) (Paine al. 2020
Yang et al., 2020 Dereventsov et al., 2021) considers the problem of choosing the best policy from a set of policies given only offline data. This problem is critical in the offline settings (i.c., offline RL and offline IL) because the online execution is often very costly and safety-aware, deploying problematic policy may damage the real-world systems (Tang & Wiens 2021). Note that existing offline RL/IL methods break the offline assumption by evaluating different policies corresponding to their rewards in online environment interactions. However. this online evaluation often infeasible and hence undermines the initial assumption of offline RI/II..
8

---

Under review as a conference paper at ICLR 2022
2443
263
3-40
3X0E
208
205
Higer #iver
1923
1463
1403
1203
353
0.4
dherrm leamned DWWDC.
Outsut clectrirater leamed DVBC
vutuutvvalle of didle CAMBC
Figure 2: Additional experiment offline policy selection by the discriminator learned by DWBC.
In this paper, propose novel offline policy selection method, simply by using learned discriminator d. More specifically, as the discriminator gets state s, action and the log probability log (aaas) s) as input, we use expert state-action pairs from De and different policy и as input. The discriminator learned by DWBC will assign large values (close to 1) when the evaluated policy is close to the policy learned by DWBC, this means that the policy is more optimal.
To validate our proposed idea, we conduct experiments in Hopper, Walker2d and Pen environ- mi xed-2, mi xed-5 and mi xed-1 0 datasets, in Pen,
ment. In Hopper and Wa ker2d, we use mi xed-2, 3 and mi xed-5 datasets. We compare three algorithms (BC-exp, ORIL and DWBC) trained in these datasets, total of policies in each environment. We first train DWBC, then we use the learned discriminator along with De to compute the value d(s, a, log ग. (als)) of each policy тj. We plot average d(s,, a, log (a(s)s versus the policy's 's true return in Figure 2 As shown, the discriminator's 's values can well reflect the rank between almost every two policies. This means that we can first train DWBC policy and then use the trained discriminator
select the best policy among our candidate policy sets, without executing them in the environment to get the actual returns.
Comparision of run time. We also evaluate the run time of training DWBC and other baseline algorithms for 250 thousand training steps (does not include evaluation run time cost). All run time experiments were executed on NVIDIA V100 GPUs. For fair comparison, we the same policy network size in BC, BCND, ORIL and DWBC. The discriminator net- work size is also kept the same in ORIL and DWBC. The results are reported in Figure 3 Unsurprisingly, we find the run time of approach only slightly more than BC, while other baselines (ORIL, BCND) are over 7 times more costly than BC. The reason that ORIL costly run is due to the additional effort to solve an offline RL problem. The high computation cost of BCND is due its inner iterations of training K policy ensembles (K 5 in our experiment), which is also mentioned in their pa- per (Sasaki & Yamashina 2021). This demonstrates the effectiveness of DWBC by only adds a limited cost to the original BC algorithm while providing substantially improved performance.
Run Time
5Gm
2.mm
ORIL
BChD
8C
DWec
Figure Run time comparison of train- ing each offline IL algorithm.
CONCLUSION AND LIMITATION
In this paper, we propose new offline imitation learning algorithm that can learn from suboptimal demonstrations without environment interactions or expert annotations. Experimental results show that our algorithm achieves higher returns and faster training speed compared to baseline algorithms, under different scenarios. One limitation of our work is that since our algorithm is based on weighted BC, the covariate shift problem of BC (Ross et al. 101I) will be inherited. That is, there is way for the policy to learn how to recover it deviates from the behavior policy to a new state not seen the demonstrations, In future work, will consider modifying the main task from action matching to state-action distribution matching, which is known to be more robust to distributional shift (Kostrikoy et al. 2019

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214 223. PMLR, 2017.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pp. 783-792. PMLR, 2019.
Daniel S Brown, Wonjoon Goo, and Scott Nickum. Better-haanddmoonttiee demonntraaoor imitation learning via automatically-ranked demonstrations. Conference on robot learning, pp. 330-359. PMLR, 2020.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv: wet..09022...... 2019.
Jonathan D Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data without great coverage. arXiv preprint arXiv:2106.03207 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.01344 2021.
Thomas Degris, Martha White, and Richard Sutton. Off-policy actor-critic. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pp. 179-186, 2012.
Anton Dereventsov, Joseph D Daws Jr, and Clayton Webster. Offline policy comparison under limited historical agent-environment interactions. arXiv preprint arXiv:2/06.03934 2021.
Marthinus Du Plessis, Gang Niu, and Masashi Sugiyama. Convex formulation for learning from positive and unlabeled data. International conference on machine learning, pp. 1386-1394.
PMLR, 2015.
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 213-220, 2008.
Alhussein Fawzi, Seyed-Mohsee Moosavi-Dezfooli, and Pascal Frossard. Robustness of classifiers: from adversarial to random noise. arXiv preprint arXiv: 08967, 2016.
Pete Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. arXiv preprint 11XXx:22009.0137 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020a.
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Alexander Novikov, Mengjiao Yang, Michael Zhang. Yutian Chen, Aviral Kumar, Cosmin Paduraru, et al. Benchmarks for deep off off-policy evaluation. In International Conference Learning Representations, 2020b.
Izrail Moiseevitch Gelfand, Richard A Silverman, et al. Calculus of variations. Courier Corporation, 2000.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems. pp. 2672-2680, 2014a.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint
2014b.
10

---

Under review as a conference paper at ICLR 2022
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems,
2016.
Alex Irpan. Deep reinfforcemmetttee learning doesn't work yet. httpss//ww..leexrrppp.. 2018.
Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning as one big sequence modeling problem. arXiv preprint arXiv:2106. 02039, 2021.
Firas Jarboui and Vianney Perchet. Offline inverse reinforcement learning. arXiv preprint
(0ti::2/660000000 2021.
Daniel Jarrett, Ioana Bica, and Mihaela van der Schaar. Strictly batch imitation learning by energy- based distribution matching. arXiv preprint M.Kiv:2006.1415:: 2020.
Edouard Klein, Bilal Piot, Matthieu Geist, and Olivier Pietquin. A cascaded supervised learning approach to inverse reinforcement learning. In Joins European conference on machine learning and knowledge discovery in databases, pp. 1-16. Springer, 2013.
Ksenia Konyushkova, Konrad Zolna, Yusuf Aytar, Alexander Novikov, Scott Reed, Serkan Cabi, and Nando de Freitas. Semi-supervised reward learning for offline reinforcement learning, arXiv preprint arXiv:2012.06689, 2020.
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations, 2019.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy
q-learning via bootstrapping error reduction. In Advances Neural Information Processing Systems, pp. 1 11761-11771, 2019.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pp. 45-73. Springer, 2012.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. A tutorial on energy-based learning. Predicting structured data, 100). 2006.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial,
review, and perspectives on open problems. arXiv preprint arXiv:2005.0166 01643, 2020.
Charles X Ling and Victor S Sheng. Cost-sensitive learning and the class imbalance problem. Encyclopedia of machine learning, 2011:231-253 2008
Daniel Lowd and Christopher Meek. Adversarial learning. Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, pp. 641-647, 2005.
Yecheng Jason Ma. From Adversarial Imitation Learning to Robust Batch Imitation Learning. PhD thesis, 2020.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359. 2020.
Behnam Neyshabur. Implicit regularization in deep learning. arXiv preprint / 2017.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations:
Theory and application to reward shaping. In Ieml, volume 99, pp. 278 287, 1999.
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume 1, pp. 2, 2000.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09053 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightee regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint arXiv: 1770.0017, 2019.
11

---

Under review as a conference paper at ICLR 2022
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Advances in neural information processing systems, pp. 305-313, 1989.
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv: 1905. 10000..0.........111111.11100118 2019.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635, 2011.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In Interna- fional Conference Learning Representations, 2021. URL htt.ptttp///opeeeeviee... net/
Forum?1d-2rT3Hcskaa
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert RG metrics...hi-divergeeee <phi-divergences and binary classification. arXiv
Lanckriet. On integral probability metrics, preprint wxXiv:0901.2694 2009.
Mingfei Sun, Anuj Mahajan, Katja Hofmann, and Shimon Whiteson. Softdice for imitation learning: Rethinking off-polic distribution matching. arXiv preprint
2021.
Richard S Sutton, Andrew G Barto, et al. Introduction reinforcement learning, volume 135. MIT press Cambridge, 1998.
Gokul Swamy, Sanjiban Choudhury, Zhiwei Steven Wu, and Andrew Bagnell. Of moments and matching: Trade-offs and treatments in imitation learning. arXiv preprint arXiv:2103.0322 2021.
Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. arXiv preprint arXiv:2107.11003. 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33, 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1x1hv:191..11311, 2019a.
Yuch-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imita- tion learning from imperfect demonstration. In International Conference on Machine Learning, pp. PMLR, 2019b.
Danfei Xu and Misha Denil. Positive-umlabeled reward learning. arXiv preprint arXiv:1911.0044 2019.
Mengjiao Yang, Bo Dai, Ofir Nachum, George Tucker, and Dale Schuurmans. Offline policy selection under uncertainty. arXiv preprint arXiv:2012.06919, 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization, arXiv preprint arXiv:2005. 13239, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-bar based policy optimization. arXiv preprint arXiv:2102.08333, 2021.
12

---

Under review as a conference paper at ICLR 2022
Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience. arXiv preprint 1..1::2011.18880 2020a.
Konrad Zolna, Chitwan Saharia, Léonard Boussioux, David Yu-Tung Hui, Maxime Chevalier- Boisvert, Dzmitry Bahdanau, and Yoshua Bengio. Combating false negatives in adversarial imitation learning. arXiv preprint arXiv:2002.000 00412, 2020b.
13

---

Under review as a conference paper at ICLR 2022
A DERIVATION DETAILS
In this section, we provide the detailed design intuition and theorectical derivation of DWBC.
A.1 DECOMPOSITION AND REFORMULATION OF LEARNING TASKS
As discussed in Section 3.1 learning from both the expert dataset D. and suboptimal dataset Do implies the need of jointl solving two tasks: a BC task and discriminator task. A straightfoward solution is to learn the two tasks separately, which solves the BC task by imitating the expert demonstrations in De and learn the discriminator via PU learning using data from both De and Da:
BC task: T(als) arg min LB
Discriminating task: d(s, <- arg min La
where LC and La are discussed and given in objectives and 4 in the main article follows:
LBC E | lo (a|s)]
(s.a)~De
La E log d(s, a)] + (s,a)~D,
E log([11 a(s 7 (s.a)~D,
E [-log(1 d(ss, a))l (s,0)~D.
Naïvely solving above two tasks separately is insufficient. First, the BC task only learn from the expert dataset D.. fails to utilize the potential valuable information in the supoptimal dataset Da- Second, both tasks lack sufficient information to improve their own performance. For example, a good discriminator could provide important information to distinguish the potential expert samples in the suboptimal dataset De, which are valuable for the learning of policy क; a well-performed policy π (als) will assing large probabilities expert actions under expert states, which could provide additional learning signal for the discriminator d to more easily contrast expert and non-expert transitions in D.-
There are two existing approaches can be used to jointly solve above two tasks, however, both of them have some drawbacks. One approach is cast the problem into a multi-task style multi-objective optimziation problem, by optimizing and d. The problem is that the BC task and the discriminating task are different tasks, the potential contradiction of the two tasks in certain settings may impede both tasks from achieving the best perforamnce. Moreover, properly selecting the hyperparameter /3 is very tricky. Another approach is to adopt GAN-style model (Goodfellow et al.) 2014a) which treats the policy as the generator and optimize it implicitly through solving a min-mac optimization problem with the discriminator loss L. However, this is very costly and is known suffer from training instability and issues such as mode collapse (Arjovsky et al. 2017). Moreover, although we have an explict loss function LB for π. it not used such a GAN GAN-style model, which results in potential loss of information.
augmented loss BL + (1 - La B (0, 1) for both
In this paper, we design an new cooperative learning mechansim to address above issues, which also results in computationally efficient practical algorithm. It includes three key ingredients: 1) sharing information between the BC task and the discriminating task to achieve cooperative learning: 2) enabling the BC task learn on both expert and suboptimal data by introduce an additional corrective loss f impacted by the discriminator outputs; 3) solving both tasks in fully supervised learning manner maintain computational efficiency. In our approach, we consider following alternative formulation to establish information sharing across the two tasks and enable cooperative learning:
BC task: r(als) + arg min aC + Lw: a
(10)
Discriminating task: d(s, a, log (als)) + arg min Ls
with the new La given in objective 5 as follows:
La =1
E
[- log d(s,, a, log (a)s)) + (s,a)-D,
|- log(1 d(s, a, log (cls)))
(s,a)-D,
7 E [- log(1 d(s, a, lagss (a.
In above reformulation, we design the information provided by the policy to discriminator as the element-wer -iiee imitation loss value log [a]s)), and the information provided the policy the
14

---

Under review as a conference paper at ICLR 2022
addional corrective loss term L computed using output values of the discriminator d samples from both De and D.. With the involvement of Cw, we can allow the policy т learning from sub-optimal data under the guidance of the discriminator. Moreover, to more robustly learn the discriminator d, we borrow the idea of adversairal learning (Lowd & Meek] 2005) and make the policy challenge d by providing as little useful information in log π (a|s) possible.
We will show in the next section that with the choice of demment-wise imitation loss log ([ass) as the form of information and the advasarial behavior of policy n, an exact form of L.: can be derived, and eventually, transforms the original BC task into a cost sensitive learning problem.
A.2 DRIVATION OF THE CORRECTIVE LOSS TERM L
In this section, we resort to functional analysis and calculus of variation derive the exact form of Lw. Under the reformulated problem (TO), both the discriminator d and its loss La are impacted by the information provided by policy æ (log π (a]s)). Hence they are now functional of q. We are interested to see how the variation of impacts Ld, and further influence d. Moreover, we can use specific form of L to alter the behavior of the learned π to achieve the desired adversarial behavior.
Denote 0g and 0g as the parameters of the policy and discriminator d respectively, As discussed in Section 3.2 to analyze the impact of ı on d, we fix the parameters 9g of d to exclude its own influence. The function of the discriminator d(0a, S, a. log ([]ss)) can thus be replaced by d(ss, a, log ((ass)). Similarly, with 0a fixed, we can define a functional (T)) for La which depends on function æ and further eliminate the effect of changes in 8, a d by integrating over and as in Eq.
J(x) | aLa(s, a, d, log π (a]s)) ds da
/ F(s,a, nI x') ds da,
Öd(s, s, a, log [(ass))
where F= /d is assumed to be a continuously differentiable function with respect to 8, a, π and n' (derivative of r). Note that 0g is fixed and not considered variable in our analysis, 0g no longer contributes gradients in the partial derivative aca/ 0d, and functional F is given as follows:
1
F(5, =
E (s.a)-D,
d(s, a, log π (a ()))
(s.a)-Do
1 - d(s, a, log π (a|s))
(11)
7
E - d(s, a, logs(als))
(s,a)>D;
To enforce the adversarial behavior of r, we want to make π challenge the discriminator d by maximizing La under current d (doing the opposite to d). By doing so, the policy will provide as little information in log x as possible such that minimizing faa becomes harder for the discriminator. This can be seen minimizing the worst-case error, which makes the robustness of the discriminator significantly improved.
Maximizing La for under current d is equivalent to finding the maxima of functional J(TT) (Ca with 0g and fixed). According to the calculus of variations (Gelfand et al. 2000), the extrema (maxima or minima) of functional (T) can be obtained by finding function where the functional derivative is equal to zero. We can show with following proposition that ensuring the functional derivative of J (IT) equal to zero requires OF/00, O.
Proposition 1. With the parameters 0. of the discriminator d fixed, the functional ( and F(s, a, n, æ²) defined as in Eq. and (77). if uniform continuity of both F and derivative is satisfied, the necessary condition for J[T] attaining its extrema is OF a, n, r'0000,, =. O.
Proof. We derive the finite difference approximation of the functional derivative of J(z) by Taylor expansion,
J(x) = (Trgg ) + J' (T(g Yor + o(57).
(12)
where ôn is the variation of n. As o(ōm) ) is higher order infinitesmall term when 0g - 00 which can be ignored. Plug Eq. into the form of (TT, we obtain the finite difference approximation of ((T) as follows:
AJ(a) J(x) J(Fag) = J'( )8T dʃ F(s, a, x') ds da 5n.
(13)
d8,
15

---

Under review as a conference paper at ICLR 2022
If the uniform continuity of both F and its derivative is satisfied, can swap the order of integration and differentiation as well as ds da and ón. The above equation can be simplified as:
AJ(T) = f [ (s, a, n, n²) ) ds da
ōn = / ( a, T,, m²) dm ds da.
(14)
a0,
00,
As π can be any function in some function space Ensuring the functional derivative of J(x) equal to 0 also requires (F a, r'000,, 0.
Given the form of functional F in Eq. I1I. we can derive the detailed form of OF/00, as presented in ( of the main article:
(F a, T')
(F a, nI, x²)
alog
OF(s, TT,
8r log
00,
i Olog
a0x
i olog π
1
E (s.a)~De
Ve, log x (a|s)
+
E (a.0)~Do
Ve, log (als)
d
d
E (s.a)-/D
Ve, lals
1-d
where in the last equation, we slightly abuse the notations and write the output value of d(s, a, log (a]s)) as d for simplicity.
As discussed in Section 3.2, above derivative can be equivalently perceived as the graident of new loss term for ग, which exactly the corrective loss term L. that are looking for. If we set BLw/06. -F/ we recover Lw in following form:
C = E (ogs((ls)) "1 (s,a)=D,
E
log (als(a)s):
1 1-d (s,a)=D.
E
logn(als)
I-d
(s.a)~D,
The introduction of the minus sign on 0F/00,, is to ensure we find the maxima of J(T) rather than its minima. Hence minimizing Ly with respect to π (make OCw, = -0F/00, = 0) will drive the functional derivative of (Tz) to zero, which also lead to the maxima of (1)) (update with gradient direction -0// Adding the new corrective loss term L back to our reformulated problem (IO), obtain the final learning objective of for our BC task (Eq ( in the main article):
log (als) 1
min
E E 1- (s,a)~D, (s.0)~D,
log T (als)
E (c.0)~D.
d( d(--d)
Note that the derivation of L.w requires uniform continuity to be satisfied in F and its derivative. The involvement of discriminator output values 1/d(s, a, log (als)) and 1/(1 d(s, a, log n (a]s)) may violate the continuity assumption. We thus clip the discriminator output values to the range of 0.9] in our practical algorithm.
B TRAINING PROCEDURE DETAILS
B.1 ALGORITHM DETAILS
In this section, we present the pseudocode of DWBC in Algorithm !
B.2 IMPLEMENTATION DETAILS
In this paper, all experiments are implemented with Tensorflow and executed on NVIDIA V100 GPUs. For all function approximators, we use fully connected neural networks with RELU activations. For policy networks, we use tanh (Gaussian) on outputs. We use Adam for all optimizers. The batch size is 256 and is 0.99. We rescale the reward to [0, 1] as r' = (r Tmin)) / (Fmax Tmim)) ). where Tmax and Tmin is the maximum and the minimum reward in the dataset. Note that any affine transformation of the reward function does not change the optimal policy of the MDP. The policy and discriminator network are all 3-layer MLP with 256 hidden units in each layer. The learning rate for the policy is le 5 and the learning rate for the discriminator network is 1e 4. We search in {1, 2, 5, 10] for best model performance. We clip the output of d to [0.1, 0.9]. We set y to 0.5 across all tasks, which the same the ORII. paper (Zolna al. 2020al
16

---

Under review as a conference paper at ICLR 2022
Algorithm Discriminator-Weiggheee Behavior Cloning (DWBC)
Require:
Dataset D. and D... hyperparameter 1),
1: Initialize the imitation policy and the discriminator
while training do
3: Sample (&e, 9e) De and (8a, 0o) ~ Do to form training batch B
4: Compute log (a]s) values for samples in B using the learned policy
5:
Compute discriminator output values d(ss, a, log [a]ss) using sampled (s, and computed ((]ss
Update by minimizing the learning objective E
7: Update by minimizing the learning objective in Eq.
8: end while
C ADDITIONAL RESULTS
C.I DATASETS DETAILS
In this section, we list all datasets used in paper and the number of trajectories and transitions in De and D., different is labeled after the dataset name.
# D
# Do.
Dataset-X
Trajectories
Transitions
Trajectories
Transitions
Hopper_mixed4
204
96,222
1,835
303,737
Hopper_mixed-5
82
39,590
1,957
360,369
Hopper_mixed-
10
19,176
1,998
380,783
Hopper_cxp-rand-3
6,993
1,003
23,720
Hlopper_exp-rane
4
3,996
1,006
26,717
Halfcheetah_mmixed22
20
19,980
182
181,818
halfcheetah_mixede
7,992
194
193,806
Halfchectah_miixdl
3,996
198
197,802
Halfcheetah
1
6,993
1,003
1001,997
aeficheetaa_exp-rrndeeee
3,996
1,006
1004,994
Walker2d 2iiied22__ixed22222
109
74,857
984
226,050
calker2d_mixed
44
31,010
1,049
269,897
Walker2d_mixede mixed-10
22
15,569
1,071
285,338
Walker2d [ilkerr2d_exp-rand23
6,993
1,003
21,874
Walker2d_ent -xxrerrdd_xpp-
3,996
1,006
24,871
Ant mixed-2
46.646
436
254,869
Ant mixed-5
20
19,209
465
282,306
Ant_mixed-10 10
10
9,866
475
29,1649
Ant_exp-rand-3
6,458
1,003
182,909
Ant_exp-rand-6
3,996
1,006
185,371
Pen human-2
S97
22
4,378
Pen 3
398
23
4,577
P
199
24
4,776
Door human-2
770
22
5,934
Door human-3
479
23
6,225
Door huma-55
255
24
6,449
Hammer hhminnerr_umna222
1,485
22
9,800
Hammer Hammer_huumn-3 human-3
844
23
10,441
hammer__unnann
483
24
10,802
Relocate human-2
1.328
22
8,589
Relocate human-3
9,055
Relocate human-S
511
24
9,406
Table 2: Dataset details.
17

---

Under review as a conference paper at ICLR 2022
C.2 LEARNING CURVES
In this section, we provide the learning curves of listed algorithms in Section 5, As the learning procedure of BC quite fast and stable, for more clearly presentation, we plot the results of BC-exp and BC-all as a horizon bar with the shaded area as the standard deviation across different seeds. The average return in De plotted the red dashed line in all plots.
Happer 2
Valuebb rmas2
Art mi
10
Hepe Flaed 5
Jealhe Fand
Vilood rlads
Art Ne.v
HarChhsbit 2rinc10
21 sprans)
ArLsppendd
Haper G
JalChers soraed
Artingaaad mass
basand
Fanneer Jumur
Junan
Doo: Jamand
Parsee Jumann
Rebodt Junann
1o
Figure 4: Learning curves of compared algorithms on different datasets.
18


Reviews:
[{"review_id": "gLEWPLo5qkg", "reviewer": "Reviewer_x4DH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The paper is well written and clearly communicates the motivation and contributions.\n+ Sufficient empirical experiments which support the core thesis of the paper. The baselines are selected carefully to provide meaningful comparison to the proposed approach.\n+ Using the jointly learned discriminator seems like a novel idea for policy optimization and evaluation. \n\n\nWeakness:\n+ The experiments on very similar datasets (e.x changing the fraction of true positives). While I agree that it is strong experiment, From Figure 1. it is clear that the performance of DWBC is not too sensitive to fraction of positive samples. It would be useful to have similar experiments on datasets collected from a random mix of optimal and suboptimal policies (instead of buffer of single learning policy). For example, mixing expert and random policies, or Adroit environments with human demonstrations.\n+ Offline Policy evaluation is evaluated on Hopper-v2 only. It would be useful to have a diversity in environments to evaluate this contribution more objectively.", "summary_of_the_paper": "The authors consider the problem of offline imitation learning in the presence of suboptimal datasets. In the presence of suboptimal data, classical baselines like behavior cloning suffer performance hits, the drop in performance often correlates positively with increase in number of suboptimal trajectories. In this work, the authors propose a novel learning objective inspired by the min-max formulation in GANs. Particularly the agent learns a discriminator to distinguish between samples from the expert and the suboptimal demonstrator. Building on prior work in cost-sensitive learning, this discriminator is used to reweight loss per sample in the offline buffer. \n\nThe proposed algorithm is evaluated on standard offline RL benchmarks, across multiple environments. In many environments, e.g Hopper-v2 style environments, the policy improves against strong imitation learning benchmarks. The discriminator (which takes action probabilities as input) is also evaluated in the context of offline policy evaluation. On Hopper-v2 datasets, the discriminator output is compared with true reward accumulated by multiple policies.", "summary_of_the_review": "Overall, the paper is well motivated and provides promising results for leveraging suboptimal datasets for effective offline imitation learning. The problem is well motivated and the authors provide some novel insights into better objectives for behavior cloning. While the authors demonstrate some improvement over the provided baselines, I would encourage the authors to consider adding couple of more ablations, particularly across datasets with a mix of human and random trajectories. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dT2CndhvoPa", "reviewer": "Reviewer_jCdL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem studied in this paper is quite relevant and important — the ability to re-use large, noisy offline datasets to solve new tasks. This paper motivates the problem well in the introduction and lays out the preliminaries clearly. The related work section also makes references to much of the relevant work, though the subsection on offline RL could be enhanced by expanding on the limitations of offline RL (something the authors already did in the introduction). The method appears to be novel and the discussion framing this method as a weighted behavior cloning objective connects nicely to prior work. Alongside these strengths, there are a number of concerns which I enumerate below:\n1. The derivation presented in section 3.2 is a bit confusing and unintuitive. First, I’m not sure I agree with the justification for the why policy outputs needs to be part of the discriminator input. The authors state that it helps the discriminator distinguish between expert and non-expert actions better, but I would like to see this design choice verified empirically — ie. comparing to a discriminator that only takes the state and action as input, possibly without the adversarial formulation. In addition, the discussion on the adversarial policy learning objective also can benefit from additional motivation and details. In particular, why is “providing as little information in $\\log \\pi$ as possible” the best way to learn a discriminator? Why is the latter necessarily equivalent to “maximizing $L_d$ for $\\pi$”? This subsection is, in my opinion, the weakest part of the paper. The discussion here can be improved by a combination of providing a more formal framework for the adversarial learning problem, complementing the formal framework with intuitive explanations, and connecting the ideas here to prior literature.\n2. I am having trouble understanding the derivation details in Appendix B. Specifically, why does $\\frac{\\partial F}{\\partial \\log \\pi}$ equal to $\\frac{\\partial L_d}{\\partial d} \\cdot \\frac{\\partial d}{\\partial \\log \\pi}$? And also I don’t see where the $\\frac{\\partial d}{\\partial \\log \\pi}$ term in incorporated into the final derivation at the end of the section. Can you please clarify these steps?\n3. One critically missing baseline is to train the discriminator defined in equation (4) once, without all the adversarial learning machinery presented in section 3.2 — and use the discriminator weights to weigh different samples when training the policy. This can help justify all of the additional complexities presented in section 3.2 of updating the discriminator and policy in an alternating optimization scheme.\n4. While the proposed method appears to work well in locomotion domains, it remains unclear how the approach would scale to more complex settings, such as the robotic manipulation datasets for the kitchen and adroit tasks in D4RL [1] and the manipulation tasks in robomimic [2]. In principle, it should not be too difficult to run experiments on these datasets as well and such experiments would certainly enhance the scope of this paper. That said, given that ICLR is primarily focused on core machine learning methods and less so on strong empirical evaluations, this is not the primary concern in this review. \n5. It is unclear to me why BC-all and BCND are constant lines, while the other baselines are curves. Shouldn’t all the baselines be shown as curves — ie. where the performance is changing across training iterations? \n6. While it can be implied from the paper as is, pseudocode or a text description of the full training scheme would be nice to have. In particular, I was wondering how often the discriminator is updated relative to the policy — does one update more frequently than the other? \n\n[1] Fu et al., D4RL: Datasets for Deep Data-Driven Reinforcement Learning, 2020\n\n[2] Mandlekar et al., What Matters in Learning from Offline Human Demonstrations for Robot Manipulation, CoRL 2021", "summary_of_the_paper": "This paper proposes an offline imitation learning framework that incorporates both optimal and suboptimal datasets to learn decision-making tasks, without requiring any reward annotations. To leverage high reward transitions from the suboptimal dataset, the authors formulate a discriminator that optimizes a positive-unlabeled learning objective, where positive samples come from the optimal dataset and unlabeled samples come from the suboptimal dataset. This discriminator is trained in an adversarial fashion along with the policy, resulting in a behavior cloning objective where samples from the optimal and suboptimal datasets are weighted differently according to the discriminator’s predictions. Experiments demonstrate that on a set of simulated locomotion domains, the proposed algorithm can leverage the suboptimal dataset to learn more performant policies compared to vanilla behavior cloning objectives and prior offline IL/RL baselines.", "summary_of_the_review": "My reaction to this paper is mixed. On one hand, the introduction, preliminaries, and related work are well laid out. On the other hand, I had several confusions about the method and have some concerns about the experiments (see the main review for specific details). As it stands, I think this paper is marginally below the acceptance threshold. I hope that the authors can diligently address the concerns that I raised, at which point I will reconsider my recommendation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Xpqe7Z1T2Z", "reviewer": "Reviewer_nHQm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- The paper is overall well written and easy to follow. The justification of the setup makes sense and is well explained.\n- The setting is very interesting and has a great potential impact on the community.\n\n**Weaknesses**\n- Method\n    - The authors fail to mention a very close work that already applied positive-unlabeled learning to the Imitation learning setup: PUGAIL (https://arxiv.org/abs/1911.00459). This greatly limits the novelty of this work.\n    - I find the theoretical derivations confusing. In particular, in appendix B, proposition 1, is “when” an “if and only if” condition or just an implication?\n    - I also don’t understand why the authors need to introduce the function F. Overall the mathematical derivations are unclear and would benefit for more details (keeping them shorter in the main text but developing them correctly in appendix for example).\n\n- Related Work\n    - I think a few more works would be worth mentioning. In particular, some previous work already considered having access to two datasets (on expert, one non-expert), like CSI  (https://hal-supelec.archives-ouvertes.fr/hal-00869804/document)  or MILO (https://arxiv.org/pdf/2106.03207.pdf)\n- Evaluation\n    - Authors write “the proposed algorithm can learn behaviors that are much closer to the optimal policies” yet they fail at providing experiments that support this claim as they only study the return of the learnt policies. I would recommend that they checkout this work (https://arxiv.org/abs/2105.12034 ) that provides insights on how to evaluate models in the context of imitation learning. What is more, the plots don’t show the average return of D_e which makes it impossible to use the return as a proxy to study how “close” the policy is from the demonstrations.\n- Experiments\n    - The experimental setup is good but a bit “light”. As written by the authors, their method is quite fast to train, so why not provide more results on different setups, e.g. 1) with very little data (e.g. like 1 or 5 trajectories only)? with very random data in D_o? with human expert data in D_e? with more complicated environments like Adroit? (All these environments/datasets are available in D4RL).\n    - Please report the average return in D_e as a horizontal bar in the plots. Otherwise it is impossible to calibrate the results as a reader.\n- Writing\n    - In the abstract, the authors say “both optimal and non-optimal expert behaviors”. I find this expression a bit confusing. It suggests that the algorithm will only work if the D_o is actually made of expert but slightly suboptimal trajectories.\n    - nit: In 3.2, “much hard” -> “much harder”, “which we denote it as” -> “which we denote as”\n", "summary_of_the_paper": "The paper deals with the following setup: offline imitation learning in the presence of both an expert dataset and a non-expert dataset. More precisely, the goal is to learn a policy as close as possible to the one(s) that generated the samples in a dataset $D_e$, while making the most of samples in a non-expert dataset $D_o$. The “reward” information is not present/used in the dataset.\nThe authors draw inspiration from the positive-unlabeled classification as well as the adversarial imitation learning literature to propose a new algorithm to tackle this problem. They interleave the training of a discriminator and a policy. The discriminator is trained to discriminate between expert and non-expert dataset (using a positive/unlabeled loss) and takes as input the state, the action and the logit of the policy $\\pi(a | s)$. The policy is trained to imitate the expert on $D_e$ and to “fool” the discriminator. \n\nThe authors present results on four environments from the Gym Mujoco suite with datasets extracted from the D4RL datasets.\n", "summary_of_the_review": "I believe this is an interesting idea in an interesting setup yet it lacks novelty and it would deserve more work, notably on the experimental part, before being published.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GGpgiVMOOtK", "reviewer": "Reviewer_xvWD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the results are quite promising, especially the fact that the algorithm can learn to mimic the expert from a really small number of expert demonstrations. For example, Figure 1 shows that it is possible to learn HalfCheetah from just 4 thousand expert samples (=4 episodes?). However, I do feel like that both the method and the derivation need some more clarity, and I also have some concerns regarding the experiments.\n\nThe derivation includes several inaccuracies and vague statements. The main innovation behind DWBC is to learn a discriminator and condition it on the policy that is being learned. The conditioning is motivated by the fact that it makes discrimination easier (given the optimal policy, one can discriminate the actions based on their probabilities only). It is then argued (after Equation 6) that learning the disciminator becomes more robust, if at the same time, the policy is optimized by minimizing the amount of useful information it can provide to the discriminator. This observation is the most important contribution of the paper, yet the justification seems insufficient and vague. Further, it is not obvious that having the derivative of the discriminator loss w.r.t the policy equal to zero (Equation 7) will be desirable, and that minimization of the final loss in Equation 8 will in fact lead to that condition to be true. That said, the results do indicate the algorithm performs well, so I'm mostly curious seeing a more rigorous derivation and justification that could help shed some light why the DWBC works well.\n\nMoreover, there are several other inaccuracies that make following the derivation hard. For example: \n* The loss $L_d$ should not depend on $s$ and $a$ as it does in Equation 6.\n* I think in Equation 8, the gradients need to be stopped from flowing to the discriminator.\n* How is the discriminator trained? Equation 8 is only minimized with respect to the policy. Is the same loss used for learning the discriminator?\n\nI also have several smaller comments on the experiments:\n\nThere seems to be something not quite right with the shading (standard deviation) in Figure 1. For some of the curves, the shaded region is really narrow compared to how much the curves change between iterations. Can you comment on that? Is the training set  different for each seed? \n\nPlease add axis labels to Figure 2.\n\nThe datasets used in the experiments have a really particular form as they are collected during online training and thus the expert and the sub-optimal data are highly related. It would be good to see a comparison where the datasets come from two completely different policies: one optimal expert policy, and another fixed but suboptimal policy. This would be a more realistic setup (i.e., if the data comes from policy that is trained online, then why do we need offline learning?).\n\nIn the offline policy selection experiment, do you use separate datasets for training the discriminator and evaluation? If not, then perhaps the discriminator is simply memorizing the training data. \n\nWhat is the return of the expert policies for the experiments in Figure 1?\n", "summary_of_the_paper": "The paper proposes a new offline imitation learning algorithm, DWBC, for datasets that combine both optimal and suboptimal demonstrations. The approach is based on a modified behavioral cloning loss that weighs expert and non-expert data based on a learned discriminator. DWBC is compared against prior methods in OpenAI Gym tasks, and it is shown to yield better policies compared to the prior work. As a by-product, the method learns a discriminator that can be used to estimate the relative performance of any policies without rolling them out in the environment.", "summary_of_the_review": "The results presented in the paper are promising, but there are several issues with the clarity of the derivation and some with the experiments, and thus the paper is not yet of sufficient quality to be published as is.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce a method for offline imitation learning in the presence of optimal and non-optimal data. In particular, they propose to learn a discriminator that can be then further used to modify the behavior cloning loss which leads to performance improvements over baselines. The reviews mention that the idea is novel and most sections of the paper are well written and self-explanatory. They do point out, however, several flaws such as the clarity of the derivation and  the thoroughness of experimental evaluation. While the paper has significantly improved during the rebuttal, its significant changes warrant another round of reviews. I encourage the authors to continue improving the paper, addressing the reviewers' feedback and resubmitting it as it has a potential to be a strong submission.
Decision: Reject