DemoDICE: Offline Imitation Learning with Supplementary Imperfect Demonstrations
Authors: Geon-Hyeong Kim
Seokin Seo
Jongmin Lee
Wonseok Jeon
HyeongJoo Hwang
Hongseok Yang
Kee-Eung Kim
Venue: ICLR 2022 Poster (2022)
Keywords: imitation learning
offline imitation learning
imperfect demonstration
non-expert demonstration

Abstract: We consider offline imitation learning (IL), which aims to mimic the expert's behavior from its demonstration without further interaction with the environment. One of the main challenges in offline IL is to deal with the narrow support of the data distribution exhibited by the expert demonstrations that cover only a small fraction of the state and the action spaces. As a result, offline IL algorithms that rely only on expert demonstrations are very unstable since the situation easily deviates from those in the expert demonstrations. In this paper, we assume additional demonstration data of unknown degrees of optimality, which we call imperfect demonstrations. Under this setting, we propose DemoDICE, which effectively utilizes imperfect demonstrations by matching the stationary distribution of a policy with experts' distribution while penalizing its deviation from the overall demonstrations. Compared with the recent IL algorithms that adopt adversarial minimax training objectives, we substantially stabilize overall learning process by reducing minimax optimization to a direct convex optimization in a principled manner. Using extensive tasks, we show that DemoDICE achieves promising results in the offline IL from expert and imperfect demonstrations.

Published as a conference paper at ICLR 2022
DEMODICE: OFFLINE IMITATION LEARNING WITH SUPPLEMENTARY IMPERFECT DEMONSTRATIONS
Geon-Hyeong Kim Seokin , Jongmin Lee', Wonseok Jeon 133,4, HyeongJoo Hwang', 1,2,5 1 School of Computing, KAIST, Daejeon, Republic of Korea
Hongseok Yangl.2., Kee-Eung Kim 1.2
2 Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea
3 Mila, Quebec AI Institute
1 "Scoool of Computer Science, McGill University
Discrete Mathematics Group, Institute for Basic Science (IBS), Daejeon, Republic of Korea
ABSTRACT
We consider offline imitation learning (IL), which aims to mimic the expert's be- havior from its demonstration without further interaction with the environment. One of the main challenges in offline IL is to deal with the narrow support of the data distribution exhibited by the expert demonstrations that cover only a small fraction of the state and the action spaces. As a result, offline IL. algorithms that rely only on expert demonstrations are very unstable since the situation easily de- viates from those in the expert demonstrations, In this paper, we assume additional demonstration data of unknown degrees of optimality, which we call imperfect demonstrations. Compared with the recent IL algorithms that adopt adversarial minimax training objectives, we substantially stabilize overall learning process by reducing minimax optimization to a direct convex optimization in principled manner. Using extensive tasks, we show that DemoDICE achieves promising re- sults in the offline IL from expert and imperfect demonstrations.
INTRODUCTION
Reinforcement learning (RL) (Sutton et al., 1998) aims learn intelligent behavioral strategy based on reward feedback. Although RL has achieved remarkable success in many challenging do- mains, its practicality and applicability are still limited in two respects. First, we need to specify the reward function, which may be non-trivial to do so in many real-world problems that require complex decision making. Second, the standard RL setting assumes online interaction with the en- vironment during the intermediate stages of learning, which is infeasible for mission-critical tasks.
Imitation learning (IL) (Pomerleau 1991 Ng & Russell 2000) addresses the first limitation of RL, where the agent is trained to mimic the expert from demonstration instead of specifying the reward function.
well known that adopting supervised learning for training the imitating agent, commonly referred to behavioral cloning (BC), is vulnerable to the distribution drift (Ross al. 201T). Thus, most of the successful IL algorithms rely on online experiences collected from the environment by executing intermediate policies during training. Recent progress made by adversarial imitation learning (AIL) (Ho & Ermon 2016 Ke et al. 2019 Kostrikov et al. 2020) achieving state- of-the-art results on challenging imitation tasks still relies on such an online training paradigm.
Unfortunately, in many realistic tasks such as robotic manipulation and autonomous driving, online interactions either costly dangerous. Offline RL. (Fujimoto al. 2019 Kumar et al.) 2019 2020 Levine et al. 2020) Wang et al.) 2020; Lee et al. [202]] Kostrikov et al. 2021) aims to ad- dress these concerns by training the agent from the pre-collected set of experiences without online interactions. To prevent an issue caused by the distributional shift, offline RL algorithms mitigate the phenomenon by constraining the shift or making a conservative evaluation of the policy being leameed.
'hhe research that is the hasis of this paper was done while the author was at Mila/McGill University, but the author is currently employed by Qualcomm Technologies Inc.

---

Published as a conference paper at ICLR 2022
In this paper, problems is tricky. For instance, naively extending the offline RL algorithms (which assume a reward function) to the offline IL setting does not work. In practice, expert demonstrations are scarce due to the high cost of obtaining them. Thus, they typically cover only a small fraction of the state and action spaces, which in turn makes the distribution drift issue the standard offline RL setting with reward function. We mitigate this issue by assuming a large number of supplementary imperfect demonstrations, without requiring any level of optimality for these imperfect demonstrations; they may contain expert or near-expert trajectories (Wu et al. [201)] Wang et al., (021) as well as non-expert ones all together. This generality covers the situations from real-world applications, but at the same time, it poses significant challenge for the design of successful offline II. algorithm.
concerned with offline IL problems. Finding effective algorithm for these
more stand out compared with
In this paper, we propose DemoDICE, a novel model-free algorithm for offline II. from expert and imperfect demonstrations. We formulate an offline II. objective which not only mitigates distribution shift from the demonstration-data distribution but also naturally utilizes imperfect demonstrations. Our new formulation allows us to compute closed form solution, which learns a policy in the space of stationary distributions but suffers from the instability issue in practice. We tackle the issue by proposing an alternative objective, which leads to a stable algorithm in practice while keeping the optimal stationary distribution. Finally, introduce a method to extract the expert policy from learned stationary distribution in a simple yet effective way. Our extensive evaluations show that DemoDICE achieves performance competitive or better than a state-of-the-an off-policy algo- rithm in the offline-IL tasks with expert and imperfect demonstrations. The code to reproduce our results is available at our GitHub repository!
PRELIMINARIES
2.1 MARKOV DECISION PROCESS (MDP)
We assume an environment modeled as a Markov Decision Process (MDP), defined by tuple M (S, A. T. R. Po. 7), where S is the set of states, A is the set of actions, T S x A - A(S) is the probability p($+41| St, ax) of making transition from state 8, to state St+1 by executing action a, at timestep i, R: S A -> IR is the reward function, po E A(S) is the distribution of the initial state 80, and 7 E [0, 1] is the discount factor. A policy т : - (AA) of MDP M is a mapping from states of M to distributions over actions. For the given policy the stationary distribution dt is
defined as follows:
d" (s. (1
= (1-1) - Σ
ppp(s = =,0== a0so ~ po()), ~ T(180-1, a)-1), ~ (+156))
1=0
We assume precollected dataset E of (s, as,,,s') s') tuples generated by the expert, and precollected imperfect dataset D (gencrated by unknown degrees of optimality). More precisely, for the (under- lying) expert policy's stationary distribution d (s,a), as (s,a) a) ~ P ~ T(-]s, a). We define D = D UD', , the union of two datasets, and denote the corresponding stationary distribution of the dataset D d² We denote the trajectories gen- erated by expert policy as expert trajectories, and trajectories generated by non-expert policies as non non-expert trajectories. The expert demonstrations consist only of expert trajectories and imperfect demonstrations consist of a mixture of expert and non non-expert trajectories. In this paper, we assume that the quality of imperfect demonstrations is unknown.
assume that (s, a, s') E DE is sampled
2.2 IMITATION LEARNING
Behavior cloning (BC) classical IL approach, which attempts to find a function that maps s via supervised learning. The standard BC finds policy x by minimizing the negative log- liceliioo:ooo::
min JBC(T) x
= min 1
Σ
lals(.
(1)
D
(s.0)ED
tat on-dice
2

---

Published as a conference paper at ICLR 2022
However, it is known to be brittle (Ross et al. 2011 when the interaction with the environment deviates from the scarce trajectories in D In such cases, BC fails to recover expert policies.
One of the notable approaches for IL is to formulate the problem distribution matching (Ho & Ermon 2016, Ke et al., 2019; Kostrikov et al. 2020). When instantiated with the KL divergence widely used in previous IL works (Ke al. 2019 Kostrikov et al. 2020), the approach amounts to finding a policy и by optimizing the following objective:
max -DL (d" ld ((aaa)dr log ds d=
Since we cannot directly access the exact value of d (s, a) and dt (8, a), we estimate their ratio using samples from d (8,a) and d= (s, a), given as follows:
max c x A (0.1)) (( [log c(s, a)] (a [log[1 c(s,
d (s.a) Based (s, (s, a) - log(1 - c² (s,, a)) = log d=(saa) on this connection between generative adversarial networks (GANs) and IL, AIL algorithms focus on recovering the expert policy (Ho & Ermon 2016 Kostrikov et al., 2019). However, the agent obtain samples from d= through interaction with environment, which is impossible in offline setting. Therefore, to tackle the offline II. problems, we should derive an alternative estimation without on-policy samples.
Here, the optimal discriminator e" recovers log c²
3 DEMODICE
In this section, we present a novel model-free offline IL algorithm named offline imitation learn- ing using additional imperfect Demonstrations via stationary DIstribution Correction Estimation (DemoDICE). Starting from regularized offline IL objective which accords with offline RL algo- rithms, we present a formulation that does not require on-policy samples. Such formulation allows us to construct a nested maximin optimization for offline II from expert and imperfect demonstrations (Section B.T). Then, we derive the closed-form solution to the sub-problem of the aforementioned optimization and obtain simple convex optimization objective (Section 3.2). Since the objective unstable practice, we transform the objective to an alternative yet still convex objective (Sec- tion 3.3). Finally, we show how to extract the policy from the learned correction term (Section 3.4)
3.1 TRANSFORM CONSTRAINED OPTIMIZATION INTO NESTED OPTIMIZATION
In the context of offline RL, most works expected return maximization with some regularization to overcome the extrapolation error in offline settings (Fujimoto et al. 2019) Kumar et al.) 2019
Nachum al. 2019b; Kumar et 2020 Lee al.) (021). In this work, we use KL divergence minimization between d" and d with KL-regularization:
arg max (d' aDKI (dFlldd)
where 2 is hyperparameter that controls the balance between minimizing KL divergence with d and preventing deviation of d from d
Many online AIL algorithms estimate divergence between expert and current policy using on-policy samples, which is not available in offline scenario. In contrast, to construct a tractable optimization problem in the offline setting, we consider a problem equivalent to Equation 4 in terms of stationary distribution d:
max - -DL(ddldF) - aDKI(dlld"
(5)
s.t Σ d(s,a) = =(1 -(1-2)p0(s) + Σ T(s)3,a)d(s,0) As.
(6)
3.a
d(s,a) 20 Vs.a.
3

---

Published as a conference paper at ICLR 2022
The constraints 7 called the Bellman flow constraints. The Lagrangian of the above strained optimization problem is
max min -Dki (dladdl 420
(8)
where v(s) are the Lagrange multipliers, (B,d)(s) Ea d(s, a) is the marginalization operator, and (s) == >.. 8 a a)d(s, a) is the transposed Bellman operator. We introduce following
derivations for the optimization to obtain tractable optimization in the offline setting:
-Drt(dldd)) - aDrr(dldd))
v(s) - 7))oos) - (TTd)(s) (B)d)
( v(s) log d(s,a)
d(s, a) a log du (U(s,
-1- (7)psspol((s)) - (s) ((Tu)(s,))
(8(00))
d (s,
-(1 + (0)))oo log d(s,a)
= (1 - 7) Ro [v})] Rd (TTu)(8, -(s) + log du (s,a)
(10)
d (U((s,))
==0(,,0)
the equality in Equation holdd from the following properties of transpose operators:
Σ (B.)(())))(s) Σ (Bu))(()) and Σ ((())(()))))>>> - ((T)
where (Bv) v(s), (T)) = =.. T(s' a))(s')), with assumption d (s, a) > 0 when d(s, > (Nachum al. 2019a). We introduce another log ratio, denoted by r(s, a) Equa-
tion to avoid using log a(s,a) which requires on-policy samples estimate. Unlike log d(s.a)
d' (s.a)
(s.a)
we can estimate r(s, a) in the offline setting using d and d", we will discuss in the next section
detail.
We change the distribution used in the expectation of Equation 10 from d to dd by following the standard trick of importance sampling as follows:
(1 - Es [s)] + (s) (r,, a) >TTu)
-(1 (BT)))s.)--((v(((. (Bv) log w(s a)]
= = using
(1 Ba [M)s + (s
- (1 aog
=: -[),v,r).
(11)
As an alternative, one can convert expectation of d to d instead of d in Equation 10 by the similar application of the trick of importance sampling.
(1 - BB [s)] + (ss,a)=a [exp(-r(s, a) w)) (A), (1 a) log (oss,)))]
In practice, due to the small number of demonstrations d there may a lack of diversity. To release this practical issue, prefer to use d instead of d=
In summary, DemoDICE solves the following maximin optimization:
max min L(w, v;T),
(12)
w z00
where r is trained by using precollected datasets. Note that the solution w* of Equation 12 with the ground-truth ratio r the ratio of two distributions, the stationary distribution d' the expert policy nt and the stationary distribution d of union of expert and imperfect demonstrations.
3.2 PRETRAINED STATIONARY DISTRIBUTION RATIO AND A CLOSED-FORM SOLUTION
To solve the problem 12 we can pretrain the log-ratio of d (s, a) and d (s, a). To this end, we train a discriminator c x A + [0, 1] using the following maximization objective:
max Fe dd)) = Ege (log c(8, a)] + Egu [log(1 c(8, a))] -=
(13)
4

---

Published as a conference paper at ICLR 2022
whose maximizer is (((,,)
d (s.a) a (s,a)+ (s,aa By using c*, can be also obtained as
r(s, a) loo (c(ssa) -1).
(14)
Since the optimization 8 is convex optimization and the strong duality holds, the optimization 8 is equal to
min max L(w, v;
(15)
w20
The closed-form solution of inner max optimization of 15 turns
be w)(s
(a((..)) for all (s, a), where Au(s, a) == =((s,)) +T a) v(s). By using w a),
exp
1+a
optimization (15) can reduced to a simple convex minimization problem as follows:
AA (A
1+a - ]]]
min L(w v:r) 7) + (1 + (s..
exp
(16)
Although L(w r) in Equation 16 is convex on v, we observe that T. [. v; in Equation 16 is numerically unstable in practice. Especially, the exponential term in L(w v2 is prone to explo- sion and so is the gradient V, LW vr)).
3.3 ALTERNATIVE CONVEX OPTIMIZATION
In order to derive a numerically-stable alternative objective, we describe the following theoretical result:
Proposition 1. Define the objective T(v;r) as
Ap a)
T(u;r)= = (1 71-7)EE-~p [(s)] + (1 + a:)log (wa))dd (
exp
(17)
1 1
Then, the following equality holds:
min L(w uzr) min L(vrr).
(The proof can be found Appendix
The objective L(w r) in Equation 16 has the instability issue since the gradient with respect to v involves an unbounded explod-ex function exp(-) contrast, the alternative objective [l:rr) ) in Equation 17 does not suffer from the same stability issue because in this case, the gradient forms soft-max and is bounded by Furthermore, we observe that the surrogate objective still convex:
Proposition 2. The objective ī(v:r) )) is convex with respect to v. (The proof is in Appendix C
To summarize, we have presented a training objective Σ(v;r) which significantly stabilizes the objective I[w r) while having the same optimal value. We converted the minimax optimization (15) to single minimization, which much more stable. Proposition is, however, limited that is not explain how to sample actions from learned E. Thus, we need method to sample action from learned and the next section presents such method.
3.4 POLICY EXTRACTION
In this section, we present a method for extracting policy from the optimal i arg min, L(vrr). Our method is based on the following optimization, which amounts to the variant of weighted BC:
min T. (s.a)) [log (als)) ] = -I -F(s, [w (s, a) (a)s)))
(18)
where w² (s,
d= (s.a) It can be estimated by E arg min, L(w, v; r). but we use ((vrr) d
in Equation[] 17 to train v. Here, from the Proposition T we introduce theoretical results to explain the relation between n² arg min, Iw r) and o arg min, (D;:r):
5

---

Published as a conference paper at ICLR 2022
Corollary 1. Let V arg min, L(v;r) r) and V = arg min, L(u v:r) be the sets Then the following equation holds: (We provide the proof in Appendix
[ [
(19)
We observe that for any D' V, there are v² V and c E R such that i -V" + c. Based on this relationship, can easily derive the following proportion: (see Appendix D for more detailed derivation)
Ap.. (s,a) (p, + a
d' (s,
Wp- (s, exp
d (s,
In other words, using 2² E V. we can only obtain unnormalized stationary distribution ratio instead of exact ratio. Therefore, we estimate Equation 18 usiigg self-normalized importance sam- pling (Owen 2013) as shown below:
min J,(D")== ((sa) ( (s, a) log [(]ss)
(20)
(e (wp- a)]
Based on the weighted BC, we simply extract the policy from i* E V. Our policy extraction is highly related to variants of weighted BC (Wang et al.) 2018 2020: Siegel et al., 2020) in offline RL. The common idea of these methods is (1) training
action- value function ((s a. and (2) estimating the corresponding advantage A(s, a). Using an increasing, non-negative function f()) (e.g. exp(-)), they perform weighted BC on precollected dataset with defining weight [(A(s, a), 8, as:
arg min (( [- ((A(( a), log
Finally, if regard the advantage of optimal v" in Equation T6 advantage function, policy extraction matches the offline RI. based on weighted BC.
RELATED WORKS
Learning from imperfect demonstrations Among several recent works (Wu et al.
2019 Brown
ctt al. 2019 2020 Brantley et al.] 2020 Tangkaratt et al.] 2020; (asaki & Yamashina 2021 Wang
al. 2021) that attempted to leverage imperfect demonstrations in II, two-step importance weight-
ing imitation learning (2IWIL), generative adversarial imitation learning with imperfect demonstra-
tion and confidence (IC-GAIL) (Wu et 2019), weighted GAIL (WGAIL) (Wang 2021). and
Behavioral Cloning from Noisy Demonstrations (BCND) (Sasaki & Yamashina 2021) are closely related work.
2IWIL, IC-GAIL and WGAIL suppose imperfect demonstrations which are composed of expert and non-expert trajectories and this definition the same that of DemoDICE. Assuming that some of trajectories in the imperfect demonstrations are provided with labels of confidence of optimali- ties, 2IWIL trains a supervisd classifier to predict confidence scores for unlabeled imperfect demonstrations and train the policy using the score. Instead of such two-step learning, IC-GAIL trains the policy in an end-to-end fashion.
Inspired from Wu et al. (2019), Wang et al. (2021) suggest a general weighted GAIL objective function. Unlike previous work, WGAIL. predicts the weight for imperfect demonstrations without information about their quality. However, 2IWIL, IC-GAIL, and WGAIL are online on-policy imi- tation learning method, whereas our algorithm, DemoDICE, an offline imitation learning method.
BCND assumes that the expert demonstrations sampled from noisy expert policy. BCND learns ensemble policies with a weighted BC objective, where the weight is policy learned by weighted BC in the previous internal iteration. As the learning progresses, the mode of the learned policy moves towards that of the noisy expert policy. Thus, the learned policy tends to converge to a non-expert policy when the non-expert trajectories comprise the majority of demonstrations.

---

Published as a conference paper at ICLR 2022
Stationary distribution corrections In RL and IL, some prior works have used distribution cor- rections. AlgaeDICE (Nchuum et al., 2019b), regularization in the space of stationary distribution augmented policy optimization objective solve affi-poliy RL problems. In addition, by ing dual formulation f-divergence and change variables (Nachum et al. 2019a), an off-policy learning objective is derived in AlgaeDICE. ValueDICE (Kostrikov al. (020) minimizes the KI. divergence between the agent and the expert stationary distributions to solve IL problems.
Similar the AlgaeDICE, ValueDICE obtains off-policy learning objective for distribution matching by using the dual formulation of KL divergence and change of variables. Both AlgaeDICE and ValueDICE objectives are optimized by nested optimization, which may suffer from numerical instability.
In contrast, OptiDICE (Lee et al. 2021) reduces the same optimization used AlgaeDICE uncon- strained convex optimization. Although it shows promising performance in offline RL tasks, directly applying offline from expert and imperfect demonstrations not trivial as we discussed in Section 3
Off policy learning from observations (OPOLO) Zhu et al. 2020) addresses off-policy IL using expert demonstrations composed only of states. It has similarities to DemoDICE in two aspects. Firstly, OPOLO adopts a discriminator estimate a state-transition distribution ratio. Furthermore, DICE-like technique is applied to the derivation of the objective of OPOLO, which is similar to ours. Thus, based on the relation between DemoDICE and OPOLO, offline using expert demonstrations containing only states be considered promising direction for future work.
EXPERIMENTS
In this section, we present the empirical performance of DemoDICE and baseline methods on Mu- JoCo continuous control environments [Toorrov et al. 2012) using the OpenAI Gym (Brockman jtt al. 2016) framework. We provide experimental results for 4 MuJoCo environments: Hopper, Walker2d, HalfCheetah, and Ant. We utilize D4RL datasets (Fu et al.) 2020 to construct expert and imperfect demonstrations for our experiments. We construct datasets, baselines, and evaluation protocol according the following procedures:
Datasets For each of MuJoCo environments, utilize three types of D4RL datasets (Fu al., 2020). whose name end with "-expert-v2", r-full_repla--v2", or "-random-v2". In the remain- der of this section, refer them by using corresponding suffixes. We regard the trajectories in expert v22 as expert trajectories. The set of expert demonstrations D consists of the first trajec- tory expert-v2.
Baselines We compare our method with three strong baseline methods, BC, BCND Sasaki & Yamashina 2021). and ValueDICE (Kostrikov et al., 2020). To consider the potential benefit of utilizing D we carefully tuned BC with 5 different values of 3 {1, 0.75, 0.5, 0.25, 0}, which controls the balance between minimizing negative likeliioddood of D and minimizing that of D = D UFU. D' follows:
min (nC = DE
Σ loga(a - (1 -1- 5)- D
Σ
log loga(als))
(s.0)ED#
(e.0)ED"
We denote those different settings by BC( BC(3=0.75), BC(B=0.5), BC(3=0.25), BC(B=0). For BCND, use all demonstrations noisy expert demonstrations and report statisties of the best performance among 5 internal iterations over 5 seeds. We provide the details for BCND in Appenedix E.1 Lastly, made ValueDICE to utilize imperfect datasets by plugging in all the demonstrations to replay buffer.
Evaluation metric The normalized scores for each environment are measured by random normalized score 100 x expert acore- random scoret where expert and random scores average returns of trajectories in expert-v2 and random-v2, respectively. We compute the average normalized score and the standard error over five random seeds. In the following subsections, provide experimental results
types of imperfect datasets.
7

---

Published as a conference paper at ICLR 2022
HoopeerMill (1400-100)
Walker2ddM1(0000110000
Ant:A014400-110
MalfCheetahMM110000
Nomalizdd Retuma
80
82
60
60
60
-0-
43
42
42
20
22
22
2.EM
1N
5.5H 22H
0.2N 0.4H
D.NN
0.0N
0.2M 0.4M
(2040-000(00
Walker2d (420004000)
Ant-A12(000:400
HallCheetth-M224004400))
Nomalted Rituma
82
60
60
60-
-0
43
42
42
20
22
22
3.2H
5.EH
1N
5.5M 52H
5.0M
0.2N
0.1M
D.NN
0.0N
0.2M 0.4M
Mcp0pe-M36600:1000
Wallker26-M(c000100000
Am0-033440
Nomaatzd Recuma
80
82-
60
60
-0-
43
42-
42-
20-
20
23-
22
5.0M 3.2M
2.MM
1N
5.M
5ON 0.1M
0.2
B.NN
IM
0.NN
0,4M D.E
Training
Training
Iterations
Training
Expert
BCND
BC 18 0)
BC (8 0.5)
BC (6=1)
ValueDICE
DemoDICE (ours)
Figure 1: Performance of DemoDICE and baseline algorithms on mixed-dataset tasks M1, M2, and M3. Especially, in the M3 task, which contains large number of bad trajectories, DemoDICE maintains performance while ValueDICE, BCND, and BC fail to achieve competitive performance. We plot the mean and the standard errors (shaded area) of the normalized scores over random seeds.
5.1 MIXED DATASET
DemoDICE mainly aims overcome distributional drift caused by the lack of expert demonstra- tions. When DemoDICE successfully learns the optimal unnormalized stationary distribution ratio, can expect that DemoDICE distinguishes expert trajectories from non-expert ones update its own policy. Based on this intuition, we hypothesize that if the same sufficiently-man expert tra- jectories are included in imperfect demonstrations, DemoDICE achieves the optimal performance invariant to the number of sub-expert trajectories in imperfect demonstrations. To see if it is the case, we use mixed datasets which have the same expert trajectories but have different numbers of non-expert trajectories imperfect demonstrations.
Experimental setup Across all environments, consider 3 tasks, each of which is called one of M1, M2, or M3 and is provided with expert and imperfect demonstrations. While sharing the expert demonstrations composed of the same single expert trajectory, imperfect demonstrations in each task are composed of expert and random trajectories with different ratios such as 1:0.25 (MI), 1:1 (M2), and 1:4 (M3). Specifically, imperfect demonstrations in MI, M2, and M3 composed of 400 expert trajectories sampled from expert-v2 and 100, 400, and 1600 random trajectories sampled from random-v2 respectively.
Result We report the result of DemoDICE and comparing methods in Figure For simplicity, BC(B3 = 0.25) and BC( 0.75) are in Appendix(F) Note that imperfect demonstrations in the tasks MI, M2, and M3 share the same expert trajectories, while having the different number of random trajectories. Ideally, if an algorithm uses only expert trajectories from imperfect demonstrations, the algorithm should have the same performance in the tasks M1, M2, and M3, as discussed at the beginning of this subsection, except convergence speed. While ValueDICE, BCND, and all the variants of BC catastrophically fail to recover expert policies in task M3 on all the environments except Walker2d, DemoDICE reaches the expert performance regardless of environments and tasks.
8

---

Published as a conference paper at ICLR 2022
Hopper-RB
Walker2d-RB
Ant-RB
Ha IChhheettt-88
100
100
83
03
02
85
Nomaltzed
3-
5:38
2.3H 0,40
0.8M
5.MM IM
5.0M
52M 0.4M
0.5%
C.NN
IN
5.0
0.2H 0.4M 0.5% D.NN
IM
D.NN 0.29 0.4 066% 0 1V
Training Iteratios
Training Ierratons
Training Iborations
Training Iteration
Expert
BCND
BC (p= 0)
BC (8 0.5)
BC (8 1)
ValueDICE
DemoDICE (ours)
Figure 2: Performance of DemoDICE and baseline algorithms for replay buffer task RB. In RB tasks, DeMoDICE achieves better or competitive performances compared to the other baselines. We plot the mean and the standard error (shaded area) of the normalized scores over random seeds.
The result strongly implies that DemoDICE succeeds to learn appropriate stationary distribution correction, which is possible only when the algorithm effectively leverages expert trajectories in im- perfect demonstrations ignoring the rest. Furthermore, it is remarkable that DemoDICE reaches the expert performance HalfCheetah in the end, which can be found in Appendix F.T To summarize, have empirically shown that DemoDICE robust any choices of the environments and tasks and significantly outperforms existing methods on offine II..
5.2 REPLAY BUFFER DATASET
When we use mixed datasets as imperfect demonstration sets, ValueDICE poorly performs in all environments except Walker2d as we observed in Section [. Since full.replay-22 consists of the replay buffer's data during off-policy training, believe it is one of the most practically- relevant datasets to ValueDICE's replay buffer. Therefore, we conduct additional experiments using this dataset as the imperfect demonstrations.
Experimental setup Applying the same expert demonstrations used in Section 5.1 we employ another task named RB, which uses full full.replay-v2 directly as its imperfect dataset.
Result Figure 2 summarizes the result of DemoDICE and baseline methods. We observe that De- moDICE performs on par with ValueDICE in Walker2d. However, on the other environments, De- moDICE strictly outperforms ValueDICE. We also observe that DemoDICE performs on par with BCND in Ant, HalfCheetah environments, and significantly outperforms in Hopper and Walker2d environments. While showing the performance competitive to the best performing baseline meth- ods in Walker2d, Ant, and HalfCheetah, DemoDICE outperforms all the methods in Hopper by significant margin.
Since the replay buffer dataset is collected by the policy which evolves (in policy training phase), it can be regarded as a set of imperfect demonstration from a wide range of sources. We emphasize that DemoDICE performs well not only when bad trajectories make up majority of the given imperfect demonstrations but also when they are generated from multiple sources.
CONCLUSION
We have presented DemoDICE, an algorithm for offline IL from expert and imperfect demonstra- tions that achieves state-of-the-art performance on various offline IL tasks. We first introduced regularized offline IL objective and reformulated the objective so as to make it natural to compute closed-form solution. We then tackled the instability coming from the naive application of closed- form solution by the alternative objective which yields not only the same optimal value but also a stable convex optimization. Furthermore, we presented the method to extract an optimal policy with simple weighted BC. Lastly, our extensive empirical evaluations showed that DemoDICE achieves remarkable performance close the optimal by exploiting imperfect demonstrations effectively.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation (NRF) of Korea (NRF- 2019RIA2CI087632 NRF-2021M331A10999
and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea govemmenn(MSIT) (No.2019. 0-00075,
No.2021-0-0200 HY was supported by the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF-201 8R1 ASA 1059921) and also by the Institute for Basic Science (IBS- R029-CI).
REFERENCES
Kianté Brantley, Wen Sun, and Mikael Henaff. Disagreemeen--regulaaized imitation learning. In International Conference Learning Representations (ICLR), 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: 2016.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International Conference on Machine Learning (ICML), pp. 783-792. PMLR, 2019.
Daniel S Brown, Wonjoon Goo, and Scott Niekum. automatically-ranked demonstrations. In Conference on Robot Learning (CoRL), pp. 330-359. PMLR, 2020.
imitation learning via
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning (ICML), pp. 2052-2062. PMLR, 2019.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of Wasserstein GANs. arXiv preprint arXiv: N00iv:/704.00000 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Liyining Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization. arXiv preprint 1rXiv:190S.12888 2019.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Discriminator-acoorcrttti: Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations (ICLR), 2019.
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off off-policy distribution matching. In International Conference on Learning Representations (ICLR), 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with Fisher divergence critic regularization. In International Conference on Machine Learning (ICML), pp. 5774-5783. PMLR, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off off-policy Q- learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Jongmin Lee, Wonseok Jeon, Byung-Jun Lee, Joelle Pineau, and Kee-Eung Kim. OptiDICE: Offline policy optimization via stationary distribution correction estimation. In International Conference Machine Learning (ICMI.), 2021.
10

---

Published as a conference paper at ICLR 2022
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01663, 2020.
Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellar duality, arXiv preprint arXiv:2001.01 016,,, 2020.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li, DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems (NeurIPS). 2019a.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. AlgaeDICE: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.00 2019b.
Andrew Y Ng and Stuart J Russell. Algorithms for inverse reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML). 2000.
Art B. Owen. Monte Carlo theory, methods and examples. 2013.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neu- ral computation, 3(1):88-97, 1991.
Stéphane Ross, Geoffrey Gordon, and Andrew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 627-635, 2011.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In Interna- tional Conference on Learning Representations (ICLR), 2021.
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki. Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. International Conference on Learning Representations (ICLR), 2020.
Richard S Sutton, Andrew G Barto, et al. Introduction reinforcement learning, volume 135. MIT press Cambridge, 1998.
Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, and Masashi Sugiyama. Variational imitation learning with diverse-quality demonstrations. In International Conference on Machine Learning (ICML), pp. 9407-9417. PMLR, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Qing Wang, Jiechao Xiong, Lei Han, Peng Sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. In Advances in Neural Information Processing Systems (NeurIPS). pp. 6291-6300, 2018
Yunke Wang. Chang Xu, Bo Du, and Honglak Lee. Learning to weight imperfect demonstrations. In International Conference on Machine Learning (ICML), pp. 10961-10970. PMLR, 2021.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg. Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. In Advances in Neural Information Processing Systems (NeurIPS). 2020.
Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imi- tation learning from imperfect demonstration. In International Conference on Machine Learning (ICML). pp. 6818-6827, 2019.
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from observa- tions. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
11

---

Published as a conference paper at ICLR 2022
A PROOF OF PROPOSITION
We provide the detailed proofs based on the previous works (Nachum & Dai 2020 Lee et al. 2021 Therefore, highly recommend read the works for better understanding. We first provide lemma to prove the proposition:
Lemma (Fenchel duality). Consider a primal problem given by
min Jp(z) = g(x) + h(Ax),
wEit
where g, h Q - R are convex, lover-semi-continnuus and A is a linear operator. Then, the dual of this problem is given by
max
Jn(y) = -9. -9.(-A.1) h.(y),
yEn
where A, is (Hermitian) adjoint operator of A, i.e., (y, Ax) = (A.y, x) for all
Proof. We borrow the proof steps from Nachum Dai 2020). Remark that the definition of conju- gate function of function f:
fv) = max(#, y) f(x),
where (.-) is inner product. Based on the definition, we can easily derive the following equations:
min (x) + h(Ax)= = min max g(x) + (y. Az) h, (3)
min g (x) + { Az) } - hx
max
max { {-max( (max(-A_1))
g(x)) } h (y)
9
max - 9a( (9 (.,) - h1 (y)
Now, we prove the proposition: Proposition 1. Define the objective L(v; r) as
(A((8,a)
i(v;r) (1- [v()] (1 B (sa
exp
(17)
1+a
Then, the following equality holds:
min L(w v;r) min L(v;r).
(The proof can be found in Appendix
Proof. We first define two functions:
g(-) (((1-2)p0)()) (-), h(-) (-r,)) ) + + a) DKL (-110))
where
d (s. a)
(s) := log du (s. 0
0 ECC,
8c(x) oo otherwise.
Then, the corresponding conjugate functions are given by
g+(g (.) Ep [t)
h. (+) a:) log Eay exp
1 a
12

---

Published as a conference paper at ICLR 2022
Now, rewrite the optimization problem 5 follows:
max -(((1))) (TT B) ( aDr
=0 ( ( B).d) ((s,))al-r(ss a)] - (1+ DKl(dddd - (TT - B).d) - h(d).
-
By Lemma we reformulate the optimization problem as:
max -9( (TT - B).d) - h((d)
= min (v) + h. ((>T - -B)
(TT B) +
= - 7) B [+] (1 + a:) log Egy exp
1 1
((T a)) (Bu) (B) +r(s,a)
= (1 - >) Spu [s)) + (1 a) log E (s.a)ad
oxp
1+a
((3,0) + (TT))(s, a) - -m(s)
(1 7) E Spp (8)] (1 a) log R (s.a)ad
exp
1
AA((s,a)
= - 7 [()s] + (1 a:) log RE (s.a)ad'
exp
1
It means that min, L(w r) and min, [(v;r) are the same optimization problems.
We obtain the conjugate function 9. using the fact that (a, y) is the conjugate of 8, (x)) We also compute h using the two facts: 8 f-( (MT") the conjugate of (a, x) + b f(x) and log E. [y)] is the conjugate of Drr (x)
13

---

Published as a conference paper at ICLR 2022
B PROOF OF COROLLARY
We remind the objectives in Equation and Equation 17
AA(8,
I(w = (1 - - [vs)) + (1 + (s.))
exp
1+a
AA(s, a)
[(urr): = (1
a)) log (e,a)
exp
1 +
where
(= (TT):)))((((s v(s),
u)( = exp 1+ -1)
A
Note that for any constant C, the advantage for Cs:
(s,+c)(s,0) r(s, (TTu))++(T+++)))+))))) (u(s)(()))))))))))(( + о)
a + ([((]s(..)) (O())+0)
(TT)),a)) 100-011-1 (1-7)C Au(s,a)
Lemma 2. For arbitrary function v and constant C, the equality
T(u;r) = T((v C;r)
holds.
Proof. From the definition of L(v;r), we derive the following equation:
T(v C;r)
(1 - Ex (1 a) log E
(s,a)ad
exp
1+a
Au(s, -
(1 -11-)00) )
(1 7) Ranpu
(1 0:) log (a,anaad
exp
I+a
7)C
(v(
(1 ? Eanpo
(1 0) log exp
exp 1+a
1+ a
AA(s,
(1 (1-
(1 Banpu (M() C (1 a:)
log log (aaa)ad
cxp
Ita
1 a
(Alssaa)
(1 [v(s] + (1 + ax) log R (aaaaad
exp
1 a
-(p:r)
Lemma 3. For any function v, the the following inequality always holds:
L(w v.r) L(v;r).
Equality holds if and only if
(s a)
-1]]
E E(
exp
1 1+a
Proof. For any 2 0.
y 12 log y:
14

---

Published as a conference paper at ICLR 2022
and equality holds if and only if y 1. From this inequality, we can derive the inequality directly:
Ap (1. a)
- 1 }
A, a)
)
exp
2 log E (o,a))d)
exp 1+a
1+a
and equality holds if and only if
Au(s,a) -1)
-1) 1.
E (s.0) exp
1+a
Finally, we obtain the following results:
Ap(s,
-1) 1
L(wi, v;r) = (1 7) [M)s + (1 a) (s,))
exp
1 I+a
2 - -1177)a-poll) (b)) (1 a) log Elsaaann
Au(s,a)
-1)
+ +11
exp
1
(A((s,a) - 1) exp((s)
2 (1 - 2 (s)s (1 a) log B(s,a)na''
exp
1+
[A,(saa)
(1 7) [m()] + (1 + a) log (e
exp
1+ a
T(urr).
The equality holds if and only if
(Au(s,a) a)
l+a -
[]]]
E(
exp
Lemma 4. For any optimal solution i0 arg min, [(v:r), there is a constant C such that i + an optimal solution of min, L(w,,
Proof. Let i* be an optimal solution of arg min,, L(v; r) and
+ a C = -
exp ( [ (), - 1
log B(a.a)aa
1+a
Then, = D + O satisfies
I+a Ab(saa)-))
E (s
exp
= =Rassayal
exp
1
(p,
-(1- 7) -1)
=Eisa)adu
exp
1
(s, 1 1-7 0* -m-1
= =El
exp
1+a
1+a
(s,
-1)
-1-200)
= =E(sa)<d'
exp
exp
1+a
AAp-(s,a)
-1] B(o.a)aad
exp ( (Ap-(s)a) (A
lta - -))13
exp
1+aa
11
Furthermore, P also an optimal solution of min, [(v;r) r) by Lemma Then, by equality condition in Lemma 3
I(w Dr L((:r) min [(v:r) < min (ww) It means, ú is optimal solution of min, L(wū,u;r) v;r)
15

---

Published as a conference paper at ICLR 2022
Lemma 5. An optimal solution v arg min, L(w2, v:r) is also optimal solution of min, L(v;r)
Proof. From the Lemma 3
min L v;r) L(wG..V": 2 :(v":r).
Lemma 4 show that min, L( v;r) and min, (w; r) have the same minimum value,
:(v:r))= S L( ;r) min [(w v; = min L(v;r)
holds. It means that T(v":r) is the minimum value of T. and therefore, v' is optimal solution of min, L(v;r).
Finally, we ready to prove the corollary:
Corollary 1. Let V arg min, ((rr)) and V arg min, L(w v;r) be the sets v. Then the following equation holds: (We provide the proof in Appendix B
V - V,c eR R}
(19)
Proof. Lemma and Lemma shows that the last equation holds.
16

---

Published as a conference paper at ICLR 2022
C PROOF OF PROPOSITION 2
Proposition 2. The objective [(v;r) is convex with respect to v. (The proof is in Appendix C
Proof. For any functions v, v' and constant E [0, 1J. we can derive the following equality:
A004(1-N12($s,0) + - ())((s) 77 - - (1- -1-x))((s))
>(r(ss + - + +
- \s
= (A) + (1 (1- (s,
means, Ay is the linear function with respect to v. Furthermore, using the convexity of log-sum-
exp.
log ((aa)add [exp(A(au++e (x,
= log E (e (exp(AAu(s,a)) = (1 (8,
S Alog Bleaa)ad [exp( (2 a))] + (1 - X) log (s,a)aa [A a)))
Therefore, Σ(v; r) is convex function with respect to v.
D POLICY EXTRACTION By the definition of A,,
A1000(600011
exp
Ita
-1 -1)
= exp
1ta
(A
(I+a (1-3c--1
=
exp
I+a
exp
Therefore,
Ag- (s,a)
(
A
-1)
d' as
exp
1 1+
exp
1+a
d (s, a)
17

---

Published as a conference paper at ICLR 2022
E EXPERIMENTAL DETAILS
Implementation detail For fair comparison, use the same learning rate to train actors of BC and DeMODICE. In DemoDICE, we applied the same regularization coefficient a 0.05 Equa- tion 17 across all the tasks and dataset configurations. We implement our network architectures for BC and DemoDICE based on the implementation of OptiDICH" For ValueDICE, we use its offi- cial implementation" without any modification to network architectures hyperparameters. Based on DAC (Kostrikov et al. 2019). we treat terminal states as absorbing states in ValueDICE and DemoDICE.
For stable discriminator learning, we use gradient penalty regularization on the r(s, a) function, which was proposed in Gulrajani al. (2017) to enforce I-Lipschitz constraint. To stabilize critic training, we add gradient L2-norm to the critic loss for the regularization. Detailed hyperparameter configurations used for our main experiments are summarized in Table I
Hyperparameters
BC
DemoDICE
(discount factor)
0.99
0.99
(regularization coefficient)
0.05
learning rate (actor)
3 x 10-5
3 x 10-6
network size (actor)
[256, 256]
[256, 256]
learning rate (critic)
3 x 10-1
network size (critic)
[256,256]
learning rate (discriminator)
x 10-4 [256,256]
network size (discriminator)
gradient L2-norm coefficient (critic)
x 10-4 10
gradient penalty coefficient (discriminator)
batch size
256
256
# of expert trajectories
# of training iterations
1,000,000
1,000,000
Table 1: Configurations of hyperparameters used in our experimental results.
E. EXPERIMENTAL DETAILS FOR BCND
We implement the BCND based on the BCND paper (Sasaki & Yamashina 2021). We shuffle all demonstrations and divide them into K disjoint sets {D*J Then, for each iteration, we perform stochastic gradient ascent to update each policy Tha using D Finally, update reward R(s, a) and using
Ms,
1 1
Σ
Tot (als),
V(s,a) D,
K
K
17 EgnD Ex Tok (por (s)[s)}
Here, we use following hyperparameters as reported in paper (Sasaki & Yamashina 2021): We use M 5 (number of iteration), K 5 (number of policy / disjoint dataset), N 128 (batch size), 500 x D for each & (k E {l,..., K}. our tasks, E [3 105, 1.1 x 10%), Te(als) = K >> Ā= Taa (als), < 10-4 (shared learning rate), Adam optimizer with learning rate Ç X 17 for gradient ascent, and Gaussian policy T = N ((gs (s), o For Guassian policy, we use neural network with size [100, 100) for Mgkk, and a trainable independent vector for ogk. as reported
in the paper.
attps:
18

---

Published as a conference paper at ICLR 2022
We measure the performance of algorithm for each internal iteration. In Figure and Figure 2 report the best performance and corresponding standard error the algorithm among the M internal iterations (we compute the average performance for each iteration and pick the maximum average performance as the best performance).
Algorithm Behavioral Cloning from Noisy Demonstrations
Require: Noisy expert demonstrations D, policy parameters (0s h learning rate C
Ensure: Ensemble policy Te = K Σ Takk
Set f(s, a) for V(s, a) E D.
Split D into disjoint sets (D', D ,...,, D*
for iteration= ...... M do
for & = ......... K do Initialize parameters 0².
for I....,
Sample minibatch B (8 )n))m= from D*
Calculate sampled gradient Σn gg log (Sn an) Moss an).
Update Oh gradient ascent with learning rate Ç y)
end for end for
Update A(ss, a) and 7 end for
19

---

Published as a conference paper at ICLR 2022
F ADDITIONAL EXPERIMENTAL RESULTS
In this section, we report additional experimental results for detailed empirical analysis of our ap- proach. We focus three experiments follows: (I) asymptotic performance of our approach on HalfCheetah environment for the mixed dataset we considered in Section 5.1 (2) behavior cloning
with more diverse selections which are omitted in Figure (3) ablation study on hyperparameter
for DemoDICE, which is appeared in Equation 17
F.1 ASYMPTOTIC PERFORMANCE OF DEMODICE ON HALFCHEETAI
To inspect asymptotic behaviors of DemoDICE on our mixed dataset tasks HalfCheetah environ- ment, extend the number of training iterations into 2 million with fixing the other hyperparameter settings. As Figure 3 shows, DemoDICE converges to the expert performance for all tasks we con- sidered in Section 5
04000000000000000
HarfCheetah-M22440-0000
MaliCheetah-M334000100
120
120
120
Normalized Returs
100
100
100
60
20
0.0M
0.5M
1M
1.5M
2M
0.0M
0.5M
1M
1.5M
2M
0.0M
0.5M
1M
1.5M
2M
Training Iterationns
Training Iterations
Training Iteration
Figure 3: Learning curves of DemoDICE during million training iterations in HalfCheetah mixed dataset tasks (M1, M2, M3).
20

---

Published as a conference paper at ICLR 2022
F.2 BEHAVIOR CLONING PERFORMANCE
In Figure [] we omitted performance of BC with 3 = 0.25 and / = 0.75 for simplicity. Figure shows performance of entire BC configurations, i.e., {0, 0.25, 0.5, 0.75, 1.0}.
(4000))-M1440011000 001
Wallar26411/0000100
Ant-MM1(400:1000 100)
Half MwllCChaaaaaMM1144000000 100)
220
120
120
222
Nermatze
2330
1.30
0.0
664 0.
201
000
0.30
0.4%
6.54
COM
0.2H
2,44
0.13
3,44
0.3M
022
0:42
DON 2
(40000002
Walker22.M22440-00000
(0L-N22(00-400)
400)
120
120
120
20-
22
3.8
1.34
0.80
002
0:22
0.49
6MH
0:8M
0.29
1,24
6.1
7BV
0.2 0.42
7TH
(3(40011000
N3(400:1600)
Haifhheetah-MM1440
Nor mormat Pamurte
100
100
200
3.6M
Happer-28
HallCheetiah-BB
120
120
320
Now Normatted Returee
40
E.EH Training
0.2M
Berattons
Training Harullnna
Trainng teratins
BCND
8C:1=00.51
BC 0.5)
0.755
(8mm)
DemoDICE
Figure 4: Performance of variants of BC with 3 E (0, 0.25, 0.5, 0.75, 1.0) and DemoDICE both on the mixed datasets (M1, M2, M3) and the replay buffer datasets. We plot the mean and the standard errors (shaded area) of normalized scores over random seeds.
21

---

Published as a conference paper at ICLR 2022
F.3 DIFFERENT QUALITY OF IMPERFECT DEMONSTRATIONS
In this section, we construct new tasks where random trajectories in mixed dataset (see 5.1 are replaced medium trajectories, named 11, 12, and I3. Here, all tasks share the expert demonstrations composed of the single expert trajectory. Imperfect demonstrations in each task are composed of 400 expert trajectories and medium trajectories, and the number of medium trajectories is 100 for the task we observed that DemoDICE performs well in these new
11, 400 for 12, and 1600 for 13. In Figure 5 tasks.
Hopper-1144001 (400::00)
Icer 20-ee/66111 2400114001100
Art-111400:100
HallCheeta h- (10-000100000
120
120
120
120
Normaltzed
60
60-
60-
3.2M
0.2M
8.0M
6.aM
10
0.0M
0.25
0.14
5
5.0M
1M
0.0N
0.NM
0.NN
DINN
CNNN
3M
0.0M
0.3M 0.4M
0. B.
1M
H4gper-22440::400
We Ioer 20-221000:400)
Ant-121400-4000
00:000heeea--22400000000
120
120
120
Normaltzss seturs
000-
1.00
40
3.56
6.2M
0.0M
6.60
0.6M
1/
0.8%
0.26
0.19
5AM
5.M
0.0N
0.5M
D.IN
D.EN
354
0.2M
Hoppes-B(4D0-160))
Wollke/2d43/440:166e
Ant-13/400:1600 6000)
TaIIChheetal-331400110000
Normaizzs Recurts
100
100
1.00
40
20
20
6.NN
Training leerations
Theiring Barailons
Traiiing
Training Iterations.
Expert
Medium
BC DC.03w0 0.50
BC 11
WHHCKE
DemoBICE curst
Figure 5: Performance of DemoDICE and baseline algorithms on alternative mixed-dataset tasks 11, 12, and 13. Bold lines and shaded areas indicate the means and the standard errors of normalized scores over random seeds respectively.
22

---

Published as a conference paper at ICLR 2022
F.4 DIFFERENT COVERAGE OF EXPERT DEMONSTRATIONS
In this section, we introduce additional experiments with different coverages of expert demonstra- tions. Concretely, constructed expert demonstrations using increasing number of expert tra- jectories, namely 2, 5, to 10. Because DemoDICE already performs well in M1 and M2 tasks, we compare M3 and RB tasks for this ablation study. In this experiments, DemoDICE is not signif- icantly better than other baselines when the expert demonstrations cover enough space. We remind the readers that when the coverage not high, DemoDICE significantly outperforms the others as shown in Figure 1 and 2 in the main text.
Hopper-MM(400:1660
Walker2d-MB/400-16000
HallCheetaa-A43400-1000
Avl--M3(400-1600
Normatee
Hopper-RB
Walker2d-RB
HallChhetah--BB
Ant-RB
Number Expert Trajectodes
Number of Expert Trajectories
Number Expert Trajectories
Number Expert Trajectories
Expert
Figure 6: We compare DemoDICE with baselines when the number of expert trajectories constitut- ing expert demonstrations is 2, 5, and 10. Each performance is measured normalized score averaged by the last 5 evaluations during training. Bold lines and shaded areas indicate the means and the standard errors of converged performances over random seed respectively.
23

---

Published as a conference paper at ICLR 2022
F.5 ABLATION TEST ON HYPERPARAMETER
As we can notice in Equation 17 DemoDICE introduces a hyperparameterr which is a regulariza- tion coefficient. In this section, aim study how does the choice of effect training processes and performance of our algorithm. We select a E {10, 1, 0.5, 0.1, 0.05, 0} and train on M3 datasets and replay buffer datasets by preserving the other settings same as before. Figure describes the results of ablation study. We point out that DemoDICE shows almost the same performance in when [0, 0.1].
Hopper-M33
HellCheetaa-AA1440--6000
Normaltzo Rocurn
00
62
63
40-
23
23
0.2M
0.4M
0.NN
0.4M
Hopper-RB
Walker2d-RB
Ant-RB
HallCheetah-RB
100
1.0
80
82
Normalized
80
63
62-
60-
40
40
40
20
Iberotions
Training Ibcrations
Traininn Itorations
Expert
a= 10
d-1 a
6=0.1
U= 0.05
a=
Figure 7: Performance of DemoDICE by varying a E {10, 1, 0.5,0.... 0.1, 0.05, 0} on the mixed datasets M3 and the replay buffer datasets. Bold lines and shaded areas indicate the means and the standard errors (shaded area) of normalized scores over random seeds respectively.
24

---

Published as a conference paper at ICLR 2022
F.6 ABLATION TEST ON PURE OFFLINE IMITATION LEARNING
In this section, we compare DemoDICE and BC using only expert demonstrations. In Figure g observe that DemoDICE and BC show similar performance. In pure offiine II, DemoDICE essentially tries to find a stationary distribution that minimizes DKL (d|[d) on the MLE MDP con-
structed by D Let d be the empirical stationary distribution of d Then, the optimal solution trivially given by d' d with DKL ( (d] 0 since d itself is valid stationary distribu- tion on the MLE MDP. Finally, extracting a policy from d reduces to simple behavior cloning: min, (s dt [log (a]ss)], which concludes that DemoDICE would not be better than BC in the pure offline II. setting.
Hopper
Hopper#22
Hopper
Hopper
122
122
120
80
Normatzed
60
60
60
20-
3
2.NM
5.5H
0.2M
0.NN
0.0N
0.4M
Walker2dit
Walker2d#2
Walker2de5
Walker2ddH1
Normal hicurns
000
102
102-
3.0M
2.3M
0.8M
0.6M
5.N
1M
3.0M
0.2M
0.49
0.9
5.0N
0.0M
0.2M
0.4M
0&6
D.NN
1M
0.0N
MalfCheetahh#1
HalfCheetah#2
HellCheetahh#5
HalfCheetahd 10
120
120
120
125
60-
Normatedd
00
02
60
3-
3.0M
3.2M
0.0M
0.69
5.5M
1M
0.0M
0.2M
0.4M
0.M
0.6M
6.0M
0.2M
0.4M
0.69
0.6M
1M
0.0N
0.2M
0.4M
0.6
Antel1
Ant#2
Ant#5
Anty
Nomatzed Ricuma
100
100
100
42
20-
0.0M 5.H
0.60 5:
6.5M
5.H
0.M
0.2M 0.4M 0.6 0.6M
6.0N
Training Ieerations
Training therations
Training lterations
Training Iherations
Exper:
BC
Demob lesport
Figure 8: Performance of DemoDICE and BC with only expert demonstrations, For each environ- ment, we construct expert demonstrations using an increasing number of expert trajectories, namely 1, 2, 5, up 10. Bold lines and shaded areas indicate the means and the standard errors of normal- ized scores over random seeds respectively.
25

---

Published as a conference paper at ICLR 2022
G DETAILED ANALYSIS OF EMPIRICAL RESULTS
In this section, we provide additional discussions about empirical results.
First, M1 is easy task where most of the trajectories from the expert, some baselines also succeed to achieve good performance.
Environment
All trajectories
> 100
>
90
>
80
>
70
> 60
> 50
Hopper Walker2d
3514
0
117
176
217
281
385
1887
0
0
325
564
638
693
Ant HalfCheetah 999
1318
333
479
576
629
672
727
0
0
0
143
467
689
Table 2: Statistics of trajectories in Full_ree ay-v2 in D4RL datasets.
To discuss about RB tasks, we show the normalized score statistics of trajectories in the replay buffer counted the number of trajectories that have normalized score greater than Rt as > R, so the numbers above are cumulative counts, Thus, this table suggests natural upper bound of performances of imitation learning algorithms, e.g. would be very hard to score around 100 for HalfCheetah task.
in the T For each environment,
In the Ant-RB task, have enough good trajectories, especially 100, both DemoDICE and BC(3 = 0) recover expert policy.
In the HalfCheetah-RB task, there is no trajectory with 80, DemoDICE eagerly distinguishes the non-expert trajectories from expert ones, throwing away non-expert demonstrations. Thus overall behaves like BC(3 0).
In the Walker2d-RB task, both ValueDICE and DemoDICE show similar performances at around 80 the end. Looking at the statistics above, since the best trajectory in RB has its normalized score less than 90, we believe that ValueDICE and DemoDICE both achieved reasonable performance levels.
26


Reviews:
[{"review_id": "cwl3MeudZdk", "reviewer": "Reviewer_5YKV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: the paper studies an important problem for imitation learning with only offline data. Several tricks in equation 11, 21 helps to improve the method over baselines. Converting the original minimax problem into a minimization problem helps to stabilize the training. Comparing with BC and ValueDICE demonstrates that the proposed DemoDICE can achieve a better performance in several MuJoCo environments. Especially, it is important to have the results of different number of combinations of expert/imperfect demonstrations. \n\nWeakness: \nThe experiments part may still need further work. \n1. In figure 1, the half cheetah environment results seem to be unfinished. The curves are not converging yet especially for M3. It would be great to see the results when the curves converge. \n2. The authors mentioned that the supplementary demonstrations are imperfect, containing expert or near-expert demonstrations. However, the experiments section used either expert or random demonstrations. It would be great to see the performance of the algorithm when sub-optimal demonstrations are provided instead of totally random trajectories. According to proposal of the paper, the work is to tackle problems where expert demonstrations are not diverse enough. Is it possible to demonstrate how the performance of the proposed method will change when the expert demonstrations are of different coverage of the state/action space? For example, intuitively, when the expert demonstrations cover enough space, the difference between BC, ValueDICE and DemoDICE may not be that significant, while when the expert demonstrations is of very limited coverage,DemoDICE will be much better than BC and ValueDICE.\n3. In the results, it seems that the more imperfect demonstrations given the same perfect demonstrations, all methods tend to have worse performance (it appears that M1,M2,M3 are using the same number of expert demos but M3 are given much more bad demos). If that is the case, why include bad demonstrations in the data given that only using good demos can achieve a better performance?\n4. In M1 task in Hopper, Walker2d, Ant, the proposed DemoDICE isn't better than the baseline BC with beta=0. Also in the case, where the RB dataset is used, DemoDICE is not better than BC for beta=0 in ant and half cheetah. It is also worse than ValueDICE in Walker2d. Is there further explanation why this happens? ", "summary_of_the_paper": "This work presents an approach that learns a policy from expert demonstrations data in an offline setting. It is trying to solve an important problem in imitation learning: the expert demonstration may not cover the entire state/action space pretty well, and many existing algorithms require further interaction with the environment. To solve the problem of distribution shift when expert data is not diverse enough, the method proposes to introduce a large number of supplementary demonstrations of various qualities. In this way, the proposed algorithm DemoDICE does not require any on-policy samples. The method adapts the OptiDICE method for its own purpose by using a transformed objective function. The theoretical analysis and empirical experiments demonstrate the improvement of the method over BC and ValueDICE.", "summary_of_the_review": "Overall the paper is of good quality and it studies an important problem. The proposed method formulation makes sense. However, the experiments part need improvement and clarification. \n\nPost rebuttal: I read the authors' rebuttal and appreciate all the feedbacks. I hope the author could incorporate the suggestions by the reviewers to improve the overall quality of the paper. I raised my rating for the rebuttal and the paper has a nontrivial contribution to the field. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_uhcp1xVD8", "reviewer": "Reviewer_SEq7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n-- The proposed setting is practical and important.\n\n-- The proposed algorithm is novel (although heavily derives from existing work).\n\n-- The empirical results & baselines are comprehensive.\n\n-- The writing is clear and easy to follow.\n\nWeaknesses/Questions:\n\n-- The 3-stage process in the algorithm is arguably more complex than baselines (e.g., BC is just a single objective, ValueDICE is a single min-max objective) and can present challenges in practice w.r.t. hyperparameter tuning of each stage.\n\n-- The derivations heavily borrow from existing work (e.g., OptiDICE, AlgaeDICE). At times, the derivations seem to be more complicated than they need to be. For example, when introducing the log-expected-exponent, it is presented as a very involved multi-step derivation. I think it could be simplified by just noting that the Fenchel dual of KL when inputs are restricted to the simplex is exactly the log-expected-exponent function; see Section 5.3.1 in https://arxiv.org/abs/2001.01866 (in fact, this existing tutorial includes a lot of derivations similar to DemoDICE).\n\n-- I have missed this in the paper, but I'd be curious to know how DemoDICE performs in the pure imitation learning setting (no imperfect demos) compared to the baselines.\n\n-- You may want to include a reference to https://arxiv.org/abs/2102.13185 which also proposes an IL algorithm that is composed of a 1st stage that does discriminator-based density ratio estimation and then uses that density ratio as a reward for a later DICE-like stage.", "summary_of_the_paper": "The paper considers the problem of imitation learning, and specifically the setting where a small number of expert demonstrations is paired with some amount of suboptimal data. To tackle this setting, the paper proposes to optimize a convex constrained optimization, with linear variables denoting d^pi, linear constraints expressing the MDP transitions, and an objective composed of a linear combination of KL from the expert and KL from the suboptimal data. The paper presents derivations transforming this convex constrained optimization to a practical, unconstrained objective. The final algorithm is composed of three stages: (1) train a discriminator to get a density ratio estimator; (2) use the density ratios as a reward in a separate log-expected-exp objective (similar to REPS or ValueDICE objectives) to learn a different density ratio w; (3) extract a policy from w using weighted max likelihood training on the suboptimal data. The algorithm is evaluated on a variety of simulated robotics (mujoco) environments, and shows favorable results compared to baselines. ", "summary_of_the_review": "Overall the paper is a good submission, and I would recommend accepting it. I hope the authors can incorporate my feedback into the final version.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WBp1RaZ_uUn", "reviewer": "Reviewer_ERsJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\nThe proposed problem is a natural and practical extension of the offline IL problem. This is a nice step towards practical applications of IL. The derivation of the method is also grounded. While the derivation steps are based on existing work, the stability improvement by using a soft-max surrogate and its accompanying proofs are good additions. \n### Weaknesses\n1) Related work\n\nIt is interesting to see that the final policy learning objective in equation 21 essentially reweighs behavioral cloning (BC) objective based on the advantage function. This result makes DemoDICE closely related to [1] which reweighs GAIL objective based on the advantage function and [2] which learns from imperfect demonstrations by reweighting a BC objective. These recent works should be discussed in the paper. \n\n2) Experiments\n\nExperiments are conducted sufficiently to demonstrate the effectiveness of DemoDICE on continuous-control benchmarks. Though, there are only two baseline methods (BC and ValueDICE) and more baselines should be evaluated (e.g., the method in [2]).\n\n3) Clarity\n\nThe paper is overall well written and well organized. However, there are unclear statements that need further clarification: \n\n  - In Section 3.3, why cannot $w^\\star$ be computed from $\\tilde{\\nu}^\\star$? In my understanding, $\\tilde{\\nu}$ is a neural network that minimizes equation (18) and the advantage function can be computed directly from it. With these, it should be straightforward to compute $w^\\star(s,a) = \\exp(\\frac{A_{\\tilde{\\nu}^\\star}(s,a)}{1+\\alpha}-1)$ on state-action samples drawn from the dataset $D^U$ to reweight the objective.\n\n  - I do not understand the sentence \"Based on the weighted BC, we simply extract the policy without training any additional network.\". It is indeed  possible to find a non-parametric $\\pi$ from equation (21). However, such $\\pi$ only defines action probabilities on states observed in the datasets and cannot be used in the environment. At the same time, the appendix does mention about a neural network for the actor. So I do not understand which  \"additional network\" the above sentence refers to.\n\n  - Proposition 1 requires $\\nu$ to be in a large family of functions. Are there restrictions of $\\nu$? Lemma 1 and 2 seem to hold for any function $\\nu$, so I think Proposition 1 should hold for any $\\nu$.\n\n### Questions\n\n1) It is intriguing to see that regularized BC does not work in the experiments but DemoDICE does given that BC is also a distribution-matching approach (without matching the state-marginals). Does the superior performance of DemoDICE come from matching the state-marginals or from the dual-program optimization?\n\n2) In theory, the policy function can be extracted from state-action distribution exactly with $\\pi(a|s) = \\frac{d^\\pi(s,a)}{\\int_A d^\\pi(s,a) \\text{d}a}$. The integration is intractable in general for continuous control tasks. However, for Mujoco tasks the action-space is bounded in $[-1,1]$. Is computing the exact policy function possible in DemoDICE with the knowledge of action-space's bounds?\n\n[1] Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to Weight Imperfect Demonstrations. ICML, 2021.\n\n[2] Fumihiro Sasaki and Ryota Yamashina. Behavioral Cloning from Noisy Demonstrations. ICLR, 2021.\n\n", "summary_of_the_paper": "The paper considers an offline imitation learning (IL) problem with an addition of supplementary imperfect demonstrations. To solve this problem, the paper proposes DemoDICE which regularizes a distribution-matching objective of IL by a KL divergence between the agent distribution and a mixed of expert and imperfect distributions. DemoDICE finds an optimal state-action distribution of this regularized objective by using a dual-program technique similar to that of OptiDICE (Lee et al., 2021) for offline RL with an improvement in terms of stability. Given the optimal state-action distribution, DemoDICE extracts the expert policy by performing weighted behavioral cloning. \nEmpirical evaluation on Mujoco tasks with D4RL datasets show that DemoDICE can efficiently and effectively solve the offline IL problem. \n\n### Contributions\n- A new learning problem combining offline IL and IL with imperfect demonstrations. \n- A new model-free offline IL method based on dual-program optimization.\n", "summary_of_the_review": "I overall like the idea and execution in the paper. I rate the paper as acceptance. Still, there are issues that need to be addressed (related work, baseline methods, and clarity). Nonetheless, these issues are minor and should be straightforward to address. \n\n** Update after rebuttal **\nI read the rebuttal and the other reviews. The rebuttal and the revision address my questions and concerns. I think the paper should be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for learning sequential decision making policies from a mix of demonstrations of varying quality. The reviewers agree, and I concur, that the method is relevant to the ICLR community. It is non-trivial, the empirical evaluations and theoretical analysis are rigorous, resulting in a novel method that produces near optimal policies from more readily available demonstrations. The authors revised the manuscript to reflect the reviewers' comments.
Decision: Accept (Poster)