Year: 2021

Abstract:
We consider offline imitation learning (IL), which aims to mimic the expert’s behavior from its demonstration without further interaction with the environment. One of the main challenges in offline IL is to deal with the narrow support of the data distribution exhibited by the expert demonstrations that cover only a small fraction of the state and the action spaces. As a result, offline IL algorithms that rely only on expert demonstrations are very unstable since the situation easily deviates from those in the expert demonstrations. In this paper, we assume additional demonstration data of unknown degrees of optimality, which we call imperfect demonstrations. Under this setting, we propose DemoDICE, which effectively utilizes imperfect demonstrations by matching the stationary distribution of a policy with experts’ distribution while penalizing its deviation from the overall demonstrations. Compared with the recent IL algorithms that adopt adversarial minimax training objectives, we substantially stabilize overall learning process by reducing minimax optimization to a direct convex optimization in a principled manner. Using extensive tasks, we show that DemoDICE achieves promising results in the offline IL from expert and imperfect demonstrations.

heading: 1 INTRODUCTION
text: Reinforcement learning (RL) (Sutton et al., 1998) aims to learn a intelligent behavioral strategy based on reward feedback. Although RL has achieved remarkable success in many challenging domains, its practicality and applicability are still limited in two respects. First, we need to specify the reward function, which may be non-trivial to do so in many real-world problems that require complex decision making. Second, the standard RL setting assumes online interaction with the environment during the intermediate stages of learning, which is infeasible for mission-critical tasks.
Imitation learning (IL) (Pomerleau, 1991; Ng & Russell, 2000) addresses the first limitation of RL, where the agent is trained to mimic the expert from demonstration instead of specifying the reward function. It is well known that adopting supervised learning for training the imitating agent, commonly referred to as behavioral cloning (BC), is vulnerable to the distribution drift (Ross et al., 2011). Thus, most of the successful IL algorithms rely on online experiences collected from the environment by executing intermediate policies during training. Recent progress made by adversarial imitation learning (AIL) (Ho & Ermon, 2016; Ke et al., 2019; Kostrikov et al., 2020) achieving stateof-the-art results on challenging imitation tasks still relies on such an online training paradigm.
Unfortunately, in many realistic tasks such as robotic manipulation and autonomous driving, online interactions are either costly or dangerous. Offline RL (Fujimoto et al., 2019; Kumar et al., 2019; 2020; Levine et al., 2020; Wang et al., 2020; Lee et al., 2021; Kostrikov et al., 2021) aims to address these concerns by training the agent from the pre-collected set of experiences without online interactions. To prevent an issue caused by the distributional shift, offline RL algorithms mitigate the phenomenon by constraining the shift or making a conservative evaluation of the policy being learned.
In this paper, we are concerned with offline IL problems. Finding an effective algorithm for these problems is tricky. For instance, naively extending the offline RL algorithms (which assume a reward function) to the offline IL setting does not work. In practice, expert demonstrations are scarce due to the high cost of obtaining them. Thus, they typically cover only a small fraction of the state and action spaces, which in turn makes the distribution drift issue even more stand out compared with
the standard offline RL setting with a reward function. We mitigate this issue by assuming a large number of supplementary imperfect demonstrations, without requiring any level of optimality for these imperfect demonstrations; they may contain expert or near-expert trajectories (Wu et al., 2019; Wang et al., 2021) as well as non-expert ones all together. This generality covers the situations from real-world applications, but at the same time, it poses a significant challenge for the design of a successful offline IL algorithm.
In this paper, we propose DemoDICE, a novel model-free algorithm for offline IL from expert and imperfect demonstrations. We formulate an offline IL objective which not only mitigates distribution shift from the demonstration-data distribution but also naturally utilizes imperfect demonstrations. Our new formulation allows us to compute a closed form solution, which learns a policy in the space of stationary distributions but suffers from the instability issue in practice. We tackle the issue by proposing an alternative objective, which leads to a stable algorithm in practice while keeping the optimal stationary distribution. Finally, we introduce a method to extract the expert policy from a learned stationary distribution in a simple yet effective way. Our extensive evaluations show that DemoDICE achieves performance competitive to or better than a state-of-the-art off-policy IL algorithm in the offline-IL tasks with expert and imperfect demonstrations.

heading: 2 PRELIMINARIES

heading: 2.1 MARKOV DECISION PROCESS (MDP)
text: We assume an environment modeled as a Markov Decision Process (MDP), defined by tuple M = 〈S,A, T,R, p0, γ〉, where S is the set of states, A is the set of actions, T : S × A → ∆(S) is the probability p(st+1|st, at) of making transition from state st to state st+1 by executing action at at timestep t, R : S × A → R is the reward function, p0 ∈ ∆(S) is the distribution of the initial state s0, and γ ∈ [0, 1] is the discount factor. A policy π : S → ∆(A) of MDP M is a mapping from states of M to distributions over actions. For the given policy π, the stationary distribution dπ is defined as follows:
dπ(s, a) = (1− γ) ∞∑ t=0 γtp ( st = s, at = a ∣∣s0 ∼ p0(·), st ∼ T (·|st−1, at−1), at ∼ π(·|st)) We assume a precollected dataset DE of (s, a, s′) tuples generated by the expert, and a precollected imperfect dataset DI (generated by unknown degrees of optimality). More precisely, for the (underlying) expert policy’s stationary distribution dE(s, a), we assume that (s, a, s′) ∈ DE is sampled as (s, a) ∼ dE , s′ ∼ T (·|s, a). We define DU = DE ∪ DI , the union of two datasets, and denote the corresponding stationary distribution of the dataset DU as dU . We denote the trajectories generated by expert policy as expert trajectories, and trajectories generated by non-expert policies as non-expert trajectories. The expert demonstrations consist only of expert trajectories and imperfect demonstrations consist of a mixture of expert and non-expert trajectories. In this paper, we assume that the quality of imperfect demonstrations is unknown.

heading: 2.2 IMITATION LEARNING
text: Behavior cloning (BC) is a classical IL approach, which attempts to find a function that maps s to a via supervised learning. The standard BC finds a policy π by minimizing the negative log-likelihood:
min π JBC(π) := min π − 1 |D| ∑ (s,a)∈D log π(a|s). (1)
However, it is known to be brittle (Ross et al., 2011) when the interaction with the environment deviates from the scarce trajectories in DE . In such cases, BC fails to recover expert policies.
One of the notable approaches for IL is to formulate the problem as distribution matching (Ho & Ermon, 2016; Ke et al., 2019; Kostrikov et al., 2020). When instantiated with the KL divergence widely used in previous IL works (Ke et al., 2019; Kostrikov et al., 2020), the approach amounts to finding a policy π by optimizing the following objective:
max π −DKL(dπ‖dE) = E(s,a)∼dπ
[ log dE(s, a)
dπ(s, a)
] . (2)
Since we cannot directly access the exact value of dE(s, a) and dπ(s, a), rwe estimate their ratio using samples from dE(s, a) and dπ(s, a), given as follows:
max c:S×A→(0,1)
E(s,a)∼dE [log c(s, a)] + E(s,a)∼dπ [log(1− c(s, a))]. (3)
Here, the optimal discriminator c∗ recovers log c∗(s, a) − log(1 − c∗(s, a)) = log d E(s,a) dπ(s,a) . Based on this connection between generative adversarial networks (GANs) and IL, AIL algorithms focus on recovering the expert policy (Ho & Ermon, 2016; Kostrikov et al., 2019). However, the agent obtain samples from dπ through interaction with environment, which is impossible in offline setting. Therefore, to tackle the offline IL problems, we should derive an alternative estimation without on-policy samples.

heading: 3 DEMODICE
text: In this section, we present a novel model-free offline IL algorithm named offline imitation learning using additional imperfect Demonstrations via stationary DIstribution Correction Estimation (DemoDICE). Starting from a regularized offline IL objective which accords with offline RL algorithms, we present a formulation that does not require on-policy samples. Such formulation allows us to construct a nested optimization for offline IL from expert and imperfect demonstrations (Section 3.1). Then, we derive the closed-form solution to the sub-problem of the aforementioned optimization and obtain a simple convex optimization objective (Section 3.2). Since the objective is unstable in practice, we transform the objective to an alternative yet still convex objective (Section 3.3). Finally, we show how to extract the policy from the learned correction term (Section 3.4).

heading: 3.1 TRANSFORM CONSTRAINED OPTIMIZATION INTO NESTED OPTIMIZATION
text: In the context of offline RL, most works use expected return maximization with some regularization to overcome the extrapolation error in offline settings (Fujimoto et al., 2019; Kumar et al., 2019; Nachum et al., 2019b; Kumar et al., 2020; Lee et al., 2021). In this work, we use KL divergence minimization between dπ and dE with KL-regularization:
π∗ := arg max π −DKL(dπ‖dE)− αDKL(dπ‖dU ), (4)
where α ≥ 0 is a hyperparameter that controls the balance between minimizing KL divergence with dE and preventing deviation of dπ from dU .
Many online AIL algorithms estimate divergence between expert and current policy using on-policy samples, which is not available in offline scenario. In contrast, to construct a tractable optimization problem in the offline setting, we consider a problem equivalent to Equation 4 in terms of stationary distribution d:
max d −DKL(d‖dE)− αDKL(d‖dU ) (5) s.t ∑ a d(s, a) = (1− γ)p0(s) + γ ∑ s̄,ā T (s|s̄, ā)d(s̄, ā) ∀s, (6)
d(s, a) ≥ 0 ∀s, a. (7)
The constraints (6-7) are called the Bellman flow constraints. The dual problem for the above constrained optimization problem is
max d≥0 min ν −DKL(d‖dE)− αDKL(d‖dU ) + ∑ s ν(s)((1− γ)p0(s) + γ(T∗d)(s)− (B∗d)(s)), (8)
where ν(s) are the Lagrange multipliers, (B∗d)(s) := ∑ a d(s, a) is the marginalization operator,
and (T∗d)(s) := ∑ s̄,ā T (s|s̄, ā)d(s̄, ā) is the transposed Bellman operator. We introduce following
derivations for the optimization (8) to obtain tractable optimization in the offline setting: −DKL(d‖dE)− αDKL(d‖dU ) + ∑ s ν(s)((1− γ)p0(s) + γ(T∗d)(s)− (B∗d)(s))
= (1− γ)Es∼p0 [ν(s)] + E(s,a)∼d [ γ(T ν)(s, a)− ν(s)− log d(s, a)
dE(s, a) − α log d(s, a) dU (s, a)
] (9)
= (1− γ)Ep0 [ν(s)] + Ed [ γ(T ν)(s, a)− ν(s) + log d E(s, a)
dU (s, a)︸ ︷︷ ︸ :=r(s,a) −(1 + α) log d(s, a) dU (s, a)︸ ︷︷ ︸ :=w(s,a)
] , (10)
the equality in Equation 9 holds from the following properties of transpose operators:∑ s ν(s)(B∗d)(s) = ∑ s,a d(s, a)(Bν)(s, a) and ∑ s ν(s)(T∗d)(s) = ∑ s,a d(s, a)(T ν)(s, a),
where (Bν)(s, a) = ν(s), (T ν)(s, a) = ∑ s′ T (s
′|s, a)ν(s′), with assumption dE(s, a) > 0 when d(s, a) > 0 (Nachum et al., 2019a). We introduce another log ratio, denoted by r(s, a) in Equation 10 to avoid using log d
E(s,a) d(s,a) , which requires on-policy samples to estimate. Unlike log dE(s,a) d(s,a) ,
we can estimate r(s, a) in the offline setting using dE and dU , as we will discuss in the next section in detail.
We change the distribution used in the expectation of Equation 10 from d to dU by following the standard trick of importance sampling as follows:
(1− γ)Es∼p0 [ν(s)] + E(s,a)∼d[r(s, a) + γ(T ν)(s, a)− (Bν)(s, a)︸ ︷︷ ︸ :=Aν(s,a)(advantage using ν) −(1 + α) logw(s, a)]
= (1− γ)Es∼p0 [ν(s)] + E(s,a)∼dU [ w(s, a) ( Aν(s, a)− (1 + α) logw(s, a) )] =: L(w, ν; r). (11)
As an alternative, one can convert expectation of d to dE instead of dU in Equation 10 by the similar application of the trick of importance sampling.
(1− γ)Es∼p0 [ν(s)] + E(s,a)∼dE [ exp(−r(s, a))w(s, a) ( Aν(s, a)− (1 + α) logw(s, a) )] .
In practice, due to the small number of demonstrations in dE , there may be a lack of diversity. To release this practical issue, we prefer to use dU instead of dE .
In summary, DemoDICE solves the following maximin optimization:
max w≥0 min ν L(w, ν; r), (12)
where r is trained by using precollected datasets. Note that the solution w∗ of Equation 12 with the ground-truth ratio r is the ratio of two distributions, the stationary distribution dπ ∗ of the expert policy π∗ and the stationary distribution dU of union of expert and imperfect demonstrations.

heading: 3.2 PRETRAINED STATIONARY DISTRIBUTION RATIO AND A CLOSED-FORM SOLUTION
text: To solve the problem (12), we can pretrain r, the log-ratio of dE(s, a) and dU (s, a). To this end, we train a discriminator c : S ×A→ [0, 1] using the following minimization objective:
min c:S×A→[0,1]
Jc(d E , dU ) := EdE [log c(s, a)] + EdU [log(1− c(s, a))], (13)
whose minimizer is c∗(s, a) = d E(s,a) dU (s,a)+dE(s,a) . By using c∗, r can be also obtained as
r(s, a) = − log ( 1 c∗(s, a) − 1 ) . (14)
Since the optimization (8) is a convex optimization and the strong duality holds, the optimization (8) is equal to
min ν max w≥0 L(w, ν; r). (15)
The closed-form solution of inner max optimization of (15) turns out to be w∗ν(s, a) = exp (Aν(s,a) 1+α −1 ) for all (s, a), whereAν(s, a) := r(s, a)+γ(T ν)(s, a)−ν(s). By using w∗ν(s, a), optimization (15) can be reduced to a simple convex minimization problem as follows:
min ν L(w∗ν , ν; r) = (1− γ)Es∼p0 [ν(s)] + (1 + α)E(s,a)∼dU
[ exp ( Aν(s, a) 1 + α − 1 )]
(16)
Although L(w∗ν , ν; r) in Equation 16 is convex on ν, we observe that L(w ∗ ν , ν; r) in Equation 16 is numerically unstable in practice. Especially, the exponential term in L(w∗ν , ν; r) is prone to explosion and so is the gradient∇νL(w∗ν , ν; r).

heading: 3.3 ALTERNATIVE CONVEX OPTIMIZATION
text: In order to derive a numerically-stable alternative objective, we describe the following theoretical result: Proposition 1. Define the objective L̃(ν; r) as
L̃(ν; r) := (1− γ)Es∼p0 [ν(s)] + (1 + α)logE(s,a)∼dU [ exp ( Aν(s, a)
1 + α
)] . (17)
Then, the following equality holds:
min ν L(w∗ν , ν; r) = min ν L̃(ν; r).
(The proof can be found in Appendix A.)
The objective L(w∗ν , ν; r) in Equation 16 has the instability issue since the gradient with respect to ν involves an unbounded easily-exploding function exp( · ). In contrast, the alternative objective L̃(ν; r) in Equation 17 does not suffer from the same stability issue because in this case, the gradient forms a soft-max and is bounded by 1. Furthermore, we observe that the surrogate objective is still convex: Proposition 2. The objective L̃(ν; r) is convex with respect to ν. (The proof is in Appendix C.)
To summarize, we have presented a training objective L̃(ν; r) which significantly stabilizes the objective L(w∗ν , ν; r) while having the same optimal value. We converted the minimax optimization (15) to a single minimization, which is much more stable. Proposition 1 is, however, limited in that it is not explain how to sample actions from learned ν̃. Thus, we need a method to sample action from learned ν̃ and the next section presents such a method.

heading: 3.4 POLICY EXTRACTION
text: In this section, we present a method for extracting a policy from the optimal ν̃∗ = arg minν L̃(ν; r). Our method is based on the following optimization, which amounts to the variant of weighted BC:
min π −E(s,a)∼dπ∗ [log π(a|s)] = −E(s,a)∼dU [w∗(s, a) log π(a|s)], (18)
where w∗(s, a) = d π∗ (s,a) dU (s,a)
. It can be estimated by ν∗ ∈ arg minν L(w∗ν , ν; r), but we use L̃(ν; r) in Equation 17 to train ν. Here, from the Proposition 1, we introduce theoretical results to explain the relation between ν∗ ∈ arg minν L(w∗ν , ν; r) and ν̃∗ ∈ arg minν̃ L(ν̃; r): Corollary 1. Let Ṽ = arg minν L̃(ν; r) and V = arg minν L(w∗ν , ν; r) be the sets of ν. Then the following equation holds: (We provide the proof in Appendix B.)
Ṽ = {ν + c|ν ∈ V, c ∈ R} (19)
We observe that for any ν̃∗ ∈ Ṽ , there are ν∗ ∈ V and c ∈ R such that ν̃∗ = ν∗ + c. Based on this relationship, we can easily derive the following proportion: (see Appendix D for more detailed derivation)
w̃ν̃∗(s, a) := exp
( Aν̃∗(s, a)
1 + α
) ∝ d ∗(s, a)
dU (s, a) .
It other words, using ν̃∗ ∈ Ṽ , we can only obtain an unnormalized stationary distribution ratio instead of exact ratio. Therefore, we estimate Equation 18 using self-normalized importance sampling (Owen, 2013) as shown below:
min π Jπ(ν̃
∗) = − E(s,a)∼dU [w̃ν̃∗(s, a) log π(a|s)]
E(s,a)∼dU [w̃ν̃∗(s, a)] . (20)
Based on the weighted BC, we simply extract the policy from ν̃∗ ∈ Ṽ . Our policy extraction is highly related to variants of weighted BC (Wang et al., 2018; 2020; Siegel et al., 2020) in offline RL. The common idea of these methods is (1) training an action-value function Q(s, a) and (2) estimating the corresponding advantage A(s, a). Using an increasing, non-negative function f(·) (e.g., exp(·)), they perform weighted BC on precollected dataset with defining weight f(A(s, a), s, a) as:
arg min π
E(s,a)∼dU [−f(A(s, a), s, a) log π(a|s)].
Finally, if we regard the advantage of optimal ν∗ in Equation 16 as an advantage function, our policy extraction matches the offline RL based on weighted BC.

heading: 4 RELATED WORKS
text: Learning from imperfect demonstrations Among several recent works (Wu et al., 2019; Brown et al., 2019; 2020; Brantley et al., 2020; Tangkaratt et al., 2020; Sasaki & Yamashina, 2021; Wang et al., 2021) that attempted to leverage imperfect demonstrations in IL, two-step importance weighting imitation learning (2IWIL), generative adversarial imitation learning with imperfect demonstration and confidence (IC-GAIL) (Wu et al., 2019), weighted GAIL (WGAIL) (Wang et al., 2021), and Behavioral Cloning from Noisy Demonstrations (BCND) (Sasaki & Yamashina, 2021) are closely related to our work.
2IWIL, IC-GAIL and WGAIL suppose imperfect demonstrations which are composed of expert and non-expert trajectories and this definition is the same as that of DemoDICE. Assuming that some of trajectories in the imperfect demonstrations are provided with labels of confidence of optimalities, 2IWIL trains a semi-supervised classifier to predict confidence scores for unlabeled imperfect demonstrations and train the policy using the score. Instead of such two-step learning, IC-GAIL trains the policy in an end-to-end fashion.
Inspired from Wu et al. (2019), Wang et al. (2021) suggest a general weighted GAIL objective function. Unlike previous work, WGAIL predicts the weight for imperfect demonstrations without information about their quality. However, 2IWIL, IC-GAIL, and WGAIL are online on-policy imitation learning method, whereas our algorithm, DemoDICE, is an offline imitation learning method.
BCND assumes that the expert demonstrations are sampled from a noisy expert policy. BCND learns ensemble policies with a weighted BC objective, where the weight is policy learned by weighted BC in the previous internal iteration. As the learning progresses, the mode of the learned policy moves towards that of the noisy expert policy. Thus, the learned policy tends to converge to a non-expert policy when the non-expert trajectories comprise the majority of demonstrations.
Stationary distribution corrections In RL and IL, some prior works have used distribution corrections. In AlgaeDICE (Nachum et al., 2019b), regularization in the space of stationary distribution is augmented to a policy optimization objective to solve off-policy RL problems. In addition, by using dual formulation of f-divergence and change of variables (Nachum et al., 2019a), an off-policy learning objective is derived in AlgaeDICE. ValueDICE (Kostrikov et al., 2020) minimizes the KL divergence between the agent and the expert stationary distributions to solve IL problems.
Similar to the AlgaeDICE, ValueDICE obtains an off-policy learning objective for distribution matching by using the dual formulation of KL divergence and change of variables. Both AlgaeDICE and ValueDICE objectives are optimized by nested optimization, which may suffer from numerical instability.
In contrast, OptiDICE (Lee et al., 2021) reduces the same optimization used in AlgaeDICE to unconstrained convex optimization. Although it shows promising performance in offline RL tasks, directly
applying it to offline IL from expert and imperfect demonstrations is not trivial as we discussed in Section 3.
Off policy learning from observations (OPOLO) (Zhu et al., 2020) addresses off-policy IL using expert demonstrations composed only of states. It has similarities to DemoDICE in two aspects. Firstly, OPOLO adopts a discriminator to estimate a state-transition distribution ratio. Furthermore, DICE-like technique is applied to the derivation of the objective of OPOLO, which is similar to ours. Thus, based on the relation between DemoDICE and OPOLO, offline IL using expert demonstrations containing only states can be considered as a promising direction for future work.

heading: 5 EXPERIMENTS
text: In this section, we present the empirical performance of DemoDICE and baseline methods on MuJoCo continuous control environments (Todorov et al., 2012) using the OpenAI Gym (Brockman et al., 2016) framework. We provide experimental results for 4 MuJoCo environments: Hopper, Walker2d, HalfCheetah, and Ant. We utilize D4RL datasets (Fu et al., 2020) to construct expert and imperfect demonstrations for our experiments. We construct datasets, baselines, and evaluation protocol according to the following procedures:
Datasets For each of MuJoCo environments, we utilize three types of D4RL datasets (Fu et al., 2020), whose name end with “-expert-v2”, “-full replay-v2”, or “-random-v2”. In the remainder of this section, we refer them by using corresponding suffixes. We regard the trajectories in expert-v2 as expert trajectories. The set of expert demonstrations DE consists of the first trajectory in expert-v2.
Baselines We compare our method with three strong baseline methods, BC, BCND (Sasaki & Yamashina, 2021), and ValueDICE (Kostrikov et al., 2020). To consider the potential benefit of utilizing DI , we carefully tuned BC with 5 different values of β ∈ {1, 0.75, 0.5, 0.25, 0}, which controls the balance between minimizing negative log-likelihood of DE and minimizing that of DU = DE ∪DI as follows:
min π JBC(β)(π) := −β ·
1 |DE | ∑
(s,a)∈DE log π(a|s)− (1− β) · 1
|DU | ∑
(s,a)∈DU log π(a|s).
We denote those 5 different settings by BC(β=1), BC(β=0.75), BC(β=0.5), BC(β=0.25), BC(β=0). For BCND, we use all demonstrations as noisy expert demonstrations and report statistics of the best performance among 5 internal iterations over 5 seeds. We provide the details for BCND in Appenedix E. Lastly, we made ValueDICE to utilize imperfect datasets by plugging in all the demonstrations to replay buffer.
Evaluation metric The normalized scores for each environment are measured by normalized score = 100 × score−random scoreexpert score−random score , where expert and random scores are average returns of trajectories in expert-v2 and random-v2, respectively. We compute the average normalized score and the standard error over five random seeds. In the following subsections, we provide experimental results on two types of imperfect datasets.

heading: 5.1 MIXED DATASET
text: DemoDICE mainly aims to overcome distributional drift caused by the lack of expert demonstrations. When DemoDICE successfully learns the optimal unnormalized stationary distribution ratio, we can expect that DemoDICE distinguishes expert trajectories from non-expert ones to update its own policy. Based on this intuition, we hypothesize that if the same sufficiently-many expert trajectories are included in imperfect demonstrations, DemoDICE achieves the optimal performance invariant to the number of sub-expert trajectories in imperfect demonstrations. To see if it is the case, we use mixed datasets which have the same expert trajectories but have different numbers of non-expert trajectories in imperfect demonstrations.
Experimental setup Across all environments, we consider 3 tasks, each of which is called one of M1, M2, or M3 and is provided with expert and imperfect demonstrations. While sharing the
expert demonstrations composed of the same single expert trajectory, imperfect demonstrations in each task are composed of expert and random trajectories with different ratios such as 1:0.25 (M1), 1:1 (M2), and 1:4 (M3). Specifically, imperfect demonstrations in M1, M2, and M3 are composed of 400 expert trajectories sampled from expert-v2 and 100, 400, and 1600 random trajectories sampled from random-v2, respectively.
Result We report the result of DemoDICE and comparing methods in Figure 1. For simplicity, BC(β = 0.25) and BC(β = 0.75) are in Appendix F. Note that imperfect demonstrations in the tasks M1, M2, and M3 share the same expert trajectories, while having the different number of random trajectories. Ideally, if an algorithm uses only expert trajectories from imperfect demonstrations, the algorithm should have the same performance in the tasks M1, M2, and M3, as discussed at the beginning of this subsection, except convergence speed. While ValueDICE, BCND, and all the variants of BC catastrophically fail to recover expert policies in task M3 on all the environments except Walker2d, DemoDICE reaches the expert performance regardless of environments and tasks.
The result strongly implies that DemoDICE succeeds to learn appropriate stationary distribution correction, which is possible only when the algorithm effectively leverages expert trajectories in imperfect demonstrations ignoring the rest. Furthermore, it is remarkable that DemoDICE reaches the expert performance in HalfCheetah in the end, which can be found in Appendix F.1. To summarize, we have empirically shown that DemoDICE is robust to any choices of the environments and tasks and significantly outperforms existing methods on offline IL.

heading: 5.2 REPLAY BUFFER DATASET
text: When we use mixed datasets as imperfect demonstration sets, ValueDICE poorly performs in all environments except Walker2d as we observed in Section 5.1. Since full replay-v2 consists of the replay buffer’s data during off-policy training, we believe it is one of the most practically-
relevant datasets to fill ValueDICE’s replay buffer. Therefore, we conduct additional experiments using this dataset as the imperfect demonstrations.
Experimental setup Applying the same expert demonstrations used in Section 5.1, we employ another task named RB, which uses full replay-v2 directly as its imperfect dataset.
Result Figure 2 summarizes the result of DemoDICE and baseline methods. We observe that DemoDICE performs on par with ValueDICE in Walker2d. However, on the other environments, DemoDICE strictly outperforms ValueDICE. We also observe that DemoDICE performs on par with BCND in Ant, HalfCheetah environments, and significantly outperforms in Hopper and Walker2d environments. While showing the performance competitive to the best performing baseline methods in Walker2d, Ant, and HalfCheetah, DemoDICE outperforms all the methods in Hopper by a significant margin.
Since the replay buffer dataset is collected by the policy which evolves (in policy training phase), it can be regarded as a set of imperfect demonstration from a wide range of sources. We emphasize that DemoDICE performs well not only when bad trajectories make up majority of the given imperfect demonstrations but also when they are generated from multiple sources.

heading: 6 CONCLUSION
text: We have presented DemoDICE, an algorithm for offline IL from expert and imperfect demonstrations that achieves state-of-the-art performance on various offline IL tasks. We first introduced a regularized offline IL objective and reformulated the objective so as to make it natural to compute closed-form solution. We then tackled the instability coming from the naive application of closedform solution by the alternative objective which yields not only the same optimal value but also a stable convex optimization. Furthermore, we presented the method to extract an optimal policy with simple weighted BC. Lastly, our extensive empirical evaluations showed that DemoDICE achieves remarkable performance close to the optimal by exploiting imperfect demonstrations effectively.

heading: A PROOF OF PROPOSITION 1
text: We provide the detailed proofs based on the previous works (Nachum & Dai, 2020; Lee et al., 2021). Therefore, we highly recommend to read the works for a better understanding. We first provide a lemma to prove the proposition:
Lemma 1 (Fenchel duality). Consider a primal problem given by
min x∈Ω JP (x) := g(x) + h(Ax),
where g, h : Ω → R are convex, lower-semi-continuous and A is a linear operator. Then, the dual of this problem is given by
max y∈Ω∗
JD(y) := −g∗(−A∗y)− h∗(y),
where A∗ is a (Hermitian) adjoint operator of A, i.e., 〈y,Ax〉 = 〈A∗y, x〉 for all x, y.
Proof. We borrow the proof steps from Nachum & Dai (2020). Remark that the definition of conjugate function of a function f :
f∗(y) := max x 〈x, y〉 − f(x),
where 〈·, ·〉 is inner product. Based on the definition, we can easily derive the following equations:
min x g(x) + h(Ax) = min x max y
g(x) + 〈y,Ax〉 − h∗(y)
= max y { min x g(x) + 〈y,Ax〉 } − h∗(y)
= max y
{ −max
x 〈−A∗y, x〉 − g(x)
} − h∗(y)
= max y −g∗(−A∗y)− h∗(y).
Now, we prove the proposition:
Proposition 1. Define the objective L̃(ν; r) as L̃(ν; r) := (1− γ)Es∼p0 [ν(s)] + (1 + α)logE(s,a)∼dU [ exp ( Aν(s, a)
1 + α
)] . (17)
Then, the following equality holds:
min ν L(w∗ν , ν; r) = min ν L̃(ν; r).
(The proof can be found in Appendix A.)
Proof. We first define two functions:
g(·) := δ{(1−γ)p0}(·), h(·) := 〈−r, ·〉+ (1 + α)DKL(·‖dU ),
where
r(s, a) := log dE(s, a)
dU (s, a) ,
δC(x) := { 0 x ∈ C, ∞ otherwise.
Then, the corresponding conjugate functions are given by
g∗(·) := (1− γ)Ep0 [·], h∗(·) := (1 + α) logEdU [ exp ( ·+ r 1 + α )] .
Now, we rewrite the optimization problem (5-7) as follows:
max d −δ(1−γ)Ep0
( − (γT − B)∗d ) −DKL(d‖dE)− αDKL(d‖dU )
= −δ(1−γ)Ep0 ( − (γT − B)∗d ) − E(s,a)∼d[−r(s, a)]− (1 + α)DKL(d‖dU )
= −g ( − (γT − B)∗d ) − h(d).
By Lemma 1, we reformulate the optimization problem as: max d −g ( − (γT − B)∗d ) − h(d)
≡ min ν g∗(ν) + h∗((γT − B)ν) = (1− γ)Ep0 [ν] + (1 + α) logEdU [ exp ( (γT − B)ν + r
1 + α )] = (1− γ)Es∼p0 [ν(s)] + (1 + α) logE(s,a)∼dU [ exp ( γ(T ν)(s, a)− (Bν)(s, a) + r(s, a)
1 + α )] = (1− γ)Es∼p0 [ν(s)] + (1 + α) logE(s,a)∼dU [ exp ( r(s, a) + γ(T ν)(s, a)− ν(s)
1 + α )] = (1− γ)Es∼p0 [ν(s)] + (1 + α) logE(s,a)∼dU [ exp ( Aν(s, a)
1 + α )] It means that minν L(w∗ν , ν; r) and minν L̃(ν; r) are the same optimization problems.
We obtain the conjugate function g∗ using the fact that 〈a, y〉 is the conjugate of δa(x). We also compute h∗ using the two facts: b · f∗(y−ab ) is the conjugate of 〈a, x〉+ b · f(x) and logEp[exp(y)] is the conjugate of DKL(x‖p).

heading: B PROOF OF COROLLARY 1
text: We remind the objectives in Equation 16 and Equation 17: L(w∗ν , ν; r) := (1− γ)Es∼p0 [ν(s)] + (1 + α)E(s,a)∼dU [ exp ( Aν(s, a) 1 + α − 1 )] ,
L̃(ν; r) := (1− γ)Es∼p0 [ν(s)] + (1 + α) logE(s,a)∼dU [ exp ( Aν(s, a)
1 + α
)] ,
where
Aν(s, a) := r(s, a) + γ(T ν)(s, a)− ν(s),
w∗ν(s, a) := exp
( Aν(s, a) 1 + α − 1 ) .
Note that for any constant C, the advantage for ν + C is:
A(ν+C)(s, a) = r(s, a) + γ(T (ν + C))(s, a)− (ν(s) + C) = r(s, a) + γ ( (T ν)(s, a) + C ) − (ν(s) + C)
= r(s, a) + γ(T ν)(s, a)− ν(s)− (1− γ)C = Aν(s, a)− (1− γ)C
Lemma 2. For arbitrary function ν and constant C, the equality
L̃(ν; r) = L̃(ν + C; r)
holds.
Proof. From the definition of L̃(ν; r), we derive the following equation:
L̃(ν + C; r) = (1− γ)Es∼p0 [ν(s) + C] + (1 + α) logE(s,a)∼dU [ exp ( A(ν+C)(s, a)
1 + α )] = (1− γ)Es∼p0 [ν(s) + C] + (1 + α) logE(s,a)∼dU [ exp ( Aν(s, a)− (1− γ)C
1 + α )] = (1− γ)Es∼p0 [ν(s) + C] + (1 + α) log [ exp ( − (1− γ)C
1 + α
) E(s,a)∼dU [ exp ( Aν(s, a)
1 + α )]] = (1− γ)Es∼p0 [ν(s) + C] + (1 + α) [ logE(s,a)∼dU [ exp ( Aν(s, a)
1 + α
)] − (1− γ)C
1 + α ] = (1− γ)Es∼p0 [ν(s)] + (1 + α) logE(s,a)∼dU [ exp ( Aν(s, a)
1 + α )] = L̃(ν; r)
Lemma 3. For any function ν, the following inequality always holds:
L(w∗ν , ν; r) ≥ L̃(ν; r).
Equality holds if and only if E(s,a)∼dU [ exp ( Aν(s, a) 1 + α − 1 )] = 1.
Proof. For any y ≥ 0, y − 1 ≥ log y,
and equality holds if and only if y = 1. From this inequality, we can derive the inequality directly: E(s,a)∼dU [ exp ( Aν(s, a) 1 + α − 1 )] − 1 ≥ logE(s,a)∼dU [ exp ( Aν(s, a) 1 + α − 1 )] ,
and equality holds if and only if E(s,a)∼dU [ exp ( Aν(s, a) 1 + α − 1 )] = 1.
Finally, we obtain the following results: L(w∗ν , ν; r) = (1− γ)Es∼p0 [ν(s)] + (1 + α)E(s,a)∼dU [ exp ( Aν(s, a) 1 + α − 1 )]
≥ (1− γ)Es∼p0 [ν(s)] + (1 + α) { logE(s,a)∼dU [ exp ( Aν(s, a) 1 + α − 1 )] + 1 } ≥ (1− γ)Es∼p0 [ν(s)] + (1 + α) log { E(s,a)∼dU [ exp ( Aν(s, a) 1 + α − 1 )] exp(1)
} = (1− γ)Es∼p0 [ν(s)] + (1 + α) logE(s,a)∼dU [ exp ( Aν(s, a)
1 + α )] = L̃(ν; r).
The equality holds if and only if E(s,a)∼dU [ exp ( Aν(s, a) 1 + α − 1 )] = 1
Lemma 4. For any optimal solution ν̃∗ = arg minν L̃(ν; r), there is a constant C such that ν̃∗+C is an optimal solution of minν L(w∗ν , ν; r).
Proof. Let ν̃∗ be an optimal solution of arg minν L̃(ν; r) and
C∗ := 1 + α
1− γ logE(s,a)∼dU
[ exp ( Aν̃∗(s, a) 1 + α − 1 )]
Then, ν̂ := ν̃∗ + C∗ satisfies E(s,a)∼dU [ exp ( Aν̂(s, a) 1 + α − 1 )]
= E(s,a)∼dU [ exp ( A(ν̃∗+C∗)(s, a) 1 + α − 1 )]
= E(s,a)∼dU [ exp ( Aν̃∗(s, a)− (1− γ)C∗ 1 + α − 1 )]
= E(s,a)∼dU [ exp ( Aν̃∗(s, a)
1 + α − 1− γ 1 + α C∗ − 1 )] = E(s,a)∼dU [ exp ( Aν̃∗(s, a) 1 + α − 1 )] exp ( − 1− γ 1 + α C∗ )
= E(s,a)∼dU [ exp ( Aν̃∗(s, a) 1 + α − 1 )]{ E(s,a)∼dU [ exp ( Aν̃∗(s, a) 1 + α − 1 )]}−1
= 1.
Furthermore, ν̂ is also an optimal solution of minν L̃(ν; r) by Lemma 2. Then, by equality condition in Lemma 3,
L(w∗ν̂ , ν̂; r) = L̃(ν̂; r) = min ν L̃(ν; r) ≤ min ν L(w∗ν , ν; r).
It means, ν̂ is an optimal solution of minν L(w∗ν , ν; r)
Lemma 5. An optimal solution ν∗ = arg minν L(w∗ν , ν; r) is also an optimal solution of minν L̃(ν; r)
Proof. From the Lemma 3,
min ν L(w∗ν , ν; r) = L(w ∗ ν∗ , ν ∗; r) ≥ L̃(ν∗; r).
Lemma 4 show that minν L(w∗ν , ν; r) and minν L̃(ν; r) have the same minimum value,
L̃(ν∗; r) ≤ L(w∗ν∗ , ν∗; r) = min ν L(w∗ν , ν; r) = min ν L̃(ν; r)
holds. It means that L̃(ν∗; r) is the minimum value of L̃ and therefore, ν∗ is an optimal solution of minν L̃(ν; r).
Finally, we ready to prove the corollary:
Corollary 1. Let Ṽ = arg minν L̃(ν; r) and V = arg minν L(w∗ν , ν; r) be the sets of ν. Then the following equation holds: (We provide the proof in Appendix B.)
Ṽ = {ν + c|ν ∈ V, c ∈ R} (19)
Proof. Lemma 4 and Lemma 5 shows that the last equation holds.

heading: C PROOF OF PROPOSITION 2
text: Proposition 2. The objective L̃(ν; r) is convex with respect to ν. (The proof is in Appendix C.)
Proof. For any functions ν, ν′ and constant λ ∈ [0, 1], we can derive the following equality:
A(λν+(1−λ)ν′)(s, a)
= r(s, a) + γ(T (λν + (1− λ)ν′))(s, a)− (λν(s) + (1− λ)ν′(s)) = r(s, a) + γλ(T ν)(s, a) + γ(1− λ)(T ν′)(s, a)− (λν(s) + (1− λ)ν′(s)) = λ(r(s, a) + γ(T ν)(s, a)− λν(s)) + (1− λ)(r(s, a) + γ(T ν′)(s, a)− λν′(s)) = λAν(s, a) + (1− λ)Aν′(s, a).
It means, Aν is the linear function with respect to ν. Furthermore, using the convexity of log-sumexp,
logE(s,a)∼dU [ exp(A(λν+(1−λ)ν′)(s, a)) ] = logE(s,a)∼dU [exp(λAν(s, a) + (1− λ)Aν′(s, a))] ≤ λ logE(s,a)∼dU [exp(Aν(s, a))] + (1− λ) logE(s,a)∼dU [Aν′(s, a))]
Therefore, L̃(ν; r) is convex function with respect to ν.

heading: D POLICY EXTRACTION
text: By the definition of Aν ,
exp
( A(ν̃∗+C)(s, a) 1 + α − 1 )
= exp
( Aν̃∗(s, a)− (1− γ)C 1 + α − 1 )
= exp
( Aν̃∗(s, a)
1 + α
) exp ( − 1− γ
1 + α C − 1
) .
Therefore,
exp
( Aν̃∗(s, a)
1 + α
) ∝ exp ( A(ν̃∗+C)(s, a) 1 + α − 1 ) = d∗(s, a) dU (s, a) .

heading: E EXPERIMENTAL DETAILS
text: Implementation detail For fair comparison, we use the same learning rate to train actors of BC and DemoDICE. In DemoDICE, we applied the same regularization coefficient α = 0.05 in Equation 17 across all the tasks and dataset configurations. We implement our network architectures for BC and DemoDICE based on the implementation of OptiDICE1. For ValueDICE, we use its official implementation2 without any modification to network architectures or hyperparameters. Based on DAC (Kostrikov et al., 2019), we treat terminal states as absorbing states in ValueDICE and DemoDICE.
For stable discriminator learning, we use gradient penalty regularization on the r(s, a) function, which was proposed in Gulrajani et al. (2017) to enforce 1-Lipschitz constraint. To stabilize critic training, we add gradient L2-norm to the critic loss for the regularization. Detailed hyperparameter configurations used for our main experiments are summarized in Table 1.
E.1 EXPERIMENTAL DETAILS FOR BCND
We implement the BCND based on the BCND paper (Sasaki & Yamashina, 2021). We shuffle all demonstrations and divide them into K disjoint sets {Dk}Kk=1. Then, for each iteration, we perform stochastic gradient ascent to update each policy πθk using DK . Finally, we update reward R̂(s, a) and η using
R̂(s, a)← 1 K K∑ k=1 πθk(a|s), ∀(s, a) ∈ D,
η ← K Es∼D[ ∑ k πθk(µθk(s)|s)] .
Here, we use following hyperparameters as reported in paper (Sasaki & Yamashina, 2021): We use M = 5 (number of iteration), K = 5 (number of policy / disjoint dataset), N = 128 (batch size), L = 500 × |Dk|/N for each Dk (k ∈ {1, . . . ,K}, in our tasks, L ∈ [3 × 105, 1.1 × 106]), πθ(a|s) = 1K ∑K k=1 πθk(a|s), ζ = 10−4 (shared learning rate), Adam optimizer with learning rate ζ × η for gradient ascent, and Gaussian policy πθk = N (µθk(s), σ2θk). For Guassian policy, we use neural network with size [100, 100] for µθk , and a trainable independent vector for σθk , as reported in the paper.
1 https://github.com/secury/optidice 2 https://github.com/google-research/google-research/tree/master/value_dice
We measure the performance of algorithm for each internal iteration. In Figure 1 and Figure 2, we report the best performance and corresponding standard error of the algorithm among theM internal iterations (we compute the average performance for each iteration and pick the maximum average performance as the best performance).
Algorithm 1 Behavioral Cloning from Noisy Demonstrations Require: Noisy expert demonstrations D, policy parameters {θk}Kk=1, learning rate ζ Ensure: Ensemble policy πθ = 1K ∑K k=1 πθk
Set R̂(s, a) = 1 for ∀(s, a) ∈ D. Split D into K disjoint sets {D1, D2, . . . , DK}. for iteration= 1, . . . ,M do
for k = 1, . . . ,K do Initialize parameters θk. for l = 1, . . . , L do
Sample a minibatch Bk = {(sn, an)}Nn=1 from Dk. Calculate a sampled gradient 1
N ∑N n=1∇θk log πθk (sn, an) · R̂(sn, an).
Update θk by gradient ascent with learning rate ζ × η end for
end for Update R̂(s, a) and η
end for

heading: F ADDITIONAL EXPERIMENTAL RESULTS
text: In this section, we report additional experimental results for detailed empirical analysis of our approach. We focus three experiments as follows: (1) asymptotic performance of our approach on HalfCheetah environment for the mixed dataset we considered in Section 5.1, (2) behavior cloning with more diverse β selections which are omitted in Figure 1, (3) ablation study on hyperparameter α for DemoDICE, which is appeared in Equation 17.
F.1 ASYMPTOTIC PERFORMANCE OF DEMODICE ON HALFCHEETAH
To inspect asymptotic behaviors of DemoDICE on our mixed dataset tasks in HalfCheetah environment, we extend the number of training iterations into 2 million with fixing the other hyperparameter settings. As Figure 3 shows, DemoDICE converges to the expert performance for all tasks we considered in Section 5.1.
F.2 BEHAVIOR CLONING PERFORMANCE
In Figure 1, we omitted performance of BC with β = 0.25 and β = 0.75 for simplicity. Figure 4 shows performance of entire BC configurations, i.e., β ∈ {0, 0.25, 0.5, 0.75, 1.0}.
F.3 DIFFERENT QUALITY OF IMPERFECT DEMONSTRATIONS
In this section, we construct new tasks where random trajectories in mixed dataset (see 5.1) are replaced by medium trajectories, named I1, I2, and I3. Here, all tasks share the expert demonstrations composed of the single expert trajectory. Imperfect demonstrations in each task are composed of 400 expert trajectories and medium trajectories, and the number of medium trajectories is 100 for the task I1, 400 for I2, and 1600 for I3. In Figure F.2, we observed that DemoDICE performs well in these new tasks.
F.4 DIFFERENT COVERAGE OF EXPERT DEMONSTRATIONS
In this section, we introduce additional experiments with different coverages of expert demonstrations. Concretely, we constructed expert demonstrations using an increasing number of expert trajectories, namely 1, 2, 5, up to 10. Because DemoDICE already performs well in M1 and M2 tasks, we compare M3 and RB tasks for this ablation study. In this experiments, DemoDICE is not significantly better than other baselines when the expert demonstrations cover enough space. We remind the readers that when the coverage is not high, DemoDICE significantly outperforms the others as shown in Figure 1 and 2 in the main text.
F.5 ABLATION TEST ON HYPERPARAMETER α
As we can notice in Equation 17, DemoDICE introduces a hyperparameter α, which is a regularization coefficient. In this section, we aim to study how does the choice of α effect to training processes and performance of our algorithm. We select α ∈ {10, 1, 0.5, 0.1, 0.05, 0} and train on M3 datasets and replay buffer datasets by preserving the other settings same as before. Figure 7 describes the results of ablation study. We point out that DemoDICE shows almost the same performance in when α ∈ [0, 0.1].
F.6 ABLATION TEST ON PURE OFFLINE IMITATION LEARNING
In this section, we compare DemoDICE and BC using only expert demonstrations. In Figure F.5, we observe that DemoDICE and BC show similar performance. In pure offline IL, DemoDICE essentially tries to find a stationary distribution that minimizes DKL(d||dE) on the MLE MDP constructed by DE . Let d̂E be the empirical stationary distribution of dE . Then, the optimal solution is trivially given by d∗ = d̂E with DKL(d||d̂E) = 0 since d̂E itself is a valid stationary distribution on the MLE MDP. Finally, extracting a policy from dE reduces to a simple behavior cloning: minπ −E(s,a)∼dE [log π(a|s)], which concludes that DemoDICE would not be better than BC in the pure offline IL setting.

heading: G ANALYSIS EMPIRICAL RESULTS
text: In this section, we provide additional discussions about empirical results.
First, M1 is an easy task where most of the trajectories are from the expert, so some baselines also succeed to achieve good performance.
To discuss about RB tasks, we show the normalized score statistics of trajectories in the replay buffer in the Table F.5. For each environment, we counted the number of trajectories that have a normalized score greater than R as > R, so the numbers above are cumulative counts.Thus, this table suggests natural upper bound of performances of imitation learning algorithms, e.g. it would be very hard to score around 100 for HalfCheetah task.
In the Ant-RB task, we have enough good trajectories, especially > 100, so both DemoDICE and BC(β = 0) recover expert policy.
In the HalfCheetah-RB task, there is no trajectory with > 80, so DemoDICE eagerly distinguishes the non-expert trajectories from expert ones, throwing away non-expert demonstrations. Thus it overall behaves like BC(β = 0).
In the Walker2d-RB task, both ValueDICE and DemoDICE show similar performances at around 80 at the end. Looking at the statistics above, since the best trajectory in RB has its normalized score less than 90, we believe that ValueDICE and DemoDICE both achieved reasonable performance levels.

Review summaries:
The paper considers an offline imitation learning (IL) problem with an addition of supplementary imperfect demonstrations. To solve this problem, the paper proposes DemoDICE which regularizes a distribution-matching objective of IL by a KL divergence between the agent distribution and a mixed of expert and imperfect distributions. DemoDICE finds an optimal state-action distribution of this regularized objective by using a dual-program technique similar to that of OptiDICE (Lee et al., 2021) for offline RL with an improvement in terms of stability. Given the optimal state-action distribution, DemoDICE extracts the expert policy by performing weighted behavioral cloning.  Empirical evaluation on Mujoco tasks with D4RL datasets show that DemoDICE can efficiently and effectively solve the offline IL problem.   ### Contributions - A new learning problem combining offline IL and IL with imperfect demonstrations.  - A new model-free offline IL method based on dual-program optimization. 

This work presents an approach that learns a policy from expert demonstrations data in an offline setting. It is trying to solve an important problem in imitation learning: the expert demonstration may not cover the entire state/action space pretty well, and many existing algorithms require further interaction with the environment. To solve the problem of distribution shift when expert data is not diverse enough, the method proposes to introduce a large number of supplementary demonstrations of various qualities. In this way, the proposed algorithm DemoDICE does not require any on-policy samples. The method adapts the OptiDICE method for its own purpose by using a transformed objective function. The theoretical analysis and empirical experiments demonstrate the improvement of the method over BC and ValueDICE.

The paper considers the problem of imitation learning, and specifically the setting where a small number of expert demonstrations is paired with some amount of suboptimal data. To tackle this setting, the paper proposes to optimize a convex constrained optimization, with linear variables denoting d^pi, linear constraints expressing the MDP transitions, and an objective composed of a linear combination of KL from the expert and KL from the suboptimal data. The paper presents derivations transforming this convex constrained optimization to a practical, unconstrained objective. The final algorithm is composed of three stages: (1) train a discriminator to get a density ratio estimator; (2) use the density ratios as a reward in a separate log-expected-exp objective (similar to REPS or ValueDICE objectives) to learn a different density ratio w; (3) extract a policy from w using weighted max likelihood training on the suboptimal data. The algorithm is evaluated on a variety of simulated robotics (mujoco) environments, and shows favorable results compared to baselines.