A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories
Authors: Kai Yan
Alex Schwing
Yu-Xiong Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: offline Imitation learning
learning from observations
positive-unlabeled learning
TL;DR: We develop a simple algorithm that effectively solves offline imitation learning from observations and example problem with possibly incomplete trajectories.

Abstract: Offline imitation from observations aims to solve MDPs where only task-specific expert states and task-agnostic non-expert state-action pairs are available. Offline imitation is useful in real-world scenarios where arbitrary interactions are costly and expert actions are unavailable. The state-of-the-art ‘DIstribution Correction Estimation’ (DICE) methods minimize divergence of state occupancy between expert and learner policies and retrieve a policy with weighted behavior cloning; however, their results are unstable when learning from incomplete trajectories, due to a non-robust optimization in the dual domain. To address the issue, in this paper, we propose Trajectory-Aware Imitation Learning from Observations (TAILO). TAILO uses a discounted sum along the future trajectory as the weight for weighted behavior cloning. The terms for the sum are scaled by the output of a discriminator, which aims to identify expert states. Despite simplicity, TAILO works well if there exist trajectories or segments of expert behavior in the task-agnostic data, a common assumption in prior work. In experiments across multiple testbeds, we find TAILO to be more robust and effective, particularly with incomplete trajectories.

A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories
Kai Yan
Alexander G. Schwing Yu-Xiong Wang University of Illinois Uramaa-Champpign (kaiyan3, aschwing, yxw)@illinois.ede edu https://github. com./o/////////2///TTT
Abstract
Offline imitation from observations aims solve MDPs where only task-specific expert states and task-agnostic non-expert state-action pairs are available. Offline imitation is useful in real- real-world scenarios where arbitrary interactions are costly and expert actions are unavailable. The state-of-the-art 'DIstribution Correction Estimation' (DICE) methods minimize divergence of state occupancy between expert and learner policies and retrieve policy with weighted behavior cloning; however, their results are unstable when learning from incomplete trajectories, due to a non-robust optimization in the dual domain. To address the issue, in this paper, Imitation Learning from Observations (TAIL.)). TAILO uses discounted sum along the future trajectory as the weight for weighted behavior cloning. The terms for the sum are scaled by the output of discriminator, which aims identify expert states. Despite simplicity, TAILO works well if there exist trajectories or segments of expert behavior the task-agnostic data, a common assumption in prior work. In experiments across multiple testbeds, find TAILO to be more robust and effective, particularly with incomplete trajectories.
we propose Trajectory-Awan
Introduction
In recent years, Reinforcement Learning (RL) has been remarkably successful on variety of tasks, from games [57] and robot manipulation [66] to recommendation systems [9] and large language model fine-tuning [53]. However, RL often also suffers from the need for extensive interaction with the environment and missing compelling rewards in real-life applications [37].
To address this, Imitation Learning (IL), where an agent learns from demonstrations, is gaining popularity recently [20, 22, 27]. Offline imitation learning, such as behavior cloning (BC) [51], allows the agent to learn from existing experience without environment interaction and reward label, which is useful when wrong actions are costly. However, similar to RI., IL also suffers when limited data is available [52]- which is common demonstrations of the target task need to be collected every time a new task addressed. this work, we consider a special but widely studied [31, 36, 60] case of expert data shortage in offline II, i.e., offline Learning from Observations (LfO) [71]. In LfO the task task-specific data only consists of few expert trajectories, key frames, or even just the goal (the latter is also known example- based IL [13]), and the dynamics must be mostly learned from task- task-agnostic data, i.c., demonstration from data not necessarily directly related to the target task. For example, sometimes the agent must learn from experts with different embodiment [41], where expert actions are not applicable.
In the field of offline LfO, researchers have explored action pseudo-labeling [33, 60], inverse RL [31, 61, 73], and similarity- reward labeling [8, 55]; for example-based IL, the benchmark
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Tws Twa-agrod cars
Dipen Straee
Kanga Giaadlatton
Tepparayy feom Thoweteiing
(a) Problem Settings
(b) Discriminator R(s) (c) Weighted Behavior Cloning
Figure 1: An illustration of our method. TAILO. Different trajectories are illustrated by different styles of arrows. TAILO consists of two parametric steps: 1) train discriminator which gives high R(s) for near-expert states and low R(s) for non-expert states, as shown in panel b); conduct weighted behavior cloning with weights calculated from R(s) along the trajectory, as shown in panel c). High transparency indicates a small weight for the state and its corresponding action.
RCE [CE[13), which uses RL with classifier-lbbeled reward. Recently, DIstribution Correction Estimation (DICE) methods, LobsDICE [23] and SMODICE [41], achieve the state of the art for both offline LfO and example-based IL. Both methods minimize the state visitation frequency (occupancy) divergence between expert and learner policies, and conduct a convex optimization in the dual space.
However, DICE methods are neither robust to incomplete trajectories in the task task-agnostic/ / task- specific data [72], nor do they excel if the task-agnostic data contains a very small portion of expert trajectories or only segments [56]. These are inherent shortcomings of the DICE-based methods, as they are indirect: they first perform optimization in a dual domain (equivalent finding the value function in RL), and then recover the policy by weighted behavior cloning. Importantly, Kullback- Leibler(KL.)-aate optimization of dual variables requires complete trajectories in the task-agnostic data to balance all terms in the objective. Note, SMODICE with x2-based optimization is also theoretically problematic (see Appendix C) and empirically struggles on testbeds [41].
To overcome the shortcomings listed above, we propose a simple but effective method for imitation learning from observations and examples, Trajectory-Awart Imitation Learning from Observations (TAILO). We leverage the common assumption that there exist trajectories or long segments that are near-optimal the task of interest in the task aask-agnoticc data. This assumption is the basis of skill- based learning [20, 46, 47], and the benchmarks of many recent works satiesfy this assumption [36, 40, 41, 56]; real-life example fulfilling this assumption is robotics: the robot often utilizes overlapping skills such as moving the robotic arm and grabbing items from other tasks to complete the current task. Based on this assumption, we discriminate/identffyy which state-action pairs could be taken by the expert, and assign large weights for trajectory segments leading to those segments in the downstream Weighted Behavior Cloning (WBC). This is simple way to make the learned policy trajectory-aware The method only consists of two parametric steps: 1) train discriminator using positive-unlabes unlabeled (PU) learning, and 2) use Weighted Behavior Cloning (WBC) on all state-action pairs in the task- aggossicc data with the weights of WBC being discounted sum over thresholded scores given by the discriminator. Note, the discounted sum propagates large scores to trajectory segments in the past far from expert states, if they lead to expert trajectory segments eventually, Meanwhile, as the task-agnostic data contains both expert and non-expert demonstrations, Positive- Unlabeled (PU) learning is better than plain binary classification. Fig. summarizes our algorithm. We found this simple solution to be surprisingly effective across multiple testbeds, especially if the task-agnostic data contains incomplete trajectories. In this latter case, baselines struggle or diverge. Moreover, we find our method to also improve if the task-agnostic data contains only a small portion of expert trajectories.
We summarize our contributions as follows: 1) We carefully analyzed the sfate-of-the- DICE methods in offline LfO, pointing out their limitations both empirically and theoretically (see Ap- pendix C for details); 2) We propose simple yet effective solution to offline imitation learning from observations; and 3) We empirically show that this simple method is robust and works better than the state the art on a variety of settings, including incomplete trajectories, few expert trajectories the task-agnostic dataset, example-base based IL and learning from mismatching dynamics.

---

2 Preliminaries
Markov Decision Process. A Markov Decision Process (MDP) a well-established framework for sequential decision-making problems. An MDP is defined by the tuple (S, A, 7)), where 5 is the state space and A is the action space. For every timestep of the Markov process, a state S; E is given, and an action a, E A is chosen by the agent according to its policy [ [t) E A(A), where (AA) is the probability simplex over A. Upon executing the action ae, the MDP will transit to a new state 8t+1 S according to the transition probability T(s411 8t, a4) while the agent receives reward r(st, ai) E R. The goal of the agent is to maximize the discounted reward Σ, 2.r""(ss.. a²) with discount factor y [0, 1] over complete run, which is called episode. The state(-action pairs) collected through the run are called a staee-acttoo)) trajectory т. Trajectory segment this continuous subsequence of trajectory The state visitation frequency (state
work is defined occupancy) of a policy π is denoted as d= (s) = (1 - 7) 7" Pr((t = 8). See Appendix A for a detailed definition of state occupancy and other occupancies.
learning [12] addresses the problem of binary classification with feature x E IK and label y E {0, when only the positive dataset Dp and the unlabeled dataset Dy are known. Our solution leverages positive prior f)p = Pr(y = 1) and negative prior 7/, (r 0), which are unknown and treated as a hyperparameter.
Positive- Unntbiee Learning.
Offline Imitation from Observations Examples. Offline imitation learning from observations requires the agent to learn good policy from two sources of data: one is the task-specific dataset DTS. which contains state trajectories TTs = {#1; 82,..., (nn)] } from the expert that directly addresses the task of interest; the other is the task-agnostic dataset DTA. which contains state-action trajectories
TIA = [((s) a1), (2), a2)...., (Sna: anz) } of unknown optimality to the task of interest. Note that the task-specific trajectory TTS be incomplete; specifically, if only the last state exists example of success, then is called imitation from examples [13, 41J.
The state of the art methods in this field are SMODICE [41] and LobsDICE [23]. SMODICE minimizes the divergence between the state occupancy from specific data dds and the learner policy I occupancy d";; for example, when using a KL--ivergence as the metric, the objective is
min (L (s)lld((ss) (s)), s.t. is feasible policy.
However, since the task-agnostic dataset is the only source of correspondence between state and action, the state occupancy of the task-agnostic dataset d must be introduced. With some derivations and relaxations, the objective is rewritten
(T)
(L((")))l s.t. x is a feasible policy. max Exnar log (x)
Here, the first term R(s) log (s) (r((e) is an indicator for the importance of the state; high R(s) means that the expert often visits state s, and s is a desirable state. Such R(s) can be trained by a discriminator c(s): positive dataset Drs (label 1) and negative dataset DrA (label 0) are used to find an 'optimal' discriminator c = c Given this discriminator, we have R(s) = log (s) SMODICE then converts the constrained Eq. (2) to its unconstrained Lagrangian dual form with dual variable (ss,, and optimizes the following objective (assuming KL-divergence the metric):
(*())
min(1 - 7) [V(s)] + log (e,,,,,) +V))))+((6)))))) - ((s)),
(3)
where y is the discount factor and Pu is the distribution of the initial state in the MDP. In this formulation, R(s) can be regarded as the reward function, while V(s) is the value function. The whole objective is an optimization a convex function with respect to the Bellman residual. With V(s) learned, the policy is retrieved via weighted behavior cloning where the coefficient is determined by (s). LobsDICE is in spirit similar, but considers the occupancy of adjacent state pairs instead of single state.
3

---

3 Methodology
3.1 Motivation and Overview
As mentioned in Sec. 2, the DICE methods for offline LfO discussed above consist of three parts: reward generation, optimization of the value function V(s), and weighted behavior cloning. Such a pipeline can be unstable for two reasons. First, the method is indirect, as the weight for behavior cloning depends on the learned value function ((s), which could be inaccurate if the task-agnostic dataset is noisy or is not very related to the task of interest. This is aggravated by the fact that (s') used in the objective, which further
I-sample estimation of (ps'p(s'ss.aa (s')) and logsumexp destabilizes learning. Second, for KL-bbaed metrics, if no state appears twice in the task-agnostic data, which is common for high-dimensional environments, the derivative of the objective with respect to (s) is determined by the initial state term, the current state term - (s)s), and the next state term (s'). Thus, if non-initial state s' missing from the trajectory, then only (s')s') remains, which makes the objective monotonic with respect to the unconstrained V(s'). Consequently, V diverges for s' (see Appendix C in the for more detailed analysis and visualization in Fig. 7).
To address the stability issue, we develop Trajectory- Imitation Learning from Observations (TAILO). TAILO also seeks to find an approximate reward R(s) by leveraging the discriminator c(s), which is empirically a good metric for optimality of the state. However, compared to DICE methods which determine the weight of behavior cloning via dual program, we adopt much simpler idea: find 'good' trajectory segments using the discounted sum of future R(s) along the trajectory following state 8, and encourage the agent to follow them in the downstream weighted BC. To do so, we assign a much larger weight, a thresholding result of the discounted sum, to the state-action pairs for 'good' segments. Mcanwhile, small weights on other segments serve as regularizer of pessimism [25]. Such a method is robust to missing steps in the trajectory. Empiricially we find that the weight need not be very accurate for TAILO to succeed. In the remainder of the section, we discuss the two steps of TAILO: 1) training a discriminator to obtain R(s) (Sec. 3.2), and 2) thresholding over discounted sums of R(s) along the trajectory (Sec. 3.3).
3.2 Positive-Unlabeled Discriminator
Following DICE, R(s) is used as metric for state optimality, and is obtained by training a discrimi- nator c(s). However, different from DICE which regards all unlabeled data as negative samples, we use -uttiie--nll (PU) learning to train c(s), since there often are some expert trajectories or segments of useful trajectories in the task-agnostic dataset. Consequently, we treat the task- lask-agnostic dataset as an unlabeled dataset with both positive samples (expert of the task of interest) and varied negative samples (non-expert).
Our training of c(s) consists of two steps: 1) training another discriminator ds(s) that identifies safe negative samples, and 2) formal training of c(s). In the first step, to alleviate the issue of treating positive samples from DTA as negatives, we use debiasing objective [29] for the training of c((s) (see Appendix A for a detailed derivation):
min L1 = min - -np c's) + max((0, EL-Dr log(1 (g(11cc(s)) - EnwDss log(1 c' (s)))), (4) d'(a)
c(())
where the bias comes from viewing unlabeled samples as negative samples. Here, positive class prior Up is a hyperparameter; in experiments, we find results to not be sensitive to this hyperparameter.
In the second step, after ds(s) is trained, R'(s) = log (-)(s) is calculated for each state in the task-agnostic data, and the states in the (possibly incomplete) trajectories with the least 81 E (0. 1) portion of average (s)s) are identified as "safe" negative samples. Note, we do not identify "safe" positive samples, because trajectory that only has a segment useful for the task of interest might not have the highest average Rs(s) due to its irrelevant part; however, an irrelevant trajectory will probably have the lowest R' (s) throughout the whole trajectory, and thus can be identified as "safe" negative sample. We collect such samples to form a new dataset Dante TA-
d'ts)
Finally, the formal training of c(s) uses states from DTs as positive samples and D TA "safe" negative samples. The training objective of c(s) is combination of debiasing objective and standard cross entropy loss for binary classification, controlled by hyperparameter 32. Specifically, we use

---

min 32L2(c) + (1 (1- (22)L,( where c(a)
L2(c) min - Lyp Drs log c(s) + max(0, Don log(1 - c()) -ap log(1 c(s)))},
<(*) Ls(c) Drs log ((s) + Dew log(1 c(s)).
(5)
In this work, only consider 82 (0.1): empirically, found that B2 is better if the agent's embodiments across Dis and DIA are the same, and B2 = is better if the embodiments differ. This is because debiasing objective assumes positive samples still exist in the safe negatives, which pushes the classification margin further from samples in Dts, and has a larger probability to classify expert segments in DAA as positive.
3.3 -raeectory-Aware Thresholding
With the discriminator css) trained and R(s) = log I-c(s) obtained, we can identify the most useful
c(a)
trajectory segments for our task. To do this, we use simple thresholding which makes the weight employed in behavior cloning trajectory-awant Formally, the weight W(s;, 0,) for weighted behavior cloning is calculated as
((s,, 04)
En exp(aR(sit)))
(6)
where (s,, a.) is the i-th step in a trajectory, and a is a hyperparameter that controls the strength of thresholding; a balances the tradeoff between excluding non-expert and including sxpert-traeectories For an i which exceeds the length of the trajectory, we set 8;tj to be the last state of the (possibly incomplete) known trajectory, the final state is of significant importance in many applications [18]. This design also allows us to conveniently address the example-based offline IL problem, where the final state important. With the weights determined, with the objective max, (r W(8, a) log π (a)s) s), where π (a|s) is the desired policy.
finally conduct weighted behavior cloning
Experiments
In this section, we evaluate TAILO on five different, challenging tasks across multiple mujoco testbeds. More specifically, study the following two questions: 1) Is the algorithm indeed robust to incomplete trajectories either task-agnostic (Sec. 4.1) or task-specific (Sec. 4.2) data, and does work with little expert data in the task-agnostic dataset (Sec. 4.3)? 2) Can the algorithm also work well in example- based IL (Sec. 4.4) and learn from experts of different dynamics (Sec. 4.5)?
Baselines. We compare TAILO to four baselines: SMODICE [41], LobsDICE [23], ORIL [73], and Behavior Cloning (BC). Since LobsDICE works with state-pair occupancy, cannot solve example- based IL; thus, we substitute LobsDICE in example-based IL with RCE [13], a state-of-the-an example-based RL method. Unless otherwise specified, we use random seeds per method each scenario.
Environment Setup. Following SMODICE [41], unless otherwise specified, we test our algorithm on four standard mujoco testheds from the OpenAl Gym [5], which are the hopper, halfcheetah, ant, and walker2d environment. We use normalized average reward! as the main metric, where higher reward indicates better performance; for environments where the final reward is similar, fewer gradient steps in weighted behavior cloning indicates better performance. We report the change of the mean and standard deviation of reward with respect to the number of gradient steps.
Experimental Setup. For all environments, we use a 1.25, 61 0.8, Лр 0.2; B2 if DTs and DIA are generated by different embodiments, and B3 = 0 otherwise. We use 7 = 0.998 unless otherwise specified. We use an exactly identical discriminator and policy network as SMODICE: for the discriminator c(s) and c'(s), we use small Multi-Layer Perceptron (MLP) with two hidden layers, width 256, and tanh activation function. For actor æ, we use an MLP with two hidden layers, width 256, and ReLU [2] activation function. For the training of c(s) and c'(s). we use a learning
Normalization standard is according D4RL [16], and identical SMODICE [41].
5

---

Hopper, 1/20
Haffcheerah 1/20
Ant 1/0
Walker2d 1/20
122
139
Hopper 1/10
Haltchectah 1/10
Ant 1/10
Walker2d 1/10
1914
121
51
Hopper
Hallcheerah 1/5
Ant 1/5
Welker2d 1/5
199
121
51
Hopper
Halfcheerah 1/3
Ant 1/3
Walker 1/3
1344
330
59
Hopper 1/2
Halfcheetah 1/2
Ant 1/2
Normalized Reward
Walker 1/2
139
300
3400 SP00KE--11
Gradient
300
Figure 2: Reward curves for offline imitation learning with incomplete trajectories in the task-agnostic dataset, where the x-axis is the number of gradient steps and the y-axis is the normalized reward. The title for each sub- figure is the format of "environment namc"+"1/f "11/x" (task-agnostic data removed), where x E {2, 3, 5, 10, 20}. We observe the proposed method to be the most stable.
rate of 0.0003 and I-Lipschitz regularizer, run 10K gradient steps for (s).. and run 40K gradient steps for c(s). For weighted BC, we use a learning rate of 10 4,-4,_ a weight decay of 10 10-5 and run 1M gradient steps. For discriminator training, we batch size of 512; for weighted BC steps, a batch size of 8192. Adam optimizer [28] is used for both steps. See Appendix D for more details and Appendix FF for sensitivity analysis regarding the batch size, a, B1, $2, Up and 7.
4.1 Learning from Task-Agnostic Dataset with Incomplete Trajectories
Dataset Setup. We modify the standard dataset settings from SMODICE to create our dataset. SMODICE uses offline datasets from D4RL [16], where single trajectory from the "expert-v2" dataset used as the task-specific data. The task- aggossic data consists of 200 expert trajectories (200K steps) from the "expert-v2" dataset and 1M steps from the "random-v2" dataset. Based on this, we iterate over the state-action pairs in the task-agnostic dataset, and remove one pair for every pairs. In this work, we test æ E (2, 3, 5, 10, 20).
Main Results. Fig. shows the result for different methods with incomplete task-agnostic trajectories, where our method outperforms all baselines and remains largely stable despite decrease of æ (i.e., increase of removed data), as the weights for each state-action pair do not change much. In the training process, we often witness SMODICE and LobsDICE to collapse due to diverging value functions (see Appendix C for explanation), which is expected; for runs that abort due to numerical eтrоr,
reward of 0 for the rest of the gradient steps. Under few cases, SMODICE with KL-divergence works decently well with larger noises (c.g., Halfcheetah, 1/3 and Walker2d :/3)); this is because sometimes the smoothing effect of the neural network mitigates divergence. However, with larger batch size (See Fig. 23 in Appendix F.4.6) and more frequent and uniform updates on each data point, the larger the noise the harder SMODICE fails.
4.2 Learning from Task-Specific Dataset with Incomplete Trajectories
Dataset Setup and Main Results. We use SMODICE's task-agnostic dataset as described in Sec. 4.1. For the task-specific dataset, only use the first steps and the last

---

Happer, heac100 tailt
Halfcheetah head100 taill
Ant head100 tail
Walcer2d heac100 tailt
139
Hopper hese9U taill0
Halfchoetah. head50 taillU
Ant head9IC tall10
Walcer2d hesesu taillo
121
Hopper heac50 tail5
Hallcheetah head50 tall50
Ant head5C zell50
Walker22 heed50 taiisc
199
51
Hopper heacl0 tail90
Halfcheetah, head10 tail0
headlC
Walker22 headl0 tail90
121
Hopper heacl tail100
Halfcheetah headl 100
Ant headl Call100
Cormanin Howardd
Walker2d headl tail100
199
2P00KE--1
Gradient
300
Figure 3: Reward curves for offiine imitation with incomplete trajectories in the task-specific dataset. The title format for each subfigure is "environment name"+"head +"+"tail y" (states remained), where (æ, {(100, 1), (90, 10), (50, 50), (10, 90), (1, 100)}. We observe the proposed method to be the most stable.
steps in the expert trajectory, and discard the remainder. In this work, we test (I; y) E {(1, 100), (10, 90), (50, 50), (90, 10), (100, 1)). Fig. shows the result with incomplete task-specific trajectories, where our method outperforms all baselines and often achieves results similar to those obtained when using the entire task-specific dataset. In contrast, SMODICE and LobsDICE expectedly unstable in this setting.
4.3 Standard Offline Imitation Learning from Observations
Environment Setup. In addition to the four standard mujoco environments specified above, we also test on two more challenging environments: the Franka kitchen environment and the antmaze environment from D4RL [16]. In the former, the agent needs control 9-DoF robot arm to complete a sequence of item manipulation subtasks, such as moving the kettle or opening the microwave; the latter, the agent needs to control robot ant crawl through U-shaped maze and get to particular location. As the kitchen environment requires less steps to finish, we use = 0.98 instead of 0.998.
Dataset Setup. As existing methods already solve the four mujoco environments with SMODICE's dataset settings in Sec. 4.1 quite well (see Appendix for result), we test more difficult setting to demonstrate TAILO's ability to work well with few expert trajectories in the lask-agnostic dataset. More specifically, we use the same task-specific dataset, but only use 40 instead of 200 expert trajectories from the "expert-v2" dataset to mix with the 1M "random-v2" steps data and form the task- aggnosic dataset. For the more challenging kitchen and antmaze environment, we use the identical dataset as SMODICE, where single trajectory is used the task-specific dataset. The lask-agnostic dataset for the kitchen environment consists expert trajectories completing different subtasks (both relevant and irrelevant) in different orders, and the task-agnostic dataset for antmaze consists of data with varied optimality. See Appendix for details.
Main Results. Fig. 4 shows the result for different methods in standard offline imitation learning from observations. We find our method outperform all baselines on hopper, halfcheetah, ant and walker2d. Results are comparable to the best baseline kitchen and antmaze. In the experiment, we found ORII. to often diverge, and the performance of SMODICE varies greatly depending on the f-divergence: hopper, halfcheetah and walker2d, KL-divergence is much better, while x-

---

Hopper
Halfcheetah
Ant
1314
370
Walker2d
Kitchen
Normalized aaiiediizddd Reward
Antmaze
a
6.4
64
0.
06
1.3
Gradient Steps
3400 CE-48
SPOCKCE-1
0%L
LobiCcE
Figure 4: Reward curves for offline imitation learning from observations with few expert trajectories in the lask-agnostic dataset. In all six environments, our method either oulperforms or is comparable to the baselines.
Pointmaze Goal
Antmaze Goal
Kitchen Microwave Goel
Normalized Reward 3.
Kitchen Kettle Goal
Gradient Steps
Figure Reward curves for offline imitation learning from examples.
divergence is better on ant and kitchen. LobsDICE is marginally better than SMODICE, as the former considers state-pair occupancy instead of single state occupancy, which is more informative. Also worth noting: none of the methods exceeds normalized reward of 60 in the kitchen environment; this is because the SMODICE experiment uses expert trajectories that are only expert for the first out of all 4 subtasks. See Sec. F.3 in the Appendix for detailed discussion.
4.4 Learning from Examples
Environment Setup. Following SMODICE, test example-based offline imitation in three different testbeds: pointmaze, kitchen and antmaze. the mujoco-base based [59] pointmaze environment the agent needs to control pointmass environment, we test two different settings where the agent is given successful examples of moving the kettle and opening the microwave respectively (denoted as "kitchen-kettle" and "kitchen-mierowave"
2D plane navigate particular direction. For the kitchen
As pointmaze requires less steps to finish, we use = 0.98 instead of 0.998.
Dataset Setup and Main Results. Following SMODICE, use small set of success examples: < 200 states for antmaze and pointmaze and 500 states for kitchen (see Appendix for details) are given the task-specific dataset. For pointmaze, the task-agnostic data contains 60K steps, generated by a script and distributed evenly along four directions (i.e., 25% expert data); for other testbeds, task-agnostic data is identical to SMODICE described in Sec. 4.1. Fig. 5 shows the results for offline imitation learning from examples on all environments tested in SMODICE; TAILO is marginally better than the baselines on pointmaze, antmaze, and kitchen michbeew,,,r and is comparable (all close to perfect) on kitchen-kettle.
4.5 Learning from Mismatched Dynamics
Environment and Dataset Setup. Following SMODICE, we test our method on three environments: antmaze, halfcheetah, and ant. We task-specific data from expert with different dynamics (e.g.. ant with leg crippled; see Sec. E for details). Task-agnostic data follows SMODICE in Sec. 4.1.
Main Results. Fig. 6 shows the results for offline imitation learning from mismatched dynamics, where our method the best in all three testbeds. Among the three environments, halfcheetah the most difficult, as the state space for halfchectah with shorter torso is unreachable by normal

---

Ant Mismatch
Halfcheetah Mismatch
Normalizes ized Reward
Antmaze Mismatch
-0-
50
25
20
0.2
0.2
0.8
0.1
C.1
0.2
ca
Gradlent Steps
160
Figure 6: Reward curves offline imitation learning from mismatched dynamics.
halfcheetah, i... the assumption that (s) > O wherever (s) > 0 in SMODICE and LobsDICE does not hold; in such a setting, our method much more robust than DICE methods.
Related Work
Offline Imitation Learning and DIstribution Correction Estimation (DICE). Offiin imitation learning aims to learn a policy only from data without interaction with the environment. This is useful where immature actions are costly, e.g., dangerous factory. The simplest solution for offline imitation learning is plain Behavior Cloning (BC) [51]. Many more methods have been proposed recently, such as BCO [60] and VMSR [33] (which pscudolabel actions), offline extensions of GAIL [3, 31, 61, 73], and samilarity-basedd reward labeling [8, 35, 55, 64]. Currently, the state-of- the-art method for offline imitation learning DIstribution Correction Estimation (DICE) [23, 27, 32, 34,41], which minimizes the discrepancy between an expert's 's and learner's s state, atate-action, or state-pair occupancy. Our method inspired by DICE, but is much simpler and more effective. More recently, two works unify offline imitation learning with offline RL, which are oflline-RL-basedd MAHALO [36] and DICE-based ReCOIL [56]. Different from SMODICE and LobsDICE, ReCOIL minimizes the divergence between learner and expert data mixed with non-expert data respectively, which removes the data coverage assumption. However, ReCOIL faces the problem discussed in Sec. 3.1 when dealing with incomplete trajectories, and MAHALO is based on state-paie sate- similar to LobsDICE, which cannot solve II. with incomplete trajectories or example-based IL like our TAILO.
Learning from Observations and Examples. Learning from observations (LfO) [60] requires the agent learn from task-specific dataset without expert action, which is useful when learning from videos [44] or experts with different embodiments [55], as the expert action is either unavailable not applicable. Learning from examples is extreme case of LfO where only the final goal is given [13]. There are three major directions: 1) pseudolabeling of actions which builds an inverse dynamic model and predicts the missing action [33, 60]; 2) occupancy divergence minimization with either DICE [23, 27, 34, 41, 71] or inverse-Rl RL style iterative update [61, 67, 73]; and 3) RL/planning with reward assignment based on state similarity (often in visual imitation) [9, 55, 64]. Our proposed TAILO solves LfO with a simple solution different from existing ones.
Discriminator Reward. The idea of training a discriminator provide rewards for states is widely used in II. including inverse RL methods [15, 22, 31, 73], DICE methods [23, 27, 27,41], and methods such 2IWIL [65] and DWBC [68]. In this work, propose simple but explicit way to take the trajectory context into account, using the output from a discriminator as reward, which differs from prior works.
Positive-Unlabeler Learning. Rositive-Unlabele (PU) [11, 12, 29, 50] learning aims to solve binary classification tasks where only positive and unlabeled data are available. It is widely used in data retrieval [54], outlier detection [38], recommendation [7] and control tasks [67]. this work, utilize two PU learning achievements: the skill of identifying "safe" negative samples [39] and debiasing [11, 29]. The closest RL work to our use of PU learning is ORIL [73], which also uses positive-unlabeled learning to train discriminator for reward estimation. However, there are three key differences between our method and ORIL: we define R(s) differently for better thresholding. different techniques for PU learning to prevent overfitting, and, importantly, the removal of value function learning. See Appendix F.6 for an ablation to demonstrate efficacy of these changes.
Reward-Wear Weighted Regression idea in our Eq. (6) of weighted behavior cloning with weights being (often exponentiated) discounted return has been widely used in the RL community [1, 45, 63], and our objective resembles that of
[48] and Advantage- Weighted Regression(AWR [45]. The

---

RWR/AWR. However, our work differs from RWR/AWR inspired works [30, 43, 49] in the following aspects:
The objective for both AWR and RWR are built upon the related payoff procedure [10, 21}, Expecaaion-Maxxiizzttonn
which introduces an Expectation- Maximization (EM) procedure for RI. However, for offline IL in our case, iteration between E-step and M-step infeasible. There two workarounds for this: importance sampling and naively using one iteration. However, the former is known to be non-robust [42] and the latter, MARWIL [62], struggles in our testbeds (see Appendix F.2).
We use neither an adaptive reward scaling term [48] in the EM framework nor parametric value estimation [45, 62], which are both widely utilized by RWR/AWR inspired works. However, the former is not guaranteed to preserve an optimal policy and thus is not an advantage [58], while the latter struggles in our setting where learning good value function is hard, as illustrated by the performance of baselines such as DICE methods and MARWIL.
For all existing RWR/AWR works, the reward labels are assumed be available, which is different from our case.
Conclusion
We propose TAILO, simple yet effective solution for offline imitation from observations by training discriminator using PU learning, applying score to each state-action pair, and then conducting weighted behavior cloning with a discounted sum over thresholded scores along the future trajectory to obtain the weight. We found our method to improve upon state-of-the- baselines, especially when the trajectories are incomplete or the expert trajectories in the task-agnostic data are few.
Societal Impact. Our work addresses sequential decision-making tasks from expert observations with fewer related data, which makes data-driven automation more applicable. This, however, could also lead to negative impacts on society by impacting jobs.
Limitations and Future Directions. Our method relies on the assumption that there exist expert trajectories at least segments in the agnostii data. While this is reasonable for many appli- cations, like all prior work with proprioceptive states, it is limited in generalizability. Thus, one promising future direction is improve the ability summarize abstract "skills" from data with better generalizability. Another future direction is learning from video demonstrations, which is major real- wollw application for imitation learning from observations. Also, our experiments are based on simulated environments such as D4RL. Thus a gap between our work and real-life progress remains. While we following the settings of many recent works, such SMODICE [41], ReCOIL [56] and OTR [40], to bridge the gap using techniques such as sim2real in the robotics community [24] is another very important direction for future work.
Acknowledgemmetss. This work was supported in part by NSF under Grants 2008387, 2045586, 2106825, MRI 1725729, NIFA award 2020-67021-32798 the Jump ARCHES endowment through the Hcalth Care Engineering Systems Center, the National Center for Supercomputing Applications (NCSA) at the University of Illinois Lirbana-Champan through the NCSA Fellows program, the IBM-Iliinois Discovery Accelerator Institute, and the Amazon Research Award.
10

---

References
[1] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Maxi- mum posteriori policy optimisation. ArXiv:1806.0663 069200 2018.
[2] A. Agarap. Deep learning using rectified linear units (relu). ArXiv: 2018.
[3] S. Y. Arobb Off-policy adversarial inverse reinforcement learning, ArXiv:2005.01138 2020.
[4] M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm.
Mutual information neural estimation. In ICML, 2018.
[5] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
[6] A. Camacho, I. Gur, M. Moczulski, O. Naschum, and A. Faust. Sparsedice: Imitation learning for temporally sparse data via regularization. In the Unsupervised Reinforcement Learning Workshop in ICML, 2021.
[7] S. Chang, Y. Zhang, J. Tang, D. Yin, Y. Chang, M. A. Hasegawa-Johnson, and T. S. Huang. Positive-unlabeles learning streaming networks. In KDD, 2016.
[8] A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from "in-the-wild" human videos. 2/t.:://03.66117. 2021.
[9] X. Chen, S. Li, Li, S. Jiang, Y. Qi, and Song- Generative adversarial user model for reinforcement learning based recommendation system. In ICML, 2019.
[10] P. Dayan and G. E. Hinton. Using expecaaionn-maxiinzzaoon for reinforcement learning. Neural Computation, 1997.
[11] M. C. du Plessis, G. Niu, and M. Sugiyama. Analysis of learning from positive and unlabeled data. In NIPS, 2014.
[12] C. Elkan and K. Noto. Learning classifiers from only positive and unlabeled data. In KDD, 2008.
[13] B. Eysenbach, S. Levine, and R. Salakhutdinov. Replacing rewards with examples: Example- based policy search via recursive classification. In NeurIPS, 2021.
[14] G. Freund, E. Sarafian, and S. Kraus. A coupled flow approach to imitation learning. In ICML, 2023.
[15] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. In ICLR, 2018.
[16] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. ArXiv:2004.072/1 2020.
[17] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In ICML., 2018.
[18] Ghasemipour, R. Zemel, and S. Gu. A divergence minimization perspective on imitation learning methods. In CoRI, 2019.
[19] A. L. Gibbs and F. E. Su. On choosing and bounding probability metrics. International statistical review, 2002.
[20] K. Hakhamaneshi, R. Zhao, A. Zhan, P. Abbeel, and M. Laskin. Hierarchical few-shot imitation with skill transition models. ICLR, 2022.
[21] G. E. Hinton. Connectionist learning procedures. 1990.
[22] J. Ho and S. Ermon. Generative adversarial imitation learning, In NIPS, 2016.

---

[23] G. hyeong Kim, J. Lee, Y. Jang, H. Yang, and K. Kim. Lobsdice: Offline learning from observation via stationary distribution correction estimation, NeurlPS, 2022.
[24] Höfer, K. Bekris, A. Handa, J. C. Gamboa, M. Mozifian, F. Golemo, C. Atkeson, D. Fox, K. Goldberg, J. Leonard, C. Karen Liu, J. Peters, S. Song, P. Welinder, and M. White. Sim2real in robotics and automation: Applications and challenges. IEEE Transactions on Automation Science and Engineering, 2021.
[25] Y. Jin, Z. Yang, and Z. Wang. Is pessimism provably efficient for offline ri? In ICML, 2021.
[26] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning. In NeurlPS, 2020.
[27] G. Kim, S. Seo, J. Lee, W. Jeon, H. Hwang, H. Yang. and K. Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In ICLR, 2022.
[28] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[29] R. Kiryo, G. Niu, M. C. du Plessis, and M. Sugiyama. Positive-unlabeled learning with non-negative risk estimator. In NIPS, 2017.
[30] Kober and Peters. Policy search for motor primitives in robotics. In NIPS, 2008
[31] I. Kostrikov, K. K. Agrawal, D. Dwibedi, Levine, and J. Tompson. Discriminator-actorr Addressing sample inefficiency and reward bias in adversarial imitation learning. In ICLR, 2019.
[32] I. Kostrikov, O. Nachum, and Tompson. Initation learning via off-policy distribution matching. In ICLR, 2020.
[33] A. Kumar, S. Gupta, and J. Malik. Learning navigation subroutines from egocentric videos. In CoRL, 2019.
[34] J. Lee, W. Jeon, B.-J. Lee, J. Pineau, and K.-E. Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In ICML, 2021.
[35] Y. Lee, A. Szot, S.-H. Sun, and J. J. Lim. Generalizable imitation learning from observation via inferring goal proximity. In NeurlPS, 2021.
[36] A. Li, B. Boots, and C.-A. Cheng. Mahalo: Unifying offline reinforcement learning and imitation learning from observations. ArXiv:2303.17156, 2023.
[37] K. Li, A. Gupta, A. Reddy, V. H. Pong, A. Zhou, J. Yu, and S. Levine. Mural: Meta-learning uncertainty-aware rewards for ourvone-driven reinforcement learning. ICML, 2021.
[38] X.-L. Li, B. Liu, and S.-K. Ng. Learning to identify unexpected instances in the test set. In IJCAI, 2007.
[39] C. Luo, P. Zhao, C. Chen, B. Qiao, C. Du, H. Zhang, W. Wu, S. Cai, B. He, S. Rajmohan, et al. Pulns: Positive-unlabeled learning with effective negative sample selector. In AAAI, 2021.
[40] Y. Luo, Z. Jiang, S. Cohen, E. Grefenstette, and M. P. Deisenroth. Optimal transport for offline imitation learning. In ICLR, 2023.
[41] Y. J. Ma, A. Shen, D. Jayaraman, and O. Bastani. Smodice: Versatile offline imitation learning via state occupancy matching. In ICML, 2022.
[42] O. Nachum, Y. Chow, B. Dai, and L. Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. NeurIPS, 2019.
[43] T. Osa and M. Sugiyama. Hierarchical policy search via return-eeighted density estimation. In AAAI, 2018.
[44] Pari, M. Shafullah, S. Arunachalam, and Pinto. The surprising effectiveness of representation learning for visual imitation. ArXiv:2112.0151 2021.
12

---

[45] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advintage-weiggted regression: Simple and scalable off-policy reinforcement learning. ArXi::1910.00117,2211
[46] K. Pertsch, Y. Lee, and J. Lim. Accelerating reiforrcmmt learning with learned skill priors. In CoRL, 2020.
[47] K. Pertsch, Y. Lee, Y. Wu, and J. J. Lim. Demonstrattio--uuidee reinforcement learning with learned skills. In CoRL, 2021.
[48] Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. In ICML, 2007.
[49] J. Peters, K. Mulling, and Y. Altun. Relative entropy policy search. In AAAI, 2010.
[50] M. Plessis, G. Niu, and M. Sugiyama. Convex formulation for learning from positive and unlabeled data. In ICML, 2015.
[51] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In NIPS, 1988.
[52] N. Rajaraman, L. Yang. Jiao, and K. Ramchandran. Toward the fundamental limits of imitation learning. In NeurlPS, 2020.
[53] R. Ramamurthy, P. Ammanabrolu, K. Brantley. Hessel, R. Sifa, C. Bauckhage, H. Hajishirzi, and Y. Choi. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. In ICLR, 2023.
[54] A. Santara, J. Datta, S. Sarkar, A. Garg, K. Padia, and P. Mitra. Punch: Positive unlabelled classification based information retrieval in hyperspectral images. ArX1v:190..04547,0 2019.
[55] Sermanet, C. Lynch, J. Hsu, and S. Levine. Time-contrastive networks: Self-supervised learning from multi-view observation. ArXiv:1704.06880 2017.
[56] H. S. Sikchi, A. Zhang, and S. Nickum. Imitation from arbitrary experience: A dual unification of reinforcement and imitation learning methods. ArXiv:2302.08566 2023.
[57] D. Silver, T. Hubert, J. Schritwieserr, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L.. Sifre, D. Kumaran, Graepel, P. Lillicrap, Simonyan, and D. Hassabis. Mastering chess and shogi by self-play with general reinforcement learning algorithm. Science, 2018.
[58] M. Strupl, F. Faccio, D. R. Ashley, R. K. Srivastava, and J. Schmidhuber. Reward-weighted regression converges to a global optimum. In AAAI, 2022.
[59] E. Todorov, T. Erez, and Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.
[60] F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. In IJCAI, 2018.
[61] Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. ICML Workshop on Imitation, Intent, and Interaction, 2019.
[62] Q Wang, Xiong, L. Han, H. Liu, T. Zhang, et al. Exponentially weighted imitation learning for batched historical data. In NeurIPS, 2018.
[63] Z. Wang, A. Novikov, K. Zolna, J. S. Merel, J. T. Springenberg, S. E. Reed, B. Shahriari, N. Siegel, C. Gulcehre, N. Heess, et al. Critic regularized regression. In NeurIPS, 2020.
[64] A. Wu, A. Piergiovanni, and M. S. Ryoo. Model-basee behavioral cloning with future image similarity learning. CoRL, 2019.
[65] Y.H. Wu, N. Charoenphakdee, H. Bao, V. Tangkaratt, and M. Sugiyama. Imitation learning from imperfect demonstration. In /CML, 2019.
[66] Xia, C. Li, R. Martin-Martin, Litany, A. Toshev, and S. Savarese. MoG.M Leveraging motion generation in reinforcement learning for mobile manipulation. In ICRA, 2020.
13

---

[67] D. Xu and M. Denil. Positive-unlabeles reward learning. In CoRL, 2019.
[68] H. Xu, X. Zhan, H. Yin, and H. Qin. Discriminattr--wiihhedd offline imitation learning from suboptimal demonstrations. NeurIPS, 2022.
[69] T. Xu, Z. Li, Yu, and Z.-Q. Luo. On generalization of adversarial imitation learning and beyond. arXiv preprint arXiv:2106. 10424, 2021.
[70] T. Yu, G. Thomas, L.. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model- based offline policy optimization. In NewrIPS, 2020.
[71] Z. Zhu, K. Lin, B. Dai, and J. Zhou. Off-policy imitation learning from observations. In NeurIPS, 2020.
[72] L. Ziniu, X. Tian, Y. Yang, and L. Zhi-Quan. Rethinking valuedice does it really improve performance? In ICLR Blog Track, 2022.
[73] K. Zolna, A. Novikov, K. Konyushkova, C. Gulcehre, Z. Wang, Y. Aytar, M. Denil, N. de Freitas, and S. E. Reed. Offline learning from demonstrations and unlabeled experience. In Offline Reinforcement Learning Workshop at NeurlPS, 2020.
14

---

Appendix: A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories
The Appendix is organized as follows: first, we summarize our key findings. Then, in Sec. A, we provide a more rigorous introduction of the key mathematical concepts, and in Sec. B we provide the pseudocode for the training process of TAILO. In Sec. C, we explain why DICE methods struggle with incomplete trajectories in the task-agnostic or task-specific data. We then list additional implementation details of our method and the baselines in Sec. D, include additional experimental settings in Sec. E,, and present additional experiment results well as ablation studies in Sec. F; after that, we report the training time and computational resources utilized by each method in Sec. G. thub. com/
Finally, we examine the licenses of assets for our code in Sec. H. See https: //ghthb.. KaiY Raiia9/29/// TAILO for our code.
The key findings of the additional experimental results summarized as follows:
Can our method discriminate expert and non-expert trajectories in the task-agnostic dataset? In Sec. F.I, we visualize the change of R(s) and behavior cloning weight W a) along the trajectory, as well as the average R(s) and Wss, a) for expert and non-expert trajectories in the task-agnostic dataset DIA. We find that our method successfully dis- criminates expert and non-expert trajectories in the task-agnostic dataset across multiple experiments.
How does our method perform compared to other offline RL/IL methods? In addi- tion to LobsDICE and SMODICE, in the appendix compare our performance to the following offline RL/IL methods: a) a very recent DICE method, ReCOIL; b) methods with extra access to expert actions reward labels, such as DWBC [68]. model- based RL methods MOReL. [26] and MOPO [70]; c) offline adaption for Advantage-Wa Weighted Regression(AWRR [45], MARWIL [62]] d) recent Wasserstein-basee offline II method, OTR [40]. We found our method to be consistently better than all these methods. See
Sec. F.2 for details.
How does our method and baselines perform on the kitchen environment with "Kitchen- Complete-v0"? The kitchen testbed adopted in SMODICE the Seitchen-complete-vee environment in D4RL, which has a different subtask sequence from the ones evaluated in the environment and the ones in the task-agnostic data ("kitchen- mixed-vO"); the two environments also have different state spaces. When changed to all baselines except BC fail, because the state space between task- agnostic data and evaluation environment is different. Nonetheless our method still works. See Sec. F.3 for details.
an expert trajectory from
with batch How sensitive is our method to hyperparameters, such as at B1, B2g Mpp and ?? We conduct a sensitivity analysis in Sec. F.4, and find our method be generally robust to reasonable selection of hyperparameters.
Our method and baselines use different batch sizes. Is this a fair comparison? In Sec. F.4.6, we test our method with batch size 512 and SMODICE/LobsDDN size 8192. We find that our method more stable with larger batch size for few settings, but SMODICE and LobsDICE do not generally benefit from an increased batch size. In addition, our method with batch size 512 still better than the baselines. Thus, the comparison is fair to SMODICE and LobsDICE.
How important the Lipschitz regularizer for training of R(s)? Lipschitz regularizer is used in many methods tested in this work, including our method, SMODICE, LobsDICE, and ORII.. In Sec. F.4.7, we found this regularizer to crucial for the generalization of the discriminator and R(s).
Our standard imitation from observation setting different from that of SMODICE. Is this fair? Compared with the SMODICE's original setting, in Sec. 4.3 we focused on more challenging experiment setting with less expert trajectories the task-agnostic dataset for the four mujoco environments (hopper, halfcheetah, ant, and walker2d). In the appendix, we also provide experimental evaluation under the SMODICE'S original setting, and find our method still outperforming basclines. See Sec. F.5 for details.
15

---

How is our method different from ORIL, and where does the performance gain come from? Our method is different from ORIL in three major aspects: definition of R(s). positive-unlabeled learning technique, and policy retrieval. In Sec. F.6, we report that each of the three aspects contributes to the reward increase; among them, policy retrieval the most important factor, and the positive-unlabed learning technique is the least.
A Mathematical Concepts
In this section, we rigorously introduce four mathematical concepts in our paper, which are state(- action/-pair) occupancy, debiasing objective for positive-unlabeled learning, f-divergences, and Fenchel conjugate. The first one is used throughout the paper, the second is used in the training of c(t), and the others are used in Sec. C.
State, State-Action, and State-pair Occupancy. Consider an infinite horizon MDP (S Tr, 7) with initial state distribution po. where the state at the t-th timestep is 8, and the action is ar- Then, given any fixed policy π, the probability Pr(s; s) of landing in any state s for any step t determined. Based on this, the state visitation frequency d' (s) with policy æ, also known as state occupancy, is defined as d" (s) Ei=1 Pr(s: s). Similarly, the state-action frequency is defined as d= (s,a) = P Pr(st 5,0, = a), and the state-pair frequency is defined as d' (s. 8') = E Pr(*;= = 81+1 = 8¹). For better readability, with a little abuse of notation, we the same d for all three occupancies throughout the paper. In this work, we refer the occupancy with the learner's policy л as d", we refer to the occupancy with average policy of the task- aask-agnostic dataset d'AA, and to the occupancy with average policy of the task-specific dataset as d!S.
Debiasing Objective for UonabieeUnllaeeled Learning. Consider a binary classification task with feature E R" and label y E {0,1). 1}. Assume we have access to both labeled positive dataset Dp and labeled negative dataset DN and the class prior, i.e., the probability of having a label when uniformly sampled from the dataset, is 7pp Pr(y 1) for positive labels and 7,, Pr(y 0) for negative samples. Then, with sufficiently many samples, the most commonly used loss function is a cross entropy loss. The average cross entropy loss can be approximated by
Up (e..)))))pp log 0] + Tn (a |- log(1 Ú)]
where g = c(x) is the output of the discriminator.
In positive-unlabeled learning, we only have access to the positive dataset Dp and unlabeled dataset Du as an unlabeled mixture of positive and negative samples. One naive approach is to regard all samples from Du negative samples. While this sometimes works, it falsely uses the loss function for positive samples in Dy, and thus introduces bias. To avoid this and get better classification results, multiple debiasing objectives [11, 29] have been proposed. Such objectives assume that the unlabeled dataset Du consists of 1hp portion of positive data and 7/o portion of negative data. Thus, we have
(xzy)) |- log 0)] = Dr 0) + (x log(1 - Ø) (8)
Correspondingly, Eq. (7) can be approximated by
hp (a.))) log + (a.)))Dy log(1 - 1) Tip (x.n)~DD log(1 - û)]
This is the formulation utilized by ORIL [73]. However, as Kiryo et al. [29] pointed out, the red part in Eq. (9) is approximation for nh (x.g)) log(11 O)], and thus should be no less than O; violation of such a rule could lead to overfitting (see [29] for details). Therefore, we apply a max operator max(-; 0) to the red part. The objective thus becomes
- -Inp log ÿ + (ax(B log(1 - () -)ppBB..))) log(1= - 2). 0)).
(10)
Note that while Kiryo et al. [29] uses an alternative update conditioning on the red term, we found the max operator to be more effective in our case. For our first step of discriminator training, we use Eq. (10), with Drs as Dr, DTA as Dy, state as feature v, and action a as label y. For the second step, use Eq. (10) again for spsk-sp data with mismatch dynamics, but this time we use
16

---

DIA safe Dy; otherwise, we use Eq. (7) with Dis as Dp, DIA as DN, state feature and action a as label y.
f-divergences. The f-divergence is the basis the DICE family of methods [23, 27, 32, 34, 41]. DICE methods minimize f-divergeecc, such the KI.-divergence between the learner's policy and the average policy on the task-specific dataset. For any continuous and convex function f. any domain X, and two probability distributions p, q on X, the f-divergence between and is defined as
Dy(plla) = Bz P(x) ((x)) )..
(11)
For example, for the Ki-deveree f(z) = zlog2, and Dy Dy(pl14) KL(pllg) = Ex (p(x)) log p(n]]
((x) {(x) Ezap log p(u), for x2 diveeeege,,e,, f(x) (x 1)?, and Dy(pllg)
((x)'
(pll)) = En p(z) - 1) Sawx (p(x)~g(z))² Note that SMODICE uses f(x) = { 1)?
((z)
9(0)
which is essentially half x² -iveereeeeee
Fenchel Conjugate. For any vector space Ω with inner product (., and convex and differentiable function f : S E, the Fenchel conjugate of f(x), x E 82 is defined
(v) = max (ar, y) f(x).
(12)
sen
In SMODICE, the derivation of the x2 divvergeeece uses f(z) = 2 (z 1)², and the Fenchel dual is (v) (y 1)2. However, such Fenchel dual is obtained with constraint x, while in SMODICE with KL-divergence, æ is constrained on probability simplex. This extra relaxation could be a factor for its worse performance on some testbeds.
B Algorithm Details
Alg. shows the pseudocode of our method.
C
Why DICE Struggles with Incomplete Trajectories?
In our experiments, we empirically find that SMODICE and LobsDICE struggle with either incom- plete task-specific or incomplete task-agnostic trajectories. section.
We give an extended explanation this
C.1 Incomplete Task-Specific Trajectory
The phenomenon that DICE struggles with incomplete expert trajectories is first discussed [72]: the work mentions that subsampled (i.e., incomplete) trajectories artifically "mask" some states and causes the failure of ValueDICE. Recent discussion indicates that such failure on subsampled task-specific trajectories is closely related to overfitting [6] and a lack of generalizability [69, 72].
C.2
Incomplete Task-Agnostic Trajectory
C.2.1 KL-Based Formulation
The reason that SMODICE [41] with KL-divergence and LobsDICE [23] struggle with incom- plete task-agnostic trajectories is more direct: consider the final objective of SMODICE with KL-divergence:
min(1 - -e [V(s)] + log (',,, exp R(s) + +V((5) - V(s),,
(13)
where Pa A(S) is the initial state distribution over the state space S, and "reward function" (s) is labeled by discriminator c(s) as described in Sec. 2.
R(s) = log [
17

---

Similarly, the final objective of LobsDICE is
(im(117)Bss-pp V(s)+ (1 log
expl( ))) Da
(R(s,s') + +VV V(s)), (14) I+a
where the "reward function" M(s s') is based on state pair instead single state, and a > is a hyperparameter. Note, both methods use 1-sample estimate for the expectation of the dual variable E. (s') for future state 8'.
Assume that no two states in DIA are exactly the same, which is common in high-dimensional continuous state space. such case, there are only two occurrences of (s)) for particular state E Dra in Eq. (13) and Eq. (14): for initial states, one in the linear term and the other in - (V(s)
inside the exp-term; for other states, one the - V(s) term and the other in the (V(s") term inside the exp-term. Consider trajectory
{(11, 41 $2), ($2, 02, $3). (Sns On. 8n+1)] } DIA.? If (82, 02 83) is missing, we have the following ways to make up:
1. Use DIA without extra handling. In this case, the only occurrence of ((5)) will be (83). as no future state exists for $2. As the objective is monotonic (which is not the case in RL) with unconstrained V(s)), the convergence of the algorithm solely relies smoothing from nearby states in DTX. This is also the method that we tested in Sec. F.
2. Let the trajectory contain ($1, a1 83). In this case, the method would learn with a wrong dynamic of the environment because $1 cannot transit to $3 by conducting action aj such a ground truth would suggest;
3. Consider 82 as a terminal state of the trajectory т, where the agent could learn to halt in the middle of an expert trajectory if is expert trajectory;
4. Train an inverse dynamic model for pseudolabeling of the action. However, such method cannot deal with more than one consecutive transition missing.
In conclusion, none of the workarounds descrihed above addresses the concern. Note that the problem described here is also valid for ValueDICE [32] and many follow-up works such as DemoDICE [27] long the Donsker- Varadhan representation [4] of the KL-diivergence is used. Fig. 7 illustrates this divergence, which empirically verifies our motivation.
wakee
(a) Minimum ((s)
(b) Maximum ((s)
(c) Normalized Reward
Figure 7: Visualization of collapsing patterns of SMODICE-E KL in Sec. 4.1 (walker 1/5 is collpasing: walker. 1/3 is not). Walker 1/5 stops early due to NaN values in training, which we denote as 0 reward in our paper. It is clearly shown that the reward decrease is closely related to Vss) divergence.
C.2.2 2- --Based Formulation
Empirically, we found that SMODICE with >d-divergence struggles on several testbeds such as halfcheetah, hopper, and walker2d, which is consistent with the results reported by SMODICE. The performance gap could be due to the following two reasons:
First, violation of Theorem I in SMODICE [41]. Theorem in SMODICE reads follows:
Theorem 1. Given the assumption that dtA (s)(s) > whenever dT O, we have
Because DICE methods utilize the next state for state-action pair (s, a). we write the state-action trajectories transition (s, a, trajectories.
18

---

(T)
(L (s)lds) I Egnd [log( (i) 11 KL (s,aal (s,
(15)
and furthermore, for any f-divergence Dj larger than KL.,
d
<L(d"ss)lld') I Bend llog(
11 Dy(d" (s)aa as)
(16)
(TS
The right hand side of Eq. (15) is the optimization objective of SMODICE with KL-divergence, and di-divereence is introduced via Eq. (16) upper bound.
However, SMODICE uses f()) 2 (xr - 1)2 instead of f(x) (x - 1) as the x2 diverggeece:
ie., the divdivergence is halved the objective. While the x divereegee is an upper bound of the KL-divergence [19], half of the d-=diveree is not. For example, consider two binomial distributions p) and 2). The KI--divergence between the two is plog + (1 p) log i-q-
and the >'ddivergence (D-9)²
(q-p)² 1-q When = 0.99 and g 0.9, the KL--divergence is
0.99 log 1.1 0.01 1.1 0.0713, and the x--divergence is 0.0g² 0.09 0.09, where half of 0.9 + 0.1 X -d-diveree is not
the x2 divergence is smaller than the KL-divergence. Thus, SMODICE with optimizing an upper bound of KL(d" (s)lld' and the performance is not guaranteed. In Fig. 8, we plot the reward curves of using halved and full >i-divergence under the settings Sec. F:5; we find that using full X² diveerggen significantly increases performance the walker2d environment and performs similarly on other env environments, though still much worse than our method.
Hopper
Normalized Reward
12-
14
60
2.
1.0
8.3
0.2
04
2.0 -22
0.6
04 59
100
Gradiant
Figure 8: Performance comparison between using halved x diveeregee (purple) adopted by SMODICE [41] and full x² -iveergee (fuchsia) with experiment settings in Sec. F.5. Full x² -iveergee performs significantly better the walker2d environment than halved --divergeenee though still much worse than our method.
Second, SMODICE with divveeegee uses fv( = [ + 1)² as the Fenchel conjugate of i.e., E R;
f(x) 1 (zr 1)2_ Such conjugate is obtained when there is no constraint however, such æ. as indicated by the derivation SMODICE with KL--divergence (Example 1 of Appendix C in SMODICE), should be the probability simplex. This relaxation could be another reason for the performance drop in SMODICE with x" d'viieergence.
C.3 Other Recent Works
We noticed that there also recent works in the DICE family that try address the issue of incomplete (i.e., subsampled) trajectories, such as SparseDICE [6] and CFIL [14] for online imitation learning; LobsDICE [23] also discusses learning with subsampled expert trajectories. However, all those works only focus on incomplete task-specific (i.e. expert) trajectories instead of task-agnostic trajectories; also, method solve example- based IL and task-specific trajectories with all but one state provided at the beginning, which differs from their settings where the sampling of expert states/state-action pairs is uniform throughout the trajectory.
D Additional Implementation Details
Our code is provided the supplementary material. We implement our algorithm from scratch, and use the implementation of SMODICE [41] (https / gitgggthhbbb..// comm/Jasoo//2206//MDDD as
19

---

the codebase for SMODICE, ORIL [73], and RCE [13]. We obtain the code for LobsDICE [23] from their publicized supplementary material on OpenReview.
Tab. summarizes the unique hyperparameters for our method, and Tab. summarizes the common training paradigms for our method and the baselines. For all baselines, if the hyperparameter values in the paper and the code are different, we record the values from the code. We discuss the influence of batch size and sensitivity of our hyperparameters in Sec. Tab. summarizes the hyperparameters specific to other methods. SMODICE, ORIL, and RCE first train a discriminator, and then jointly update actor and critic, while LobsDICE jointly updates all three networks.
E
Additional Experimental Settings
In this section, we describe detail how the environment setup and how the dataset is generated.
E.1
Offiine Imitation Learning from Task-Agnostic Dataset with Incomplete Trajectories
Environment Settings. Sec. 4.1 tests four mujoco environments: hopper, halfcheetah, ant, and walker2d. The detailed configuration for each enviromment is described as follows:
Hopper. this environment, an agent needs to control a single-legged robot to jump forward by controlling the torques on its joints. The action space A is [-1,1] one dimension for each joint; the state space angle and velocity. Note that the r-coordinate is not part of the state, which means that the expert state approximately repeats itself periodically. Thus, in Fig. 10 the expert R(s) is periodic. For this environment well halfcheetah, ant, and walker2d, the reward is gained by surviving and moving forward, and the episode lasts 1,000 steps.
11-dimensional, which describes its current
Similar to hopper, an agent needs to control a 2D cheetah-like robot with torques on its joints to move forward. The action space A is [-1, 1]6, and the state space S is 17-dimensional describing its coordinate and velocity.
Halfcheetah.
Ant. In this environment, the agent controls a four-legged robotic ant to move forward in 3D space with a 11 111-dimensional state space describing the coordinate and velocity of its joints, as well as contact forces each joint. The action space is 1|.
Walker2d. The agent in this environment controls 2D two-legged robot to walk forward, with state space being 27-dimensional and action space being [-1
Fig. 9 shows an illustration of each environment.
Dataset Settings. We generate our dataset on the basis of SMODICE. SMODICE uses trajectory (1, 000 states) from the "expert-v2" dataset in D4RL [16] the task-specific dataset Drs. and concatenates 200 trajectories (200K state-action pairs) from the "expert-v2" dataset and the whole "random-v2" dataset D4RL (which contains 1M state-action pairs) be the task-agnostic dataset DrA-
Based on this, take the dataset from SMODICE, concatenate all state-action pairs from all trajectories into an array, and remove a state-action pair for every x steps; we test x {2, 3, 5, 10, 20) in this work.
the task-specific dataset from SMODICE. For the task-agnostic dataset,
E.2 Offline Imitation Learning from Task-Specific Dataset with Incomplete Trajectories
Environment Settings. Sec. 4.2 tests hopper, halfcheetah, ant, and walker2d with the same environ- ment settings those discussed in Sec. E.1.
Dataset Settings. We use the task-agnostic dataset from SMODICE as described in See. E. For the task-specific dataset, we take the first x and last y steps from the task-specific dataset of SMODICE as the new task-specific dataset, and discard the other state-action pairs; in this work, we test (z, y) {(1, 100), (10, 90), (50, 50), (90, 10), (100,
20

---

E.3 Standard Offline Imitation Learning from Observation
Environment Settings. addition to the four mujoco environments tested in Sec. E.1, Sec. 4.3 tests the kitchen and antmaze environments. The detailed configuration for the two environment is described as follows:
Kitchen, In this environment, the agent controls a 9-DoF robotic arm to complete sequence of 4 subtasks; possible subtasks include opening the microwave, moving the kettle, turning on the light, turning on the bottom burner, turning on the top burner, opening the left cabinet, and opening the right cabinet. The state space is 60-dimensional, which includes the configuration of the robot, goal location the items to manipulate, and current position of the items.
Antmaze. In this environment, the agent controls a robotic ant with 29-dimensional state space and 8-dimensional action space move from one end of a u-shaped maze the other end.
Fig. 9 illustrates the two environments.
(a) Hopper
(c) Ant
(c) Kitchen
(b) Halfcheetah
(d) Walker2d
Antmaze
Figure 9: Illustration of environments tested in Sec. 4.3 based on OpenAl Gym [5] and D4RL. [16].
Dataset Settings. For the four mujoco environments, we take the task-specific dataset from SMODICE described Sec. E. existing methods already perform well on those environ- ments with SMODICE's dataset settings (see Sec. F.5 for results), we introduce more challenging task-agnostic dataset concatenating 40 trajectories (40K state- acttin pairs) instead of 200 from the "expert-v2" dataset and the whole "random-v2" dataset in D4RL the task-agnostic dataset DTA-
For the more challenging kitchen and antmaze environments, as SMODICE. For the kitchen environment, we use trajectory (184 states) from the "kitchen- complete-vo" dataset as the task-specific dataset Drs and the whole siitceen-mixd-vO"" dataset as the task-agnostic dataset DTA. which contains 33 different task sequences with total of 136, 950 state-action pairs. For the antmaze environment, we the expert trajectory of length 285 given by SMODICE as the task-specific dataset Dis. We use the dataset collected by SMODICE as the task- agnostic dataset DIA, which contains 1, 348, 687 aate-action pairs.
identical dataset settings
E.4 Offline Imitation Learning from Examples
Environment Settings. Sec. 4.4 tests antmaze and kitchen in the same environment settings as the ones discussed in Sec. E.3, but only require the agent complete particular subtask instead
21

---

of all four (we test "opening the microwave" and "moving the kettle"). Additionally, Sec. 4.4 tests the pointmaze environment, which is simple environment where the agent controls pointmass to move from the center of an empty 2D plane to particular direction. The action is 2-dimensional; the state is 4-dimensional, which describes the current coordinate and velocity of the pointmass.
Dataset Settings. For the antmaze environment, we use the same task-agnostic dataset as the one discussed in Sec. E.3, and we use 500 expert final states selected by SMODICE as the task-specific dataset. For the kitchen environment, we use the same task-agnostic dataset as the one discussed in Sec. E.3, and randomly select 500 states from the "kitchen- mixed-vO" dataset that has the subtask of interest completed as the task- speciiii dataset. For the pointmaze environment, we use trajectory of 603 expert trajectories with 50K state-action pairs as the task agnostic dataset, where the trajectories moving up, down, left and right each take up 1/4. We use the last state of all trajectories that move left as the task-specific dataset.
E.5 Offiine Imitation Learning from Mismatched Dynamics
Environment Settings. Sec. 4.1 tests halfcheetah, ant, and antmaze with the same environment settings as the ones discussed in Sec. E.1 and Sec. E.3.
Dataset Settings. We use the task-agnostic dataset from Sec. E.1 (ant, halfcheetah) and See. E.3 (antmaze). For the task-specific dataset, we use the data generated by SMODICE, which is a single trajectory conducted by an agent with different dynamics. More specifically, for halfcheetah, we use an expert trajectory of length 1, 000 by cheetah with a much shorter torso; for ant, we use an expert trajectory of length 1, 000 by an ant with one leg crippled; for antmaze, we use an expert trajectort of length 173 by a pointmass instead of an ant (and thus only the first two dimensions describing the current location are used for training the discriminator). See Appendix H of SMODICE [41] for an illustration of the environments.
F
Additional Experiment Results
F.1
Can Our Method Discriminate Expert and Non-Expert Data in the Task-Agnostic Dataset?
In Sec. 4, an important problem related to our motivation is left out due to page limit: Does the algorithm really succeed in discriminating expert and non-expert trajectories and segments in the task-agnostic data? We answer the question here with Fig. 10. It shows R(s) and the weight for behavior cloning ((s, a)) along Sec. 4.3, hopper with first 50 step and last 50 step as task task-specific dataset in Sec. 4.2, and pointmaze in Sec. 4.4. We also plot the average R(s) and weight (ss.a) for state-action pairs in all expert trajectories and non-expert trajectories in DTA- The result clearly shows that, even if the discriminator cannot tell the expert states from the non-expert states the beginning of the episode because states from all trajectories are similar to each other, the weight successfully propagates from the later expert state to the early trajectory, and thus the agent knows where to go even when being close to the starting point; also, the weight (s, a) is smoother than the raw R(s), which prevents the agent from being lost in the middle of the trajectory because of few steps with very small weights.
expert and a non-expert trajectory in DíA on halfcheetah in
F.2 Comparison to Other Baselines
Besides the baselines tested the main paper, we compare our method to a variety of other offline RL/IL methods. The tested methods include: a) a very recent method, ReCOIL; b) methods with extra access to expert actions or reward labels, such as DWBC [68]. model-based RL methods MOReL [26] and MOPO [70]; c) offline adaption for Advantage- edvantage-Weighted Regression (AWR) [45], MARWIL [62]; d) a recent Wasserstein- based offline IL method, OTR [40]. As we do not have the code for ReCOIL, and model- based RL methods takes long time to run on our testbed, we test our method on their testbeds and use their reported numbers for comparison; for other methods, we test on our testbed (both under SMODICE setting in Sec. F.5 and standard Lf0 setting in Sec. 4.3). We found our method to consistently outperform all methods, even those with extra access to reward labels or expert actions.
22

---

Hattcheetah Minghe (ES.N)
Horper
Portmmaze
Enviecement Staps
Enviromment Staps
Envronmert Stops
Figure 10: Illustration of R(s), behavior cloning weight ((s, a). and their average expert and non-expert trajectories DTA. Curves for expert trajectories are plotted red color, while non-expert trajectories are plotted in green color. The non-expert trajectory on the hopper environment short because the agent topples quickly and the episode is terminated early. It is clear that our design of weights propagates the large weight from later expert states and smoothes the weight along the trajectory. Hopper has large periodic change of R(s), because the velocity vector for the joints of the agent periodic.
F2211 ReCOIL
Similar to Sec. 4.1, we test on the four common mujoco environments, which are hopper, halfcheetah, ant, and walker2d; the results we tested include random + expert (identical to SMODICE), random + few expert (30 expert trajectories instead of 40 in Sec. 4.3), and medium expert (substitute random dataset medium dataset; see ReCOIL for details). The result random + expert is illustrated in Fig. 26, where our method outperforms ReCOIL. The result of random + few expert is illustrated in Fig. 11, and the result medium expert is illustrated in Fig. 12. On all test beds, our method is significantly better than ReCOIL.
Hopger
Hellcheerah
Ant
Welkerzd
199
6
20-
0.5
2.2
1.0
010
134
3.11
Gradient Steps
Figure 11: Performance comparison between our method and ReCOIL on the "random+few expert" testbed of ReCOIL.
Happer
Hairchestah
Ant
Walker2d
Narmal noward
B
06
210
05
N6COL
Gracient Steps
Figure 12: Performance comparison between our method and ReCOIL on the "medium+expert" testhed of ReCOIL.
23

---

F.2.2 Model-Based RL
In this section, we compare our method to MOReL [26] and MOPO [70] which are provided with ground-truth reward labels, i.e., MOReL and MOPO have an advantage. The trajectory with the best return is provided as the task-specific dataset to our method. Tab. 4 shows the performance comparison between our method, MOReL and MOPO; despite being agnostic to reward labels, our method is still marginally better than MOReL (74.3 vs. 72.9 reward averaged over widely tested environments (halfcheetah, hopper, walker2d) x (medium, medium-replay, medium-expert)) and much better than MOPO (74.3 vs. 42.1 average reward).
F.2.3 DWBC, MARWIL and OTR
In this section, additionally compare our method three other baselines offline IL: DWBC [68] that also trains discriminator using Dositive-Unlabeles (PU) learning. MARWIL [62] that is a naive adaptation of Reward-Weightee Weeighted Regression [48] to offline scenarios with a similar actor objective as TAILO, and the recent Wasserstein- based method OTR [40] which computes Wasserstein distance between the task-specific and task agnossti trajectories, assigns reward label based on optimization result and conducts offline RL. Among those methods, MARWIL and OTR can be directly applied to our scenario, while DWBC requires extra access to expert actions. Fig. and Fig. 14 illustrate the result, respectively on the settings of Sec. F.5 and Sec. 4.3.
Hopper
HoRchestah
Waleer2d
AzE
Figure 13: Performance comparison between our method and DWBC, MARWIL and OTR on settings of Sec. F.5. Our method outperforms all other methods, which all struggle in our settings.
Herger
Wallorri
14
Grndlent
Figure 14: Performance comparison between our method and DWBC, MARWIL and OTR on settings of Sec. 4.3.
Our method outperforms all other methods, which all struggle in our settings.
F.3 Kitchen Environment Evaluated on KKitchen-Complete VO"
In Sec. 4.3, we mentioned that the SMODICE experiment uses expert trajectory that is only expert for the first of 4 subtasks. More specifically, SMODICE uses the expert trajectory in the "kicchen-completevoo" environment in D4RL the task-specific dataset, and uses data from the "kitchen-mixed-v0" environment as the task-agnostic dataset; they also evaluate on seitchen-mixed vO." However, the subtask list to complete in cettchen-complet--ene Switch, Slide Cabinet), while the subtask list to complete in "kitchen-mixed-vo"" Kettle, Bottom Burner, Light Switch). 3 Thus, the list of subtasks that is accomplished in expert
is [Microwave, Kettle, Light
[Microwave,
*See https:/ sithub..... com/ ottp://eocccaatttoooo/////////tn py in D4RL and lines 29, 164 and 208 in http:://.hhhhcccccce/e/eaaaat/tt oil_ www.orrttiiinsss.omooo py for details.
com/

---

trajectory (from "kitchen "kitchen-completer complete-v0") and the list of subtasks evaluated by the environment (from "kicchen-mixed-vO"" are only identical the first two subtasks. We follow their setting Sec. 4.3, and this is the reason why the normalized reward for any method is hard to reach 60 in Fig. 4, i.e., reach an average completion 2.4 tasks.
We now evaluate on saiichen-compeee--voe and Fig. 15 shows the result when evaluating on this environment. The other settings remain identical the ones discussed in Sec. 4.3. Note, this new evaluation setting introduces another challenge since Kiichen-coomplte-vOO" has a different set of subtasks to accomplish, the dimensions masked zero differ; to be more specific, the 4lst and 42nd dimensions are paro-masked in "kitchen-cee "kitchen-complete complete-vii but not in "kitchen- "kitchen-mixed-ee mixe---xxxdvv and vice versa for the 49th dimension. Thus, the -gnossti data are in a different state space from the environment for evaluation. This significantly increases the difficulty of the environment, because the states met by the agent in evaluation are out of distribution from the task-agnostic data. Even under such a setting, our method as well as BC remains robust, while all the other baselines struggle.
Normalized Reward
amredcaeel smodice-chi
-0
LOGSDACE
0.l
0.2
C.4
0.0
3.0
Gradient Steps
200
Figure 15: Reward curves for the kitchen environment evaluated "kilchen-complete-vO., only our method and are robust to the change state space from task-agnostic data.
E4 Hyperparameter Sensitivity Analysis
In this section, we conduct sensitivity analysis for the hyperparameters in our method.
F.4.1 Effect of a
Fig. 16 illustrates the reward curves tested on the settings of Sec. 4.3 with different (for scaling of
R())), where the value we used throughout the paper a = 1.25. While an extreme selection of the hyperparameters leads to a significant decrease of performance (e.g., = 2.0 for antmaze), our method is generally robust to the selection of a.
F.4.2 Effect of B1
Fig. 17 shows the reward curves tested on the settings of Sec. 4.3 with different 31 (ratio of "safe" negative samples), where the default setting is 31 0.8. Our method again generally robust the selection of B1. Obviously the reward decreases with an extreme selection of B3, such as 81 = 0.3. Intuitively, such robustness comes from two sources: 1) with small ratio of expert trajectories task-agnostic dataset (< 5%), thus there is little expert data erroneously classified safe negatives; 2) the hipschitz-smootlle discriminator yields good classification margin, and thus the 60%- trajectories with lower average reward represent all non-expert data well.
the
F.4.3 Effecc of B2
In this work, we use 83 when the embodiment of the task-specific dataset Dis different from that of the task-agnostic dataset DIA- We use 3g = otherwise. Ideally, after selecting safe negatives from the first step of the discriminator training, the obtained DTA safe should consist of (nearly) 100% non-expert data, and thus we use B2 0: however, when DTs is collected from different embodiment, the recognition of "safe negatives" will be much harder, because the states reachable by the task-specific expert could be different from the agent that was used to collect task-agnostic
25

---

Hopper
Halfcheotah
Art
390-
370
Walkerrd
130
Hormllzed Rewward
07
300
Figure 16: Reward curves tested in settings of Sec. 4.3 with different a. The default 1.25 plotted red, lower a in blue, and higher a in green. The further a deviates from the default, the deeper the color. Our method is generally robust to the selection of a.
Hallcheetah
131
370-
Vallerrd
Kitchen
Antmaze
339
Normalized Rewer
W
6.6
Gradiant Steps
21.0000
23-33
$2-03
11-0.
Figure 17: Reward curves tested in the settings of Sec. 4.3 with different 31 The default 81 is plotted in red, lower in blue and higher in green, with deeper color indicating further deviation from the default. Our method is generally robust to the selection of B1.
data. Thus, the debiasing objective in the formal training step, i.e., B2 = We compare the reward of using B2 = and 32 1 in Fig. 18, where 32 = 1 works significantly better than B2 on halfcheetah with mismatched dynamics.
Ant Mismatch
Halfcheetah Mismatch
Normalized Reward
Antmaze Mismatch
73
120
110
20
100
0.0
0.2
0.6
0.8
0.0
0.6
0.0
02
0.4
0.6
Gradient Steps
Figure 18: Reward curves for 3n 1 (i.e., debiasing objective) and A2 0 (i.e., normal cross entropy loss)
environments with mismatching dynamics. 82 better than 82 0 on the halfchectah environment with mismatched expert dynamics.
F.4.4 Effect T
Fig. 19 shows the reward curves tested in the settings Sec. 4.3 with different 7p. where the default setting is Ωp = 0.2. The results show that our method is robust to the selection of πp.
26

---

Sopper
Halfcheotah
Art
170
330
120
Normalizer Rerward
50-
Gradient
1eh
Figure 19: Reward curves tested on the settings Sec. 4.3 with different TJp. The default dp plotted lower in blue and higher in green, with deeper color indicating further deviation from default. Our method robust to the selection of
F.4.5
Effect of
Fig. 20 shows the reward curves tested in the settings of Sec. 4.3 with different (decaying factor for weight propagation along the trajectory); our default setting is y 0.98 for the kitchen and pointmaze environment, and 0.998 otherwise. We found that when is too low (e.g., 0.95), weights from the future expert state cannot properly propagated to initial states where most trajectories are similar, and thus the algorithm is more likely to fail; more extremely, when y = 0, our method works much worse as shown Fig. both Sec. F.5 and Sec. 4.3 settings, which illustrates the necessity of propagation of future returns. Generally however, our method is robust to the selection of y.
Hopper
Halfchootah
Art
300
Walkerr2d
Kitchen
Antmaze
135
Normalized Reward
W
x
Grodient Staps
0000
$114
Figure 20: Reward curves tested in the settings of Sec. 4.3 with diffcrent 7. The default y is plotted in red, lower in blue and higher in green, with deeper color indicating further deviation from the default; note, kitchen uses default y of 0.98 and other environments 0.998. Our method is robust to the selection of
F.4.6 Effect of Batch Size
Since we use different batch size (8192) than SMODICE and LobsDICE (512) for weighted behavior cloning, one possible concern is that the comparison might be unfair to the DICE methods, as we process more data. To address the concern, we test our algorithm with batch size 512 and SMODICE/LobsDIC
with batch size 8192 Sec. 4.1, Sec. 4.2, and Sec. 4.3. The results are illustrated in Fig. 22, Fig. 23, and Fig. 24. Generally, we found that most testbeds, our method with a batch size of 512 is slightly less stable than the default batch size 8192, but still on par or better than the baselines; meanwhile, SMODICE and LobsDICE with batch size 8192 work
27

---

(a) SMODICE (Sec. F.5) setting
(b) Sec. 4.3 setting
Figure 21: Ablation of our method with y struggles both experiment settings of Sec. F.5 and
Sec. 4.3, showing that > 0 crucial to performance of our method.
slightly better on Sec. 4.3 than batch size 512, but less stable in other scenarios such as Sec. 4.1 and Sec. 4.2. In conclusion, there is no general performance gain by increasing the batch size for SMODICE and LobsDICE. Thus, our comparison is fair to SMODICE and LobsDICE.
Hooper
Halfcheecah
Ant
We Ieer2d
130
Nun 222 Fanaae
27
26
300
Gradient Steps 300
50100
212
107 2012
Figure 22: Performance comparison of SMODICE, LobsDICE, and our method with batch size 512 and 8192 on mujoco testbeds of standard offline imitation from observation (Sec. 4.3). Non-default settings (SMODICE/LobsDICE with batch size 8192, deviations, while default settings (SMODICF/L DOCCD with hatch size 512, with batch size 8192) with standard deviations. Our method with batch size 512 works slightly worse, is still hetter than baselines; haselines with batch size 8192 work slightly better than use a batch size of 512.
with batch size 512) single lines without standard
F.4.7 Effect of Lipschitz Regularizer
Lipschitz regularizer on the discriminator used for many methods tested this paper, such as our method, SMODICE, LobsDICE, and ORIL. Intuitively, the regularizer makes the classification margin of the discriminator c(s) smooth, and thus can give higher weights to expert trajectories in the task-agnostic dataset DTA instead of overfitting to the few states given in the task-specific dataset Drs. In practice, we found this regularizer to be crucial; Fig. 25 shows the change of average
c(a) R(s) = log T-c[s] of all expert trajectories under the setting of Sec. 4.3 with respect to gradient steps. The result shows that R(s) for the expert trajectories extremely high when the Lipschitz regularizer is removed, which indicates severe overfitting and significantly worse performance.
28

---

Hopper 1/20
Haffcheerah, 1/20
Ant. 1/0
Walker2d Malker2d_1/20
190-
139
Hopper 1/10
Halfcheerah 1/10
Art 1/10
Walker2d 1/10
139-
Hopper
Halfchecrah 1/5
Ant 1/5
Walker2d 1/5
1994
131
Hopper 1/3
Hallchocrah 1/3
Ant 1/3
Walkerza 1/3
130-
50
Hopper 1/2
Halfchocrah 1/2
Ant 1/2
50 Sonooaa Steword
Walker2d 1/2
1991
5
51 Gradient
365
SHA0 5322
SWO0 0150
LE An-312
Figure 23: Performance comparison of SMODICE, LobsDICE and our method with batch size 512 and 8192 on mujoco testbeds of offline imitation with incomplete task agnostic trajectories (Sec. 4.1). Note that SMODICE and LobsDICE with a larger batch do not necessarily work better; contrast, larger batch size often leads less stability on environments such as Hopper_ 1/2, Walker2d, 1/2 and Ant_ 1/3, Our method with batch size 512 works slightly worse than 8192, but is still on par or better than the basclines.
F,5 Performance Comparison Under Identical SMODICE Settings
In Sec. 4.3, use less expert trajectory DIA than the SMODICE setting. Fig. 26 shows the result where we use identical dataset settings as SMODICE, where our method is still the best among all baselines. Such result is consistent with that reported in SMODICE, For ReCOIL [56], the code is not available, we plot their average reward straight line; our method significantly outperforms ReCOIL..
F.6 Ablation Comparing Our Method and ORIL
While our method and ORIL both utilize a discriminator-base R(s) with posittie-uunlabled learning, there are three major differences:
1. Definition of R(s). For discriminator c(s) trained with non-expert samples as label (S(a)
and expert sample as label 1, we use R(s) = log (li(a) = log T-c(s] while ORIL uses R(s) = css).
o(s)
2. Positive-Unlabeled Learning Techniques. The training of discriminator C((s) in ORIL is only one step, and uses Eq. (9) as the training objective. contrast, we first use Eq. (10) to find safe negative samples, and then use Eq. (10) or Eq. (7) (see Sec. A for details) to train the final discriminator c(s) and consequently obtain R(s).
3. Policy Retrieval. ORIL conducts RI. on the offline dataset with reward labeled by R(s), while our method uses non-paraeetric approach calculate coefficients for weighted behavior cloning.
29

---

Happer, heac100 tailt
Halfcheetah, head100, taill
Ant head100 tail
Wallerrd heac100 tailt
199
Hopper heacod taillo
Halfcheetah, headSo tall10
Ant headiC tall10
Walcer2d head90 taill
121
Happer heac50 tailso
Halfcheetah, head50 tall50
Ant head5C tall50
Walver2d heac50 taiisc
199
131
Hopper heaciu tailoo
Hallchectah nea510 tangy
Ant headiC ta190
Walkerid heedil tailo
121
5
Happor heael tall100
Halfchectah headl tal 100
Ant headl tall100
800 malizaa Rowarrd
Walker2d headi, tail10c
139
0.1
Gradient 366
SHODKGE 0132
SPOCKSS
LabiKt 8312
Figure 24: Performance comparison of SMODICE, LobsDICE and our method with batch size 512 and 8192 mujoco testbeds of offine imitation with incomplete saek-ipecific trajectories (Sec. 4.2). Again, larger batch size sometimes worse for SMODICE and LobsDICE, e.g.. Halfcheetah headl tail100 and Hopper_heae headl t0il Our method batch size 512 works slightly worse than 8192, but remains on par or better than the baselines.
We compare eight variants of the methods, denoted as ours, ours-VI, ours V2, ours V3, ORIL-logR- VI, ORIL-logR-VE V2, ORIL-01-VI, and ORIL-01-V2 V2. Tab. summarizes the differences between the eight variants.
The results illustrated in Fig. 27. Generally, all variants except "ours" work least marginally worse than "ours," which shows that all three differences matter in the final performance. Among the three variants of "ours." ours-v2 has the closest performance "ours," which indicates that the positive-unlabeled learning technique is the least important factor, though it still makes difference on environments such halfcheetah with mismatched dynamics as shown in Fig. 28. Moreover, while learning technique)
ours-vl (different R(s)) and ours-v3 (different R(s) ORIL positive-unlabele are significantly worse than "ours," ORIL-01-VI. ORIL-0I-V2, ORIL-logR- VI, and ORIL-logR- V2 perform even worse and are only marginally different from each other. This indicates that retrieval of policy (RL vs. non non- con-parametric)
the most important factor for the performance gap.
Additionally, in order to illustrate the importance of -nnttbledd (PU) learning in general, we test the performance of our method without PU learning and find significant performance decrease on some scenarios; Fig. 29 illustrates two of the failure cases without PU learning.
G Computational Resource Usage
All our experiments are conducted on an Ubuntu 18.0 18.04 server with 72 Intel Xeon Gold 6254 CPUs @ 3.10GHz and a single NVIDIA RTX 2080Ti GPU. Given these resources, our method requires about 4 hours to finish training in the standard offline imitation from observation scenario, while BC needs 3 4 hours, ORII, SMODICE, and LobsDICE need about 2.5 3 hours. In our training process, training of R(s) (both steps included) requires only 15-20 minutes. The inference speed for all methods is similar as the actor network is the same, and is thus not bottleneck.
30

---

10-
Sucerage hust Expert Thppetory
2000
4002
5002
5000
10202
cradieent Sheps
Figure 25: The average R(s) = log T-e(s] of the expert trajectories in the task-agnostic datasct with respect to the gradient steps of discriminator training on the halfcheetah environment. use the settings discussed in Sec. 4.3. The run without Lipschitz regularizer (red curve) overfits to the expert state, and thus has much higher average R(s). Such run diverges to infinity in later training.
cia)
Hopper
Haltcheerah
Ant
Walker2d
22
02
0900002
S40 CM
$L000
OC
Gradlent Steps
Figure 26: Reward curves for hopper, halfcheetah, ant, and walker2d evaluated under the identical settings as SMODICE: our method is the best among all methods including ReCOIL.
H Dataset and Algorithm Licenses
Our code is developed upon multiple algorithm repositories and environment testbeds.
Algorithm Repositories. We implement our method and behavior cloning from scratch. We test RCE, ORIL and SMODICE from the SMODICE repository, which has no license. We get LobsDICE code from their supplementary material the publicized OpenReview submission, which also has no license.
Environment Testbeds. We utilize OpenAl gym [5], mujoco [59], and D4RL [16] as testbed, which have an MIT license, an Apache-2.0 license, and an Apache-2.0 license respectively.
31

---

hopper
Hellcheetah
AVt
130
Walkar2d
Kitchen
M
No mamizzzd Rewarr
180
Gradlent
340
Figure 27: Ablation between method and ORIL: Tab. 5 for reference to the legend. While ours-V2 with ORIL-style positive-unlabeles learning has the closest performance to our final algorithm, still marginally worse. ORIL-01 and ORIL-logR fail, which shows that retrieval of the policy is the most important factor for the performance gap.
OUTS V2
Normalized Reward
Gradient Steps
Figure 28: Comparison of ours and ours-V2 on the halfcheetah environment with mismatched dynamics
(Sec. 4.5). Ours-V2 V2 fails on this environment.
Ha JallCheetah hiilCheethhmmmmmtthh mismatch
Hopper head50 tail50
Normalized Reward
60
| l |
60-
60-
0.4
0.0
de
1ce
Gradient Steps
205
Figure 29; Examples Failure cases of our method without learning; on the left is the halfcheetah environment with mismatched dynamics 4.5, and on the right is the hopper environment with only first and last 50 steps in the task-specific dataset (Sec. 4.2). In both cases, our method without PU learning struggles while method with PU learning succeeds.
32

---

// Sgeal the last state in T3 Algorithm 1: Our Algorithm, TAILO
Input
:state-only task-specific dataset DTs, state-action lask-agnostic dataset DTA containing
m7 trajectories
Input
Discounted sum rate 7 ratio of "safe" negative sample 31, positive prior Tip :Nuberr of gradient steps for pretraining R1, formal training n2,, and number of epoches for weighted behavior cloning rg
Input
Input
:Randomly initialized discriminator c(s) and c((s), parameterized by w and w'
Input
:Learning rate a Learned policy (a(ss)) parameterized by 0 begin
// Pretraining Discriminator
for i {1, .,. n1 do
Sample DTS, DTA L -np co (s(1) + max(0, d((2)) 7p log(11 c((s)))]
w + w - PL
// Labeling Safe Negative
for i E (1. m] do foreach E T, do
// Iterating through trajectories
(ss) (s) log I-c
R(T:)) E.,, R(a) mm
// average trajectory
4 31 quantile of R
DealoTA - (T) <g}
// Formal training of Discriminator
for i E (1, .2) do
10
Sample 81 DTS, DeateTA if Expert embodiment is different then
11
(mp log c(s1) + lo 7pp log(1 d ((1))))
else
12
< - ((1) + log(1 c(82))
Al
13
t w a
// Assi dssignment of Weights for i (1, , m do foreach E T, do
// Iterating through trajectories
15
16
R(s) +- log (
c(s)
17
(s, + (xp(ea((p))) R()())
R((mo)
18
U+
I-7
19
foreach $ E T; do W(8, a) u 7u + (s,
// This time in reverse
21
// average over trajectory
// Weighted Behavior Cloning for j E {1.2, (1.2..... M2} do foreach (s, Dra do
22
// for loop over epochs
23
// for each data point
24
I - (as
0 0 0L
33

---

Hyperaaameter
Value
Meaning
1.25
Scaling factor for calculation of weights
61
0.8
Estimated ratio of safe negative samples
(mismatch), (others)
Whether debiasing objective in formal training
0.2
Positive prior
(kitehen, pointmaze), 0.908 (others)
docay factor in weight propagation along the trajectory Table 1: Hyperparameters specific to our method.
Type
Hyperparameter
ours
BC
LobsDICE
SMODICE Tanh
ORIL
RCE
Network
256.
256]
Activation Function
Tanh
N/A
ReL.U
Tanh
Tanh
Learning
0.0003
0.0003 0
0.0003
0.0003
Weight Decay
0
N/A
0 1K steps 256
0
Training Length
(10+401K steps
N/A
1M steps
1K steps
IK steps
Batch Size
512
N/A
512
256
256
Optimizer
Adam
N/A
Adam
Adam
Adam
Adam
Actor
Network
[256, 256]
256]
[256,
[256,
[256, 256]
[256,
Activation Function
ReI.U
Rel.U
ReI.U
Rel.U
ReL.U
ReL.U
Learning Rate
0,0001
0.0001
0.0003
0.0003
0.0003
0.0003
Weight Decay
10-
10-2
0
Truining length
1M steps
IM steps
1M steps
1M steps
IM steps
IM sleps
Batch Size
8192
512
512
Optimizer
Adam
Adam
Adam
Adam
Adam
Adam
Tanh Squashed
Yes
Yes
Yes
Yes
Yes
Yes
Critic
Netwark Size
N/A
N/A
[256, 256]
[256, 256]
[256, 256]
1256, 256]
Activation Function
NIA
N/A
ReLU
ReLU
ReLU
ReLU
Learning
NIA
0.0003
0.0003
Weight Decay
NIA
NIA
0.0001
0.0001
0
Training Length
steps
steps
steps
steps
Batch Size
N/A
N/A
512
512
512
512
Optimizer
NIA
N/A
Adam
Adam
Adam
Adam
Discount Factor
N/A
0.99
0.99
0.99
0.99
Table Training paradigms for method and baselines; Disc. is the abbreviation for discriminator: [256, 256]
network size means network with two hidden layers and width 256. For our method, (10 + 40)K mcans
10K gradient steps for the first step and 40K for the second step. tanh-squashed means a Tanh applied at the end of the output to ensure the output action is legal.
Method LobsDICE RCE, ORIL
Hyperparameter
Value
Notation
Regularization factor
0.1
RL algorithm
TD3 [17]
Policy Update Frequency
2
Policy Noise
0.2
Noise Clip
0.5, 0.5]
Target Network Update Rate
0.005
Table 3: Unique hyperparameters for other methods.
Environment
MOReL
MOPO
TAILO (Ours)
Halrcheetah-Medinn
42.1
42.3
39.8
Hopper-Mediun
95.4
28
56.2
Walker2d- Mediuum
77.8
17.8
71.7
40.2
53.1
42.8
Hopper-Mcdduu--Rgee
93.6
67.5
83.4
Walker2d-MMdiuumRRe
Replay
49.8
39.0
61.2
Mallchhctta--Meddun-EE-ee
53.3
63.3
94.3
Hopper- Seerrr-Medduuupppppeee
108.7
23.7
111.5
Walker2d-Meeium-
95.6
44.6
108.2
Average
72.9
42.1
74.3
Table 4: The performance comparison between our method and model- RL methods on D4RL mujoco
offline datasets. Our method works even better than the offline RL methods with extra access to the underlying reward label.
34

---

Name
R(s)
PU learning
Policy Retrieval
log I-c(s)
c(a)
two-step with max(-;
non-parametric
ours-VI
10c(s)
two-step with max(- 0)
non-parametric
ours-V2 V2
log (-c(a)
c(s)
one step without max(:, 0)
non -on-aaraettric
ours-V3 V3
10c(s)
one step without max
non-parametric
ORIL-01-VI
c(s)
one step without max(-,0)
RL
ORIL-logR-V
log [-o(a)
one step without max(:.0)
RL
ORIL-01-V2
c(s)
two-step with max(-.0)
RL
ORIL-logR-V2 log I-c(8)
c(s)
two-step with max(-;
RL
Table 5: Configurations of the variants ablated in Fig. 27. Ours-VI I and ours-V3 10c(s) instead of c(s) scale the behavior cloning weight so be similar "ours." "
35


Reviews:
[{"review_id": "5gOo2UuWsR", "reviewer": "Reviewer_iGFb", "summary": "This paper proposes a simple weighted BC algorithm for offline RL with missing data. The authors first study that DICE family of algorithms suffer from missing data due to inaccurate value estimation or sparsity of observations which causes undesired monotonicity. The authors propose training a reward model from PU data and use MC value estimation with BC to train a policy. Experimental results on offline RL benchmarks show that this method is more robust to missing data and outperforms previous DICE family of methods.", "questions": "Please see above for specific questions.", "limitations": "The authors study some limitations but a discussion on whether progress on D4RL reflects progress on real life would be helpful.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper discusses limitations of DICE family of models in a practical way. The proposed method is also practical in a real setting. Results show robustness and overall improvement in missing/noisy data regime.", "weaknesses": "- The paper studies DICE family of algorithms with noisy data but the proposed method is mainly independent of these findings. It is a new objective without any value function or duality, which feels like it is not focusing on circumventing issues with DICE.\n\n- Model based methods are missing from the comparison.\n\n-  The proposed method is similar to [1] in which the authors use exponential of advantage with BC for batch RL setting. While they don’t study missing data setting, I believe the objective is more general and can be applied to yours as well. A detailed comparison would be helpful.\n\n- Which one is more crucial: learning a better reward using PU data or using discounted future rewards for the value estimate? What is the performance if you used one-level PU learning without safe examples or two-level PU learning with 1-step reward?\n\n- Eq (6) doesn’t handle missing data explicitly. What happens if some $i+k,k>0$ is missing from the trajectory? Do you just ignore that from the summation?\n\n- I would have expected that with more noise, DICE methods perform worse. But in Figure-2, SMODICE-KL performs well Walker2d_1/3 and worse with Halfcheetah_1/5 while for others it is the opposite. Could you please explain why?\n\n\n\n\n\n[1] Exponentially Weighted Imitation Learning for Batched Historical Data. Qing Wang, Jiechao Xiong, Lei Han, peng sun, Han Liu, Tong Zhang.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the valuable feedback.\n\n### Q1. The paper studies DICE with noisy data, but the proposed method is not focusing on circumventing issues with DICE methods.\n\nThough our method looks very different, our motivation is to solve the issues in DICE methods as simply as possible. Below is a summary of our motivations:\n\n**1. We want to adopt the reward design of SMODICE [1] and LobsDICE [2], as they are empirically effective.** However, with expert segments in the task-agnostic dataset, the reward design with normal binary classification objective sometimes fails. For this reason we improve by introducing Positive-Unlabeled (PU) learning.\n\n**2. We want to address the problem of SMODICE and LobsDICE that they require complete task-agnostic trajectories to work well.** We found theoretically (**Sec. C** in the appendix) that SMODICE and LobsDICE formulations are brittle if incomplete trajectories are present. We also observed this empirically as the value function update is often non-robust (which is also true for ORIL [3]). \n\nBased on this, we find that simply using the discounted sum of future exp(R(s)) is a good solution to address both points and improve on all testbeds of SMODICE. Thus, though the final methods are very different, our idea is in spirit an improvement over DICE methods.\n\n### Q2. Comparison to model-based methods.\n\nGreat suggestion. We additionally compare our method to two model-based offline RL methods MOReL [4] and MOPO [5]. Notably, MOReL is much slower to train, requiring 2 days as opposed to 4 hours of our approach on our machine (NVIDIA RTX 2080Ti). To improve efficiency, we compare our method to MOReL and MOPO which are provided with ground-truth reward labels, i.e., MOReL and MOPO have an advantage. The trajectory with the best return is provided as the task-specific dataset to our method. Despite being agnostic to reward labels, our method is still **marginally better than MOReL** (**74.3 vs. 72.9** reward averaged over 9 environments {halfcheetah, hopper, walker2d} * {medium, medium-replay, medium-expert}), and **much better than MOPO** (**74.3 vs. 42.1** average reward).\n\nBelow is the detailed performance comparison:\n\n| Dataset | MOReL | MOPO | TAILO (ours) |\n|---|---|---|---|\n|halfcheetah-Medium(M)|42.1|**42.3**|39.8|\n|hopper-M|**95.4**|28|56.2|\n|walker2d-M|**77.8**|17.8|71.7|\n|halfcheetah-Medium-Replay(MR)|40.2|**53.1**|42.8|\n|hopper-MR|**93.6**|67.5|83.4|\n|walker2d-MR|49.8|39.0|**61.2**|\n|halfcheetah-Medium-Expert(ME)|53.3|63.3|**94.3**|\n|hopper-ME|108.7|23.7|**111.5**|\n|walker2d-ME|95.6|44.6|**108.2**|\n|average|72.9|42.1|**74.3**|\n\n### Q3. Add comparison to MARWIL [7].\n\nWe additionally compare to MARWIL as requested, and we find that consistent with prior work [8], **MARWIL performs similar to plain BC and worse than our method.** See the pdf in the global response for reward curves. \n\n### Q4. The effect of using PU learning vs. using future reward for value estimates, and more ablations on PU learning.\n\nPart of the answer can be found in the appendix **Sec. F.6**; we additionally conduct experiments for the remaining questions and summarize here (see the pdf of the global response for reward curves).\n\n**1. Using future rewards for value estimates is more crucial, but one-level PU without safe examples also leads to a performance drop in some environments.** The comparison between ours and ORIL variants in **Sec. F.6** shows that even with rewards from PU learning, RL often fails. Using 1-step PU learning without safe examples causes a significant performance drop in some environments, as Fig. 23 and 24 depict. \n\n**2. We additionally test the result with 2-step PU learning + 1-step reward and without PU learning.** The result shows that while performance does drop without PU learning, it is less important than using the discounted sum of future reward instead of 1-step reward (i.e., $\\gamma=0$) as value estimation.\n\n### Q5. About handling missing data in the task-agnostic dataset.\n\nYes, we ignore missing data from the summation in Eq. 6. The idea is that the weighted sum does not need to be very accurate to work well; e.g., if every other step is missing, the result will be summing over every other future state with $\\gamma’=\\gamma^{0.5}$. Because $\\gamma$ is close to $1$ in long-horizon environments, the result does not change much.\n\n### Q6. The unintuitive behavior of SMODICE-KL. \n\n**1. Collapses of SMODICE-KL come from value function divergence.** As stated in **Sec. C.2.1** in the appendix, SMODICE-KL diverges with missing data in the middle of the task-agnostic trajectories. When this happens (e.g., halfcheetah_1/5, walker2d_1/5 in Fig. 2), SMODICE-KL collapses; otherwise, it performs decently well (see pdf in the global response for visualization). \n\n**2. The smoothing effect of the Neural Network (NN) mitigates the divergence, but the strength of the effect varies.** As the data distribution differs, the smoothing effect varies across datasets. Thus, the time it takes for divergence differs.\n\n**3. With more and uniform updates, the abnormality caused by NN smoothing disappears.** As shown in **Fig. 19**, when the batch size is larger, the updates on each data point are more frequent and uniform. With such a batch size, the trend that increased noise leads to worse performance on SMODICE-KL (thin blue line) is more obvious. \n\n### Q7. Limitations on the gap between D4RL and real-life progress.\n\nGreat advice. We will add the following discussion to the limitation section in the camera-ready version:\n\nA limitation of our work is that our experiments are based on simulated environments such as D4RL. Thus a gap between our work and real-life progress remains. While we are following the settings of many recent works, such as SMODICE [1], ReCOIL [9] and OTR [10], to bridge the gap using techniques such as sim2real in the robotics community [11] is another very important direction for future work. \n\n### References\nSee global response."}, {"review_id": "PPsRgrtrOb", "reviewer": "Reviewer_rs8a", "summary": "The authors propose TAILO, Trajectory-Aware Imitation Learning from Observations, a method to solve MDPs from offline data in the form of task-specific expert states and task-agnostic state-action trajectories. The method addresses the instabilities of algorithms like DICE, takes the context of a trajectory into account even when it may be incomplete, and demonstrate across many Mujoco environments that TAILO outperforms baselines especially with incomplete trajectories. ", "questions": "-", "limitations": "-", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ [Clarity] The paper is well-written, organized, and easy to follow. \n+ [Clarity] The authors include relevant background about DICE and document baselines clearly.\n+ [Originality] The authors propose a relatively simple solution, which is novel to the best of my knowledge.\n+ [Quality] Experiments on several Mujoco tasks demonstrate compelling performance. The authors additionally report results on the Franka kitchen environment.", "weaknesses": "-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our work and are excited about the positive feedback. Thanks a lot."}, {"review_id": "AByWReOwQu", "reviewer": "Reviewer_d8U7", "summary": "This paper studies offline imitation from observations, assuming a small amount of task-specific expert states and task-agnostic non-expert state-action pairs are available. The method is to learn a discriminator to identify expert states in the task-agnostic dataset and then apply weighted behavior cloning to imitate states. Empirical results show that the proposed method TAILO outperforms state-of-the-art methods (the family of DICE), particular for datasets with incomplete trajectories.", "questions": "1) About the role of c'(s) in Section 3.2, if we directly learn c(s) without the help of c'(s), how much it will hurt the performance of TAILO? \n2) In lines 150-152, intuitively, the value of hyper-parameter \\beta_1 should highly affect the final performance. Could you show an ablative study on this? Do you have any intuitive explanation why the same \\beta_1 works well across different experiments (as mentioned in line 194)?\n3) Could you please show experimental results on datasets exactly the same as the SMODICE paper? On these modified datasets, the performance of SMODICE looks poor. It either always keeps a low score or suddenly crashes. I'm wondering about the performance of TAILO on the standard benchmark datasets. It is great to clarify, given a specific dataset, how should we choose between SMODICE and TAILO? Is TAILO always the better choice?", "limitations": "This work is based on the assumption that there exist expert trajectories/segments for the task of interest in the task-agnostic dataset. I'm concerned about the reliability or generalizability of this assumption. In real-world problems, when collecting demonstration dataset, there seems a small chance that this assumption holds.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The idea behind this approach is intuitive, well-motivated, and conceptually simple.\n2) Empirically, TAILO performs well in learning from datasets with incomplete trajectories, significantly better than state-of-the-art approaches.\n3) The empirical evaluation is extensive. Considering different ways and hyper-parameters to modify datasets, the authors show good performance across all these datasets. It is impressive that the same set of hyper-parameters for TAILO performs well in all these experiments.\n", "weaknesses": "1) Compared with DICE methods (and the previous state-of-the-art), the proposed method lacks a theoretical foundation. The objective functions in equations 4,5,6 look mathematically complex, but we do not have a theory to support them.\n2) The ablation study is missing. Thus, the importance of each component in TAILO is unclear. See the section of Questions for more details.\n3) In experiments, the datasets are mostly manipulated, different from the original dataset used in previous work. This seems not standard for benchmarking and comparing approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for valuable feedback.\n\n### Q1: Lack of theoretical foundation, especially for Eq. 4, 5 and 6.\n\n**1. Eq. 4 and 5 do not lack theoretical foundations.** The objective functions in Eq. 4 and 5 come from established Positive-Unlabeled (PU) learning works [24]. Eq. 4 is a binary classification objective that estimates the loss from the negative data using a mixed, unlabeled data distribution, with a max operator that limits biased estimates; Eq. 5 is a selection between Eq. 4 and a standard binary classification loss. See **Sec. A** in the appendix (starting from line 509) for a detailed derivation of Eq. 4 and 5.  \n\n**2. Eq. 6 is immediately intuitive.** Eq. 6 states that the weight for Behavior Cloning (BC) is determined by the discounted future return in the trajectory from the current state; the exp function serves as a thresholding method to emphasize the importance of trajectories with high return.\n\n**3. TAILO is based on intuitive improvements of some carefully studied DICE methods and it is effective.** The idea of TAILO is that we want to follow the reward function design used in SMODICE and LobsDICE, but we do not want their non-robust weight update (see **Sec. C** for the reasons of non-robustness). Instead, we find that a weight as simple as thresholded future reward along the trajectory works better. Furthermore, we want to improve the classifier training of SMODICE and LobsDICE. As some examples are unlabeled, PU learning is introduced.\n\n### Q2: Ablation studies are missing.\n\nAll ablations mentioned in the question section were already provided in the appendix. \n\n**1. The role of c’(s).** The result of directly learning c(s) without the help of c’(s) is the “ours-V2” variant in **Sec. F.6** (see Tab. 4 in the appendix for the meaning of each variant). As Fig. 23 suggests, there is a significant gain in the Antmaze environment and a marginal gain in the kitchen environment with c’(s).\n\n**2. The effect of $\\beta_1$.** The ablation study is in **Sec. F.4.2**. According to Fig. 14, The reward will only decrease with extreme selection of $\\beta_1$. Intuitively, there are two reasons why the same $\\beta_1$ works well for different experiments: 1) the ratio of expert trajectories in our task-agnostic dataset is small (<5%), thus there is little expert data erroneously classified as safe negatives; 2) the Lipschitz-smoothed discriminator yields a good classification margin, and thus the 60%+ trajectories with lower average reward represents all non-expert data well.\n\n**3. Experimental results on datasets identical to those in the SMODICE paper.** The result is in **Sec. F.5.** As Fig. 22 suggests, our method is still better. Note SMODICE works well in their paper because they report the better result among using KL and chi-squared divergence (and in Sec. E.2. of the SMODICE appendix one can see the divergence chosen is crucial), while we report them separately. On all the datasets that SMODICE was tested on, TAILO improves, as TAILO is at least comparable to the better variant of SMODICE, while removing the need of selecting a divergence.   \n\n### Q3. Using manipulated dataset from prior work is not standard for benchmarking.\n\n**1. We also improve on standard benchmarks.** As stated in Q2 point 3, our method also improves on standard benchmarks. We also compare to offline RL methods on standard D4RL datasets (Q2 of reviewer iGFb), where the other methods have extra access to ground-truth reward labels. Nonetheless, the proposed approach performs well.\n\n**2. The original benchmark is close to being solved and cannot reflect the challenges discussed in our work, e.g., incomplete trajectories.** Thus, we construct new benchmarks to show that our method has even larger benefits against those challenges, and We think that researchers should go beyond the standard benchmarks.\n\n**3. Our comparison is fair as our testbed covers that of SMODICE’s.** SMODICE tests standard imitation from observation (Sec. F.5 in our work), imitation from examples (Sec. 4.4), and learning from mismatched dynamics (Sec. 4.5). We test TAILO on **every environment in every setting with exactly the same condition** and find consistent, significant gains. We further prove that our method is robust to incomplete trajectories (Sec. 4.2, 4.3) and task-agnostic datasets with fewer expert data (Sec. 4.1).  \n\n**4. Our modification for demonstration of advantages is minimal.** Except for mismatched dynamics where we strictly follow SMODICE, we do not make any modification to the environment itself; we only modify the dataset.\n\n### Q4. The assumption that the task-agnostic dataset contains expert segments does not hold in real-world problems.\n\n**1. The assumption is common in prior works.** This assumption is the basis of prior works in the skill-based learning community, e.g., SPiRL [12], SKiLD [13] and FIST [14], where they assume that there are many action sequences from the task-agnostic data that can be utilized for the task of interest. Practically, the benchmarks of many recent works in offline imitation learning satisfy the property in the assumption (note mujoco medium and medium-replay data also contain expert trajectories, as shown in Fig. 4 of OTR [10] and Fig. 4 of decision transformer [6]), such as SMODICE [1], LobsDICE [2], OTR [10], ReCOIL [9], MAHALO [15], and decision transformer [6].  \n\n**2. The assumption has real-world applications.** For example, in the robotics community, the robot often needs to utilize overlapping skills, i.e., trajectory segments observed in other tasks, such as moving the robotic arm to a particular place and grabbing the item, to complete the current task. This is demonstrated in our kitchen environment, where the robotic arm needs to combine different skills (e.g. opening the cabinet, moving the kettle) to complete a particular procedure of interest.  \n\n### References \nSee global response.\n"}, {"review_id": "0FgHLodxN2", "reviewer": "Reviewer_6nYj", "summary": "This paper provides an offline imitation learning method upon a specific problem setting where task-specific expert states (observations) are restrictively available and task-agnostic non-expert state-actions are supplementarily available. \n\nIn this problem setting, the authors follow the line of works using DICE-based imitation learning methods, and present TAILO, Trajectory-Aware Imitation Learning from Observations. Specifically, TAILO employs the 2-step positive-unlabeled learning scheme to have the state discriminator, and then it uses rewards calculated by the discriminator as the weight for weighted behavior cloning. \n\nThis procedure in TAILO hinges on the assumption that in the pool of task-agnostic data, there exist trajectories or long segments that are closely aligned to the optimal expert trajectories for the specific target task. \n\nThe authors focus on different use cases of offline imitation learning on incomplete trajectories and show the benefits of TAILO in such context for those cases through experiments. \n", "questions": "Could the authors compare the recent work [1] in offline imitation learning?\n[1] Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations (ICML 2022)\n\nCould the authors introduce some specific real-world application scenarios for the problem settings of incomplete trajectories? \n\nIs TAILO the first that applies PU in the context of offline imitation learning? Could the authors further clarify the novelty of their work?\n\n", "limitations": "The limitation is described in the conclusion section. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper contributes to the area of offline imitation learning, particularly in scenarios where offline data conditions exhibit variability due to incomplete trajectories. The robustness of the proposed method, Trajectory-Aware Imitation Learning from Observations (TAILO), is thoroughly demonstrated in different data conditions through the experiments detailed in Section 4.\n\nThese experimental conditions encompass scenarios with incomplete expert trajectories, incomplete task-agnostic trajectories, observations, and examples. The tests conducted appear to be comprehensive and the obtained results align well with the authors' focus, thereby solidifying the consistency and potential effectiveness of TAILO.\n", "weaknesses": "TAILO, as outlined in the paper, comprises two primary techniques, each of which is described in Section 3.2 and 3.3. One shortcoming of this work, however, is the absence of ablation studies that evaluate these proposed techniques separately.\n\nTo clarify the effectiveness of the Positive-Unlabeled (PU) learning method, alternative PU algorithms could be implemented and compared with the proposed technique. It would also be instructive to remove some losses as detailed in Equations (4) and (5) and assess the performance impact.\n\nIn a similar vein, various weighting strategies could be trialed in the context of behavior cloning, including cases where no weight is applied at all. Carrying out these ablation studies could provide a clearer understanding of the individual effectiveness and contributions of each method within the TAILO framework.\n \nMinor errors:\n- The legend of Figure 5 could be relocated without overlap.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for valuable feedback.\n\n### Q1. The absence of ablation studies in Sec. 3.2 and 3.3. \n\nPart of the requested ablation studies are in **Sec. F.6.** We test the remaining ablations and summarize here. \n\n**1. The ablation of Sec. 3.2.** In Sec. F.6, we compare ours and ours-V2, where we study the effect of using the ORIL-style Positive-Unlabeled (PU) learning technique instead of ours. As Fig. 23 and 24 suggest, there are significant differences in the Antmaze and halfcheetah_mismatch environment and marginal differences in the kitchen environment. We additionally test using no PU learning at all, which fails on halfcheetah_mismatch and hopper_head50_tail50 environments (see pdf in the global response for reward curves).\n\n**2. The ablation of Sec. 3.3.** In Sec. F.6, we compare ours and ORIL-logR-V2, and ours and ours-V1. We find that the non-parametric retrieval of weights for Behavior Cloning (BC) is much better than the RL process introduced by ORIL, while ORIL is non-robust and often diverges. We also find that the log design of $R(s)=\\log\\frac{c(s)}{1-c(s)}$ is much better than the linear design $10c(s)$, as the former further emphasizes the reward given by states that are clearly classified as expert states.    \n\n\n### Q2. Alternative Positive-Unlabeled (PU) learning algorithms and removal of some losses in Eq. 4 and 5.\n\n**The requested ablation studies can be found in point 1 in Q1.** As stated in Q1, the result shows that our design of PU learning exhibits a significant improvement on certain environments like halfcheetah_mismatch and hopper_head50_tail50.\n\n### Q3. Ablations on BC weighting strategies, including unweighted strategy. \nWe test three different weighting strategies in the main paper and the appendix. We additionally test another two weighting strategies and summarize here (see pdf in the global response for reward curves).\n\n1. We tested SMODICE [1], LobsDICE [2], ORIL [3] and ReCOIL [9] (**Sec. F.2**) as representatives of value-based weights (the dual variable of DICE methods can be seen as an equivalent of the value function).\n\n2. We use plain, unweighted behavior cloning as a standard baseline in all experiments.\n\n3. In **Sec. F.6** in the appendix, we test Ours-V1 to see whether the design of $R(s)=\\log\\frac{c(s)}{1-c(s)}$ instead of being proportional to the discriminator output is reasonable.\n \n4. We additionally test OTR [10] as the representative of Wasserstein-based weights, which does not work well in case of small portions of expert data in the task-agnostic data.\n\n5. We additionally test MARWIL [7] in the global response as the representative of advantage-based weights. It performs similarly to plain BC. This is consistent with prior work [8].\n\n**We found TAILO to significantly outperform all five strategies.**\n\n### Q4. Minor Errors on the legend of Fig. 5.\n\nThanks for the advice. As this year’s NeurIPS does not allow modification of the original draft, we will modify the figure in the camera-ready revision.\n\n### Q5. Comparison to DWBC [16]\n\n**1. DWBC is not applicable to offline Learning from Observations (LfO).** In DWBC, the input of the discriminator takes state, action and $\\log\\pi(a|s)$ from the task-specific and task-agnostic data as input, the latter two of which are not computable in LfO as expert actions aren’t available. Further, DWBC uses three terms in its policy loss (see Eq. 5 of DWBC), but we are unable to calculate the first two terms due to the lack of expert actions in LfO data. The adaptation of DWBC to offline LfO is beyond the scope of this work. \n\n**2. Even with extra access to expert action, DWBC is still unable to learn well from task-agnostic data with a very small portion of expert trajectories.** See pdf in the global response for reward curves.\n\n\n### Q6. Real-World Applications for Incomplete Trajectories\n\nThere are at least two possible applications for incomplete trajectories:\n\n**1. Learning from corrupted task-agnostic data.** In real-world scenarios, task-agnostic data are often accumulated from many different experiences, possibly with recording devices turned on and off in the middle, or even from the wild with unfamiliar sources and different formats (in which case alignment is required [17]). Therefore, it is very likely that some of the data are corrupted with a few steps missing in the middle of a long trajectory.\n\n**2. Learning from expert key frames and goals.** In robotic tasks such as navigation, it is common that a robot needs to move to another point with a few waypoints or the goal location as clues [18]; in physical simulation, we need to generate the desired agent behavior [19] by a few frames designated by the artists, as manually determining the position of every frame is prohibitively labor-intensive. In both cases, the task-specific dataset is incomplete with only examples or a few key frames.\n\n### Q7. The novelty of using Positive-Unlabeled (PU) learning.\n\n1. TAILO is not the first work to apply PU learning in offline Imitation Learning (IL); there is one prior work, ORIL [3], that also applies PU learning in offline IL. However, the two methods are significantly different; a thorough ablation in **Sec. F.6** shows that each difference between TAILO and ORIL leads to our performance gains.\n\n2. The novelty of TAILO can be summarized as follows:\n\na) a novel offline imitation from observation method that works well on a variety of problems;\n\nb) a simple and non-parametric way of determining weights for weighted behavior cloning;\n\nc) a novel way of acquiring a discriminator-based reward, inspired by positive-unlabeled learning designs from prior work, which is empirically shown to be effective.\n\nd) both theoretical (**Sec. C**) and empirical analysis on the shortcomings of state-of-the-art SMODICE and LobsDICE.\n\n### References\nSee global response.\n"}, {"review_id": "j0Z5oMRlTq", "reviewer": "Reviewer_Re2g", "summary": "This paper introduces Trajectory-Aware Imitation Learning from Observations (TAILO) for offline imitation learning. TAILO tackles the problem of learning from incomplete trajectories, where other state-of-the-art (SOTA) methods fail. Specifically, TAILO proposes a simple yet effective solution. It first learns to recover the task reward function by distinguishing the expert demonstrations and the sub-optimal data. In addition, TAILO treats the non-expert data as an unlabeled dataset, and relabelles the reward of sub-optimal transitions to provide more training data for the policy. Next, TAILO performs policy learning by weighted behavior cloning with accumulative return along a successful trajectory segment. In the experiments, TAILO successfully outperforms a series of baselines on offline imitation learning with incomplete trajectories.", "questions": "- In Sect. 3.2, the authors propose to learn two discriminators to label the non-expert trajectories and learn the reward function in an inverse reinforcement learning manner. However, it is common that solving a min-max game is naturally unstable and hard to optimize. Coupling two min-max objectives would even make it worse in my opinion. Are there any tricks to improve the stability? What are the failure cases of such an optimization scheme?\n- In Eqn. 6, the authors propose to weight the log-likelihood by $\\sum\\limits_{j=0} \\gamma^j \\exp(\\alpha R(s_{i+j}))$. This seems a bit odd to me. If we treat $R(s)$ as the exact reward function and solve the policy improvement by RL as inference, we will end up with exactly the AWR [1] objective, which is $\\exp(Q(s, a) - V(s)) = \\exp(\\sum\\limits_{j=0}\\gamma^j R(s_{i+j}) - V(s_i))$ for the weight. Thus, I’m wondering if there is a detailed derivation of this objective in Eqn. 6?\n- In Fig. 2, the results look quite interesting and promising. One thing I’m curious about is that SMODICE actually performs worse in some cases with less data removed. For example, SMODICE completely fails on Walker2d_1/20, 1/10, 1/5, but achieves a fairly strong performance on Walker2d_1/3. Could you provide some additional explanations on this?\n- The experiments mainly focus on continuous control tasks, where the agent has repeating cyclic patterns for its actions. With this in mind, it is a bit unclear to me why incomplete trajectories would be an issue, if the dropped trajectory segments are repeating the remaining ones. In this case, does the performance gain come from the use of unlabeled data or the better Q-value approximation?\n\nReferences:\n\n[1] Peng, X. B., Kumar, A., Zhang, G., & Levine, S. (2019). Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177.\n", "limitations": "As discussed above, this paper has certain limitations regarding its presentation, novelty, the mathematical correctness, and its novelty. I do appreciate the authors’ efforts on the extensive experiments and detailed appendix. However, I would think the paper is not ready for publication yet and major revision is needed for it to be published in another venue.\n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper studies an interesting problem with potential real-world applications.\n- The empirical performance looks promising.\n", "weaknesses": "There are several main weaknesses of the paper.\n\nFirstly, the presentation of the paper still has room for improvement. Specifically, I would suggest the authors double check the literature and be careful when making statements. There are several important but unfortunately erroneous statements in the paper. For example, the very first sentence in the abstract:\n\n> Offline imitation from observations aims to solve MDPs where only task-specific expert states and task-agnostic non-expert state-action pairs are available.\n\nThis statement is inaccurate as by its definition, offline imitation learning discusses the case where a policy is learned with only off-policy data, excluding the on-policy environment interactions. Consider the standard behavior cloning, for example. The use of non-expert transitions is just one of the solutions to assist policy learning, as in ValueDICE [1], DemoDICE [2], and etc.\nAlso, in line 113, the authors state that there are three main steps for DICE methods:\n\n> DICE methods consist of three parts: reward generation, optimization of the 114 value function V(s), and weighted behavior cloning.\n\nThis is inaccurate as well. For example, ValueDICE [1] directly learns the value function, rather than the rewards.\nBesides the presentation issues, the contribution of the paper is relatively minor. The use of cumulative return / advantage function is standard in reinforcement learning literature. Mathematically, only by maximizing the cumulative return would give a correct policy improvement step. If we consider the specific form of weighted behavior cloning, or more precisely, advantage weighted regression, we can still find many existing works with similar structure [3, 4, 5]. \n\nThere are some potential issues with the mathematical correctness and experiment details, which I will reserve for the questions.\n\nReferences:\n\n[1] Kostrikov, I., Nachum, O., & Tompson, J. (2020). Imitation learning via off-policy distribution matching. In International Conference on Learning Representations.\n\n[2] Kim, G. H., Seo, S., Lee, J., Jeon, W., Hwang, H., Yang, H., & Kim, K. E. (2022, October). Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations.\n\n[3] Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., & Riedmiller, M. (2018). Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920.\n\n[4] Peng, X. B., Kumar, A., Zhang, G., & Levine, S. (2019). Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177.\n\n[5] Wang, Z., Novikov, A., Zolna, K., Merel, J. S., Springenberg, J. T., Reed, S. E., ... & de Freitas, N. (2020). Critic regularized regression. Advances in Neural Information Processing Systems, 33, 7768-7778.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for valuable feedback.\n\n### Q1. Inaccurate statements.\n\n- Offline imitation from observations aims to solve MDPs where only task-specific expert states and task-agnostic non-expert state-action pairs are available.\n\nThis is correct. Note, we are discussing offline **imitation (Learning) from Observations (LfO)**, not **Imitation Learning (IL), i.e., no expert actions.** ValueDICE [20], DemoDICE [21] and Behavior Cloning (BC) all require expert actions missing in LfO. With only expert states and no interactions, the **Task-Agnostic (TA)** dataset is the only source for MDP dynamics.\n\n- (As mentioned in Sec. 2,) DICE methods consist of three parts: reward generation, optimization of the value function V(s), and weighted behavior cloning.\n\n**1. This sentence needs to be put in the paper’s context.** As we write right before, “As mentioned in Sec. 2”, the “DICE methods” here refers to those discussed in Sec. 2. We only mention SMODICE [1] and LobsDICE [2] in Sec. 2, and they both satisfy the statement. \n\n**2. ValueDICE requires expert actions; it is not for LfO discussed in the paper.** One can adapt ValueDICE to the LfO setting as shown by OPOLO [22]. But such a baseline is improved upon by OPOLO, which is in turn outperformed by our baseline LobsDICE.\n\n3. To prevent confusion, we will change the sentence to “the DICE methods for offline LfO discussed above”. However, we think the message within the context has no factual error.\n\n### Q2. Limited contribution beyond Advantage Weighted Regression (AWR) style [23] methods.\n\n**1. Our method is not a variant of AWR.** Our goal is to learn more from expert trajectories in TA datasets. We adopt the reward from SMODICE / LobsDICE and improve  with Positive-Unlabeled (PU) learning. We adopt the pessimism principle in offline RL by giving small weights to non-expert data, and use the exp function for thresholding. We sum over future returns, as the policy does not depend on the past in an MDP. The discount factor balances expert experience propagation (line 56-58) and avoidance of non-expert segments in the middle. AWR is totally different: it iteratively maximizes the (approximated) advantage from policy in the last iteration with divergence constraint.\n\n**2. The AWR methods listed in the review learns a value function, which we try to avoid.** We verify empirically that the value function is hard to obtain in our settings, e.g. missing steps in TA datasets. Also, **consistent with prior work [8], MARWIL [7] (single-iteration AWR) struggles in our settings.** See pdf in the global response for reward curves.\n\n3. **We provide theoretical (Sec. C) and empirical evidence on the downsides of SOTA methods, SMODICE and LobsDICE,** which is another novelty.\n\n### Q3. Stability issue of discriminator learning with coupling min-max objectives. Tricks?\n\n**1. The discriminator training is not a min-max game.** Only one moving part (c(s) or c’(s)) exists in Eq. 4 and 5 respectively. The max operator inside the min objective is a clipping technique for debiasing (see Sec. A for details). \n\n2. Lipschitz regularizer is used for better stability (see Sec. F.4.7).\n\n3. Optimization fails with extreme hyperparams or no Lipschitz regularizer. (see Sec. F.4.2, F.4.3 and F.4.7).\n\n### Q4. Missing derivations for Eq. 6.\n\nAs stated in Q2, our method is not a variant of AWR. Thus, Eq. 6 is immediately intuitive: the weight for BC is determined via a discounted sum of future returns; the exp function serves as a thresholding method upweighting expert data, as opposed to AWR’s closed-form solution.\n\n### Q5.  Abnormal SMODICE-KL behavior in ablation.\n\n**1. Collapses of SMODICE-KL come from value function divergence.** As stated in Sec. C.2.1 in the appendix, SMODICE-KL diverges with missing data in the task-agnostic trajectories. When this happens (e.g., halfcheetah_1/5), SMODICE-KL collapses; otherwise, it performs well (see pdf in the global response for visualization). \n\n**2. The smoothing effect of the Neural Network (NN) mitigates the divergence, but the strength of the effect varies.** As the data distribution differs, the smoothing effect varies across datasets. Thus, the time it takes for divergence differs.\n\n**3. With more and uniform updates, the abnormality caused by NN smoothing disappears.** As shown in Fig. 19, with larger batch size, the updates on each data point are more frequent and uniform. Thus, the trend that increased noise leads to worse performance on SMODICE-KL (thin blue line) is obvious. \n\n### Q6. Is the performance gain coming from the use of unlabeled data or better Q-estimation, as the environment has cyclic patterns?\n\n**1. Our method performs well in non-cyclic environments.** Kitchen (Sec. 4.3, 4.4), antmaze (Sec. 4.3-4.5) and pointmaze (Sec. 4.4) are non-cyclic environments; our method works well on all of them. \n\n**2. Cyclic patterns are more beneficial to baselines.** With cyclic patterns, the states are closer to each other, and NN smoothes out the diverging terms in SMODICE / LobsDICE more easily. In non-cyclic environments, our method gets better R(s) as the expert trajectories lead to a different direction from non-expert ones in the state space, but the divergence terms in SMODICE / LobsDICE remain. \n\n3. We are unsure about what the reviewer refers to with “the use of unlabeled data”.\n\n**a) If it is the TA dataset, then it is always crucial.** As mentioned in Q1 point 1, TA dataset is the only source for MDP dynamics in offline LfO.\n\n**b) If it is PU learning that leverages the unlabeled essence of the TA dataset, then there is indeed a performance gain. In Sec. F.6**, our PU learning is much better than ORIL’s (and no PU; see pdf in the global response) on environments like halfcheetah_mismatch.\n\n**4. Better Q-estimation is important. Sec. F.6** shows that ORIL with our reward still works badly. In fact, our motivation is to avoid learning it and choose a non-parametric but robust approach.\n\n### References \nSee global response.\n"}]

Author rebuttal:
We genuinely thank the reviewers for their valuable opinions and advice. We are delighted to see that the reviewers have carefully evaluated our work, given many valuable feedbacks, and highlight that 

1) we study the DICE family of algorithms in offline LfO, pointing out that they suffer from missing data due to inaccurate value estimation or sparsity of observations which causes undesired monotonicity (cited from reviewer iGFb);

2) the solution proposed in our paper is simple and effective, and

3) we have extensive empirical evaluations and a detailed appendix, which thoroughly demonstrate that our method improves upon baselines in many different scenarios.  

We respond to questions in the individual replies. We also include a pdf with additional figures. Finally, we use the following references in our replies:

[1]  Y. J. Ma et al. Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching. In ICML, 2022.

[2] G. H. Kim. et al. LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation. In NeurIPS, 2022.

[3] K. Zolna et al. Offline Learning from Demonstrations and Unlabeled Experience. In Offline Reinforcement Learning Workshop at NeurIPS, 2020.

[4] R. Kidambi et al. MOReL: Model-based Offline Reinforcement Learning. In NeurIPS, 2020.

[5] T. Yu et al. MOPO: Model-based Offline Policy Optimization. In NeurIPS, 2020.

[6] L. Chen et al. Decision Transformer: Reinforcement Learning via Sequence Modeling. In ICML, 2022.

[7] Q. Wang et al. Exponentially Weighted Imitation Learning for Batched Historical Data. In NeurIPS, 2018.

[8] X. Chen et al. BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning. In NeurIPS, 2020.

[9] H. S. Sikchi et al. Imitation from Arbitrary Experience: A Dual Unification of Reinforcement and Imitation Learning Methods. In ArXiv, 2023.

[10] Y. Luo et al. Optimal Transport for Offline Imitation Learning. In ICLR, 2023.

[11] S. Höfer et al. Sim2Real in Robotics and Automation: Applications and Challenges. in IEEE Transactions on Automation Science and Engineering, 2021.

[12] K. Pertsch et al. Accelerating Reinforcement Learning with Learned Skill Priors. In CoRL, 2020.

[13] K. Pertsch et al. Demonstration-guided reinforcement learning with learned skills. In CoRL, 2021.

[14] K. Hakhamaneshi et al. Hierarchical few-shot imitation with skill transition models. In ICLR, 2022.

[15] A. Li et al. MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations. In ICML, 2023.

[16] H. Xu et al. Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations. In ICML, 2022.

[17] M. Chang et al. Semantic Visual Navigation by Watching YouTube Videos. In NeurIPS, 2020.

[18] D. S. Chaplot et al. Neural Topological SLAM for Visual Navigation. In CVPR, 2020.

[19] X. Peng et al. DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills. In ACM Transactions on Graphics, 2018.

[20] I. Kostrikov et al. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations. In ICLR, 2020.

[21] G. H. Kim et al. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In ICLR, 2022.

[22] Z. Zhu et al. Off-Policy Imitation Learning from Observations. In NeurIPS, 2020.

[23] X. B. Peng et al. Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. In ArXiv, 2019.

[24] R. Kiryo et al. Positive-unlabeled learning with non-negative risk estimator. In NIPS, 2017.


Meta-review: The submission here identifies an important shortcoming of the DICE family of methods applied to imitation from observation and proposes an simple but interesting method for addressing this shortcoming that should be of interest to the NeurIPS community. The authors have engaged in a productive conversation with the reviewers during the discussion phase and the submission here should now be revised in order to provide the same additional results and context that was provided to the reviewers (eg, the additional ablation study, the clarity regarding motivation, and the positioning with respect to related work).
Decision: Accept (poster)