LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation
Authors: Geon-Hyeong Kim
Jongmin Lee
Youngsoo Jang
Hongseok Yang
Kee-Eung Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: offline learning from observation
learning from observation
imitation from observation
imperfect demonstration
imitation learning

Abstract: We consider the problem of learning from observation (LfO), in which the agent aims to mimic the expert's behavior from the state-only demonstrations by experts. We additionally assume that the agent cannot interact with the environment but has access to the action-labeled transition data collected by some agents with unknown qualities. This offline setting for LfO is appealing in many real-world scenarios where the ground-truth expert actions are inaccessible and the arbitrary environment interactions are costly or risky. In this paper, we present LobsDICE, an offline LfO algorithm that learns to imitate the expert policy via optimization in the space of stationary distributions. Our algorithm solves a single convex minimization problem, which minimizes the divergence between the two state-transition distributions induced by the expert and the agent policy. Through an extensive set of offline LfO tasks, we show that LobsDICE outperforms strong baseline methods.


LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation
Geon-Hyeong Kim 1.3, Jongmin Lee".", Youngsoo Jang!.
Hongseok Yang- 1,3 Kee-Eung Kim
School of Computing, KAIST
Kim Jaechul Graduate School of AI, KAIST
LG AI Research
4 University of California, Berkeley
Discrete Mathematics Group, Institute for Basic Science (IBS)
Abstract
We consider the problem of learning from observation (LIO), in which the agent aims to mimic the expert's behavior from the state-only demonstrations by experts. We additionally assume that the agent cannot interact with the environment but has access the action-labeled transition data collected by some agents with unknown qualities. This offline setting for LfO is appealing in many real-world scenarios where the ground-truth truth expert actions environment interactions are costly or risky. In this paper, we present LobsDICE, offline LfO algorithm that learns imitate the expert policy via optimization in the space of stationary distributions. Our algorithm solves a single convex minimization problem, which minimizes the divergence between the two state- transition distributions induced by the expert and the agent policy, Through an extensive set of offline LfO tasks, we show that LobsDICE outperforms strong baseline methods.
inaccessible and the arbitrary
Introduction
The ability to learn from experience is one of the core aspects of an intelligent agent. Reinforcement learning (RL) [36] provides framework to acquire such intelligent behavior autonomously through interactions with the environment while receiving reward feedback. However, the practical applicability of RI real-world domains has been limited for reasons. First, designing suitable reward function for complex tasks can be extremely difficult. The RL agent learns behaviors incentivized by the reward function rather than the ones intended, which can be nontrivial to specify in terms of reward. Second, the need for online interaction with the environment during the RL training loop has hindered its adoption in many real-world domains, where environment interactions are costly or risky.
33 34 circumvents the difficulty of reward design in RL by leveraging demonstrations given by experts, where the goal is to mimic the expert's 's behavior. However, the standard IL requires the expert demonstrations to contain not only the state information (e.g. robot joint angles) but also the precise action (c.g. robot joint torques) executed by the expert at each time step. This demand for explicit action labels is in contrast to the way human imitates (e.g. learning by watching videos) and precludes leveraging a massive amount data in which the action label is missing. Therefore, developing imitation learning algorithm that learn from observing the
Imitation learning (IL) 1
Equual contribution.
Work done while the authors were students at KAIST.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

experts' state-only trajectories is a promising direction for creating practical, autonomous intelligent
agent. Learning from observation (LfO) [2] 14 26
39 45 concerns this particular learning scenario
and has been gaining interest in recent years.
In this paper, are particularly interested in solving the L.O problem in an offline setting: Given the state-only demonstrations by experts and abundant state-action demonstrations by imperfect agents of arbitrary levels of optimality, we aim to find policy that follows the sequence of expert's states without further interaction with the environment. This problem setting is appealing in many practical situations where environment interactions are costly or risky. Yet, it cannot be straightforwardly tackled by existing approaches. Most of the existing LfO methods are on on-policy algorithms [25] 33| 39 [0). which not directly applicable to the offline setting. One of the few exceptions BCO 38 which performs behavior cloning on the inferred action via an inverse dynamics model. This method could be suboptimal unless the inverse dynamics disagreement is always zero, i.e. the underlying environment dynamics is deterministic and injective. Another one is OPOLO 45 an off-policy L.f algorithm, but relies on nested min-max optimization as well as distriibiitiiioo (OOD) action values, which can be unstable especially in the offline setting. IQ-Learn [8] solves Inverse RL and can be applied to the offline LIO by learning a state-only reward function. However, it suffers from numerical instability due to using OOD action values in the offline setting. Lastly, RCE [5], an example-base control algorithm, can in principle be applied to the offline LfO setting by providing the expert trajectories successful example states; however, its empirical performance is known to be limited without online data collection.
We present an offline Lf0 algorithm that minimizes the divergence between state-transition dis- tributions induced by the expert and the learned policy, without requiring an inverse dynamics model. Our algorithm, offline Learning from OBServation via stationary DIstribution Correction Estimation (LobsDICE), essentially optimizes in the space of state-action stationary distributions and state-transition stationary distributions, rather than in the space of policies. We show that our formulation can be reduced to a single convex minimization problem that can be solved efficiently in practice, unlike existing imitation learning (from observation) algorithms that rely on nested min-max optimization. In the experiments, we demonstrate that LobsDICE can successfully recover the state visitations by the expert, outperforming strong baseline methods.
2 Preliminaries
2.1 Markoy decision process
We consider an environment modeled as a Markov Decision Process (MDP), defined by M (S, A, T, R. po: 7) (36), where S is the set of states, A is the set of actions, T S x A + A(S) is the transition probability, R: S x A - IR is the reward function, po E A(S) is the distribution of the initial state, and 7 E (0, 1) is the discount factor. The policy S + A A) is mapping from states to distribution over actions. For the given policy π,, its state-action stationary distribution d' (ss a) and state-transition stationary d (s,, s') are defined as:
d' (= Σ Pr(s, = =sa,- = a), ds = (1 7)) Σ 1
Pr(s, = S, =s),
0=
where 80 Pa. Og ~ T(: and 81+1 ~ T(11s2, a:) for all timesteps t 2 0. For brevity, the bar notation (*) will be used to denote the distributions for s'),,), e.g. d= (s,
We assume offline LfO setting, where direct, online interactions with the environment are not allowed, and the policy should be optimized solely from a pre-collected dataset. We denote the dataset of state-only demonstrations collected by experts as DE = ((8, 8') 3 Ng and the dataset of state-
action demonstrations by some imperfect agents as D = {(ss a, s') ) That is, we do not have information about the actions taken by the expert, but instead, we have additional action-labeled transition data collected by some other agents with unknown levels of optimality. We denote the corresponding distributions of the datasets D and D' by d and d', respectively. For brevity, we will abuse notation d' to represent (s, ~ d', (s, a, s') d' and (s) ~ d' unless ambiguous.
2

---

2.2 Imitation learning and learning from observation
Imitation learning (IL) aims to mimic the expert policy from its state-action demonstrations. can be naturally formulated as a distribution matching problem that minimizes the divergence between state-action stationary distributions induced by the expert and the target policy [T] (3). For example, one can consider minimizing KI. Kl.-divergence 17]
min DEL (s, (s, = (( log
d' d'
2 (s.0a)
However, the standard IL requires action labels in the expert demonstrations, which may be too strong requirement for various practical situations. Learning from observation (LiO) relaxes the requirement on action labels, and aims to imitate the expert's behavior only from the state observations. Since the expert's action information is missing in the demonstrations, the distribution matching for state-action stationary distribution is no longer readily applicable. Therefore, LfO is reformulated as another distribution matching problem that minimizes the divergence between state-transition
stationary distributions induced by the expert and the target policy (39
40 45
min (d (s, s')ll' ss)s = )-,s")= log
('(ss') (z.0)))
(2)
Still, optimizing Eq. in a purely offline manner is challenging since naively estimating the
expectation would require the knowledge of T(-|s, a) for the OOD action TT() by the target
policy, which inaccessible in the offline LfO setting; see Section 9.8 in (45) for more discussions.
OPOLO [45], an off off-po LfO algorithm, mitigates this challenge by minimizing the following upper bound of the divergence DRL. (d'ddd(ss (s,e') (8,))):
(x,))))) log
(s.s')
(dL (s,
(3)
(*.e")
and applying DuaIDICE trick [29] to the RHS of 3 The upper bound gap is given by the inverse dynamics disagreement between the target policy and the imperfect demonstrator:
(KL(dd s')la' s())),
which usually gets larger as the stochasticity of the environment transition increases.
3 LobsDICE
We present offtine Learning from OBServation via stationary DIstribution Correction Estimation (LobsDICE), a principled offline LíO algorithm that further extends the recent progress made by the DIstribution Correction Estimation (DICH) methods for offline RL. LobsDICE essentially optimizes the stationary distributions of the target policy to match the expert's state visitations.
In the context of offline RL, the policy constraint principle (i.e. prevent deviating too much from the data support) is one of the common approaches to avoid severe performance degradation 7 12 18 22 30). In the same manner, we use KL divergence minimization between d= (2s(s, s') and de (s, (s,8')
with additional KL regularization on the deviation from d':
min DKL (d (s, s' (s, s')) + a (d (s, a)lldd
(5)
Here the hyperparameter > 0 balances between encouraging tate-trransiton matching and prevent- ing distribution shift from the distribution of imperfect demonstrations. All the proofs can be found in Appendix F
3.1 Lagrange dual formulation
The derivation of our algorithm starts by rewriting the (regularized) distribution matching problem (5) in terms of directly optimizing stationary distribution, rather than policy:
max d,d20
DKL(d((s dd)lld (s,8') aD (d(s, a))ld' (s.
(6)
s.t. Id(s'a = (1 (1- 7))o((s) + M (lsss))Tii (Ts) ss
(7)
8.0
Σ d(s = d(sss') Vs,s',
(8)
'See Appendix Afror discussion of why d= (s, s' ( matching is preferable d' (s)-maatnin
3

---

The Bellman flow constraint ensures d(ss, a) to be valid state-action stationary distribution of some policy, where d(ss a) can be interpreted as a normalized occupancy measure of (s, a). The marginalization constraint ( enforces d(s,, s') to be the state-transition stationary distribution that directly induced by d(s, a). In essence, the constrained optimization problem stationary distributions of an optimal policy, which best matches the state-transition trajectories of
(6 seeks the
the expert. Once we have computed the optimal solution (d" dd), its corresponding optimal policy
can also be obtained by normalizing d' for each state 32
π (a|s) Σn (=,a)
d' (s.o)
Note that DemoDICE 15 considers similar optimization problem to ours, but it deals with the offline IL (i.e. state-action stationary distribution matching), whereas we consider the offline LfO (e.g teate-rransttion stationary distribution matching). Accordingly, the optimization variable d(ss and the marginalization constraint (8) are newly added in our formulation.
We then consider the Lagrangian for the constrained optimization d(a,'') [=(0)))] - aEs log ('.0)))
16 8
min max. MA d,500
Ed log
+
Σ cuss s'ss,'')d(ss,s)) - d(s,, ())''ls
8,8"
(1)))((01) + E d(s, a)T(s"|ss - Ed(s', 8
a,a
where v(s) E R are the Lagrange multipliers for the Bellman flow constraints 7 and pls, s') e I. are the the Lagrange multipliers for the marginalization constraint (8). Note that the Lagrangian
( cannot be naively optimized an offline manner since requires evaluation of T('lss, (la a) for
(8, a) d, which is not accessible in the offline LfO setting. Therefore, we rearrange the terms in (
to eliminate the direct dependence on and d, introducing new optimization variables w and that denote stationary distribution correction ratios for (s, a) and (s, 8'), respectively:
Log
dia.e') (o.o*)
p
min max (1 Eso [M0)]] + E(s.erya (((,, log 7t (x,8))
dis,a')
+lgg (s:s))
d" (a,a')
H.M d,d20
=:)(e,e")
=:r(e,0')
+ E(ssa)nd
Eor Earl- + )w((')) - ((ss) -a log
d(s,a)
(10)
(8.00)
=:w((s.))
min max (1 B [u(s]]] + (ss."") [0(s, s (F(s,)) + ((ss log i6 (s, s')]] H.M 10,102
+ (s.aaaad' [wrss a) (enm a log wn(s, a))] =: L(w, W. v).
(11)
Similar to the assumption of full coverage which is fairly standard across broad set of recent offline RL approaches [99 22 27), we make a milder assumption that d' (s, > 0 whenever d (s,8') > O.
This assumption is necessary to recover the expert's 's behavior successfully. We introduce the log ratio
d (s.e') in (1O) to take an expectation under d' (instead of d which is assumed to have a broader support than d This log ratio r(s, 8') can be easily estimated using pretrained discriminator for two datasets D and D., which will be explained in detail in the following section.
r(s, s') = log (=
In summary, LobsDICE aims to solve the minimax optimization,
min max C(w, W0, J4, v). R. w,
(12)
The optimal solution (w" w0 of (2) represents stationary distribution corrections of an optimal á= (a,x")
policy n": w² (s
d= (s,a) and Tn (s, d' (s,s5)
d' (s.a)
3.2 Log ratio estimation via a pretrained discriminator
an estimate of the log ratio r(s, = log (s.s') 3 (s.s') is required. The log ratio
To optimize
estimation is straightforward for tabular MDPs since we can use empirical distributions from the datasets to estimate ds, s') and d' (s, For continuous MDPs, we train discriminator c S x S + [0, 1] by solving the following maximization problem [9] 45]:
c² arg max (e. / (log c(s, s')] + E (e) [log(1 c(s, 8)))
(13)
c: S=$

---

(8,8)))
It is easy to show that the optimal discriminator satisfies c² (s, a (o,d") Thus, s') can be derived from the optimal discriminator c* as,
r(s, s') = 1 log (5T6,67)) 1).
(14)
3.3 Minimax to min: closed-form solution
Exploiting the strict convexity log z, we can derive a closed-form solution to the inner maximiza- tion for (w, ( in (IT).
Proposition 3.1. For any (1s, v), the closed-form solution to the inner maximization of i.e. (Wp p = arg max [(w, W. ui v), is given by:
==xpss")=exp(((s.. Wpp (s. exp ( (,, a) 1) and mp(s,s') exp(r(s, 8') + p(s,s') 1).
(15)
Based on the above result, we reduce the nested min-max optimization of [T to a single minimization by plugging the closed-form solution (W,,,, (y into L(w,, w, v1;
min f(wwer Wp U. v) = (1 - = [+(s]] + (ss (exp (r(s, + p(s,8') -1] HeN
+ (e,o exp (n, -))]
(16)
We can even show that [(wppr ue 1, a convex function of μ and v. Proposition 3.2. [(wp,r Wpr p1 v) is convex with respect μ and v.
In short, by operating in the space of stationary distributions, offline LfO can, in principle. be resolved by solving convex minimization problem. This is in contrast the existing LfO algorithms, which typically involve either an adversarial training that optimizes the policy and the discriminator [14] 39 40 or nested min-max optimization for the policy and the critic (45)
3.4 Policy extraction
So far, we have derived an algorithm that essentially solves the state-transition distribution matching problem via convex minimization. However, we obtain (n* (p²,,²) as the solution (6)). instead of the optimal policy x². The remaining problem is to extract the optimal policy from (μ²", -²)). The first step
is to see that we can obtain the state-action stationary distribution correction Wp--er (s, a) = d= (s.o) ('(a)))
of the optimal policy from the closed-fore solution (15). Among many possibilities extract the policy from the state-action distribution correction, we adopt weighted behavior cloning (WBC):
max EE (s.))>=='' [log क (a[s)] [ss.a))d'' [W "" (8, a) log [a]ss]
(17)
which aims to maximize the predicted probability of actions chosen by optimal policy π². This is done by BC on the offline dataset D' where each sample (s, a) is weighted by " (s, a). For tabular MDPs, we can formally show that WBC extracts an optimal policy x (Appendix(C)
3.5 Practical algorithm with sample-based approximation
In practice, we estimate C(w,, w. u1 v) in (T using samples from distribution d', Let ExED(S(x)]
াo ExED [x) be a Monte-Carlo estimate of [x)) where D {x:)111 We denote
each sample (s, (5,a,) s') in DJ as x for brevity.
min max E(w, w. p, v) = (1 E) [p(s))] pau w.
(18)
É EDED [iss, (').s')((6)) (r(s. s') + pls, s') log (ssss')) ) + w ((er,)((e a log wss a])]
where (s, == ((s, ) + 7p('') v(s) is a single-sample estimate of en a). Note that this basele objective function L(w,, w. n, v) can be estimated only from samples in the offline dataset D and is an unbiased estimator of L(w, w. v) as long as every sample in DJ was collected by interacting with the underlying MDP. To the best of our knowledge. this is the first
5

---

result to directly solve the state-transitinn distribution matching problem in fully offline manner. In contrast, OPOLO [45] relies on the (potentially loose) upper bound the objective in (3).
To reduce the minimax optimization to single minimization, we apply the non-parametric closed- form solution for each sample (s, s') in D
(x) = exp (1 (2e as, s') 1) and (x)x)) exp(r(s,s)) + u(s.s') - 1).
(19)
15). Plugging this result into (8) yields sample-based objective function for which is analogous minimization
min D) = -1-1) () DD ((())) + EXED oxp (r(s, s') + p(s, -1) PA1
(20)
exp (d - 1)]
Still, the variable μ RRSS is much higher dimensional than R So, works the main bottleneck for the overall optimization. Fortunately, we can further simplify (20) by eliminating its dependence on y via exploiting additional closed-form solution.
Proposition 3.3. For any v. the closed-form solution to the minimization 20 with respect μ. i.e. Hy = arg min,, C(u,, v),
po(s, = Ita
-
+ (o(()
(21)
Using the above result in (20) we obtain the following minimization problem:
min 2(0) = (1 - 7) E ((0)) + (1 + a B) 8
exp I+a (o(s,,,8) -)]
(22)
where (s, a, 8¹) = r(s,'') + yw(s') vss). The remaining issue is that optimizing 22 is
practical because exp(-) ) often causes numerical instability and gradient explosion. To address this, we use numerically-stabe
alternative of 22)
Proposition 3.4. Let L(T) be the function:
2(D) = (1 - ED (7(s)) + (1 +1+ a)) log Ezen exp ( (ita Ao(s, a, s')]]
(23)
Then, ming [(D) ming [(D) holds. Also, [(D) is conver with respect to D.
In order see why minimizing ((i) no longer suffers from numerical instability, note that the gradient V, log Ex-p [ex)())))) = Ezzpl Ep-p [22] V=h(z)) normalizes exp(-) by softmax and thus tames large numerical values. Finally, we can show the following connection between in and D*:
exp(h(x))
Proposition 3.5. Let V and V be the sets arg ming L(D) and arg min; L(D), respectively. Then,
V = (0* C CCI 10 V,C E IR} holds.
Proposition 3.5 implies that unnormalized stationary distribution corrections of optimal policy would be obtained from 7 V.
ug- D. (x: exp 1 ug (8, 7000(8))) - -1)
exp
( A (s, 5') -1)
(by 2T
x exp
1 1. (s, 1+a
W.- (1)
(by Proposition 3.5)
'In contrast to the results in the previous sections where 16 and are identical, Z( v) in 20 is
an upper bound of max... a Z(w, W, u, in (8) due to applying per-sample closed-fon solutions. While unbiasedness has been sacrificed at the cost of eliminating the nested optimization. it enables much stable optimization in practice. The upper bound gap vanishes when the transition dynamics are deterministic.

---

Policy extraction We must take caution when using ügg (z) since it is an unnormalized density ratio, i.c. Ex (gg (z)] * 1. Therefore, to extract a policy, we perform weighted BC using self-normalized importance sampling [31]:
max Zxent Wi- (1r) (a]s)
(24)
Exent Wig. (x)
which completes the derivation of the practical version of LobsDICE. To sum up, LobsDICE solves
7 arg ming [(z) of (23) via gradient descent, and extracts a policy via self-normalized weighted
BC of (2 Pseudocode for LobsDICE can be found in Appendix G.2
4 Related Work
Learning from observation (LfO) Recent approaches for LfO are mostly on-policy 25 35
39 40] algorithms and are not directly applicable to the offline LfO setting considered in this work. MobILE [4] is a model-based LfO algorithm, but encourages uncertainty for online exploration, which is not suitable for the offiie setting. BCO 38 uses inverse dynamics model (IDM) to infer the missing expert actions and performs BC on the generated expert's state-action dataset. In addition to common issues by vanilla BC, BCO is not guaranteed to recover the expert's 's behavior in general. OPOLO (45) is a principled oolcpo LfO algorithm, but it solves a nested optimization and requires evaluation OOD action values during training, which suffers from numerical instability in the offline setting. IQ-Learn [8] solves online and offline IL problems while avoiding adversarial training by learning a single Q-function. However, it also suffers from numerical instability in the offline setting by overestimating Q due to using OOD action values. RCE [5] aims to solve example-base based control tasks in which a collection of example success states is assumed to be provided instead of entire expert trajectories. While RCE can be applied to LfO in principle, it tends to stay in a few expert states that are easy reach in the offline setting, as discussed in Section C.2 of (5)
Stationary DIstribution Correction Estimation (DICE) DICE-family algorithms perform station-
ary distribution estimation, and many of them have been proposed for off-policy evaluation [ [29 42
44. Other lines of works consider reinforcement learning [22 23 30 offline policy selection 41
ValueDICE [17 and OPOLO 45) derive off-policy IL and LfO objectives using DICE. However, they suffer from numerical instability in the offline setting due to the nested min-max optimization and OOD action evaluation. DemoDICE [15 is an offline IL. algorithm that directly optimizes stationary distribution as ours and reduces to solving convex minimization. Yet, it requires expert action labels and is not directly applicable to LfO. Concurrent to our work, SMODICE (28) is an offline LfO method, aiming to match stationary state distributions by minimizing an objective similar to OPOLO 445 in (3). Thus also relies on potentially loose upper bound of the divergence. addition, matching the state distributions may not be sufficient to recover the expert's behavior, on which we provide detailed discussion Appendix A and Remark B.I
Experiments
In this section, we evaluate LobsDICE on both tabular and continuous MDPs. We use four baseline methods in tabular MDPs: BC on imperfect demonstrations, BCO [38], and OPOLO [45]. Addi- tionally, we designed strong baseline DemoDICEfO, which extends the state-of-the-art offline IL algorithm, DemoDICE [55. DemoDICEfO trains an inverse dynamics model, uses it to fill the missing actions in the expert demonstrations, and then runs DemoDICE using the approximate expert demonstrations and the imperfect demonstrations. For continuous control tasks, we use two additional baselines: IQ-Learn [8] and RCE (5).
5.1 Random MDPs
We first evaluate LobsDICE and baseline algorithms on randomly generated finite MDPs using a varying number of expert/imperfect trajectories and different degrees of stochasticity in the environ- ment. We follow the experimental protocol in previous offline RL. works 22 but with additional control on the stochasticity of transition probabilities. We conduct repeated experiments for 1K runs. For each run, (1) a random MDP is generated, (2) expert trajectories and imperfect trajectories are collected, and (3) each offline LfO algorithm is tested on the collected offline dataset. We evaluate
7

---

BC
BCO
DemoDIcEro
OPOLO
LobsDICE (ours)
Ng 10,3= 0.01
100, 0.01
No 1000, A= 0.01
N1 10000, 0.01
(4r)) (19))
10
10"
TVV
10"
10
10
10"
10
10
10
10"
10"
10"
10
10")
10"
10
10"
10"
10"
10
10
10"
10"
10"
10
10
10"
10"
10
10"
10'
10"
10"
10"
N2 10, 0= 0.1
N8
100, /0 0.1
Ng 1 1000, / 0.1
N2 10000, , 0.1
10
10
10
10
TV
10"
10
10
10"
10
10
10
10
10
10"
10
10
10"
10
10
10"
10
10
10"
10"
10"
10
10'
10"
10"
10"
10"
10'
10"
10"
10"
Ag
10, 9
Agg
100, 3=
1/g = 1000, /
Ag
10000,
9=
10
10
10"
10"
10"
10"
10"
10"
10"
10"
10
10
10
10"
10
10"
10
10"
10
10
10"
10"
10"
10"
10
10"
imperfect domonstrations
of imperfect demonstrations
# of imporfect demonstratons
imperfact demonstrations
Performance of tabular LobsDICE and baselines in randomly generated MDPs. The first row indicates near-deterministic dynamics and the last row indicates highly stochastic dynamics. As the level of stochasticity increases, baselines fall into suboptimal, even the number of state-only expert demonstrations and imperfect demonstrations increases, while LobsDICE goes to optimal. For each algorithm, we measure the performance using total variation between saatetrannsiion stationary distributions of expert and learned policy. We plot the mean and standard error of total variations (d (s, s'), d= (s, s')) over 1000 random seeds.
Figure I:
the performance of each algorithm by measuring the total variation distance between state-transition stationary distributions by the expert policy and the learned policy, i.c. Drv (d" (s, ( (s)ld') (s, s')). For the tabular MDP experiments, we adopt tabular methods but not function approximation. Our tabular LobsDICE optimizes (16) on the empirical MDP model constructed from the action-labeled dataset D while extracting the policy through (I7). The pseudocode for tabular LobsDICE can be found in Appendix G.I The detailed experimental setup such random MDP generation and offline
dataset generation can be found in Appendix L.
Results Figure D the results in random MDP experiments, where / is the hyperparameter that controls the stochasticity of the underlying MDP. The first row corresponds to the case when 3 3= 0.01 (nearly deterministic MDP). In this situation, the inverse dynamics disagreement will be close to zero, i.c DKL (dt) (a|s, s")ll (a|s, 5²)) z O for any two policies т and πT. Thus, the algorithms whose performance directly relies on the IDM's accuracy (i.e. BCO and DemoDICErO) even perform very well since it is very easy to learn a perfect inverse dynamics model in this scenario. OPOLO's 's upper bound gap (4) will also be close to zero, thus OPOLO directly minimizes the divergence of state-transitin transition distributions. As result, there is no performance gap among different algorithms, except for BC whose performance determined by the quality of imperfect demonstrations. Also, the performance of all algorithms (except for BC) improves as more data is given, which is natural.
The second row and the third row in Figure 1 presents the result when B = 0.1 (weakly stochastic MDP) and 8 = 1.0 (highly stochastic MDP) respectively. In the stochastic MDPs, the IDM trained by the imperfect demonstrations faces a challenge in predicting the expert's 's actions accurately (more challenging as 3 gets larger). As a consequence, BCO gets suboptimal and its suboptimality cannot be improved even if more data is given. DemoDICEfO performs better than BCO since it additionally considers the distributional shift by considering state distribution matching. However, it is still suboptimal due to its nature that directly depends on the quality of inferred action by the learned IDM. OPOLO does not rely on the learned IDM and outperforms both BCO and DemoDICEFO. Still, OPOLO can be inherently suboptimal due to its nature of optimizing the upper bound unless the underlying transition dynamics are deterministic and injective. This upper bound gap 44) is not controllable by the algorithm and implies that OPOLO can be suboptimal even given infinite
8

---

Expert
BC
BCO
RCE
10 Learn
DeMO
OPOLO
LabsDICE (ours)
Hopper (100,100.100)
Walker2d (100,100,100)
Ant 10000000000
HalfCheetah (400.100.100)
Norma Rocums
100
103
103
105-
0-
2.38
0.49
5.8M
5.08
23M
0.28
0.8N
0.0N
Hopper (100,500,500)
Walker2d (100,500,500)
Ant (100,500,500)
HalfCheetah (400,500,500
122
Normaltzed Roturns
100
100
103
102
00-
20-
120
Hopper (100,1080,1000)
120
Walker2d (100,1000,1000
122
Ant
122
HolfCheetah (400,1080,
Normazedd heturre
100-
100
100
100
00-
62-
62-
62-
Training Derrationa
Training Hurations
Training Iterations
Training Herations
Performance of LobsDICE and baseline algorithms on various MuJoCo control tasks. We build state-only expert demonstrations using 5 trajectories from expert-v2. For each task (X, Y, Z) we construct imperfect demonstrations using X, Y, and Z trajectories from expert-v2. medium-v2, and random-v2, respectively. We plot the mean and the standard errors (shaded area) of the normalized scores over five random seeds.
Figure 2:
amount of data with sufficient dataset coverage, which can be seen in the rightmost figures. Finally, tabular LobsDICE using (16) essentially solves the exact state-transition distribution matching problem (as a - 0). LobsDICE is the only offline LfO algorithm that can asymptotically recovers the expert's state demonstrations even though the underlying MDP is stochastie.
5.2 Continuous control tasks (Gym-MuuoCoo)
We present the empirical performance of LobsDICE and baselines MuJoCo 37 continuous control tasks using the OpenAl Gym [3] framework. We utilize the D4RL dataset [6] for offline LIO tasks in four MuJoCo environments: Hopper, Walker2d, HalfCheetah, and Ant. Implementation details for LobsDICE and baseline algorithms such as hyperparameters and evaluation metric are provided in Appendix 1.2
Task setup For each MuJoCo environments, employ expert-v2, medi um-v2, and random-v2 datasets
from datasets (6. Across all environments, we consider three tasks, each of which uses different imperfect demonstrations while sharing the same expert observations. First, construct the state- only expert demonstration D {(s, Ne using the first 5 trajectories expert-v2. Then, we use trajectories in expert-v2 -2, medium-v2s v2, and random-v2 v2 to construct imperfect demonstrations with different ratios. We denote the composition of imperfect demonstrations as (XX, Y, Z) in the title of each subplot in Figure which means that the imperfect dataset consists of X trajectories from expert-v2, trajectories from medium- v2. 2, and Z trajectories from random-v2.
Results Figure summarizes that the empirical results of LobsDICE and baselines on continuous control tasks. We first remark that LobsDICE (blue) significantly outperforms OPOLO (green) in all tasks across all domains, although both LobsDICE and OPOLO are DICE-based algorithms. The failure of OPOLO comes from its numerical instability due to its dependence on nested optimization and using OOD action values during training. contrast, LobsDICE solves a single minimization (23 while it does not involve any evaluation on OOD actions, thus it is optimized stably. IQ-Learn (purple) also suffers from numerical instability due to the usage of OOD action values during training, showing poor performance similar OPOLO. RCE (brown) tends to stay in a few states that are easy reach

---

Kee-Eung in the offline setting, rather than following the entire expert trajectories. Naive BC on imperfect demonstrations (black) is inherently suboptimal since it does not consider distribution maaribu with the expert's observation at all. While BCO (orange) exploits the expert's demonstrations with the inferred actions by the IDM, its policy learning done only on the very scarce expert dataset (i.e. 5 trajectories), which makes the algorithm perform not well. DemoDICEfO (red) exploits both expert demonstrations (where the missing actions are filled with the IDM) and the abundant imperfect demonstrations, but its performance is affected by the quality of the learned IDM. We empirically observe that the IDM error (on the true expert data) increases as the proportion of the non-expert data (i.e. medium-v2 and random-v2) increases, resulting in performance degradation of both BCO and DemoDICEfO. Finally, LobsDICE is the only algorithm that was able to fully recover the expert's 's performance regardless of the increase of non-expert data in the imperfect demonstrations, significantly outperforming baseline algorithms. This result highlights the effectiveness of our method that solves tate-transition stationary matching problem in principled manner. We provide additional experiments in Appendix H
6 Conclusion
We presented LobsDICE, an algorithm for offline learning from observations (LfO), which success- fully achieves oftc-of-thheart performance on various tabular and continuous tasks. We formulated the offline LiO as a state-transition stationary distribution matching problem, where the stationary distribution is optimized via convex minimization. Experimental results demonstrated that LobsDICE achieves promising performance in both tabular and continuous offline LfO tasks.
Acknowledgments and Disclosure of Funding
This work was supported by National Research Foundation (NRF) of Korea (NRF- 2019RIA2C1086644 Field-oriented Technology Development Project for Customs Administration through National Research Foundation (NRF) of Korea funded by the Ministry of Science & ICT and Korea Customs Service MEFF20221311A1099
Institute of Information & communica- tions Technology Plamning & Evaluation (IITP) grant funded by the Korea govermment (MSIT) (No.2020-0-00944, Foundations of Safe Reinforcement Learning and Its Applications to Natural Language Processing; No.2022-0-003 11, Development of Goal-Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects; f0.2009-00000073 Artificial Intelligence Graduate School Program (KAIST); No.2021-00 2021-0-02068, Artificial Intelligence Innova- tion Hub) and Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government (22ZS1 100,, Core Technology Research for Self-Improving Integrated Artificial Intelligence System). Hongseok Yang was supported by the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF-2018R SA888RRAAA11199999 and also by the Institute for Basic Science Kim was supported by KAIST-NAVER Hypercreative AI Center.
References
[1] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469-45) 2009.
[2] D. C. Bentivegna, C. G. Atkeson, and G. Cheng. Learning tasks from observation and practice. Robotics and Autonomous Systems, 47(2): 2004. Robot Learning from Demonstration.
[3] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint 1606.01440011 2016.
[4] B. Dai, O. Nachum, Y. Chow, L. Li, C. Szepesvari, and D. Schuurmans. CoinDICE: Off- policy confidence interval estimation. Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 9398 1011991111110011 2020.
[5] B. Eysenbach, S. Levine, and R. R. Salakhutdinov. Replacing rewards with examples: Example- based policy search via recursive classification. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
10

---

[6] J. Fu. A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4RL: Datasets for deep data- reinforcement learning, 2020.
[7] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning (ICML), pages 2052-2062. PMLR, 2019.
[8] D. Garg. S. Chakraborty, C. Cundy, J. Song, and S. Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems (NeurlPS), 34, 2021.
[9] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
[10] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. Improved training of Wasserstein GANs. arXiv preprint 0rXiv:1704.000266 2017.
[11] J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in Neural Informa- tion Processing Systems (NeurIPS), 2016.
[12] Y. Jin, Z. Yang, and Z. Wang. Is pessimism provably efficient for offline rI? In Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 5084-5096, 18-24 Jul 2021.
[13] L. Ke, S. Choudbury, M. Barnes, W. Sun, G. Lee, and S. Srinivasa. Imitation learning as f-divergence minimization. In International Workshop the Algorithmic Foundations of Robotics, pages 313-329. Springer, 2020.
[14] R. Kidambi, J. D. Chang, and W. Sun. MobILE: Model-based imitation learning from observa- tion alone. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. Rtm4rYnIIL.
[15] G.-H. Kim, S. Seo, J. Lee, W. Jeon, H. Hwang, H. Yang, and K.-E. Kim. DemoDICE: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations (ICLR), 2022.
[16] I. Kostrikov, K. K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson. Discriminator-actor: critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations (ICLR), 2019.
[17] I. Kostrikov, O. Nachum, and Tomppon. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations (ICLR), 2020.
[18] A. Kumar, Fu, M. Soh, G. Tucker, and S. Levine Stabilizing off-policy Q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[19] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[20] R. Laroche, P. Trichelair, and R. T. Des Combes. Safe policy improvement with baseline bootstrapping. In International Conference on Machine Learning (ICML), pages 3652-3661. PMLR, 2019.
[21] B. Lee, Lee, P. Vranex, D. Kim, and K.-E. Kim. Batch reinforcement learning with hyperpa- rameter gradients. In International Conference on Machine Learning (ICML), pages 5725-5735 PMLR, 2020.
[22] J. Lee, W. Jeon, B.-J Lee, J. Pineau, and K.-F. Kim. OptiDICE: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning (ICML), 2021.
11

---

[23] Lee, C. Paduraru, D. J. Mankowitz, N. Heess, D. Precup, K.-E. Kim, and A. Guez. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations, 2022. URL htttps://openrevier net/forum? net/Forum??dd=PAA55Mae 1d=PLA55mBeee
[24] Z. Li, Xu, Yu, and Z.-Q. Luo. Rethinking valuedice: Does it really improve performance? arXiv preprini arXiv:
2022.
[25] F. Liu, Z. Ling, T. Mu, and H. Su. State alignment-based imitation learning. In International Conference on Learning Representations (ICLR), 2019.
[26] Y. Liu, A. Gupta, P. Abbeel, and S. Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 1 1118-1125. IEEE, 2018.
[27] Y. Ma, D. Jayaraman, and O. Bastani. Conservative offiine distributional reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[28] Y. J. Ma, A. Shen, D. Jayaraman, and O. Bastani. Smodice: Versatile offline imitation learning via state occupancy matching. arXiv preprint arXiv:2202.02433. 2022.
[29] O. Nachum, Y. Chow, B. Dai, and L. Li. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[30] O. Nachum, B. Dai, I. Kostrikov, Y. Chow, L. Li, and D. Schuurmans. AlgaeDICE: Policy gradient from arbitrary experience. arXiv preprint arXiv: 191iv:192..020000 2.02074, 2019.
[31] A. B. Owen. Monte Carlo theory, methods and examples. 2013.
[32] M. L. Puterman, Markov decision processes: Discrete stochastic dynamic programming, 1994.
[33] S. Ross, G. Gordon, and J. A. Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pages 627-635, 2011.
[34] S. Schaal et al. Learning from demonstration. Advances in neural information processing systems (NeurIPS), pages 1040-1046, 1997.
[35] W. Sun, A. Vemula, B. Boots, and D. Bagnell. Provably efficient imitation learning from observation alone. In International conference on machine learning. pages 6036-6045 PMLR, 2019.
[36] R. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT Press, 1998.
[37] E. Todorov, T. Frez, and Y. Tassa. Mujoco: physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[38] F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. In International Joint Conference on Artificial Intelligence (IJCAI), 2018.
[39] F. Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. ICML Workshop on Imitation, Intent, and Interaction (I3), 2019.
[40] C. Yang, X. Ma. W. Huang, Sun, H. Liu, J. Huang, and C. Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[41] M. Yang, B. Dai, O. Nachum, G. Tucker, and D. Schuurmans. Offline policy selection under uncertainty, 2020.
[42] M. Yang, O. Nachum, B. Dai, L. Li, and D. Schuurmans. Off-policy evaluation via the regularized lagrangian. In Advances Neural Information Processing Systems (NeurIPS), 2020.
12

---

[43] R. Zhang, B. Dai, L. Li, and D. Schuurmans. GenDICE: Generalized offline estimation of sta- tionary values. In Proceedings of the 8th International Conference on Learning Representations (ICI.R), 2020.
[44] Zhang, B. Liu, and Whiteson. GradienDDICE: Rethinking generalized offline estimation of stationary values. In Proceedings of the 35th International Conference on Machine Learning (ICML), 2020.
[45] Z. Zhu, K. Lin, B. Dai, and J. Zhou. Off-policy imitation learning from observations. In Advances Neurul Information Processing Systems (NeurIPS), 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See abstract and introduction (Section []
(b) Did you describe the limitations of your work? [Yes] Due to space limit, we discuss it
in the supplementary material (Section L
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Due to
space limit, we discuss it in the supplementary material (Section N
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] In appropriate places.
(b) Did you include complete proofs of all thcoretical results? [Yes] Due to space limit, provide proofs in the supplementary material. See Section
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We submit the code supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We specify them in the supplementary material. See Section
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We report them in each figure. See Figure[andd] (d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] We specify them in the supplementary material. See Section
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite Random MDP protocols and D4RL dataset. See Section
(b) Did you mention the license of the assets? [Yes] D4RL dataset is licensed under the
Apache 2.0. See Section[N M
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] All the datasets we used are public.
(c) Did you discuss whether the data you are using/curating contains personally identifiable
information offensive content? [N/A] None of datasets contain identifiable contents.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] Not applicable.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] Not applicable.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [NA] Not applicable.
13


Reviews:
[{"review_id": "nLerHrGMyPU", "reviewer": "Reviewer_cB6F", "summary": "This paper deals with offline LfO with imperfect demonstrations. The main contribution of this paper includes solving a distribution matching problem without  requiring an inverse dynamics model. The reduction of the problem is simple. But the algorithm and the theory heavily borrow from DemoDICE[15]. and the contribution of this paper is incremental.", "questions": "Figure 1 shows that the performance gap between LobsDICE and other baselines becomes larger as the level of stochasticity increases. And there is no gap when the environment is nearly deterministic. \n\nHowever, the Mujoco environments in Figure 2 are deterministic within my knowledge. So how to explain the performance of LobsDICE in figure 2?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n  - The paper reduces the regular minimax problem in matching state distribution to a single convex minimization problem to make the proposed algorithms more stable than prior work.\n  - There are sufficient baselines in the experiments to support the performance of LobsDICE.\n\n- Weaknesses\n  - The algorithm and the theory heavily borrow from DemoDICE[15].", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zob7bR-B5QP", "reviewer": "Reviewer_zkgU", "summary": "The authors propose an algorithm for offline learning from _observations_, i.e. for learning from observed expert trajectories where only the states, not the actions can be observed. Additionally, similar to some prior work, the authors assume access to a large set of data from suboptimal policies, but containing actions.\n\nOverall, the paper follows the DemoDICE paper very closely, both in structure of the paper as well as in terms of the algorithm, but with the crucial difference than DemoDICE assumes access to expert actions.\n\nInterestingly, the optimization target for both algorithms ends up looking exactly alike:\n\nEquation (23) from LobsDice: \n\n$$\n\\widetilde{\\mathcal{L}}(\\widetilde{\\nu})=(1-\\gamma) \\hat{\\mathbf{E}}\\_{s\\_{0} \\in D_{0}}\\left[\\widetilde{\\nu}\\left(s_{0}\\right)\\right]+(1+\\alpha) \\log \\hat{\\mathbf{E}}\\_{x \\in D^{I}}\\left[\\exp \\left(\\frac{1}{1+\\alpha} \\widehat{A}_{\\widetilde{\\nu}}\\left(s, a, s^{\\prime}\\right)\\right)\\right]\n$$\n\nEquation (17) from DemoDice: \n\n$$\n\\widetilde{L}(\\nu ; r):=(1-\\gamma) \\mathbb{E}\\_{s \\sim p\\_{0}}[\\nu(s)]+(1+\\alpha) \\log \\mathbb{E}\\_{(s, a) \\sim d^{U}}\\left[\\exp \\left(\\frac{A\\_{\\nu}(s, a)}{1+\\alpha}\\right)\\right]\n$$\n\nwith, as far as I can tell, the only difference being whether the discriminator is learned on tuples $(s,s')$ or $(s,a)$.\n\nHowever, as far as I can tell, it is not obvious a-priori that the solutions will end up looking the same, hence making the derivations in the paper necessary. \n\nPerformance is evaluated against several baselines (BC, BCO, DemoDICE with inverse dynamics model, OPOLO) on both a tabular environment and several MuJoCo environments (Hopper, Walker2d, Ant and HalfCheetah), showing good performance.", "questions": "Please see my suggestions above regarding small changes to the presentation of the paper. \nI don't have any question and I think this is a solid paper which should be accepted.\nI'm not rating it stronger than \"accept\", due to the similarity (in approach and solution) to DemoDICE. ", "limitations": "N/A.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strenghts\n\n* I found the writing accessible and good to follow, as well as the mathematical derivation well explained. I do think the paper might be a bit harder to follow if one is not familiar with similar algorithms (e.g. the *DICE family), so a short recap of their their core ideas in the background section would be helpful. Similarly, a recap of DemoDICE would be helpful and make the close connection clearer.\n* The topic of offline IL is relevant, as is the extension to LfO\n* I liked the experimental section and found both the figures and discussion easy to follow as well as insightful.\n\n## Weaknesses\n\n* I think the presentation of the derivation of the algorithm could be slightly improved. While each step was easy to follow, it took me a while (including going back and forth between LobsDICE and other *DICE papers) to get a decent high level motivation for the individual steps. It certainly does not require any major changes, but maybe a bit of additional signposting could help.\n* I think the relationship to DemoDICE should be made clearer. The authors already mention that DemoDICE solves a related problem, but I would have found it helpful if also the similarities and differences in the solution would be discussed.\n* One could argue that the \"delta\" in terms of research compared to DemoDICE is limited. However, I believe there is sufficient difference to warrant publication. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-kyuFsdgEcE", "reviewer": "Reviewer_s7rN", "summary": "This paper algorithmically extends recent advances to the DICE literature in the offline RL setting to the IFO setting. They present a novel extension to match state-transition distributions as well as a more stable minimization objective to achieve SOTA performance for Offline IFO on the Mujoco benchmark task. ", "questions": "This is not a critique, but from the experiments it looked like you used 5 expert demonstrations. I'm curious about how low could you go? BC is able to achieve expert performance with as low as 1 trajectory in these environments (I understand BC also gets actions so has a definite advantage), but it would be really cool to see how LobsDICE does with less and less expert demonstrations. ", "limitations": "Yes the authors have sufficiently addressed limitations and societal impacts in their paper.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. Technical contribution: Extends much of the recent advancements in the DICE literature to the IFO setting to do (s, s') distribution matching. \n2. Definitely excited to see the stabilization of the DICE objective to be a single minimization than a min-max. On a side note, this makes me wonder if this sheds some light into scaling up to higher dimensional benchmarks?\n3. Strong experimentation showing the effective state-transition minimization with state-of-the-art performance on the Mujoco benchmarks. I know D4RL does not provide datasets for Humanoid but would be interesting to fully finish up the benchmark suite for IFO.\n\n### Weaknesses\n1. One small nitpick would be compare against an Offline IL method such as MILO [1] slightly modified for IFO (train discriminator on (s, s') rather than (s, a)). This would be a comparison against a principled offline method for adversarial IL. This would provide more proof of the stability contribution of this work of having just a minimization objective rather than a min-max objective - a benefit the authors mention throughout the text.\n\n\n[1] Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage, 2021 ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-Vr0RjgaH_e", "reviewer": "Reviewer_iQ5k", "summary": "This paper propose an offline learning from observation (LfO) algorithms, which imitates the expert policy by stationary distribution matching without interacting with the environment. The authors presented how to transfer the original complex minimax optimization problem to a simple convex minimization problem in detail, which avoids the instability issue in the original DICE based algorithms. \nEmpirical experiments show the effectiveness of the proposed algorithms and shows competitive performance compared with baseline algorithms.", "questions": "I have a long existing fundamental question for this line of works, which try to use DICE techniques to perform offline imitation learnings, ever since ValueDICE, and also for this work.\n\n If we consider Equation (1) or final Equation (5) in the paper for offline imitation learning, what is the distribution of $\\bar{d}^{E}(s, s^\\prime)$, $d^{I}(s,a)$ or $d^{E}(s,a)$? Are they the **discounted** stationary distributions (the distribution would involve initial state distribution and a discounted factor $\\gamma < 1$) or just the **average** visitation stationary distribution ($\\gamma =1$ or the distribution you just uniformly sample from the empirical dataset).\n\nThis question is important, because when we consider to use the **discounted** stationary distribution $\\bar{d}(s, s^\\prime)$ or $d(s, a)$ of the current learning policy to match the target distributions ($d^{I}(s, a)$ or $\\bar{d}^{E}(s, s^\\prime)$), the target distributions should also be the **discounted** ones. As a result, $d^{I}(s, a)$ or $\\bar{d}^{E}(s, s^\\prime)$ must be the **discounted distribution**, which are not the **same as the distributions we uniformly sample from the empirical dataset**. \n\nFrom what the authors describe in the paper, I think the authors just use the empirical data distribution as $d^{I}(s, a)$ or $\\bar{d}^{E}(s, s^\\prime)$, which are actually not right because the uniformly sampling procedure would give us the average visitation stationary distribution ($\\gamma = 1$). \n\nThere are two ways to fix the flaw:\n1. Consider to define $\\bar{d}(s, s^\\prime)$ or $d(s, a)$  as the average visitation stationary distribution ($\\gamma = 1$), thus Equation (7) is different, and the whole derivation of this paper need to change. \n\n2. Consider to change the sampling procedure such that $d^{I}(s, a)$ or $\\bar{d}^{E}(s, s^\\prime)$ are the discounted stationary distribution, which involves the initial distribution $p_{0}(s)$. One possible simple solution is to sample transition data according to $\\gamma^t$, where $t$ is the time step. \n\n-----\nEmpirically I think there might not be significant impact of above concern, and please correct me if I misunderstand anything in the paper, and I will change my score. \n\n\n\n\n\n\n\n\n  ", "limitations": "The authors adequately addressed the limitations and there are no negative societal impact. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- **Originality**.  \nThe proposed algorithm is based on previous DICE techniques including DualDICE, OptiDICE for offline policy evaluation or improvement, and the authors extended the idea to the offline learning from observations settings. As far as I know, the techniques are not new but there are very few prior works that consider extending the idea to the offline LfO setting. \n\n-  **Quality**.  \nThe paper is technically sound and the empirical experiments are conducted thoroughly. One concern I have is that, as the authors mentioned, there are some concurrent works (SMODICE) related to this topic. In the appendix the authors discussed potential drawbacks of the concurrent work, while there are no empirical experiments to validate and support the claim, which I think the authors should make clear comparison and discussion in both the experiments and the related work sections (I checked with their paper, it seems that they already release the code?). \n\n- **Clarity**.  \nOverall the paper is well presented and it is easy to follow. The derivation presented in the main content are easy to follow if the readers know the basic DICE techniques in the previous work.\n\n- **Significance**.  \nThe presented techniques are a little bit incremental, but I think the proposed algorithm is a good addition to the offline LfO settings. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept