Adversarial Imitation Learning from Visual Observations using Latent Information
Authors: Vittorio Giammarino
James Queeney
Ioannis Paschalidis
Venue: Accepted by TMLR ()

Abstract: We focus on the problem of imitation learning from visual observations, where the learning agent has access to videos of experts as its sole learning source. The challenges of this framework include the absence of expert actions and the partial observability of the environment, as the ground-truth states can only be inferred from pixels. To tackle this problem, we first conduct a theoretical analysis of imitation learning in partially observable environments. We establish upper bounds on the suboptimality of the learning agent with respect to the divergence between the expert and the agent latent state-transition distributions. Motivated by this analysis, we introduce an algorithm called Latent Adversarial Imitation from Observations, which combines off-policy adversarial imitation techniques with a learned latent representation of the agent's state from sequences of observations. In experiments on high-dimensional continuous robotic tasks, we show that our model-free approach in latent space matches state-of-the-art performance. Additionally, we show how our method can be used to improve the efficiency of reinforcement learning from pixels by leveraging expert videos. To ensure reproducibility, we provide free access to all the learning curves and open-source our code.

Published in Transactions on Machine Learning Research (05/2024)
Adversarial Imitation Learning from Visual Observations using Latent Information
Vittorio Giammarino
vgiammar@bu.edu edu
Division of Systems Engineering
Boston University
James Queeney
queeney@merl. com
Milsubishi Electric Research Laboratories
Joannis Ch. Paschalidis
yannisp Abu. edu
Department of Electricol and Computer Engineering
Division of Systems Engineering
Faculty Computing & Data Sciences
Boston University
Reviewed on OpenReview: https: // openreview. net/ forum? id- ydPHjgf6h0
Abstract
We focus on the problem of imitation learning from visual observations, where the learning agent has access to videos experts as its sole learning source. The challenges of this frame- work include the absence of expert actions and the partial obscrvability of the environment, the ground-truth states can only be inferred from pixels. To tackle this problem, we first conduct theoretical analysis of imitation learning in partially observable environments. We establish upper bounds the suboptimality of the learning agent with respect to the divergence between the expert and the agent latent state-transition distributions. Moti- vated by this analysis, we introduce an algorithm called Latent Adversarial Imitation from Observations, which combines off-policy adversarial imitation techniques with a learned latent represcntation of the agent's state from sequences of obscrvations. In experiments on high- digeesiienni continuous robotic tasks, we show that our model-free approach in show how method
latent space matches state-of-the-art performance. Additionally, can be used to improve the efficiency of reinforcement learning from pixels by leveraging expert videos. To ensure reproducibility, we provide free access to all the learning curves and open-source our code.
1 Introduction
Learning from videos represents a compelling opportunity for the future, as it offers a cost-effective and efficient way to teach autonomous agents new skills and behaviors. Compared to other methods, video recording a faster and more flexible alternative for gathering data. Moreover, with the abundance of high-quality vidcos available on the internet, lcarning from vidcos has bocome incrcasingly accessible recent years. However, despite the potential benefits, this approach remains challenging as involves several technical problems that must be addressed simultaneously in order to succeed. These problems include representation learning, significant computational demands due to high-dimensional observation space, the partial observability of the decision process, and lack of expert actions. Our objective is to establish algorithms capable of overcoming all of these challenges, enabling the learning of complex robotics tasks directly from videos of experts.
Formally, our focus is on the problem of Visual Imitation from Observations (V-IfO). In V-IfO, the learning agent does not have access to pre-specified reward function, and instead has to learn by imitating

---

Published in Transactions on Machine Learning Research (05/2024)
Table 1: A summary of imitation from experts: Imitation Learning (IL), Imitation from Observations (lfO),
Visual Imitation Learning (V-IL), and Visual Imitation from Observations (V-IfO).
IL
IrO
V-IL
V-IIO
Fully observable environment Access to expert actions
/ X
x
X
/
X
expert's behavior. Additionally, in V-IfO, expert actions are not accessible during the learning process, and the pixel-based observations we obtain from video frames result in partial observability. The absence of expert actions and the partial observability of the environment distinguish V-IIO from other types of imitation from experts. Specifically, we identify three other frameworks previously addressed in the literature:
Imitation Learning (IL) (Atkeson & Schaal
1997: Abbeel & Ng 2004 Ross & Bagnell 2010 Roske et al. 2021
Giammarino et al. 2023a where stales are fully observable and expert stale-action pairs are accessible, Visual
2021 which explores the idea of imitating directly from pixels but still assumnes that expert actions are provided to the learning agent, and Imitation from Observations (IfO)
Imitation Learning (V-IL) RRfaillvv et al.
Torabi et al.) 2018b ap which retains full observability but considers only the availability of expert states.
Table summarizes these frameworks.
In order to address the V-1fO problem, this paper introduces both theoretical and algorithmic contributions. First, we provide theoretical analysis of the problem and demonstrate that the suboptimality of the learning agent can be upper bounded by the divergence between the expert and the agent latent state- transition distributions. Our analysis motivates the reduction of the V-IfO problem to two subproblems: (i) estimating a proper latent representation from sequences of observations and (ii) efficiently minimizing the divergence between expert and agent distributions in this latent space. Next, we propose practical solutions to these subproblems. By doing so, we formalize a novel algorithm called Latent Adversarial Immitation from Observations (LAIJO), which tackles the divergence minimization step using off-policy adversarial imitation techniques (Chasemipour et al. 2020) and recovers latent representation of the ground-truth state by means of observations stacking (Mnih et al. 2013 2015) and data augmentation (Laskin et al. 2020b Kostrikov ct al. 2020 Yarats et al.) 2021). ). We evaluate our algorithm on the DeepMind Control Suite (Tssa et al. 2018), demonstrating that our model-free approach in latent space achieves state-of-the-arl performance. We conclude by showing how LAIfO can be used on challenging environments, such as the humanoid from pixels
Tassa et al. 2018). to improve Reinforcement Learning (RL) efficiency by leveraging expert videos.
The remainder of the paper organized follows: Section provides summary of the most related works to this paper. Section 3 introduces notation and background RL and IL. Section 4 provides theoretical
analysis of the V-IfO problem. Section 5 introduces our algorithm, LAIfO, and outlines how it can leverage expert videos to improve data efficiency of RL from pixels. Finally, Section G presents our experimental results and Section 7 concludes the paper providing a general discussion on our findings.
2 Related work
In recent years, several studies have focused on the IL problem (Atkeson & Schaal 1997 Abbeel & Ng. 2004;
Ross & Baaneel 2010 Reddy et al., 2019 Reske et al.
2021
Giammarino et al. 2023a) and, in particular, on 2016) which has emerged as one of the most promising
the generative adversarial II. framework (Ho & Ermon
approaches for IL. Adversarial IL builds upon a vast body of work on inverse RL (Russell, 1998 Ng al,,
2000 Abbeel & Ng. 2004 [yydd & Schapire 2007 Ziebart et al. 2008 Syed et al. 2008). The primary goal of inverse RL is to identify reward function that enables expert trajectories (i.e., state-action pairs) to be optimal. The reward function obtained from the inverse RL step is then used to train agents in order to match the expert's 's expected reward. In the fully observable setting, adversarial IL was originally formalized in Ho & Ermon (2016) and Fu et al.) (2017). It was later extended to the observation only setting in Torabi ct al. (2018b) and to the visual sctting Karnan et al. (2022). Furthermore, the adversarial IfO problem has been theoretically analyzed in Yang et al. (2019) and Cheng et al. (2021). Note that all of these studies are built. upon on-policy RT (Schulman et al. 2017). which provides good learning stability but is known
2

---

Published in Transactions on Machine Learning Research (05/2024)
for poor sample efficiency. In recent works, this efficiency issue has been addressed by using off-policy RL algorithms in the adversarial optimization step (Haarnoja et al. 2018 Lillicrap et al. 2015). These include DAC (Kostrikov et al. 2018), SAM (Blondé & Kalousis 2019 and ValueDICE (Kostrikov et al., 2019) for the adversarial IL problem, and OPOLO (Zhu et al. 2020) and MobILE (Kidambi et al. |021] for the adversarial TIO problem. Another line of research has lackled TIO by directly estimatting expert act and
subsoquently deploying IL, techniques on the estimated state-action pairs (Torabi et al., 2018a Liu et al.
2018
Behbahani et al., 2019 Zhang & Ohn-Bar [021 Zhang et al.) 2022 Shaw et al. 2023 Yang al.,
2023
Finally, recent studies have investigated offine alternatives to the adversarial IL framework (Dadashi
et al. 2020).
All of the aforementioned works consider fully observable environments modeled as Markov Decision Processes (MDPs). However, when dealing with pixels, individual observations alone are insufficient for determining
optimal actions. As a result, recent works (Rafailov et al., 2021 Hu et al., 2022) have treated the V-II.
problem as Partially Observable Markov Decision Process (POMDP) (Astrom 1965 J. In particular, Rafailov
et al. 2 addressed the V-IL problem by proposing a model-based extension (Hafner et al. 2019a b) of generative adversarial IL called VMAIL. The work in Gaangaani et al. (2020) also considered IL in a POMDP in order to handle missing information in the agent state, but did not diroctly focus on lcarning from pixels. The more difficult V-IfO problem, on the other hand, has received less attention in the literature. To the best of our knowledge, this problem has only been considered by the recent algorithm PatchAII, (Liu et al. 2023), where off- off-policy adversarial IL is performed directly on the pixel space. Different from Liu et al. (2023) we first study V-IfO from a theoretical perspective, which motivates an algorithm that performs imitation on a latent representation of the agent state rather than directly on the pixel space in PatchAlL. This difference is crucial to ensure improved computational efficiency.
Our work is also related to the RL from pixels literature which tackles the challenge of maximizing an agent's expected return end-to-end, from pixels to actions. This approach has proven successful in playing Atari games MMiih et al. 2013 2015). Recently, RL from pixels has also been extended to tackle continuous action space lasks, such as robol locomotion, by leveraging either data augmentation techniques (Laskin et al.,
2020a b Kostrikov et al.] 2020
Lee et al. 2020:
Raileanu et al. 2021 Yarats et al. 2021) or variational
inference (Hafner ct al. 2019a b Lee et al. 2020
Hafner et al. 2020
Finally, another line of research has focused on the visual imitation problem in the presence of domain mismatch, also known as third-person imitation learning (Stadie et al. 2017 Okumura et al.) 2020 Cetin & Celiktutan 2021; Giammarino et al. (2023b). This paradigm relaxes the assumption that the agent and the expert are defined on the same decision process and represents a generalization of the imitation from experts frameworks introduced in Table 1
3 Preliminaries
Unless indicated otherwise, we use uppercase letters (e.g., St) for random variables, lowercase letters (e.g., st) for values of random variables, script letters (c.g., S) for sets, and bold lowercase letters (c.g., 0) for vectors. Let [t, : te] be the set of integers t such that t1 I S t2j we write S, such that t1 S t S to Stytt. We denote with E[-] expectation, with PP(-) probability, and with (s(;))) an f-divergence between two distributions of which the total variation (TV) distance, DTv(-;)) and the Jensen- Shanmon divergence, Bas(-I1-), are special cases.
We model the decision process as an infinite-horioon discounted POMDP described by the tuple x T,U, U, R. po, 7), where S is the set of states, A is the set of actions, and X' is the set of obser- vations. S x A + P(S) is the transition probability function where P(S) denotes the space of probability distributions over S, u : S + P(X)) is the observation probability function, and R S x A R is the reward function which maps stato-action pairs to scalar rewards. Alternatively, the reward function can also be expressed as R : S x S + R mapping stale-transition pairs to scalar rewards rather than state-action pairs. Finally, Po E P(S) is the initial state distribution and y E [0, the discount factor. The true environment state s E S is unobscrved by the agent. Given an action a E A, the next state is sampled such that s' T(-|s, a). an observation is generated as x' ~ U(-ss'), and a reward R(s, a) or R(s, s') is computed. Note that an MDP is a special case of POMDP where the underlying state is directly observed.
3

---

Published in Transactions on Machine Learning Research (05/2024)
Reinforcement learning Given an MDP and stationary policy S P(A), the RL objective is to
maximize the expected total discounted return J(x) = Er, 7 (s)) where Tx = (so, d0, 81, 01,,......
A stationary policy π induces a normalized discounted state visitation distribution defined as d (s) =
(1 -) Ei PP((s; = sipo: n, T): and we define the corresponding normalized discounted state-action
visitation ion distribution as Px Pr a) = dg (s)m) Finally, we denole the state value function of i as V* =
Er, [E=n (s,, 4t)[so = s) and the state-action value function as Q* (s, a) Er, (Ei= 7 R( at)]so =
8, Ao - a.. When a function is parameterized with parameters 0 E C R we write we.
Generative adversarial imitation learning Assume we have a set of expert demonstrations TE (SO:T, (0:7) generated by the expert policy TE, a set of trajectories TO generated by the policy тa, and a discriminator network Dx= : S x - [0, 1] parameterized by X. Generative adversarial IL. (Ho & Ermon 2016 optimizes the min-max objective
min max Ergllog(Dx a))] E.g [log(1 Dx(s, X
(1)
Maximizing | with respect to X is effectively an inverse RL step where a reward function, in the form of the
discriminator x is inferred by leveraging TE and Tg. On the other hand, minimizing ( with respect to can be interpreted as an RL step, where the agent aims to minimize its expected cost. It has been demonstratod that optimizing the min-max objective in is equivalent to minimizing IDss (P (s. a)]le= (8, a)); we are recovering the expert state-action visitation distribution (Ghasemipour et al. 2020
Latent representation in POMDP ae based on a single observation Te E X is likely to perform poorly since Ie lacks enough information about the unobscrvable true state st. It therefore beneficial to estimate a distribution of the true state from the full history of prior experiences. To that end, we introduce a latent variable 2, E Z such that 2 = Φ( a<t), where é maps the history of observations and actions to Z. Alternatively, when actions are not observable, we have & ((xg)). The latent variable 2 should be estimated such that (s ace)) z P(5)(zz), meaning that 2 represents a sufficient statistic of the history for estimating a distribution of the unobscrabble true stale 8t- It is important to clarily that this does not imply 2 = S.
When dealing with a POMDP, a policy Te(at) that selects an action
4 Theoretical analysis
Recall that we consider the V-IIO problem where expert actions are not available and the ground-truth states 8 are not observable (see Table D] As result, a latent representation 2 E is inferred from the history of observations and used by the learning agent to make decisions.
Throughout the paper we make the following assumptions: (i) the expert and the agent act on the same POMDP and (fi) the latent variable z can be estimated from the history of observations as 2; ((zg() such that P(s 22, ax) = 2())22))= = P( ace). Assumption (i) is instrumental for both our derivations and experiments. Relaxing this assumption would lead to dynamics mismatch (Gangwani et al., 2022 and
visual domain adaptation problems (Giammarino et al., 2023b), which represent interesting extensions for future work. On the other hand, assumption (ii) explicitly states the characteristics required by the latent variable 2; i.e., 21 can be successfully estimated from the history of observations dςg in order to approximate sufficient statistic of the history. Note that this is common assumption in the IL. literature for POMDPs (Gangwani et al.) 2020 Rafailov et al. [002],, , and estimating such a variable is a non-trivial problem that we address in the next section. We further discuss the importance of this assumption from a theoretical perspective in Appendix B (Remark
On the latent space Z, we can define the normalized discounted latent state visitation distribution as (z) = (1 - -)) Ee P(, = z|po, no, T2)) and the normalized discounted latent state-action visitation distribution Proo P (2,a) = d (a)z)) Further, we define the latent state-transition visitation distribution as Pmo (2, dza (2) [A P(z'|2, mo (ā) z) and the normalized discounted joint distribution as Pao Pmp a, =
Pro (z P( 12,1121,00 a), where
P(1llz, a) 16 / P(''=', : s, (d ds.
(2)
4

---

Published in Transactions on Machine Learning Research (05/2024)
Finally, we obtain Pro (a)2,2²)
1, (a)z)
P (a/2, = Ia P(='l (a (a)=)dā
Note that we write P Pao: with Fe as subscript, order to explicitly denote the dependency on the policy and omit the subscript, as in (2 when such probability depends only on the environment.
We start by considering the case in which R: : S x A + R and J(x) = E,, 1> -t= RR((s, The following Theorem shows how the suboptimality of TH be upper bounded by the TV distance between latent statc-transition visitation distributions, roducing the V-IfO problem to divergence minimization problem in the latent space 2.
Theorem 1. Consider a POMDP, and let R. S x A + R and žt = (((5)) such that P(s ae) = P((8lzz)) =
P(selag,, act). Then, the following inequality holds:
2mmax
[J(TE) - - 1-
DTV
Pum (xo(2 + C,
where Rmax = max(s,)) [R(s, a)l and
22maxx
C 1-y Epro (2.2²) [DTV 2'), (alz,
Proof. Using the definition of J(me), we first upper bound the performance difference between expert and agent by Dry (pro ....)) a)). Next, we bound the latter divergence by Dry (2,o(2,)) Paw (ax(2, a)) using the assumption P(ss/2x, dx) = (s (ss)2 a)) and noticing that (((x)=+)) is policy independent. Finally, we bound this last divergence in terms of Drv (Pro (2, Pare (z, (Lemma 3 in Appendix B). We provide the full derivations Appendix
Theorem addresses the challenge of considering rewards that depend on actions without the ability observe expert actions. Consequently, in our setting, we cannot compute C in (3). Similar to the MDP case (Yang al. 2019), sufficient condition for C= indicating that there is only one action corresponding to a given latent state transition. This property ensures that P(a|22 z²) remains unaffected by different exccuted policies, ultimately reducing C' to zero. For the sake of completeness, we formally state this result in Appendix C However, in our setting, it is difficult to guarantee the injectivity of [F(2'|z, a) due to its dependence on both the environment through U(='|s') and T(s'ls, |s, a), and the latent variable estimation method through P'z''|z', a, 2) and [s]]z). Instead, we demonstrate in Theorem how redefining the reward function as : S x S + R, which commonly observed in robotics learning, allows us to reformulate the result in Theorem [ without the additive term C in 33
0 is the injectivity of P(='|z,0) in (2) with respect to a,
Theorem 2. Consider POMDP, and let R S S RR and 21 ((x()) such that PP 2t, ae) = (P((e/=)) (())) Then, the following inequality holds:
5 2R DTv (pro z'), Pre (2,
1-7
where Rmax- (xX RR(s,')|.
Proof. The proof proceeds similarly to the one for Theorem by using that P(s, s's,s' is not characterized by the policy but only by the environment. We show the full proof in Appendix
In summary, Thcorems I and 2 show that, assuming we have latent space 2 that can effectively approximate a sufficient statistic of the history, the imitation problem can be performed entirely on this latent space. Note that this in contrast with the existing literature (Liu et al.) 2023), where imitation is performed on the obsorvation space X. As a result of this analysis, our algorithm characterized by two main ingredients: a practical method to estimate 2 E Z from sequences of observations, and an efficient optimization pipeline minimize Bry (p (z, 2²), Pag (2,xg(,,2))
5

---

Published in Transactions on Machine Learning Research (05/2024)
5 Latent Adversarial Imitation from Observations
In the following, we introduce the main components of our algorithm LAIfO. Motivated by our theoretical analysis in the previous section, our algorithm combines techniques for adversarial imitation from observations and latent variable estimation. First, we outline our adversarial imitation pipeline in the latent space Z, which leverages ofl-policy adversarial imitation from observations (Kostrikov al., 2018: Blondé & Kalousis
2019 Zhu et al.) 2020) in order to minimize the divergence between the latent state-transition visitation distributions of the agent and expert. Then, we describe a simple and effective approach for estimating
the latent state that makes use of observations stacking (Mnih et al. 2013 2015) and data augmentation
Laskin et al. 2020b; Kostrikov et al.) 2020 Yarats et al.) 2021). Finally, we show how LAIfO can leverage
expert videos to enhance the efficiency of RL from pixels in a number of highly challenging tasks.
Off-policy adversarial imitation from observations Based on the results in Section 4 given a latent. variable z that captures a sufficient statistic of the history, we can minimize the suboptimality of the policy Te by solving the minimization problem
min (P (z, Prg Pa (2,2))) 0
(4)
We propose to optimize the objective in 4 using off-policy adversarial IfO. We initialize two replay buffers Bg and B to respectively store the sequences of observations generated by the expert and the agent policies, from which we infer the latent state-transitions (z, 2²). Note that we write (2, ~ B to streamline the notation. Then, given a discriminator Dx Z x Z + [0, 1], we write
logg (xx (2,2222 21i + + (6..2)-0olg log(1 max ((2,"))=B llg
(5)
where 9(-) is defined in G As mentioned, alternating the maximization of the loss in (5) with an RL step leads to the minimization of Iass (Pao (2)22"((pxs (2,2)) (Goodfellow et al. 2020). Since D-s(-I1-) can be used to upper bound - (cf. Lemma in Appendix B this approach effectively minimizes the loss in (4). In order to stabilize the adversarial training process, it is important to ensure local Lipschitz-continnity of the
learned reward function BBondé et al. 2022 Therefore, as proposed in Gulrajani et al. (017), we include in the gradient penalty term
g Px) = (B [(l ((IIV 1)
(6)
where À is hyperparameter, and (Pee,e) is defined such that 22,2") are sampled uniformly along straight lines between pairs of transitions respectively sampled from BE and B. For additional details on the importance of the term in 16 for improved stability, refer to our ablation experiments in Appendix D and to Gulrajani et al. (2017). Finally, from a theoretical standpoint, note that we should perform importance sampling correction in order to account for the effect of off-policy data when sampling from B (Qucency et al. 2021 2022 However, neglecting off-policy correction works well in practice and does not compromise the stability of the algorithm Kostrikov et al. 2018).
Latent variable estimation from observations Note that the problem in (4] is defined on the latent space 2. Therefore, we now present a simple and effective method to estimate the latent variable from sequences of observations. Inspired by the model-free RL from pixels literature, we propose combine the successful approaches of observations stacking (Mnih et al.) 2013 2015) and data augmentation (Laskin et al. 2020b Kostrikov et al.) 2020 Yarats et al. 2021). We stack together the most recent d N observations, and provide this stack as an input to a feature extractor which is trained during the RL step. More specifically, we define a feature ure extractor Φs : X + Z such that = (a(xz=a) where / - 1 + = d. When learning from pixels, we also apply data augmentation to the observations stack to improve the quality of the extracted features as Kostrikov et al. (2020). We write aug(r(-a) to define the augmented stack of observations. The latent representations & and x' are then computed respectively as (aug((r-)) and z' Φs (auug(#(++1++1) We train the feature extractor de with the critic networks Qu. (I: = 1,2) in order

---

Published in Transactions on Machine Learning Research (05/2024)
to minimize the loss function
bb (B) = B (Qpp (2, 2) y = xx(2,2) 2 +
(7)
min Q a²),
k=
"x(2,2)) 2') = Dx(2,
(8)
where Dx(2, z') is the discriminator optimized in 15 7 a is an action stored in B used by the agent to interact with the enviroument, while a' = (=(=²) + where ~ clip(N(O,, (0, - -c, c) is a clipped exploration noise with the clipping parameter and (0 o²) a univariate normal distribution with zero mean and o standard deviation. The reward function (x(2, ) is defined in (8), and 4/1 and 1/2 are the slow moving weights for the target Q networks. We provide more implementation details and the complete pseudo-code for our algorithm in Appendix D]
Note that the feature extractor os is shared by both the critics Qwa, the policy no, and the discriminator xx However, we stop the backpropagation of the gradient from To and Dx into Φg. The logic of this choice involves obtaining a latent variable that not biased towards any of the players in the aevesaaiaal IfO game in 5), but only provides the information necessary determine the expert and agent expected performance. This design is motivated by our theoretical analysis which shows how, provided Φ8 x Z where 2 approximates sulficient statistic of the history, all the networks in the adversarial IIO game can be directly defined on Z as Dx : Z x Z + [0, 1], Q Z x A + IR and To : Z + P(A). This is in contrast with the current literature on V-IfO where all the networks are defined on the observation space X' as Dx : X x X + [0, 1], Que X'' x A + I and Te X P(A).
Finally, note that latent variable estimation is an active research area, and it is possible to apply other techniques such as variational inference (Lee et al. 2020) and contrastive learning (Chen al.) 2020 Grill et al. 2020). However, will show in our experiments that the straightforward approach of observations stacking and data augmentation leads to strong performance in practice, without the need for more complicated estimation procedures. We include an ablation study on the importance of data augmentation in Appendix L
Improving RL from pixels using expert videos We have so far considered the pure imitation setting where reward function can only be estimated from expert data. However, for many real-world tasks simple objective can often be provided to the learning agent. Assuming that videos of experts are also available, we show how we can use LAIfO to accelcrate the RL learning process.
We combine the standard RL objective with our V-IfO objective in (4), leeading to the combined problem
max 0
ET
R(s,, ae) - Drv -Dro (2, Pag (z,z²)) z²))
(9)
Using the adversarial IfO pipeline presented in (5), we can rewrite
9 as
>' (R(s), + (r (2(
max
E,
(10)
with rx in (8). By learning 'x with LAIIO and optimizing the problem in 110 throughout training, we will show that we able to significantly improve sample efficiency challenging humanoid from pixels tasks TTssa et al. 2018 compared to state-of-the-art RL from pixels algorithms Yaratt et al. 2021
6 Experiments
In this section, we conduct experiments that aim to answer the following questions:
(1) For the V-IIO problem, how does LAIIO compare to PatchAIL (Liu al. 2023). a state-offthe-art approach for V-IfO, in terms of asymptotic performance and computational efficiency?
(2) How does the V-IL version of LAIfO with access to expert actions, named Latent Adversarial Imitation Learning (T.AII.) compare to VMAIL (Rafailov et al. 2021 state-of-the-art approach for V-IL?
7

---

Published in Transactions on Machine Learning Research (05/2024)
Table 2: Experimental results for V-1fO (i.e., imitation from experts with partial observability and without access to expert actions). We use DDPG to train experts in a fully observable setting and collect 100 episodes of expert data. All of the expert policies can be downloaded by following the instructions in our code repository. BC is trained offline using expert observation-action pairs for 10 gradient steps. All the other algorithms are trained for 3 x 10 frames in walker run, hopper hop, cheetah run, quadruped run, and quadruped walk, and 10 frames for the other tasks. We evaluate the learned policies using average performance over 10 episodes. We run each experiment for 6 seods. In the third, fourth and fifth columns, we report mean and standard deviation of final performance over seeds. In the last column, we report the ratio of wall-clock limes between LATIO and PatchAlL to achieve 75% of expert performance. For each task, we highlight the highest asymptotic performance between LAIFO and PatchAIL.
Wall-clock time ratio to 75% expert performance (LATHO (our) / PTIchATI--W
PatchAIL-W
Expert
BC
LATIO (our)
Liu et al. 2023
Cup Catch
980
971 + 9.7
967 = 7.6
804 + 357
0.69
Finger Spin
932
542 = 219
926 10.7
885 + 30.8
0.67
Cartpole Swingup
881
329 + 25.0
873 3.6
842 6.7
0.63
Cartpole Balance
990
648 36.4
878
966 5.5
0.61
Pendulum Swingup
845
427 1: 142
786 1 70.4
829 23.7
0.90
Walker Walk
960
723 137
7.0
0.15
Walker Stand
980
871 77.7
961 2000
971 10.5
0.27
Walker Run
640
133 27.8
1.6
5
0.22
Hopper Stand
920
398 96.4
800 46.7
867 = 33.9
0.16
Hopper Hop
217
45.9 22.1
206 8.5
191 13.0
0.16
Quadruped Walk
970
337 50.5
594 + 92.9
263 + 242
NA
Quadruped Run
950
340 1 75.2
516 132
471 1: 219
NA
Cheetah Run
900
106 + 26.3
773 = 41.2
695 + 312
0.46
* 75% of the expert performance was not achieved by any algorithm.
(3) What is the impact performance due to partial observability and the absence of expert actions?
(4) Can LAIFO leverage expert videos improve the efficiency of RL from pixels in high-dimensional continuous robotic tasks?
For more details about the hardware used to carry out these experiments, all the learning curves, additional ablation studies, and other implementation details, refer to Appendix D and to our (code.
Visual Imitation from Observations In order to address Question (1), we evaluate LATIO and PatchAlL (Liu et al.) 2023), in its weight regularized version denoted by PatchALL-W, on 13 different tasks from the DeepMind Control Suite (Tassa et al.) 2018). We also compare these algorithms to Behavioral Cloning (BC) (Pomerleau, 1988) for reference. Note that BC uses observation-action pairs. The results
are summarized in Table Figure [ and Figure 2 Table 2 includes the asymptotic performance of each algorithm, as well as the ratio of wall-clock times between LAIFO and PatchAlL to achieve 75% of expert performance. Figure [ depicts the average return per episode throughout training as a function of wall-clock time. Moreover, we include in Figure 2 plots showing the average return per episode as a function of training steps. These results demonstrate that LAIfO can successfully solve the V-IfO problem, achieving asymptotic performance comparable to the state-of-the-art baseline PatchAIL. Importantly, LAIfO is signéficantly more computationally efficient than PatchAIL. This is well highlighted both in Table 2 and in Figure ! where we show that LAIFO always converyes faster than PatchAlL in terms of wall-clock time. This improved computational efficiency is the result of performing imitation on the latent space Z, instead of directly on the high- figh-dimensional observation space X (i.e., pixel space) as in PatchAlL. Finally, in Table we examine the impact of the amount of expert data on performance. Throughout these experiments, LATOO does not exhibit. a tangible drop performance due to the decrease of available expert data, and consistently outperforms PTtchAAT..
8

---

Published in Transactions on Machine Learning Research (05/2024)
Cup Catch
Finger Spin
Cartpole Swingup
Cartpole Balance
Pendulum Swingup
1220
2000
Return
100
1020
820
www
820-
020
#20
600
820
620
410
230
200
400
220
10
Hours
Hours
Hours
Hours
Hours
Walker Walk
Walker Stand
Walker Run
Hopper Stand
Hopper Hop
1020
820
200
822
900
130-
620
Epicade
420-
130-
400
200
250
220
30
20
30
20
30
25
50 Hours
75
300
20
Hours
Hours
Hours
Hours
Quadruped Walk
Q Quadruped Run
Cheetah Run
LAIFO (our)
Episoda Recur
PatchAlL-W BC
000
100
200
expert
Hours
Hours
Hours
Figure 1: Learning curves for the V-1fO results in Table 2 Plots show the average return per episode a function of wall-clock time. Our algorithm LATfO achieves statc-of-the-art asymptotic performance, and siguificantly reduces computation time compared to PatchAIL.
Catch
Finger Spin
Cartpole Swingup
Cartpole Balance
Pendulum Swingup
1220
3000
12:0
1020
Raturn
000
800
610
620
250-
230
0.50
0.23
5.50
0.75 1.00
3.50
0.5
0.50
0.75
1.00
2.00
0.55
050
0.75
1:0
0.00
0.23
0350
0.75
1.00
5.50
025
0.50
0.75
1.00
Steps (x10%)
Steps (x109)
Steps (x (x10')
Steps (x10%)
Steps (x109)
Walker Walk
Walker Stand
Walker Run
Hopper Stand
Hopper Hop
3070
600
820
500
6:0
400
Eplcode
500
300
420
420
230
200
200
20
0.30
0.5
0:30 Staps (x10%)
Stops (x10%)
Steps (x10*)
Stops (x
Steps (x107)
Quadruped Walk
Quadruped Run
Cheetah Run
LAIFO (our)
Episode Return
820
300
600
PatchAIL-W
820
420
BC
220
200
expert
Steps (x10%)
Steps (x10f)
Steps Dx (x10%)
Figure 2: Learning curves for the V-IfO results in Table function of training steps.
Plots show the average return episode as

---

Published in Transactions on Machine Learning Research (05/2024)
Table 3: Experimental results for V-IL (i.e., imitation from experts with partial observability and access to expert actions). We use DDPG to train exports in a fully observable setting and collect 100 episodes of expert data. The experiments are conducted as in T In the third, fourth and fifth columns, we report mean and standard deviation of final performance over sceds. In the last column, we report the ratio of wall-clock times between the two algorithms to achieve 75% of expert performance. For each task, we highlight the highest asymptotic performance between LAIL and VMAIL.
Wall-clock time ratio to 75% expert performance (LAII. (our) VMATI)
VMAIL
Expert
BC
LAIL (our)
Rafailov et al. 2021
Cup Catch
980
971 9.7
962 + 18.0
939 + 37.4
0.31
Finger Spin
932
542 : 219
775 - 345
476 + 352
0.21
Cartpole Swingup
881
329 25.0
873 + 3.0
512 4 291
0.13
Cartpole Balance
990
648 36.3
982 + 1.6
868 + 104
0.10
Pendulum Swingup
845
427 142 142
825 45.1
723 143
1.43
Walker Walk
960
723 137
946
939 9.8
0.40
Walker Stand
980
871 77.7
106
+ 309
0.82
Walker Run
640
133 + 27.8
5.1
224
0.58
Hopper Stand
920
398 1 96.4
111
72.3 285
0.12
Hopper Hop
217
22.1
208 3.1
0.23
Quadruped Walk
970
337 + 50.5
500 182
223 + 51.9
NA
Quadruped Run
950
340 75.2
697 + 102
127 + 66.0
NA
Cheetah Run
900
106 26.3
811 67.9
539 + 367
0.83
75% of the expert performance was not achieved by any algorithm.
** 75% the expert performance was not achieved by VMAII..
Visual Imitation Learning To answer Question (2), we test LAIL, the V-IL version of LAIfO, and VMAIL (Rafailov et al., 2021) using the same experimental setup that we considered in the V-15O setting. As for V-IfO, wC also compare these algorithms to BC for reference. VMAIL stands for Variational Model Adversarial Imiitation Learning, and represents model-based version of generative adversarial IL built upon the variational models presented in Hafner et al.
2019b a 2020). LAIL is obtained by simply defining the x 2
discriminator in (5) as Dx: : 2 x A > [0, rather than Dx 2
[0, as in LATFO. The results for these experiments are summarized in Table 3 Figure 3 and Figure A Compared to VMAIL, we see that
LAIL achieves better usymplotic performance and better compulational efficiency. While both algorithis perform imitation on latent space Z, LAIL is model-free algorithm that requires a lower number of learnable parameters compared to the modcl-based VMAIL. VMAIL must learn accurate world model during training, which can be challenging and computationally demanding task. The model learning process contributes to higher wall-clock times, and can also lead to instability in the training process for some environments (cf. Figure 4.. On the other hand, the model-free approach of LAIL results in stable training that yields faster convergence and better efficiency. Finally, in Table 4 we examine the impact of the amount of expert data on the final results. Throughout these experiments, both LAIL, and VMAIT. exhibit reliable results and no tangible decrease in performance observed due relying on less expert data.
A Ablation study In order to answer Question (3), we compare performance for each type of imitation from experts in Table I For the partially observable setting, we consider our algorithms LAIL and LAIFO. For the fully observable setting, we consider DAC (Kostrikov et al. 2018) and our implementation of DAC from Observations (DACFO). We provide the full learning curves for DAC and DACIO in Appendix L (cf. Table P and Figure 7. The results are summarized in Figure 5 which shows the average normalized return obtained by each algorithm throughout the different tasks in Table 2 These experiments highlight how our algorithms can successfully address the absence of expert actions and partial obsorvability, suffering only marginal performance degradation due to these additional challenges. As explained in our theoretical analysis in Section 4 partial observability is addressed by estimating a latent state representation that successfully approximates a sufficient statistic of the history. On the other hand, marginal degradation due to the absence of expert actions occurs either because we are in the context described by Thcorem 2 where the environment reward function does not depend on actions, or because C in Theorem becomes negligible.
10

---

Published in Transactions on Machine Learning Research (05/2024)
Cup Catch
Finger Spin
Cartpole Swingup
Cartpole Balance
Pendulum Swingup
2000
830-
820
020
620
420
220
200
200
420
230
10
23
Hours
Hours
Hours
Hours
Hours
Walker Walk
Walker Stand
Walker Run
Hopper Stand
Hopper Hop
:200
020
0003
620
400
100
420
400
220
15
20
40
90
10
10
40
Hours
Hours
Hours
Hours
Hours
Quadruped Walk
Quadruped Run
Cheetah Run
10000
LAIL (our)
Episoda Retum
VMAIL
620
BC
220
200
expert
Hours
Hours
Figure 3:
Learning curves for the V-IL results Table 3
Plots show the average return per episode as
a function of wall-clock time. LAIL outperforms VMAIL in terms of both asymptotic performance and computational efficiency.
Catch
Finger Spin
Cartpole Swingup
Cartpole Balance
Pendulum Swingup
1220
1200
1000
1020
Raturn
.100
600
1020
000
500
620
600
430
230
0.50
0.23
0.50
0.75
1.00
2.50
0.5
0.50
0.55
1.00
2.00
0.55
050
0.75
1:0
0.50
0.23
0.75 0.30
1.00
2.50
0.5
0.30
0.75
1.00
Steps (x10
Steps (x10%)
Steps (x (x10")
Steps (x10%)
Steps (x109)
Walker Walk
Walker Stand
Walker Run
Hopper Stand
Hopper Hop
1030
1000
910
200
920
820
150
Eplcode
0.0
420
420
200
200
220
Stops (x10")
Steps (x10')
Stops (x
Steps (x10")
Steps (x10*)
Quadruped Walk
Quadruped Run
Cheetah Run
1000
LAIL (our)
Episode Return
820
100
VMAIL
420
BC
expert
Steps (x10%)
Steps (x10f)
Steps Ox 10')
Figure 4: Learning curves for the V-IL results in Table Plots show the average return per episode as function of training steps.
11

---

Published in Transactions on Machine Learning Research (05/2024)
Table 4: Performance for different numbers of expert episodes on Walker Run. Except for the number of expert episodes, the experiments are conducted as in Table and Table 3 We report mean and standard deviation after training for 10 frames. For both the V-IL and the V-1f0 settings, highlight the highest performance.
Expert episodes
Walker Run
10
20
50
100
BC
70.7
87.0 + 1.8
113
133 1.8
VMAIL
Rafailov et al.
2021
520 + 221
630 + 21.9
517 +: 221
619 + 11.8
521 4: 222
LAIL (our)
614 7.6
626 7.3
627 6.8
611 + 22.5
613 + 20.3
PatchAlL-V W [Liu et al.
2023
494 + 199
561 36.0
586 = 39.7
604 + 22.5
598 + 34.7
LAIfO (our)
612 - 22.8
620 + 6.4
623 + 7.6
618 - 10.1
621 + 4.8
normalized retum
Texpert performance
0.97
0.-
0.89
0.89
0.92
0.
Imitation
Imitation
Imitation without expert actions and full observability (IOO)
Imitation with expert actions and full observability (IL)
without expert actions and partial abservability and partial observability (V-IL)
with expert a actions
(V-IfO)
Figure 5: Normalized returns obtained by each type of imitation from experts the tasks in Table For each task and random seed, normalized returns represent performance divided by the expert performance for the considered task (Expert column in Table 2 and Table 3.. For each type of imitation from experts, we plot mean and standard deviation over the full set of rums. The performance of our algorithms in the partially observable setting are comparable to the performance in the fully observable setting, and the absence of expert actions and partial observability leads only to marginal performance degradation.
Improving RL using expert videos We answer Question (4) by applying LAIfO to the problem in for the humanoid from pixels environment. We consider the statc-of-the-art model-free RL from pixels algorithms, DrQv2 (Yarats et al. 2021) as a baseline. The results are illustrated Figure G By leveraging expert videos, we see that our algorithm significantly outperforms the baseline terms of sample efficiency. This result highlights the value of leveraging expert videos to improve the sample efficiency of RL algorithms, in particular when dealing with challenging tasks.
Humanoid Stand
Humanoid Walk
Humanoid Run
Episode Return
300
500
500
200
100
0
20 Steps (x106)
10
30
0
10
20
30
0
20 Steps (x10%)
10
30
Steps (x10%)
RL + LAIFO
DrQ v2
Figure 6: Performance using the multi-objective RL framework in 19 on the humanoid environment. The experiments are designed as in Table [ We report mean and standard error over seeds. For DrQv2, note that we use as comparison the curves provided in the official Github repository of the paper.
12

---

Published in Transactions on Machine Learning Research (05/2024)
7 Conclusion
In this work, we formally analyzed the V-1fO problem and introduced our algorithm LAIfO effective sonution. We experimentally showed that our approach matches the performance of state-of-the-arl V-IL and V-If0 methods, while requiring significantly less computational effort due to our model-free approach in latent space. Furthermore, we showed how LAIf0 can be used to improve the efficiency and asymptotic performance of RL methods by leveraging expert videos.
Limitations and future work Despite the advancement in addressing the V-IfO problem, it is important understand the limitations of our approach. The primary limitation arises from the assumption that the expert and the agent act within the same POMDP. In realistic scenarios, such aligmment rarely occurs, emphasizing the need for methods that can handle dynamics mismatch and visual domain adaptation. This is a crucial next step towards enabling successful learning from expert videos. Furthermore, throughout this work we have used adversarial learning for divergence minimization between distributions. Adversarial learning can introduce optimization challenges and stability issues. While we propose practical solutions to mitigate these problems, exploring alternatives to this framework offers another interesting avemue for future research. Additionally, from an experimental standpoint, our emphasis has been on robotics control tasks. In the future, we plan to address navigation tasks, considcring not only third-view perspectives but also egocentric camera viewpoints. In this context, challenging and relevant consideration the correspondence problem, ie., the problem of enabling egocentric policy learning directly from third-view videos of experts. Finally, we are interested in testing our algorithms on more realistic scenarios that go beyond simulated environments. Our long-term goal is to provide solutions for real-world problems, such as vehicle navigation and robotic ic manipulation, and to enable imitation directly from videos of biological systems such as humans and animals.
8 Acknowledgements
Vittorio Giammarino and Ioannis Ch. Paschalidis were partially supported by the NSF under grants IIS- 1914792, (C0-020,052, and ECCS-2217079, by the ONR under grant N00014-19-1-251 by the DOE under grant DE=AA002-05CH1222 by the NIH under grant UL54 TR004130, and by Boston University. James Queency was exclusively supported by Mitsubishi Electric Research Laboratorics.
References
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first International Conference on Machine Learning, pp. 1, 2004.
Karl J. Astrom. Optimal control of markov decision processes with incomplete state estimation. J. Math. Anal. Applic., 10:174 205, 1965.
Christopher G. Atkeson and Stefan Schaal. Robot learning from demonstration. In Proceedings of the fourteenth International Conference on Machine Learning, pp. 12 22, 1997.
Feryal Behbahani, Kyriacos Shiarlis, Xi Chen, Vitaly Kurin, Sudhanshu Kasewa, Ciprian Stirbu, Joao Gomes, Supratik Paul, Frans A. Olichock, Joao Messias, al. Learning from demonstration in the wild. In 2019 International Conference on Roboties and Aulomation (ICRA), pp. 775 781. IEEE, 2019.
Lionel Blondé and Alexandros Kalousis. Sample-efficiet imitation learning via generative adversarial nets. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 3138-3148 PMLR, 2019.
Lionel Blondé, Pablo Strasser, and Alexandros Kalousis. Lipschitzness all you need to tame off-policy generative adversarial imitation learning. Machine Learaing, 111(4):143) 1521, 2022.
Edoardo Cetin and Oya Celiktutan. Domain-robust visual imitation learning with mutual information constraints. arxiv preprint ar urXiw:2103.05007 2021.
13

---

Published in Transactions on Machine Learning Research (05/2024)
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMTR, 2020.
Zhihao Cheng, Liu Liu, Aishan Läu, Hao Sun, Meng Fang, and Dacheng Tao. On the guaranteed almost. equivalence between imitation learning from observation and demonstration. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imitation learning. arXiv preprint arXiv:2006.0467. 2020.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248, 2017.
Tanmay Gangwani, Joel Lehman, Qiang Liu, and Jian Peng. Learning belief representations for imitation learning in POMDPs, In Uncertainty in Artificial Intelligence, pp. 1061-1071. PMLR, 2020.
Tanmay Gangwani, Yuan Zhou, and Jian Peng. Imitation learning from observations under transition model disparity, arXiv preprint or 22w.2224411444. 2022.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective imitation learning methods. In Proceedings of the Conference Robot Learning, pp. 1259 1277. PMLR, 2020.
Vittorio Giammarino, Matthew F Dunne, Kylie N Moore, Michael E Hasselmo, Chantal E Stern, and Ioannis Ch Paschalidis. Combining imitation and deep reinforcement learning to human-level performance on virtual foraging task. Adaptive Behavior, 2023a.
Vittorio Giammarino, James Queeney, Lucas C. Carstensen, Michael E. Hasselmo, and Ioannis Ch. Paschalidis. Opportunities and challenges from using animal videos in reinforcement learning for navigation. IFAC- PapersOnLine, 56(2):9056 9061, 2023b. 22nd IFAC World Congress.
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 139-144, 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemoud, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 13:21271-21286 2020.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of Wasserstein GANs. Advonces in Neural Information Processing Systems, 30, 2017.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint Xr 05905, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1912: 2.06003, 2019a.
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the thirty-sinth International Conference on Machine Learning, pp. 2555 2565. PMLR, 2019b.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. ar preprint or orXim:2010.02198, 2020.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems, 29, 2016.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zak Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learning for urban driving. or preprinl arXiu:2210.07729, 2022.
Haresh Karnan, Faraz Torabi, Garrett Warnell, and Peter Stonc. Adversarial imitation learning from video using a state observer. In 2022 International Conference on Robolics and Aulomation (ICRA), pp- 2452 2458. IEEE, 2022.
Rahul Kidambi, Jonathan Chang, and Wen Sun. Mobile: Model-based imitation learning from observation alone. Advances in Neural Information Processing Systems, 34:28598 28611, 2021.
Nya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-tttii Addressing sample inelficiency and reward bias in adversarial imitation learning. arxiv preprint arv:: 1009. 1809. 029202 2018.
llya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off off-policy distribution matching. arXiv preprint 19Xiv:112..0000 2019.
lya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649 2020.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representations for reinforcement learning. In Proceedings of the thirty-seventh International Conference on Machine Learning, pp. 5639-5650. PMLR, 2020a.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems, 33:19884-1998 2020b.
Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. Advances in Neural Information Processing Systems, 33:741 1 2020.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint Xrv: 029900297, 2015.
Minghuan Liu, Tairan He, Weinan Zhang, Shuicheng Yan, and Zhongwen Xu. Visual imitation learning with patch rewards. arXiv preprint ar Xrx:22302. 20062000000 2023.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate bchaviors from raw video via context translation. In 2018 IEEE International Conference on Robolics and Aulomation (ICRA), pp. 1118 1125. IEEE, 2018.
Volodymyy Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglon, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement. learning. arXie preprint arxim: 1E12:33155003 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529- 533, 2015.
Andrew Y. Ng, Stuart Russell, el al. Algorithms for inverse reinforcement learning. In Proceedings of the seventeenth International Conference on Machine Learning, volume 1, pp. 2, 2000.
Ryo Okumura, Masashi Okada, and Tadahiro Taniguchi. Domain-udverrarial and-conditional state space model for imitation learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5179-5186, IEEE, 2020.
Yury Polyanskiy and Yihong Wu. Information theory: From coding to learning, 2022.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in Neural Information Processing Systems, 1, 1988.
James Queeney, Ioannis Ch. Paschalidis, and Christos G. Cassandras. Generalized proximal policy optimization with sample reuse. Advances in Neural Information Processing Systems, 34:11909 19111999--1199,, 2021.
James Queeney, Ioannis Ch. Paschalidis, and Christos G. Cassandras. Generalized policy improvement algorithms with theoretically supported sample reuse. arXiv preprint arXiv:2206.13714 2022.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Visual adversarial imitation learning using variational models. Advances in Neural Information Processing Systems, 34:3016-380 6-2088 2021.
Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Πlya Kostrikov, and Rob Fergus. Automatic data
augmentation for generalization in reinforcement learning. Advances in Neural Information Processing Systems, 34:5402 5415, 2021.
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. In International Conference on Learning Representations, 2019.
Alexander Reske, Jan Carius, Yuntao Ma, Farbod Farshidian, and Marco Hutter. Imitation learning from mpe for quadrupedal multi-gait control. In 2021 IEEE Internalional Conference on Robolics and Aulomation (ICRA), pp. 5014-5020. TEFF, 2021.
Stéphane Ross and Drew Bagnell. Elficient reductions for imitation learning. In Proceedings of the thirleenth International Conference on Artificial Intelligence and Statistics, pp. 561-668. JMLR. Workshop and Conference Procoodings, 2010.
Stuart Russell. Learning agents for uncertain environments. In Proceedings of the eleventh Annual Conference on Computational Learning Theory, pp. 101-103, 1998.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alee Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint or Xivx:17770063/7 2017.
Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. Videodex: Learning dexterity from internet videos. In Proceedings of The sirth Conference Robot Learning, pp. 654-665. PMLR, 2023.
Bradly C. Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. arXiv preprint 1.Xiv:1703... 01703, 2017.
Umar Syed and Robert E. Schapire. A game-theoretic approach to apprenticeship learning. Advances in Neural Information Processing Systems, 20, 2007.
Umar Syed, Michael Bowling, and Robert E. Schapire. Apprenticeship learning using linear programming. In Proceedings of the twenly-fifth International Conference on Machine Learning, pp- 1039,1039, 2008
Yuval Tassa, Yolam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv.1801.00690, 2018.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. ar preprint arXiv:1805 008800050.11 2018a.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. Xiv preprint arXio:1807.06155 2018b.
Chao Yang, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. Advances in Neural Information Processing Systems, 32, 2019.
Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Picter Abboel, and Dale Schuurmans. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint ar Xrx::2300.00 04129, 2023.
16

---

Published in Transactions on Machine Learning Research (05/2024)
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprinl or oriim:1077006433 2021.
Jimuyang Zhang and Eshed Ohn-Bar. Learning by watching. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pp- 12711-12721, 2021.
Jimuyang Zhang, Ruizhao Zhu, and Eshed Ohn-Bar. Selfd: self-learning large-scale driving policies from the web. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17316-17326, 2022.
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from observations. Advances in Neural Information Processing Systems, 33:12402 12413, 2020.
Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, Anind K. Dey, et al. Maximum entropy inverse reinforcement learning. In Twenly- Third AAAT Conference on Arlificial Intelligence, volume 8, pp- 1433-1438. Chicago, IL, USA, 2008.
17


Reviews:
[{"review_id": "j3sCJBB5S8", "reviewer": "Reviewer_mYfo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is overall well-written, e.g. the abstract+introduction provides a clear view on the setting and the contribution (and Table 1 is useful).\n- The contribution is clear and the experimental results that are provided are convincing\n- code is provided\n\nWeaknesses\n- An assumption that $z_t$ effectively represents a sufficient statistic of the history is made. This assumption essentially circumvents the biggest difficulty of working with a POMDP. The theoretical results seem relatively limited in terms of novelty.\n- There seems to be some technical errors/improvements", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The study focuses on learning from expert videos for imitation tasks, addressing challenges of absent expert actions and partial observability. They propose Latent Adversarial Imitation from Observations is an algorithm that combines off-policy imitation techniques with learned latent representations. A few theoretical results are given and the experiments on robotic tasks show competitive performance. Code for reproducibility is provided.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Technical error? Theorem 1 and 2: $R_{max}$ is defined as the max of the rewards but it seems that the reward function is not defined as strictly positive and I thus think that a factor 2 is missing in the bounds (see Apeendix C from the first line to the second line of the proof).\n- (minor) a few elements are not defined such as $\\rho_0$ end of page 3\n- (would be nice to have) study of the importance of using Equation 7 in the pipeline", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wjG1Evx512", "reviewer": "Reviewer_JQsb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- LAIfO tackles an interesting problem - imitation learning without full-observability or access to expert actions. This is an important problem, especially in robotics, where collecting data with robots is expensive. Leveraging passive videos to bootstrap policy learning has a lot of future potential. \n- LAIfO achieves efficiency gains over PatchAIL and VMAIL in DMC. These results were averaged over 6 random seeds. The authors also promise to release their code, so these results should be easily reproducible. \n- The experiments include a good set of ablations that investigate how LAIfO performs in the absence of expert actions and partial observability. Typically, papers only benchmark against methods with similar assumptions, so it’s nice to see comparisons to methods beyond these assumptions, i.e. fully-observable and other BC baselines. \n- The introduction does a good job of motivating key problems and positioning the paper with respect to related work and methods. \n\nWeaknesses:\n- The connection between the theoretical and empirical result is poor. In practice, LAIfO just uses frame-stacking (Mnih et al 2013) and data augmentation (Laskin et al, 2020)? Both these techniques are commonly used in both IL and RL methods to provide history and learn generalizable state representation. Is there anything empirically novel that makes LAIfO effective?\n- All experiments are limited to the DMC Control Suite. While learning directly from videos can be very powerful, using DMC rollout videos to train DMC agents doesn’t show the true potential (or limitation) of the proposed approach. Did the authors consider training across simulators or domains? Like training on MetaWorld (Yu et al, 2019) and evaluating on RLBench (James et al, 2020)? \n- Computational efficiency is not the biggest concern for DMC tasks. What if PatchAIL was scaled-up with more compute and lots of parallel DMC environments? Additionally, LAIfO requires an expert DDPG agent that was presumably trained with even more environmental interactions, again inside DMC. Efficiency would be a concern if agents were trained in the real-world, where environmental interactions are expensive and dangerous, but LAIfO is only set in a simulated setting. \n- The ablation experiments could benefit from an investigation on how the number of demonstrations affects performance. It seems like only N=100 was investigated. Is LAIfO competitive with PatchAIL in low-data and large-data regimes?\n- The experiments are missing a simple BC baseline. Training a BC agent on DDPG rollouts assumes access to expert actions, but nonetheless, it could provide a reference point as the simplest imitation learning algorithm for comparison. \n\nReferences:\n-  Yu et al, 2019 https://arxiv.org/abs/1910.10897\n- James et al, 2020 https://arxiv.org/abs/1909.12271", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents LAIfO, an imitation-learning algorithm for learning from action-less observations. LAIfO is set in an adversarial setting like GAIL (Ho et al.), where the objective is to generate trajectories against a discriminator that tries to discern between expert and learned policies. LAIfO addresses two key problems: partial-observability and no access to expert actions. Based on some theoretical insights, these two problems are addressed with (1) frame-stacking, and (2) data augmentation of observations. All evaluations are conducted in the DMC Control Suite. Simulated experiments indicate that LAIfO matches the performance of PatchAIL-W and VMAIL (prior state-of-the-art methods), while being more efficient. Further ablation studies show that LAIfO is almost as effective as agents with full observability and access to expert-actions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Summarizing points from above:\n- Make the connection between the theoretical and empirical result more explicit. \n- Explain the novel contributions of LAIfO in terms of practical implementations. \n- Show experiments beyond DMC, across domains or simulators.\n- Explain why efficiency is a key concern in DMC.\n- Add an ablation experiment on how the number of demos affects performance.\n- Add a simple BC baseline for reference.", "broader_impact_concerns": "Nothing of immediate concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JBUjsQPuuY", "reviewer": "Reviewer_PpXg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The presented approach of learning a latent representation and imitation learning based on the latent state is sound.\n2. The proof appears to be correct, with detailed lemma and remarks.\n3. The claims are substantiated, and main limitations (e.g. that expert and learner act in the same POMDP) are clearly communicated.\n4. The presentations are mostly clear and easy to follow.\n\nWeaknesses:\n1. While learning from observations is an important problem, the current work should discuss more clearly its potential applications. As it stands, the current setup does not address the correspondence problem in the sense that the agent does not learn to convert expert demonstrations (i.e., input from 3rd-person perspective) into an egocentric policy (i.e., policy from 1st-person perspective). While I understand that the correspondence problem is a separate issue, I believe that the current work should discuss more clearly the potential applications for the existing setup and the proposed algorithms.\n\n2. The LAIL algorithm is unclear. The algorithm variant with access to expert actions is barely discussed in the paper and only introduced at the start of the experiment. It is unclear what changes are needed to convert LAIfO to LAIL. This needs to be elaborated.\n\n3. It is unclear why the LAIL algorithm does not outperform LAIfO. From Table 2 and 3, it appears the LAIL outperforms LAIfO for about half the tasks while losing the rest. However, LAIL appears to provide strictly more information to the agent than LAIfO. It is rather puzzling why LAIL does not perform strictly better and could the authors clarify on this?\n\n4. Clarification on the learned reward function. The proposed algorithm appears to use log(D(z, z’)) as the reward function during the RL step, but uses D(z, z’) for the critic update and RL + LAIfO. Could the authors further discuss why two forms of the reward functions are needed?\n\n5. RL + LAIfO. The results appear misleading as Fig 4 only shows the initial part of the training and the performance there is nowhere near the expert performance. In addition, the comparison is fundamentally unfair since RL + LAIfO has additional expert demonstrations compared to other methods. Possible ways to make comparison fairer include 1) initialize RL agent with behavioral cloning before applying existing RL methods, or incorporating existing methods with approaches such as Deep Q-Learning from Demonstrations [1]. Lastly, DrQ-v2 results are significantly worse than what is being reported in the original paper and this should be discussed in the manuscript.\n\n     The RL + LAIfO sections are rather limited and do not add much to the central theme of the paper. The space for it could be better used for more ablation on LAIL and LAIfO. See below.\n\n6. The experiment results do not fully address why LAIfO performs better than existing methods. For instance, compared to VAMIL, the key difference for LAIfO appears to be how the latent representation is learned. Consequently, a natural question would be what happens to LAIfO if it adopts the latent representation learning from VMAIL, while keeping all the other components unchanged. Given the modular nature of the proposed algorithms, many important ablations should be performed, (e.g., matching the RL algorithm with those used by VAMIL or PatchAIL, adding data augmentation to VAMIL or PatchAIL). Such ablations would greatly improve the understanding of the proposed method.\n\n[1] Hester, Todd, et al. \"Deep q-learning from demonstrations.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an adversarial imitation learning algorithm from visual observations of expert demonstrations. Two variants of the algorithm are introduced, with one (named LAIL) having access to expert actions and one (named LAIfO) without. For LAIfO, the paper shows that, under certain assumptions, the suboptimality of the imitation learning agent is bounded by the divergence of the state-transition between the agent and expert. The paper presents several comparisons with state-of-the-art algorithms and demonstrates improvements in the majority of the experiments.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the requested changes in the section above.", "broader_impact_concerns": "N.A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I am recommending that the paper be accepted with minor revisions, however, it is worth mentioning that it is unclear whether minor improvements in the proposed method are sufficient to exceed the performance of state of the art methods. In particular, I thought the following issues should still need to be addressed in the latest draft of this submission:  

1. The central assumption that the learned embedding has to be predictive of the real state of the system seems like it is a heavy assumption. How does one guarantee that the latent representation indeed satisfies this property, without ever having access to the true state? 

2. Right before section 4 in the paper, it is necessary to clarify how the spaces Z and S differ, and whether the policy has the form  pi(a|z) or pi(a|x), where x is the observation.

3. In Eqn 2, where do we get the model p(z' | x, a z) and why is x not redundant in this model?

4. In Eqn 9, the reward has access to the state s, but the imitation learning objective does not. How do we know s?

5. The fact that the proposed algorithms have better runtime efficiency than existing algorithms that work directly on observations is not surprising at all, since they are working learning on embeddings and not the images themselves. Please remove this as one of the main contributions of the paper.

6. Fig 1 plots performance (episode return) as a function of wall-clock time. In my opinion, this is quite unusual in deep RL, and what matters more is data efficiency. Looking at the analogous figure in Appendix E, Fig 6, which shows episode return as a function of training steps, we observe very little difference between the proposed method and PatchAIL, the state of the art baseline that is being compared with. Please move the relevant appendix figures to the main body of the paper.

7. The results of the baselines used in this paper seem to be sensitive to the number of expert demonstrations (see Table 4). I would like to see the # of expert demonstrations be listed for every comparison made in the paper between the proposed method and existing baselines. 

8. Unclear how normalization was done in Fig 3. Please clarify.

9. I'd also like to see a few more references in the related works discussion:

- Adversarial Imitation Learning from Video Using a State Observer, Karnan et al

- SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards, Reddy et al
Decision: Accept (poster)