Visual Adversarial Imitation Learning using Variational Models
Authors: Rafael Rafailov
Tianhe Yu
Aravind Rajeswaran
Chelsea Finn
Venue: NeurIPS 2021 Poster (2021)
Keywords: imitation learning
visual imitation learning
deep reinforcement learning

Abstract: Reward function specification, which requires considerable human effort and iteration, remains a major impediment for learning behaviors through deep reinforcement learning. In contrast, providing visual demonstrations of desired behaviors presents an easier and more natural way to teach agents. We consider a setting where an agent is provided a fixed dataset of visual demonstrations illustrating how to perform a task, and must learn to solve the task using the provided demonstrations and unsupervised environment interactions. This setting presents a number of challenges including representation learning for visual observations, sample complexity due to high dimensional spaces, and learning instability due to the lack of a fixed reward or learning signal. Towards addressing these challenges, we develop a variational model-based adversarial imitation learning (V-MAIL) algorithm. The model-based approach provides a strong signal for representation learning, enables sample efficiency, and improves the stability of adversarial training by enabling on-policy learning. Through experiments involving several vision-based locomotion and manipulation tasks, we find that V-MAIL learns successful visuomotor policies in a sample-efficient manner, has better stability compared to prior work, and also achieves higher asymptotic performance. We further find that by transferring the learned models, V-MAIL can learn new tasks from visual demonstrations without any additional environment interactions. All results including videos can be found online at https://sites.google.com/view/variational-mail

Visual Adversarial Imitation Learning using Variational Models
Rafael Rafailov! Tianhe Yu
Aravind Rajeswaran 2.3 Chelsea Finn' (rafailov, tianheyu, [ofinn]@sstafooln edu, aravraj0fb. com com Stanford University, University of Washington, 3 Facebook Al Research
Abstract
Reward function specification, which requires considerable human effort and itera- tion, remains major impediment for learning behaviors through deep reinforce- ment learning. In contrast, providing visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We consider a setting where an agent is provided a fixed dataset of visual demonstrations illustrating how perform task, and must learn to solve the task using the provided demonstra- tions and unsupervised environment interactions. This setting presents number of challenges including representation learning for visual observations, sample com- plexity due to high dimensional spaces, and learning instability due to the lack of a fixed reward or learning signal. Towards addressing these challenges, we develop a variational model- based adversarial imitation learning (V-MAIL.) algorithm. The model-based approach provides strong signal for representation learning, enables sample efficiency, and improves the stability of adversarial training by enabling on- policy learning. Through experiments involving several vision-based locomotion and manipulation tasks, find that V-MAIL learns successful visuomotor policies in a sample-effice efficient manner, has better stability compared to prior work, and also achieves higher asymptotic performance. We further find that by transferring the learned models, V-MAIL can lear new tasks from visual demonstrations without any additional environment interactions. All results including videos be found online at peetps://siter s/tte///oossssslelleeeeeeee cww/viiw/ www.iiiwwaaaiat...o onal onal-man -mail.
Introduction
The ability of reinforeement learning (RL) agents to autonomously learn by interacting with the environment presents a promising approach for learning diverse skills. However, reward specification has remained major challenge in the deployment of RL in practical settings [1. 2. 3]. The ability imitate humans or other expert trajectories allows us to avoid the reward specification problem, while also circumventing challenges related to task- task-specific exploration in RL. Visual demonstrations can also be more natural way teach robots various tasks and skills real-world applications. However, this setting is also fraught with number of technical challenges including representation learning for visual observations, sample complexity due to the high dimensional observation spaces, and learning instability [4, 5, 6] due to lack of a stationary learning signal. We aim to overcome these challenges and to develop an algorithm that can learn from limited demonstration data and scale to ligh-dimensional observation and action spaces often encountered in robotics applications.
Behaviour cloning (BC) is a classic algorithm imitate expert demonstrations [7], which uses supervised learning to greedily match the expert behaviour at demonstrated expert states. Due to environment stochasticity, covariate shift, and policy approximation error, the agent may drift away from the expert state distribution and ultimately fail to mimic the demonstrator [8]. While wide initial state distribution [9] or the ability interactively query the expert policy [8] can cireumvent
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Expert Trajectory
Model training full replay buffer of unsupervised environment interactions
Dy
D2
D2
B Fg
Policy rollout in model latent space
Figure I: Left: the variational dynamics model, which enables joint representation learning from visual inputs and latent space dynamics model, and the discriminator which is trained to distinguish latent states of expert demonstrations from that of policy rollouts. Dashed lines represent inference and solid lines represent the generative model. Right: the policy training, which uses the discrim- inator as the reward function, so that the policy induces a latent state visitation distribution that is indistinguishable from that of the expert. The learned policy network is composed with the image encoder from the variational model to recover a visuomotor policy.
these difficulties, such conditions require additional supervision and are difficult to meet in practical applications. An alternate line of work based on inverse RL [10, 11] and adversarial imitation learning [12, 13] aims to not only match actions at demonstrated states, but also the long term state visitation distribution of the expert [14]. Adversarial imitation learning approaches explicitly train a GAN-based classifier [15] to distinguish the visitation distribution of the agent from the expert, and use substantial improvement over behaviour cloning without additional expert supervision, they are difficult to deploy in realistic scenarios for multiple reasons: (I) the objective requires on-policy data collection leading to high sample complexity; (2) the reward function changes as the RL agent learns; and (3) high-dimensional observation spaces require representation learning and exacerbate the optimization challenges.
reward signal for training the agent with RL. While these methods have achieved
Our main contribution in this work is the development of algorithm, variational model-based adversarial imitation learning (V-MAIL), which aims to overcome each of the aforementioned chal- lenges within single framework. As illustrated in Figure V-MAIL. trains variational latent-space dynamics model and a discriminator that provides a learning reward signal by distinguishing latent rollouts of the agent from the expert. The key insight of our approach is that variational models can address these challenges simultancously by (a) making it possible to collect on-policy roll-outs inside the model without environment interaction, leading an efficient and stable optimization process and (b) providing rich auxiliary objective for efficiently learning compact state representations and which regularizes the discriminator. Furthermore, the variational model also allows V-MAIL to perform zero-shot transfer to new imitation learning tasks. By generating on-policy rollouts within the model, and training the discriminator using these rollouts along with demonstrations of new task, V-MAIL. can learn policies for new tasks without any additional environment interactions.
Through experiments vision-based locomotion and manipulation tasks, we find that V-MAIL can successfully learn visuomotor control policies from limited demonstrations. In particular, V-MAIL exhibits stable and near-monotonic learning, highly sample efficient, and asymptotically matches the expert level performance on most tasks. In contrast, prior algorithms exhibit unstable learning and poor asymptotic performance, often achieving less that 20% of expert performance on these vision- based tasks. We further show the ability to transfer the model to novel tasks, acquiring qualitatively new behaviors using only a few demonstrations and additional environment interactions.
2 Preliminaries
We consider the problem setting of learning in partially observed Markov decision processes (POMDPs), which can be described with the tuple: M (S, X, R, T.T.T. T,,, where S

---

is the state space, a E A is the action space, X' is the observation space and R a is reward function. The state evolution is Markovian and govemnd by the dynamics as s' ~ s. Finally, the observations are generated through the observation model æ ~ ((-ss)) The widely studied Markov decision process (MDP) a special case of this 7-tuple where the underlying state directly observed in the observation model.
In this work, study imitation learning in unknown POMDPs. Thus, we do not have access the underlying dynamics, the true state representation of the POMDP, or the reward function. In place of the rewards, the agent is provided with a fixed set of expert demonstrations collected by executing an expert policy x², which we assume is optimal under the unknown reward function. The agent can interact with the environment and must learn policy T( (x<<() that mimics the expert.
2.1 Imitation learning as divergence minimization
In line with prior work, we interpret imitation learning as a divergence minimization problem [12, 14, 16]. For simplicity of exposition, we consider the MDP case in this section, and discuss POMDP extensions in Section 3.2. Let PM (s, a) = (1 -?) Er P(s, a = a) be the discounted state-action visitation distribution of a policy π MDP M. Then, divergence minimization objective for imitation learning corresponds to
min D(P
where p is the discounted visitation distribution of the expert policy nh and D is divergence measure between probability distributions such as KL-divergence, -ensen-Shannon divergence, or generic f divverggence. To see why this is a reasonable objective, let ( denote the expected value of policy π in M. Inverse RL [17, 12, 13] interprets the expert as the optimal policy under some unknown reward function. With respect to this unknown reward function, the sub-optimality of any policy can be bounded as:
[( M -
< Rmaxx Drv (PM. 1-7 MM- J JO, MT/
since the policy performance is (1 -)) ( () [r(s,a) a)). We use Drv denote total variation distance. Since various divergence measures related to the total variation distance, optimizing the divergence between visitation distributions in state space amounts to optimizing a bound on the policy sub-optimality.
2.2 Generative Adversarial Imitation Learning (GAIL)
With the divergence minimization viewpoint, any standard generative modeling technique including density estimation, VAEs, GANs etc. in principle be used minimize Eq. 1. However, in practice, use of certain generative modeling techniques can be difficult. A standard density estimation technique would involve directly parameterizing P34 say through auto-regressive flows, and learning the density model. However, policy that induces the learned visitation distribution in M is not guaranteed to exist and may prove hard recover. Similar challenges prevent the direct application of a VAE based generative model as well. In contrast, GANs allow for a policy based parameterization. since it only requires the ability to sample from the generative model and does not require the likelihood. This approach was followed in GAIL,, leading to the optimization
max min ((,a)~= [- log (s a)] + E(a.))=o [- lloo log (1 (1- D)(s
(2)
where DR is a discriminative classifier used to distinguish between samples from the expert dis- tribution and the policy generated distribution. Results from Goodfellow et al. [15] and Ho and Ermon [12] suggest that the learning objective in Eq. 2 corresponds to the divergence minimization objective in Eq. with consen-Shanne divergence. In order estimate the second expectation in Eq. we require on-policy samples from π, which is often data-inefficient and difficult to scale to high-dimensional image observations. Some off off-policy algorithms [18, 19] replace the expectation under the policy distribution with expectation under the current replay buffer distribution, which allows for off- off-policy training, but can no longer guarantee that the induced visitation distribution of the learned policy will match that of the expert.
3

---

3 Variational Model-Based Adversarial Imitation Learning
Imitation learning methods based on expert distribution matching have unique challenges. Improving the generative distribution of trajectories (through policy optimization, as we do not have control over the environment dynamics) requires samples from f. which requires rolling out я in the environment. Furthermore, the optimization landscape of a saddle point problem (see Eq. 2) can require many iterations of learning, each requiring fresh on-policy rollouts. This is different from typical generative modeling applications [15, 20] where sampling from the generator is cheap. To overcome these challenges, we present a model-based imitation learning algorithm. Model-based algorithms utilize a large number of syntheric on-policy rollouts using the learned dynamics model, with periodic model correction. auxiliary task for state representation learning, making policy learning easier and more sample efficient. For conceptual clarity and ease of exposition, we first present our conceptual algorithm in the MDP setting in Section 3.1, and then extend this algorithm the POMDP case in Section 3.2. Finally, we present a practical version of our algorithm in Sections 3.3 and 3.4.
addition, learning the dynamics model serves as a rich
3.1 Model-Basee Adversarial Imitation Learning
Model-based algorithms for RL and IL involve learning an approximate dynamics model I using environment interactions. The learned dynamics model can be used to construct an approximate MDP M. our context of imitation learning. learning a dynamics model allows us to generate samples from M as a surrogate for samples from M, leading to the objective:
min (p PM)
(3)
which can serve as good proxy to Eq. as long as the model approximation is accurate. This intuition can be captured using the following lemma (see the appendix for proof).
Lemma 1. (Simultaneous policy and model deviation) Suppose have a-approximate dynamies
model given by TDr (T a), T(s, a)) S a V(s, a). Let Rmax max().a) R(s, a) be the maxi- mum of the unknown reward in the MDP with unknown dynamics T. For any policy we can bound the sub-optimality with respect to the expert policy T as:
(TV(P p + a. Rmax 1--
Rmax
J( (M) - (TT
(4)
21-
Thus, the divergence minimization in Eq. 3 serves as an approximate bound on the sub-optimality with bias that is proportional to the model error. Thus, we ultimately propose to solve the following saddle point optimization problem:
max min Do
B(Sa)~EN [- log (s a)] + (( llog (1- D (s,(
(5)
which requires generating on-policy samples only from the learned model M. We can interleave policy learning according to Eq. 5 with performing policy rollouts in the real environment to iteratively improve the model. Provided the policy is updated sufficiently slowly, Rajeswaran al. [21] show that such interleaved policy and model learning corresponds to stable and convergent algorithm, while being highly sample efficient.
3.2 Extension POMDPs
In POMDPs, the underlying state is not directly observed, and thus cannot be directly used by the policy. In this case, we typically use the notion of belief state, which is defined to be the filtering distribution P(s)(h,), where we denote history with h, := (æ act). By using the historical information, the belief state provides more information about the current state, and can enable the learning of better policies. However, learning and maintaining an explicit distribution over states can difficult. Thus, we consider learning a latent representation of the history 2, = q(h,), so that P(s.hh: x P().)zz). To develop an algorithm for the POMDP setting, we first make the key observation that imitation learning in POMDPs can reduced to divergence minimization in the latent belief state representation. To formalize this intuition, introduce Theorem 1.

---

Algorithm V-MAIL: Variational Model-Based Adversarial Imitation Learning
1: Require: Expert demos Bg, environment buffer B..
2: Randomly initialize variational model {qu, To policy and discriminator De.
3: for number of iterations do
4: // Environment Data Collection
5: for timestep 1
6:
Estimate latent state from the belief distribution 2, ~ ga(-|201, 11 ai-1)
7:
Sample action a ~ TT [a)
8:
Step environment and get observation #+++1
9:
Add data (Z1:T, Q1:T-1} to policy replay buffer B, for number of training iterations do
10:
// Dynamics Learning Sample batch of trajectories (m1-T. Q1-T-1} from the joint buffer Bg U B
Optimize the variational model {ga, To using Equation 7
14:
// Adversarial Policy Learning
15:
Sample trajectories from expert buffer {æi.r. 13 Bg
16:
Infer expert latent states #1:7 ~ ge(-|affr. air--) using the belief model qo
17:
Generate latent rollouts TT using the policy Ty from forward model T
18:
Update the discriminator D3 with data zf.r. 21:Ht using Equation 6
Update policy #y to improve the value function in Equation
Theorem 1. (Divergence in latent space) Consider a POMDP M, and let 2t be a latent space representation of the history and belief state such that P(s, Izt act)) P(s,lz)) Let the policy class be such that a, ~ <(-lZ)), ) so that P(sr|zz, ax) = P(s z,). Let Dj be generic fidieergence. Then the following inequalities hold:
Dalema(a ()llp a)) < Dy(pMu(s, ()) ( a)) < ())
The condition P(s zr, az)) P()((z)) essentially states that the actions of both the agent and the expert do not carry additional information about the state beyond what is available in the history. This will be true of all agents trained based on some representation of the history. and only excludes policies trained on ground truth states. Since we cannot hope compete with policy classes that fundamentally have access to more information like the ground truth state, we believe this is benign assumption. Theorem 1 suggests that the divergence of visitation distributions in the latent space represents an upper bound of the divergence in the state and observation spaces. This is particularly useful, since we do not have access to the ground-truth states of the POMDP and matching the expert marginal distribution in the high-dimensional observation space (such images) could be difficult.
Furthermore, based on the results in Section 2.1, minimizing the state divergence results in minimizing a bound on policy sub-optimality as well. These results provide a direct way to extend the results from Section 3.1 to the POMDP setting. If we can learn encoder z; g( a<4)) that captures sufficient statistics the history, and a latent state space dynamics model 1++1 ~ [- at), then can learn the policy by extending Eq. to the induced MDP in the latent space
max min ((E.a))pE((E [- log Do a)] + E(E.a)-0 Do
[- log (1 D)(z (6)
Once learned, the policy can be composed with the encoder for deployment in the POMDP. Similar approach was also taken by [22], however they only use the model for representation purposes in low-dimensional domains and do not carry out model-based training.
3.3 Practical Algorithm with Variational Models
The divergence bound of Theorem allows to develop practical algorithm if we can learn good belief state representation. Following prior work [23, 24, 25, 26, 27, 28] we optimize the ELBO:
max Ego
log (z)(2)))) DKL 21 a Σ 1=1
a.-))
(7)
reconstmution
forward model
5

---

Multi-Task Replay Buffer
Multi-Task Imitation
Learned Model
New Task
Environment roll-cuts
Policy Training
Palicy Training
Training Phase
Transfer Phase
Figure 2: Illustration of our transfer learning approach. In the training phase, we learn a multiple tasks with a shared replay buffer and model. Subsequently, in the transfer and evaluation phase, the agent learns new task using expert demonstrations and the learned model, without any additional interactions with the environment.
where do is state inference network, To is latent dynamics model and i an observation model. Here we jointly train a belief representation with network qo and latent dynamics model To Given the learned latent model we can use any on-policy RL algorithm to train the policy using Eq. 6, however in our setup, the RL objective differentiable function of the policy, model, and discriminator parameters. We can then optimize the policy by directly back-propage pooppapppgggiing through To using the objective:
W+K-1 Σ
max Vik T
Vo,g (2,) max E Tig
--- log Dulzt aar) + +
(8)
Finally, we train the discriminator D,, using Eq. with on-policy rollots from the model ÿ. Our full approach is outlined Algorithm I, for more details, see the appendix.
3.4 Zero-Shot Transfer to New Imitation Tasks
Our model-based approach is well suited to the problem of zero-shot transfer to new imitation learning tasks, i.e. transferring to new task using modest number of demonstrations and additional samples collected in the environment. In particular, we assume a set source tasks (T',, each with buffer of expert demonstrations B'g. Each source task corresponds to different POMDP with different underlying rewards, but shared dynamics. During training, the agent can interact with each source environment and collect additional data. At test time, we' 'e introduced with new target task T with corresponding expert demonstrations Bg and the goal is to obtain a policy that achieves high reward without additional interaction with the environment.
Our proposed method is illustrated in Fig. 2. The key observation is that we can optimize Eq. under our model and still obtain an upper bound on policy sub- sub-optimality via Eq. 4. Furthermore, the sub-optimality is bound by the accuracy of our model over the marginal state-action distribution of the target task expert. Specifically, training a single shared variational model across the tasks. By fine- tuning that model on data that includes the target task expert demonstrations hope is that can get accurate model and thus a high-quality policy. Similarly to Algorithm 1, we then train a discriminator and policy for the target task using only model rollouts. This approach outlined in Algorithm 2.
first train all of the source tasks using Algorithm
Algorithm 2 Zero-Shot Transfer with V-MAIL
1: Require: Expert demos Byg for each source task, expert demos B5 for target task
Randomly initialize policy Ty. and discriminator D
3: Train Alg on source tasks, yielding shared model {go, To and aggregated replay buffer B,
4: for number training iterations do
5: // Dynamics Fine-Tunina using Expert Trajectories
Update variational model {go. To} using Equation with data from Bg Ba
7: // Adversarial Policy Learning
8: Update discriminator Dy and policy with Equations 6 and 8.

---

Cheetah Run
walker Walk
Car Racing
608
880
Return
200
400
208
0
9
280
480
280
488
288 Samp les (x10²)
489
Samples (x10²)
Samples (x10²)
Claw Rotate
Booding Balls
1860
12.5
500
18.0
Mmm
Return
2.5
-1800
50 75
180
360
408
Samples (x107)
Samples (x107)
DA SQIL
DA- DAC
Behaviour Cloning
Expert
Figure 3: Learning curves showing ground truth reward versus number of environment steps for V-MAIL (ours). prior model-free imitation learning approaches, and behavior cloming on five visual imitation tasks. Wc find that V-MAIL consistently outperforms prior methods in terms final performance and stability, particularly for the first four environments where V-MAIL reaches near-expert performance. In the most challenging visual Baoding Balls task, which is notably difficult even with ground-truth state, only V-MAIL is able to make some progress, but all methods struggle. Confidence intervals shown with SE over 6 runs.
4 Experiments
In our experiments, we aim to answer four questions: (1) can V-MAIL successfully solve environments with image observations, (2) how does V-MAIL compare seate-of-the-are model-free imitation approaches, (3) can V-MAIL solve realistic manipulation tasks and environments with complex physical interactions, and (4) can V-MAIL enable zero-shot transfer new tasks? All experiments were carried out on single Titan RTX GPU using an internal cluster for about 1000 GPU hours.
4.1 Single-Task Experiments
Comparisons. To answer question (2), we choose to compare V-MAIL to model-free adversarial and non-adversarial imitation learning methods. For the former, we choose DAC [18] as a representative approach, which we equip with DrQ data augmentation for greater performance on vision- based tasks. For the latter, we consider SQIL [29], also equipped with DrQ training. We refer to each approach with data augmentation as DA-DAC and DA-SQIL respectively. Both of these methods are off- off-policy algorithms, which we expect to considerably more sample efficient than on-policy methods like GAIL [12] and AIRL [11]. For implementation details, the appendix.
Environments and Demonstration Data. To answer the above questions, we consider the five visual control environments. These consist of two locomotion environments from the DeepMind Control Suite [30], the classic Car Racing environment from OpenAl Gym [31] and two dexterous manipulation tasks using the D' Olaaw [32] and Shadow Hand platforms. For full details on these environments and the expert data, see the appendix.
Results. Experiment results are shown in Figure 3. To answer questions (1) and (2), we compare V- MAIL to DA-SQIL and DA-DAC on the Cheetah and Walker tasks. We find that V-MAIL efficiently and reliably solves both tasks; in contrast, the model-free methods initially outperform V-MAIL, but their performance has high variance across random seeds and exhibits significant instability. Such stability issues have also been observed by Swamy et al. [33], which provides some theoretical explanation in the case of SQIL and the suggestion of early stopping as mitigation technique. In the case of DAC, the reasons for instability are less clear. Motivated by instability we observed in the critic loss for DADDAC, we experimented with a number of mitigation strategies in attempt
7

---

Cheetah Run
Walker Walk
Wa Walker Walk
600
888
Retura
688
686
480
400
200
209
208
6
200
480
200
480
200
400
Samp les (x10*)
Samples (x10²)
Samples (x10²)
10 demos
demos
demo
VDAC
BC (10 demos)
Expert
Figure 4: Ablation experiments for the VMAIL model. The left and middle graph show the efficacy of learning from different number of demonstrations where VMAIL outperforms baseline models even with single demo. The final figure compares VMAIL to model which used the variatonal model for representation purposes only (labeled VDAC). We see that VMAIL achieves 30% higher sample efficiency. Confidence intervals are shown with I SE over runs.
improve DA-DAC, including constraining the discriminator, varying the buffer and batch sizes, and separating the convolutional encoders of the discriminator and the actor/critic; however, these techniques didn't 't fully prevented the degradation performance.
On the Car Racing environment, we find that DA-SQIL and DA-DAC can reach or outperform behavior cloning, but struggle to reach expert-level performance. In contrast, V-MAIL stably and reliably achieves near-expert performance in about 200k environment steps. Note that Reddy et al. [29] report expert-level performance on this task, but an easier setting with double the number of expert demonstrations available (20 vs. 10). Given that tracks are randomly generated per episode demanding significant generalization, is not surprising that the problem becomes considerably more difficult with only 10 demonstrations.
Finally, answer question (3), consider the D' D'Claw and Baoding Balls tasks. In the D' D'Claw environment, SQIL fails to make progress, while DA-DAC makes significant progress initially but quickly degrades. V-MAIL. solves the task in less than 100k environment steps. In the most challenging visual Baoding Balls problem, involving a 26-dimensional control space, V-MAIL. the only algorithm reach any success.
4.2 Ablation Experiments
Results for our ablation experiments are shown in Fig. 4.
Number of Demonstrations. In the first set of experiments evaluate VMAIL's capability to learn from a limited number of demonstrations. We deploy our model on the two locomotion environments using 10, 5 and a single demonstration trajectory. VMAIL shows only minor deterioration in performance on the Cheetah Run environment, as well as on the Walker Walk environment when using only 5 demos, but it struggles to reach expert-level performance the walker task when provided with a single trajectory. However, all cases VMAIL outperforms the asymptotic results of our baselines, which use the full 10 demonstrations.
Effect of Representation Learning. We also evaluate the effect of representation learning on model performance. In this scenario we still train variational model, but similar to [25] only use it for for representation purposes. We then train a standard Discriminator-Acoo the learned latent space, which we denote as Variational DAC or VDAC. Results are shown in the last graph of Fig. VDAC exhibits more stable performance as compared to learning directly from pixels, however it is still 30% less sample efficient than VMAIL.
Critic model on top of

---

4.3 Transfer Experiments
Transfer Scenarios. To evaluate V-MAIL's ability to learn new imitation tasks in zero-shot way (i.e. without any additional environment samples) we deploy Algorithm 2 on two domains: in locomotion experiment we train on the Walker Stand and Walker Run (target speed greater than 8) tasks and and evaluate transfer to the Walker Walk (target speed between 2 and 4) task from the Deep Control suite. In a manipulation scenario, we use set of custom D'Claw Screw tasks from the Robel suite [32]. We train our model on the 3-prong tasks with clockwise and counter-clockwise rotation, as well as the 4-prong task with counter-clockwise rotation and evaluate transfer to the 4-prong task with clockwise rotation.
Comparisons. We devise several points of comparison. First, we compare directly applying the policy learned in the most related source task to the target task. This tests whether the target task demands qualitatively distinct behavior. Second, we compare to an offline version of DAC, augmented with the CQL approach [34], where samples collected from the source task are used to update the policy, with the target task demonstrations used to learn the reward. Finally, we also compare to behavior cloning on the target task demonstrations (without leveraging any source task data), and an oracle that performs V-MAIL on the target task directly.
The results in Table 1. Policy transfer performs poorly, suggesting that the target task indeed requires qualitatively different behaviour from the few training tasks available. Further, behavior cloning on the target demonstrations is not sufficient to Iearn the task. Offline DAC also shows poor performance. Finally, we see that V- MAIL almost matches the performance of the agent explicitly trained on task, indicat- ing the learned model and the algorithm for training within that model can be used not just for efficient visual imitation learning, but also for zhro-shot transfer to new tasks.
Results.
Method
Walker Walk
Claw Rotate
Offine DAC
8.8%
-0.7%
Bchavior cloning
26.8%
8.3%
Policy transfer
21.3%
5.6%
V-MAIL (ours)
92.7%
97.9%
Target task IL (oracle)
98.2%
102.3%
Table I: Zero- shot transfer performance to new imitation learning task as percent of expert return. Each method provided with 10 demonstrations of the target task, and zero additional environment samples. V-MAII. can solve the target tasks within its learned model without any additional samples, while model-free transfer learning approaches fail.
Related Work
Imitation Learning. Recent model-free imitation learning can be categorized as either adversarial or non-adversarial. Adversarial methods inspired by GANs [15] train an explicit classifier between expert and policy behaviour and optimize the agent in two-player minimax game. GAIL [12] and AIRL [11] are two such algorithms; however they often have poor sample efficiency due to the requirement of on-policy rollouts in the environment. To address sample efficiency issues, off-policy variants such as DAC [18] and SAM [19] have been developed, however they suffer from an objective mismatch when using off off-policy data [35], often resulting in learning instability [36].
An alternate line of research attempts to forego adversarial training: SQIL [29] frames the problem as regularized behaviour cloning and trains an off policy algorithm with rewards of for expert trajectories and for policy ones. RCE [37] uses a very similar approach, but derives it through a different objective. ValueDICE [35] uses the same key result for iterative distribution matching as RCE to obtain an off-policy distribution matching algorithm. In Swamy et al. [33] the authors derive distribution matching as a bound on policy peder-performance, similar to our analysis in Section 3.1 and propose a practical non-adversarial algorithm AdVIL, however in reported experiments it does not outperform behaviour cloning. In Berseth et al. [38] the authors consider variational encoder, which embeds expert video demonstrations in a latent space, and optimize a mis-match objective using on-policy model-free training.
few papers have considered model-bases imitation learning as well: Baram et al. [39] is an adversarial algorithm conceptually similar to our approach, but only focuses on low-dimensional state-based tasks and train the discriminator using off-policy replay buffer, which does not allow it to generalize to new tasks. Related to our method is Finn et al. [10] which uses a similar reward learning in combination with locally linear dynamics model, which leads to trajectory-centric algorithms that cannot transfer the model to new tasks. Das et al. [40] considers similar setting for inverse

---

RL using simplified parameterization of the cost function. In this work we develop end-to-end model for adversarial imitation learning in high-dimensional POMDPs and generalization to novel tasks without hand-designed features. A related line of research is learning from observations alone (without access to expert actions). The BCO approach Torabi et al. [41] learns an inverse dynamics model to infer and copy expert actions, however it still suffers from the drawbacks of behaviour cloning. In a concurrent work [42] explore similar objective to [43] and VMAIL, using only states for training the discriminator objective, however they only focus on low dimensional domains. With a particular variational model specification VMAIL can be also be extended to work only with expert state observations, which we leave for further work.
Finally, related to zero-shot imitation learning, concurrent work by Chang et al. [43] extends a model-based offline RL algorithm [44, 45] to imitation learning. The focus in their work is primarily theoretical analysis, with empirical results in tasks with compact state representations. our work aims to develop a stable and efficient imitation learning algorithm that can handle high- dimensional observation spaces (like visual inputs), in both online and offline learning settings.
contrast,
Reinforcement Learning From Images with Variational Models. Reinforcement learning from images is inherently difficult task, since the agent needs learn meaningful visual representations to support policy learning. A recent line of research [26, 27, 25, 28, 46] train a variational model of the image-based environment as an auxiliary task, either for representation learning only [26, 25] or for additionally generating on on-policy data by rolling out the model [28]. Our method builds upon these ideas, but unlike these prior works, considers the problem of learning from visual demonstrations without access to rewards.
6 Conclusion
In this work we presented V-MAIL, model-based imitation learning algorithm that works from high-dimensional image observations. V-MAIL. learns model of the environment, which serves as strong seffsuupevi signal for visual representation learning and mitigates distribution shift by enabling synthetic on on-policy rollouts using the model. Through experiments, we find that V-MAIL achieves better asymptotic performance, is more stable, and matches the sample efficiency of prior model-free approaches. By effectively re-using the learned model, V-MAIL is also successful in zero-shot imitation learning, capable of learning new tasks using a small number of demonstrations, without any additional interactions with the environment.
Limitations. VMAIL uses low-dimensional dynamies model. explicitly trained with image prediction. This might make our method vulnerable to adversarial visual perturbations. Although successful in domains with complex dynamics, our approach relies on variational models with compact, single-level, latent state spaces. It is possible that such model class may not have sufficient capacity to represent complex scenes with multiple cluttered objects or deformable objects. A different selection of training objectives or further developments and improvements in variational generative modeling can potentially address both of these limitations.
Future Work. We believe this work opens the door for multiple potential developments. One direction is to train our procedure using only expert observations without access expert actions, which is an even more realistic scenario. This setup is quite difficult for model-free approaches, since expert actions usually serve as strong supervision. Another direction is to use on-policy model based rollouts to efficiently train other algorithms that inherently require on-policy data, such as multi-modal imitation [47, 48]. We showed the transfer capabilities of our algorithm to new tasks in a zero-shot imitation learning formulation. However, V-MAIL can in principle utilize any previously collected data for model-training, enabling potential applications in offline imitation learning in conjunction with offline RL algorithms like Rafailov et al. [46].
Acknowledgements
This work was supported in part byt ONR grants N00014-20-1-2677. and N00014-21-1-2685 as well as Intel Corporation. Chelsea Finn is a CIFAR Fellow in the Learning in Machines and Brains program. Part of this work was completed when Aravind Rajeswaran was at the University of Washington, where he was supported through the J.P. Morgan PhD Fellowship in AI (2020-21).
10

---

References
[1] Dario Amodei, Chris Olah, J. Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. ArXiv, abs/I 1aw./66066065555 2016.
[2] Tom Everitt and Marcus Hutter. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. ArXiv, abs/1908.00l 007344444 2019.
[3] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforce- ment Learning and Demonstrations. In Proceedings of Robolics: Science and Systems (RSS), 2018.
[4] Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre Yves Oudeyer. Auto- matic curriculum learning for deep rl: A short survey. ArXiv, abs/2003.0466m 2020.
[5] Khimya Khetarpal, Matthew Riemer, I. Rish, and Doina Precup. Towards continual reinforce- ment learning: A review and perspectives. ArXiv, 111990002.14490, 2020.
[6] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compeiityy environments. In NIPS, 2017.
[7] Dean A Pomerleau. Alvinn: an autonomous land vehicle in neural network. In Proceedings of the 1st International Conference on Neural Information Processing Systems, pages 305-313, 1988.
[8] Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. AISTATS, 2011.
[9] Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and J. Andrew Bagnell. Feedback in imitation learning: The three regimes of covariate shift. ArXiv Preprint, 2021.
[10] Chelsea Finn, Sergey Levine, and Picter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49-58. PMLR, 2016.
[11] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. International Conference on Learning Representations, 2018.
[12] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Conference on Neural Information Processing Systems, 2016.
[13] Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. ArXiv Preprint, 2016.
[14] Scyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimiza- tion perspective on imitation learning methods. Conference on Robot Learning, 2019.
[15] Ian J. Goodfellow, Jean Pouget-Abadie, M. Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
[16] Liyiming Ke, Mat Barnes, W. Sun, Gilwoo Lee, Sanjiban Choudhury, and S. Srinivasa. Imitation learning as f-divergence minimization. ArXiv, abs/ has//195.10 12888, 2019.
[17] Brian D. Ziebart, Andrew L. Maas, J. Bagnell, and A. Dey. Maximum entropy inverse reinforce- ment learning. AAAI, 2008
[18] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. eeteriiinntor-acto--cciie
Addressing sample inefficiency and reward bias in adversarial imitation learning. International Conference on Learning Representations, 2019.
[19] Lionel Blondé and Alexandros Kalousis. Sample-efficient imitation learning via generative adversarial nets. AISTATS, 2019.

---

[20] Andrew Brock, Jeff Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. ArXiv, abs/1809.11096 2019.
[21] Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. Game Theoretic Framework for Model-Based Reinforcement Learning. In ICML, 2020.
[22] Tanmay Gangwani, Joel Lehman, Qiang Liu, and Jian Peng, Learning belief representations for imitation learning in pomdps. Conference on Uncertainty Artificial Intelligence, 2020.
[23] Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. Conference on Neural Information Processing Systems, 2015.
[24] Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J. Johnson, and Sergey Levine. Solar: Deep structured representations for model-based reinforcement learning, Inter- national Conference on Machine Learning. 2019.
[25] Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Conference on Neural Information Processing Systems, 2020.
[26] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G. Bellemare. Deep- mdp: Learning continuous latent space models for representation learning. International Conference on Machine Learning, 2019.
[27] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. International Conference on Machine Learning, 2019.
[28] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. International Conference on Learning Representa- tions, 2020.
[29] Siddharth Reddy, Anca D. Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. International Conference on Learning Representations, 2020.
[30] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy Lillicap,, and Martin Riedmiller. Deepmind control suite, 2018
[31] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
[32] Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. ROBEL: RObotics BEnchmarks for Learning with low-cost robots. In Conference on Robot Learning (CoRL), 2019.
[33] Gokul Swamy, Sanjiban Choudhury, Zhiwei Steven Wu, and J. Andrew Bagnell. Of moments and matching: Trade-offs and treatments in imitation learning. International Conference on Machine Learning, 2021.
[34] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative g-learning for offline reinforcement learning. Conference on Neural Information Processing Systems, 2020.
[35] Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribu- tion matching. International Conference on Learning Representations, 2020.
[36] Lionel Blondé, Pablo Strasser, and Alexandros Kalousis. Lipschitzness all you need to tame off-policy generative adversarial imitation learning, 2020.
[37] Benjamin Eysenbach, Sergey Levine, and Ruslan Salakhutdinov. Replacing rewards with examples: Example-based policy search via recursive classification, 2021.
[38] Glen Berseth, Florian Golemo, and Christopher Pal. Towards learning to imitate from a single video demonstration, 2021.
12

---

[39] Nir Baram, Oron Anschel, and Shie Mannor. Model-based adversarial imitation learning. Conference on Neural Information Processing Systems, 2016.
[40] Neha Das, Sarah Bechtle, Todor Davchev, Dinesh Jayaraman, Akshara Rai, and Franziska Meier. Model-based inverse reinforcement learning from visual demonstrations. Conference Robot Learning, 2020.
[41] Faraz. Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. Interna- fional Joint Conference on Artificial Intelligence, 2018.
[42] Rahul Kidambi, Jonathan Chang, and Wen Sun. Mobile: Model-based imitation learning from observation alone, 2021.
[43] Jonathan D. Chang, Masatoshi Uchara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Miti- gating covariate shift imitation learning via offline data without great coverage, 2021.
[44] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In NeurlPS, 2020.
[45] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. CoRR, abs/2005.13234 2020.
[46] Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. arXiv preprint arXiv:2012.11547, 2020.
[47] Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. Conference on Neural Information Processing Systems, 2017.
[48] Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph Lim. Multi- modal imitation learning from unstructured demonstrations using generative adversarial nets, 2017.
13


Reviews:
[{"review_id": "yHfZy9BkeEW", "reviewer": "Reviewer_ESTA", "summary": "The authors propose V-MAIL, a variational model-based imitation learning algorithm. The transition dynamics model is learned by optimizing a relevant variational lower bound under the assumption of partial observability, while discriminators for imitation learning are trained over the latent-action space. The empirical results show that V-MAIL outperforms baselines including SQIL and DAC (where the data-augmentation technique is applied for both baselines). The authors additionally check the generalization ability (transfer learning) for V-MAIL. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although the proposed idea outperforms their baselines, I think more baselines and related works should have been considered, which makes me vote for rejection of this work.\n- The submission is not the first work considering a latent representation in imitation learning, e.g., Gangwani et al., \"Learning Belief Representations for Imitation Learning in POMDPs\". In Gangwani et al., the inequalities similar to Theorem 1 of this submission have been considered. \n- The submission is not the first work on image-based imitation learning, e.g., Pathak et al., \"Zero-shot visual imitation\", Torabi et al., \"Imitation Learning from Video by Leveraging Proprioception\", Liu et al., \"Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation\", etc. I couldn't write all here, but I could easily find out vision-based imitation learning algorithms that refer to GAIL paper in Google Scholar. (FYI, my keywords were \"visual based learning\", \"pixel\", \"image\" under the papers citing GAIL.) While most of them didn't consider model-based imitation, I think comparing V-MAIL with model-free vision-based imitation learning is necessary. (If not possible, authors need to state why some of those submissions cannot be compared with V-MAIL.)\n- The contribution seems marginal. The proposed method combines well-known approaches in vision-based RL to improve the performance of imitation learning. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No societal impact was concerend.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "tFEa3ifLiIn", "reviewer": "Reviewer_LYjg", "summary": "This paper introduces Variational model-based adversarial imitation learning (V-MAIL). V-MAIL considers a model-based setup for adversarial imitation learning: a latent dynamic model is learned and used to generate on-policy trajectories for training the RL policy generally used in the GAIL setup. It manages to achieve state-of-the-art performance on multiple tasks from visual observations, including MuJoCo, classic car racing, and two other robot manipulation tasks.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is well-written and easy to follow. Combining the model-based learning with GAIL-like imitation learning has indeed given extra benefit: it works like a regulizer for the policy, and provides nice on-policy rollouts for the policy learning. The experiment results also look quite promising. I think the interesting part of the paper is that it combines several interesting ideas and manages to prove that they can significantly improve the performance of imitation learning in the challenging scenarios. \n\nHowever, I still have certain concerns about the paper, which make me feel that the paper is not ready for publication yet. Please find the detailed comments below.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have certain concerns about the paper.\n\n1/ I think the contribution of the paper is a bit limited. V-MAIL combines several existing ideas, e.g., latent imagination, GAIL, variational state space model, etc., and achieves a good performance. However, how these components affect each other and how they contribute to the final performance are not clear. An ablation study is also missing from the paper. In this case, it would be hard to get inspiration from reading it.\n\n2/ Although the paper has provided theoretical analysis about the model-based adversarial imitation learning (sect. 4.1 and sect. 4.2), they are disconnected from the practical implementations (sect. 4.3). In particular, Theorem 1 shows that the divergence of the visitation distributions in a MDP can be upper bounded by a divergence of the visitation distributions in a POMDP. However, in the practical implementation, a variational state space model captures the belief, rather than the visitation distribution of the belief. In addition, it feels difficult to compute the visitation frequency of a belief, whose size is exponential to the size of the history and the state space. I believe the proposed algorithm indeed has its merit, but I don’t think Theorem 1 provides a correct justification of the optimization objective used in this paper.\n\n3/ I feel the author should be careful when making certain claims. For example, from line 39 to line 48, the authors are analyzing the limitations of the existing IRL methods and adversarial imitation learning methods. “These approaches explicitly train a GAN-based classifier [17] to distinguish the visitation distribution of the agent from the expert, and use it as a reward signal for training the agent with RL….” However, not all IRL methods are adversarial imitation learning. In fact, most of them don’t train a GAN-based classifier and do RL afterwards. Instead, a lot of them recover the reward and do planning instead.\n\n4/ The authors claimed that V-MAIL achieves zero-shot transfer to novel environments. However, the policy is fine-tuned with additional expert demonstrations, as shown in Alg. 2. Why is this zero-shot? In addition, I guess the transferability might be limited by the real difficulty of the source task / target task. Walker-run is clearly harder than walker-walk, so the policy transfer here is possible. Also, for the manipulation scenario, 3-prong task with both clockwise / counter clockwise rotations, together with one 4-prong task, actually provides sufficient information about the target task. I guess it might be difficult to transfer a policy from simpler tasks to more complex tasks. This has to be made clear in the paper. Otherwise, it is quite misleading.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "o3nzZDXeSil", "reviewer": "Reviewer_K7pU", "summary": "The authors present a novel method for imitation learning from a limited set of expert demonstrations. The method is heavily based on the DREAMER model-based RL framework and extends this with a discriminator to match a learner policy to an expert rollout.\nI find the paper overall well-written and a worthwhile contribution but there are some technicalities holding the paper back from being a top submission. I am overall still in favor of getting this work accepted since I believe most things can be addressed easily and I'm happy to change my rating if the authors can address some of my concerns.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** \n\nThe work combines a GAIL-like discriminator between expert and agent trajectory with a much more sample-efficient world model-style approach, for which the authors used DREAMER. There are many other trajectory embedding approaches to imitation learning, most of which are cited or even used as a baseline but one noteworthy omission is VIRL [1], which has a similar network structure but lower sample efficiency. I think it would be good to at least acknowledge that work in the related works section.\n\n**Quality:** \n\nThe method is explained in great detail but to a fault. In my personal opinion, the POMDP formulation is not necessary at all. While technically, visual control tasks in 3D should always be considered POMDPs, in most contemporary works, they aren't. And accordingly, I don't think you gained anything by including section 4.2 in the main body of the paper (rather than the appendix), and instead I would prefer more ablation studies. \n- Specifically, I'd be curious to see the influence of the number of expert samples on the training performance. \n- I also would like to see more than 3 seeds in the result plots since there is such an incredibly high variance. Please see this tutorial on picking a number of seeds that is statistically significant: https://arxiv.org/pdf/1806.08295.pdf\n- I think there is some issue in the implementation of the baseline methods: Neither SQIL nor DAC are nearly as unstable in their original publications as they are depicted here, which leads me to believe that the author-added DrQ modification may have had negative impacts on stability. This needs to be addressed urgently.\n\n**Clarity:** \n\nThe paper is clearly written and the authors get huge bonus points from me for including working code with instructions for running it.\n\n**Significance:** \n\nI think this work is a very meaningful extension of DREAMER and definitely a valuable new imitation learning method that does not need excessive environment interactions while learning. I didn't think about using model-based RL methods in this way, but I really like the idea.\n\n**Nitpicks and questions:**\n1. line 16 \"better stability compared to prior work\" - not at any statistical significance with only 3 seeds and not with the unexplained performance deterioration of the baselines\n2. line 26 ...circumventing exploration... - it doesn't really circumvent exploration. If the expert demonstration only covers a tiny part of the state space, the imitation learning has no chance of generalizing outside of that.\n3. Figure 1 right side is great and communicates all major points! The bold and dash lines on the left side are probably more confusing than anything. Maybe something like Fig 3a from the dreamer paper would be better.\n4. About the zero-shot/few-shot: I think this is only valid when both tasks share mostly the same dynamics, so the forward rollout model is identical given a state and action. But if you're going from one task to the other, e.g. driving as fast as possible while avoiding obstacles on CarRacing vs. making a lot of sharp turns in CarRacing, then I don't think this holds up. This wasn't really explored in the transfer section and probably needs a sentence. In your examples, the environment dynamics of walker-walk are included in walker-run and the environment dynamics of the 4-pronged D'Claw clockwise are too similar to the other tasks.\n5. line 237 \"scale\" - scale from what?\n6. line 241 1000 GPU hours total or each or how are they distributed?\n7. line 254 CarRacing-v0 is not terribly hard to solve and for example Ilya Kostrikov's PPO implementation (https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail/tree/master/a2c_ppo_acktr) can get to a mean episodic reward of ~550 and 700 within 500k and 1M steps respectively. \n8. Figure 3: \"consistently outperforms\" - Not sure this is correct, looking at the top performance, that's not true for Cheetah Run, and looking at the early performance, that's not true for Claw, Car Racing, and Walker.\n9. line 295 how does that work if you have 2 different action spaces - 3-pronged D'Claw has a smaller action space than 4-pronged D'Claw, right?\n10. line 334 \"Finally, the experiments suggest that this algorithm is efficient enough to be applied to real robots,\" - I don't know how this sentence is justified.\n\n\n**References:**\n\n- [1] Berseth, Glen, and Christopher J. Pal. \"Visual Imitation Learning with Recurrent Siamese Networks.\" arXiv preprint arXiv:1901.07186 (2019).), https://arxiv.org/abs/1901.07186", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would like to commend the authors for being the only one in my 6 reviews that actually followed instructions and put the limitations into their own little subsection in the conclusion. \nOne additional limitation that I can see is resilience to visual distractors. Since your state is based on image to image reconstruction, all pixels, including meaningless but moving background and other actors will be reconstructed and therefore have to be captured in the latent state. Just a single distractor agent that's doing something task-independent could throw this method off, no?", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "jY3wZNkfj4d", "reviewer": "Reviewer_uqtX", "summary": "Motivated by several challenges that face the use of adversarial imitation learning (AIL) in practice, the authors propose V-MAIL—a model-based algorithm for visual AIL. V-MAIL addresses the challenges of sample inefficiency and difficult numerical optimization by learning and leveraging both a variational observation model and a variational forward dynamics model. The authors provide/prove interesting theoretical performance bounds with respect to the proposed formulation, and propose V-MAIL as a practical solution to that formulation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS:\n\n(S1) The proposed approach is novel and proven effective empirically in standard benchmark domains.\n        \n(S2) The authors have provided a relevant and interesting theoretical analysis that motivates the proposed algorithm.\n    \nWEAKNESSES:\n\n(W1) The paper is missing discussion of important related literature in imitation from observation (IfO). While the authors should draw connection to the area as a whole, they also need to discuss the similarities/differences with BCO [Torabi et al., 2018] in particular, since it’s extremely similar. For example, BCO _also_ contains a model-learning phase followed by an imitation phase. To be clear, the proposed algorithm is unique in its own right, the authors just need to address the close relationship here.\n\nPOST-DISCUSSION COMMENTS:\n\nThanks to the authors for agreeing to revise the paper to make the connection to IfO more explicit. I'm happy to continue recommending acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I found the authors’ treatment of these topics in the paper to be sufficient.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents V-MAIL, a model-based algorithm for visual Adversarial Imitation Learning. V-MAIL learns both a variational observation model and a variational forward dynamics model. The algorithm is motivated by theoretical performance bounds, proposing the algorithm as a practical solution to the formulation. All the reviewers evaluated positive about the paper, and voted for acceptance. 
Decision: Accept (Poster)