Imitation Learning from Pixel Observations for Continuous Control
Authors: Samuel Cohen
Brandon Amos
Marc Peter Deisenroth
Mikael Henaff
Eugene Vinitsky
Denis Yarats
Venue: ICLR 2022 Submitted (2022)
Keywords: imitation learning
optimal transport
GAIL
adversarial learning

Abstract: We study imitation learning using only visual observations for controlling dynamical systems with continuous states and actions. This setting is attractive due to the large amount of video data available from which agents could learn from. However, it is challenging due to $i)$ not observing the actions and $ii)$ the high-dimensional visual space. In this setting, we explore recipes for imitation learning based on adversarial learning and optimal transport. A key feature of our methods is to use representations from the RL encoder to compute imitation rewards. These recipes enable us to scale these methods to attain expert-level performance on visual continuous control tasks in the DeepMind control suite. We investigate the tradeoffs of these approaches and present a comprehensive evaluation of the key design choices. To encourage reproducible research in this area, we provide an easy-to-use implementation for benchmarking visual imitation learning, including our methods.

Under review as a conference paper at ICLR 2022
IMITATION LEARNING FROM PIXEL OBSERVATIONS FOR CONTINUOUS CONTROL
Anonymous authors Paper under double-blee -oiid review
ABSTRACT
We study imitation learning using only visual observations for controlling dynami- cal systems with continuous states and actions. This setting is attractive due to the large amount of video data available from which agents could learn from. However, it is challenging due to i) not observing the actions and ir) the high-dimensional visual space. this setting, we explore recipes for imitation learning based on adversarial learning and optimal transport. A key feature of methods is use representations from the RL encoder to compute imitation rewards. These recipes enable visual continuous control tasks in the DeepMind control suite. We investigate the tradeoff's of these approaches and present comprehensive evaluation of the key design choices. To encourage reproducible research in this area, we provide an easy-to-use implementation for benchmarking visual imitation learning, including our methods'.
scale these methods attain expert-level performance on
INTRODUCTION
Learning continuous control policies directly from pixel observations is an important problem due to its potential impact on fields like robotics, autonomous driving and video games. These domains have rich resources of data available of humans performing expert-level demonstrations that our software agents do not leverage as they are often trained from scratch without any knowledge of how humans think about these problems. However, using unlabeled video data is challenging as requires distilling a representation of the world into the policy of an agent, and ii) we do not know the underlying actions and reasoning process of the expert. This renders common algorithms like canonical behavioral cloning (Pomerleau, 1988; 1991) useless in the no-action setting.
Recently the community has advanced our understanding learning visual representations and learning to imitate demonstrations provided as proprioceptive states. Visual representation learning has been crucial in recent advancements for sample-efficient reinforcement learning (RL) directly from pixels in continuous spaces, e.g. with reconstruction (Finn et al., 2016; Yarats et al., 2019), contrastive learning (Srinivas et al., 2020; Stooke et al., 2020), unsupervised pre-training (Liu & Abbeel, 2021; Yarats et al., 2021b; Seo al., 2021), world models (Hafner al., 2018; 2019; 2020) and data augmentation (Yarats et al., 2021c; Raileanu et al., 2020; Laskin et al., 2020).
These approaches require known reward signals from the environment, which not always available or well-defined. When expert demonstrations are available, imitation learning (IL) and inverse RL (IRL) methods overcome the issue of not having reward signal and seek to recover the expert agent (Ng & Russell, 2000). These methods are very effective when low-dimensional proprioceptive states and actions are available and typically consist in learning based on the mismatch between the expert and agent's 's state(-action) distributions.
In this paper, we combine the budding areas of model-free image-based reinforcement learning and state-action imitation learning to control non-trivial continuous dynamical systems from pixel demonstrations. We extend two leading proprioceptive-stin approaches to comparing the pixel trajectories of the learner and expert- see fig. for an illustration. The first approach, pixel sinkhorn imitation learning, P-SIT,, extends optimal transport approaches for imitation learning to the image
Code httphhtt///// anonymous 4open. aci eeweee/c///ee/eeeee imi tateFromPt 1 CLRR-OOR

---

Under review as a conference paper at ICLR 2022
Target Encoder
P-SIL
oe
iv)
C
fΦ
iix)
'''
"T
fø
o
P-DAC
oe
Encoder
fo aug
fa
Cccoa
fo
ii)
iit
Loss
iv)
fa
fø aug
fd cOooO
0"
Discriminator
1 17'2
''
Figure I: Summary of our proposed methods P-SIL and P-DAC. Top: P-SIL, i) encodes the agent 6 and expert of trajectories into a latent space, úi) computes map x² between these to iv) produce imitation rewards T1:r. Bottom: P-DAC, i) encodes the (data-augmented) agent o and expert of trajectories into latent space, ii) passes them through the discriminator, and mii) evaluates the DAC loss. iv) Rewards F1:7 can then be produced with the discriminator.
matrix C and iii) transport
setting (Papagiannis Li, 2020; Dadashi et al., 2021). A key component here is how we learn and compare latent representations using the cosine distance target encoder that is updated with the RL encoder's weights. Our second approach, pixel discriminator actor critic, P-DAC, is GAlL-based method for pixels (Ho Ermon, 2016; Kostrikov al., 2019; Torabi al., 2018). We propose several key modifications, including data augmentation and using the RL encoder for representations, that enable P-DAC scale non-trivial control tasks without needing access to expert actions. The imitation rewards from these methods are optimized with DrQ-v2 as an underlying image-based RL backbone (Yarats al., 2021a).
We demonstrate the versatility and strong performance of both approaches on DeepMind control suite tasks, and find that both P-DAC and P-SIL are able to recover expert performance while out- performing canonical extensions of corresponding state-based IL methods, such as SIL (Papagiannis Li, 2020) and DAC (Kostrikov et al., 2019; Ho & Ermon, 2016) terms of performance and sample efficiency. Our extensive ablation study reveals insights into the key design decisions for pixel imitation learning. Finally, we also provide an easy-to-use implementation these methods with unified DrQ-v2 backbone to make simple for practitioners build upon in this under-explored area.
BACKGROUND AND RELATED WORK
2.1 REINFORCAMENT LEARNING (RL) WITH PIXEL OBSERVATIONS
Pixel reinforcement learning can be instantiated as an infinite-horizon Markov decision process (MDP) (Bellman, 1957; Sutton & Barto, 2018), where the agent's state is approximated by a stack of consecutive RGB frames (Mnih et al., 2015). The MDP is the form (O, A, P, R, y, do) where O is the observation space, A is the action space, P O A A(O) is the transition function (A(O) is probability distribution over O), R: O A + IR is the reward function, 7 E [0, 1) is the discount factor and dg is the initial state distribution. The RL problem consists of finding optimal
2

---

Under review as a conference paper at ICLR 2022
policy π 0 + ((A) that maximizes the expected long-term reward E,, |i= 2R ai)l, where Ou ~ do, 01 ~ o() and 01+1 ~ P(-lor, az).
2.2 IMITATION LEARNING
In imitation learning, agents do not have access to the environment reward R. Instead, they are provided with dataset of multiple expert trajectories which the agent aims imitate, where each trajectory is of the form o = (oi..., o%- E O More formally, agent trajectories o = (o 0f) under the policy π have to be close to expert trajectories of under some metric between trajectories. We note these trajectories are empirical proxies for the occupancy distributions under the learner and expert policies (Ho & Ermon, 2016; Dadashi et al., 2021). This paper follows the line of research in imitation learning via agent-expert density matching, initially studied by Ng & Russell (2000); Boularias et al. (201 1); Englert et al. (2013). Here, imitation-reward signals are derived from trajectories obtained by an expert policy; the agent is then trained using RL methods, which use these imitation rewards as learning signals.
Generative Adversarial Learning for IL P-DAC builds on the budding area of state(-action)- based generative adversarial imitation learning, originally initiated by Ho & Ermon (2016) with the GAIL method, which uses adversarial learning combined with RL. A significant body of work built upon GAIL, notably AIRL and WGAIL (Fu et al., 2018; Xiao et al., 2019; Kostrikov et al., 2020), which propose alternative losses relying on other probabilistic divergences. Also, Torabi et al. (2018) extend GAIL the no-action setting, a setting which consider in this paper well. DAC (Kostrikov et al., 2019) significantly improves upon GAIL by turning it into an offline algorithm, and adding gradient penalty. Our P-DAC approach directly builds upon DAC, extending it to the visual setting. A key difference, which allowed us to stabilize visual DAC, was to store imitation rewards in the replay buffer and to not recompute them with up-to-date discriminators and encoders. Also, while DAC parameterizes the discriminator directly on the (proprioceptive) observation space, we compose the discriminator with the RL image encoder to leverage representations learned in the RL. pipeline.
GAIL-based methods in visual spaces are nascent, and contain works leveraging auxiliary losses (Cetin & Celiktutan, 2021). Toyer et al. (2020) propose a robust visual benchmark in which they evaluate mainly adversarial approaches in the discrete action setting. Finally, Rafailov et al. (2021) concurrently propose a model-based extension of GAIL, which does adversarial learning in the latent space of world model. As baseline, they spaces by parameterizing a discriminator in image space, which does not perform well on nearly all tasks they consider, while our P-DAC extension achieves expert performance on all considered tasks (including tasks considered by Rafailov et al. (2021), such as walker walk and cheetah run).
direct extension of DAC to visual
Optimal Transport for IL P-SIL extends to the visual case recent SOTA works on imitation learning via optimal transport (OT), namely Dadashi et al. (2021) and Papagiannis & Li (2020). They both leverage optimal transport matching to define rewards. The former uses a greedy approximation of the Wasserstein while the latter leverages the entropic Wasserstein with cosine cost. However, the latter also learns an embedding for states adversarially, while we leverage the encoder of the RL algorithm, avoiding the need for minimax optimization, and allowing us to avoid any computational slowdown. Finally, most of the adversarial algorithms in the GAIL line (Ho & Ermon, 2016; Fu et al., 2018) can also be interpreted from the standpoint of the minimization of an OT functional between the state(-action) occupancy distribution of the agent and expert (Xiao et al., 2019). App. A reviews relevant background on optimal transport.
3 RECIPES FOR VISUAL IMITATION LEARNING
We now describe our IL. approaches, which consist of alternative ways of defining imitation rewards that an RL backbone algorithm can learn from, and significantly improve upon canonical extensions of proprioceptive-state methods. In this paper, we leverage DrQ-v2 as the underlying RL algorithm (Yarats et al., 2021a). The data augmentation strategy throughout the paper (including for baselines) hence consists of random shifts with padding and random crop to restore the original image dimension, followed by bilinear interpolation. For all methods, we gather episodes under the current policy, evaluate reward trajectories, and update the replay buffer with such episodes, replacing
3

---

Under review as a conference paper at ICLR 2022
environment rewards by imitation rewards. We summarize the overall process in algorithm 1. We also illustrate our two proposed methods in fig. I.
3.1 PIXEL SINKHORN IMITATION LEARNING (P-SIL)
Our first approach extends imitation learning algorithms based optimal transport to a setting where only pixel observations are available. We define imitation-reward signals via the negative entropic Wasserstein distance between embedded agent and experts' image trajectories.
We interpret an image trajectory o (oi,...,oT) as a discrete probability measure of the form Mo Σ 6o., where images E RC are atoms weighted uniformly over time. Optimal transport distances directly on image observations provide weak signal given that i) the metric between individual images does not take into account spatial relationships between individual pixels, and ii) the sample complexity of OT grows exponentially with the number of dimensions (the number of samples in each trajectory necessary for good estimate of the OT plan π) (Genevay et al., 2019).
To alleviate these challenges, we embed image trajectories using DrQ-v2's encoder:
0.0 = folo)).... fo(ot)
o = [ffoloi......
(1)
In order be agnostic to the scale of the encoded states, consider the cosine distance metric de between encoded visual observations, similarly to Papagiannis & Li (2020),
(0, a.a
Ce. = do 9. oo
c.d =
(2)
1|0;
We estimate the entropic Wasserstein distance with the cosine cost between embedded trajectories
(H Hoc.o) min Σ
Crevie H(b),
H(v)
Σ fP==1
,r. log te,e'
DEV 1,1==1
where = (6 E RXT, +1 of 11} is the set of coupling matrices, in order to obtain an optimal alignment sta Finally, we extract rewards for each of the agent's 's states as
r(o,
If we are provided with multiple expert trajectories 0 ..... NN we infer the nearest-neighbor expert by computing the embedded entropic Wasserstein distance with cosine cost between each expert trajectory and the agent's 's rollout. We then set rewards with the alignment to the closest expert:
e, = arg min W (Moniss x)
r(o, = Σ
dolo a. o0 yot.
(5)
The rewards computed in (5) are non-stationary as the encoder fo is updated to optimize the critic loss. To increase the stability of P-SIL, we use a target network før which is updated every Tupdate environment steps with the weights of DrQ-v2's encoder. We use for to estimate rewards via (5). Also, we rescale P-SIL rewards so that the first episode normalized have a total return 10 to account for rapid changes in encoder representations during initial training.
important benefit of P-SIT that computing imitation rewards does not add any computational the encoder of the RL agent. This is in contrast
burden the cost of (5) is negligible, and to adversarial approaches (Papagiannis Li, 2020; Ho & Ermon, 2016), which require training a discriminator by solving inner-loop maximization problem and then inferring rewards using the discriminator. We thus preserve the attractive computational efficiency of the base algorithm DrQ-v2.
In summary, compute imitation rewards the entropic Wasserstein discrepancy between the agent's 's and expert's 's encoded states for the dingleddemonstration case. For multiple expert demonstra- tions, use the transport cost from the closest expert demonstration in the OT' sense.
4

---

Under review as a conference paper at ICLR 2022
3.2 PIXEL DISCRIMINATOR ACTOR CRITIC (P-DAC)
We now propose and study P-DAC as a GAlL-based method for pixels that builds on the work by Ho & Ermon (2016); Torabi al. (2018); Kostrikov et al. (2019). The key differences with previous works are that i) we compose the discriminator with the RL encoder to leverage its representations, til we apply data augmentation to discriminator imputs and iii) we store imitation rewards in the buffer rather than recomputing them at each iteration. We now describe the main components of P-DAC in more detail.
In contrast to P-SIL, it is necessary to train discriminator, along with DrQ-v2's other networks (actor, critic and encoder). To train the discriminator, we sample batches oi and oNN of size N from both the expert buffer and the policy buffer, respectively, apply data augmentation, and encode augmented observations using DrQ-v2's encoder, which then yields:
[ a aug(of),..... So aug(o
o = Jo aug(oi),.... aug(or)) (6)
Per common practice (Kostrikov et al., 2019), we omit importance sampling when estimating the policy state occupancy via replay buffer. We then maximize
log C0 + - log(1 )) E [| - 1]
with respect to the discriminator's weights and add a gradient penalty as recommended by Kostrikoy al. (2019). Here, o are sampled along straight lines between agent and expert's encoded observations. To compute rewards at the end of episodic rollouts, we embed the observation trajectory with DrQ-v2's 's encoder and then apply the discriminator the encoded observations to obtain rewards
r(o,) log (r D(o, logg (1 - a D(o,:)))
where a is the sigmoid function. We then add the episode with imitation rewards computed in (8) to the replay buffer. This differs from the original GAIL implementation (Ho & Ermon, 2016), where training is online, and from its follow-up offline extension DAC (Kostrikov et al., 2019) that recomputes rewards at each training iteration. We will show in the experiment section that data augmentation is a key component to scaling to challenging environments. Finally, in contrast with previous works that parameterize the discriminator directly in observation space (Ho & Ermon, 2016; Torabi et al., 2018; Kostrikov et al., 2019), we compose it with the RL encoder, which allows us to leverage its representations.
EXPERIMENTS
In this section, we empirically evaluate the proposed algorithms on tasks from the DeepMind control suite, aiming to contrast their strengths and weaknesses. We also provide an extensive ablation study highlighting the key design choices that enable solving challenging tasks with imitation learning from visual observations only.
Our set of experiments was designed with the aim of answering the following questions:
1. Are P-DAC and P-SIT able to achieve expert performance, and how do they compare in terms of sample efficiency (number of interactions with the environment required solve task) and computational efficiency?
Yes. P-DAC is more sample efficient than P-SIL, see fig. while P-SIL has a 37% faster computational footprint than P-DAC, see fig. 3. Furthermore, our methods allow to learn policies that are closer to expert policies than baselines under the embedded Wasserstein metric, see fig. 4.
2. Are P-DAC and P-SIL robust to the number of expert demonstrations they are provided? Yes, see fig. 5.
3. Is data augmentation as essential and effective as is in pixel-based reinforcement learning? Yes, agents cannot approach expert performance on nearly all task without data augmentation, see fig. 6.
5

---

Under review as a conference paper at ICLR 2022
Cartpole Swingup
Finger Spin
Cup Catch
Acrobot Swingup
Episade Return
800
1000
1000
400
600
750
750
300
500
500
200
200
250
250
100
1.5
1.5
0.0
Frames (x10')
Cheetah Run
Walker Stand
Walker Walk
Walker Run
Episade Return
800
800
800
400
400
500
400
200
200-
250
200
0
2
Frames (x10")
Frames (x109)
Frames (xx10f)
Frames (x109)
Expert
BC
DAC
SIL
P-DAC (Ours)
P-SIL (Ours)
Figure Our agents P-DAC and P-SIL demonstrate superior performance over vanilla instantiations of DAC and SIL, as well as BC over a set of challenging image-based tasks from DMC. In many cases P-DAC and P-SIL are able recover the expert performance by using only expert trajectories. Note, DAC and BC have privileged access to expert actions, while other methods do not.
4. Is optimal transport providing a gain over simpler metrics without alignment (e.g., cosine distance between trajectories)?
Yes, there a significant performance gain when leveraging OT alignments, see fig. 7
5. Are expert actions required to solve challenging tasks or are pixel observations enough?
No. P-DAC without actions performs par better than P-DAC with actions all environments, see fig. 8.
4.1 EXPERIMENTAL SETUP
Environments We consider Mujoco (Todorov et al., 2012) tasks the DeepMind control suite (Tassa et al., 2018). The selected tasks are distinct enough to demonstrate the versatility and robustness P-DAC and P-SIL. Environment observations are stacks of three consecutive 84 x 84 RGB images. We evaluate the agents with the environment rewards, but they are not provided to the agents during training. For the simplest tasks, we allow a budget 1.5M environment frames, while for hard tasks allow 5M environment frame in main results. In ablations, allow 3M for hard tasks.
Expert Demonstrations We gather expert demonstrations by training DrQ-v2 using the true environment rewards. We run 10 seeds and pick the seed that achieves highest episodic reward to generate expert trajectories (sequences of image observations).
Baselines As baselines, we consider our two proposed approaches, P-DAC and P-SIL, along with a behavioral cloning (BC) baseline that we strengthen by leveraging the same data augmentation as in our approaches. We also consider canonical extensions of DAC (Kostrikov et al., 2019) and SIL (Papagiannis & Li, 2020), in which the discriminator is directly defined on pixel-space, and without the tricks proposed in this paper (e.g., no data-augmentation applied to discriminator inputs, no target encoder for SIL). DAC and SIL baselines are implemented with the same DrQ-v2 backbone as P-DAC and P-SIL, hence also benefit from data- augmentation in the RL training loop, thus making the comparison setup fair. We note that the BC and DAC baselines are privileged since they access expert actions while our baselines do not.
Setup In all experiments, we 10 seeds under each configuration and average results, while providing a 90% confidence interval. We compare agents with respect to two distinct scores. Firstly, use the episodic return metric to verify whether the agent solves the task; secondly, average the entropic Wasserstein distance with cosine cost between agent rollouts and expert demonstrations, both embedded via common random encoder. This shows that agent rollouts close to the expert

---

Under review as a conference paper at ICLR 2022
Cheetah Run
Walker Stand
Walker Walk
Walker Run
Episode Return
800
1000
6:0
800
750
400
600
500
400
200
200
250
200
10 Hours
20
20
10 Hours
20
10
Hours
Hours
Expert
P-DAC (Ours)
P-SIL (Ours)
Figure 3: Performance of P-DAC, P-SIL, and data-augmented BC on DeepMind control suite tasks from pixel observations only with wall-clock time axis. We notice P-SIL and P-DAC solve all tasks in less than 4 hours.
1.2
Entropic Wass. to Expert
1.0
0.8
0.6
0.0
Cartpole Swingup
Finger Spin
Cup Catch
Acrobot Swingup
Cheetah Run
Walker Stand
Walker Walk
Walker Run
BC
DAC
SIL
P-DAC (Ours)
P-SIL (Ours)
Figure 4: Entropic Wasserstein distance to the expert demonstrations (the lower the better) for P-DAC, P-SIL, SIL, DAC and data-augmented BC on DeepMind control suite tasks from pixel observations only, relative to random policy. We notice P-SII. and P-DAC are on par in terms of closeness expert demonstrations while BC, SIL and DAC are worse.
demonstrations they are trained on. Finally, in ablation bar-plots, we show episodic reward averaged over the final 10 environment episodes. App. further describes all hyperparameters.
4.2 MAIN RESULTS
Sample efficiency We begin by comparing baselines with respect sample efficiency, i.e., the number of environment interactions required to solve task. We observe in fig. 2 that both P-DAC and P-SII. achieve expert performance on all environments besides acrobot swingup, which is hard exploration task. Both approaches significantly surpass the performance of SII,, DAC and data-augmented behavioral cloning even though the latter two are privileged by knowing expert actions. We also note that P-DAC is more sample efficient than P-SIL.
In fig. 4, we show the entropic Wasserstein between embeddings of agent rollouts and expert trajectories; results correlate well with those under episodic return (see fig. 2). Again, P-SIL and P-DAC outperform duta-au BC under this alternative metric, i.e., trajectories under these closer in the embedded entropic Wasserstein sense than those of BC.
Computational efficiency We contrast the computational efficiency of our approaches fig. 3; the unified backhone. Both of approaches solve
comparison is fair methods are implemented the hard tasks in around wall-clock hours on a single V100 GPU. This can be explained by the fact that there no loss frames per second (FPS) for P-SIL over its RL backbone (FPSpsIL. 101), while P-DAC's FPS is impacted by the cost coming from discriminator training (FPSSPAC = 74). As result, both approaches have comparable computational efficiency.
4.3 ABLATIONS
We now provide extensive ablation contrasting the approaches and highlighting key design choices.

---

Under review as a conference paper at ICLR 2022
1000
Episode Return
400
200
Carpooee Swingup
Angeer Spin
Cup Catch
Swingup Run
Stand
Walk
P-DAC ( (
10
50)
P-SIL (
10
50)
BC
(
10
50)
Figure 5: Ablation study the number of demonstration. We train all P-DAC, P-SIL and BC on 1, 10 and 50 demonstrations. We observe P-DAC and P-SIL are robust to the number of demonstrations, while BC is not, and fails most tasks for the I-demonstration setting.
Number of demonstrations We first evaluate the robustness of baselines to the number of expert demonstrations provided to the learner. We train agents on 1, 10 and 50 demonstrations, on all tasks, and plot episodic return averaged across seeds in fig. 5. We observe that approaches robust to the number of demonstrations, with comparable performance for 1, 10 and 50 demonstrations across tasks (besides finger spin). By contrast, the data-augmented BC baseline is non-robust, and requires large number of demonstrations (typically 50) to approach expert performance.
1000
Episode Return
600
600
Cartpole Swingup
Angger Spin
Cup Catch
Acrobot Swingup
Cheetah Run
Walker Stand
Walker Walk
Walker Run
P-DAC ( DA
DA
P-SIT
No DA
DA
BC
No DA
DA
Figure 6: Ablation study on data augmentation (DA). We compare P-DAC, P-SIL and BC with and without data augmentation. We observe the gap in performance between data- autta--ugeeted (DA) and no-data- augmented (No DA) implementations to be large. For instance, our baselines do not recover any meaningful behaviors on finger spin, cheetah run, acrobot swingup, and walker environments without DA.
Data augmentation Next, evaluate whether data augmentation is necessary achieve expert performance. We observe that without augmentation, both P-DAC and P-SIL are not able to recover solid performance on nearly all tasks, especially locomotion ones. We note that behavioral cloning is less sensible to it. This may be explained by the fact that augmentation is necessary to obtain good representations seen in Laskin et al. (2020); Yarats al. (2019; 2021a), and that our imitation approaches leverage such representations to design reward signals. As result, without good representations the agents can not recover any meaningful behaviors.
Metrics We continue with analyzing of the key design choices behind P-SIL, i.e., whether an optimal transport alignment of the agent and expert trajectories is required over simply leveraging an or cosine distance (no alignment). We observe in fig. that OT is essential achieving expert performance on most tasks. We believe this is due to the high variance in the initial state distribution of agent and expert, which leads to unaligned trajectories, and hence uninformative rewards in the absence of OT alignment.
Observations vs. Observation-actions GAIL-like methods are often assumed to require expert actions recover non-trivial performance. We hence compare P-DAC with privileged implementa-
8

---

Under review as a conference paper at ICLR 2022
Episode Return
600
400
Cartpole Swingup Ringer Spin
Cup Swingup
P-SIL
Cosine
Euclidean
Sinkhorn + Cosine (P-SIL))
Figure 7: Ablation study the distance between trajectories for P-SIL. We compare approach (OT alignment with cosine cost), to the cosine distance and the Euclidean distance between trajectories (without OT). We observe that OT significantly outperform non-OT approaches. In particular, baselines without OT do not recover any meaningful behaviors on finger spin and cheetah run, and weaker on acrobot swingup and walker run. Hence, OT alignment is also an essential component of P-SIL.
1000
Episode Return
800
600
400
Cartpole Swingup Angeer Spin
Cup Catch
Swingup
P-DAC Observations and Actions
Observations
Figure 8: Ablation study the need for expert actions. We train P-DAC with and without actions concatenated to observations (the former is privileged). We observe the performance gap to be marginal, hence actions not necessary in the considered environments, contrarily to popular beliefs on GAlL-like algorithms.
tion of P-DAC which accesses expert actions. In the latter case, we concatenate embedded image observations, and actions before passing these to the discriminator. We observe in fig. 8 that our approach without expert actions performs on par with the action-baseline highlighting that on the considered environments, actions are not required to achieve strong performance. This is encouraging given that there are massive amounts of unlabeled video data that these agents could potentially learn from.
CONCLUSION
We propose effective methods for imitation learning directly from pixels without expert actions, extending state- base- adversarial and optimal transport approaches, and significantly outperforming canonical extensions of these. This steps closer toward leveraging the rich amount of expert-level data on non-trivial environments for the control continuous systems. We envision that continuing scale in this space will involve further understanding how to recover representation of the high-dimensional data, e.g. with unsupervised pretraining, and how to beyond settings where the expert and learner live in the same space.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Richard Bellman. A markovian decision process. Indiana Univ. Math. J., 1957.
Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In
Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp- 182-189. JMLR Workshop and Conference Proceedings. 2011.
Edoardo Cetin and Oya Celiktutan. Domain-robust visual imitation learning with mutual information constraints. In International Conference Learning Representations, 2021. URL https: //openrevies net/ forum?
Robert Dadashi, Leonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein
imitation learning. International Conference on Learning Representations, 2021. URL https:
///ppenreveww.neeeeeeeeeeeeeeeee wetee/fouuteeeeeeeeeeeeeeeeeeeet forum? Id-ItYSUU9zge
Peter Englert, Alexandros Paraschos, Jan Peters, and Mare Peter Deisenroth. Model-based imitation learning by probabilistic trajectory matching. In 2013 IEEE International Conference on Robotics and Automation, pp. 1922-1927, 2013.
doi: 10. 11110//CRA.2013.. 203366638....
Chelsea Finn, X. Tan, Yan Duan, Trevor Darrell, Sergey Levine, and P. Abbeel. Deep spatial autoencoders for visuomotor learning. 2016 IEEE International Conference on Roboties and
Automation (ICRA), pp. 512-519, 2016.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse inforcement learning. International Conference on Learning Representations, 2018. URL
https://openrevaa
net forum?id-rkHywl.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018, 2018.
Aude Genevay, Lénaic Chizat, Francis Bach, Marco Cuturi, and Gabriel Peyré. Sample complexity of sinkhorn divergences. The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1574-1583, PMLR, 2019.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: I81 1.04551, 2018.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 19Xiv:1912.01600) 2019.
Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. CoRR, 2020.
Jonathan Ho and S. Ermon. Generative adversarial imitation learning. In NIPS, 2016.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-oooi Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Leurning Representations, 2019. URL https:// openreview net forum? www.orummmmdttk44oooooooo.... fpoA5km.
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations, 2020. URL https:// openreview. net. forum? d
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data, 2020.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, 2015.
10

---

Under review as a conference paper at ICLR 2022
Hao Liu and Pieter Abbeel. Unsupervised active pre-training for reinforcement learning, openreview; 2021. URL htttps://openrevieeee http::/www.....oorrrmddddoooooo.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 51 (277544)))5593333 2015.
Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML '00, pp. 663-670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072.
Georgios Papagiannis and Yunpeng Li. Imitation learning with sinkhorn distances. ArXiv, abs/2008.09167, 2020.
G. Peyré and Marco Cuturi. Computational optimal transport. Found. Trends Mach. Learn., 11: 355-607, 2019.
D. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In NIPS, 1988.
D. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3:88-97, 1991.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Visual adversarial imitation learning using variational models. JCML 2021 Workshop on Unsupervised Reinforcement Learning, 2021. URL https: wpwpreeieeeewweeeeepee net forum?id=z 6333666666
Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in deep reinforcement learning. CoRR, 2020.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, P. Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In ICML, 2021.
Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21:343-346, 1967.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint XXiv:2004.04136 2020.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. arXiv preprint arXiv, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind control suite. Technical report, DeepMind, January 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv: 0118800066555 2018.
Sam Toyer, Rohin Shah, Andrew Critch, and Stuart Russell. The MAGICAL benchmark for robust imitation. In Advances in Neural Information Processing Systems, 2020.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Huang Xiao, Michael Herman, Joerg Wagner, Sebastian Ziesche, Jalal Etesami, and T. H. Linh. Wasserstein adversarial imitation learning. ArXiv, abs/1906.08 13, 2019.
Denis Yarats. Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.011 2019.
11

---

Under review as a conference paper at ICLR 2022
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021a.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. CoRR, 2021b.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. In 9th International Conference on Learning Representations, ICLR 2021, 2021c.
12

---

Under review as a conference paper at ICLR 2022
A BACKGROUND ON OPTIMAL TRANSPORT
Optimal transport (Villani, 2009; Peyré & Cuturi, 2019) tools allow us to compare probability measures while incorporating the geometry of the space. The entropic 2-Wasserstein distance
between two discrete measures prm = Σ sz. and By = Σラ= py is
T
Myy
min NEW 1,t==1
Σ
(xr Ve ma H(),,
H(b) - E
War log Va,er
(9)
6,==1
where 1/ = fob E RTXT 11 : 1 1 is the set of coupling matrices, and is a metric. Adding the entropic term has two main benefits: i) the induced problem can be solved in quadratic time via Sinkhorn's algorithm (Sinkhorn & Knopp, 1967), and ii) the resulting distance is smooth the measures' samples. Intuitively, the optimal coupling matrix & provides an alignment of the samples of Ma, My which minimizes the cost of transport between these. The distance then consists of the weighted sum of distances between aligned samples, along with an entropic penalty.
B DRQ-V2
We use DrQv2 (Yarats al., 2021a), which is a actor-critic method for continuous control based on the deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015). Given a replay buffer D, it learns simultaneously Q-function Qo and policy Tp. Qe is trained by clipped double- Q- learning (Fujimoto et al., 2018) with n-step returns. is trained via deterministic policy gradient. DrQv2 employs data augmentation in the form of random shifts with padding and random crop to restore the original image dimension, followed by bilinear interpolation. Data augmentation acts as regularization and reduces the variance of the Q estimates. Images are embedded into the latent space via an encoder fo after being augmented. This encoder is trained to minimize the critic loss only.
C ALGORITHMS
Algorithm Inverse reinforcement learning core. Different methods can be instantiated by changing the rewarder function.
Require: Expert demonstrations {Horm ) replay buffer D, initialized policy network Q- network Q and encoder f. For DAC-like baselines, also requires a discriminator D.
for t. Trotal do
done then
FI.T = rewarder(episode)) for instance (5) for P-SIL, (8) for P-DAC Update episode with fı.r and add all quadruples [or, az Tr to D.
O, env.reset(), done False, episode
end if
a, π (-(o,) + oc+1, done env.step env.step(a:). episode. append (o;, at, o,+1)) Update DrQ-v2's s actor and critic, and rewarder-specific functions using D.
end for
13

---

Under review as a conference paper at ICLR 2022
D HYPERPARAMETERS AND FURTHER EXPERIMENTAL DETAILS.
We provide a list of all hyperparameters table 1. The only environment-speciffe variation is that the number of n-steps set for walker tasks. For P-DAC, we the DrQ-v2's critic encoder, while for P-SIL we use the actor's s encoder, noting that both share the same convolutional layer weights.
Agent
Parameter
Value
Common
Replay buffer size Learning rate
150000
le-4
Exploration Schedule
linear (L, 0.1, 500000)
Discount
0.99
n-Step returns
Action repeat
Frame stack
Seed frames
2000
Exploration steps
4000
Mini-batch size
256
Agent update frequency
Critic soft-update rate
0.01
Features dim
50
Hidden dim
1024
Optimizer
Adam
P-SII.
Target update frequency
10000
Reward scale factor
10
P-DAC
Gradient penalty coefficient
10
Table 1: List hyperparameters.
14

---

Under review as a conference paper at ICLR 2022
E
EXTRA ABLATION PLOTS
Episode Return
Cartpole Swingup
Episode Return Episode Return
1000
Finger Spin
Episode Return
1000
Cup Catch
Episode Return
Acrobot Swingup
500
500
200
0.0
1.0
1.5
0.0
0.5
1.0
0.0
1.0
1.5
Frames (x10*)
Frames (x10f)
Frames (x 10*)
Frames (x10')
Cheetah
Walker Stand
Walker Walk
Walker
Episode Return
Episode Return
1000
Episode Return
500
500
500
500
250
Frames (x10')
Frames (x10')
Frames (x10f)
Frames (x10%)
P-DAC
10
50
P-SIL (
10
50
BC
10
50
Figure 9: Ablation study on the number of demonstration. We train all P-DAC, P-STI. and BC on 1, 10 and 50 demonstrations. We observe P-DAC and P-SIL robust to the number of demonstrations, while BC is not, and fails on most tasks for the I-demonstration setting.
Episode Return
Cartpole Swingup
Episode Return
1000
Finger Spin
Episode Return Episode Return
1000
Cup Catch
Episode Return
Acrobot Swingup
500
500
500
200
0.0
0.5
1.0
1.5
0.0
0.5
1.0
0.0
0.5
1.0
1.5
2
Frames
(x10*)
Frames (x
Frames (x 10*)
Frames (x100)
Episode Return
Cheetah Run
Episode Return
1000
Walker Stand
1000
Walker Walk
Episode Return
Walker Run
500
500
500
Frames (x10*)
Frames (x10*)
Frames (x 10
Frames (x100)
P-DAC ( No DA
DA
P-SIL ( No DA
DA
BC
No DA
DA
Figure 10: Ablation study on data augmentation (DA). We compare P-DAC, P-SIL and BC with and without data augmentation. We observe the gap in performance to be significantly larger for P-DAC and P-SIL, showing that DA is a crucial component of our methods.
15

---

Under review as a conference paper at ICLR 2022
Episode Return
Cartpole Swingup
Episode Retur
Finger Spin
Episode Retum
1000
Cup Catch
Episode Retum
Acrobot Swingup up
200
500
500
500
100
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
Frames (x10")
Frames (x10')
Frames
(x10f)
Frames (x10%)
Episode Return
Cheetah Run
Episode Return
1000
Walker Stand
Episode Return
Walker Walk
Episode Return
Walker Run
500
400
500
200
0:
Frames (x10")
Frames (x10*)
Frames (x10*)
Frames (x10")
P-SIL
Cosine
Euclidean
Sinkhorn + Cosine (P-SIL))
Figure 11: Ablation study on the distance between trajectories for P-SIL. We compare our approach (OT alignment with cosine cost), to the cosine distance and the Euclidean distance hetween trajectories (without OT). We observe that OT significantly outperform non-OT approaches. Hence, OT alignment also an essential component P-SIL.
Episode Return
Cartpole Swingup
Episode Return
1000
Finger Spin
Episode Return
1000
Cup Catch
Episode Retum
400
Acrobot Swingup
500
500
500
200
0.0
0.5
1.0
1.5
0.0
0.5
1.0
0.0
0.5
1.0
1.5
Frames (x10")
Frames (x11%)
Frames (x10*)
Frames (x10*)
Cheetah Run
Walker Stand
Walker Walk
Walker Run
Episode Return
Episode Return
1000
Episode Return
1000
Episode Return
500
500
500
500
250
0-
Frames (x10%)
Frames (x10")
Frames (x10*)
Frames (x110f
P-DAC
Observations and Actions
Observations
Figure 12: Ablation study on the need for expert actions. We train P-DAC with and without actions concatenated to observations (the former is privileged). We observe the performance gap to be marginal, hence actions are not necessary in the considered environments, contrarily to popular beliefs on GAIL-like algorithms.
16


Reviews:
[{"review_id": "td4_wyzkQg", "reviewer": "Reviewer_R6AN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n\nTwo imitation learning methods based on RL encoder are proposed. The experimental results show that both of the methods outperform the DAC and SIL baselines. \n\nCons: \n1. Data augment is important for improving the performance of P-DAC. However, it is not clear how to augment the data in practice.\n2. It is not always possible to used the RL encoder for many tasks. This is the key issue of this work.  \n3. The It is necessary to compare the proposed methods with other extentions of DAC and SIL.  ", "summary_of_the_paper": "This paper introduces two methods for imitation learning from pixel observations based on adversarial learning and optimal transport. Representation from the RL encoder are adopted for calculation imitation rewards. P-SIL (Pixel Sinkhorn Imitation Learning) learns and compares latent representations of the encoded expert and agent behaviors; and P-SAC (Pixel Discriminator Actor Critic) augments the representations of the agents and the experts. The key component of this work is the RL encoder used in both P-SIL and P-SAC. ", "summary_of_the_review": "Imitation learning can be applied in tasks where a reward function is hard to specify or too sparse to be used in practice. The methods introduced in this work is useful for imitation learning from images directly. However, the learning from pixels can be ambiguous in certain cases since the actions cannot be fully described in a video. \n\nA paper to appear in the Proceeding of Deep Reinforced Learning Workshop in NeurIPS 2021 has the same tile with this submission: Imitation Learning from Pixel Observations for Continuous Control (https://nips.cc/Conferences/2021/Schedule?showEvent=21848). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iBIn18G2O5", "reviewer": "Reviewer_JN38", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ The paper is well-motivated. It is attractive to enable agents to learn from demo videos without actions.\n+ The paper is well-written and easy to follow.\n+ Two approaches (P-SIL and P-DAC) are proposed based on stinkhorn imitation learning (SIL) and discriminator actor-critic (DAC), respectively. The introduced data augmentation method is simple yet effective in improving performance.\n+ The designed experiments comprehensively answered common questions on effectiveness, efficiency, robustness, and necessity.\n\nWeakness:\n- The setting is impractical. The proposed methods rely on an RL expert for computing the reward. However, such an RL expert is unavailable in most video demos. Notably, if the RL expert is available, it will be convenient to access the expert action for training. This is my main concern for this paper. \n- It seems that the data augmentation method contributes most to the improvement in the performance. How about the performance of DAC and SIL with the data augmentation methods?\n", "summary_of_the_paper": "This paper focuses on learning visual policies by imitating video data without observing the expert actions. The authors introduce a method to compute the imitation reward based on the representation from the expert RL encoder. Besides, a data augmentation method is proposed to scale to non-trivial control tasks. They conduct comprehensive experiments on DeepMind control suite tasks, showing the versatility and strong performance of the two approaches (P-DAC and P-SIL).", "summary_of_the_review": "My main concern for this paper is the setting. In my view, it is unreasonable to use an encoder from the RL-based expert for learning from videos without actions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NVHWRgqtvW", "reviewer": "Reviewer_wukE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- Easy to implement and understand algorithms, albeit knowing the algorithms they are built upon.\n\n- Good results on pixel-based imitation learning\n\n- Open-source implementation is provided.\n \n##########################################################################\n\nCons:\n\n- Adding data augmentation and DrQ-v2 as backbone is a sound and effective approach but there is little novelty in this paper, as auxiliary losses and target encoders to improve pixel observations have been studied before. To my understanding the contributions for P-SIL with respect to SIL are using the DrQ-v2 RL encoder to embed states instead of an adversarial approach. Instead for P-DAC, the contributions with respect to DAC are again composing it with the DrQ-v2 RL encoder and computing the rewards at the end of episodic rollouts instead of recomputing them at each training iteration.  \n\n- Paper is hard to read:\n    - pseudocode of algorithm is not detailed enough to understand your contribution. You could expand on the `rewarder` function which should be your actual contribution.\n    - pseudocode of algorithm doesn't have any initialization and definition of variables.\n    - in equation 3, $\\Psi$ and $\\psi$ are not defined.\n    - a background on DrQ-v2 would be highly appreciated to understand how the training pipeline works and the gradient flows, which I had to hypotesize after reading DrQ and DrQ-v2.\n\n##########################################################################\n\nQuestions during rebuttal period:\n\n- Address and/or clarify the cons above.", "summary_of_the_paper": "This paper presents two algorithms for imitation learning P-SIL and P-DAC which are built on top of SIL (Papagiannis & Li, 2020) and DAC (Kostrikov et al, 2019). The first one is an algorithm based on the Sinkhorn distance and the second one uses an adversarial approach, by training a discriminator. These two approaches use state-only trajectories to generate reward sequences to match the agent state-trajectories and learn through that reward. By combining the advances in image-based reinforcement learning by the usage of data augmentation and target encoders, the two new algorithms, P-SIL and P-DAC, are tested and reach good results on visual control tasks.\n", "summary_of_the_review": "I am towards rejection of this paper. Sound and interesting approach which achieves good results, but there is little novelty. Furthermore, paper is quite hard to read and to understand what are the main contributions, which makes an approach that seems easy quite complex.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3qeqsG2No2A", "reviewer": "Reviewer_omV5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS\n\n(S1) The stated thrust of the paper--that of exploring the design choices and tradeoffs within the space of algorithms designed for imitation learning from visual observations--is of great importance to the community studying these methods. To date, I can think of no paper that provides a compelling discussion of these topics.\n\n(S2) The authors have provided an open-source implementation of their methods, which bolsters the story of the paper being one of exploring \"recipes\" for visual imitation.\n\nWEAKNESSES\n\n(W1) Unfortunately, I don't feel that the authors really lived up to the goals that they stated at the outset of the paper. That is, even with proposing and experimenting with P-SIL and P-DAC, I still don't have a clear understanding of the \"design choices and tradeoffs\" in this algorithmic space. Nowhere in the paper can I find a clear and explicit discussion of these things--the five experimental questions stated at the beginning of Section 4 seem to ultimately focus more on whether or not P-SIL and P-DAC will perform well, as opposed to quantifying or discovering which design choices are more important. For example, with respect to data augmentation, Figure 6 really only serves to show that P-DAC and P-SIL are better than BC rather than exploring the effect of data augmentation as a specific design choice across the suite of available methods. Moreover, with respect to the experiment depicted in Figure 7, it does not seem to me that the presented data matches up with the conclusion. Many of the domains show that all the methods compared perform about as well no matter what distance function is used--this doesn't at all support the claim that the authors have made that \"OT alignment significantly outperform [sic] non-OT approaches.\" Finally, if P-SIL and P-DAC are both built atop DrQ-v2 and the other methods compared against are not, what evidence is there that the performance gain is not simply due to the use of DrQ-v2 as opposed to the \"representation sharing\" and other more general factors that the authors claim account for this difference? I do note that there _is_ unused space at the end of Page 9--I wonder if the authors might be able to address this comment with an enhanced Conclusion for example.\n\n(W2) Again, for a paper purporting to discuss design choices and tradeoffs, I was disappointed that more experimental details weren't given. For example, with respect to the data augmentation experiments, I could not find anywhere in the paper where it was specified what exact types of data augmentation were performed for these experiments.\n\nMINOR COMMENTS\n\n(MC1) The bottom half of Figure 1 failed to render in an application on my iPad, though it did render correctly in Chrome. Perhaps the authors could pursue a more reliable way to generate that figure to avoid this problem in the next version of the paper.", "summary_of_the_paper": "The authors have considered the problem of imitation learning using only visual observations, and they've explored the paradigms of using methods based on adversarial learning and optimal transport to solve this problem. In particular, the authors propose two new methods--P-SIL and P-DAC--presumably as a way to explore design choices and tradeoffs in this space. Some experiments designed to demonstrate the superiority of P-SIL and P-DAC are presented.", "summary_of_the_review": "I agree wholeheartedly with the goal that the authors have stated in the early portions of the paper, but I think the work that has been done--or at the very least, the way it has been presented--fails to live up to that goal. I'd encourage the authors to focus more on the design decisions and tradeoffs in the existing space of algorithms rather than trying to show that their proposed algorithms dominate others.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Learning policies from video demonstrations alone without paired action data is a promising paradigm for scaling up Imitation Learning. As such the paper is well-motivated. Two approaches P-SIL and P-DAC train rewards for RL training, based on learning Sinkhorn distances between trajectory embeddings and an adversarial approach.  The reviews brought up lack of clarity in presentation and experimental results and ablation studies falling short of convincingly demonstrating value of distance functions used and other design tradeoffs. As such the paper does not meet the bar for acceptance at ICLR.
Decision: Reject