Year: 2021

Abstract:
We study imitation learning using only visual observations for controlling dynamical systems with continuous states and actions. This setting is attractive due to the large amount of video data available from which agents could learn from. However, it is challenging due to i) not observing the actions and ii) the high-dimensional visual space. In this setting, we explore recipes for imitation learning based on adversarial learning and optimal transport. A key feature of our methods is to use representations from the RL encoder to compute imitation rewards. These recipes enable us to scale these methods to attain expert-level performance on visual continuous control tasks in the DeepMind control suite. We investigate the tradeoffs of these approaches and present a comprehensive evaluation of the key design choices. To encourage reproducible research in this area, we provide an easy-to-use implementation for benchmarking visual imitation learning, including our methods1.

heading: 1 INTRODUCTION
text: Learning continuous control policies directly from pixel observations is an important problem due to its potential impact on fields like robotics, autonomous driving and video games. These domains have rich resources of data available of humans performing expert-level demonstrations that our software agents do not leverage as they are often trained from scratch without any knowledge of how humans think about these problems. However, using unlabeled video data is challenging as it i) requires distilling a representation of the world into the policy of an agent, and ii) we do not know the underlying actions and reasoning process of the expert. This renders common algorithms like canonical behavioral cloning (Pomerleau, 1988; 1991) useless in the no-action setting.
Recently the community has advanced our understanding in learning visual representations and learning to imitate demonstrations provided as proprioceptive states. Visual representation learning has been crucial in recent advancements for sample-efficient reinforcement learning (RL) directly from pixels in continuous spaces, e.g. with reconstruction (Finn et al., 2016; Yarats et al., 2019), contrastive learning (Srinivas et al., 2020; Stooke et al., 2020), unsupervised pre-training (Liu & Abbeel, 2021; Yarats et al., 2021b; Seo et al., 2021), world models (Hafner et al., 2018; 2019; 2020) and data augmentation (Yarats et al., 2021c; Raileanu et al., 2020; Laskin et al., 2020).
These approaches require known reward signals from the environment, which are not always available or well-defined. When expert demonstrations are available, imitation learning (IL) and inverse RL (IRL) methods overcome the issue of not having a reward signal and seek to recover the expert agent (Ng & Russell, 2000). These methods are very effective when low-dimensional proprioceptive states and actions are available and typically consist in learning based on the mismatch between the expert and agent’s state(-action) distributions.
In this paper, we combine the budding areas of model-free image-based reinforcement learning and state-action imitation learning to control non-trivial continuous dynamical systems from pixel demonstrations. We extend two leading proprioceptive-state approaches to comparing the pixel trajectories of the learner and expert – see fig. 1 for an illustration. The first approach, pixel sinkhorn imitation learning, P-SIL, extends optimal transport approaches for imitation learning to the image
1Code at https://anonymous.4open.science/r/ImitateFromPixelsICLR-B0EF
setting (Papagiannis & Li, 2020; Dadashi et al., 2021). A key component here is how we learn and compare latent representations using the cosine distance on a target encoder that is updated with the RL encoder’s weights. Our second approach, pixel discriminator actor critic, P-DAC, is a GAIL-based method for pixels (Ho & Ermon, 2016; Kostrikov et al., 2019; Torabi et al., 2018). We propose several key modifications, including data augmentation and using the RL encoder for representations, that enable P-DAC to scale to non-trivial control tasks without needing access to expert actions. The imitation rewards from these methods are optimized with DrQ-v2 as an underlying image-based RL backbone (Yarats et al., 2021a).
We demonstrate the versatility and strong performance of both approaches on DeepMind control suite tasks, and find that both P-DAC and P-SIL are able to recover expert performance while outperforming canonical extensions of corresponding state-based IL methods, such as SIL (Papagiannis & Li, 2020) and DAC (Kostrikov et al., 2019; Ho & Ermon, 2016) in terms of performance and sample efficiency. Our extensive ablation study reveals insights into the key design decisions for pixel imitation learning. Finally, we also provide an easy-to-use implementation of these methods with unified DrQ-v2 backbone to make it simple for practitioners to build upon in this under-explored area.

heading: 2 BACKGROUND AND RELATED WORK

heading: 2.1 REINFORCEMENT LEARNING (RL) WITH PIXEL OBSERVATIONS
text: Pixel reinforcement learning can be instantiated as an infinite-horizon Markov decision process (MDP) (Bellman, 1957; Sutton & Barto, 2018), where the agent’s state is approximated by a stack of consecutive RGB frames (Mnih et al., 2015). The MDP is of the form (O, A, P , R, γ, d0) where O is the observation space, A is the action space, P : O × A → ∆(O) is the transition function (∆(O) is a probability distribution over O), R : O ×A → R is the reward function, γ ∈ [0, 1) is the discount factor and d0 is the initial state distribution. The RL problem consists of finding an optimal
policy π : O → ∆(A) that maximizes the expected long-term reward Eπ[ ∑∞ t=0 γ
tR(ot,at)], where o0 ∼ d0, at ∼ π(·|ot) and ot+1 ∼ P (·|ot,at).

heading: 2.2 IMITATION LEARNING
text: In imitation learning, agents do not have access to the environment reward R. Instead, they are provided with a dataset of multiple expert trajectories which the agent aims to imitate, where each trajectory is of the form oe = (oe0, . . . ,o e T ) ∈ OT . More formally, agent trajectories oa = (oa0 , . . . ,o a T ) under the policy π have to be close to expert trajectories o
e under some metric between trajectories. We note these trajectories are empirical proxies for the occupancy distributions under the learner and expert policies (Ho & Ermon, 2016; Dadashi et al., 2021). This paper follows the line of research in imitation learning via agent-expert density matching, initially studied by Ng & Russell (2000); Boularias et al. (2011); Englert et al. (2013). Here, imitation-reward signals are derived from trajectories obtained by an expert policy; the agent is then trained using RL methods, which use these imitation rewards as learning signals.
Generative Adversarial Learning for IL P-DAC builds on the budding area of state(-action)based generative adversarial imitation learning, originally initiated by Ho & Ermon (2016) with the GAIL method, which uses adversarial learning combined with RL. A significant body of work built upon GAIL, notably AIRL and WGAIL (Fu et al., 2018; Xiao et al., 2019; Kostrikov et al., 2020), which propose alternative losses relying on other probabilistic divergences. Also, Torabi et al. (2018) extend GAIL to the no-action setting, a setting which we consider in this paper as well. DAC (Kostrikov et al., 2019) significantly improves upon GAIL by turning it into an offline algorithm, and adding a gradient penalty. Our P-DAC approach directly builds upon DAC, extending it to the visual setting. A key difference, which allowed us to stabilize visual DAC, was to store imitation rewards in the replay buffer and to not recompute them with up-to-date discriminators and encoders. Also, while DAC parameterizes the discriminator directly on the (proprioceptive) observation space, we compose the discriminator with the RL image encoder to leverage representations learned in the RL pipeline.
GAIL-based methods in visual spaces are nascent, and contain works leveraging auxiliary losses (Cetin & Celiktutan, 2021). Toyer et al. (2020) propose a robust visual IL benchmark in which they evaluate mainly adversarial approaches in the discrete action setting. Finally, Rafailov et al. (2021) concurrently propose a model-based extension of GAIL, which does adversarial learning in the latent space of a world model. As a baseline, they use a direct extension of DAC to visual spaces by parameterizing a discriminator in image space, which does not perform well on nearly all tasks they consider, while our P-DAC extension achieves expert performance on all considered tasks (including tasks considered by Rafailov et al. (2021), such as walker walk and cheetah run).
Optimal Transport for IL P-SIL extends to the visual case recent SOTA works on imitation learning via optimal transport (OT), namely Dadashi et al. (2021) and Papagiannis & Li (2020). They both leverage optimal transport matching to define rewards. The former uses a greedy approximation of the Wasserstein while the latter leverages the entropic Wasserstein with cosine cost. However, the latter also learns an embedding for states adversarially, while we leverage the encoder of the RL algorithm, avoiding the need for minimax optimization, and allowing us to avoid any computational slowdown. Finally, most of the adversarial algorithms in the GAIL line (Ho & Ermon, 2016; Fu et al., 2018) can also be interpreted from the standpoint of the minimization of an OT functional between the state(-action) occupancy distribution of the agent and expert (Xiao et al., 2019). App. A reviews relevant background on optimal transport.

heading: 3 RECIPES FOR VISUAL IMITATION LEARNING
text: We now describe our IL approaches, which consist of alternative ways of defining imitation rewards that an RL backbone algorithm can learn from, and significantly improve upon canonical extensions of proprioceptive-state methods. In this paper, we leverage DrQ-v2 as the underlying RL algorithm (Yarats et al., 2021a). The data augmentation strategy throughout the paper (including for baselines) hence consists of random shifts with padding and a random crop to restore the original image dimension, followed by bilinear interpolation. For all methods, we gather episodes under the current policy, evaluate reward trajectories, and update the replay buffer with such episodes, replacing
environment rewards by imitation rewards. We summarize the overall process in algorithm 1. We also illustrate our two proposed methods in fig. 1.
3.1 PIXEL SINKHORN IMITATION LEARNING (P-SIL)
Our first approach extends imitation learning algorithms based on optimal transport to a setting where only pixel observations are available. We define imitation-reward signals via the negative entropic Wasserstein distance between embedded agent and experts’ image trajectories.
We interpret an image trajectory o = (o1, . . . ,oT ) as a discrete probability measure of the form µo = 1 T ∑T t=1 δot , where images ot ∈ RC×H×W are atoms weighted uniformly over time. Optimal transport distances directly on image observations provide weak signal given that i) the metric between individual images does not take into account spatial relationships between individual pixels, and ii) the sample complexity of OT grows exponentially with the number of dimensions (the number of samples in each trajectory necessary for a good estimate of the OT plan π) (Genevay et al., 2019).
To alleviate these challenges, we embed image trajectories using DrQ-v2’s encoder: oa,φ = [ fφ(o a 1), . . . , fφ(o a T ) ] , oe,φ = [ fφ(o e 1), . . . , fφ(o e T ) ] . (1)
In order to be agnostic to the scale of the encoded states, we consider the cosine distance as metric dc between encoded visual observations, similarly to Papagiannis & Li (2020),
Ct,t′ = dc(o a,φ t ,o e,φ t′ ) = 1−
〈oa,φt ,o e,φ t′ 〉
||oa,φt || ||o e,φ t′ ||
. (2)
We estimate the entropic Wasserstein distance with the cosine cost between embedded trajectories
W2 (µoa,φ , µoe,φ) = min ψ∈Ψ T∑ t,t′=1 Ct,t′ψt,t′ −H(ψ), H(ψ) = − T∑ t,t′=1 ψt,t′ logψt,t′ , (3)
where Ψ = {ψ ∈ RT×T : ψ1 = ψT1 = 1T 1} is the set of coupling matrices, in order to obtain an optimal alignment ψ?. Finally, we extract rewards for each of the agent’s states as
r(oa,φt ) = − T∑ t′=1 Ct,t′ψ ? t,t′ . (4)
If we are provided with multiple expert trajectories oe1 , . . . ,oeN , we infer the nearest-neighbor expert by computing the embedded entropic Wasserstein distance with cosine cost between each expert trajectory and the agent’s rollout. We then set rewards with the alignment to the closest expert:
e? = arg min n∈{1,...,N}
W2 (µoa,φ , µoen,φ) r(o a,φ t ) = − T∑ t′=1 dc(o a,φ t ,o e?,φ t′ )ψ ? t,t′ . (5)
The rewards computed in (5) are non-stationary as the encoder fφ is updated to optimize the critic loss. To increase the stability of P-SIL, we use a target network fφ′ , which is updated every Tupdate environment steps with the weights of DrQ-v2’s encoder. We use fφ′ to estimate rewards via (5). Also, we rescale P-SIL rewards so that the first episode is normalized to have a total return of −10 to account for rapid changes in encoder representations during initial training.
An important benefit of P-SIL is that computing imitation rewards does not add any computational burden as the cost of (5) is negligible, and we use the encoder of the RL agent. This is in contrast to adversarial approaches (Papagiannis & Li, 2020; Ho & Ermon, 2016), which require training a discriminator by solving an inner-loop maximization problem and then inferring rewards using the discriminator. We thus preserve the attractive computational efficiency of the base algorithm DrQ-v2.
In summary, we compute imitation rewards as the entropic Wasserstein discrepancy between the agent’s and expert’s encoded states for the single-demonstration case. For multiple expert demonstrations, we use the transport cost from the closest expert demonstration in the OT sense.
3.2 PIXEL DISCRIMINATOR ACTOR CRITIC (P-DAC)
We now propose and study P-DAC as a GAIL-based method for pixels that builds on the work by Ho & Ermon (2016); Torabi et al. (2018); Kostrikov et al. (2019). The key differences with previous works are that i) we compose the discriminator with the RL encoder to leverage its representations, ii) we apply data augmentation to discriminator inputs and iii) we store imitation rewards in the buffer rather than recomputing them at each iteration. We now describe the main components of P-DAC in more detail.
In contrast to P-SIL, it is necessary to train a discriminator, along with DrQ-v2’s other networks (actor, critic and encoder). To train the discriminator, we sample batches oe1:N and o a 1:N of size N from both the expert buffer and the policy buffer, respectively, apply data augmentation, and encode augmented observations using DrQ-v2’s encoder, which then yields:
oa,φ = [ fφ ◦ aug(oa1), . . . , fφ ◦ aug(oaN ) ] , oe,φ = [ fφ ◦ aug(oe1), . . . , fφ ◦ aug(oeN ) ] . (6)
Per common practice (Kostrikov et al., 2019), we omit importance sampling when estimating the policy state occupancy via replay buffer. We then maximize
max D N∑ n=1 logD(oa,φn ) + N∑ n=1 log(1−D(oe,φn ))− Eōφ [∥∥∇D(ōφ)∥∥− 1]2 , (7)
with respect to the discriminator’s weights and add a gradient penalty as recommended by Kostrikov et al. (2019). Here, ōφs are sampled along straight lines between agent and expert’s encoded observations. To compute rewards at the end of episodic rollouts, we embed the observation trajectory with DrQ-v2’s encoder and then apply the discriminator to the encoded observations to obtain rewards
r(oat ) = log ( σ ◦D(oa,φt ) ) − log ( 1− σ ◦D(oa,φt ) ) , (8)
where σ is the sigmoid function. We then add the episode with imitation rewards computed in (8) to the replay buffer. This differs from the original GAIL implementation (Ho & Ermon, 2016), where training is online, and from its follow-up offline extension DAC (Kostrikov et al., 2019) that recomputes rewards at each training iteration. We will show in the experiment section that data augmentation is a key component to scaling to challenging environments. Finally, in contrast with previous works that parameterize the discriminator directly in observation space (Ho & Ermon, 2016; Torabi et al., 2018; Kostrikov et al., 2019), we compose it with the RL encoder, which allows us to leverage its representations.

heading: 4 EXPERIMENTS
text: In this section, we empirically evaluate the proposed algorithms on tasks from the DeepMind control suite, aiming to contrast their strengths and weaknesses. We also provide an extensive ablation study highlighting the key design choices that enable solving challenging tasks with imitation learning from visual observations only.
Our set of experiments was designed with the aim of answering the following questions:
1. Are P-DAC and P-SIL able to achieve expert performance, and how do they compare in terms of sample efficiency (number of interactions with the environment required to solve a task) and computational efficiency? Yes. P-DAC is more sample efficient than P-SIL, see fig. 2 while P-SIL has a 37% faster computational footprint than P-DAC, see fig. 3. Furthermore, our methods allow to learn policies that are closer to expert policies than baselines under the embedded Wasserstein metric, see fig. 4.
2. Are P-DAC and P-SIL robust to the number of expert demonstrations they are provided? Yes, see fig. 5.
3. Is data augmentation as essential and effective as it is in pixel-based reinforcement learning? Yes, agents cannot approach expert performance on nearly all task without data augmentation, see fig. 6.
4. Is optimal transport providing a gain over simpler metrics without alignment (e.g., cosine distance between trajectories)? Yes, there is a significant performance gain when leveraging OT alignments, see fig. 7
5. Are expert actions required to solve challenging tasks or are pixel observations enough? No, P-DAC without actions performs on par or better than P-DAC with actions on all environments, see fig. 8.

heading: 4.1 EXPERIMENTAL SETUP
text: Environments We consider 8 Mujoco (Todorov et al., 2012) tasks in the DeepMind control suite (Tassa et al., 2018). The selected tasks are distinct enough to demonstrate the versatility and robustness of P-DAC and P-SIL. Environment observations are stacks of three consecutive 84×84 RGB images. We evaluate the agents with the environment rewards, but they are not provided to the agents during training. For the simplest tasks, we allow a budget of 1.5M environment frames, while for hard tasks we allow 5M environment frame in main results. In ablations, we allow 3M for hard tasks.
Expert Demonstrations We gather expert demonstrations by training DrQ-v2 using the true environment rewards. We run 10 seeds and pick the seed that achieves highest episodic reward to generate expert trajectories (sequences of image observations).
Baselines As baselines, we consider our two proposed approaches, P-DAC and P-SIL, along with a behavioral cloning (BC) baseline that we strengthen by leveraging the same data augmentation as in our approaches. We also consider canonical extensions of DAC (Kostrikov et al., 2019) and SIL (Papagiannis & Li, 2020), in which the discriminator is directly defined on pixel-space, and without the tricks proposed in this paper (e.g., no data-augmentation applied to discriminator inputs, no target encoder for SIL). DAC and SIL baselines are implemented with the same DrQ-v2 backbone as P-DAC and P-SIL, hence also benefit from data-augmentation in the RL training loop, thus making the comparison setup fair. We note that the BC and DAC baselines are privileged since they access expert actions while our baselines do not.
Setup In all experiments, we run 10 seeds under each configuration and average results, while providing a 90% confidence interval. We compare agents with respect to two distinct scores. Firstly, we use the episodic return as a metric to verify whether the agent solves the task; secondly, we average the entropic Wasserstein distance with cosine cost between agent rollouts and expert demonstrations, both embedded via a common random encoder. This shows that agent rollouts are close to the expert
demonstrations they are trained on. Finally, in ablation bar-plots, we show episodic reward averaged over the final 10 environment episodes. App. D further describes all hyperparameters.

heading: 4.2 MAIN RESULTS
text: Sample efficiency We begin by comparing baselines with respect to sample efficiency, i.e., the number of environment interactions required to solve a task. We observe in fig. 2 that both P-DAC and P-SIL achieve expert performance on all environments besides acrobot swingup, which is a hard exploration task. Both approaches significantly surpass the performance of SIL, DAC and data-augmented behavioral cloning even though the latter two are privileged by knowing expert actions. We also note that P-DAC is more sample efficient than P-SIL.
In fig. 4, we show the entropic Wasserstein between embeddings of agent rollouts and expert trajectories; results correlate well with those under episodic return (see fig. 2). Again, P-SIL and P-DAC outperform data-augmented BC under this alternative metric, i.e., trajectories under these are closer in the embedded entropic Wasserstein sense than those of BC.
Computational efficiency We contrast the computational efficiency of our approaches in fig. 3; the comparison is fair as methods are implemented on a unified backbone. Both of our approaches solve the hard tasks in around 4 wall-clock hours on a single V100 GPU. This can be explained by the fact that there is no loss in frames per second (FPS) for P-SIL over its RL backbone (FPSPSIL = 101), while P-DAC’s FPS is impacted by the cost coming from discriminator training (FPSPDAC = 74). As a result, both approaches have comparable computational efficiency.

heading: 4.3 ABLATIONS
text: We now provide an extensive ablation contrasting the approaches and highlighting key design choices.
Number of demonstrations We first evaluate the robustness of baselines to the number of expert demonstrations provided to the learner. We train agents on 1, 10 and 50 demonstrations, on all tasks, and plot episodic return averaged across seeds in fig. 5. We observe that our approaches are robust to the number of demonstrations, with comparable performance for 1, 10 and 50 demonstrations across tasks (besides finger spin). By contrast, the data-augmented BC baseline is non-robust, and requires a large number of demonstrations (typically 50) to approach expert performance.
Data augmentation Next, we evaluate whether data augmentation is necessary to achieve expert performance. We observe that without augmentation, both P-DAC and P-SIL are not able to recover solid performance on nearly all tasks, especially locomotion ones. We note that behavioral cloning is less sensible to it. This may be explained by the fact that augmentation is necessary to obtain good representations as seen in Laskin et al. (2020); Yarats et al. (2019; 2021a), and that our imitation approaches leverage such representations to design reward signals. As a result, without good representations the agents can not recover any meaningful behaviors.
Metrics We continue with analyzing one of the key design choices behind P-SIL, i.e., whether an optimal transport alignment of the agent and expert trajectories is required over simply leveraging an l2 or cosine distance (no alignment). We observe in fig. 7 that OT is essential to achieving expert performance on most tasks. We believe this is due to the high variance in the initial state distribution of agent and expert, which leads to unaligned trajectories, and hence uninformative rewards in the absence of OT alignment.
Observations vs. Observation-actions GAIL-like methods are often assumed to require expert actions to recover non-trivial performance. We hence compare P-DAC with a privileged implementa-
tion of P-DAC which accesses expert actions. In the latter case, we concatenate embedded image observations, and actions before passing these to the discriminator. We observe in fig. 8 that our approach without expert actions performs on par with the action-baseline, highlighting that on the considered environments, actions are not required to achieve strong performance. This is encouraging given that there are massive amounts of unlabeled video data that these agents could potentially learn from.

heading: 5 CONCLUSION
text: We propose effective methods for imitation learning directly from pixels without expert actions, extending state-based adversarial and optimal transport approaches, and significantly outperforming canonical extensions of these. This steps us closer toward leveraging the rich amount of expert-level data on non-trivial environments for the control of continuous systems. We envision that continuing to scale in this space will involve further understanding how to recover a representation of the high-dimensional data, e.g. with unsupervised pretraining, and how to go beyond settings where the expert and learner live in the same space.

heading: A BACKGROUND ON OPTIMAL TRANSPORT
text: Optimal transport (Villani, 2009; Peyré & Cuturi, 2019) tools allow us to compare probability measures while incorporating the geometry of the space. The entropic 2-Wasserstein distance between two discrete measures µx = 1T ∑T t=1 δxt and µy = 1 T ∑T t=1 δyt is
W2 (µx, µy) = min ψ∈Ψ T∑ t,t′=1 d2(xt,yt′)γt,t′ −H(π), H(ψ) = − T∑ t,t′=1 ψt,t′ logψt,t′ , (9)
where Ψ = {ψ ∈ RT×T : ψ1 = ψT1 = 1T 1} is the set of coupling matrices, and d is a metric. Adding the entropic term has two main benefits: i) the induced problem can be solved in quadratic time via Sinkhorn’s algorithm (Sinkhorn & Knopp, 1967), and ii) the resulting distance is smooth in the measures’ samples. Intuitively, the optimal coupling matrix ψ provides an alignment of the samples of µx, µy which minimizes the cost of transport between these. The distance then consists of the weighted sum of distances between aligned samples, along with an entropic penalty.

heading: B DRQ-V2
text: We use DrQv2 (Yarats et al., 2021a), which is a actor-critic method for continuous control based on the deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015). Given a replay buffer D, it learns simultaneously a Q-function Qθ and a policy πη. Qθ is trained by clipped double-Qlearning (Fujimoto et al., 2018) with n-step returns. πη is trained via deterministic policy gradient. DrQv2 employs data augmentation in the form of random shifts with padding and a random crop to restore the original image dimension, followed by bilinear interpolation. Data augmentation acts as regularization and reduces the variance of the Q estimates. Images are embedded into the latent space via an encoder fφ after being augmented. This encoder is trained to minimize the critic loss only.

heading: C ALGORITHMS
text: Algorithm 1 Inverse reinforcement learning core. Different methods can be instantiated by changing the rewarder function. Require: Expert demonstrations {µoen }Nn=1, replay buffer D, initialized policy network π, Q-
network Q and encoder f . For DAC-like baselines, also requires a discriminator D. for t ∈ Ttotal do
if done then r1:T = rewarder(episode), for instance (5) for P-SIL, (8) for P-DAC Update episode with r1:T and add all quadruples [ot,at,ot+1, rt] to D. ot = env.reset(), done = False, episode = [ ] end if at ∼ π(·|ot)→ ot+1,done = env.step(at), episode.append([ot,at,ot+1]) Update DrQ-v2’s actor and critic, and rewarder-specific functions using D.
end for

heading: D HYPERPARAMETERS AND FURTHER EXPERIMENTAL DETAILS.
text: We provide a list of all hyperparameters in table 1. The only environment-specific variation is that the number of n-steps is set to 1 for walker tasks. For P-DAC, we use the DrQ-v2’s critic encoder, while for P-SIL we use the actor’s encoder, noting that both share the same convolutional layer weights.

heading: E EXTRA ABLATION PLOTS

Review summaries:
This paper introduces two methods for imitation learning from pixel observations based on adversarial learning and optimal transport. Representation from the RL encoder are adopted for calculation imitation rewards. P-SIL (Pixel Sinkhorn Imitation Learning) learns and compares latent representations of the encoded expert and agent behaviors; and P-SAC (Pixel Discriminator Actor Critic) augments the representations of the agents and the experts. The key component of this work is the RL encoder used in both P-SIL and P-SAC. 

The authors have considered the problem of imitation learning using only visual observations, and they've explored the paradigms of using methods based on adversarial learning and optimal transport to solve this problem. In particular, the authors propose two new methods--P-SIL and P-DAC--presumably as a way to explore design choices and tradeoffs in this space. Some experiments designed to demonstrate the superiority of P-SIL and P-DAC are presented.

This paper presents two algorithms for imitation learning P-SIL and P-DAC which are built on top of SIL (Papagiannis & Li, 2020) and DAC (Kostrikov et al, 2019). The first one is an algorithm based on the Sinkhorn distance and the second one uses an adversarial approach, by training a discriminator. These two approaches use state-only trajectories to generate reward sequences to match the agent state-trajectories and learn through that reward. By combining the advances in image-based reinforcement learning by the usage of data augmentation and target encoders, the two new algorithms, P-SIL and P-DAC, are tested and reach good results on visual control tasks. 

This paper focuses on learning visual policies by imitating video data without observing the expert actions. The authors introduce a method to compute the imitation reward based on the representation from the expert RL encoder. Besides, a data augmentation method is proposed to scale to non-trivial control tasks. They conduct comprehensive experiments on DeepMind control suite tasks, showing the versatility and strong performance of the two approaches (P-DAC and P-SIL).