Domain-Robust Visual Imitation Learning with Mutual Information Constraints
Authors: Edoardo Cetin
Oya Celiktutan
Venue: ICLR 2021 Poster (2021)
Keywords: Imitation Learning
Reinforcement Learning
Observational Imitation
Third-Person Imitation
Mutual Information
Domain Adaption
Machine Learning

Abstract: Human beings are able to understand objectives and learn by simply observing others perform a task. Imitation learning methods aim to replicate such capabilities, however, they generally depend on access to a full set of optimal states and actions taken with the agent's actuators and from the agent's point of view. In this paper, we introduce a new algorithm - called Disentangling Generative Adversarial Imitation Learning (DisentanGAIL) - with the purpose of bypassing such constraints. Our algorithm enables autonomous agents to learn directly from high dimensional observations of an expert performing a task, by making use of adversarial learning with a latent representation inside the discriminator network. Such latent representation is regularized through mutual information constraints to incentivize learning only features that encode information about the completion levels of the task being demonstrated. This allows to obtain a shared feature space to successfully perform imitation while disregarding the differences between the expert's and the agent's domains. Empirically, our algorithm is able to efficiently imitate in a diverse range of control problems including balancing, manipulation and locomotive tasks, while being robust to various domain differences in terms of both environment appearance and agent embodiment.

Published as a conference paper at ICLR 2021
DOMAIN-ROBUST VISUAL IMITATION LEARNING WITH MUTUAL INFORMATION CONSTRAINTS
Edoardo Cetin & Oya Celiktutan
Centre for Robotics Research
Department of Engineering, King's College London
(edoardo. cet ietiin, oya. ce kel.kttta)))kcl..e aac.:....kl..............
ABSTRACT
Human beings are able to understand objectives and learn by simply observing others perform task. Imitation learning methods aim to replicate such capa- bilities, however, they generally depend on access to a full set of optimal states and actions taken with the agent's 's actuators and from the agent's 's point of view. In this paper, we introduce new algorithm called Disentangling Generative Adversarial Imitation Learning ( (DisentanGAlL)- with the purpose of bypassing such constraints. Our algorithm enables autonomous agents to learn directly from high dimensional observations of expert performing task. by making use of adversarial learning with latent representation inside the discriminator network. Such latent representation is regularized through mutual information constraints to incentivize learning only features that encode information about the completion levels of the task being demonstrated. This allows to obtain shared feature space to successfully perform imitation while disregarding the differences between the expert's and the agent's domains. Empirically, our algorithm is able efficiently imitate in a diverse range of control problems including balancing, manipulation and locomotive tasks, while being robust to various domain differences terms of both environment appearance and agent embodiment.
INTRODUCTION
Recent advances demonstrated the strengths of combining reinforcement learning (RL) with power- ful function approximators to obtain effective behavior for high dimensional control tasks (Lillicrap al., 2015; Schulman al., 2017: Haarnoja et al., 2018a). However, RL's reliance reward function introduces a fundamental limitation as reward specification and instrumentation can bring about a great design burden to potential users aiming train an agent for a novel problem.
An alternative approach for addressing this limitation to recover a learning signal through expert demonstrations. Most of the past work exploring this area focused the problem setting where demonstrations are provided directly from the agent's 's point of view and through the agent's actu- ators, which we refer as agent-centric imitation. However, applying agent-centric imitation for real-world robot learning would demand users to provide a diverse range of kinesthetic or teleoper- ated demonstrations to a robotic platform, leading to an unnatural user-agent interaction process.
In this paper, we focus instead on learning effective policies solely from set of external, high dimensional observations of different expert agent executing task. We refer this problem formulation as observational imitation. Solving this requires disentangling the expert's intentions from the observations' context, which has been a challenging problem for prior research, and often relied on additional assumptions about the environment and expert data (Torabi al., 2019).
We propose a novel algorithm, called Disentangling Generative Adversarial Imitation Learning (DisentanGAIL), to acquire effective agent behavior without such limitations. Our technique is based on the framework of inverse reinforcement learning, yet, it enables an agent to learn with only access to observations collected by watching structurally different expert. DisentanCAlL utilizes an off-policy learner alongside a novel discriminator with a latent representation bottleneck, regularized to represent domain invariant space the agent's and expert's sets of observations.

---

Published as a conference paper at ICLR 2021
This is achieved by enforcing two constraints on the estimated mutual information between the latent representation and the origin collected observations.
In particular, the contribution of this work for solving observational imitation is threefold:
We propose a discriminator making use of novel mutual information constraints, and provide techniques to adaptively and consistently ensure their enforcement.
We identify the problem domain information disguising when estimating mutual information and propose structural modifications to our models for its prevention.
We show that, unlike prior work, our algorithm can scale to high dimensional tasks while being robust to domain differences in both environment appearance and agent embodiment, by testing on a novel diverse set of tasks with varying difficulty.
RELATED WORK
Agent-centric imitation has been long-studied problem setting. Behavior cloning (Pomerleau, 1989 1991; Ross et al., 2011) was first proposed to approach imitation from supervised learning perspective. Particularly, an agent trained to maximize the likelihood of executing set of recorded optimal actions from the states encountered by the expert. Inverse reinforcement learning (IRL) (Ng al., 2000; Abbeel & Ng. 2004; Ratliff et al., 2006) was more recently proposed as an alternative two-step solution to imitation and has been often shown to be more effective. First, IRL. aims to recover a reward function by parameterizing a discriminator, trained to be representative of the objective portrayed by the expert demonstrations. Second, it tries to learn behavior to accomplish such objective, utilizing RL. To effectively understand and represent the expert's s intentions, modern instantiations of IRL combined the maximum entropy problem formulation (Ziebart al., 2008) with deep learning (Wulfmeier et al., 2015; Finn et al., 2016b), and proposed a direct connection with adversarial learning (Ho & Ermon, 2016; Finn et al., 2016a; Kostrikov al., 2018). This allowed for successful imitation in complex control tasks with few expert demonstrations. Related our algorithm, Peng et al. (2018) implemented a variational bottleneck limit information flow in the discriminator, for tackling the discriminator saturation problem (Arjovsky & Bottou, 2017). Additionally, Zolna et al. (2019) proposed to optimize the discriminator to be maximally uncertain about uninformative sets of data as a form of regularization to disregard irrelevant features.
different line of research instead considered imitating from observations of an expert performing a task, in problem settings resembling observational imitation. Earlier methods proposed to use hand- engineered mappings to domain invariant features (Gioioso et al., 2012; Gupta et al., 2016), while more recent works proposed to learn such mappings, relying on specific prior data obtained under both the agent and expert perspectives. These techniques include using aliened demonstrations (Gupta et al., 2017: Sermanet et al., 2018; Liu et al., 2018; Sharma et al., 2019), or multiple tasks where the agent and the expert already achieved expertise (Smith et al., 2019; Kim et al., 2019).
While effective, these methods for observational imitation make considerable assumptions about the task structure and the available data. Therefore, they have limited applicability for arbitrary prob- lems, where environment instrumentation and prior knowledge are minimal. The work most related to ours is by Stadie et al. (2017), where domain invariant representation is learned through utilizing domain confusion loss that requires two different expert policies for sampling failure and success demonstrations in the expert domain. While this approach was also adopted in recent works (Oku- mura et al., 2020; Choi et al., 2020), empirically, yielded successful imitation results only when working in low dimensional control tasks and with the agent and expert domains differing solely in their appearance. On the contrary, our algorithm only requires a limited set of expert demonstrations and allows for successful imitation in both low and high dimensional control tasks, with the expert's and agent's 's domains differing in both environment appearance and agent embodiment.
BACKGROUND AND PRELIMINARIES
3.1 ADVERSARIAL IMITATION LEARNING
In imitation learning, the agent is provided set of expert demonstrations B5 = {Ti, T2..., TN); where each 7 (8o, 0g 81, .1,,..,,
..) 87) represents trajectory collected with expert policy from
2

---

Published as a conference paper at ICLR 2021
the agent's point of view. These demonstrations used to provide the learning signal for the agent to improve its policy я. In Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016) this learning signal is obtained through pseudo-reward function Rp. derived from discriminator network D trained to discern between 'expert' and 'agent' state-action-nex state triplets:
arg max EBg (ss; ai 8(+1)) + Bp-(r) log(1 - D(s,, O2y 8i+1));
where (T) is the distribution of trajectories encountered by the agent, stemming from both the environment's dynamics and its own current policy Reinforcement learning methods then applied for x to adversarially maximize the sum of encountered pseudo-rewards:
RD( ce sc+1) (0.6+1)))))
arg max Bpe (T)
(2)
where RD(S; Qg, 8:+1) log D( az. (1+1)) log(1 D(s; az 8;41)). Ho & Ermon (2016) pro- posed to iteratively execute the adversarial optimization steps described in Eqs. 1 and 2. optimizing through the Trust Region Policy Optimization algorithm (Schulman et al., 2015).
3.2 Observational IMITATION LEARNING
In the problem setting of observational imitation, we are concerned with learning without knowl- edge of the states visited and the actions taken by the expert. We purely rely observations, O, provided in a set of expert demonstrations Bg = {Tí. T2... NN containing visual tra- jectories 7i = (o), .i., oyy)) Each visual trajectory of is a sequence of observations obtained by watching the expert act
domain. Here, the discriminator D will be optimized to discern between the expert demonstrations in Bg and 'agent' observations in Br, a set of visual trajec- tories collected by watching the agent act according to x. We consider each observation to be an RGB image of the environment at the current time-step, hence, containing only partial and highly- entangled information about the true state. We define the expert's and agent's domains as distinct Partially Observed Markov Decision Processes (POMDPs), ME = (Sg. Ag, Og, P, Po: R) and MA = AA. A. P. Po. R). respectively. The main challenge in observational imitation that Og in the expert's POMDP do not neces-
the states s E Sg, actions Ag and observations sarily match the states E SA. actions a E A and observations o E O in the agent's POMDP.
3.3 MUTUAL INFORMATION
To overcome the differences between the expert's and agent's POMDPs, our algorithm relies on esti- mating the mutual information between different sets of variables. Mutual information is a statistical measure that represents the level of dependency between two random variables and quantifies how much information each random variable is expected to contain about the other (Kinney & Atwal, 2014). In other words, the mutual information between X and Z measures the difference between the entropy of X and the conditional entropy of X given Z:
T(X, H(X) - ( = H(2) - H(Z
Belghazi et al. (2018) showed that the mutual information can be effectively estimated through the Mutual Information Neural Estimator (MINE). This consists in lower bounding the mutual informa- tion through the Donskher- Varadhan dual representation of the KL divergence between two random variables distributions (Donsker & Varadhan, 1975). by searching for a parameterized function Tş:
I(X, 2) 2 sup FPXx.2) )) log (Ep()),PIz) [To(a.)))
(4)
de#
4 DISENTANGAIL
DisentanGAIL utilizes several algorithmic components to address the problem of observational im- itation. Particularly, its discriminator, D, is regularized by the enforeement of two mutual infor- mation constraints between the domain origin and a specific latent representation the collected observations. This enables the pseudo-rewards, Rp, to disregard the domain differences between the
3

---

Published as a conference paper at ICLR 2021
4
T
Zy-i
Por
Z--3
S0z
Dg
Figure 1: Simplified discriminator optimization structure: for each time-step the four most recent observations 0u:-33 are processed independently by the preprocessor Pe,, outputting the correspond- ing latent representations Zt:--3. The latent representations are then concatenated and fed jointly into the invariant discriminator Sog, and fed individually into the statistics network Tø. outputting respectively the GAIL objective Jg. and the penalty loss I.g or T.x-
expert's 's and agent's domain and provide a meaningful learning signal to improve the agent's policy, и. Together with the visual trajectories in BE and B., DisentanGAIL also exploits sets of prior data collected in the expert's and agent's 's domains, denoted by B and Bp.- respectively. Such data is obtained by recording observations from the expert's and agent's domains, while neither expert nor agent is attempting to perform the target task, e.g., while they are acting randomly.
4.1 DISCRIMINATOR COMPONENTS
Our algorithm for observational imitation utilizes a convolutional neural network discriminator Dos optimized for outputting the probability that input sequence of observations occurred conse- quence of expert behavior. As illustrated Fig. I, our discriminator can be divided into two distinct sub-models, namely, the preprocessor Pe, and the invariant discriminator Seg:
Dg So Po.
We define the preprocessor parameterized multivariate Gaussian distribution with diagonal co- variance Pg, = {ue, : Ee, J. from which a latent representation is sampled for each input observation z; N (Ho, Eo, (o,)). The preprocessor's 's objective is project each observation into latent space containing information about the achievement state of the goal, disregarding the irrelevant information about the inherent differences between the expert's and agent's domains. The Gaussian representation ensures that, for any input, the support over the possible latent representations is infinite, moreover, it allows the model to directly reduce the information any of the independent dimensions of z by increasing the corresponding variance in Σe, (o:).
Based on these latent representations, the invariant discriminator Se, is tasked to output the dis- criminator score for the observed behavior. To classify behavior at any time-step, Sş, takes as input the concatenated sequence of the latent representations of the four most recent observations, 2, concat 2,, 2;-1, 2t-2, 2t-3). Feeding concatenation of the latent representations over mul- tiple time-steps serves the purpose of facilitating the recovery of information regarding the true unobserved state of the POMDP from the observations. It also allows the discriminator to reason di- rectly with goal-completion progress throughout different consecutive observations. To understand the necessity of this practice, consider navigation problem where the agent's 's task is reach target position. Only given access to information about multiple visual observations showing the location of the agent different time-steps, the discriminator will be able assess if the agent is approaching the target and retrieve higher-order state information about its motion.
Both the preprocessor and invariant discriminator trained end-to-end through the reparameteri- zation trick (Kingma& & Welling, 2013), to optimize the GAIL objective JG to discern behavior from the set of expert demonstrations BE against behavior from the of recent agent observations B,:
arg max Jo(0, (0, BE, Bx) arg max LE Bg, log(Sa, (2:)) + EB., log(1 Sa, (2:)). (6)
4.2 MUTUAL INFORMATION CONSTRAINTS
To obtain an invariant latent space from the preprocessor's output, we propose to enforce two different constraints on the mutual information between the observation's 's latent representations ~ Po, (o:) and a corresponding set of domain labels. Each domain label d, is a binary vari- able representing whether the associated observation o, was collected in the expert POMDP, i.e.,
4

---

Published as a conference paper at ICLR 2021
d, d= BB. To estimate the mutual information, make use of the MINE estimator and utilize statisties network To optimized maximize the objective in Eq. 4 between the latent representations and the domain labels for the observations in BE and Bx:
(Re(U)).(4)) ( (eFolando)) (7)
arg max To(z, d, Bx) arg max PP(4),
Expert demonstrations constraint. The first mutual information constraint is for the latent repre- sentations of the observations from the union of Bg with B,. We propose to constraint the estimated mutual information of these latent representations with the corresponding domain labels be less than 1 bit: (z, d:|Bp U Bx) < 1. We define two kinds of information that the preprocessor Po can encode into the latent representations to aid the invariant discriminator So₂ in discerning transitions Out-33 from Bg and _ (i) domain information, from the visual differences of the two environments (labeled by dz), or (ii) noal-completion information, from the expected progress shown in the observations towards achieving the goal demonstrated by the expert in B5 (represented by the variable Gg). By constraining the mutual information of the latent representations with the domain labels d, to be less than bit, we prevent the invariant discriminator to exclusively rely informa- tion inherent to the domain origin to make its classification decision. Therefore, we force to seek goal-completion information about G, to fully optimize its objective from Eq. 6. We empirically evaluate this constraint against tighter constraints in Appendix D.
Prior data constraint. An additional mutual information constraint is for the latent representations of the observations from the union of prior data sets collected independently in both agent and expert domains, By. and Br... The observations collected in these sets expected come from observing both expert's and agent's 's domains, while neither expert or agent are attempting to perform the target task. Hence, by assuming that the goal-completion levels observed in these two sets approximately match, we can constraint the mutual information of the relative latent representations with the domain labels to be near namely I& (z, di Bp. U Br.=)= = This mutual information constraint implicitly optimizes for mapping which makes the distributions of latent representations generated from the observations in the two prior sets of data equivalent. Hence, allows for the utilization of great amounts of cheaply collected unsupervised data to provide an additional learning signal regarding the information which should be discarded by the preprocessor.
Comparison with prior efforts. Stadie et al. (2017) proposed to constraint the mutual information between the encoded observations and the domain labels in BgUB, Br to be O. We argue that enforcing such constraint would unnecessarily limit the information in the latent representations and impair learning. This constraint assumes that some observable factor determining o; is independent of the domain labels d,, otherwise, no information about c, can be encoded in the latent representations. However, given that in BE U Ba we have og E Bg + d4 = 1, such assumption seldom holds, as it requires the distributions of some goal-completion information about et present in the observations in Bg and B, to exactly match. However, unlike this work, the algorithm proposed by Stadie et al. (2017) does not attempt to enforce such constraint precisely. Instead, it simply penalizes a measure proportional to the mutual information via a domain confusion loss with fixed weight coefficient. We further discuss the implications of this practice and provide a toy example where truly enforcing such constraint would prevent any learning in Appendix A.
4.3 OFF-POLICY LEARNING
To learn effective behaviour, we combine our regularized DisentanGAIL discriminator with the off- policy Soft-Actor Critic (SAC) algorithm by Haarnoja et al. (2018b). To optimize parameterized agent policy, Tw. SAC maximizes the expected sum of entropy regularized pseudo-rewards:
arg max (ww) arg max E (mm RD 01 0-)) anw (cels))
(8)
E
IMPLEMENTATION
5.1 ENFORCING THE MUTUAL INFORMATION CONSTRAINTS
We implement two different techniques to enforce the mutual information constraints proposed in Section 4.2, given set of observations B with the corresponding domain labels. Both techniques
5

---

Published as a conference paper at ICLR 2021
make use of single arper-parameter maz, which represents the upper limit on the estimated infor- mation about the domain labels that we allow the latent representations to retain.
Adaptive penalty Lg. The first technique is having supplementary loss function for the preproces- sor Po.. penalizing it proportionally to the estimated mutual information in the latent representations. We use an adaptive parameter / to ensure the mutual information is within the desired range:
(g B) (Z, de BB)
We design our updates of follow similar pattern to the adaptive penalty coefficient proposed by Schulman et al. (2017), utilizing Iaa and updating:
- 3 1.5, if (z,,, d:(B) Lmas
3+ 1.5, L (o(z,, Imaz + 2.
Dual penalty Lx. The second technique consists in different supplementary loss function penal- izing the preprocessor Pe. proportionally to the violation of the upper limit. In this case, we ensure constraint enforcement through the introduction of a non-negative Lagrange multiplier variable A:
(0 B) = A (T& (z,, d: BB Imaa))
(10)
where À is updated to maximize L,, approximating dual gradient descent (Boyd et al., 2004):
A max (0, A (Lg (Zi, d)\ mma)))
(11)
In practice, the dual penalty enforces precisely the mutual information constraints, but the dual variable À stabilizes in more iterations than the adaptive parameter 3. Our final implementation
makes use of the adaptive penalty when enforcing the expert demonstrations constraint with Lmaz 0.99 and the dual penalty when enforcing the prior data constraint with Lmax = 0,001. Hence, our penalized discriminator objective augments the original diseriminator objective in Eq. 6s:
arg max Jo(0, Bg U Br)) Lg(0r; Be U Bx) (01, Br. U Br.))
(12)
5.2 DOMAIN INFORMATION DISGUISING
The discriminator Dg is updated to maximize Eq. 12, given a mutual information estimate from a fixed-sized statistics network Tg. Hence, if the optimization of Tę temporarily converges sub-optimal local minimum, Do could encode domain information into the latent representations z without the statistics network detecting We refer to this phenomenon as domain information disguising, and we utilize two further techniques to prevent this issue.
Double statisties network. The first technique takes inspiration from the work of Van Hasselt et al. (2016) and consists of learning independently two statistics neural networks, namely To, and TØp- Thus, the mutual information is estimated through taking the maximum prediction of the indepen- dent models over the same set of observations, [2, B) = max(Ie, (z;, d, B), )), Io,(2, (z,, B)). This change makes it impractical for the discriminator disguise domain information, as the gra- dient from each of the regularization losses can only contain information about a single statistics network at time (from the max operation). Therefore, this gives statistics network reaching sub-optimal local minimum the chance to recover, without affecting the mutual information esti- mates great extent. This practice has also the benefit of providing a better prediction of the current mutual information, counteracting the effects of epistemic uncertainty on the optimization.
Invariant diseriminator regularization. The second technique comprises regularizing the invariant discriminator So, to be approximately 1-Lipschitz. A great part of the GAN literature (Arjovsky al.. 2017: Gulrajani et al., 2017; Miyato et al., 2018) showed the effectiveness of this practice when regularizing discriminator networks. In our specific problem setting, it has the further benefit of restricting the expressivity of the invariant discriminator, preventing it from capturing domain information not captured by our mutual information estimator. To enforce this, utilize spectral normalization, a regularization technique proposed by Miyato et al. (2018).
DisentanGAIL trains all the models end-to-end, in three main learning steps: (i) Discriminator learn- ing, where the discriminator's 's parameters 0 are updated to maximize Eq. 12: (ii) Mutual information learning, where the statistics network's parameters é are updated maximize Eq. 7; (iii Agent learning, where the learner's ' parameters w are updated to maximize Eq. 8 with SAC. We provide further implementation details and formal summary of the algorithm in Appendix B.

---

Published as a conference paper at ICLR 2021
Poskinm
Sarre
22
Steer
Adutad
Alate' Rocker
Same
Diselamfikl
DisntlanntAil without prior deln
10
aneettnn-All with domain confusion lowa
No Idbett representid in reguler melmm
Figure 2: Performance curves for the Inverted Pendulun (Top) and Reacher (Bottom) realms. Dis- entanGAIL the only algorithm consistently achieving performance close the 'expert' agent's.
Table Results summary for the Inverted Pendulum and Reacher environment realms
Discronees betwent the agent and the expest donatts
No differvares
Embhodhmont
Appecnance
Emfoddio and appeamsnve
Algorithms evalated:
Reacher
Inserted Pendslan
Reacher
Inverted Fonalitls
Reacher
Inserted Pondolum
Regther
Inveried Fondulis
Dommust--L
0.073 0.10:: 0.104
1721
11.03
03541
0.045
0.964 0.081
0.880 = 0.064
0,354
002551
0.860 = 0.081
0.918 0.114
GAl ALL (No proor)
1,004
1:0.012
1015 0.023
0.517
0.044
0.914 0:1.
0.596 1:555
0.791
0.255
0.578
0,160
0.587
TPR.
0.251
0.111
0.81200.66 0.162
0.185
0.000
0.215
0.119
0.276
0.217
0.330
0.122
0.255
0.154
0 0.256 0.100
TPIL experience)
0.883
0.48
0.296
0.279
0.585
1.255
0.510
0,626
0.282
DosmuodAA, (DCL)
1.024
0.026
0.169
0.550
0.523
No engalarizaiios
0.388
1.088
0.200
0.281
0.677
0.386
0.1882
EXPERIMENTS
To evaluate our algorithm, we design six different environment realms, simulated with Mujoco (Todorov al., 2012), extending the environments from Brockman al. (2016): Inverted Pendu- lum, Reacher, Hopper, Half-Cheetah, 7DOF-Pusher and 7DOF-Striker. We define an environment realm as of environments with shared semantic goal but with significant differences in terms of appearance and agent embodiment. For each of the experiments, we select source environment and target environment within one environment realm. Thus, we train 'expert' agent and collect set of visual trajectories in the source environment. An 'observer' agent will then use these visual trajectories to perform imitation the target environment, without access to the reward function. In all our experiments, each epoch corresponds to the 'observer" agent collecting 1000 time-steps of experience in the targel environment. We report at each epoch the mean and standard error over five experiments of the maximum expected cumulative reward recorded so far. We obtain the expected cumulative reward by averaging the performance of the 'observer' agent over five trajectories. We scale the cumulative rewards such that 0 represents the performance from random behavior, and represents the performance obtained by the 'expert' agent. We provide a detailed description of the different environment realms in Appendix C.
Can DisentanGAlL efficiently solve the problem of observational imitation with both appear- and embodiment mismatches? We first evaluate the performance of algorithm the

---

Published as a conference paper at ICLR 2021
Pode
10w
124
15
emproseettsiii regi
rspresontation regulanizaton source envirommennt
Figure 3: Performance curves for the Hopper (Top-left), Half-Cheede (Bottom-left), 7DOF-PPstie (Top-right) and 7DOF- (Bottom-righht environment realms.
Inverted Pendulum and Reacher realms. We test eight different combinations source and targer environments for each of these realms. We allow the 'observer' agent to train for maximum of 20 epochs. evaluate effectiveness of our proposed constraints and techniques in solving problem of observational imitation, compare the performance of the following algorithms:
DisentunGAl:: The full proposed algorithm, as described in Section 5.
DisentanGAlL without prior data (No prior): DisentanGAlL without the prior data constraint.
TPIL: The original implementation the algorithm from Stadic et al. (2017).
DisentanGAIL with domain confusion loss (DCL): Re-implementation of the domain confusion loss by Stadie et (2017) applied DisentanGAlL, substituting the proposed constraints.
No latent representation regularization (No regularization): DisentanGAIL model without any loss constraint to prevent encoding domain information in its latent representations.
We present the performance curves in Fig. 2 and summary of the results in Table 1. Particularly, the full DisentanGAlL algorithm outperforms all other algorithms when considering any domain difference and consistently achieves performance close the 'expert' agent. In comparison, TPIL severely under-performs, even when evaluated given five times the amount experience. Applying the domain confusion loss to DisentanGAlL significantly and consistently deteriorates the perfor- mance, validating the effectiveness of our proposed constraints. However, this version of Disen- tanGAIL still vastly outperforms the original TPIL implementation, underlying the superiority of our proposed model and optimization. DisentanGAlL with no prior data, performs well in most experiments, but under performs when faced with drastic changes in domain appearance, indicat- ing that the utilization of sets of prior data is important when strong visual cues about environment correspondences are missing. We report additional ablation studies for our model in Appendix D.
Can DisentanGAlL. scale to more challenging, high dimensional control tasks? We evaluate our algorithm on the four remaining realms, which consist of substantially harder problems, nar- rowing the evaluation gap with agent-centric imitation algorithms. We refer to the environments in these realms as 'high dimensional' since their state and action spaces are significantly larger than the state and action spaces of the environments explored in prior work making confusion loss (Stadie et al., 2017: Okumura et al., 2020; Choi et al., 2020). Namely, we explore two locomotion realms, Hopper and Half-Cheetale and two manipulation realms, 7DOF-Pusher and
of the domain
8

---

Published as a conference paper at ICLR 2021
Table 2: Results summary for the 'high dimensional' environment realms
Enviroamen: realms
Algwrihhms evalonted:
Hopper
Half-Cheettk
7DOF-Pusher
700F-Stnkke
DismtanGAAI..
0.026
0.901 0.044
DisemtoGAAL (No prioe)
0.622
0.051
0.8G0
00231
0.677 0.072
0.707 0.050
TPIL
0.302 + 0.U57
0.066 = 0.020
-0.020 = 0.06a
0.081 + 0.065
DisentaeGAII (DCT.)
0.619 0.036
0.486 0.058
0.747 4 0.054
0.554 0.134
No reguleriuttonn
0.039
0.247
0.062
0.637
0.604
No regulorization (source)
0.866 +: 0.100
0.914 + 0.041
0.901 +: 0.044
0.837 0,038
7DOF-Striker. We use DisentanGAIL to perform observational imitation with the 'source' and "tar- get' environments differing greatly both terms of appearance and agent embodiment, as detailed in Appendix C. We compare DisentanGAlL. with the rreviously-inrroduee baselines. To provide an upper bound on the expected performance of DisentanGAll, we additionally evaluate the No la- tenl representation regularization baseline with the 'observer' agent learning in the original 'source' environment, i.c., imitating with no domain differences.
We present the performance curves in Fig. 3 and a summary of the results in Table 2. Remarkably, DisentanGAIL able to recover close to the expert's performance both manipulation realms, with at least the same efficiency the No latent representation regularization baseline learning in the 'source' environment. In the locomotion realms, the performance appears to converge more slowly to similar but lower value than the expert. We hypothesize this is because the main objec- tives in the locomotion realms are based on the agents continuously executing a particular stream of actions rather than reaching target state. Thus, the analyzed shifts in agent's 's embodiment, modifying the action-spaces dimensionality, strongly increase the discriminator ambiguity on re- warding the best possible way to solve the tasks. The performance gap of DisentanGAlL with the rest of the baselines is considerably greater than in the previous set of experiments. In particular, TPIL and the No latent representation regularization baseline fail to recover meaningful behavior in any experiment. Similarly, applying the domain confusion loss to DisentanGAIL degrades consid- erably the performance across all problems. Additionally, removing the prior data constraint from DisentanGAIL also appears to degrade the performance. Yet, DisentanGAlL. with no prior data still outperforms all other baselines in three environment realms and is able to almost match the full DisentonGAIL performance in the performing observational imitation in high dimensional environments and show the effectiveness of proposed constraints and optimization.
realm. These results highlight the complexity of
7 CONCLUSION
We proposed DisentanGAll- - novel algorithm to effectively solve the problem of observational imitation. Our method makes use of two mutual information constraints for latent representa- tion inside the discriminator network to encode goal-completion information and discard domain information about the observations. Unlike prior work, our experiments show DisentanGAIL's effectiveness at dealing with various domain differences, both in terms of environment appear- ance and agent embodiment, and at scaling more complex high dimensional tasks. We believe our work might have strong implications for future real-world imitation learning, as it could al- low users teach agents tasks by simply being observed, leading to natural human-robot interactions. To facilitate future efforts, we share the code for our algorithms and environments: https://gitee www.:/uw.ooooaaa..o.//aaooooooee dww.aeroooooooooooooooiioaiiiiea ceaabuut-iiuua-iii
ACKNOWLEDGMENTS
Edoardo Cetin would like to acknowledge the support from the Engincering and Physical Sciences Research Council [EP/R513064]

---

Published as a conference paper at ICLR 2021
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. ACM, 2004.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arxiv e-prints, art. arXiv preprint arXiv: 1ww.11711....... 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein
gan. arXiv preprint
arXiv: 2017.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. International Conference Machine Learning, pp. 531-540 2018.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge uni- versity press, 2004.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1606v 01 540, 2016.
Sungho Choi, Seungyul Han, Woojun Kim, and Youngchul Sung. Cross-domain imitation learning with a dual structure. arXiv preprint arXiv:2006.0144 01494, 2020.
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time, i. Communications on Pure and Applied Mathematics, 28(1):1-47, 1975.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852. 2016a.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pp. 49-58, 2016b.
Guido Gioioso, Gionata Salvietti, Monica Malvezzi, and Daniele Prattichizzo. An object-based approach map human hand synergies onto robotic hands with dissimilar kinematics. Robotics: Science and Systems VIII. pp. 97-104, 2012.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In Advances neural information processing systems, pp. 2767-5777,2011
Abhishek Gupta, Clemens Eppner, Sergey Levine, and Pieter Abbeel. Learning dexterous manip- ulation for soft robotic hand from human demonstrations. In 2016 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 3786-3793. IEEE, 2016.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. arXiv preprint 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint 1rXiv:1801.01290, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05905. 2018b.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pp. 4573, 2016.
Kun Ho Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Cross domain imita- tion learning. arXiv preprint 1rXiv:1910.00 2019.
10

---

Published as a conference paper at ICLR 2021
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.698)/ 2014.
arXiv preprint
Diederik P Kingma and Max Welling, arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Justin B Kinney and Gurinder Atwal. Equitability, mutual information, and the maximal informa- the National Academy of Sciences, 1110)):334-3359, 2014.
tion coefficient. Proceedings the
Iya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Diseriminator-actorrrriiic Addressing sample inefficiency and reward bias in adversarial imitation learning. 2018.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv 2015.
preprint
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learn ing to imitate behaviors from raw video via context translation. In 2018 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 1118-1125. IEEE, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 180.05780,, 2018
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning.. In Ieml, volume 1, pp. 2, 2000.
Ryo Okumura, Masashi Okada, and Tadahiro Taniguchi. Domain-adversarial and-conditional state space model for imitation learning. arXiv preprint arXiv:2001.11628, 2020.
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbecl, and Sergey Levine. Variational dis- criminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining infor- mation flow. arXiv preprint arXiv:/810.008: 2018.
Dean A Pomerleau. Alvinn: An autonomous land vehicle a neural network. In Advances in neural information processing systems, pp. 305-313, 1989.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neu-
ral computation, 3(1):88-97, 1991.
Nathan D Ratliff, Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In Proceedings of the 23rd international conference Machine learning, pp. 729-736. ACM, 2006.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635, 2011.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-18999 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg
Klimov. URL
Proximal policy optimization algorithms.
CoRR, abs/1707.06347, 2017.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 113--114. IEEE, 2018.
Pratyusha Sharma, Deepak Pathak, and Abhinav Gupta. Third- visual imitation learning via decoupled hierarchical controller. In Advances in Neural Information Processing Systems, pp. 2593-2603, 2019.
11

---

Published as a conference paper at ICLR 2021
Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, and Sergey Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprint arXiv:1912.0444; 2019.
Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. arXiv preprint arXiv:1703.01703, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-500 50333. IEEE, 2012.
Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from obser- vation. arXiv preprint arXiv:/905. 13566, 2019.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double qr learning, Thirtieth AAAI conference on artificial intelligence, 2016.
Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforce- ment learning. arXiv preprini urXiv:1507.0455 2015.
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. 2008.
Konrad Zolna, Scott Reed, Alexander Novikov, Sergio Gomez Colmenarej. David Budden, Serkan Cabi, Misha Denil, Nando de Freitas, and Ziyu Wang. Task-relevant adversarial imitation learn- ing, arXiv preprint arXiv:1910.01077 2019.
12

---

Published as a conference paper at ICLR 2021
A
ALTERNATIVE EXPERT DEMONSTRATIONS CONSTRAINT
Table 3: Example visual trajectories collected in the sample task
visual trajectories
{zooo, 21, (2882)
Bg:
{10, 11,
100,11,11
(10,
{10,
Bx:
{00, 00,000,00
(00,01,00)
(00, 00,
00, 00, 00)
Previous work (Stadie et al., 2017) proposed to optimize the GAIL objective described in section 6, subject to constraining the mutual information to be O:
arg max Jo(0, Be U Br) s.1. T(zi de BB U B₂) = 0.
9
In practice, however, the algorithm proposed by Stadie al. (2017) enforces such constraint very loosely via a domain confusion loss. This is achieved by introducing second classifier, Co. on top of the preprocessor Pa.. to predict the domain labels de from the latent representations 2;.
JDC 01, B5, Bx)) B.
log( CCo (2.)) + EB,,,e, log(1 Ca(2)))
The GAIL objective JG is then augmented by the domain confusion loss, JDCL, where the prepro- cessor adversarially trained to minimize the information about the domain labels de useful for C.- This optimization is then regulated by a fixed weight coefficient X:
arg max arg min Jc(0, Bg U BB) - XJD 01, BE, Bx).
As consequence. in practice the domain confusion loss acts domain labels information, contained in the single latent representations, rather than attempting to enforce precise constraint. Below, we provide a toy example where truly enforcing [(z,, d;) would prevent the preprocessor from encoding any useful information about the observations.
heuristic to minimize the
Consider simple task where the objective is for agent to reach and remain in a target state. In this setting, we let the agent and expert POMDPs differ in their observation spaces. Specifically, we define the observations collected to be composed of two binary variables, 0, = rzfft The value of the first variable I; is for any observation in the expert POMDP. and it is 0 for any observation in the agent POMDP. The value of the second variable y/s if the visited state is target state, and it is 0 otherwise. Therefore, for any observation o; the first binary variable z; contains domain information about d, which should be discarded (as xt = d,). while the second binary variable y: contains useful goal-completion information about c, which should be encoded in z,
Consider an instance of this problem with a task horizon of and with four visual trajectories in BE and B, described in Table 3. In this example, unlike the visual trajectories collected by the agent in B., the expert demonstrations in Bg accomplish the goal of this task as they show successfully reaching and remaining in target state. Therefore, the distribution of goal states encountered in the observations from Bg and B2 are different, and consequently, y; is not statistically independent from T,. We show this by computing the conditional probabilities:
p(E; 1 = 1) = 4/5 # p(z, = 1/y; 0) = 2/7 = p(z; 1lys) # p(æ; 1) 1/2.
Thus, the observable goal completion information about c, present in y:s are not statistically inde- pendent from the domain labels d.
13

---

Published as a conference paper at ICLR 2021
For simplicity, consider deterministic preprocessor encoding the latent representations Z, P(o:) (as proposed by Stadie et al. (2017)). Enforcing [ d,) = 0 means that the value z; must be independent of d, and since d; = 24, we must have z; P(o,) However, we claim that this also implies that T(z,, = 0. We can easily show this by contra- diction, assume that T(z; C4) 0, since yi is the only observable source of information about G2 then (y = 0) + Pa Py = 1). Therefore, P3 must be invertible or in other words there exists function such that Py (Va) = y- However, we then have that p(d; = p(d (z:)) = p(drly)) = p( * p(æ,) = p(d)), therefore z, is not independent of de and we must have T(z;, de) 0.
P(zy)V2 Py
B
ALGORITHM DETAILS
B. PRIOR DATA
The prior data sets utilized to enforce the prior data constraint are collected by executing random behavior in both 'source' and 'target' environments. To add diversity the discriminator's learn- ing signal, we also use samples from both prior sets as additional negative examples for Jg. The baselines making use of the domain confusion loss also make use of prior data, analogously to how 'failure' data is used in the original TPIL algorithm (Stadie et al., 2017). On the other hand, the baselines DisentanGAlL with no prior data and No latent representation regularization are evalu- ated without access prior data.
B.2 DISCRIMINATOR
Given observation o;, to sample the latent representation z; N (no, (oi), Ea, (o:)) we flatten the output the preprocessor and split the resulting K Kidiimnsionna vector in two halves. We utilize the first half of the variables to obtain the latent representation's 's mean, while we apply Tanh nonlinearity, exponentiate and scale the second half to obtain the latent representation' 's covariance:
do. (o.) Pn, (0.)).
Eg, (o.) = (ediagezp((anh(()) (0))</.k)) + 2)).
To obtain non-stochastic learning signal for the policy, when calculating the pseudo-reward Ro 0;-1, 9-2, 0:-3) = log( Do 0,-1, 0;-2, 0;-2; log(1 - 0i we set the Gaussian noise to zero, equivalently substituting Do(o;, 0;- 0 02-3) with:
Daaa (ot, 01-1; 0/-, 0)33) Soz ( Te, (0--1), oo, (0--2) og (01-3))).
B.3 TRAINING SPECIFICATIONS
The discriminator loss from Eq. 12 and the MINE estimator objective from Eq. 7 are approximated utilizing batches of transitions of observations b, sampled uniformly from the corresponding sets of visual trajectories B. Specifically, for all optimizations, a fixed size agent set of visual trajectories B, and evict old transitions when reaching full capacity, thus, effectively acting as a replay buffer (similarly to Kostrikov et al. (2018)). Throughout all the ex- periments, we utilize the same 2 hidden-layer fully-connected policy and Q-networks with 256 units and ReLU nonlinearities. We keep other model architectures structurally consistent, with fully- convolutional preprocessor Po,, mully-connected invariant discriminator Se, and mully-connected statistics networks To.. We only vary the depth of the models and the number of filters/units in each layer depending on the environment realm. To avoid having a biased pseudo-reward which could provide a learning signal to the agent even without any meaningful discriminator (Kostrikov al., 2018), modify the environments by removing terminal states. Thus, each collected visual trajec- tory has fixed length, equal to the task horizon [T]. We provide the utilized sitroomennt-speiffe hyper-parameters in Table 4, where we specify the buffer sizes in terms of total/maximum number of observations.
set the batch size bb 128. We utilize
We train each model through the Adam optimizer (Kingma & Ba, 2014) with unique learning rate a = 0.001 and momentum parameters Bn 0.9, B2 = 0.999. We altemnte the collection
of single episode using the current policy nw. with repeating (i) Discriminator learning and (ii)
14

---

Published as a conference paper at ICLR 2021
Table 4: Environment-realm specific hyper-parameters
Realm
Poz
S62
T
Bgl
[Rp.zl
[R,
Inverted Pendulion
x [16 x (3. a))oou,, Yank, (2, M (1 x (3, 3) cone, (2. MaxPoot)
2 [32-FC. ReLL)) (1.EC, Signold)
(32-FC, Xinb] (1-AC)
10000
10000
10000
50
Reacher
2 x {16 (3, 3) comr, Tank, (2. 2) Machwol)
2 {32 FC, ReLU) {1-FC, Signorid)
2 {32 Tinb)
10000
10000
10000
50
{1 (3, -)-omm, (2, 2) MarPPol!
{1-FC)
Hopper
[16 x (3. 3) com. Tank, (2. 2) MaxPool)
2 (100-FC, ReLU) [1-FC, Sigmoid)
2 {128 FC, Janb)
20000
20000
100000
200
(24 3)-oww, Tonk, (2, Pooc)
(1-FC)
-ooco, Tank, M
[48 (3, 3) comv, (2, MaccPoo))
Chefthhhetihh
{16 (3, 3) com,, Zink, (2. 2) MaxPaol)
2 (100-FC, RoLUy [I-FC, Signond)
2 x (126-FC, Xank) [I-FC)
20000
20000
100000
200
24 x (3, 3 3oomm, Tank, () MaxPool!
(22 (3. 3oomm, Tank, [2, Part
comy, (2, 2) MaxPwol
7DDr-PPshoy
Raok, (2, MucPPol)
(100-FC, (1-FC, Signold)
(128-FC, Tonk]
10000
10000
100000
200
322 comn Nank,
(L-FC)
[40 x (3. 3)ooww, Yank, ( Poolll
[64 (3, -omm,
(2, MacPool)
(24 x (3, 3) 3) Yonk, (2, Phool
2 (100-FC, Rell)) {I-FC, Signoid)
2 (126-PC, Timb)
10000
10000
100000
200
(32 comy Tonk, (2, 2) MaxPool
(1-FC)
440 3) Tonk, MaMMolll
(3, voon, (2, PaaMa
Mutual information learning for many iterations the number of time-steps collected. Then, by averaging the mutual information estimates accumulated from performing (ii), we update the coefficients and A. Finally, we also perform (iii) Agent learning for many iterations as the number of time-steps collected. We utilize the same agent set of visual trajectories B, all different learning steps. A formal summary of DisentanGAlL reported below in Algorithm 1.
Algorithm DisentanGAlL
1: Input: expert demonstrations Bg, prior expert and agent observations Bp Bp
2: Initialize B, +- (
3: for i = 1, 1, .... do
4: Sample T (or, dt, s=) with
5: Bμ - B U
6:
for 1,2, |T| do
7:
Sample by, by. br. by E Bg, Br, Br By
v Discriminator learning
8:
Lp -- by. b-) Lg (02, by Ubz) (x(01, b Ubp
9:
Update with Adam
for 1, do
Mutual information learning
11:
Sample bæ, be, bp. bp E BE, Ba, Bp., Bp
In Io_ (z,, d:lbg bb)
1. Len (z,, de bp. U
L1 - 1
15:
Update On with Adam
16:
15at max (4, 12)
Ifestt max (If I2") 7
18:
Update B using |+| Σ =1 1est
Update À using . Σト IfPeat
20: Update Tw, with SAC, sampling from B, for |r| steps
D Agent learning
C ENVIRONMENTS DESCRIPTION
We evaluate the algorithms six different environment realms designed to test the proposed meth- ods for a diverse range task difficulties and domain difference, extending the environments in OpenAl Gym (Brockman al., 2016):
15

---

Published as a conference paper at ICLR 2021
Table 5: Description of the implemented environment realms
Realm
Fnvironmant
Charaderisiios citi ()) dim((O)
Somontic gool
Peoduion
i-Linked Invested Peoduius
link, coloring
82
Keeping
2 Linked Inverted Pendulim
links, sundard coloring
vertically balaaced
1 Linked Colored Inverted Penduiiu
link, altemative coloring
above the moving cur..
2-Linled Colored
Produiin
altemative
Reacher
Linked Reacher
links, stindand camern
2
48
48 Approsching and fixsking torget goal the end effector.
3.1 Liineed Reacher
standand
2-Linked Tikeed Reacher
2 links, camera filted 14.1 degrees
3-Linked Tilted Reacher
3 links, camera illted 14.1 degrees
Hopper
Happer
movable joins, standrdd coloring additional thigh joint, alteraaivve thigh coloring
x 64 Hopping forward without failling
Fleaible Hupper
Holf-Cheeuah
Holf-Chwriih
mowable joints, standard coloring fixed feet joints, aiternative feet coloring
6
64 x 64 x
Masimizing Forward
Inmubilier fest Half-CCwvttl
distance covered.
2D0F-Puber
7D0F-Pushi
standard link sizes, standard coloring. 5-Linktdd haand alernative altennatvve coloring. linkeed hand
48 48
Grabtiing and pasihing item turget goal.
Demonstrator 7U0FF-Pusher
7DOF-Smike
StiFeStriber
staddrd link sizes, staddrdd coloring altemativve link altemotive coluring
7
48 48 3 Striking hall into
Demonstrator 7DOF-Striker
target goal.
This environment realm consists of four variations of the original balancing task. The variations explore changing the color of the agent and adding a second link to be balanced on top of the moving cart.
Inverted Pendulum:
Reacher: This environment realm consists of four variations of the original 2-D reaching task. The variations explore augmenting the number of joints and changing the observer's camera recording angle.
Hopper: This environment realm consists of two environments, including the original Hopper environment and an alternative version, in which the agent has additional joint splitting its thigh link two, with the new link also appearing in a different color.
Half- Chiffhhhetth: This environment realm consists two environments, including the original Half- Cheetah environment and alternative version, in which the agent has immobilized feet joints and the connected links appearing in a different color.
Each of these environment realms consists of two environments, including the original Pusher/Striker environments and alternatvee version, in which the agent's model is modified in its appearance and structural configuration, to make resemble very sim- plified human operator.
To collect observations, we render the environments with Mujoco and down-scale the renderings to different dimensions with the purpose of having efficient representations but still preserving the relevant details about the observations. We provide further environment-specife descriptive details in Table 5.
D SUPPLEMENTARY RESULTS
D.1 LATENT REPRESENTATIONS COUPLING
After performing the reported experiments, utilize the learnt models understand what features are encoded into our constrained latent representations 2;- Particularly, we use the output of the trained preprocessors map observations between the 'expert' agent's and the 'observer' agent's sets of visual trajectories. We achieve this by taking four different observations from B, and com- puting their latent representations. Then, we match these observations with the four observations in Bg having the closest latent representations, computed by taking the relative L1-distances. We repeat this process, matching four observations in Bpxx with four observations in Bp.E. We show the produced couplings for six different experiments with DisentanGAIL in Fig. 4. From the sults, it can be inferred that the mutual information constraints successfully guide the preprocessor to encode features which are agnostic the agent's 's embodiment and the environment's appearance, yet preserving information about the goal-completion levels displayed the observations. For the Inverted Pendulum realm, the preprocessor appears to be encoding information about the relative
16

---

Published as a conference paper at ICLR 2021
Freddtive
(Znns Posksse
Hiteee Rosdke
8.
>
Rocke
Bopee
Paedhi Dayer
Hosas-lle 1DOR
Figure 4: Couplings produced by matching the observations collected in the 'observer' agent's environment with the observations collected in the 'expert' agent's environment, to minimize the L1-distance between the latent representations produced by DisentanGAlL. On the right we show the results between the agent set visual trajectories B, and the set expert demonstrations BE, and on the left we show the results between the prior set of agent observations Bp.- and the prior set of expert observations Bp.g- From top to bottom, we show the results the Inverted Pendulum, Reacher, Hopper, Half- Cheetah, 7DOF-Pusher and IDOF-Striked environment realms. We produce the couplings six sample observational imitation problems considering domain differences in terms of both environment appearance and agent embodiment.
17

---

Published as a conference paper at ICLR 2021
Teleed Froddtio
Ac
CSnnin wtw.k//w/w///////////////a Postiwe
Tieeel Irannl ResMaa
R:
2-toeed Iranal PreAtiv
24lolel Scanel PesMiss
Figure 5: Couplings produced by matching the observations collected in the 'observer' agent's environment with the observations collected in the 'expert" agent's environment, to minimize the Li-distance between the latent representations produced by DisentanGAlL with domain confusion loss. This visualization is analogous to Fig. 4
18

---

Published as a conference paper at ICLR 2021
0.0
Avenge estmattedddeeee 12 Feue
Figure 6: Average mutual information between the latent representations Ttrue throughout performing observational imitation, for differents value of the expert demonstra- tion hyper-parameter maaz- We evaluate this measure for the experiments in the Reacher and verted Pendulum environment realms considering domain differences in both appearance and em- bodiment.
the default rewards
angle between the tip of the links and the moving cart, together with some information about the cart's position. For the Reacher realm, the preprocessor appears to be encoding information about the position of the tip of the reacher, ignoring the orientation of the individual links. For the Hop- per and Celf-Cheetah realms, the preprocessor appears to be encoding the orientation of the joints movable both domains, together with the current height and position the agent with respect to the floor tiles, allowing the discriminator to assess whether the agent is hopping/advancing. For the 7DOF-Pusher and 7DOF-Striker realms, the preprocessor appears to be encoding the location of the item/ball, together with the location and orientation of the agent's hand actuator.
We also show the produced couplings for six different experiments with DisentanGAIL. with domain confusion loss in Fig. 5. From the results, can be inferred that the features encoded by the domain confusion loss are similar to the ones encoded by the mutual information constraints, yet not as consistently interpretable. This is especially evident in the more challenging high dimensional envi- ronments. Particularly, for the Hopper and Half-Cheetah Cheetah realms, the preprocessor does not appear be always encoding the agent's relative position to the floor tiles, but rather focusing either on the angle of particular joints or the agent's 's overall appearance. Additionally, for the 7DOF-Pusher and PDDOF-Strike realms, the preprocessor does not appear to be consistently encoding the location of the item/ball, but rather focusing some less interpretable feature about the agent's appearance.
D.2 EXPERT DEMONSTRATION CONSTRAINT
We evaluate the effects of enforcing tighter expert demonstration constraints DisentanGAlL. in the low-dimensional environments. This is achieved by running our algorithm with lower values for the hyper-parameter regulating the upper-limit on the estimated mutual information, Imazs in the adaptive penalty loss Lg (described in Section 5.1).
First, analyze the effects that tighter constraints have the amount of goal-completion in- formation encoded in our latent representations
Particularly, for different values of I Iaaz- we utilize MINE estimate the mutual information between å and default environment rewards Ttrue: [(2, Ttrue). We argue that this measure is a good heuristic about the goal-completion information contained in 2 since Ttrue can be effectively used recover policy to solve the task all environ- ments. Specifically, in Fig. 6, we show the average and standard deviation of the mutual information collected throughout different experiments considering domain differences both appearance and embodiment. This data shows that there is positive correlation between Imaz and I [(2, Ttrue), indi- cating that tighter mutual information constraints are detrimental. This is explained since, especially at the beginning of training, e, and d, are highly dependent. Hence. looser constraint allow greater amounts information about c; to encoded within These findings are also consistent with our arguments from Appendix A.
19

---

Published as a conference paper at ICLR 2021
Argn
Ma Calond Poskior
Sanny
Nage
3 Lasha
H.lba Booierr
Noccer
Mdlcea Rracow
D001
Figure 7: Performance curves from enforcing tighter expert demonstration constraints in the Inverted Pendulum (Top) and Reacher (Bottom) environment realms.
20

---

Published as a conference paper at ICLR 2021
Table 6: Results summary for DisentanGAll. with tighter expert demonstration constraints
Diiferences
squat
expent
differences Inverted Pendavant
Embeddiment
Appecrance
Embodiimet
appearince
Algorithmss cvalnated:
Reocher
Inveice Fendiiss
Reucher
Inveried Pennieton
Reacher
Jeeried
Di -l..
0.004
0.023
0.04%
0.904
0.081
0.885
0:251
0.520 0.051
0.11
0.111
DisemtaGGAl Joar 0.75 0.583
0.057
1.019 1 0.021
0.841 0.113
0,892
131
0.903 1 060
0.857
0,180
0.897
100056
0.772
0.200
DivemanGAlL- Jaas -..00..86 9.87
0.045
1013 0.0.6
0.885
104
0.855
208
0.861 1. 1.0.077
0.962
1 0.131
0.837
0.007
0.700
0.251
DismturGAll--an 00225 0.975
1.025
1.020
= 0,025
0.227
+: 0.002
0,882
0.178
0,861
4: 0.0888
0.300
4 0.155
0.848
= 11:85
0.721
11.251
DovmunGAM- -0.1 0.921
0.091
0.582 + 0.041
0.905 + 0.067
0.790 + 0.901
0.652 + 0.1888
0.580 + 0.234
0.576 + 0.144
0.500 00.46
Table 7: Results summary for DisentanGAIL with missing domain information disguising preven- tion techniques
Dilferences between the ageni and the expert domains
No differences
Embodiment
Appesrance
Embodiment and appearance
Algurithms evalnated:
Reacher
Inverted Pendulum
Reacher
Inverted Pondulus
Reacher
Inserted Fondulum
Reacher
Inverted Pendulom
DisentanGAAL
0.973 0.174
0.941
044
0.564 0.081
0,885
0.066
(1880 ausi
0.115
DisemtuGAALL (No 0.957 0.084
1011 1
0.844
0.105
0.853
0,092
0.861
0.864
DismunGGAlL (No 250 0.982 0.676
1,020 +
0.907
0.109
0.900
0.156
0.857
0.127
0.728 + 1991
0.838 0,014
0.789
DisemtanAAl. (No Prevj 0.049 J.1
1,002 1: 0.022
0.866 0.056
060
0.088
0.768
0:115 0.778 1: 0.235
0.763 1: 0.100
0.750 1: 0.225
Second, we analyze directly the effects that tighter constraints have the performance of Disen- tanGAIL. The performance curves for different values of maz are shown in Fig. 7 and a summary of the results given in Table 6. Overall, DisentanGAIL appears to be quite robust to all settings tested, excluding the extreme mas = 0.01. In general, however, a lower mutual information upper- limit appears to have a negative effect on the performance in most experiments, especially when the "expert' agent's 's embodiment and the 'observer' agent's is embodiment differ. This is likely because a tighter constraint does not permit the discriminator to utilize enough information about single ob- servations, thus, providing less informative learning signal to finetune the agent's 's behavior. The effects of varying Iaa appear to be less accentuated in the experiments performed in the Reacher realm. This is likely because the exploratory policy in this environment covers a greater range of states than in the Inverted Pendulum realm, with a more diverse range of goal-completion levels. Thus, encoding features carrying goal-completion information necessitates to carry less information about the domain labels.
D.3 DOMAIN INFORMATION DISGUISING
We also perform an ablation study to understand the effects of the techniques proposed to counteract domain information disguising. described in Section 5.2. We compare the proposed DisentanGAlL algorithm with alternative versions: (i) with no spectral normalization (No SN) (ïi) with no dou- ble statistics network (No 2St) (iii) with domain information disguising prevention (No Prev- making use of neither spectral normalization or double statistics network). The performance curves are shown in Fig. and a summary of the results given in Table 7. Overall, both techniques contribute positively to the final performance. Particularly, the double statistics network appears to have a slightly greater positive effect. This is especially evident in the Inverted Pendulum realm. Additionally, removing spectral normalization from the invariant discriminator's 's layers makes the agent initially learn slightly faster, indicating that there might be a trade-off between convergence speed and training stability.
D.4 ROBUSTNESS TO BACKGROUND DIFFERENCES
We examine whether DisentanGAlL's performance is affected by larger visual domain differences, solely concerning the background appearance. Particularly, most of the tested domain differences in our environment realms involved changing the appearance and morphology of the agents themselves. Hence, we test DisentanGAll. on two alternative target environments Pusher environment realms, with very distinct backgrounds from the relative source environments. Particularly, the target environment of the Hopper realm has a much darker floor, where the tiles are difficult to discern. Additionally, the target environment in the 7DOF-Pusher realm includes a green table and a white floor, both of which differ greatly in appearance to the grey table and black floor of the source environment. We compare the performance of DisentanGAlL performing imitation in these two alternative target environments with the performance in the original target environments to evaluate its robustness.
the Hopper and 7D0F-
21

---

Published as a conference paper at ICLR 2021
Aape
Seuror envoane
Argni arivesa
Jodeed Necder
$4.0e footer
2dMRN Poude
Anader
Arante
Sww. N. spediall sorma Dation
DesertaCAA W8th Socal cuiidiis sebecc
Diseban whti danai Infumatti clagu (TANNESO
Figure 8: Performance curves for the domain information disguising ablation performed in the In- verted Pendulum (Top) and Reacher (Bottom) environment realms.
Segn
DisontanGALL
sontar GAIL original target envirommnt
Figure 9; Performance curves for the Hopper (Left) and 7DOF-Pusher (Right) environment realms the alternative target environments.
22

---

Published as a conference paper at ICLR 2021
Table 8: Results summary for the experiments considering further background domain differences in the 'target' environments
Environment realms
Algorilhm: evalusted:
Hopper
7DOF-Pacher
DismtanGALL.
DismimGAAM (original lorget)
0.901 0.044
We present the performance curves Fig. and a summary of the results Table The ob- tained results show that background domain differences have limited effect DisentanGAlL's final performance. However, they have more prominent effect on DisentanGAlL's efficiency, mak- ing it converge in increased number of epochs. This is particularly noticeable in the Hopper realm's 's results. We hypothesize this is because the new target environments there is greater amount of domain information that requires be 'disentangled" from the useful goal-completion information. For example, in the locomotion realms, the pré-processor encodes goal-completion information about the relative position of the agent with respect to the floor tiles the two envi- ronments (as empirically suggested in Section D.I). In the new target environment of the Hopper realm, this information needs be also disentangled from domain information regarding the tiles' appearance.
23


Reviews:
[{"review_id": "xYa7oPKsaKj", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThe paper proposes a novel approach for third-person visual imitation learning from observations, ie for imitating a different agent in a potentially different environment purely from visual demonstrations. The main difference over prior work (eg Stadie et al, 2017) that used domain confusion objectives is the introduction of new regularization objectives on the learned representation of the visual scene. Empirically, the paper shows that the proposed regularizations can improve performance across a wide range of third-person visual imitation tasks, transferring between simulated agents with different appearances and/or morphologies.\n\n\n## Strengths\n- the paper is very clearly written and easy to follow, necessary background is adequately covered, all components of the model are explained in detail\n- all novel design choices are ablated in the experimental section\n- experiments are conducted across multiple environments and source/target variations, both appearance and morphology -- empirical performance improvements are demonstrated\n- experiments include higher-dimensional, robotic manipulation environments, more complex than those tested in prior work\n\n\n## Weaknesses\n- **not fully fair comparison to baselines**: since the main novelty lies in the introduction of novel regularization objectives, the \"DisentanGAIL w/ domain confusion loss\" is the main comparison method since the only difference to the proposed method is the representation regularization function. However, this baseline does not have access to the additional collected prior datasets used for the introduced \"prior regularization\" loss. Looking at Figure 2, it seems that the performance of DisentanGAIL w/o Prior Data and DisentanGAIL w/ domain confusion loss are near-equivalent which suggests that the access to the additional prior data might be the main factor that contributes to DisentanGAIL's superior performance --> an additional baseline \"DisentanGAIL w/ domain confusion loss & prior data\" is needed, which additionally trains the domain confusion objective on the prior data collected for the DisentanGAIL prior regularization objective, to allow for fair comparison of both regularization approaches with access to the same data\n- **missing baseline results on harder tasks**: on the harder tasks shown in Fig 3 there is no evaluation of the baseline methods, which makes it hard to judge how hard these tasks actually are for prior third-person visual imitation approaches\n\n\n## Questions\n- from my understanding the issue of prior work that constrains the domain-related MI to 0 (described in Appendix, section A) appears for *any* two domains (since, particularly in the beginning of training, there will always be differences in the goal-reaching distributions of expert and policy data). However, it seems that DisentanGAIL w/ domain confusion loss, which applies the MI=0 loss from Stadie et al. 2017, works well on many of the tested domains. How does that fit to the explanation made in Section A + the quant results in Fig 5?\n- what differences result in the substantial performance difference between TPIL and DisentanGAIL w/ domain confusion loss? \n- why does the paper report the \"max cumulative reward so far\" not the expected cumulative reward of the current episode? the latter could give a better idea of the training stability of the different algorithms\n- out of curiosity: the shown scenarios are visually still pretty similar between source and target (eg background etc) --> how do you think an approach like the proposed one would scale to visually substantially different environments, eg from one kitchen to another?\n\n## Suggestions to improve the paper\n- add an additional baseline \"DisentanGAIL w/ domain confusion loss & prior data\", as discussed in the \"weaknesses\" section, particularly for the transfer tasks on the bottom right of Fig 2 in which the discrepancy between DisentanGAIL and the baselines is the largest\n- add evaluation of baselines (particularly DisentanGAIL w/ domain confusion loss (w/ and w/o prior data)) to the harder manipulation environments in Fig 3 to show the benefits of the introduced regularizations\n- since the proposed method addresses a concrete problem of prior work (as explained in appendix Section A) with a clear intuition, it could be nice to add a toy experiment early in the paper that demonstrates this effect empirically for an easy-to-analyze imitation problem, showing that for MI=0 the agent cannot properly learn to imitate since it is unable to capture the relevant information --> such an experiment could help to further motivate the need for the new regularizations\n- the additional assumption of a \"prior dataset\" collected in both domains for additionally constraining the latent representation is first mentioned in section 4.2 --> since this is a different assumption from prior work on cross-domain imitation it would be good to mention this earlier, maybe in a dedicated \"Problem Statement\" section\n- for qualitative matching results in Fig 4 in the appendix it would be nice to show the corresponding matches found when using the domain confusion loss instead of the proposed regularizations to see whether some of the failure cases are interpretable\n- I wonder whether it would be possible to show imitation across agents with more drastic morphology differences in the most challenging 7DOF robotic manipulation tasks. Right now, while there are differences in the link dimensions, the main difference still seems to be in the visual appearance (at least from looking at the provided pictures) --> maybe one could try to eg try to transfer between robots with different gripper morphologies (eg one agent has U-shaped gripper vs another agent with T-shaped \"gripper\" for the pusher task), or from a 4DOF arm to a 7DOF arm etc.\n- as mentioned in one question above: while the agent appearance and morphology changes between the experiments, the largest part of the observation, the background, is usually constant across source and target environment --> it would be interesting to see experiments with different background appearances to see the robustness of the method\n\n## Overall Recommendation\nThe paper is well written and the experiments are covering a wide array of third-person visual imitation problems on which the proposed method shows strong results. The experimental evaluation seems thorough, all design choices are ablated. My main doubts are about the fairness of comparison to the baselines (particularly with regard to the additional data available to the proposed method), and some lacking baseline results on the harder tasks. Therefore I cannot fully support acceptance yet, but if the authors are able to provide the additional evaluations and answer the questions posed above adequately, I am willing to increase my score and vote for acceptance.\n\n\n## Post-Rebuttal Comments\nI thank the authors for their detailed feedback. Particularly the clarifications about the usage of prior data in the baselines were very helpful and the added results with background differences are interesting!\n\nI am not sure whether it is standard to show the learning curves with the \"max achieved return so far\" in the GAIL literature, but if not I still think the true reward per episode should be shown to properly reflect the training stability of the algorithm. Potentially, the stability could also be increased by adding a learning rate decay schedule?\n\nOverall, the rebuttal addressed my questions and incorporated many of the suggestions, therefore I am increasing my score and vote for acceptance.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZmGA5_PnZX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n--\nThis paper proposes a method for performing observational imitation learning -- an existing task that seeks to enable an agent to learn from visual observations of expert behavior in order to roughly imitate the expert behavior. The method employs an adversarial imitation learning objective function that incorporates proposed mutual information constraints that are intended to force the representation space to be invariant to the domain of the data sources, and instead only encode goal-completion information.\n\nExperiments illustrate that the method was able to learn and be performant despite visual and embodiment differences in the expert and agent domain on various mujoco environments, exhibiting substantially better performance to a different observational IL method. Experiments also illustrate good performance on a slightly higher dimensional task.\n\nClarity and Correctness\n--\nThe paper has a few somewhat painful clarity issues that make it difficult to understand.\n\n- Section 4 starts with a subsection that describes components of the architecture, when instead it should start with a clear high-level description of the approach.\n\n- S4.2 The relationship between B_P.E and B_P.\\pi and B_E and B_\\pi is unclear. The former two terms were never defined in the preliminaries, so it's not clear whether they constitute additional data that was not mentioned in the assumptions. This ties into my comment about about Section 4 needing an introductory paragraph to lay out the high-level assumptions, inputs and outputs, and idea of the method. The notion of \"prior data\" is used in the paper, but I cannot find a clear description of it anywhere.\n\nOriginality\n--\nAs mentioned in the paper, this paper is closely related to Stadie et al. (2017). There's description of the differences to it in various places throughout the paper, but it would be nice if there were a clear section on the comparison of objective functions.\n\nSignificance\n--\nThe significance of this paper is that it demonstrates a more performant method for performing observational imitation learning, which is potentially more applicable than standard IL approaches that require state or state-action traces.\n\n- Why was TPIL not used as a method of comparison in Fig 3?\n\n- The results shown in Fig 2 have rather short x-axes, with a maximum of 20,000 steps of training. The comparisons would be more informative if training were run for more steps (e.g. 1e6 or 1e7). It is possible that the other approaches end up matching expert performance as well, without any great loss in efficiency.\n\nOther comments\n--\n- S4.2 prior data constraint -- the inequality is backwards (MI is nonnegative)\n- S3.2, trajectories are sequences, use () not {}\n- The logical and notation in S4.2 is very confusing -- the statements inside the parentheses that define d_i aren't boolean truth values, so it makes the definition of d_i confusion. My suggestion is just to use d_i = 1(o \\in B_e), since B_e and B_\\pi are disjoint.\n- Be precise / define what you mean by \"high dimensional\". \"High\" is subjective, and is perhaps not the most appropriate adjective to describe 7 dimensional tasks. Also, make it clear that the high dimensionality is in the action space.\n\nPost-rebuttal comments\n--\nAfter reading the authors' response and the updated components of the manuscript, I thank the authors for addressing nearly all of my concerns. The inclusion of a clearer motivation, more discussion w.r.t. TPIL, and comparison to TPIL, all enhance my understanding of the contributions of the paper beyond my original review enough for me to increase my score from a 6 to a 7.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "K433eqZVgM", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a visual imitation learning algorithm that can handle domain shifts between the expert demonstrations and the data generated by the agent. The proposed method is built upon GAIL and scales to image inputs. The authors handle the domain shift problem by learning a domain-invariant discriminator and a statistics network. The invariant discriminator takes in a concatenation of a sequence of latent representations of the observations and tries to predict without relying on the domain information. The statistics network is used for estimating the mutual information constraint between the latent representation and the domain labels, which the authors are trying to minimize in order to attain domain invariant predictions. The authors optimize the standard GAIL objectives along with the mutual information constraints as regularizations jointly. The experiments are done in both discrete and continuous control tasks in Mujoco and the proposed approach, DisentanGAIL, outperforms the prior method TPIL.\\\n\nFor pros, I think the experimental setup is reasonable and the authors conduct relatively thorough ablation studies to show the usefulness of various components such as the prior data constraint, double statistics network, and spectral normalization regularization, which is helpful for understanding the method. The paper is also well written and easy to understand.\n\nHowever, I have a few concerns about this approach, which I will list as follows.\n\n1. Regarding the novelty of this paper, I feel like the approach is not particularly novel. The method is similar to TPIL (Stadie et al. 2017) and the main difference between DisentanGAIL and TPIL is that in DisentanGAIL, the authors set the mutual information constraint to be less than 1 bit, and in TPIL, the mutual information is constrained to 0. I can definitely see why the softer regularization works better, but I wonder if the contribution of this work is substantial given such a simple tweak. Moreover, the authors also propose many techniques to make DisentanGAIL work better, such as using unsupervised data with some regularization, double statistics networks, spectral norm, and etc.. However, these seem to be more like implementation tricks rather than major contributions and I'm unsure if they make the contribution substantial enough for the standard of an ICLR paper. Moreover, adding these additional components definitely seem to make the approach much more complex and might require much more tuning to get it to work, which is another concern.\n\n2. Another point related to novelty and related works, there are several papers that also consider domain-adaptive imitation learning that seem pretty similar to this paper, such as [1, 2, 3]. [1, 2] consider learning domain-invariant features, while [3] proposes a unifying theoretical framework for domain-adaptive imitation learning. I think these methods could serve as comparisons to DisentanGAIL.\n\n3. Furthermore, since the approach seems to be a bit incremental, it would be nice to have the theoretical analysis that could justify the incremental changes and guarantee the convergence to the optimal policy (e.g. attaining similar results in [3]). Unfortunately, this is missing in the paper.\n\n4. The challenging, high-dimensional environments used in the paper are all in locomotion tasks in Mujoco. It would be nice to see more realistic environments such as robotic manipulation tasks like ROBEL and  Adroit etc., which would make the domain adaptation more appealing. \n\nOverall, based on the arguments above, I would recommend a reject for this paper.\n\n------------------------------------------------------------------------------------------------------------------\nPost-rebuttal updates:\n\nAfter reading the author response and other reviews, I agree that the difference between the paper and prior works is now much more clear and the empirical evidence shows that the new method works well, though I'm still concerned about the part where the authors add many components and make the algorithm much more complex and potentially hard to work in practice. Nevertheless, I've increased my score to a 6.\n\n[1] Okumura, Ryo, Masashi Okada, and Tadahiro Taniguchi. \"Domain-Adversarial and-Conditional State Space Model for Imitation Learning.\" arXiv preprint arXiv:2001.11628 (2020).\n[2] Lu, Yiren, and Jonathan Tompson. \"ADAIL: Adaptive Adversarial Imitation Learning.\" arXiv preprint arXiv:2008.12647 (2020).\n[3] Kim, Kuno, et al. \"Domain adaptive imitation learning.\" arXiv preprint arXiv:1910.00105 (2019).", "extra_scores": {}, "extra_text": {}}, {"review_id": "9lA4fOmgn9P", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies observational imitation learning, a problem setting in which the agent wishes to learn from expert observations, but the state, action, and observational spaces of the expert and agent domains can vary. They introduce DisentanGAIL to tackle this problem by introducing two mutual information constraints in the GAIL framework to learn domain-invariant representations of observations. In particular, DisentanGAIL aims to discard domain information in the presentation, while retaining relevant goal completion information. \n\nOverall, the problem setting is very practical. The performance gain is greatest in the domains where the camera angle changes, which is promising. I like the intuition provided in Appendix A and think some of it should be included in the main text. It motivates well the distinction between domain information and goal-completion, and highlights why previous methods like domain confusion loss can fail. Moreover, I would’ve liked to see some analysis of the experiments to support this hypothesis. For example, a version of Appendix D1 would be valuable to include, and perhaps also another visualization demonstrating that other baselines fail to encode goal completion information. Some of the implementation details in Section 5 can be shortened/moved to the appendix instead.\n\nSome questions:\n\n- What is the difference between DisentanGAIL with DCL and TPIL? The text says that the DCL substitutes the mutual information constraints, so the only difference I see would be learning a distribution over latent representations rather than a deterministic feature extractor. Any intuition why this alone leads to such a big difference in Table 1?\n\n- Without any regularization, I’m surprised DisentanGAIL learns anything at all. Wouldn’t the discriminator then be able to tell the two domains apart (especially if one of the differences is color) and fail to provide a meaningful learning signal to the agent?\n\n- If I understand Figure 3 correctly, the orange line is without any regularization (lower bound) and the green is learning without domain differences (upper bound). How come DisentanGAIL can eventually outperform the green line in some of the experiments?\n\n- What’s the gap between DisentanGAIL and the expert policy? Does the expert always achieve a reward of 1?\n\nOther comments\n\n- Please clean up all Figures and Tables. The text is too small and impossible to read without zooming in very close. \n\n- “Learning” typo in last line of page 2.\n\n----------\n\nPost-rebuttal comments\n\nThank you for answering my all questions and updating the manuscript with some of my suggestions. The additions in section 4 clarify the motivation much better and also highlight the differences with prior work. I've increased my score from 6 to 7.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers raised a number of concerns about the novelty of the paper and comparisons. The authors were able to address the concerns regarding the comparisons in the response, and the reviewers unanimously agree that the paper should be published. I do think however that this paper is quite borderline. I agree with the reviewers that the updated experiments are convincing in terms of the provided comparisons. However, the reservations I have about the work can perhaps best be stated as follows: There is quite a bit of work in the area of imitation from observations, which makes a range of different assumptions and utilizes a variety of different domain adaptation techniques. Much of this work is in the robotics domain (which is cited in the paper), and much of it demonstrates results in fairly realistic settings, often with real humans and real robots. In comparison, the experiments in this paper are quite simplistic, using toy domains and "demonstrations" obtained from a computational oracle (i.e., another policy). Given the maturity of this field and the current state of the art, I am skeptical of this evaluation, and I think TPIL is a very weak baseline. That said, I would  defer to the reviewers in this case -- I do think the particular technical contributions that the paper makes are a valuable addition to the literature, though somewhat incremental. I am also sympathetic to the authors in that much of the more successful prior work in this area that does evaluate under realistic conditions makes subtly different assumptions, or utilizes different techniques for which it is difficult to provide an apples-to-apples comparison.

One thing I would request of the authors for the camera ready though is: Please tone down the claims. "Human-like 7 DOF Striker" is not human-like, it's a crudely simulated robotic arm that was recolored. It would of course be better to have a realistic evaluation (as many prior papers in this field indeed have), but in the absence of that, it is best not to overclaim and be upfront that the evaluation is on relatively simple simulated tasks under conditions that are not necessarily realistic (and have nothing to do with actual humans), but meant rather to evaluate in an apples-to-apples manner the particular algorithmic innovations in the method.
Decision: Accept (Poster)