Seeing Differently, Acting Similarly: Heterogeneously Observable Imitation Learning
Authors: Xin-Qiang Cai
Yao-Xiang Ding
Zixuan Chen
Yuan Jiang
Masashi Sugiyama
Zhi-Hua Zhou
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Imitation Learning
Heterogeneous Observation Space
Importance Weighting
Learning with Rejection

Abstract: In many real-world imitation learning tasks, the demonstrator and the learner have to act under totally different observation spaces. This situation brings significant obstacles to existing imitation learning approaches, since most of them learn policies under homogeneous observation spaces. On the other hand, previous studies under different observation spaces have strong assumptions that these two observation spaces coexist during the entire learning process. However, in reality, the observation coexistence will be limited due to the high cost of acquiring expert observations. In this work, we study this challenging problem with limited observation coexistence under heterogeneous observations: Heterogeneously Observable Imitation Learning (HOIL). We identify two underlying issues in HOIL, i.e., the dynamics mismatch and the support mismatch, and further propose the Importance Weighting with REjection (IWRE) algorithm based on importance-weighting and learning with rejection to solve HOIL problems. Experimental results show that IWRE can successfully solve various HOIL tasks, including the challenging tasks of transforming the vision-based demonstrations to random access memory (RAM)-based policies in the Atari domain, even with limited visual observations.

Seeing Differently, Acting Similarly: Heterogeneously Observable Imitation Learning
Anonymous Author(s)
Affiliation Address email
Abstract
In many real-world imitation learning tasks, the demonstrator and the learner have to act under totally different observation spaces. This situation brings significant obstacles to existing imitation learning approaches, since most of them learn poli- cies under homogeneous observation spaces. On the other hand, previous studies under different observation spaces have strong assumptions that these two obser- vation spaces coexist during the entire learning process. However, in reality, the observation coexistence will be limited due to the high cost of acquiring expert observations. In this work, we study this challenging problem with limited observa- tion coexistence under heterogeneous observations: Heterogeneously Observable Imitation Learning (HOIL). We identify two underlying issues in HOIL, i.c., the dynamics mismatch and the support mismatch, and further propose the Impor- tance Weighting with REjection (IWRE) algorithm based on importance- -eeetance-wegghtnng and learning with rejection solve HOIL problems. Experimental results show that IWRE can successfully solve various HOIL tasks, including the challenging tasks of transforming the vision-based demonstrations random access memory bbse) policies in the Atari domain, even with limited visual observations.
2
6
8
10
12
14
15
16
Introduction
18.
Imitation Learning (IL) studies how to learn a good policy by imitating the given expert demonstra- tions [6 1. and has achieved great success many domains such as autonomous driving [8) video 19
20
games 77), and continuous control [19]. In real-world IL. applications, the expert and the learner usually have their own observations the same underlying states from the environment.
21
example,
in [ D autonomous agent is learning to drive by imitating human expert. The expert takes her actions mainly based on auditory and visual observations, which are familiar to human beings. However, the learning agent does not necessarily use the same way to observe: it can utilize more maphine-capable sensors such as a LiDAR, radar, and bird-eye view (BEV) map to generate its observations (20). The key features behind this example are two-fold: First, both the expert and the learner have their totally different observations of the same state of the environment. Thus they essentially have to choose the same action if acting optimally. Second, the observation space of the expert often of high cost for the learner to utilize [6] [0]. We call this problem Heterogeneously Observable Imitation Learning (HOIL).
23
24
25
27
29
31 There are two lines of research studying the related problems. The first line relates to domain adaptation: the observation space of the expert and the learner are the homogeneous, while some 33 typical mismatches distributions could exist: morphological mismatch, viewpoint mismatch, and dynamics mismatch [30] I7 266. However, these approaches are invalid when the observation spaces
35 for experts and learners are completely different as in HOIL.
36 The second line studied II. under different observations similar to HOIL, and some representative
37 works include Partially Observable Imitation Learning (POIL) [4 36 and Learning by Cheating
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

Rurva Expert
Agen
Figure I: Autonomous driving: an example of the HOIL problem. Figures 1, and include some illustrations and pictures from the Internet (source: www. vect com).
Spnce
Oberralie
Prescn
F
J
Frace
f
Proce
(a) POIL
(b) LBC
(c) HOIL (our work)
Figure 2: Comparisons different IL. processes under different observation spaces. The targets are all to learn 22 based on the second observation space with an auxiliary policy कı from corresponding roll-out data T and T. (a) POIL mainly emphasized that the expert can view full observations, while the observations for the learner are partial.
(b) LBC assumed that the expert's observations contain more privileged information than the learner's. s. Both POIL and LBC can observe expert's observations all along. (c) HOIL limits the amount of expert's observations.
38 (LBC) 8 as depicted in Figure 2
Both POIL and LBC assume that the expert's observations can be easily accessed by the learner without any budget limit. However in practice, differrnn from the 40 learner observations, the access to expert's observations might be of high cost and invasive 10] hindering the wide application of these methods.
42 In this paper, we initialize the study of the HOIL problem. We propose a learning process across observation spaces of experts and learners for solving this problem, and analyze the underlying issues 44 of HOII., i.e., the dynamics mismatch and the support mismatch. To tackle both two issues, we resort to the techniques of aepportance-eigghting [12 and learning with rejection [5] for active querying
43
46 to propose the Importance Weighting with REjection (IWRE) approach. We evaluate the effectiveness of the IWRE algorithm in continuous control tasks of MuJoCo [33]. and the challenging tasks of learning random access memory (RAM)-based policies given vision-based expert demonstrations in Atari B games. The results demonstrate that IWRE can significantly outperform existing IL algorithms in HOIL tasks, with limited access to expert observations.
48
50
Related Work
51
52
Domain-Shifted H., For the standard II. process, where the learner and the expert share the same
53 observation space, current state-of-the-arr methods tend to learn the policy in an adversarial style (7),
54
like GAIL [16]. When considering the domain mismatch problem, i.c., Domain-Shifter IL (DSIL),
55 the research aims addressing the static distributional shift of the optimal policies resulted from
56 the environmental differences but still under homogeneous observation spaces. Stadie et al. [30]
57
Sermanet et 22), and Liu et (23] studied the situation where the demonstrations are in view
58 of third person. Kim et al. [19] and Kim et al. [18] addressed the IL problem with morphological
59
mismatch between the expert's and learner's environment. Stadie et al. (30], Tirinzoni et al. [32] and
Desai et al. [1] focused on the calibration for the mismatch between simulators and the real world
61
through some transfer learning styles. There are two major differences between HOII. and DSII.:
One is that HOIL considers heterogeneows observation spaces instead of homogeneous ones; another

---

63
is that without observation heterogeneity, DSIL can directly align two fixed domains, which may not be realistic for solving HOIL when two observation spaces are totally different. Thus HOIL is a significantly more challenging problem than DSII.. Besides, Chen et al. g learned a lision-based agent from a privileged expert. But it can obtain expert's observations throughout the whole learning process, so it cannot handle the problem of the support mismatch under HOII..
64
65
66
67
68
POMDP. The problem of POMDPs, in which only partial observations are available for the agent(s), has been studied in the context of multi-agent [25] [36] and imitation learning [44 36] problems. But distinet from HOII,, in a POMDP, the learner only have partial observations and share a same underlying observation space with the expert, which would become an obstacle for them to make decisions correctly. For example, Warrington et al. [36] assumed that the observation of the learner is partial than that of the expert. Instead, in HOIL, expert's 's and learner's 's observations are totally
69
70
71
72
73
74 different from each other, while the learner's s observations are not belong to a part of the expert's. s. For HOIL, the main challenge is to deal with the mismatches between the observation spaces, especially when the access to expert's observations is strictly limited.
76
3 The HOIL Problem
77
In this section, we first give formal definition of the HOIL setting, and then introduce the learning process for solving the HOIL problem.
79
80
3.1 Setting Definition
81
A HOIL problem is defined within Markov decision process with mutiple observation spaces, i.e., (S, {O}, A. A.P. 7),), where S denotes the state space. {O} denotes a set of observation spaces, AA denotes the action space, P : S x A S + R. denotes the transition probability distribution of the state and action, and 7 E (0, 1] denotes the discount factor. Furthermore. a policy over an observation space O is defined as a function mapping from to A, and we denote by To the set of all policies over O. In HOII., both the expert and the learner have their own observation spaces, which are denoted as Og and Og respectively. Both OE and Or are assumed to be produced by two bijective mappings fE S + OE. ft. S - OL. which are unknown functions mapping the
82
83
84
85
86
87
88
89 underlying true states to the observations. It is obvious to see that by this assumption, any policy over OF has unique correspondence over Or.. This makes HOIL possible since the target of HOIL is to find the corresponding policy of the expert policy under Or..
90
91
92 A state-action pair a). denoted by æ, is called an instance. Also, trajectory T = {x;},i [m]
98 is set of m instances. For each observation space, . E T C Og A and I E T S OL x A, where Og- = (5) and OL = fL) Furthermore, we define the occupancy measure of a policy π under the state space S as Pa S x A + R such that p.r (z) x (r(o)) Ei y'Pr(s:
94
95
r).
96
Under HOIL, the learner accesses the expert demonstrations Tag a set of instances sampled from Prg. The goal of HOIL is to learn a policy म the corresponding policy of me over OL. If Og OL. HOIL degenerates to standard IL. GAIL [16] is one of the state-of thet---rrt IL approaches under this situation, which tries to minimize the divergence between the learner's 's and the expert's 's occupancy measures d(pt, P.x) The objective of GAIL is
97
98
99
100
min max Eu-p-g [log D [+) + Egwps [lo((1 -
where H()) is the causal entropy performed regularization term, and D Og A - (0, 1] is the discriminator of ग. and म. GAIL solved Equation (I) by alternatively taking a gradient ascent step train the discriminator Dw.. and minimization step to learn policy से based off-the-shelf
102
103
104
RL algorithm with the pseudo reward- - log (T).
105
3.2
The Learning Process for Solving HOIL
106
In HOIL, we need to cope with the absence of the learner's observations in demonstrations and the high cost of collecting the expert's observations while learning. So we introduce a learning process with pretraining across two different observation spaces for solving HOIL, as abstracted in Figure 3
107
108
109
Pretraining. Same to LBC R we assume that we can obtain an auxiliary policy क based on Or, at the beginning. Ii can be directly provided by any sources, or trained by GAIL or behavior cloning
110
111 as did LBC. Besides, we use this ग, to sample some data Tx which contain both observation
under Op (i.e.. Ta,) and Or (i.e., Tx)). ) in order to connect these two different observation spaces.
112
113 We name T., (Tx,, T x1 } the initial data.
3

---

Spoce
Bils 2
Dencesntions
Obacrstoa Sp
Oper
Lemiing
Putrasikg
T
Proces
Treking
T
Choevoties Cvestrevee
Figure 3: Illustration of a learning process across two different observation spaces for solving HOIL. T1 an auxiliary policy that additionally provided.
Training. Here we learn a policy #2 from the initial data T H, and the collected data T x2. under OL only. Besides, the learner is allowed for some operation of observation coexistence (OC): At some steps of learning, besides the observations OL, the learner could also request Tna from the corresponding observations Og (e.g.. from the human- unduansndddtttdabbeeee sensors). The final objective of
114
115
116
117
118
HOIL is to learn a good policy л2 under Or..
119
In practical applications, the auxiliary policy गi can also come from simulation training or direct imitation. But since и is additionally provided, it is more practical to consider и as non-optimal policy. During training, OC is an essential operation for solving HOIL, which helps the learner address the issues of the dynamics mismatch and the support mismatch (especially the latter one). Also, in reality, we do not need an oracle for actions, which still needs OC for obtaining expert observations first, as in many active querying research
120
121
122
123
124
so its cost will be relatively lower.
125
Besides, the related work ( also required an initialized policy Ti to solve their problem, which act teacher under privileged OF: in the pretraining and then learned vision-based student from the guidance of the teacher under both O, and Op. Their setting can be viewed as a variety of HOIL with optimal ni, unlimited Op, and unlimited OC operations, HOIL actually more practical learning framework.
127
128
129
4
Imitation Learning with Importance- Weinhhtnn and Rejection
130
In HOII,, the access frequency Og is strictly limited, it is unrealistic learn T2 in Dataset Aggregation (DAgger) style (27 as in LBC. Therefore, we resort to learning т2 with a learned reward
132
133
function by inverse reinforcement learning | in adversarial learning style [6 [3]
134
In addition, both Og and OL are assumed to share the same latent state space S as introduced in Section 3.1 the following analysis will be based on S, while the algorithm will handle the problem based on Og and Og specifically.
136
4.1
Dynamics Mismatch and Importance- Imporlance-Weightinng Weighting
138
To analyze the learning process, we let Prgg Pri: and PTą be the occupancy measure distributions of the expert demonstrations, the initial data, and the data during training respectively. Since need to consider the sub-optimality of æ1, Pay should be mixture distribution of the expert Pigg and non-expert PUNE i.e., there exists some (0, 1) such that PT: 8pmm + (1 8)PTNK:
139
140
141
142
as depicted in Figure 4a demonstrations. To this end, the original objective of reward function Dwy, for T2 is to optimize
During training, the original objective of क2 is to imitate me through
max Egmpe, [log D Dwv. (z) + Expppg [log(1 -Dw, (T))]
But the expert demonstrations are only available under Op. While during training, we can only utilize the initial data T.,. a, Px, to learn T2 and Dwy Besides, m is sub-optimal, directly imitating T m. could reduce the performance of the optimal T2 to that of कj. So we use the apporraace-weighting to calibrate this dynamics mismatch, i.c..
146
max L(D,mz) Ex-pes, [og Dwn, (x)) Exwpe, logx) D (z)) 362
(4)

---

Pru
Phu
Pr.
Probebiitts
Probeliitt
Protebiing
11
N
+1
The ideal situation
(b) The real situation
(c) The output of (T(D.)).
Figure 4: The comparisons among the distributions of expert demonstrations Prg. initial data Pr1* and non-expert data Pxxe- The red and blue regions denote the expert and non-expert parts of Pr. respectively. H, O, and N denote the latent demonstration, the observed demonstration, and the non-expert data respectively. (a) The ideal situation, where supp(pre) supp(p=;) situation, where H == supp(Prg) supp(pm)) # Ø in Pry. (c) The target output of the combined model IT [g]""
Ø; (b) The real
The output +1, 0, and regions correspond to H, O, and N respectively.
Pey is an importance-weigheee weighting factor So the current issue lies in how to 149 estimate PeR under OF. To achieve this purpose, we need to bridge the expert demonstrations and these two data sets train an adversarial model Dw, in the 151 same way as Dw, in the pretraining:
where axx) & Pag (2) 148
Pri
the initial däta. Therefore, here
max L(Dw,) - Ez-pE, [log Dw, (x) + Bewprg [log(1 - Dw, (T))..
(5)
If we write the training criterion (5) in the form of integral, i.e.,
max L(Dw,)) 1nm log Dwn, + Pag log(1 Dan }]dx,
(6)
then, by setting the derivative of the objective
6 to 0
ac DD
0), obtain the optimum Dwy:
D
PT
Px1 + Pre
154 in which the order of differentiation and integration was changed by the Leibniz rule. Besides, we can sufficiently train Dw. using the initial data Tx, and the expert demonstrations T.n Then D... 196 will good enough to estimate the importance-weighting factor, i.e.,
PTg
1 1
D*
D Wr
(((x)
Pri
D wg
D (7)
157 In this way, we can use Dwy, which can connect demonstrations and initial data, to calibrate the learning process of Dwn. The final optimization objective for Dws is
1 1 Dw,, Dwn
max C(Dw)) Exppen log Dwn ( + Earepo
log[1 Dw (7)).
159
In this way, Dmn can effectively dig out the expert part of Prz and produce efficient rewards for m2-
180
4.2
Support Mismatch
So far the challenges have still been similar to homogeneously observable imitation learning. However, our preliminary experiments demonstrated that merely importance--eighhingg weighting is not enough to fix the problem that occurred by the absence of interactions under Og. So there exist some other issues between the expert demonstrations and the initial data. find out the underlying issues, we plot the t-Distributed Stochastic Neighbor Embedding (t-SN)) K4 visualizations of these two empirical distributions under OF on Hopper and Walker2d, as shown Figure Twenty trajectories were collected for both the expert demonstrations and the initial data. We can observe that there exist some
162
163
164
186
5

---

168
high-density regions of demonstrations in which the initial data do not cover; that is, there exist some regions of the demonstrations that J did not explore. Wang al. [35 found a similar phenomenon in the standard II. setting. On the other hand, the amportance-weighhiee
189
170
cannot calibrate this situation
where Pag
171
Dc.
172
To formulate this problem, here we introduce the Support
173
Set of the occupancy measure:
174
Definition 1 (Support Set). The support set of occu-
Dypr
175
pancy measure p is the subset of the domain containing
178
the elements which are not mapped to zero:
supp(p) = fo E S x Alp(z) + 0}.
(10)
177
Due the sub-optimality of IT supp(pre) supp(pn.) + Ø (see Figure 4b). We call this part the Latent Demonstra- tion, defined as:
(a) Hopper
(b) Walker2d
178
Figure 5: t-SNE visualizations of expert demonstrations and collected data of n under Og.
179
180
Definition 2 (Latent Demonstration). The latent demon-
181
stration is the set of those x E S x A that belong to the relative complement of supp(ps) in supp(pxe):
182
H = {x E S x Alsupp((tm) \ supp(pm))
(11)
183
Also, another part of the demonstration is named the Observed Demonstration, defined as: Definition 3 (Observed Demonstration). The observed demonstration O is the set of those S X A that belong to the complement of И in supp(pag):
185
0 {= S Alsupp(p=e) supp(ps.))
(12)
186
Besides, the data outside of demonstrations should be non-expert data: Definition 4 (No--xxers Data). The non-expert data N is the set of those x S A that out of supp(Prg):
197
N {ar E S x (z) 03- 0}.
(13)
In other words, the sub-optimality of will cause not only the dynamics mismatch, but also the appearance of the latent demonstration H. We call the latter one the problem of Support Mismatch. Intuitively, when π22 > TE, we have H + 22, monotonously. So in order to fix the support mismatch between Pag and Pm.. guiding Te to find out H is the key.
190
192
193
In addition, the support mismatch problem can be viewed as an inverse problem of the Out Of Distribution (OOD) problem that frequently occurred in offline RL setting [21], in which they tried to avoid supp(p=.) ) supp(pm..) instead.
194
195
198
4.3
Imitation Learning with Rejection
We can observe that HUOUN S A. So is desirable to filter out H from O and N. Meanwhile, Dw, and Dw, can only classify U H and N, under OE and Or respectively. Therefore, here we
198
199
design two models g1 Og A + 13 and g2 : Or, x A + {0, 1} (Output O: O and output otherwise), so that given x ~ (corresponding ā ~ T and F ~ T) they can satisfy:
200
H {x S A|D (T) ID (x)))(()) +1},
(14)
201
19 0), O fo S A|D m/ (7)]g) (z) D[D u22
(15)
- = fi E x A|D w. (7) ( I[D 7
= 1},
(16)
203 respectively, where I[:] takes +1 . > 0.5, and -1 otherwise. The target combined model
204 I[D (x:)]9" (Ir) is depicted in Figure4c
205 To this end, both 9i and g2 should be able to cover O meanwhile g2 can be adaptive to continuously
change of P.oa due to the update of n2. Here we learn øi and g2 in rejection form, to reject O from

---

207 OU H. (where I((w,) +1). Concretely, the rejection setting is the same as that in Cortes et al. 19] Also inspired by Geifman al. [5], the optimization objective of the combination D,, and g is
208
[(Dw. i(D....) 9) + max(0, c (0))))))))))))
(17)
209
where denotes the target coverage, and denotes the factor for controlling the relative importance of rejection. Besides, the empirical coverage ((g) is defined as
210
اوg) x)
g(x)),
(18)
where a batch of data X = [ E [ma]. The empirical rejection risk ((D...)) g) is the ratio between the covered risk of the discriminator and the empirical coverage:
211
212
i(Dw: g) A m E (C.D.. g(:)))
(19)
((g)
Meanwhile, both Dwn and 9l can access Prg under OF directly. So given T under OL. once (T(Dwr (I)), (2(3) = +1, we can query the corresponding observations T. of x through OC operation and use (T( (3) 91 (T)) to calibrate the output of d2 and Dwn In this way, 92 and D Pwz can be entangled together and adaptively guide To to find out the latent demonstrations H under OL.
213
214
218
217
4.4 IWRE
Here we combine the importance-weighting weighting and rejection into unified whole, to propose novel 218
algorithm named Importance Weighting with REjection (IWRE). Concretely, in HOIL process:
220
Pretraining. We train discriminator Dwn by Equation (5) and its corresponding rejection model gi by Equation (17) using the initial data and the expert demonstrations.
221
Training. We train a discriminator Dwy by the combination of Equation (9) and Equation (7) as well as its corresponding rejection model 92 by Equation (17), using the initial data, the data collected by n2, and the output of Dw, with Φ1 through OC operation. Also, T2 will be updated with Dw, and
223
225 G2 asymmetrically as GAIL.
The pseudo-code of our algorithm is provided in the supplementary material.
5 Experiment
227
228 In this section, we validate our algorithm in Atari 2600 3
(GPL License) and MuloCo
33
229
(Academic License) environments. The experiments were designed to investigate:
1) Can IWRE achieve significant performance under HOIL tasks?
2) Can IWRE deal with the support mismatch problem?
232
3) During training, is active querying for HOII. indeed necessary?
238
Below we first introduce the experimental setup and then investigate the above questions. More results and experimental details are included in the supplementary material.
235
5.1
Experimental Setup
236
Environments. We choose three pixel-memory based games in Atari and five continuous control objects in MuJoCo on OpenAl platform [5] (MIT License). Details as below:
238
Pixel-memory Atari games. Oz: 84 x 84 x raw pixels; On: 128-byte random access
memories (RAM). Expert: converged DQN-based agents (24]. Atari games contain two totally isolated views: raw pixels and RAM, under the same state. Through these envi- ronments, want to investigate whether the agent can learn effective policy from demonstrations under completely different observation spaces. Moreover, IL with visual observations only is already very difficult 77, while learning RAM-based policy can be even more challenging 3 31 so few II. research reported desirable results on this task.
240
242
244
245
2.
Continuous control MuJoCo objects. Op: half of original observation features; O,.: an- other half of original observation features. Expert: converged DDPG-based agents (2). The
7

---

Borok
Reaant
Egliedex
(a) (Atari)
ChopperCommand
(b) Kangaroo (Atari)
(c) Qbert (Atari)
(d) Hopper (MuloCo)
(c) Humanoid (MuloCo)
(f) Reacher (MuJoCo)
(g) Swimmer (MuloCo)
(h) Walker2d (MuloCo)
Figure The learning curves of each method, where the shaded region indicates the standard deviation.
247
features of MuJoCo contain monotonous information like the direction, position, velocity, etc., of an object. Through these environments, we want to investigate whether the agent can learn from demonstrations with complementary signals under observations with missing information. Meanwhile, we make sure RL algorithms can obtain comparable performances under Og and OL. More details are reported in the supplementary material.
248
219
250
251
Besides, twenty expert trajectories were collected for each environment. Each result contains five trials with different random seeds. All experiments were conducted on server clusters with NVIDIA Tesla V100 GPUs The summary of the environments is gathered in the supplementary material.
253
255
Baselines. Six basic contenders were included in the experiments: Vanilla GAIL [6], GAIL with importance-weighting [22 (IW), third-person II. [30] (TPIL), generative adversarial MDP alignment 19 (GAMA), behavioral cloning 2 (BC), and learning by cheating g (L.BC). For IW, we utilized the discriminator Dw., trained in the pretraining to calculate the importance weight; also the optimization objective for Dw, during training is the same as Equation () TPIL learns the third-person demonstrations by leading the cross-entropy loss into the update of the feature extractor;
257
259
261
GAMA learns a mapping function e in view of adversarial training to align the observation of the target domain into the source domain, and thereby can utilize the policy in the source domain for zero-shot imitation. For fairness, we allowed the interaction between the policy and the environment
263
for GAMA under HOIL; LBC uses Tj learned from privileged states as teacher to train 22 in DAgger (27 style, so here we allowed LBC to access Og during the whole IL process. In Atari, to investigate whether our method could achieve good performance for RAM-based control, we further included a contender PPO-RAM, which uses proximal policy optimization (PPO) (2 to perform RL directly with environmental true rewards under the RAM--assdd observations. More detailed setup including query strategies for TPIL and GAMA, network architecture, and paramm-rrreeterss are reported in the supplementary material.
265
267
269
270
271
Learning process. To simulate the situation that OF is costly, the steps for training mi was set as 1/4 of that for training T2, using GAIL HeeyHad [7] under the Og space for MuJoCo/Atari environments. The learning steps were 10" for MuJoCo and x 10yh for Atari environments. In the pretraining, we sampled 20 trajectories from TT and the data from each trajectory had both Og and Og observations. In the training, each method learned 4 x 10? steps for MuJoCo and 2 x 107 steps for Atari under the OL space obtain π2.
273
275
276
5.2
Results
Experimental results are reported in Figure 6 Since the mapping function is hard to learn when input is RAM and output is raw images, we omit the results of GAMA in Atari. We can observe that while IW is better than GAIL in most environments, both GAIL and IW can hardly outperform mi
279
8

---

281
Because they just imitated the performance of т instead of Tg. even with auportance-weighting for calibration. For TPIL, its learning process was extremely unstable on Hopper, Swimmer, and Walker2d due to the continuous distribution shift. Furthermore, the performance of GAMA was not satisfactory in Hopper and Walker2d because its mapping function is hard to learn well when the support mismatch appears. The results of TPIL and GAMA demonstrate that DSIL methods will be invalid under heterogeneous observations as in HOIL tasks. On Atari environments, OE contains more privileged information than OL so LBC can achieve good performance. But when OE
282
284
285
286
288
is not more privileged than OL, like in most environments of MuJoCo, its performance will decrease due to the support mismatch, which would make it even worse than BC. Finally, IWRE obtained the best performance on 6/8 environments, and comparable performance with LBC on Reacher, which shows the effectiveness of our method even with limited access to OE (LBC can access to
290
292
OF all the time). Besides, we can see that the performance differences hetween the GAIL/IW and
293
IW WWR//PIL/GAAAALBB are huge (especially on Reacher) because of the absence of queries, which demonstrates that the query operation is indeed necessary for HOIL problems.
294
Moreover, even learned with true rewards, PPO-RAM surprisingly failed to achieve comparable performance to IWRE, which shows that IWRE could possibly learn more effective rewards than true environmental rewards in RAM-input tasks. The results verify that, IWRE provides powerful approach for tackling HOIL problems, even under the situation that the demonstrations are gathered from such a different observation space, meanwhile Og is strictly limited during training.
296
298
300
E-SNE visualization of P.., and PrE under Og. In Sec- tion 4.2 we point that the sub-optimality of will cause the problem of support mismatch, which embodied as the appearance of the latent demonstration H during train-
302
303
304
ing. Also the empirical results in Figure Hopper and Walker2d verify the existence of this problem. So we want to investigate whether the superiority of IWRE indeed comes from successfully tackling the support mismatch problem. To this end, we plotted the t-SNE visualization of the same expert demonstrations as in Section 4.2 and the collected data of π2 by IWRE under Og (Og is hidden to T2). All setups are the same as in Section 4.2 From the results shown in Figure 7 we can see that even under OE, which cannot be obtained by π2, almost all high-density regions of the demonstrations were covered by the col- lected data. Meanwhile, the latent demonstration H/ is dug out nearly. The results demonstrate that IWRE basically solves the problem of support mismatch and thereby performs well in these environments.
306
308
(a) Hopper
(b) Walker2d
Figure 7: t-SNE visualizations of expert demonstrations and collected data of m under Og. The high-density regions of the expert demonstrations were covered by the collected data of x2 of IWRE.
310
312
313
314
315
316
317
318 Besides, some collected data of T2 of IWRE were out of the distribution of the demonstrations, which means T2 slightly overly explored the environment. Since OF is hidden to πT, the reward function will encourage T2 to explore more areas to fix the support mismatch problem. Meanwhile, the -nttof-diitrrbution problem in HOIL is not as severe in the offline RL settings (1), so this ever-exploration phenomenon makes sense.
320
322
Conclusion
323
324
In this paper, we proposed a new learning framework named Heterogeneeuusly Observable Imitation Learning (HOIL), to formulate the situations where the observation space of demonstrations is different from that of the imitator while learning. We formally modeled a learning process of HOII,, in which the access the observations of an expert is limited due to the high cost. Furthermore. we analyzed underlying challenges during training, i.c., the dynamics mismatch and the support mismatch, on the occupancy distributions between the demonstrations and the policy. To tackle these challenges, we proposed new algorithm named Importance Weighting with RÉjection (IWRE), using mmpottnce-weiihhting and learning with rejection. Experimental results showed that the direct imitation and domain adaptive methods could not solve this problem, while our approach obtained promising results. the future, we hope to involve the theoretical guarantee for our algorithm
325
326
327
328
329
330
331
332
333
IWRE and investigate how many OF do we need to query to learn a promising æ2. Furthermore,
335
we hope to use the learning framework of HOIL. and IWRE to tackle more learning scenarios with
demonstrations in different spaces.
9

---

References
337
338
[1] Pieter Abbeel and Andrew Y. Ng. Inverse reinforcement learning. In Encyclopedia of Machine Learning, pages 554-558. 2010.
339
340
(2] Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelli- gence 15, pages 103-129, 1996.
342
[3] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res., 17:253-2799 2013.
343
344
[4]
Kianté Brantley, Hal Daumé III, and Amr Sharaf. Active imitation learning with noisy guidance. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July
346
347
5-10, 2020, pages 2093-2105. Association for Computational Linguistics, 2020.
348
[5]
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. CoRR, abs/1606.00 10066.1540 01540, 2016.
349
350
66 Alberto Broggi, Michele Buzzoni, Stefano Debattisti, Paolo Grisleri, Maria Chiara Laghi, Paolo Medici, and Pietro Versari. Extensive tests of autonomous driving technologies. IEEE Trans. Intell. Transp. Syst., 14(3):1403-141, 2013.
352
77] Xin- Cai, Yao-Xiang Ding, Yuan Jiang, and Zhi-Hua Zhou Imitation learning from pixel-level demonstrations by hashreward. In Proceedings of the 20th International Conference on Autonomous Agents and Mulli-Agent Systems (AAMAS), page 279-287, 2021.
354
355
356
[8]
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Leslie Pack Kaelbling. Danica Kragic, and Komei Sugiura, editors, 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 66-75. PMLR, 2019.
357
358
359
380
[9] Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In Ronald Ortner, Hans Ulrich Simon, and Sandra Zilles, editors, Algorithmic Learning Theory 27th International Conference. ALT 2016. Bari, Italy, October 19-21, 2016, Proceedings. volume 9925 of Lecture Notes in Computer Science, pages 67-82, 2016.
361
382
383
364
[10] Mark Cutler, Thomas J. Walsh, and Jonathan P. How. Reinforcement learning with multi-fidelity simulators. In 2014 IEEE International Conference on Robotics and Automation, ICRA 2014, Hong Kong, China, May 31- June 7, 2014, pages 3888-3895, IEEE, 2014.
385
386
387
Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, and Peter Stone. An imitation from observation approach to transfer learning with dynamics mismatch. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Floring Balcan, and Hsuan-Tien Lin. editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020.
388
389
370
371
372
373
[12]
Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking importance weighting for deep learning under distribution shift. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Esuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
374
375
377
378
[13] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018
379
380
[14] Tanmay Gangwani, Joel Lehman, Qiang Liu, and Jian Peng. Learning belief representations for imitation learning in pomdps. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Thirty-Fifih Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, volume 1 15 of Proceedings of Machine Learning Research, pages 1061-1071. AUAI Press, 2019.
381
383
10

---

385
[15] Yonatan Geifman and Ran El- El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In Kamalika Chaudhuri and Ruslan Salakhutdinov. editors, Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2151-2159. PMLR, 2019.
386
386
389
390
[16] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4565-4573, 2016.
381
382
388
[17] Shengyi Jiang, Jing-Cheng Pang, and Yang Yu. Offline imitation learning with a misspecified simulator. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 12, 2020, virtual, 2020.
394
395
398
397
[18]
Kun Ho Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Cross domain imitation learning. CoRR, abs/1910.00105, 2019.
398
399
[19] Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imitation learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, pages 5286-5295, 2020.
401
[20] Bangalore Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Kumar Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. CoRR, abs/2002.00444 2020.
403
405
[21] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01643, 2020.
406
407
[22] Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez. Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
408
409
410
411
412
[23]
Yuxuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, pages 1118-1125, 2018.
413
414
415
416
[24] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5502,, 2013.
417
418
[25] Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 11 August 2017, pages 2681 2681-2690, 2017.
420
421
422
[26] Dripta S. Raychaudhuri, Sujoy Paul, Jeroen van Baar, and Amit K. Roy- foy-Chowdhury. Cross- domain imitation from observations. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021. Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9912-8912. PMLR, 2021.
424
426
428
[27] Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011, pages 627-635, 2011.
430
432
(28) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347 2017.
11

---

434
[29] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Tome-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robolics and Automation, ICRA 2018, Brishane, Australia, May 21-25, 2018, pages 1134-1141, 2018.
435
437
2018.
438
[30] Bradly C. Stadie, Pieter Abbeel, and Ilya Sutskever. Third person imitation learning. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
440
441
[31] Jakub Sygnowski and Henryk Michalewski. Learning from the memory of atari 2600. In Tristan Cazenave, Mark H. M. Winands, Stefan Edelkamp. Stephan Schiffel, Michael Thielscher, and Julian Togelius, editors, Computer Games - 5th Workshop on Computer Games, CGW 2016, and 5th Workshop on General Intelligence in Game-Playine Agents, GIGA 2016, Held in Conjunction with the 25th International Conference on Artificial Intelligence, IJCAI 2016, New York City, NY, USA, July 9-10, 2016, Revised Selected Papers, volume 705 of Communications in Computer and Information Science, pages 71-85, 2016.
443
445
447
448
[32] Andrea Tirinzoni, Andrea Sessa, Matteo Pirotta, and Marcello Restelli. Importance weighted transfer of samples in reinforcement learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stock- holmsmässan, Stockholm, Sweden, July 10-15, 15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4943 4952. PMLR, 2018.
449
450
451
452
[33] Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEF/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, pages 5026-5033, 2012.
454
[34] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using Journal of Machine Learning Research, 9:2579- 22092 2008
457
458
[35] Ruohan Wang, Carlo Ciliberto, Pierluigi Vito Amadori, and Yiannis Demiris. Random expert distillation: Imitation learning via expert policy support estimation. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6536-6544, PMLR, 2019.
459
480
482
(36] Andrew Warrington, Jonathan Wilder Lavington, Adam Scibior, Mark Schmidt, and Frank 463
464
Wood. Robust asymmetric learning in pomdps. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 11013-11023, PMLR, 2021.
465
486
487
Checklist
469
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
471
contributions and scope? [Yes]
472
(b) Did you describe the limitations of your work? [Yes] See Section 6
478
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See supplementary material.
474
475
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
476
2. If you are including theoretical results...
478
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
479
(b) Did you include complete proofs of all theoretical results? [N/A]
480
3. If you ran experiments..
481
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Section
12

---

483
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See supplementary material.
484
485
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section
488
489
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
480
(a) your work uses existing assets, did you cite the creators? [Yes]
491
(b) Did you mention the license of the assets? [Yes]
402
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
494
(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable
487
information or offensive content? [N/A]
496
5.
If you used crowdsourcing or conducted research with human subjects...
499
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
500
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
502
509
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "z2ZHW58JBx", "reviewer": "Reviewer_msPH", "summary": "The authors consider the problem of imitation learning, focusing on the challenging scenario where (i) the demonstrator and the learner act under different observation spaces, and (ii) there is only limited access to the demonstrator observations. The focus of the paper is thus on a paradigm the authors call \tHeterogeneously  Observable Imitation Learning: they propose methods for overcoming the challenges of the setting, and experimentally demonstrate their good behaviour on different tasks.", "questions": "- “…another is that in HOIL, instead of only considering the difference between two fixed policies, dealing with the dynamic change of the learner’s policy during learning is essential.”. I got the point but the sentence is unclear\n- “… While in HOIL, expert’s and learner’s observations are totally different from each other, in the sense that the observations themselves are not the obstacle for acting optimally.” This is unclear to me\n- AN state-action pair \n- A Trajectory should be probably defined as a set T = {x_i} I=…\n- X hat and h bar might become x_E and x_L?\n- Why pi hat instead of pi_L?\n- GAIL should be explicitly mentioned in the related works\n- Dynamics mismatch and the support mismatch are often mentioned but with no introduction. A definition may be beneficial for the non-expert user\n- On the experiments, some guide on how to read and interpret the plots may be useful for the understanding. Also it would be good to have a visual impression on the “category” of the other approaches (e.g. do they consider heterogenous states? Do they have access to the whole experts observations?) for an immediate comparison\n- Also 20 trajectories seem to me not enough to reach conclusions on the benefit of the method. Some comments on this point are needed", "limitations": "There are no comments on the  potential negative societal impact of the work. It is suggested to touch the point, focusing in particular on possible application domains of interest (e.g. autonomous driving as suggested by Fig. 1). ", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To the best of my knowledge, the work has a good level of originality on the theoretical side. Overall the structure of the manuscript is very good, with a clear storytelling. However, some parts are more clear than others. I find for instance the theoretical formulations a bit hard to follow (although I recognise that the theoretical part of the work requires it) while the experiments may be probably presented in a more interpretable way for the unfamiliar reader.\nMy impression is that the task considered by the authors present interesting challenges and deserves attention, and the advances with respect to existing IL approaches are convincing in Fig. 2. However, the authors should take care of the section of related works, in which the writing is not that clear.\nWhen needed, I clarify my observations in the Questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dYcoK-V4Fuz", "reviewer": "Reviewer_3iqV", "summary": "The authors propose a new challenging task for imitation learning: heterogeneously observable imitation learning. It is common that the demonstrator and the learner have totally different observation spaces, but it will make the imitator act sub-optimally because of dynamics mismatch and support mismatch. The authors propose the Importance Weighting with Rejection (IWRE) to solve the dynamics mismatch with importance sampling and support mismatch via rejection learning. They conduct experiments on Mujoco and Atari games and their method gets the best imitation performance.", "questions": "1. There is no explicit ablation study on your method. The effects of importance weighting and rejection learning are not clear from your current experiments.\n2. The experiment design does not clearly verify the arguments in the paper. It is not clear why the method works only from the imitation performance and a simple t-SNE visualization.\n3. Some visualization or analysis results to demonstrate that $I(D(x))$ and $g(x)$ correctly find the $H$, $O$, and $N$.\n4. Why is there no standard deviation of IWRE and GAIL in Humanoid result (Figure6 (e))?\n5. The Mujoco setup is strange. I think it is better to conduct experiments in naturally designed HOIL environments, such as BEV and first-person-view like LBC paper.\n", "limitations": "The limitation is mainly in experiment design and setup. Please see above.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe proposed heterogeneously observable imitation learning is interesting and challenging. The proposed method is novel and performs well across many different environments.\n\nWeaknesses:\n\n1. The experiment setup for Mujoco is strange. I’m sure that the last half of the original observations in MuJoCo are the features of velocity. I think it is unlikely to predict the actions only from the velocity information. Would you please make more clarification about this?\n2. The paper lacks sufficient analytical experiments to demonstrate their motivation and arguments. Why the method works is not clear.\n\nMinor issues:\n\n1. The $w_{2}$ under the $max$ should be $w_{1}$ in Eq(5) and Eq(6).\n2. A \")\" is missing in Eq(14), Eq(15) and Eq(16).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bZ3EnifInd", "reviewer": "Reviewer_pD56", "summary": "This paper proposes an algorithm for Heteogeneously Observable Imitation Learning (HOIL), a variant of imitation learning in which the demonstrator and imitator have different observation spaces, and it is expensive to obtain paired samples that depict corresponding imitator and demonstrator states. Specifically, in the HOIL setting, the data available to the agent includes:\n\n1. A small number of paired training samples (corresponding observations under the two observation spaces, along with actions) obtained from some policy $\\pi_1$ (which may not be the expert policy).\n2. The ability to roll out policies under the demonstrator's observation space alone.\n3. The ability to query which expert observation corresponds to a given demonstrator observation.\n\nThe proposed method is a form of GAIL that uses importance sampling to reweight the demonstrator/novice observations from data source (2) so that they approximately follow the marginal distribution of the expert policy in the novice observation space. When a given imitator observation falls outside the support of the previously recorded paired data (i.e. when the importance ratio is infinite), the agent actively queries for the corresponding demonstrator observation and adds it to the available paired training data. Experiments on MuJoCo and Atari against various baseline methods for dealing with different observation spaces suggest that this approach yields higher return with fewer paired (expert/novice) observation samples.", "questions": "Please see the questions above. I'll copy the specific questions most important to my score here to help the authors prioritise:\n\n- Thoroughness: on L271, how was the ratio of 1/4 chosen? What happens under different ratios?\n- Thoroughness: why is PPO-RAM performing worse than IWRE?\n- Significance: when would we expect sampling from the expert observation space to be higher cost than getting equivalent samples from the demonstrator observation space? When would the cost of active expert observation queries be lower than that of active expert action queries?\n- Limitations: below I ask for failure cases for IWRE. It would be good to know whether the authors encountered any environments where IWRE didn't work well for reasons that are intrinsic to the approach (as opposed to, e.g., a lack of hyperparameter tuning).", "limitations": "The paper is explicit in setting out its assumptions, but does not do much to delineate the set of problems where IWRE can be expected to perform well. In particular, it's not clear how the environments in Figure 6 were chosen from their respective benchmark suites (Atari/Gym MuJoCo)---what happened on other environments? Are there environments where IWRE does particularly poorly? I know that it is common to only report successful environments in RL papers, but this does lead to a warped view of how effective algorithms actually are on novel problems.\n\nI do not see any obvious ethical issues or anticipate negative social impact specific to this method.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality:** There is a substantial amount of existing work on imitation learning from demonstrations that are provided in a different observation space to the one available at evaluation time for the imitator. As I understand it, the novelty of this paper is in (1) explicitly treating the required number of paired observations (i.e. corresponding imitator/demonstrator observations) as something that should be minimised by the learning algorithm, and (2) proposing a combination of importance sampling and active learning to achieve this goal. The proposed approach to this problem seems quite novel to me. However, I'm not convinced of the desirability of minimising the amount of paired data, or of the assumption that active querying for paired data is cheap (this is mentioned more in \"significance\", below).\n\n**Clarity:** The paper is a bit notationally heavy (here are many subscripts and accents without obvious mnemonic meanings), so it took me a while to digest the meat of the method. Here are some suggestions for things that could be clarified in the paper:\n\n- L6/L115: the paper refers to observation spaces \"coexisting\". After reading the paper, I think \"the two observation spaces coexist\" means \"the agent can sample paired observations from both observation spaces\"; however, I found this terminology confusing at first, since observation spaces are just sets, and it's not clear what it means for two sets to \"coexist\".\n- L11: \"dynamics mismatch\" is referred to many times but doesn't actually seem to be defined in the paper. What does this term mean?\n- Figure 2: it's not clear at this point in the paper what $\\pi_1$, $\\pi_2$, etc. are, what the script $T$s are, or why they have bars over them. Explaining this in section 1 or moving Figure 2 to later in the paper would be helpful.\n- L62-64: \"dealing with the dynamic change of the learner's policy during learning is essential\" --- what does this actually mean? In what sense does adversarial domain adaptation only need to consider the difference between \"two fixed policies\"?\n- L113: \"evolving data\" is an odd name for ${\\mathcal T}_{\\pi_1}$ given that the dataset in question seems to be fixed at the beginning of training, and doesn't \"evolve\" in any obvious way.\n- Figure 4: the customary symbol for the empty set is $\\varnothing$ (`\\varnothing`), not the Greek letter $\\phi$ (`\\phi`). The latter is currently used throughout the paper.\n- Figure 4: $H$, $O$ and $N$ should be defined at the beginning of the caption, before the explanation of the figure.\n- Equations (14)-(16): is $\\langle \\mathbb I(D^*_w), g^*\\rangle$  actually an inner product between two vectors, or is it just multipyling two scalars together? I thought the latter, but the notation suggests they are higher dimensional. (same issue for all three equations)\n- Definition (3): IMO it would be easier to read this as $\\operatorname{supp}(\\rho_{\\pi_E}) \\cap \\operatorname{supp}(\\rho_{\\pi_1})$ (or at least have the equivalence made explicit).\n- Eqn. (18): where are the $x_i$s sampled from?\n- In general: it would be helpful to have a table of notation in the appendix.\n\n**Thoroughness/rigour:**  After getting through the notation, I felt that the proposed method made sense. I appreciate that the method explicitly allows for active querying in situations where the support of the expert observations and novice observations do not overlap; without this, it seems like it would be theoretically impossible to solve the proposed problem with importance weighting. I also appreciated the range of expert baselines that take very different approaches to the problem of observation space mismatch (the proposed method is quite complicated, and so it's good to see that complexity justified by a comparison to a broad range of baselines).\n\nSome concerns about thoroughness and rigour:\n\n- L271: how was the ratio of 1/4 chosen? This choice feels somewhat arbitrary, and I feel like Figure (6) does not do a great job of communicating the tradeoff between taking samples from $\\pi_1$ and $\\pi_2$. Re-plotting the graphs with different ratios would be useful, or perhaps using a different metric like \"number of samples of $\\pi_1$ vs. number of samples of $\\pi_2$ required to get to performance level $x$\".\n- Figure 6: why is PPO-RAM regularly performing worse than IWRE in most of these plots? It seems like that method should provide an upper bound on performance, given that it doesn't have to learn a reward function.\n\n**Significance:** The empirical results of the paper are quite strong according to the chosen metric (return as a function of samples, with paired/expert observations being more costly). The overall significance of the paper to the field depends on how meaningful this metric is. I feel that this could be better justified in the beginning of the paper. Specifically:\n\n- L28 claims that the observation space of the expert is \"often high cost\" for the learner to sample from. For what applications in this true? Are these also applications where active querying of expert observations is easy? Just having a motivating application to think about would be useful. I don't really understand the current self-driving car motivation at the beginning of the paper because it doesn't seem like sticking a camera on the dashboard or recording the instrument panel is especially expensive (it's also very hard to actively query \"expert\" observations in this setting).\n- L124 claims that the cost of querying for expert observations is \"much lower\" than querying for expert actions in a given state (like DAgger does). It's not clear to me when or why this would be the case (in part because I'm not sure what would be \"expensive\" about reconstructing the expert observation). Again, a motivating example would be useful as an intution pump.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject