Seeing Differently, Acting Similarly: Heterogeneously Observable Imitation Learning
Authors: Xin-Qiang Cai
Yao-Xiang Ding
Zixuan Chen
Yuan Jiang
Masashi Sugiyama
Zhi-Hua Zhou
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Imitation Learning
Heterogeneous Observation
Importance Weighting
Learning with Rejection

Abstract: In many real-world imitation learning tasks, the demonstrator and the learner have to act under different observation spaces. This situation brings significant obstacles to existing imitation learning approaches, since most of them learn policies under homogeneous observation spaces. On the other hand, previous studies under different observation spaces have strong assumptions that these two observation spaces coexist during the entire learning process. However, in reality, the observation coexistence will be limited due to the high cost of acquiring expert observations. In this work, we study this challenging problem with limited observation coexistence under heterogeneous observations: Heterogeneously Observable Imitation Learning (HOIL). We identify two underlying issues in HOIL: the dynamics mismatch and the support mismatch, and further propose the Importance Weighting with REjection (IWRE) algorithm based on importance weighting and learning with rejection to solve HOIL problems. Experimental results show that IWRE can solve various HOIL tasks, including the challenging tasks of transforming the vision-based demonstrations to random access memory (RAM)-based policies in the Atari domain, even with limited visual observations.

Published as a conference paper at ICLR 2023
SEEING DIFFERENTLY, ACTING SIMILARLY: HETEROGENEOUSLY OBSERVABLE IMITATION LEARN-
ING
Xin-Qiang Cail2 Yao-Xiang Ding' Zi-Xuan Chen'
Yuan Jiang"
Masashi Sugiyama
Zhi-Hua Zhou'
National Key Laboratory for Novel Software Technology, Nanjing University
The University of Tokyo
State Key Laboratory for CAD&CG, Zhejiang University
*RIKEN Center for Advanced Intelligence Project
ca i @ms. k. k.u-tokyo ac jp, di ngyx. gm0 i 1. com, chenzx@ n ju edu. cn, edu. en, sugi @k u-tokyo . ac jp, zhouzhē 1 lamda nju. .edu..du... cn
jiangyelamdn n eduu
ABSTRACT
In many real-world imitation learning tasks, the demonstrator and the learner have to act under different observation spaces. This situation brings significant ob- stacles to existing imitation learning approaches, since most of them learn poli- cies under homogeneous observation spaces. On the other hand, previous studies under different observation spaces have strong assumptions that these two ob- servation spaces coexist during the entire learning process. However, in reality, the observation coexistence will be limited due to the high cost of acquiring ex- pert observations. In this work, we study this challenging problem with limited observation coexistence under heterogeneous observations: Heterogeneouslly Ob- servable Imitation Learning (HOIL). We identify two underlying issues in HOIL: the dynamics mismatch and the support mismatch, and further propose the Impor- tance Weighting with REjection (IWRE) algorithm based on importance weighting and learning with rejection to solve HOIL problems. Experimental results show that IWRE can solve various HOIL tasks, including the challenging tasks of trans- forming the vision-based demonstrations to random access memory (RAM)-based policies in the Atari domain, even with limited visual observations.
INTRODUCTION
Imitation Learning (IL), which studies how to Iearn a good policy by imitating the given demonstra- tions (Xu et al., 2020 Chen et al.] (2022). has made significant progress in real-world applications such as autonomous driving (Chen et al. 2019). health care (Iyr et al.] 202T). and continuous control (Wang al. 2023). In tradition, the expert and the learner assumed to the same observation space. However, nowadays, many real-world IL tasks demand to remove this assump- tion (Chen et al.) 2019: Warrington al. (021), such in autonomous driving (Chen al. 2019). recommendation system (Wu et al.) 2019). ), and medical decision making (Wang et al.) 2021a) la). Tak- ing AI for medical diagnosis as an example in Figure ! A medical AI learning to make medical decisions based on expert doctor demonstrations. To ensure demonstration quality, the expert may use high-cost observations such as CT, MRI, and B-ultrasound. In contrast, the AI learner is ideal to use only low-cost observations from cheaper devices, which could be newly designed ones that have not been used previously by the expert. Meanwhile, to ensure reliability, it also reasonable to allow the learner to access the high-cost observations during training under limited budget (Yu) et al. 2019). The above examples share three characteristics: (i) Even though pair of expert and learner observations can be different, they are under the same state of the environment, leading to similar policies; (ii) The learner's 's new observations are not available to the expert when generat- ing demonstrations; (iii) During training, the learner can only access expert observations under limited budget, in special the high-cost ones, since it is also important to minimize the usage of the
"Corresponding author.

---

Published as a conference paper at ICLR 2023
Expert demonsitation
Leamer
Figure 1: Medical decision making: example of the HOIL problem. Figures 1, 2, and 3 include some illustrations and pictures from the Internet (source: https:: /wwww flat con. com/)
Table 1: Comparisons between different II processes. OE and Or, denote the observation spaces for experts and learners respectively. N/A denotes not applicable.
Setting
DSIL
Pou.
LBC
HOIL((our)
Oh Cr
X
demonstrations not include Ot. The leamner does not require Op4Cr all the time Ou not more privileged than O2
NIA
NIA
expert observations during training to avoid unnecessary costs. We name such IL tasks Heteroge- neously Observable Imitation Learning (HOIL). Among them, we focus on the most challenging HOIL setting in which the expert and learner observation spaces have no overlap.
There are two lines of research studying the related problems, summarized in Table I and Figure 2 The first one relates to Domain-shifted IL (DSIL): the observation spaces of experts and learn- are the homogeneous, while some typical distribution mismatches could exist: morphological, viewpoint, and dynamics mismatch (Stadie et al., 2017]] RRaycaaudhrri et al.] [02I]. However, the approaches for DSIL are invalid when the observation spaces are heterogeneous in HOIL.
The second line studied H. under different observations similar to HOIL. Some representative works include Partially Observable IL (POIL) (Gangwani et al., 2019 Warrington al.) (021) and Learn- ing by Cheating (LBC) (Chen et al. 2019). Both POIL and LBC assume that the learner can easily access the expert observations without any budget limit. However, in practice, different from the learner observations, the access to expert observations might be of high cost, invasive, and even unavailable Yu al. 2019). which hinder the wide application of these methods.
In this paper, we initialize the study of the HOIL problem. We propose learning process across observation spaces of experts and learners to solve this problem. We further analyze the underlying issues of HOIL, i.e., the dynamics mismatch and the support mismatch. To tackle both two issues, we use the techniques of importance weighting (Shimodaira] 2000) Fang et al., 2020) and learning with rejection (Cortes et al., 2016; Geifman & El- Yaniv, 2019) for active querying to propose the Importance Weighting with REjection (IWRE) approach. We evaluate the effectiveness the IWRE algorithm in continuous control tasks of MuloCo (Todorov al., 2012), and the challenging tasks of learning random access memory (RAM)-based policies from vision- based expert demonstrations in Atari (Bellemare et al. 2013) games. The results demonstrate that IWRE can significantly out- perform existing IL algorithms in HOIL tasks, with limited access to expert observations.
2 RELATED WORK
FESL.. Recently, to deal with the significant challenges in open environment learning (Zhou 2022 there are emerging studies of feature evolvable stream learning (FESL) (Hou et al.) 2017 Hou & Zhou 2018 Zhang et al., (2020), which are among major inspirations of our work. FESL focuses on supervised learning with heterogeneous feature space. There are also significant differences between existing FESL approaches and HOIL. In (Hou et al. 2017 Hou & Zhou 2018), they assume that data features are generated from heterogeneous but fixed and static data distributions, while in HOIL, the data distributions are dynamically changing. FESL with dynamically changing distributions studied in (Zhan et al. 2020), while they assume that the changes are passive to the learner. This is different from HOII., in which the data distribution changes are actively decided by the learner due to the nature of decision-making learning.
DSIL. For the standard IL process, where the learner and the expert share the same observation space, current state-of-the--rt methods tend to learn the policy in adversarial style (Wang et al.) 2021b[c), like Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon! 2016). When con- sidering the domain mismatch problem, i.e., DSIL, the research aims addressing the static dis- tributional shift of the optimal policies resulted from the environmental differences but still under homogeneous observation spaces. Stadie al. (2017), Sermanet et al. (2018). and Liu et al. (2018)
2

---

Published as a conference paper at ICLR 2023
Oberrttte Spoce
Oberzzaa Spara
Oberrzake Spate
Lombo
Surila
Lortia
260
(a) DSIL
(b) POIL
(c) LBC
(d) HOIL (our work)
Figure 2: Comparison sketches of IL with respect to observation spaces. The target of (a) is to learn
a policy based on the same observations of the expert, while that of (b), (c), and (d) are learn
policy based on the second observation space only. The detailed differences can be found in Table T
studied the situation where the demonstrations
in view of third person. Kim et al. (2020) and
Kim et al. (2019) addressed the IL problem with morphological mismatch between the expert's s and
learner's 's environments. Tirinzoni et al. (2088, (.18),,Jaana Jiang al. 2020), Desai et al. (2020) focused on the calibration for the mismatch between dimulators and the real world through some transfer learn- ing styles. There are two major differences between HOIL and DSIL: One is that HOIL considers heterogeneous observation spaces instead of homogeneous ones; another is that without observation heterogeneity, DSIL can directly align two fixed domains, which may not be realistic for solv- ing HOIL when two observation spaces are totally different. Thus HOIL is a significantly more challenging problem than DSIL. Besides, [Cen et al. (2019) learned a vision-based agent from a privileged expert. But it can obtain expert observations throughout the whole learning process, so it cannot handle the problem of the support mismatch under HOIL.
POMDP. The problem of Partially Observable Markov Decision Process (POMDPs), in which only partial observations are available for the agent(s), has been studied in the context of multi- agent (Omidshafiei et al., 20177; Warrington al. (021) and IL (Gangwani et al., 2019; Warrington let al. (021)) problems. But distinct from HOIL, in a POMDP, the learner only has partial obser- vations and shares the same underlying observation space with the expert, which would become an obstacle for them to make decisions correctly. For example, Warrington et al. (202T) assumed that the observation of the learner partial than that of the expert. Instead, in HOIL, expert's and learner's 's observations are totally different from each other, while the learner's observations do not belong to the expert's. For HOIL,, the main challenge is to deal with the mismatches between the observation spaces, especially when the access to expert observations is strictly limited.
THE HOIL PROBLEM
In this section, we first give a formal definition of the HOIL setting, and then introduce the learning process for solving the HOIL problem.
3.1 SETTING DEFINITION
A HOIL problem is defined within Markov decision process with mutiple observation spaces, i.e.. (S, (O), A, P,7), where S denotes the state space, {0} denotes a set of observation spaces, A denotes the action space, P S x A x S - R denotes the transition probability distribution of the state and action, and 7 E (0, 1] denotes the discount factor. Furthermore, a policy и over an observation space O is defined as a function mapping from O to A, and we denote by Ill the set of all policies over O. In HOIL, both the expert and the learner have their own observation spaces, which denoted as OE and OL respectively. Both Og and Og two bijective mappings fE S + OB, ft. : S + Or., which are unknown functions mapping the underlying true states to the observations. is obvious see that by this assumption, any policy over Og has a unique correspondence over OL. This makes HOIL possible since the target of HOIL is to find the policy corresponding to the expert policy under OL.
assumed to be produced by
A state-action pair (s, a). denoted by x, is called an instance. Also, a trajectory T = (r.),i {1...., m) is set of m instances. For each observation space, 3 + C Og x and E E T C A, where Og = fe) and O2 = (S). Furthermore, we define the occupancy measure of a policy under the state space S as PT S x A + IR. such that (=(x) x (alo)) E
s|TT). Under HOIL, the learner accesses the expert demonstrations Tng a set of instances sampled from Pag. The goal of HOIL is to leam policy Ω as the corresponding policy of тp over OL. If OE = Or., HOIL degenerates to standard IL. GAIL (Ho & Ermon] 2016) is one of the off
art II. approaches under this situation, which tries minimize the divergence between the learner's
3

---

Published as a conference paper at ICLR 2023
Obscrvvation Syuce
MIN
Desonstrationy
Fey
Fritraning
Process
Trateing
122
Figure 3: Illustration of learning process across two different observation spaces for solving HOII.. Ti is an auxiliary policy that is additionally provided.
and the expert's 's occupancy measures d(p=, Prm)) The objective of GAIL is
min max max [log ww (3) + Eg [log(1 Dw (w) H
(1)
where H(+) is the causal entropy performed as regularization term, and D., OF A - (0, 1] is the discriminator of गe and म. GAIL solved Equation (I by alternatively taking gradient ascent step to train the discriminator Dw. and minimization step to learn policy à based on an off-the-shelf reinforcement learning (RL) algorithm with the pscudo reward log D (2).
3.2 THE LEARNING PROCESS FOR SOLVING HOIL
In HOIL, we need to cope with the absence of the learner's observations in demonstrations and the high cost of collecting expert observations while learning. So we introduce a learning process with pretraining across two different observation spaces for solving HOIL, as abstracted in Figure 3
Pretraining. Same as LBC (Chen al., 2019), we assume that we can obtain an auxiliary policy Ti based on OF: at the beginning. can be directly provided by any sources, trained by GAIL or behavior cloning (Michie al. 1990) from online data stream (Cai et al., (201)) or offline demon- strations (Sasaki & Yamashina T02T as did in LBC. Besides, we use this Ti to sample some data TR,, which contain both observation under Og (i.e., Tr, and OL (i.e., T=,), in order to connect these two different observation spaces. We name T = (T Tx,)
} the initial data.
Training. Here we learn a policy T2 from the initial data T and the collected data T under OL only. Besides, the learner is allowed for some operation of observation coexistence (OC); At some steps of learning, besides the observations OL, the learner could also request T from the corresponding observations OE (e.g., from the human-unders audlet sensors). The final objective of HOIL is to leam good policy n2 under OL.
In practical applications, the auxiliary policy и1 can also come from simulation training or direct imitation. But since nı is additionally provided, it is more practical to consider ι non-optimal policy. During training, OC is an essential operation for solving HOII,, which helps the learner address the issues of the dynamics mismatch and the support mismatch (especially the latter one). Also, in reality, we do not need an oracle for actions, which still needs OC for obtaining expert observations first, in many active querying research (Brantley al. 2020 Chen al. 2019). so its cost will be relatively lower.
Besides, the related work (Chen al. 2019) also required an initialized policy मi solve their problem, which acts as a teacher under privileged Og in the pretraining and then learned a vision- based student from the guidance of the teacher under both Or and Og. Their setting can be viewed as a variety of HOII, with optimal unlimited Op, and unlimited OC operations, so HOIL. is actually a more practical learning framework.
IMITATION LEARNING WITH IMPORTANCE WEIGHTING AND REJECTION
In HOIL. the access frequency to Og is strictly limited, so it is unrealistic to learn Tि in a Dataset Aggregation (DAgger) style (Ross et al. 2011) in LBC. Therefore, we resort to learning To with a learned reward function by inverse RL (Abbeel & Ng 2010) in an adversarial learning style (Ho
& Ermon 2016 Fu et al. 2018).
4

---

Published as a conference paper at ICLR 2023
Prt
Piu.
Dh.
Protuaalliyy
Protedalliy
Probebiltty
k2x
H
0
N
541
+1
(a) The idcal situation
The real situation
(c) The output of [w(Dw): g)
Figure 4: The comparisons among the distributions of expert demonstrations Prg. initial data Pri and non-expert data PaNE- The red and blue regions denote the expert and non-expert parts Pax respectively. 11, O, and N denote the latent demonstration, the observed demonstration, and the non-expert data respectively. (a) The ideal situation, where supp(Prg) supp((r.) = Ø; (b) The real situation. where = supp(pTR) (supp((=> + Pau (c) The target output of the combined model l"."". The output +1, O, and -1 regions correspond to H, O, and N respectively.
In addition, both Og and O, are assumed to share the same latent state space as introduced in Section [.I]] so the following analysis will be based on S, while the algorithm will handle the problem based on OE and OL specifically.
4.1 DYNAMICS MISMATCH AND IMPORTANCE WEIGHTING
To analyze the learning process, we let PTg. Pri, and Pra the occupancy measure distributions of the expert demonstrations, the initial data, and the data during training respectively. Since we need to consider the sub-optimality of TT. Pa. should be mixture distribution of the expert Prg and non-expert PENE: i.c., there exists some (0, 1) such that
Pr: = 6pmz + (1 8)PTNE:
as depicted in Figure 4a demonstrations. To this end, the original objective of reward function Dun for Ta is to optimize
During training, the original objective of T2 is to imitate Te through
max Bx-p.ng [log Dwng (x) Expp.gg. Dog(1 Dww(())) W2
But the expert demonstrations are only available under Og. During training, we can only utilize the initial data T x, ~ Ps, to learn and D Dwy. Besides, as is sub-optimal, directly imitating T could reduce the performance of the optimal 72 that of ni So use the importance weighting to calibrate this dynamics mismatch, i.e.,
max C(Dwa) = Ee-pey [log Dw,, (7) Ezop-, [a(x) log(1 - Dwy (7)) 10g
(4)
where axx) Pag. (x)
is an importance weighting factor (Fang et al. 2020). So the current issue lies
Peg (a)
in how to estimate fee. under Op. To achieve this purpose, we need to bridge the expert demonstra-
Prj
tions and the initial data. Therefore, here these two data sets to train adversarial model Dw, in the same way as Dw, in the pretraining:
max w() A Eg=pp, [log Dw,, (7) + E Enwopr [log(1 - Dw, (+))
(5)
we write the training criterion (5) in the form of integral, i.c.
max L(Dw))) (pm log Dwn + png log(1 Dmn }]dæ,
(6)
1/1
then, by setting the derivative of the objective to 0 (DD,
ac 0), we can obtain the optimum D Dwy;
Px:
D w1
PFi + PrE
5

---

Published as a conference paper at ICLR 2023
in which the order of differentiation and integration was changed by the Leibniz rule. Besides, sufficiently train Dw, using the initial data Tx, and the expert demonstrations T Then Duy will be good enough to estimate the importance weighting factor, i.c.
PTE 1 - D D
1 Dam (z) D (x)
a(x
(8)
Pr1
In this way, we can use Day, which can connect demonstrations and initial data, to calibrate the learning process of Dwn. The final optimization objective for Dwv is
1 1 Dwn (3) log[1 Dwy (+))-
max ((2 Exupez log Dwn (I) + Exwepr
W22
Dae
In this way, Dwy can effectively dig out the expert part of Pr. and produce efficient rewards for π2.
4.2 SUPPORT MISMATCH
So far, the challenges have still been similar to homo- geneously observable IL. However, our preliminary ex- periments demonstrated that mere importance weighting is not enough to fix the problem that occurred by the absence of interactions under Og. So there exist some other issues between the expert demonstrations and the initial data. To find out the underlying issues, we plot- ted the t-distributed Stochastic Neighbor Embedding (t- SNE) (van der Maaten & Hinton 20088) visualizations of these two empirical distributions under OE on Hopper and Walker2d Figure 5 Twenty trajectories were col- lected for both the expert demonstrations and the initial data. We can observe that there exist some high-density regions of demonstrations in which the initial data do not cover; that is, there exist some regions of the demonstrations that did not explore. Wang et al. (2019) found similar phenomenon the standard IL setting. On the other hand, the importance weighting a cannot calibrate this situation where Pag 30.
(a) Hopper
(b) Walker2d
Figure 5: t-SNE visualizations of expert demonstrations and collected data of m under OE.
To formulate this problem, here introduce the support set of the occupancy measure:
Definition (Support Set). The support of an occupancy measure p is the subset of the domain containing the elements which are not mapped to zero:
supp(ø) = =xx
Ex E x Alp(x) + 0).
(10)
Due to the sub-optimality of wı, supp(pre) supp(pm.) + Ø (see Figure 4b We call this part the latent demonstration, defined as:
Definition 2 (Latent Demonstration). The latent demonstration H is the of the domain that belongs to the relative complement of supp(pe,) in supp(pag):
H = (x E S x Alsupp(Prg) supp(ps,)):
(11)
Also, another part of the demonstration is named the observed demonstration, defined as: Definition 3 (Observed Demonstration). The observed demonstration O is the set of the domain that belongs the complement of H in supp(Pag):
{x S Alsupp((ze) supp(p=)))-
(12)
Besides, the data outside of demonstrations should be non-expert data: Definition 4 -Non-Expers Data). The non-expert data N is the set of the domain out of supp (ppp)(=m)):
N {ar 8 A PTg (x) 0}.
(13)

---

Published as a conference paper at ICLR 2023
In other words, the sub-optimality of will cause not only the dynamics mismatch, but also the appearance of the latent demonstration H. We call the latter one the problem of support mismatch. Intuitively, when π² NE, we have H Ø. monotonously. So in order to fix the support mismatch between Pag and Pm1 guiding 22 to find out is the key.
In addition, the support mismatch problem
be viewed as inverse problem of the out-of- distribution (OOD) problem that frequently occurred in offline RL setting (Levine et al. 2020), in which they tried avoid supp(Prr) supp(Prg) instead.
4.3 IMITATION LEARNING WITH REJECTION
We can observe that HOOUN = S A. So is desirable to filter out H from O and N. Meanwhile, Dwy and Dw, can only classify 0 H and N, under Og and On respectively. Therefore, here we design two models 91 OØ x A + }} and g2 OL x A + {0, 1} (Output O: E O and output I: otherwise), that given T (corresponding T T and H T), they can satisfy
= {i E A|T (F) (z) D (3) (x) +1},
(14)
= {x E S x AlIlD (1) (T) = I[D w2 (g) = 0},
(15)
N = (x E S x A|D (3) (z) = I[ 30 (7) (3) = -1},
(16)
where I[:] takes +1 if > 0.5, and - otherwise. (1)lg (1) is depicted in Figure 4c
To this end, both 9i and s2 should be able to cover O meanwhile ø2 can be adaptive to the continuous change of pxa due to the update of π2. Here we learn 91 and g2 in a rejection form, to reject from U H (where Il(,,.) +)). Concretely, the rejection setting is the same in Cortes et al. 20166)
Also, inspired by Geifman & EI- Yaniv (2019), the optimization objective of D and g is
C(D 9) & i(D... 9) + À ma c ((g))²
(17)
where e >> 0 denotes the target coverage, and À denotes the factor for controlling the relative impor- tance of rejection. Besides, the empirical coverage ò(g) is defined as
x(g]X)
Σ g(z.),
(18)
with a batch of data X { [m]] The empirical rejection risk i(D g) is the ratio between the covered risk of the discriminator and the empirical coverage:
i(Dw. g) m (I.)) ò(g)
(19)
Meanwhile, both Dwn and 9I can access Pag under Op: directly. So given T Tx, under O,. once (I(Duw (T)), (T)) +, we can query the corresponding observations of 7 through OC operation and use (I(Dw, (T)). gi (ō)) to calibrate the output of ge and Dwy. In this way, 92 and D can be entangled together and adaptively guide π2 to find out the latent demonstrations H under OL.
4.4 IWRE
Here combine importance weighting and rejection into unified procedure, resulting in novel algorithm named Importance Weighting with REjection (IWRE). Concretely, in HOIL process:
Pretraining. We train a discriminator Dw, by Equation (5) and its corresponding rejection model by Equation (17 using the initial data and the expert demonstrations.
Training. We train discriminator Dan by the combination of Equation 9 and Equation 17 as well as its corresponding rejection model by Equation (17), using the initial data, the data collected by и2. and the output of D.., with 9i through OC operation. Also, л2 will be updated with Dws, and g2 asymmetrically as in GAIL.
The pseudo-code of our algorithm is provided in the appendix.
7

---

Published as a conference paper at ICLR 2023
Rowards
Rowards
Rowards
CIML GALAA
Episodes
Episodes
Episedes
Episodes
(a)
ChopperCommand
(b) Kangaroo (Atari)
(c) Qbert (Atari)
(d) Hopper (MuJoCo)
(Atari)
Rewards
Rowards
Rowardh
Rowards
Kplisndes
Episodes
Epioodes
Epicodes
(c) Humanoid (MuJoCo) (f) Reacher (MuJoCo)
(g) Swimmer (MuloCo) (h) Walker2d (MuloCo)
Figure 6: The learning curves of each method. Shaded regions indicate the standard deviation.
5 EXPERIMENT
In this section, we validate our algorithm Atari 2600 (Bellemare et al.]20133) 2013) (GPL. License) and
MuJoCo (Toooovv et al. 2012) (Academic License) environments. The experiments were designed to investigate: 1) Can IWRE achieve significant performance under HOIL tasks? 2) Can IWRE deal with the support mismatch problem? 3) During training, is active querying for HOIL indeed necessary? Below we first introduce the experimental setup and then investigate the above questions. More results and experimental details are included in the appendix.
5.1 EXPERIMENTAL SETUP
Environments. We choose three pixel-memory based games in Atari and five continuous control objects MuloCo on OpenAl platform (Brockman et al., 2016) (MIT License). For pixel-memory Atari games, Og: 84 X 84 raw pixels; Oz. 128-byte random access memories (RAM). For continuous control MuJoCo objects. Og: half of original observation features; OL: another half of original observation features. Besides, twenty expert trajectories were collected for each environ- ment. Each result contains five trials with different random seeds. All experiments were conducted on server clusters with NVIDIA Tesla V100 GPUs. Further details of the environments are included in the appendix.
Baselines. Six basic contenders were included in the experiments: Vanilla GAIL (Ho & Ermon 2016), GAIL with importance weighting (Fang et al.] 2020) (IW, an ablation study of TWRE), third- person IL (Stadie al. 2017) (TPIL), generative adversarial MDP alignment (Kim al.) 2020) (GAMA), behavioral cloning Bain & Sammut 1996) (BC), and learning by cheating (Chen et al., 2019 (LBC). For IW, we utilized Da,, trained in the pretraining to calculate the importance weight; also the optimization objective for Dw., during training is the same as Equation (9: TPIL learns the third-person demonstrations by leading the cross-entropy loss into the update of the feature extractor: GAMA learns mapping function & in view of adversarial training to align the observation of the target domain into the source domain, and thereby can utilize the policy in the source domain for zero-shot imitation. For fairness, we allowed the interaction between the policy and the environment for GAMA under HOIL; LBC uses Hi learned from privileged states as a teacher to train TT in DAgger (Ross et al., 2011) style, so here we allowed LBC to access OE during the whole IL. process. In Atari, to investigate whether our method could achieve good performance for RAM- based control, we further included contender PPO-RAM, which uses proximal policy optimization (PPO) (Schulman et al., (017) to perform RL directly with environmental true rewards under the RAM-based observations. More detailed setup including query strategies for TPIL and GAMA, network architecture, and ppeemmerraeeerss are reported in the appendix.
5.2 RESULTS
Experimental results are reported in Figure 6 Since the mapping function is hard to learn when input is RAM and output is raw images, omit the results of GAMA in Atari, We can observe
8

---

Published as a conference paper at ICLR 2023
that while IW is better than GAIL in most environments, both GAIL and IW can hardly outperform T1. Because they just imitated the performance of πΙ instead of Tg. even with importance weighting Hopper, Swimmer, and Walker2d due to the continuous distribution shift. Furthermore, the performance of GAMA was not satisfactory in Hopper and Walker2d because its mapping function is hard to learn well when the support mismatch appears. The results of TPIL and GAMA demonstrate that DSIL methods will be invalid under heterogeneous observations as in HOIL tasks, On Atari environments, OE contains more privileged information than OL so LBC can achieve good performance. But when Og is not more privileged than OL, like in most environments of MuloCo, its performance will decrease due to the support mismatch, which would make it even worse than BC. Finally, IWRE obtained the best performance on 6/8 environments, and comparable performance with LBC on Reacher, which shows the effectiveness of our method even with limited access to OE (LBC can access to Og all the time). Besides, we can see that the performance differences between the GAIL/IW and TWRE/PPILGAMA/LLI which demonstrates that the query operation is indeed necessary for HOIL problems.
for calibration. For TPIL, its learning process was extremely unstable
are huge (especially on Reacher) because of the absence of queries,
Moreover, even learned with true rewards, PPO-RAM surprisingly failed to achieve comparable performance to IWRE, which shows that IWRE could possibly learn more effective rewards than true environmental rewards in RAM-input tasks. The results verify that, IWRE provides powerful approach for tackling HOIL, problems, even under the situation that the demonstrations are gathered from such a different observation space, meanwhile Og is strictly limited during training.
(-SNE visualization of Pr₂ and pig under Og. In Sec- tion 4.2 we point that the sub-optimality of क will cause the problem of support mismatch, which is embodied as the appearance of the latent demonstration H during training. Also the empirical results in Figure 5 on Hop- per and Walker2d verify the existence of this problem. So we want to investigate whether the superiority of IWRE indeed comes from successfully tackling the support mis- match problem. To this end, we plotted the t-SNE vi- sualization of the same expert demonstrations as in Sec- tion 4.2 and the collected data of π2 by IWRE under Og (Op is hidden to T2). All setups are the same as in Sec- tion From the results shown in Figure 7 we can see that even under OF. which cannot be obtained by 72, al- most all high-density regions of the demonstrations were covered by the collected data. Meanwhile, the latent demonstration H is dug out nearly. The results demonstrate that IWRE basically solves the problem of support mismatch and thereby performs well in these environments.
(a) Hopper
(b) Walker2d
Figure 7: t-SNE visualizations of expert demonstrations and collected data of ma under Og. The high-density regions of the expert demonstrations were covered by the collected data of πх of IWRE.
Besides, some collected data of T2 of IWRE were out of the distribution of the demonstrations, which means T2 slightly overly explored the environment. Since Og is hidden to T2. the reward function will encourage to explore more areas to fix the support mismatch problem. Meanwhile, the -utto--distribution problem HOIL not as severe as in the offline RL, settings (Levine et al., 2020), so this over-exploration phenomenon makes sense.
CONCLUSION
In this paper, we proposed new learning framework named Heterogeneouslly Observable Imita- tion Learning (HOIL), to formulate situations where the observation space of demonstrations differs from that of the imitator while learning. We formally modeled the learning process of HOIL. in which access to expert observations is limited due to the high cost. Furthermore, we analyzed the un- derlying challenges of HOIL: the dynamics mismatch and the support mismatch, on the occupancy distributions between the demonstrations and the policy. To tackle these challenges, we proposed new algorithm named Importance Weighting with REjection (IWRE), using importance weighting and learning with rejection. Experimental results showed that the direct imitation and domain adap- tive methods could not solve this problem, while our approach obtained promising results. In the future, we hope give the theoretical guarantee for our algorithm IWRE and investigate how many OF we need to query to learn a promising ny. Furthermore, we hope to use the learning framework of HOII. and IWRE to tackle more learning scenarios with demonstrations in different spaces.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENT
This research was supported by: National Key Research and Development Program of China (2020AAA0109401); National Science Foundation of China (62176117, 61921006, 62206245); the Institute for AI and Beyond, UTokyo; JST SPRING, Grant Number JPMISP2108. The au- thors would like to thank Guoqing Liu, Yoshihiro Nagano, and the anonymous reviewers for their insightful comments and suggestions.
REFERENCES
Pieter Abbeel and Andrew Y. Ng. Inverse reinforcement learning. In Encyclopedia of Machine Learning. pp. 558. 2010.
Michael Bain and Claude Sammut. framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1996.
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning envi- ronment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Kianté Brantley, Hal Daumé III, and Amr Sharaf. Active imitation learning with noisy guidance. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, July 5-10, 2020, Online, pp. 2093-2105. Association for Computational Linguistics, 2020.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.01 540, 2016.
Xin-Qiang Cai, Peng Zhao, Kai-Ming Ting, Xin Mu, and Yuan Jiang. Nearest neighbor ensembles: An effective method for difficult problems in streaming classification with emerging new classes. In Jianyong Wang. Kyuseok Shim, and Xindong Wu (eds.), 2019 IEEE International Conference Data Mining, ICDM 2019, November 8-11, 2019, Beijing, China, pp. 970-975. IEEE, 2019.
Xin- Xin-Qiang Cai, Yao-Xin Xiang Ding, Yuan Jiang, and Zhi-Hua Zhou. Imitation learning from pixel- level demonstrations by hashreward. In Frank Dignum, Alessio Lomuscio, Ulle Endriss, and Ann Nowé (eds.), AAMAS 21: 20th International Conference on Autonomous Agents and Multiagent Systems, May 3-7, 2021, Virtual, United Kingdom, pp. 279-287. ACM, 2021.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), 3rd Anmual Conference on Robot Learning, CoRL. 2019, October 30 November 1, 2019, Osaka, Japan, volume 100 of Proceedings of Machine Learning Research, pp. 66-75. PMIR, 2019.
Zi-Xuan Chen, Xin-Qiang Cai, Yuan Jiang, and Zhi-Hua Zhou. Anomaly guided policy learning from imperfect demonstrations. In Piotr Faliszewski, Viviana Mascardi, Catherine Pelachaud. and Matthew E. Taylor (eds.), 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, May 9-13, 2022, Auckland, New Zealand, pp. 244-252. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In Ronald Ortner, Hans Ulrich Simon, and Sandra Zilles (eds.), Algorithmic Learning Theory 27th International Conference, ALT 2016, October 19-21, 2016, Bari, Italy, volume 9925 of Lecture Notes in Com- puter Science, pp. 67-82, 2016.
Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, and Peter Stone. An imitation from observation approach to transfer learning with dynamics mismatch. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines, 2017.
10

---

Published as a conference paper at ICLR 2023
Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking importance weighting for deep learning under distribution shift. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tice Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Justin Fu, Katie Luo, and Sergey Levine Learning robust rewards with adverserial inverse rein- forcement learning. In 6th International Conference on Learning Representations, ICLR 2018, April 30 May 3, 2018, Vancouver, BC, Canada. OpenReview.net, 2018.
Tanmay Gangwani, Joel Lehman, Qiang Liu, and Jian Peng- Learning belief representations for imitation learning in pomdps. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, volume 115 of Proceedings of Machine Learning Research, pp. 1061-1071. AUAI Press, 2019.
Yonatan Geifman and Ran EI- Yaniv. Selectivenet: deep neural network with an integrated reject option. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Inter- national Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2151-21555 PMLR, 2019.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neu- ral Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, NIPS 2016, December 5-10, 2016, Barcelona, Spain, pp. 4565-4573, 2016.
Bo-Jian Hou, Lijun Zhang, and Zhi-Hua Zhou Learning with feature evolvable streams. In Is- abelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1417-1427, 2017.
Chenping Hou and Zhi-Hua Zhou One-pass learning with incremental and decremental features. IEEE Trans. Pattern Anal. Mach. Intell.,
2792, 2018.
Sankaran Iyer, Alan Blair, Laughlin Dawes, Daniel Aaron Moses, Christopher White, and Arcot Sowmya. Organ localisation using supervised and semi supervised approaches combining rein- forcement learning with imitation learning. CoRR, abs/21 1ass/2122.0326.. 2021.
Shengyi Jiang, Jing-Cheng Pang, and Yang Yu. Offline imitation learning with a misspecified sim- ulator. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, volume 33, pp. 8510 5510-8520. Curran Associates, Inc., 2020.
Kun Ho Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Cross domain imita- tion learning. CoRR, abs/1910.00100 2019.
Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imita- tion learning. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual, volume 1 119, pp. 5286-5295. PMLR, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. CoRR, abs/2005.01643. 2020.
Timothy Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, May 2-4, 2016, San Juan, Puerio Rico, 2016.
Yuxuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learn- ing to imitate behaviors from video via context translation. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, May 21-25, 2018, Brisbane, Australia, pp- 1118-1125. IEEF, 2018
11

---

Published as a conference paper at ICLR 2023
Donald Michie, Michael Bain, and J Hayes- Hayes-Miches, Cognitive models from subcognitive skills. IEE control engineering series, 44:71-99, 1990.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In Pro- ceedings of the 34th International Conference on Machine Learning, ICML. 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2681- 2690. PMLR, 2017.
Dripta S. Raychaudhuri, Sujoy Paul, Jeroen van Baar, and Amit K. Roy-Chowdhury, coms-domain imitation from observations. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual, volume 139 of Proceedings of Machine Learning Research, pp.
PMLR, 2021.
Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011 April 11-13, 2011, Fort Laud- erdale, USA, volume 15 of JMLR Proceedings, pp. 627-635. JMLR.or.. .rg. 2011.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Austria, May 3-7, 2021, Virtual. OpenReview.net, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1700.006447 2017.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE Inter- national Conference Robotics and Automation, ICRA 2018, May 21-25, 2018, Brisbane, Aus- tralia, pp. 1144-1141, IEEE, 2018
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Bradly C. Stadie, Pieter Abbeel, and Ilya Sutskever. Third person imitation learning. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, April 24-26, 2017, Toulon, France. OpenReview.net, 2017.
Jakub Sygnowski and Henryk Michalewski. Learning from the memory of atari 2600. In Tristan Cazenave, Mark H. M. Winands, Stefan Edelkamp, Stephan Schiffel, Michael Thielscher, and Julian Togelius (eds.), Computer Games 5th Workshop on Computer Games, CGW 2016, and 5th Workshop on General Intelligence in Game-Playing Agents, GIGA 2016, Held in Conjunction with the 25th International Conference on Artificial Intelligence, IJCAI 2016, July 9-10. 2016, New York Ciry, NY, USA, volume 705 of Communications in Computer and Information Science, pp. 71-85, 2016.
Andrea Tirinzoni, Andrea Sessa, Matteo Pirotta, and Marcello Restelli. Importance weighted trans- fer of samples in reinforcement learning. In Jennifer G. Dy and Andreas Krause (eds.), Pro- ceedings of the 35th International Conference on Machine Learning. ICML 2018, July 10-15, 2018, Stockholmsmässan, Stockholm, Sweden, volume 80 of Proceedings of Machine Learning Research, pp. 4943 4952. PMLR, 2018
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based con- trol. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, October 7-12, 2012, Vilamoura, Algarve, Portugal, pp. 5026-5033, 2012.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605, 2008.
12

---

Published as a conference paper at ICLR 2023
Linjing Wang, Tianlan Mo, Xuetao Wang, Wentao Chen, Qiang He, Xin Li, Shuxu Zhang, Ruimeng Yang, Jialiang Wu, Xuejun Gu. Jun Wei, Peiliang Xie, Linghong Zhou, and Xin Zhen. A hier- archical fusion framework integrate homogeneous and heterogeneous classifiers for medical decision-making. Knowledge-Basel Systems, 212: 2021a.
Ruohan Wang, Carlo Ciliberto, Pierluigi Vito Amadori, and Yiannis Demiris. Random expert distil- lation: Imitation learning via expert policy support estimation. In Kamalika Chaudhuri and Rus- lan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICM1.2015 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 33544.44.. PMLR, 2019.
Yunke Wang, Chang Xu, and Bo Du. Robust adversarial imitation learning via caaptively-sclectee demonstrations. In Zhi-Hor thi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Confer- ence on Artificial Intelligence, IJCAI 2021, 19-27 August 2021, Virtual Montreal, Canada, pp- 3155-3161. ijcai.org, 2021b.
Yunke Wang. Chang Xu, Bo Du and Honglak Lee. Learning to weight imperfect demonstrations. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual, volume 139 of Proceedings of Machine Learning Research, pp. 10961-10970. PMLR, 2021c
Yunke Wang. Bo Du, and Chang Xu. Unlabeled imperfect demonstrations adversarial imitation learning. CoRR, 062/22 2023.
Andrew Warrington, Jonathan Wilder Lavington, Adam Šcibior, Mark Schmidt, and Frank Wood. Robust asymmetric learning in pomdps. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual, volume 139 of Proceedings of Machine Learning Research, pp. 11013-11023. PMLR, 2021.
Meng WU, Ying Zhu, Qilian Yu, Bhargav Rajendra, Yunqi Zhao, Navid Aghdaie, and Kazi A. Za- man. A recommender system for heterogeneous and time sensitive environment. In Toine Bogers, Alan Said, Peter Brusilovsky, and Domonkos Tikk (eds.), Proceedings of the 13th ACM Confer- ence on Recommender Systems, RecSys 2019, September 16-20, 2019, Copenhagen, Denmark, pp. 210-218. ACM, 2019.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, pp. 15737- 15749, 2020.
Chao Yu, Jiming Liu, and Shamim Nemati. Reinforcement learning in healthcare: A survey. CORR, abs/1990.08796, 2019.
Zhenyu Zhang, Peng Zhao, Yuan Jiang, and Zhi-Hua Zhou. Learning with feature and distribution evolvable streams. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 11317-11327. PMLR, 2020.
Zhi-Hun Hui Zhou. Open-environment machine learning, National Science Review, (8): 123, 2022.
13

---

Published as a conference paper at ICLR 2023
A NOTATIONS
The notations of the main paper are gathered in Table
Table 2: The notations of the main paper.
Notation
Meaning
S
State space
Action space
O OE
Observation space
Observation space of the expert's view
OL P
Observation space of the learner's view
Transition Probability
Discounted factor Expert policy under O Auxiliary policy under OF Target policy under Ol Mapping function S - Oz
TE
Th
fE
fL
Mapping function S - Or.
THE
Trajectory sampled by TE under OF (demonstrations)
T
Trajectory sampled by л under OF Trajectory sampled by π under OE Trajectory sampled by W1 under Or Trajectory sampled by π2 under Or. An instance of state-action pair An instance of observation-action pair under OE An instance of observation-action pair under OL Occupancy measure of the expert policy TE Occupancy measure of the auxiliary policy π Occupancy measure of the target policy πτ
T
T m
x2
Per
Pri
Pr2
D w
Adversarial model Tmp and T Adversarial model on T m and T x2 Importance weighting factor Latent demonstration
22
H
O
Observed demonstration
N
Non-expert data
91
rejection model under OB rejection model under Ol
g2
14

---

Published as a conference paper at ICLR 2023
B ALGORITHM
The pseudo codes of our algorithm are illustrated in Algorithms I and
Algorithm IWRE Pretraining
Input: Auxiliary policy лт Expert demonstrations Tag Output: Evolving data (Ta): T>, }; Discriminator Dmn Rejection model gi. 1: function P.RRE.RETRAININNGT
2: Sample the evolving data (T T Pri by w1.
Train Dwy and 91 by Equation (5) and (17) respectively using Txe and Tmy
4: return T n2 D1
5: end function
Algorithm IWRE. Training
Input: Expert demonstrations Txg Evolving data Tx Discriminator Dwy Rejection model g1- Output: Target policy T2.
1: function IWRE. TRAININNN T (
2: Initialize и2, Dwy and 92.
3: for each step t do
4: Sample TT ~ Pra by m2.
5: for each mini-batch {T>z and {T from T xo and T do
6:
Update T2 by RL algorithms (such as PPO (Schulman et al. 2017)) using instances and pseudo rewards { log Dw,, (Inz))-
[mm²}
7:
Update Dwy by Equation (9) using negative instances (Tmp) and positive ones
{Iml J.
8:
if (I(D us (Tx,)), }), (2
then Query the Og observation of Tmy. i.c., Tr.. through OC operation.
9:
10:
Update Dw, and 92 by Equation (17) using the instance F and the correspond-
ing label (I(Du,, (T.,)), )), 91 (T,,))-
end if
12:
end for
end for
14:
return 22
end function
C DEFINITIONS
The challenges of HOIL, i.e., dynamics mismatch and support mismatch, illustrated as below.
Definition 5 (Dynamics Mismatch), The dynamics mismatch between the demonstrations and the initial data denotes the situation that:
Png TE(alo ()
=| #
(20)
Pri T1 (a(o) PPr(s= =
Definition (Support Mismatch). The support mismatch between the demonstrations and the initial data denotes the situation that:
supp(prg) suppp(p,) = fa E x Alp (x) #0) E Alp (④) +
(21)
15

---

Published as a conference paper at ICLR 2023
Table 3: Environmental summary of the tasks.
Environment
Ohservation Space OF
Observation Space Or.
Expert Rewards
Qhert
4750.00 = 50.51
ChopperCommanl
84 x 84 x 4(image)
128tunsigned int)
3135.00 + 145.86
Kangaroo
4175.00 = 94.21
Hopper
8
709.96 1 75.54
Humanoid
539.20 + 26.26
Reacher
5
i 0.54
Swimmer
5
52.24 = 1.29
Walker2d
188
188
929.97 + 24.09
Table 4: Comparisons between all contenders and IWRE in HOIL.
Algorithm
Considering beterogencous observations
Being able query
Not requiring Og all along
GAIL
GAIL-Rand
X
D DETAILED SETUP FOR THE EXPERIMENTS
Environment and Contenders. For the environments that used in the main body of the paper:
1. Pixel-memory Atari games. Og: 84 84 raw pixels; OL: 12tebyte random access memories (RAM). Expert: converged DQN-based agents (Mnih et al. 2013). Atari games contain two totally isolated views: raw pixels and RAM, under the same state. Through these environments, we want to investigate whether the agent can learn effective pol- icy from demonstrations under completely different observation spaces. Moreover, IL with visual observations only is already very difficult (Cai et al. 2021). while learning a Rased policy can be even more challenging (Bellemare et al. [2013] Sygnowski & Michalewski 2016). few IL research reported desirable results this task.
2. Continuous control MuJoCo objects. Og: half of original observation features; OL: another half of original observation features. Expert: converged DDPG-based agents (Lil-) licrap et al.) 2016). The features of MuloCo contain monotonous information like the direction, position, velocity, etc., of an object. Here we want to investigate whether the agent can learn from demonstrations with complementary signals under observations with missing information. Mcanwhile, we make sure RL algorithms can obtain comparable performances under Og and OL.
The details of the environments are reported in Table 3 Also, the detailed comparisons of the contenders (both in the main paper and the appendix) and IWRE are gathered in Table
Learning process. To simulate the situation that Og is costly, the steps for training was set as 1/4 of that for training π2, using GAIL (Ho Ermon 2owr (Cai al.) 202T under the OE space for MuJoCo/Atai Atari environments. The learning steps were 107 for MuJoCo and 5 106 for Atari environments.
the pretraining, we sampled 20 trajectories from Ti and the data from each trajectory had both Og and OL observations. In the training, each method learned 4 x 10" steps for MuJoCo and x 107 steps for Atari under the Or space to obtain 72.
Query Strategy. For TPIL and GAMA, if the output of the domain invariant discriminator is larger than 0.5, which means the encoder fails to generate proper features to confuse its discriminator, then would query Og of this data update the encoder. For IWRE, the threshold of the rejection model and the discriminator D was also 0.5, which means that 92(x) > 0.5 meanwhile Dung (x)
0.5, OF of this data would be queried. Dang 22, and the encoder (for TPIL/GAMA)
16

---

Published as a conference paper at ICLR 2023
pretrained for 100 epochs for all methods using evolving data during pretraining. The basic RL algorithm is PPO, and the reward signals of all methods were normalized into [0. 1] to enhance the performance of RL (Dhariwal 5000. Each time the buffer is full, the encoder and the rejection model will be updated for 4 epochs; also LBC will update л2 for 100 epochs with batch size 256 using the cross-entropy loss for Atari and the mean-square loss for MuJoCo. We set all architectures of the policy part the same as Dhariwal al. (2017). Besides, the hyper-parameters of the discriminator for all methods were the same: The rejection model and discriminator were updated using Adam with a decayed learning rate of x 10 +; the batch size was 256. The ratio of update frequency between the learner and discriminator was 3: 1. The target coverage e in Equation (17) was set as 0.8. À in Equation (17) was 1.0.
al. 2017). The buffer size for TPIL and IWRE was set as
update frequency, and network
E RL PERFORMANCE UNDER THE DIVISIONS OF MUJoCo
Here report the performance under the division of Og and O in MuJoCo. The details of the division are reported in Table We use DDPG-based (Lillicrap et al. 2016) agent with 102 training steps and repeat 10 times with different random seeds. The results are shown in Figure We can that the agent can obtain comparable performance under Og and OL. So for MuloCo environments, the fairness of the division in HOIL can be guaranteed, and Og is not more or less privileged than OL.
Table 5: The observation division into OE and OL in MuJoCo. The numbers denote the randomly selected observation indexes in the corresponding MuloCo environment on OpenAl Gym (Brock- man et al. 2016) platform.
OE
O2
Walker2d Swimmer Reacher Hopper
[S. 7,8, 8. 10, 11, 14, 15, 16]
[0. 2, 12, 13] [1.2.4.5]
[0. 6, |0, 7, 8, 10]
5,
3, 10]
[0, 21 4,
[2. 5,
10,
11. 12, 13, 16, 18, 19, 22,
[0. 4.
9,
14.
15,
20,
21,
24,
26,
27.
23. 25,
31,
34,
44 45,
28,
33,
35,
38.
39,
42,
46,
50,
52, 53,
47, 48
51,
54,
56,
61,
63,
66 67.
55, 58,
59,
62,
64,
70,
72,
73, 74,
68. 77.
78,
82,
86
87,
89
90
93,
95.
97,
75. 76.
79, 80,
81.
83,
84,
85,
88,
91.
92,
96
98. 99, 102,
103.
108. 110, 112, 113.
117. 119.
100, 101. 104. 105, 106,
107,
109
111. 114. 115.
120, 121, 122, 123,
124, 126,
127, 128, 133, 135,
116, 118, 125, 129, 130,
131, 132,
134,
136,
137,
144, 146, 147, 148, 151, 152
153, 158, 160, 161,
138, 139, 140, 141, 142, 143,
145,
149,
150,
154.
162. 166, 167, 170, 171. 173. 174, 176, 177. 178.
155, 156. 157. 159, 163, 164,
165, 168. 169, 172.
Humanoid
180, 181, 184, 185, 187,
191, 194, 198, 199,
175, 179, 182, 183,
189,
190, 192, 193, 195.
200, 201, 202, 207. 208.
209.
210, 211. 212,
214.
196, 197, 203 204, 205 206
213, 216
217. 218.
215, 219, 223, 227, 228, 229,
231, 232, 233, 234,
220, 221, 222, 224, 225, 226
230,
235.
240.
236, 237, 238, 242,
244,
246 248, 251, 253, 257,
241, 243, 245, 247, 249, 250, 252,
254,
255, 256.
258, 259, 260, 262 264.
265. 267, 268 271 272.
261, 263, 266, 269, 270, 274, 276 277.
283,
273, 275, 278, 279, 280, 281, 285, 287, 289, 290,
284, 286, 288, 292, 295, 297, 298, 300,
301, 303,
291, 293, 294, 296, 299, 302 304, 305, 306, 307,
309, 310, 314, 317, 318, 320, 321,
324,
308. 311, 312, 313. 315. 316. 319, 322, 326, 328.
327, 330, 331, 333, 334, 335, 336, 338, 340.
340.
329, 332, 337, 342,
343, 344, 345,
349, 358, 361.
341, 346, 347, 348, 350, 351, 352, 353, 354, 355,
362, 364, 365, 366, 368, 370, 372,
373, 375]
356, 357, 359, 360, 363, 367, 369, 371,
F ESTIMATION OF H, O, AND N BY I[Duz192
To investigate the ability of IWRE to distinguish the areas of latent demonstrations H. observed demonstrations O, and non-expert data N during policy learning, we recorded the accuracy and estimated ratio of each part on Hopper and Walker2d. The calculations of each curve are shown as below:
Accuracy H E. [1[2w, (i.)l9 (ā.) == 16& ==
(22)
D [T[Des )9
m
Ee [I[D (3 91 (ā.) = (F) = 0}
Accuracy 0
(23)
Em {1[ (3.))91 =
17

---

Published as a conference paper at ICLR 2023
Obsrrvation
Obeeration
Oburvaaton
(a) Reacher
(b) Hopper
(c) Humanoid
Obearvatten
Oburraaton Squces
(d) Swimmer
(c) Walker2d
Figure 8: The performance of RL methods under the division of Og and OL in MuJoCo. The agent can obtain comparable performances under OF and O,,, so that we can make sure the faimess the experiment of HOIL in the main paper.
Ee (I[De, (z:)lg1
I&&II& (3.
11
Accuracy_N
(24)
E1 {I[ u1 (T. 1 (T:) = =
> m
{I wy (x. )1 (T:) = 1}
Ratio H
==
(25)
m
Ratio_O = Σm {T[D wi ( (T:) 0}
(26)
Ratio_N Eis1 (T[D,o (Tx) gn (Ta)
-1
(27)
m
in which [ā. ~ Pa2 denotes batch of data sampled by T2. The results are shown Figure The results depicted not only the accuracies of 10w22192 but also the changes of these three areas during the policy learning. We can see that the accuracies decrease first. While at the same time, the ratio of H will increase. This is because the successful detection of will decrease the estimated ratio of O and reduce the accuracy of I[Dw,g19.. With the help of query operations, the accuracy of [[[Dwg] 192 will gradually increase. Also, followed by the learning procedure of the policy т2, more and more H will be recognized as O, with less and less N. This is why in the following period, the ratios of H and N will decrease while that of O will increase. These results also verify that our algorithm IWRE can indeed detect H, O and N successfully as the learning process of the policy.
each area and the ratio of O will
18

---

Published as a conference paper at ICLR 2023
10-
Estimated Ratioos
m Cnmn
Accuracy
0.6
0.0
02-
20000 40003
50003
D0022
100000
:20003 240020 100020
20300 40200 60000 30000 100000 1200ce 140000 100000
Episodes
Episodes
(a) Accuracy on Hopper
(b) Ratio on Hopper
Estimated Rutios
0.6
0.0
20020 42000
B0000 Episodes
55000
100000
220000 140000
22000
40020
02000 Episodes
BAMGD 100020
120000 240030
Accuracy Walker2d
(d) Ratio Walker2d
Figure 9: The accuracies and ratios H, O, and calculated by T[D (3:)l91 and I[D.n mn (7. 1992(1)))))))) during policy learning.
G INVESTIGATIONS ON ChopperCommand PERFORMANCH
We observed some interesting phenomenons ChopperConmand performance. Here investi- gate the behavior of a random policy, the IWRE policy, and the expert policy, shown in Figure I0 On ChopperConmant black circle) to avoid attacks from enemy aircraft meanwhile shooting them. So the policy this environment be split into two parts:
the agent needs to imitate the expert to manipulate helicopter (noted by the
1. The movement of the helicopter;
2. The shoot of the helicopter.
While capturing the image-level semantic information with only RAM input is quite difficult. Even with RAM input, IWRE still successfully learned the expert's movement of the helicopter (see the rows of "IWRE 2e7 steps" and "Expert Policy" in Figure TO But it is much harder capture the shot bullet in the image, not to mention in the RAM. So IWRE policy did not shoot enemy aircraft successfully. However, the environmental reward of ChopperConmande number of enemy aircraft shot down, regardless of the distance the helicopter has flown. Therefore, in Figure 6al IWRE policy can obtain higher rewards than the random policy the beginning of the training (1e5 steps). Then the environmental reward fluctuates within specific range. On the other hand, IWRE indeed learned some expert's 's policy of manipulating the helicopter in view the trajectory shown in Figure [0] So we believe that only considering the reward value
only related the
evaluation
This interesting phenomenon also reveals that the reward function learned from IL can be very different from the environmental reward functions in RL tasks, since the IL reward can capture expert behaviors that are not reflected by the environmental reward signals. Figure 10 can also be used metric. We also provide the
insufficient to reflect the degree of imitation on ChopperCommane
19

---

Published as a conference paper at ICLR 2023
full videos of these four policies ChopperCommand in this anonymous link: https: www. dropbox. com/ sh. Ez? 7xpbt 4y8t Jumz httpa//accccaaaccccccccaaa. OOnha? ?al=0
Random
Policy
(RAM input)
IWRE 1e5 steps
(RAM Input)
IWRE 2e7 steps (RAM Input)
Expert Policy
(pixel imput)
Figure 10:
The sequence comparisons between a random policy, the IWRE policy with 1c5/2e7 steps, and the expert policy. The agent needs to manipulate a helicopter (noted by the black circle) to avoid attacks from enemy aircraft meanwhile shooting them. The timestamp for each column of images and seed for each environment is the same.
H QUERY EFFICIENCY
We also investigate whether our query strategy is efficient. To this end, we allocate the query budget, i... limiting the query ratio for each method. For TPIL, preferentially queries those data with low D output; for our method IWRE, it preferentially queries those data with high (Dwg) .22 output. Besides, since GAIL and IW cannot directly perform queries, we design a seletton strategy for them GAIL-Rand and IW-Rand: for each batch of data, randomly select data and input the Og observations of these data to Dw.- If Dw, (T) > 0.5, which means Dw, regard this data being belonging the expert demonstrations, then would label this data the expert data update Dwy- The results are depicted in Figure
We can observe that the random strategy does not always improve the performance of GAIL and IW. For GAIL-Rand, without importance weighting to calibrate the learning process the reward function, its performance become even queried information enhances the discrimination ability of reward function, making it even more impossible for the agent to obtain effective feedbacks; for IW-Rand, its performance better than GAIL-Rand on most environments, and is reinforced on Hopper, Reacher, and Walker2d, which further demonstrate that the query operation indeed necessary for HOIL problem, but still fails compared with our method; for TPIL, it is comparable with IW-Rand, however, its performance improvement very limited as the budget increases, and on Swimmer and Walker2d there even exist performance degradations, which suggests that its query strategy is very unstable; for GAMA, it has good start point, but the performance gain very limited while the budget increases; for our method, its performance is almost the same that of IW-Rand without query most environments. When it is allowed to query Og observation, our method outperforms other methods with a large gap, which shows that the query strategy of our method is indeed more efficient.
on Hopper, Swimmer, and Walker2d, because the
I IMITATION WITH DIFFERENT NUMBER OF EXPERT TRAJECTORIES
The performances of different numbers of expert trajectories of all contenders reported in Fig- ure [2] Each experiment conducted 5 trials with different random seeds. We can observe that even with very limited number of trajectories, algorithm achieves better performance than other algorithms in most environments.
20

---

Published as a conference paper at ICLR 2023
Bedact Radto
Badad Batk
Bedad Bate
(a) Reacher
(b) Hopper
Humanoid
Budqut
(d) Swimmer
(e) Walker2d
Figure 11: The final rewards of each method on MuJoCo with different budget ratios, where the shaded regions indicate the standard deviation. The red horizontal dotted line represents the averaged performance of the expert.
IFS.
#baperi Unjectetoo
#lager Unjesketo
Tnjesteko
(a) Reacher
(b) Hopper
(c) Humanoid
(d) Swimmer
(e) Walker2d
Figure 12: The learning curves each method in MuJoCo environments with different numbers of expert trajectories, where the shaded region indicates the standard deviation.
21


Reviews:
[{"review_id": "Xc1qXsPf5u", "reviewer": "Reviewer_w9cE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The bulk of existing work on imitation learning focuses on settings in which the expert demonstrator and learner operate under the **same observation space**; however, in many real-world settings, this isn’t realistic. The work formalizes the problem of heterogeneously observable imitation learning (HOIL) and present an algorithm — Importance Weighting with Rejection (IWRE) to address this.\n\nConcretely, given a set of expert demonstrations in a given observation space, the HOIL setting this work looks at decomposes learning into collecting a set of (suboptimal) “matching correspondence” data online, running a policy that allows one to correlate expert observations with learner observations (even when there’s no explicit overlap in observables — e.g., Atari ALE RAM state vs. visual features). The proposed IWRE framework then tries to address two key problems in this learning setup via an online IRL-like procedure (similar in nature to GAIL); first, the problem of optimality/dynamics mismatch — the “expert demonstrations” are assumed to be optimal, while the correspondence data is not; as a result, directly imitating from the union of both datasets would result in a bad policy. To get around this, similar to prior work in offline RL, this work proposes using importance sampling (the IW in IWRE) to correct for this mismatch, prioritizing the expert demonstrations.\n\nThe second problem IWRE addresses is that of a support mismatch; the “correspondence matching” data is again problematic because its suboptimality might mean that the corresponding data only sees a fraction of the state space. As a result, IWRE uses rejection sampling to drive the learner’s behavior policy towards the support of the actual expert observation distribution (exploiting a learned discriminator that partitions the expert demos and correspondence demos), therefore circumventing this issue.\n\nThe IWRE approach is evaluated on a set of (albeit somewhat synthetic) environments from the Atari Arcade Learning Environment (expert observations — frames, learner observations — RAM state), and Mujoco Continuous control (expert/learner see disjoint halves of the Mujoco observation space). Compared to traditional imitation learning baselines and reinforcement learning upper bounds (for ceiling results), the IWRE approach is strong, performant, and seems to generalize across environments.\n", "summary_of_the_review": "I believe this to be strong work, with an evaluation that demonstrates the viability of the proposed approach and difficulty of the problem setting. However, I’d love to see more ecologically viable/realistic evaluations, rather than the somewhat synthetic tasks studied in this work.\n", "strength_and_weaknesses": "I really like this approach, and the motivation behind the HOIL problem. I think it’s definitely true in many robotics settings that there is a mismatch in observation spaces between the expert and the learner, and addressing this is important.\n\nIWRE as an algorithm is also well-motivated, and follows naturally from a large body of prior work; combining importance weighting and rejection sampling in this setting is natural and novel, and clearly the results speak for themselves.\n\nThe one weakness I have of this work is with the evaluation; technically, to prove the viability of the algorithm, the existing “synthetic” observation splits in the Atari and Mujoco environments make sense, but I’d really love to see a more realistic evaluation where heterogeneous observations are actually ecologically viable — perhaps in settings in robotic manipulation for example.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TZrUNNqcGw", "reviewer": "Reviewer_j2UZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for imitation learning (IL) when the expert and agent operate from different observation spaces. The paper proposes a new algorithm for dealing with this setting, which relies on two components: an importance weighting correction which modifies the standard GAIL objective, and a rejection mechanism to address potential support mismatch between expert and policy distributions. The algorithm is evaluated on 5 Mujoco and 3 Atari environments.\n\nThe algorithm operates in two stages. First, GAIL is run in the expert's MDP to learn a decent, but not necessarily optimal policy. Then, the same policy is run in the expert and learner MDPs (this is assumed to be possible), giving a dataset of aligned trajectories. This is then used to compute the importance weighting correction factor. Finally, a classifier is trained to filter out the non-expert parts of the policy, which is used for the rejection sampling mechanism.", "summary_of_the_review": "Overall, I do not recommend acceptance for this paper due to:\n- the overly strong assumptions made by this work, which somewhat undermine the otherwise interesting setting they consider\n- the complexity of the algorithm            ", "strength_and_weaknesses": "# Strengths:\n- The goal of learning from demonstrations gathered in a different observation space is definitely interesting\n- To my knowledge, this combination of importance weighting and rejection sampling is novel.\n\n\n# Weaknesses:\n- I'm not convinced by the setup they're considering here. They make assumptions which seem unrealistic to me: in particular, they assume it is possible to interact with *both* the expert and agent MDPs to some extent. This is necessary for gathering what they call the evolving dataset, which is a set of trajectories where the observations in both MDPs are aligned. I find this to be a strong assumption, and one which other methods (e.g. [1]) do not make - they assume expert trajectories are given and only interaction with the learner's MDP is allowed.\n\nFor this reason, although I think the general goal of IL from heterogeneous observations is very interesting (for example, learning from YouTube videos where the perspective of the expert is different from that of the agent), this paper makes overly strong assumptions that one can gather trajectories from both MDPs. This strong assumption limits the significance and impact of this work. Also, the fact that they are only considering discriminator-based approaches and not others (e.g. non-adversarial ones such as RED [2] or DRIL [3]) limits the applicability.\n\nAlso, the algorithm proposed is fairly complicated: there are two different learning phases with GAIL subroutines as well as training a classifier on intermediate data. I believe the community would be more open to adopting a simpler, more end-to-end approach.\n\n[1] https://arxiv.org/pdf/1703.01703.pdf\n[2] http://proceedings.mlr.press/v97/wang19d/wang19d.pdf\n[3] https://openreview.net/pdf?id=rkgbYyHtwB", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Dfkbo7T65t", "reviewer": "Reviewer_ntXD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for imitation learning with heterogeneous observations. Specifically, the dynamics mismatch and the support mismatch are handled using importance weighting and rejection. Experiments show the effectiveness of the proposed IWRE method on both Atari and MuJoCo tasks.", "summary_of_the_review": "The paper proposes an effective and novel algorithm for handling both dynamics and support set mismatch in imitation learning. The experiments justify the claims that the proposed method helps with a better coverage of the expert data as well as the effectiveness in facing the heterogeneous observations. Some questions need to be answered well in the paragraph. Adding ablation studies will make it a better work.", "strength_and_weaknesses": "Strength:\n\nThe proposed method is explained with details. Both dynamics and support set mismatch are handled with the proposed novel algorithm. Experiments show evidence for the improved covering rate in expert data by the learned policy. Explanations for the experimental results are thorough.  \n\n\nWeakness:\n\nFor the t-SNE plot in Fig. 5, is it true that the support set of $\\pi_1$ also has some part outside the expert data? How is this part handled by the method? This does not seem to be captured by Fig. 4. \n\nAdding ablation studies with only importance weighting or the rejection will help to understand their effects better. However, no such ablation study is conducted in the current paper.\n\nThe fonts in the figures are too small to read.\n\nIn Fig. 6 (a), the ChopperCommand results are a bit weird. Is it too easy or too hard to learn? It seems the reward becomes large very quickly but decreases along the training. Why is this happening? Please explain more in the paragraph.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9oYaFtLtwCF", "reviewer": "Reviewer_N1Ya", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to solve a branch of imitation learning scenario that the expert’s and the learner’s feature (observation) spaces are different. This scenario is quite difficult compared with existing imitation learning work under the same observation space. Moreover, the related works relied on the expert’s and learner’s observations can be obtained during the whole learning process, which can be costly in many real-world applications. So this paper tried to solve this problem with limited observations coexistence.\n\nAt first, this paper formulated the learning problem as heterogeneously observable imitation learning (HOIL), where some expert demonstrations and an auxiliary policy were given under the expert’s observation space, and the target was to learn a policy under the learner’s observation space. Since these two spaces can be arbitrarily different, they needed the auxiliary policy to sample some data containing both two observations as the bridge between the expert demonstrations and the learned policy. Later on, they limited the number of observation coexistence to lower the cost.\n\nBy analyzing this problem, they found the underlying challenges, i.e., the dynamics mismatch and the support mismatch. These two mismatches appeared due to the imperfection of the auxiliary policy. To solve this problem, they proposed an algorithm of importance weighting and rejection (IWRE), where the IW solved the dynamics mismatch and the RE solved the support mismatch.\n\nTo evaluate the performance of IWRE, they did some experiments on MuJoCo and Atari benchmarks. Compared with the ablation, domain-adaptation imitation learning methods, and other heterogeneous observation methods, IWRE obtained the best performance among these environments.", "summary_of_the_review": "This paper studied a challenging but more realistic imitation learning scenario, in which the expert’s and the learner’s observations are heterogeneous. Also, the authors proposed a novel and reasonable method IWRE to solve this problem. The contributions of this work are sufficient. So I recommend this paper be accepted.\n\nAfter rebuttal:\n\nI have gone through the responses and discussions between the co-reviewers and the authors. I think the authors have clearly answered the questions of other reviewers, and thus believe this paper is a good submission. So I increased my score and stand for accepting it.", "strength_and_weaknesses": "Strengths：\n* This paper studied a challenging but more general learning scenario, with weaker assumptions than related works. Also, the HOIL problem is well-defined.\n\n* For the algorithm part, the analysis of the dynamics mismatch and the support mismatch is quite convincing, with clear mathematical formulations and visualization empirical studies. Meanwhile, the motivated IWRE makes sense.\n\nWeaknesses：\n\nThe notations of this work are somehow heavy in this paper. It would be better to simplify some notations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%