Published as a conference paper at ICLR 2023

S EEING D IFFERENTLY, ACTING S IMILARLY:
H ETEROGENEOUSLY O BSERVABLE I MITATION L EARN ING
Xin-Qiang Cai1,2 Yao-Xiang Ding3 Zi-Xuan Chen1 Yuan Jiang1 ∗Masashi Sugiyama4,2 Zhi-Hua Zhou1
1
National Key Laboratory for Novel Software Technology, Nanjing University
2
The University of Tokyo
3
State Key Laboratory for CAD&CG, Zhejiang University
4
RIKEN Center for Advanced Intelligence Project
cai@ms.k.u-tokyo.ac.jp,dingyx.gm@gmail.com,chenzx@lamda.nju.edu.cn,
jiangy@lamda.nju.edu.cn,sugi@k.u-tokyo.ac.jp,zhouzh@lamda.nju.edu.cn

A BSTRACTIn many real-world imitation learning tasks, the demonstrator and the learner haveto act under different observation spaces. This situation brings significant obstacles to existing imitation learning approaches, since most of them learn policies under homogeneous observation spaces. On the other hand, previous studies
under different observation spaces have strong assumptions that these two observation spaces coexist during the entire learning process. However, in reality,
the observation coexistence will be limited due to the high cost of acquiring expert observations. In this work, we study this challenging problem with limited
observation coexistence under heterogeneous observations: Heterogeneously Observable Imitation Learning (HOIL). We identify two underlying issues in HOIL:
the dynamics mismatch and the support mismatch, and further propose the Importance Weighting with REjection (IWRE) algorithm based on importance weighting
and learning with rejection to solve HOIL problems. Experimental results showthat IWRE can solve various HOIL tasks, including the challenging tasks of transforming the vision-based demonstrations to random access memory (RAM)-based
policies in the Atari domain, even with limited visual observations.

1

I NTRODUCTION

Imitation Learning (IL), which studies how to learn a good policy by imitating the given demonstrations (Xu et al., 2020; Chen et al., 2022), has made significant progress in real-world applications
such as autonomous driving (Chen et al., 2019), health care (Iyer et al., 2021), and continuous
control (Wang et al., 2023). In tradition, the expert and the learner are assumed to use the sameobservation space. However, nowadays, many real-world IL tasks demand to remove this assumption (Chen et al., 2019; Warrington et al., 2021), such as in autonomous driving (Chen et al., 2019),
recommendation system (Wu et al., 2019), and medical decision making (Wang et al., 2021a). Taking AI for medical diagnosis as an example in Figure 1: A medical AI is learning to make medical
decisions based on expert doctor demonstrations. To ensure demonstration quality, the expert may
use high-cost observations such as CT, MRI, and B-ultrasound. In contrast, the AI learner is ideal
to use only low-cost observations from cheaper devices, which could be newly designed ones thathave not been used previously by the expert. Meanwhile, to ensure reliability, it is also reasonable
to allow the learner to access the high-cost observations during training under a limited budget (Yuet al., 2019). The above examples share three characteristics: (i) Even though a pair of expert andlearner observations can be different, they are under the same state of the environment, leading tosimilar policies; (ii) The learner’s new observations are not available to the expert when generating demonstrations; (iii) During training, the learner can only access expert observations under alimited budget, in special the high-cost ones, since it is also important to minimize the usage of the∗Corresponding author.

1

Published as a conference paper at ICLR 2023

Expert demonstration

Learner

ActionActionFigure 1: Medical decision making: an example of the HOIL problem. Figures 1, 2, and 3 include
some illustrations and pictures from the Internet (source: https://www.flaticon.com/).
Table 1: Comparisons between different IL processes. OE and OL denote the observation spaces
for experts and learners respectively. N/A denotes not applicable.
Setting

DSIL

OE ̸= OL
The demonstrations do not include OL
The learner does not require OE +OL all the time
OE is not more privileged than OL

N/A
N/A
N/A

#

POIL

!
#
#
#

LBC

!
!
#
#

HOIL(ours)

!
!
!
!

expert observations during training to avoid unnecessary costs. We name such IL tasks Heterogeneously Observable Imitation Learning (HOIL). Among them, we focus on the most challenging
HOIL setting in which the expert and learner observation spaces have no overlap.
There are two lines of research studying the related problems, summarized in Table 1 and Figure 2.
The first one relates to Domain-shifted IL (DSIL): the observation spaces of experts and learners are the homogeneous, while some typical distribution mismatches could exist: morphological,
viewpoint, and dynamics mismatch (Stadie et al., 2017; Raychaudhuri et al., 2021). However, theapproaches for DSIL are invalid when the observation spaces are heterogeneous as in HOIL.
The second line studied IL under different observations similar to HOIL. Some representative works
include Partially Observable IL (POIL) (Gangwani et al., 2019; Warrington et al., 2021) and Learning by Cheating (LBC) (Chen et al., 2019). Both POIL and LBC assume that the learner can easily
access the expert observations without any budget limit. However, in practice, different from thelearner observations, the access to expert observations might be of high cost, invasive, and even
unavailable (Yu et al., 2019), which hinder the wide application of these methods.
In this paper, we initialize the study of the HOIL problem. We propose a learning process acrossobservation spaces of experts and learners to solve this problem. We further analyze the underlying
issues of HOIL, i.e., the dynamics mismatch and the support mismatch. To tackle both two issues,
we use the techniques of importance weighting (Shimodaira, 2000; Fang et al., 2020) and learningwith rejection (Cortes et al., 2016; Geifman & El-Yaniv, 2019) for active querying to propose theImportance Weighting with REjection (IWRE) approach. We evaluate the effectiveness of the IWRE
algorithm in continuous control tasks of MuJoCo (Todorov et al., 2012), and the challenging tasksof learning random access memory (RAM)-based policies from vision-based expert demonstrationsin Atari (Bellemare et al., 2013) games. The results demonstrate that IWRE can significantly outperform existing IL algorithms in HOIL tasks, with limited access to expert observations.

2

R ELATED W ORK

FESL. Recently, to deal with the significant challenges in open environment learning (Zhou, 2022),
there are emerging studies of feature evolvable stream learning (FESL) (Hou et al., 2017; Hou &Zhou, 2018; Zhang et al., 2020), which are among major inspirations of our work. FESL focuses onsupervised learning with heterogeneous feature space. There are also significant differences between
existing FESL approaches and HOIL. In (Hou et al., 2017; Hou & Zhou, 2018), they assume thatdata features are generated from heterogeneous but fixed and static data distributions, while in HOIL,
the data distributions are dynamically changing. FESL with dynamically changing distributions isstudied in (Zhang et al., 2020), while they assume that the changes are passive to the learner. This isdifferent from HOIL, in which the data distribution changes are actively decided by the learner due
to the nature of decision-making learning.
DSIL. For the standard IL process, where the learner and the expert share the same observation
space, current state-of-the-art methods tend to learn the policy in an adversarial style (Wang et al.,2021b;c), like Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016). When considering the domain mismatch problem, i.e., DSIL, the research aims at addressing the static distributional shift of the optimal policies resulted from the environmental differences but still underhomogeneous observation spaces. Stadie et al. (2017), Sermanet et al. (2018), and Liu et al. (2018)2

Published as a conference paper at ICLR 2023

Observation Space

0.1
0.2

Learning
Process

0.5
0.2
0.6
0.8
0.9

0.2
0.6

0.7
0.4
0.3
0.1
0.5

(a) DSIL

……
……

……
……
……
……

……

……

0.1
0.3
0.7

Observation Space

Observation Space

0.5
0.7

0.1
0.2

0.2
0.6

……
……

0.2

0.4

……

0.5

Learning
Process

0.6

0.8

0.4

0.9

0.9

0.7

0.3

0.1
0.5

……

0.

……

0.

……

……

……

0.
0.
0.
0.

0.5
0.7

……
……

0.3

……

0.1
0.7
0.4
0.9

……

𝒯𝒯�𝜋𝜋2……

Learning
Process

……

0.1
0.2

0.2
0.6

……
……

0.5
0.7

0.2

0.4

……

0.3

0.5
0.6
0.8

……

……

0.9

(b) POIL

0.7
0.3
0.1
0.5

……
……
……

……

……

0.1

0.2

0.3

0.7

0.1

0.9

0.4
0.9

(c) LBC

0.4
0.3
0.7

0.7
0.6
0.2

Observation Space……
……

𝒯𝒯�𝜋𝜋2

……
……

……

……

0.8
0.6
0.3
0.2
0.1

0.1
0.2

Learning
Process

0.5

0.8

0.2
0.6

0.7

0.1

……
……

……

……

0.5
0.7

0.1

0.4

0.2

0.3

……

0.8

0.1

0.9

……

0.3

0.4
0.3
0.7

0.7
0.6
0.2

……
……

……

……

0.6
0.2
0.1

(d) HOIL (our work)

Figure 2: Comparison sketches of IL with respect to observation spaces. The target of (a) is to learna policy based on the same observations of the expert, while that of (b), (c), and (d) are to learn a
policy based on the second observation space only. The detailed differences can be found in Table 1.
studied the situation where the demonstrations are in view of a third person. Kim et al. (2020) andKim et al. (2019) addressed the IL problem with morphological mismatch between the expert’s andlearner’s environments. Tirinzoni et al. (2018), Jiang et al. (2020), and Desai et al. (2020) focused on
the calibration for the mismatch between simulators and the real world through some transfer learning styles. There are two major differences between HOIL and DSIL: One is that HOIL considers
heterogeneous observation spaces instead of homogeneous ones; another is that without observation
heterogeneity, DSIL can directly align two fixed domains, which may not be realistic for solving HOIL when two observation spaces are totally different. Thus HOIL is a significantly morechallenging problem than DSIL. Besides, Chen et al. (2019) learned a vision-based agent from a
privileged expert. But it can obtain expert observations throughout the whole learning process, so itcannot handle the problem of the support mismatch under HOIL.
POMDP. The problem of Partially Observable Markov Decision Process (POMDPs), in which
only partial observations are available for the agent(s), has been studied in the context of multiagent (Omidshafiei et al., 2017; Warrington et al., 2021) and IL (Gangwani et al., 2019; Warrington
et al., 2021) problems. But distinct from HOIL, in a POMDP, the learner only has partial observations and shares the same underlying observation space with the expert, which would become
an obstacle for them to make decisions correctly. For example, Warrington et al. (2021) assumed
that the observation of the learner is partial than that of the expert. Instead, in HOIL, expert’s andlearner’s observations are totally different from each other, while the learner’s observations do notbelong to the expert’s. For HOIL, the main challenge is to deal with the mismatches between theobservation spaces, especially when the access to expert observations is strictly limited.

3

T HE HOIL P ROBLEM

In this section, we first give a formal definition of the HOIL setting, and then introduce the learning
process for solving the HOIL problem.
3.1

S ETTING D EFINITIONA HOIL problem is defined within a Markov decision process with mutiple observation spaces, i.e.,⟨S, {O}, A, P, γ⟩, where S denotes the state space, {O} denotes a set of observation spaces, A
denotes the action space, P : S × A × S → R denotes the transition probability distribution of
the state and action, and γ ∈ (0, 1] denotes the discount factor. Furthermore, a policy π over anobservation space O is defined as a function mapping from O to A, and we denote by ΠO the setof all policies over O. In HOIL, both the expert and the learner have their own observation spaces,
which are denoted as OE and OL respectively. Both OE and OL are assumed to be produced by
two bijective mappings fE : S → OE , fL : S → OL , which are unknown functions mapping theunderlying true states to the observations. It is obvious to see that by this assumption, any policy over
OE has a unique correspondence over OL . This makes HOIL possible since the target of HOIL is
to find the policy corresponding to the expert policy under OL .
A state-action pair (s, a), denoted by x, is called an instance. Also, a trajectory T = {xi }, i :={1, . . . , m} is a set of m instances. For each observation space, xe ∈ Te ⊆ OE ×A and x ∈ T ⊆ OL ×
A, where OE = fE (S) and OL = fL (S). Furthermore, we define the occupancy measure
P∞ of a policy
π under the state space S as ρπ : S × A → R such that ρπ (x) = π(a|o)Pr(o|s) t=0 γ t Pr(st =s|π). Under HOIL, the learner accesses the expert demonstrations TeπE , a set of instances sampledfrom ρπE . The goal of HOIL is to learn a policy π̂ as the corresponding policy of πE over OL . If
OE = OL , HOIL degenerates to standard IL . GAIL (Ho & Ermon, 2016) is one of the state-of-theart IL approaches under this situation, which tries to minimize the divergence between the learner’s
3

Published as a conference paper at ICLR 2023

Observation Space

Observation Space

Learning
Process𝒯𝒯�𝜋𝜋𝐸𝐸𝒯𝒯�𝜋𝜋1

Pretraining

Training?

𝒯𝒯�𝜋𝜋2

Observation Coexistence

Observation Space𝒯𝒯�𝜋𝜋1𝒯𝒯�𝜋𝜋2𝜋𝜋2Figure 3: Illustration of a learning process across two different observation spaces for solving HOIL.
π1 is an auxiliary policy that is additionally provided.
and the expert’s occupancy measures d(ρπ̂ , ρπE ). The objective of GAIL is
min max Ex∼ρπE [log Dw (ex)] + Ex∼ρπ̂ [log(1 − Dw (ex))] − H(π̂),
π̂

w

(1)

where H(π̂) is the causal entropy performed as a regularization term, and Dw : OE × A → [0, 1] isthe discriminator of πE and π̂. GAIL solved Equation (1) by alternatively taking a gradient ascent
step to train the discriminator Dw , and a minimization step to learn policy π̂ based on an off-the-shelfreinforcement learning (RL) algorithm with the pseudo reward − log Dw (ex).
3.2

T HE L EARNING P ROCESS FOR S OLVING HOIL

In HOIL, we need to cope with the absence of the learner’s observations in demonstrations and thehigh cost of collecting expert observations while learning. So we introduce a learning process withpretraining across two different observation spaces for solving HOIL, as abstracted in Figure 3.
Pretraining. Same as LBC (Chen et al., 2019), we assume that we can obtain an auxiliary policy
π1 based on OE at the beginning. π1 can be directly provided by any sources, or trained by GAIL or
behavior cloning (Michie et al., 1990) from online data stream (Cai et al., 2019) or offline demonstrations (Sasaki & Yamashina, 2021) as did in LBC. Besides, we use this π1 to sample some data
Tπ1 , which contain both observation under OE (i.e., Teπ1 ) and OL (i.e., T π1 ), in order to connect
these two different observation spaces. We name Tπ1 = {Teπ1 , T π1 } the initial data.
Training. Here we learn a policy π2 from the initial data T π1 and the collected data T π2 , under
OL only. Besides, the learner is allowed for some operation of observation coexistence (OC): At
some steps of learning, besides the observations OL , the learner could also request Teπ2 from thecorresponding observations OE (e.g., from the human-understandable sensors). The final objectiveof HOIL is to learn a good policy π2 under OL .
In practical applications, the auxiliary policy π1 can also come from simulation training or direct
imitation. But since π1 is additionally provided, it is more practical to consider π1 as a non-optimal
policy. During training, OC is an essential operation for solving HOIL, which helps the learneraddress the issues of the dynamics mismatch and the support mismatch (especially the latter one).
Also, in reality, we do not need an oracle for actions, which still needs OC for obtaining expert
observations first, as in many active querying research (Brantley et al., 2020; Chen et al., 2019), so
its cost will be relatively lower.
Besides, the related work (Chen et al., 2019) also required an initialized policy π1 to solve theirproblem, which acts as a teacher under privileged OE in the pretraining and then learned a visionbased student from the guidance of the teacher under both OL and OE . Their setting can be viewed
as a variety of HOIL with optimal π1 , unlimited OE , and unlimited OC operations, so HOIL is
actually a more practical learning framework.

4

I MITATION L EARNING WITH I MPORTANCE W EIGHTING AND R EJECTIONIn HOIL, the access frequency to OE is strictly limited, so it is unrealistic to learn π2 in a Dataset
Aggregation (DAgger) style (Ross et al., 2011) as in LBC. Therefore, we resort to learning π2 witha learned reward function by inverse RL (Abbeel & Ng, 2010) in an adversarial learning style (Ho& Ermon, 2016; Fu et al., 2018).
4

Published as a conference paper at ICLR 2023

𝑯𝑯(a) The ideal situation

𝑶𝑶

+1

𝑵𝑵

(b) The real situation0-1(c) The output of ⟨I(Dw ), g⟩

Figure 4: The comparisons among the distributions of expert demonstrations ρπE , initial data ρπ1 ,and non-expert data ρπNE . The red and blue regions denote the expert and non-expert parts of ρπ1
respectively. H, O, and N denote the latent demonstration, the observed demonstration, and thenon-expert data respectively. (a) The ideal situation, where supp(ρπE ) \ supp(ρπ1 ) = ∅; (b) The
real situation, where H := supp(ρπE )\supp(ρπ1 ) ̸= ∅ in ρπE ; (c) The target output of the combined∗ ∗model I[Dw]g . The output +1, 0, and −1 regions correspond to H, O, and N respectively.
In addition, both OE and OL are assumed to share the same latent state space S as introducedin Section 3.1, so the following analysis will be based on S, while the algorithm will handle theproblem based on OE and OL specifically.
4.1

DYNAMICS M ISMATCH AND I MPORTANCE W EIGHTING

To analyze the learning process, we let ρπE , ρπ1 , and ρπ2 be the occupancy measure distributionsof the expert demonstrations, the initial data, and the data during training respectively. Since weneed to consider the sub-optimality of π1 , ρπ1 should be a mixture distribution of the expert ρπE andnon-expert ρπNE , i.e., there exists some δ ∈ (0, 1) such thatρπ1 = δρπE + (1 − δ)ρπNE ,(2)

as depicted in Figure 4a. During training, the original objective of π2 is to imitate πE throughdemonstrations. To this end, the original objective of reward function Dw2 for π2 is to optimize
max Ex∼ρπ2 [log Dw2 (x)] + Ex∼ρπE [log(1 − Dw2 (x))].
w2(3)

But the expert demonstrations are only available under OE . During training, we can only utilize theinitial data T π1 ∼ ρπ1 to learn π2 and Dw2 . Besides, as π1 is sub-optimal, directly imitating T π1
could reduce the performance of the optimal π2 to that of π1 . So we use the importance weighting
to calibrate this dynamics mismatch, i.e.,max L(Dw2 ) = Ex∼ρπ2 [log Dw2 (x)] + Ex∼ρπ1 [α(x) log(1 − Dw2 (x))],
w2
ρ

(4)

(x)

where α(x) ≜ ρππE (x) is an importance weighting factor (Fang et al., 2020). So the current issue lies1

ρin how to estimate ρππE under OE . To achieve this purpose, we need to bridge the expert demonstra1
tions and the initial data. Therefore, here we use these two data sets to train an adversarial model
Dw1 in the same way as Dw2 in the pretraining:max L(Dw1 ) ≜ Ex∼ρπ1 [log Dw1 (ex)] + Ex∼ρπE [log(1 − Dw1 (e
x))].
w1

If we write the training criterion (5) in the form of integral, i.e.,Z
max L(Dw1 ) = [ρπ1 log Dw1 + ρπE log(1 − Dw1 )]dx,
w1

(5)

(6)

x

∂L
then, by setting the derivative of the objective to 0 ( ∂D= 0), we can obtain the optimum Dw1 :
w
1

ρπ 1
∗
Dw
=
,
1
ρπ1 + ρπE5(7)Published as a conference paper at ICLR 2023

in which the order of differentiation and integration was changed by the Leibniz rule. Besides, we
can sufficiently train Dw1 using the initial data Teπ1 and the expert demonstrations TeπE . Then Dw1
will be good enough to estimate the importance weighting factor, i.e.,α(x) ≜∗
1 − Dw(ex)
ρπE
1 − Dw1 (e
x)
1
≈
=
.
∗
x)
ρπ 1
Dw1 (eDw1 (ex)(8)

In this way, we can use Dw1 , which can connect demonstrations and initial data, to calibrate thelearning process of Dw2 . The final optimization objective for Dw2 is
max L(Dw2 ) = Ex∼ρπ2 log Dw2 (x) + Ex∼ρπ1w2

1 − Dw1 (ex)
log[1 − Dw2 (x)].
Dw1 (ex)(9)

In this way, Dw2 can effectively dig out the expert part of ρπ1 and produce efficient rewards for π2 .
4.2

S UPPORT M ISMATCH

So far, the challenges have still been similar to homogeneously observable IL. However, our preliminary experiments demonstrated that mere importance weighting
is not enough to fix the problem that occurred by theabsence of interactions under OE . So there exist someother issues between the expert demonstrations and theinitial data. To find out the underlying issues, we plotted the t-distributed Stochastic Neighbor Embedding (t(a) Hopper(b) Walker2d
SNE) (van der Maaten & Hinton, 2008) visualizations ofthese two empirical distributions under OE on Hopperand Walker2d in Figure 5. Twenty trajectories were col- Figure 5: t-SNE visualizations of expert
lected for both the expert demonstrations and the initial demonstrations and collected data of π1data. We can observe that there exist some high-density under OE .
regions of demonstrations in which the initial data do notcover; that is, there exist some regions of the demonstrations that π1 did not explore. Wang et al.
(2019) found a similar phenomenon in the standard IL setting. On the other hand, the importanceρweighting α cannot calibrate this situation where ρππE = ∞.
Expertπ1

1

To formulate this problem, here we introduce the support set of the occupancy measure:
Definition 1 (Support Set). The support set of an occupancy measure ρ is the subset of the domain
containing the elements which are not mapped to zero:supp(ρ) := {x ∈ S × A|ρ(x) ̸= 0}.

(10)

Due to the sub-optimality of π1 , supp(ρπE ) \ supp(ρπ1 ) ̸= ∅ (see Figure 4b). We call this part thelatent demonstration, defined as:
Definition 2 (Latent Demonstration). The latent demonstration H is the set of the domain thatbelongs to the relative complement of supp(ρπ1 ) in supp(ρπE ):H := {x ∈ S × A|supp(ρπE ) \ supp(ρπ1 )}.

(11)

Also, another part of the demonstration is named the observed demonstration, defined as:
Definition 3 (Observed Demonstration). The observed demonstration O is the set of the domainthat belongs to the complement of H in supp(ρπE ):O := {x ∈ S × A|supp(ρπE ) ∩ supp(ρπ1 )}.

(12)

Besides, the data outside of demonstrations should be non-expert data:
Definition 4 (Non-Expert Data). The non-expert data N is the set of the domain out of supp(ρπE ):N := {x ∈ S × A|ρπE (x) = 0}.
6(13)

Published as a conference paper at ICLR 2023

In other words, the sub-optimality of π1 will cause not only the dynamics mismatch, but also theappearance of the latent demonstration H. We call the latter one the problem of support mismatch.
Intuitively, when π2 → πE , we have H → ∅, monotonously. So in order to fix the support mismatch
between ρπE and ρπ1 , guiding π2 to find out H is the key.
In addition, the support mismatch problem can be viewed as an inverse problem of the out-ofdistribution (OOD) problem that frequently occurred in offline RL setting (Levine et al., 2020), in
which they tried to avoid supp(ρπ1 ) \ supp(ρπE ) instead.
4.3

I MITATION L EARNING WITH R EJECTION

We can observe that H ∪O∪N = S ×A. So it is desirable to filter out H from O and N . Meanwhile,
Dw1 and Dw2 can only classify O ∪ H and N , under OE and OL respectively. Therefore, here we
design two models g1 : OE × A → {0, 1} and g2 : OL × A → {0, 1} (Output 0: x ∈ O and output
1: otherwise), so that given x ∼ T (corresponding xe ∼ Te and x ∼ T ), they can satisfy∗∗
H = {x ∈ S × A|I[Dw(ex)]g1∗ (ex) = I[Dw(x)]g2∗ (x) = +1},
1
2

(14)

∗
∗
O = {x ∈ S × A|I[Dw(ex)]g1∗ (ex) = I[Dw(x)]g2∗ (x) = 0},
1
2

(15)

∗
∗
N = {x ∈ S × A|I[Dw(ex)]g1∗ (ex) = I[Dw(x)]g2∗ (x) = −1},
1
2

(16)

∗
(x)]g ∗ (x) is depicted in Figure 4c.
where I[·] takes +1 if · > 0.5, and −1 otherwise. I[Dw

To this end, both g1 and g2 should be able to cover O, meanwhile g2 can be adaptive to the continuouschange of ρπ2 due to the update of π2 . Here we learn g1 and g2 in a rejection form, to reject O fromO ∪ H (where I(Dw ) = +1). Concretely, the rejection setting is the same as in Cortes et al. (2016).
Also, inspired by Geifman & El-Yaniv (2019), the optimization objective of Dw and g isL(Dw , g) ≜ ˆl(Dw , g) + λ max(0, c − ϕ̂(g))2 ,(17)

where c > 0 denotes the target coverage, and λ denotes the factor for controlling the relative importance of rejection. Besides, the empirical coverage ϕ̂(g) is defined asmϕ̂(g|X) ≜1 X
g(xi ),
m i=1

(18)

with a batch of data X = {xi }, i ∈ [m]. The empirical rejection risk ˆl(Dw , g) is the ratio between
the covered risk of the discriminator and the empirical coverage:Pm1ˆl(Dw , g) ≜ m i=1 ⟨L(Dw (xi )), g(xi )⟩ .
(19)
ϕ̂(g)
Meanwhile, both Dw1 and g1 can access ρπE under OE directly. So given x ∼ T π2 under OL ,once ⟨I(Dw2 (x)), g2 (x)⟩ = +1, we can query the corresponding observations x
e of x through OC
operation and use ⟨I(Dw1 (ex)), g1 (ex)⟩ to calibrate the output of g2 and Dw2 . In this way, g2 and Dw2
can be entangled together and adaptively guide π2 to find out the latent demonstrations H under OL .
4.4

IWRE

Here we combine importance weighting and rejection into a unified procedure, resulting in a novel
algorithm named Importance Weighting with REjection (IWRE). Concretely, in a HOIL process:Pretraining. We train a discriminator Dw1 by Equation (5) and its corresponding rejection model
g1 by Equation (17) using the initial data and the expert demonstrations.
Training. We train a discriminator Dw2 by the combination of Equation (9) and Equation (17),as well as its corresponding rejection model g2 by Equation (17), using the initial data, the datacollected by π2 , and the output of Dw1 with g1 through OC operation. Also, π2 will be updated with
Dw2 and g2 asymmetrically as in GAIL.
The pseudo-code of our algorithm is provided in the appendix.
7

Published as a conference paper at ICLR 2023

2000

1200

1200

1000

1000

400

200

0
0

2000

800

1250
1000

600

400

200

6000

Episodes

8000

500

10000

20000

30000

40000

50000

Episodes

60000

70000

600

400

200

0

0
0

(a)
ChopperCommand(Atari)

750

250

0
4000

Rewards

GAIL
IW
TPIL
GAMA
IWRE
PPO-RAM
BC
LBC

600

800

1500

Rewards

Rewards

Rewards

1750

800

−200
0

(b) Kangaroo (Atari)

2500

5000

7500

10000 12500 15000 17500 20000

0

Episodes(c) Qbert (Atari)

20000

40000

60000

80000 100000 120000 140000 160000

Episodes(d) Hopper (MuJoCo)

−10
800

1500
40
−20

1250

−30

400

−40

300

Rewards

500

Rewards

600

Rewards

Rewards

700

1000

20

0

750
500
250

200
−20

−50

100

0

0

50000 100000 150000 200000 250000 300000 350000

Episodes(e) Humanoid (MuJoCo)

0

100000 200000 300000 400000 500000 600000 700000 800000

0

Episodes(f) Reacher (MuJoCo)

5000

10000 15000 20000 25000 30000 35000 40000

Episodes(g) Swimmer (MuJoCo)

0

20000

40000

60000

80000

100000 120000 140000

Episodes(h) Walker2d (MuJoCo)

Figure 6: The learning curves of each method. Shaded regions indicate the standard deviation.

5

E XPERIMENTIn this section, we validate our algorithm in Atari 2600 (Bellemare et al., 2013) (GPL License) and
MuJoCo (Todorov et al., 2012) (Academic License) environments. The experiments were designed
to investigate: 1) Can IWRE achieve significant performance under HOIL tasks? 2) Can IWRE
deal with the support mismatch problem? 3) During training, is active querying for HOIL indeed
necessary? Below we first introduce the experimental setup and then investigate the above questions.
More results and experimental details are included in the appendix.
5.1

E XPERIMENTAL S ETUP

Environments. We choose three pixel-memory based games in Atari and five continuous control
objects in MuJoCo on OpenAI platform (Brockman et al., 2016) (MIT License). For pixel-memory
Atari games, OE : 84 × 84 × 4 raw pixels; OL : 128-byte random access memories (RAM). For
continuous control MuJoCo objects, OE : half of original observation features; OL : another half oforiginal observation features. Besides, twenty expert trajectories were collected for each environment. Each result contains five trials with different random seeds. All experiments were conducted
on server clusters with NVIDIA Tesla V100 GPUs. Further details of the environments are includedin the appendix.
Baselines. Six basic contenders were included in the experiments: Vanilla GAIL (Ho & Ermon,
2016), GAIL with importance weighting (Fang et al., 2020) (IW, an ablation study of IWRE), thirdperson IL (Stadie et al., 2017) (TPIL), generative adversarial MDP alignment (Kim et al., 2020)(GAMA), behavioral cloning (Bain & Sammut, 1996) (BC), and learning by cheating (Chen et al.,2019) (LBC). For IW, we utilized Dw1 trained in the pretraining to calculate the importance weight;
also the optimization objective for Dw2 during training is the same as Equation (9); TPIL learns thethird-person demonstrations by leading the cross-entropy loss into the update of the feature extractor;
GAMA learns a mapping function ψ in view of adversarial training to align the observation of thetarget domain into the source domain, and thereby can utilize the policy in the source domain forzero-shot imitation. For fairness, we allowed the interaction between the policy and the environmentfor GAMA under HOIL; LBC uses π1 learned from privileged states as a teacher to train π2 ina DAgger (Ross et al., 2011) style, so here we allowed LBC to access OE during the whole IL
process. In Atari, to investigate whether our method could achieve good performance for RAMbased control, we further included a contender PPO-RAM, which uses proximal policy optimization(PPO) (Schulman et al., 2017) to perform RL directly with environmental true rewards under the
RAM-based observations. More detailed setup including query strategies for TPIL and GAMA,
network architecture, and hyper-parameters are reported in the appendix.
5.2

R ESULTS

Experimental results are reported in Figure 6. Since the mapping function is hard to learn when
input is RAM and output is raw images, we omit the results of GAMA in Atari. We can observe
8

Published as a conference paper at ICLR 2023

that while IW is better than GAIL in most environments, both GAIL and IW can hardly outperform
π1 . Because they just imitated the performance of π1 instead of πE , even with importance weighting
for calibration. For TPIL, its learning process was extremely unstable on Hopper, Swimmer, andWalker2d due to the continuous distribution shift. Furthermore, the performance of GAMA was notsatisfactory in Hopper and Walker2d because its mapping function is hard to learn well when the
support mismatch appears. The results of TPIL and GAMA demonstrate that DSIL methods will be
invalid under heterogeneous observations as in HOIL tasks. On Atari environments, OE contains
more privileged information than OL , so LBC can achieve good performance. But when OE is
not more privileged than OL , like in most environments of MuJoCo, its performance will decrease
due to the support mismatch, which would make it even worse than BC. Finally, IWRE obtained
the best performance on 6/8 environments, and comparable performance with LBC on Reacher,which shows the effectiveness of our method even with limited access to OE (LBC can accessto OE all the time). Besides, we can see that the performance differences between the GAIL/IWand IWRE/TPIL/GAMA/LBC are huge (especially on Reacher) because of the absence of queries,
which demonstrates that the query operation is indeed necessary for HOIL problems.
Moreover, even learned with true rewards, PPO-RAM surprisingly failed to achieve comparable
performance to IWRE, which shows that IWRE could possibly learn more effective rewards thantrue environmental rewards in RAM-input tasks. The results verify that, IWRE provides a powerfulapproach for tackling HOIL problems, even under the situation that the demonstrations are gatheredfrom such a different observation space, meanwhile OE is strictly limited during training.
t-SNE visualization of ρπ2 and ρπE under OE . In Section 4.2, we point that the sub-optimality of π1 will cause
the problem of support mismatch, which is embodied
as the appearance of the latent demonstration H during
training. Also the empirical results in Figure 5 on Hopper and Walker2d verify the existence of this problem. So(a) Hopper(b) Walker2d
we want to investigate whether the superiority of IWRE
indeed comes from successfully tackling the support mismatch problem. To this end, we plotted the t-SNE vi- Figure 7: t-SNE visualizations of expert
sualization of the same expert demonstrations as in Sec- demonstrations and collected data of π2tion 4.2 and the collected data of π2 by IWRE under OE under OE . The high-density regions of(OE is hidden to π2 ). All setups are the same as in Sec- the expert demonstrations were coveredtion 4.2. From the results shown in Figure 7, we can see by the collected data of π2 of IWRE.
that even under OE , which cannot be obtained by π2 , almost all high-density regions of the demonstrations were covered by the collected data. Meanwhile,the latent demonstration H is dug out nearly. The results demonstrate that IWRE basically solves
the problem of support mismatch and thereby performs well in these environments.
Expertπ2(IWRE)

Besides, some collected data of π2 of IWRE were out of the distribution of the demonstrations,which means π2 slightly overly explored the environment. Since OE is hidden to π2 , the reward
function will encourage π2 to explore more areas to fix the support mismatch problem. Meanwhile,
the out-of-distribution problem in HOIL is not as severe as in the offline RL settings (Levine et al.,2020), so this over-exploration phenomenon makes sense.

6

C ONCLUSION

In this paper, we proposed a new learning framework named Heterogeneously Observable Imitation Learning (HOIL), to formulate situations where the observation space of demonstrations differsfrom that of the imitator while learning. We formally modeled the learning process of HOIL, inwhich access to expert observations is limited due to the high cost. Furthermore, we analyzed the underlying challenges of HOIL: the dynamics mismatch and the support mismatch, on the occupancy
distributions between the demonstrations and the policy. To tackle these challenges, we proposed a
new algorithm named Importance Weighting with REjection (IWRE), using importance weighting
and learning with rejection. Experimental results showed that the direct imitation and domain adaptive methods could not solve this problem, while our approach obtained promising results. In the
future, we hope to give the theoretical guarantee for our algorithm IWRE and investigate how many
OE we need to query to learn a promising π2 . Furthermore, we hope to use the learning frameworkof HOIL and IWRE to tackle more learning scenarios with demonstrations in different spaces.
9Published as a conference paper at ICLR 2023

ACKNOWLEDGMENT
This research was supported by: National Key Research and Development Program of China(2020AAA0109401); National Science Foundation of China (62176117, 61921006, 62206245);
the Institute for AI and Beyond, UTokyo; JST SPRING, Grant Number JPMJSP2108. The authors would like to thank Guoqing Liu, Yoshihiro Nagano, and the anonymous reviewers for theirinsightful comments and suggestions.

R EFERENCES
Pieter Abbeel and Andrew Y. Ng. Inverse reinforcement learning. In Encyclopedia of Machine
Learning, pp. 554–558. 2010.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence15, pp. 103–129, 1996.
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research,
47:253–279, 2013.
Kianté Brantley, Hal Daumé III, and Amr Sharaf. Active imitation learning with noisy guidance. In
Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th
Annual Meeting of the Association for Computational Linguistics, ACL 2020, July 5-10, 2020,
Online, pp. 2093–2105. Association for Computational Linguistics, 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, andWojciech Zaremba. Openai gym. CoRR, abs/1606.01540, 2016.
Xin-Qiang Cai, Peng Zhao, Kai-Ming Ting, Xin Mu, and Yuan Jiang. Nearest neighbor ensembles:
An effective method for difficult problems in streaming classification with emerging new classes.
In Jianyong Wang, Kyuseok Shim, and Xindong Wu (eds.), 2019 IEEE International Conferenceon Data Mining, ICDM 2019, November 8-11, 2019, Beijing, China, pp. 970–975. IEEE, 2019.
Xin-Qiang Cai, Yao-Xiang Ding, Yuan Jiang, and Zhi-Hua Zhou. Imitation learning from pixellevel demonstrations by hashreward. In Frank Dignum, Alessio Lomuscio, Ulle Endriss, and Ann
Nowé (eds.), AAMAS ’21: 20th International Conference on Autonomous Agents and Multiagent
Systems, May 3-7, 2021, Virtual, United Kingdom, pp. 279–287. ACM, 2021.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In
Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), 3rd Annual Conference on
Robot Learning, CoRL 2019, October 30 - November 1, 2019, Osaka, Japan, volume 100 ofProceedings of Machine Learning Research, pp. 66–75. PMLR, 2019.
Zi-Xuan Chen, Xin-Qiang Cai, Yuan Jiang, and Zhi-Hua Zhou. Anomaly guided policy learningfrom imperfect demonstrations. In Piotr Faliszewski, Viviana Mascardi, Catherine Pelachaud, andMatthew E. Taylor (eds.), 21st International Conference on Autonomous Agents and Multiagent
Systems, AAMAS 2022, May 9-13, 2022, Auckland, New Zealand, pp. 244–252. International
Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In Ronald Ortner,
Hans Ulrich Simon, and Sandra Zilles (eds.), Algorithmic Learning Theory - 27th International
Conference, ALT 2016, October 19-21, 2016, Bari, Italy, volume 9925 of Lecture Notes in Computer Science, pp. 67–82, 2016.
Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, and Peter Stone.
An imitation from observation approach to transfer learning with dynamics mismatch. In Hugo
Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin(eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural
Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines, 2017.
10

Published as a conference paper at ICLR 2023

Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking importance weighting fordeep learning under distribution shift. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell,
Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing
Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020,
December 6-12, 2020, virtual, 2020.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In 6th International Conference on Learning Representations, ICLR 2018,
April 30 - May 3, 2018, Vancouver, BC, Canada. OpenReview.net, 2018.
Tanmay Gangwani, Joel Lehman, Qiang Liu, and Jian Peng. Learning belief representations forimitation learning in pomdps. In Amir Globerson and Ricardo Silva (eds.), Proceedings of theThirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July
22-25, 2019, volume 115 of Proceedings of Machine Learning Research, pp. 1061–1071. AUAI
Press, 2019.
Yonatan Geifman and Ran El-Yaniv. Selectivenet: A deep neural network with an integrated reject
option. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California,
USA, volume 97 of Proceedings of Machine Learning Research, pp. 2151–2159. PMLR, 2019.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing
Systems 2016, NIPS 2016, December 5-10, 2016, Barcelona, Spain, pp. 4565–4573, 2016.
Bo-Jian Hou, Lijun Zhang, and Zhi-Hua Zhou. Learning with feature evolvable streams. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N.
Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems
30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017,
Long Beach, CA, USA, pp. 1417–1427, 2017.
Chenping Hou and Zhi-Hua Zhou. One-pass learning with incremental and decremental features.
IEEE Trans. Pattern Anal. Mach. Intell., 40(11):2776–2792, 2018.
Sankaran Iyer, Alan Blair, Laughlin Dawes, Daniel Aaron Moses, Christopher White, and Arcot
Sowmya. Organ localisation using supervised and semi supervised approaches combining reinforcement learning with imitation learning. CoRR, abs/2112.03276, 2021.
Shengyi Jiang, Jing-Cheng Pang, and Yang Yu. Offline imitation learning with a misspecified simulator. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural
Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, volume 33,
pp. 8510–8520. Curran Associates, Inc., 2020.
Kun Ho Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Cross domain imitation learning. CoRR, abs/1910.00105, 2019.
Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imitation learning. In Proceedings of the 37th International Conference on Machine Learning, ICML
2020, 13-18 July 2020, Virtual, volume 119, pp. 5286–5295. PMLR, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01643, 2020.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa,
David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua
Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR
2016, May 2-4, 2016, San Juan, Puerto Rico, 2016.
Yuxuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International
Conference on Robotics and Automation, ICRA 2018, May 21-25, 2018, Brisbane, Australia, pp.
1118–1125. IEEE, 2018.
11

Published as a conference paper at ICLR 2023

Donald Michie, Michael Bain, and J Hayes-Miches. Cognitive models from subcognitive skills. IEE
control engineering series, 44:71–99, 1990.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan
Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR,
abs/1312.5602, 2013.
Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep
decentralized multi-task multi-agent reinforcement learning under partial observability. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW,
Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2681–
2690. PMLR, 2017.
Dripta S. Raychaudhuri, Sujoy Paul, Jeroen van Baar, and Amit K. Roy-Chowdhury. Cross-domain
imitation from observations. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th
International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual, volume
139 of Proceedings of Machine Learning Research, pp. 8902–8912. PMLR, 2021.
Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning andstructured prediction to no-regret online learning. In Proceedings of the Fourteenth International
Conference on Artificial Intelligence and Statistics, AISTATS 2011, April 11-13, 2011, Fort Lauderdale, USA, volume 15 of JMLR Proceedings, pp. 627–635. JMLR.org, 2011.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In 9th International Conference on Learning Representations, ICLR 2021, Austria, May 3-7, 2021, Virtual.
OpenReview.net, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy
optimization algorithms. CoRR, abs/1707.06347, 2017.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey
Levine. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, May 21-25, 2018, Brisbane, Australia, pp. 1134–1141. IEEE, 2018.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the loglikelihood function. Journal of statistical planning and inference, 90(2):227–244, 2000.
Bradly C. Stadie, Pieter Abbeel, and Ilya Sutskever. Third person imitation learning. In 5th International Conference on Learning Representations, ICLR 2017, April 24-26, 2017, Toulon, France.
OpenReview.net, 2017.
Jakub Sygnowski and Henryk Michalewski. Learning from the memory of atari 2600. In Tristan
Cazenave, Mark H. M. Winands, Stefan Edelkamp, Stephan Schiffel, Michael Thielscher, andJulian Togelius (eds.), Computer Games - 5th Workshop on Computer Games, CGW 2016, and
5th Workshop on General Intelligence in Game-Playing Agents, GIGA 2016, Held in Conjunction
with the 25th International Conference on Artificial Intelligence, IJCAI 2016, July 9-10, 2016,
New York City, NY, USA, volume 705 of Communications in Computer and Information Science,
pp. 71–85, 2016.
Andrea Tirinzoni, Andrea Sessa, Matteo Pirotta, and Marcello Restelli. Importance weighted transfer of samples in reinforcement learning. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, July 10-15,
2018, Stockholmsmässan, Stockholm, Sweden, volume 80 of Proceedings of Machine Learning
Research, pp. 4943–4952. PMLR, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012,
October 7-12, 2012, Vilamoura, Algarve, Portugal, pp. 5026–5033, 2012.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine
Learning Research, 9:2579–2605, 2008.
12

Published as a conference paper at ICLR 2023

Linjing Wang, Tianlan Mo, Xuetao Wang, Wentao Chen, Qiang He, Xin Li, Shuxu Zhang, Ruimeng
Yang, Jialiang Wu, Xuejun Gu, Jun Wei, Peiliang Xie, Linghong Zhou, and Xin Zhen. A hierarchical fusion framework to integrate homogeneous and heterogeneous classifiers for medical
decision-making. Knowledge-Based Systems, 212:106517, 2021a.
Ruohan Wang, Carlo Ciliberto, Pierluigi Vito Amadori, and Yiannis Demiris. Random expert distillation: Imitation learning via expert policy support estimation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning,
ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine
Learning Research, pp. 6536–6544. PMLR, 2019.
Yunke Wang, Chang Xu, and Bo Du. Robust adversarial imitation learning via adaptively-selected
demonstrations. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, 19-27 August 2021, Virtual / Montreal, Canada, pp.
3155–3161. ijcai.org, 2021b.
Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to weight imperfect demonstrations.
In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on
Machine Learning, ICML 2021, 18-24 July 2021, Virtual, volume 139 of Proceedings of Machine
Learning Research, pp. 10961–10970. PMLR, 2021c.
Yunke Wang, Bo Du, and Chang Xu. Unlabeled imperfect demonstrations in adversarial imitation
learning. CoRR, abs/2302.06271, 2023.
Andrew Warrington, Jonathan Wilder Lavington, Adam Ścibior, Mark Schmidt, and Frank Wood.
Robust asymmetric learning in pomdps. In Marina Meila and Tong Zhang (eds.), Proceedings ofthe 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual,
volume 139 of Proceedings of Machine Learning Research, pp. 11013–11023. PMLR, 2021.
Meng Wu, Ying Zhu, Qilian Yu, Bhargav Rajendra, Yunqi Zhao, Navid Aghdaie, and Kazi A. Zaman. A recommender system for heterogeneous and time sensitive environment. In Toine Bogers,
Alan Said, Peter Brusilovsky, and Domonkos Tikk (eds.), Proceedings of the 13th ACM Conference on Recommender Systems, RecSys 2019, September 16-20, 2019, Copenhagen, Denmark,pp. 210–218. ACM, 2019.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. In Hugo
Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin(eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural
Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, pp. 15737–
15749, 2020.
Chao Yu, Jiming Liu, and Shamim Nemati. Reinforcement learning in healthcare: A survey. CoRR,
abs/1908.08796, 2019.
Zhenyu Zhang, Peng Zhao, Yuan Jiang, and Zhi-Hua Zhou. Learning with feature and distribution
evolvable streams. In Proceedings of the 37th International Conference on Machine Learning,
ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning
Research, pp. 11317–11327. PMLR, 2020.
Zhi-Hua Zhou. Open-environment machine learning. National Science Review, 9(8):nwac123, 2022.

13

Published as a conference paper at ICLR 2023

A

N OTATIONSThe notations of the main paper are gathered in Table 2.
Table 2: The notations of the main paper.

Notation

Meaning

S
A
O
OE
OL
P
γ
πE
π1
π2
fE
fL
TeπE
Teπ1
Teπ2
T π1
T π2
x
x
e
x
ρπE
ρπ1
ρπ2
Dw1
Dw2
α
H
O
N
g1
g2

State space
Action space
Observation space
Observation space of the expert’s view
Observation space of the learner’s view
Transition Probability
Discounted factor
Expert policy under OE
Auxiliary policy under OE
Target policy under OL
Mapping function S → OE
Mapping function S → OL
Trajectory sampled by πE under OE (demonstrations)Trajectory sampled by π1 under OE
Trajectory sampled by π2 under OE
Trajectory sampled by π1 under OL
Trajectory sampled by π2 under OL
An instance of state-action pair
An instance of observation-action pair under OE
An instance of observation-action pair under OL
Occupancy measure of the expert policy πE
Occupancy measure of the auxiliary policy π1
Occupancy measure of the target policy π2Adversarial model on TeπE and Teπ1
Adversarial model on T π1 and T π2
Importance weighting factor
Latent demonstration
Observed demonstration
Non-expert data
rejection model under OE
rejection model under OL

14

Published as a conference paper at ICLR 2023

B

A LGORITHM

The pseudo codes of our algorithm are illustrated in Algorithms 1 and 2.
Algorithm 1 IWRE.Pretraining
Input: Auxiliary policy π1 ; Expert demonstrations TeπE .
Output: Evolving data {Teπ1 , T π1 }; Discriminator Dw1 ; Rejection model g1 .
1: function IWRE.P RETRAINING(π1 )2:Sample the evolving data {Teπ1 , T π1 } ∼ ρπ1 by π1 .
3:Train Dw1 and g1 by Equation (5) and (17) respectively using TeπE and Teπ1 .
4:return T π1 , Dw1 , g1
5: end function
Algorithm 2 IWRE.Training
Input: Expert demonstrations TeπE ; Evolving data T π1 ; Discriminator Dw1 ; Rejection model g1 .
Output: Target policy π2 .
1: function IWRE.T RAINING(TeπE , T π1 , Dw1 , g1 )2:
Initialize π2 , Dw2 , and g2 .
3:
for each step t do
4:
Sample T π2 ∼ ρπ2 by π2 .
5:for each mini-batch {xπ2 } and {xπ1 } from T π2 and T π1 do
6:Update π2 by RL algorithms (such as PPO (Schulman et al., 2017)) using instances{xπ2 } and pseudo rewards {− log Dw2 (xπ2 )}.
7:
Update Dw2 by Equation (9) using negative instances {xπ2 } and positive ones
{xπ1 }.
8:
if ⟨I(Dw2 (xπ2 )), g2 (xπ2 )⟩ = +1 then9:
Query the OE observation of xπ2 , i.e., x
eπ2 , through OC operation.
10:
Update Dw2 and g2 by Equation (17) using the instance xπ2 and the corresponding label ⟨I(Dw1 (exπ2 )), g1 (exπ2 )⟩.
11:
end if
12:
end for13:
end for14:
return π2
15: end functionC

D EFINITIONS

The core challenges of HOIL, i.e., dynamics mismatch and support mismatch, are illustrated as
below.
Definition 5 (Dynamics Mismatch). The dynamics mismatch between the demonstrations and theinitial data denotes the situation that:P∞
πE (a|o) t=0 γ t Pr(st = s|πE )
ρπ E
P∞
=
̸= 1.
(20)
ρπ1
π1 (a|o) t=0 γ t Pr(st = s|π1 )Definition 6 (Support Mismatch). The support mismatch between the demonstrations and the initial
data denotes the situation that:supp(ρπE ) \ supp(ρπ1 ) = {x ∈ S × A|ρπE (x) ̸= 0} \ {x ∈ S × A|ρπ1 (x) ̸= 0} ̸= ∅.

15

(21)

Published as a conference paper at ICLR 2023

Table 3: Environmental summary of the tasks.
Environment
Qbert
ChopperCommand
Kangaroo
HopperHumanoid
Reacher
SwimmerWalker2d

Observation Space OE

Observation Space OL

84 × 84 × 4(image)

128(unsigned int)

8
4
5
5
188

9
4
6
6
188

Expert Rewards
4750.00 ± 50.51
3135.00 ± 145.86
4175.00 ± 94.21
709.96 ± 75.54
539.20 ± 26.26-8.99 ± 0.54
52.24 ± 1.29
929.97 ± 24.09

Table 4: Comparisons between all contenders and IWRE in HOIL.
Algorithm
GAIL
GAIL-Rand
IW
IW-Rand
TPIL
GAMA
BC
LBC
PPO-RAM
IWRE

D

Considering heterogeneous observations

#
#
#
#
#
#
#
!
#
!

Being able to query

#
!
#
!
!
!
#
!
#
!

Not requiring OE all along

!
!
!
!
!
!
!
#
!
!

D ETAILED S ETUP FOR THE E XPERIMENTS

Environment and Contenders. For the environments that used in the main body of the paper:
1. Pixel-memory Atari games. OE : 84 × 84 × 4 raw pixels; OL : 128-byte random accessmemories (RAM). Expert: converged DQN-based agents (Mnih et al., 2013). Atari games
contain two totally isolated views: raw pixels and RAM, under the same state. Throughthese environments, we want to investigate whether the agent can learn an effective policy from demonstrations under completely different observation spaces. Moreover, IL
with visual observations only is already very difficult (Cai et al., 2021), while learning
a RAM-based policy can be even more challenging (Bellemare et al., 2013; Sygnowski &Michalewski, 2016), so few IL research reported desirable results on this task.
2. Continuous control MuJoCo objects. OE : half of original observation features; OL :another half of original observation features. Expert: converged DDPG-based agents (Lillicrap et al., 2016). The features of MuJoCo contain monotonous information like thedirection, position, velocity, etc., of an object. Here we want to investigate whether theagent can learn from demonstrations with complementary signals under observations withmissing information. Meanwhile, we make sure RL algorithms can obtain comparable
performances under OE and OL .
The details of the environments are reported in Table 3. Also, the detailed comparisons of thecontenders (both in the main paper and the appendix) and IWRE are gathered in Table 4.
Learning process. To simulate the situation that OE is costly, the steps for training π1 was set as 1/4of that for training π2 , using GAIL (Ho & Ermon, 2016)/HashReward (Cai et al., 2021) under the
OE space for MuJoCo/Atari environments. The learning steps were 107 for MuJoCo and 5 × 106 forAtari environments. In the pretraining, we sampled 20 trajectories from π1 , and the data from eachtrajectory had both OE and OL observations. In the training, each method learned 4 × 107 steps for
MuJoCo and 2 × 107 steps for Atari under the OL space to obtain π2 .
Query Strategy. For TPIL and GAMA, if the output of the domain invariant discriminator is larger
than 0.5, which means the encoder fails to generate proper features to confuse its discriminator, thenwe would query OE of this data to update the encoder. For IWRE, the threshold of the rejection
model g and the discriminator Dw2 was also 0.5, which means that if g2 (x) > 0.5 meanwhile
Dw2 (x) > 0.5, OE of this data would be queried. Dw2 , π2 , and the encoder (for TPIL/GAMA)
16

Published as a conference paper at ICLR 2023

were pretrained for 100 epochs for all methods using evolving data during pretraining. The basic
RL algorithm is PPO, and the reward signals of all methods were normalized into [0, 1] to enhance
the performance of RL (Dhariwal et al., 2017). The buffer size for TPIL and IWRE was set as
5000. Each time the buffer is full, the encoder and the rejection model will be updated for 4 epochs;
also LBC will update π2 for 100 epochs with batch size 256 using the cross-entropy loss for Atariand the mean-square loss for MuJoCo. We set all hyper-parameters, update frequency, and networkarchitectures of the policy part the same as Dhariwal et al. (2017). Besides, the hyper-parametersof the discriminator for all methods were the same: The rejection model and discriminator were
updated using Adam with a decayed learning rate of 3 × 10−4 ; the batch size was 256. The ratio of
update frequency between the learner and discriminator was 3: 1. The target coverage c in Equation(17) was set as 0.8. λ in Equation (17) was 1.0.

E

RL P ERFORMANCE UNDER THE D IVISIONS OF M U J O C O

Here we report the performance under the division of OE and OL in MuJoCo. The details of thedivision are reported in Table 5. We use DDPG-based (Lillicrap et al., 2016) agent with 107 training
steps and repeat 10 times with different random seeds. The results are shown in Figure 8. We can seethat the agent can obtain comparable performance under OE and OL . So for MuJoCo environments,
the fairness of the division in HOIL can be guaranteed, and OE is not more or less privileged thanOL .
Table 5: The observation division into OE and OL in MuJoCo. The numbers denote the randomly
selected observation indexes in the corresponding MuJoCo environment on OpenAI Gym (Brockman et al., 2016) platform.
Walker2d
Swimmer
Reacher
HopperHumanoidFOE[5, 7, 8, 10, 11, 14, 15, 16][0, 3, 6, 7][0, 1, 7, 8, 10][1, 3, 6, 7, 9, 10][2, 3, 5, 6, 7, 10, 11, 12, 13, 16, 18, 19, 22,
23, 25, 29, 31, 32, 34, 36, 37, 40, 43, 44, 45,
47, 48, 49, 51, 54, 56, 57, 61, 63, 65, 66, 67,
68, 77, 78, 82, 86, 87, 89, 90, 93, 94, 95, 97,
98, 99, 102, 103, 108, 110, 112, 113, 117, 119,
120, 121, 122, 123, 124, 126, 127, 128, 133, 135,
144, 146, 147, 148, 151, 152, 153, 158, 160, 161,
162, 166, 167, 170, 171, 173, 174, 176, 177, 178,
180, 181, 184, 185, 187, 188, 191, 194, 198, 199,
200, 201, 202, 207, 208, 209, 210, 211, 212, 214,
215, 219, 223, 227, 228, 229, 231, 232, 233, 234,
236, 237, 238, 242, 244, 246, 248, 251, 253, 257,
258, 259, 260, 262, 264, 265, 267, 268, 271, 272,
273, 275, 278, 279, 280, 281, 285, 287, 289, 290,
291, 293, 294, 296, 299, 302, 304, 305, 306, 307,
308, 311, 312, 313, 315, 316, 319, 322, 326, 328,
329, 332, 337, 342, 343, 344, 345, 349, 358, 361,
362, 364, 365, 366, 368, 370, 372, 373, 375]OL[0, 1, 2, 3, 4, 6, 9, 12, 13][1, 2, 4, 5][2, 3, 4, 5, 6, 9][0, 2, 4, 5, 8][0, 1, 4, 8, 9, 14, 15, 17, 20, 21, 24, 26, 27,
28, 30, 33, 35, 38, 39, 41, 42, 46, 50, 52, 53,
55, 58, 59, 60, 62, 64, 69, 70, 71, 72, 73, 74,
75, 76, 79, 80, 81, 83, 84, 85, 88, 91, 92, 96,
100, 101, 104, 105, 106, 107, 109, 111, 114, 115,
116, 118, 125, 129, 130, 131, 132, 134, 136, 137,
138, 139, 140, 141, 142, 143, 145, 149, 150, 154,
155, 156, 157, 159, 163, 164, 165, 168, 169, 172,
175, 179, 182, 183, 186, 189, 190, 192, 193, 195,
196, 197, 203, 204, 205, 206, 213, 216, 217, 218,
220, 221, 222, 224, 225, 226, 230, 235, 239, 240,
241, 243, 245, 247, 249, 250, 252, 254, 255, 256,
261, 263, 266, 269, 270, 274, 276, 277, 282, 283,
284, 286, 288, 292, 295, 297, 298, 300, 301, 303,
309, 310, 314, 317, 318, 320, 321, 323, 324, 325,
327, 330, 331, 333, 334, 335, 336, 338, 339, 340,
341, 346, 347, 348, 350, 351, 352, 353, 354, 355,
356, 357, 359, 360, 363, 367, 369, 371, 374]

E STIMATION OF H , O, AND N BY I[Dw2 ]g2

To investigate the ability of IWRE to distinguish the areas of latent demonstrations H, observed
demonstrations O, and non-expert data N during policy learning, we recorded the accuracy andestimated ratio of each part on Hopper and Walker2d. The calculations of each curve are shown as
below:
Pm
Accuracy H =x )]g (ex ) == 1&&I[Dw2 (xi )]g2 (xi ) == 1}i=1 {I[Dw1 (ePim 1 i,{I[Dxi )]g1 (exi ) == 1}
w1 (e
i=1

(22)

Pm
Accuracy O =x )]g (ex ) == 0&&I[Dw2 (xi )]g2 (xi ) == 0}i=1 {I[Dw1 (ePim 1 i,{I[Dxi )]g1 (exi ) == 0}
w1 (e
i=1
17

(23)

Published as a conference paper at ICLR 2023

−10
800
500

700

−8

−4

400

Rewards

Rewards

Rewards

600
−6

500
400
300

300

200

200

−2

100
100

0

E

Observation Spaces

0

L

E

(a) Reacher

Observation Spaces

0

L

E

(b) Hopper

Observation Spaces

L(c) Humanoid

1000
50
800

Rewards

Rewards

40

30

20

600

400

200

10

0

E

Observation Spaces0L(d) Swimmer

E

Observation Spaces

L(e) Walker2d

Figure 8: The performance of RL methods under the division of OE and OL in MuJoCo. The agent
can obtain comparable performances under OE and OL , so that we can make sure the fairness of theexperiment of HOIL in the main paper.
Pm
Accuracy N =xi )]g1 (ex ) == −1&&I[Dw2 (xi )]g2 (xi ) == −1}i=1 {I[Dw1 (e
Pm i
,
{I[D
xi )]g1 (exi ) == −1}
w1 (e
i=1
Pm
xi )]g1 (exi ) == 1}i=1 {I[Dw1 (eRatio H =,m
{I[Dw1 (exi )]g1 (exi ) == 0}
Ratio O = i=1,m
Pm{I[Dw1 (exi )]g1 (exi ) == −1}
Ratio N = i=1
,
m

(24)
(25)

Pm

(26)
(27)

ei } ∼ ρπ2 denotes a batch of data sampled by π2 . The results are shown in Figure 9.
in which {xi , x
The results depicted not only the accuracies of I[Dw2 ]g2 , but also the changes of these three areas
during the policy learning. We can see that the accuracies in each area and the ratio of O will
decrease at first. While at the same time, the ratio of H will increase. This is because the successful
detection of H will decrease the estimated ratio of O and reduce the accuracy of I[Dw2 ]g2 . With
the help of query operations, the accuracy of I[Dw2 ]g2 will gradually increase. Also, followed by
the learning procedure of the policy π2 , more and more H will be recognized as O, with less andless N . This is why in the following period, the ratios of H and N will decrease while that of O
will increase. These results also verify that our algorithm IWRE can indeed detect H, O, and N
successfully as the learning process of the policy.

18

Published as a conference paper at ICLR 2023

1.0

1.0

H
O
N

0.8

Estimated Ratios

Accuracy

0.8

H
O
N

0.6

0.4

0.2

0.6

0.4

0.2

0.0
0

20000

40000

60000

80000 100000 120000 140000 160000

0

Episodes(a) Accuracy on Hopper

1.0

40000

60000

80000 100000 120000 140000 160000

Episodes(b) Ratio on Hopper

1.0

H
O
N

H
O
N

0.8

Estimated Ratios

0.8

Accuracy

20000

0.6

0.4

0.2

0.6

0.4

0.2

0.0

0.0
0

20000

40000

60000

80000

Episodes100000 120000 140000

0(c) Accuracy on Walker2d

20000

40000

60000

80000

Episodes100000 120000 140000(d) Ratio on Walker2d

Figure 9: The accuracies and ratios of H, O, and N calculated by I[Dw1 (exi )]g1 (exi ) andI[Dw2 (xi )]g2 (xi ) during policy learning.

G

I NVESTIGATIONS ON ChopperCommand P ERFORMANCE

We observed some interesting phenomenons on ChopperCommand performance. Here we investigate the behavior of a random policy, the IWRE policy, and the expert policy, shown in Figure 10.
On ChopperCommand, the agent needs to imitate the expert to manipulate a helicopter (noted by theblack circle) to avoid attacks from enemy aircraft meanwhile shooting them. So the policy in thisenvironment can be split into two parts:
1. The movement of the helicopter;
2. The shoot of the helicopter.
While capturing the image-level semantic information with only RAM input is quite difficult. Even
with RAM input, IWRE still successfully learned the expert’s movement of the helicopter (see therows of “IWRE 2e7 steps” and “Expert Policy” in Figure 10). But it is much harder to capture
the shot bullet in the image, not to mention in the RAM. So IWRE policy did not shoot enemy
aircraft successfully. However, the environmental reward of ChopperCommand is only related to thenumber of enemy aircraft shot down, regardless of the distance the helicopter has flown. Therefore,
in Figure 6a, IWRE policy can obtain higher rewards than the random policy at the beginning of
the training (1e5 steps). Then the environmental reward fluctuates within a specific range. On theother hand, IWRE indeed learned some expert’s policy of manipulating the helicopter in view of the
trajectory shown in Figure 10. So we believe that only considering the reward value as an evaluation
is insufficient to reflect the degree of imitation on ChopperCommand. This interesting phenomenon
also reveals that the reward function learned from IL can be very different from the environmental
reward functions in RL tasks, since the IL reward can capture expert behaviors that are not reflectedby the environmental reward signals. Figure 10 can also be used as a metric. We also provide the19

Published as a conference paper at ICLR 2023

full videos of these four policies on ChopperCommand in this anonymous link: https://www.
dropbox.com/sh/fz7xpbt4y8t3umz/AACw5cMq5eG8sCl04x9_OOnha?dl=0.

RandomPolicy(RAM input)

IWRE
1e5 steps(RAM input)

IWRE
2e7 steps(RAM input)

ExpertPolicy(pixel input)

Figure 10: The sequence comparisons between a random policy, the IWRE policy with 1e5/2e7
steps, and the expert policy. The agent needs to manipulate a helicopter (noted by the black circle)
to avoid attacks from enemy aircraft meanwhile shooting them. The timestamp for each column ofimages and seed for each environment is the same.

H

Q UERY E FFICIENCY

We also investigate whether our query strategy is efficient. To this end, we allocate the query budget,i.e., limiting the query ratio for each method. For TPIL, it preferentially queries those data with low
Dwϕ output; for our method IWRE, it preferentially queries those data with high ⟨Dw2 , g2 ⟩ output.
Besides, since GAIL and IW cannot directly perform queries, we design a random-selection strategy
for them as GAIL-Rand and IW-Rand: for each batch of data, we randomly select data and input the
OE observations of these data to Dw1 . If Dw1 (x) > 0.5, which means Dw1 regard this data being
belonging to the expert demonstrations, then we would label this data as the expert data to update
Dw2 . The results are depicted in Figure 11.
We can observe that the random strategy does not always improve the performance of GAIL and
IW. For GAIL-Rand, without importance weighting to calibrate the learning process of the reward
function, its performance become even worse on Hopper, Swimmer, and Walker2d, because thequeried information enhances the discrimination ability of reward function, making it even more
impossible for the agent to obtain effective feedbacks; for IW-Rand, its performance is better than
GAIL-Rand on most environments, and is reinforced on Hopper, Reacher, and Walker2d, whichfurther demonstrate that the query operation is indeed necessary for HOIL problem, but still fails
compared with our method; for TPIL, it is comparable with IW-Rand, however, its performance
improvement is very limited as the budget increases, and on Swimmer and Walker2d there even exist
performance degradations, which suggests that its query strategy is very unstable; for GAMA, it
has a good start point, but the performance gain is very limited while the budget increases; for our
method, its performance is almost the same as that of IW-Rand without query on most environments.
When it is allowed to query OE observation, our method outperforms other methods with a large
gap, which shows that the query strategy of our method is indeed more efficient.

I

I MITATION WITH D IFFERENT N UMBER OF E XPERT T RAJECTORIESThe performances of different numbers of expert trajectories of all contenders are reported in Figure 12. Each experiment is conducted 5 trials with different random seeds. We can observe thateven with a very limited number of trajectories, our algorithm achieves better performance thanother algorithms in most environments.
20

Published as a conference paper at ICLR 2023

1000

−10

500
800
400

−30

−40

GAIL-Rand
IW-Rand
TPIL
GAMA
IWRE

−50

0.00

0.10

0.25

0.40

600

Rewards

Rewards

Rewards

−20

400

200

200

100

0

0.50

300

0.00

0.10

0.25

0.40

0.50

0.00

0.10

0.25

Budget Ratio

Budget Ratio

Budget Ratio(a) Reacher(b) Hopper(c) Humanoid

0.40

0.50

1000
60
800
40

Rewards

Rewards

600
20

0

400

200

−20

0

−40

−200
0.00

0.10

0.25

0.40

0.50

0.00

0.10

0.25

0.40

Budget Ratio

Budget Ratio(d) Swimmer(e) Walker2d

0.50

Figure 11: The final rewards of each method on MuJoCo with different budget ratios, where theshaded regions indicate the standard deviation. The red horizontal dotted line represents the averaged
performance of the expert.

900

800

−10

800
700

−20

700
600
600

−40
−50

−70
−80
4

10

400
300

500
400
300

GAIL
IW
TPIL
GAMA
IWRE

−60

500

Rewards

Rewards

Rewards

−30

200
200
100
100
0
20

4

10

20

4

10

# Expert Trajectories# Expert Trajectories# Expert Trajectories(a) Reacher(b) Hopper(c) Humanoid

20

1200
40
1000

800

Rewards

Rewards

20

0

600

400
−20
200

0

−40
4

10

20

4

10

# Expert Trajectories# Expert Trajectories(d) Swimmer(e) Walker2d

20

Figure 12: The learning curves of each method in MuJoCo environments with different numbers ofexpert trajectories, where the shaded region indicates the standard deviation.

21