Main Track

AAMAS 2021, May 3-7, 2021, Online

Imitation Learning from Pixel-Level Demonstrations by
HashReward
Xin-Qiang Cai, Yao-Xiang Ding, Yuan Jiang, Zhi-Hua Zhou
National Key Laboratory for Novel Software Technology
Nanjing University{caixq,dingyx,jiangy,zhouzh}@lamda.nju.edu.cn

ABSTRACT

Nevertheless, since collecting expert demonstrations is costly,
the number of expert trajectories for IL is usually limited in practice. This may significantly increase the risks of over-fitting. Oneof the key ideas to improve generalization performance is to learna proper reward function from expert demonstrations. The learnercan get high rewards only when it generates behaviors similar to thedemonstrations. Guided by such rewards, the learner is encouraged
to mine out the expert’s policy by RL algorithms, instead of directly
performing behavior cloning [1, 13]. This idea motivates the emergence of adversary-based IL. Under this family of approaches, thepolicy and the adversarial discriminator are jointly trained during
learning, in order to force the learner to minimize the discrepancy
between the generated distribution and the demonstrated data. One
representative approaches among them, generative adversarial imitation learning (GAIL) [13], achieves state-of-the-art performancein many tasks with relatively low-dimensional state spaces. However, it has been verified that the performance of such approaches,
even deep neural network-based GAIL, degenerates seriously intasks with high-dimensional state spaces [26, 29]. On the other
hand, it is well known that deep RL algorithms can achieve evensuperhuman performance under these domains [18].
In this work, we propose thorough studies towards understanding why adversary-based algorithms fail in high-dimensional IL
environments. For obtaining a good policy, the discriminator plays
an important role. The cause for performance degeneration in highdimensional space can be interpreted by the hardness of properly
dealing with the discrimination-rewarding trade-off in the discriminator learning process. Because the number of demonstration samples is limited, under high-dimensional state space, the learned
discriminator may bias towards the discrimination side, providing
meaningless rewarding signals for policy learning. Experimental observations and theoretical results are provided to support the above
interpretations: They disclose that an ideal balance between performing dimensionality reduction (DR) and discriminative training
is essential for learning.
Based on the above findings, we study a practical approach toenhance discriminator learning. It is non-trivial to guarantee that
discriminative information in high-dimensional space can be preserved after performing DR. This suggests that directly utilizing
off-the-shelf unsupervised DR algorithms, such as unsupervised
autoencoder and hashing, is not the right choice because they makethe processes of DR and discriminator learning totally isolated. As
a result, essential information for discrimination can be heavily
lost due to DR, making discriminator training biased towards discrimination or rewarding only. To address this issue, we propose anovel IL algorithm named HashReward, which utilizes a supervisedhashing strategy to incorporate DR and discriminator training into

One of the key issues for imitation learning lies in making policy
learned from limited samples to generalize well in the whole stateaction space. This problem is much more severe in high-dimensional
state environments, such as game playing with raw pixel inputs.
Under this situation, even state-of-the-art adversary-based imitation learning algorithms fail. Through empirical studies, we findthat the main cause lies in the failure of training a powerful discriminator to generate meaningful rewards in high-dimensional
environments. Although it seems that dimensionality reduction canhelp, a straightforward application of off-the-shelf methods cannot
achieve good performance. In this work, we show in theory thatthe balance between dimensionality reduction and discriminative
training is essential for effective learning. To achieve this target, we
propose HashReward, which utilizes the idea of supervised hashing
to realize such an ideal balance. Experimental results show that
HashReward could outperform state-of-the-art methods for a large
gap under the challenging high-dimensional environments.

KEYWORDS
Imitation Learning; High-Dimensional Environments; Hashing
ACM Reference Format:
Xin-Qiang Cai, Yao-Xiang Ding, Yuan Jiang, Zhi-Hua Zhou. 2021. Imitation
Learning from Pixel-Level Demonstrations by HashReward. In Proc. of the20th International Conference on Autonomous Agents and Multiagent Systems(AAMAS 2021), Online, May 3–7, 2021, IFAAMAS, 9 pages.

1

INTRODUCTION

In recent years, reinforcement learning (RL) has achieved a great
breakthrough in many domains including robot controlling andgame playing [18, 20]. In spite of remarkable success, there are
two main issues unsolved. First, in common situations, RL algorithms rely so much on the well-specified reward functions andexploration strategies, which require delicate designs in many complex problems. Second, the sample and computational complexities
for practical RL algorithms are usually large, making it an unacceptable choice in solving many practical problems. On the other
hand, imitation learning (IL), aiming at learning a good policy fromdemonstrations, enables the possibility of sample efficient policy
learning without the need of designing rewarding and exploration
strategies by hand.

Proc. of the 20th International Conference on Autonomous Agents and Multiagent Systems(AAMAS 2021), U. Endriss, A. Nowé, F. Dignum, A. Lomuscio (eds.), May 3–7, 2021, Online.
© 2021 International Foundation for Autonomous Agents and Multiagent Systems(www.ifaamas.org). All rights reserved.

279

Main Track

AAMAS 2021, May 3-7, 2021, Online

a unified procedure. Experiments show that HashReward achieves
significant improvement comparing to other state-of-the-art IL approaches under high-dimensional environments, which are broadly
recognized to be challenging for IL algorithms.
The rest of the paper is organized as follows. Section 2 discusses
related work. Section 3 provides preliminaries. Section 4 demonstrates how to solve the underlying discrimination-rewarding tradeoff problem for adversary-based IL methods, and then introduces
HashReward algorithm. Section 5 reports the experimental setup
and results. Finally, Section 6 concludes the paper.

2

exploration. Though unsupervised hashing seems to be promisingfor improving IL, we observe that the performance is not that satisfactory in the experiments. This phenomenon is reasonable sincedirect unsupervised hashing may lead to the risk of losing discriminative information in the original state space. This motivates us topropose HashReward to address this issue.

3

PRELIMINARIESIn policy learning problems, a Markov Decision Process (MDP) canbe represented by a tuple ⟨S, A, P, γ , r,T ⟩, in which S denotes theset of states, A denotes the set of actions, P : S × A × S → R denotes the transition probability distributions of the state and action
pairs, γ ∈ (0, 1] denotes the discount factor, r : S → R denotes the
reward function, S 0 : S → R denotes the initial state distribution
and T denotes the horizon. The objective of RL is to learn a polÍicy π to maximize the expected total rewards E[ t∞=0 γ t r (st , at )]obtained by π .
Different from RL, in IL, the learner has no access to r . Instead,
there are m expert demonstrations {τ E,1, τ E,2, . . . , τ E,m } available,
where τ E,i , i ∈ [m] is an expert trajectory (a series of state-action
pairs) drawn independently from expert’s trajectory distribution µ πE ,induced by the expert’s policy πE , initial state distribution S 0 andthe transition probability distribution P. The goal of the learner isto generate π G such that the induced learner’s trajectory distributionµ πG matches µ πE .
Instead of directly minimizing the discrepancy between trajectory distributions to solve IL problem, existing adversary-based
methods turn to the equivalent goal of minimizing the distance
between the learner and expert occupancy measures d(ρ πG , ρ πE ),Íwhere ρ π : S×A → R is defined as ρ π (s, a) = π (a|s) t∞=0 γ t Pr (st =s |π ). Based on the idea of generative adversarial training, they perform policy learning by solving a min-max optimization problem, i.e.
minπG maxD D(ρ πE , ρ πG ), in which D is the discriminator. Amongthese approaches, GAIL [13] achieves state-of-the-art performancein many task environments. The objective of GAIL is

RELATED WORK

Recently, adversary-based methods have achieved great successin a wide range of IL scenarios, including robotic control [13],game playing [12], and simulating environments [22]. As the outputs from the discriminator are utilized as reward signals for thelearner, they can be treated as a generalization of the family of
IRL approaches. Among them, GAIL [13] achieves state-of-the-art
performance in handling low-dimensional IL problems, while its
performance degenerates significantly in high-dimensional environments as discussed in Section 1. Though effective high-dimensional
IL is challenging, there are a few works related to this topic, whichassume additional signals besides provided demonstrations. [12]and [3] augmented RL to learn from both environment rewards
and expert demonstrations, meanwhile [9], [7] and [14] utilized
human preference to enhance IL. In comparison, we consider pureIL without using additional environmental signals, which is morechallenging. There are some related works proposed for this scenario, e.g., CNN-AIRL [26], which uses an adversarial IRL method
to play Atari game Enduro with pixels inputs. But it utilizes autoencoder as the DR method, whose unsupervised information is notenough illustrated in the experiments. D-REX [8] improves [7] bylearning a behavior cloning model to generate the ranked samples,and has achieved promising performance under the IL problems
with suboptimal demonstrations. But in order to train the behavior cloning model efficiently, D-REX requires samples from quite
different experts, which are not available in the settings of most
IL tasks. GIRIL [29] is a non-GAIL method, and deals with thehigh-dimensional IL problem from a different angle than DR, which
encodes action signals into VAE and utilizes the mechanism of
curiosity to produce reward signals. But as will be discussed insubsequent sections, appropriate dimensionality reduction is inevitable for solving high-dimensional IL problems. VAIL [19] is themost closely related work to ours. It improves GAIL by employing
information-theoretic regularization to learn a better feature representation as the input to the discriminator, and successfully used
image feature in a continuous domain. Nevertheless, VAIL does not
utilize explicit supervised loss in DR, which is crucial as shown in
our experiments. In summary, the high-dimensional IL problem
remains challenging for existing IL approaches.
Several recent works [5, 23, 24, 28] have shown that the latenthashing features with unsupervised information learned by autoencoder can help address the exploration issue in challenging RL
problems. Through hashing, a high-dimensional state is effectively
discretized to make similar states mapping into the same hashingcode [15], leading to the convenience for utilizing counting-based

min max Eρ∼ρ πE [log D(ρ)] + Eρ∼ρ πG [log(1 − D(ρ))],
πG

D

(1)

where the discriminator D : S ×A → [0, 1] has the formulation of aclassifier trying to discriminate state-action pairs generated by thelearner and the expert. It is proved that by GAIL, the learned π G canminimize the regularized version of Jensen-Shannon divergence,i.e.,
πG = arg min −H(π ) + d J S (ρ π , ρ πE ),
(2)
π ∈Π

where Π denotes the policy set, d J S (ρ π , ρ πE ) is the Jensen-Shannon
divergence between ρ π and ρ πE , and H(π ) is the causal entropy
used as a policy regularizer. GAIL solves Equation (1) by alternatively taking a gradient ascent step to train the discriminator D anda minimization step to learn policy π G based on off-the-shelf RL algorithm which utilizes − log D(s, a) as the pseudo reward function.

4

HIGH-DIMENSIONAL IMITATIONLEARNING BY HASHREWARD

As learning an effective discriminator plays an essential rule inadversary-based methods, in this section, first we analyze the reason why existing state-of-the-art adversary-based methods fail by

280

Main Track

AAMAS 2021, May 3-7, 2021, Online

even, i.e., D ∈ D ′ implies −D ∈ D ′ . Meanwhile ∀D ∈ D ′ , ∥D ∥ ∞ ≤∆, in which ∥D ∥ ∞ = supτ ∈ T ′ |D(τ )|.

proposing a generalization bound about learning a discriminatorin IL scenarios. Inspired by the theoretical conclusion, then we
propose HashReward to solve the high-dimensional IL problem.

4.1

Assumption 2. dˆD ′ (µ̂ πE,m , µ π̂G ) ≤ η.

Balancing the Discrimination-Rewarding
Trade-Off

Assumption 1 is easily satisfied by general neural network models. Furthermore, if π̂G is sufficiently trained w.r.t. dˆD ′ , then η inAssumption 2 is also small. We then have the following sample
complexity result, whose proof is included in the supplementarymaterial.

Intuitively, a well-trained discriminator should balance the following two capabilities. On the one hand, the discriminator should
be powerful enough, in order to generate negative rewards on
trajectories that are significantly different from the expert’s demonstrations. On the other hand, the discriminator should not be toostrong to discriminate trajectories that are sufficiently similar to
the expert’s demonstrations. This will ensure that positive rewards
are generated on these good trajectories, and encourage the policy to improve in the right direction. We identify this problem
as the discrimination-rewarding trade-off. The reward curves on
Atari game Qbert in Figure 5 evince that the deficiency of state-ofthe-art adversary-based methods (i.e., GAIL) in high-dimensional
environments is due to the tendency of learning too powerful discriminators. Intuitively, this is due to the curse of dimensionality:
We usually cannot collect sufficient expert’s demonstration data
to meet the demand of high-dimensional learning, leading to overfitting of discriminator training. In such a case, dimensionality
reduction (DR) alone is not hard to think of, but the key to findingout a solution is how to do the DR. Thus we start from a theoreticalanalysis motivated by the generalization theory of GAN [30] over
the trajectory learning. Let µ̂ πE,m be expert’s empirical trajectory
distribution obtained from m expert trajectories τ E,i , i ∈ [m], over
the trajectory space T . Generally, we assume that a feature transformation ϕ(τE ), which is a bijective mapping from T to anothertrajectory space T ′ exists. We can see that ϕ plays a crucial rolein the following discussions. For simplicity, we assume that thelearner directly minimizes the neural distance [2] over trajectory
distributions under the mapped feature space, i.e,min [d D ′ (µ̂ πE,m , µ πG )],Theorem 1. Let ∆1 = |d D (µ πE , µ π̂G ) − d D ′ (µ πE , µ π̂G )|, ∆2 =|dˆD ′ (µ̂ πE,m , µ π̂G ) − d D ′ (µ̂ πE,m , µ π̂G )|. Given expert trajectory data X
which consists of m trajectories τ πE ∈ T , if m ≥ 3∥ϕ(X)∥F R, thenwith probability at least 1 − δ , we haverlog(2/δ )d D (µ πE , µ π̂G ) ≤ ∆1 + ∆2 + 6∆2m
(5)
24 ∥ϕ(X)∥ F R
m+(1 + log) + η,m
3 ∥ϕ(X)∥ F R
where R denotes the spectral normalized complexity of D ′ as definedin the supplementary material.
The above result reveals the key factors to learn a good policy.
First, see the sample complexity terms involving m. Under a properly chosen D ′ which ensures small R, the key for sharpening
the bound is to control the Frobenius norm of ϕ(X), which canbe achieved by learning a good DR version of ϕ. Furthermore, ∆1measures the gap of optimal discriminator between D and D ′ .
This shows that ϕ should also be distance-preserving. These two
observations show the advantage of learning a hashing function
as ϕ: It is well-known that hashing mappings usually perform DR
with sparsity as well as distance-preserving property [11], thus fit
for our needs desirably. We should also pay attention to ∆2 , whichmeasures the quality of discriminator training in D ′ . We find thisshould be stressed not only for learning the discriminator under D ′mapped after ϕ, but also for learning ϕ itself. These observations
from Theorem 1 motivate our HashReward approach.

(3)

πG ∈ G

where
d D ′ (µ̂ πE,m , µ πG ) = sup {EτE ∼ µ̂ π
D ∈ D′

4.2[D(ϕ(τE ))]HashReward

E,mAccording to Theorem 1, learning ϕ in a proper way is essential
to our task: A good ϕ should reduce the input norm (∥ϕ(X)∥ F )meanwhile preserving discrimination properties (∆1 and ∆2 ). Thisparticularly shows that directly applying off-the-shelf unsupervised DR, like autoencoder or unsupervised hashing, could lead tounsatisfying results for risks of losing discriminative information.
To achieve a good balance, we propose HashReward, which isa novel adversary-based IL approach utilizing supervised hashing for learning effective discriminators to achieve the balance
between discrimination and rewarding. The key idea lies in using
unsupervised reconstructive information to learn the hashing code
(reducing ∥ϕ(X)∥ F and ∆1 ) as well as leading supervised discriminative information into the whole DR part (reducing ∆2 ), so thatthe hashing code could obtain the ability to represent the originalhigh-dimensional states. To generate such effective representation,
the network structure utilized for the autoencoder and discriminator training is illustrated in Figure 1. We utilize autoencoder totrain the hashing code that maintains reconstructive information(4)− EτG ∼µ πG [D(ϕ(τG )]},

in which G is the policy hypothesis space, and D ′ is the neuralnetwork based discriminator hypothesis space under trajectory
space T ′ . To proceed the analysis, we utilize D to denote the hypothesis space under the original trajectory space T , which isdifferent from D ′ only in input dimension. By solving Equation(3), we expect to obtain a πG that minimizes the expected neural distance in the original trajectory space, i.e., d D (µ πE , µ πG ) =supD ∈D {EτE ∼µ πE [D(ϕ(τE ))] − EτG ∼µ πG [D(ϕ(τG )]} by minimizingd D ′ (µ̂ πE,m , µ πG ), which would guarantee a good πG when D isrich enough. In practice, the optimization in Equation (3) may not
be exactly solved, thus we utilize dˆD ′ , π̂G to denote the resulted
neural distance and policy after training. We further introduce thefollowing assumptions.
Assumption 1. D ′ is a class of neural networks, whose definition
could be found in the supplementary material. Furthermore, D ′ is281

Main Track

AAMAS 2021, May 3-7, 2021, Online

Algorithm 1 HashReward

Overall, the output of HashReward network is D(s, a) ∈ (0, 1), andwe utilize −loдD(s, a) as the pseudo reward for the agent. This process in all experiments can be compared to the network architecturein the supplementary material.
It can be seen that by utilizing Equation (7), the separability of
the hashing codes, which are the inputs to the discriminative layers,
can be reliably preserved even when the dimension of hashing layer
is much smaller than the original input dimension. Instead of fixing
the parameters of the hashing code layer in the training process, theHashReward model and the policy are updated alternatively during
learning. By this way, the training of the hashing code layer andthe discriminator module are coupled together. In order to achieve
faster convergence, a pretraining stage with samples from expert
demonstrations and the random policy for the autoencoder module
is included in learning. The learning procedure of HashReward isillustrated in Algorithm 1.

Input: Expert demonstrations τE ∼ µ πE ; Initialized learner’s policy πG,0 .
1: Pretrain autoencoder with samples from expert demonstrations
and the random policy.
2: for iteration t = 1, 2, · · · ,T do
3:
Utilize πG,t −1 to generate learner’s trajectories, i.e. τG ∼µ πG .
Sample a mini-batch of state-action pairs {(s, a)}t from both4:
τG and τE .
5:
Update HashReward network by Equation (6) using{(s, a)}t , then generate rewards rˆ for all state-action pairs in{(s, a)}t .
6:πG,t −1 → π G,t using rˆ by RL update.
7: end forof the original pixels. Meanwhile, the action signal is concatenatedto the hashing code to formulate the input of the discriminatortraining. By this way, the supervised discriminative information is
directly propagated back to learn hashing codes. The loss function
for discriminator training is divided into two parts, i.e,
L = LH + LD ,5 EXPERIMENT
5.1 Experimental Setup
Environment. We choose 15 games in Arcade Learning Environment [4] and 5 simulators in MuJoCo [25]. The experiment is implemented in OpenAI Gym platform [6], which contains Atari 2600
video games with high-dimensional observation space (raw pixels),
and pixels of each state in MuJoCo are obtained by a camera. The
input states for the learner are set as low-dim continuous control
inputs in MuJoCo. We train converged DQN-based agents as experts in Atari, and DDPG-based [16] agents in MuJoCo. 20 expert
trajectories are collected for each game, also 3 trials with different
random seeds are conducted for each environment. All experiments
are conducted on server clusters with NVIDIA Tesla K80 GPUs.
Contenders. There are five basic contenders in the experiment,i.e., GAIL [13], VAIL [19], GIRIL [29], GAIL with autoencoder (GAILAE) which utilizes only the first two autoencoder loss terms in Equation (7), and GAIL with unsupervised hashing (GAIL-UH) which
utilizes only the first four unsupervised hashing loss terms in Equation (7). The codes of GAIL-AE and VAIL are real numbers, while
those of GAIL-UH and HashReward belong to {−1, 1}. We initialize
autoencoder pretraining for 40M frames of updates for GAIL-AE,
GAIL-UH, GIRIL, and HashReward in Atari (1M in MuJoCo). To
find out whether keeping autoencoder stable during training will
increase the performance of GAIL-AE and GAIL-UH, we conduct
experiments of GAIL-AE and GAIL-UH with updating autoencoder
during training as GAIL-AE-Up and GAIL-UH-Up. Besides, to show
the necessity of hashing for HashReward, we remove the third andfourth terms in Equation (7) as HashReward-AE. The basic RL algorithm is PPO [21], and the reward signals of all methods are scaled
into [0, 1] to enhance the performance of RL part. We set all hyperparameters and network architectures of the policy part the sameto [10]. Also, the hyper-parameters of DR and discriminator for all
methods are the same: The DR and discriminator updates using
Adam with a decayed learning rate of 3e-4; the batch size is 256;
λ is 0.01. We implement VAIL and GIRIL with the recommended
hyper-parameters in their paper. The ratio of update frequency
between the learner and discriminator is 3: 1.

(6)where L H denotes the hashing training loss, which propagateserror for training the autoencoder and hashing code layer, and L D
denotes the discriminator training loss, which is utilized for training
the discriminator layers as well as enhancing the supervision of
DR part. L D is similar to the inner maximization in Equation (1),except that the input state s is replaced by the binary hashing codeb(s). Inspired by Liu et al. [17], we define the hashing loss L H as
L H ({si , yi }, {s j , y j }) = si − si′ 2 + s j − s j′
2

2
2


+ λ ∥1 − |b(si )| ∥22 + ∥1 − b(s j ) ∥22
1
2+ I(yi j ) b(si ) − b(s j ) 22
1
2+ (1 − I(yi j )) max(2l − b(si ) − b(s j ) 2 , 0),2(7)in which l denotes the length of the hashing code and I(yi j ) is theindicator function which takes 1 if yi equals to y j , and 0 otherwise.
In Equation (7), (si , yi ), (s j , y j ) denote a pair of state-label instance.
For one state-label instance (s, y), we utilize s to denote a state
sampled from the learner’s policy or a state from the trajectory
generated by the expert. Furthermore, we utilize y to indicate wheres is sampled, such that y = 1 if s is sampled from the demonstrationand y = 0 otherwise. The first two terms of L H represent the reconstruction error, making the reconstructed states s ′ similar to theoriginal states s. The next two terms (regularization terms weightedby λ) are used to enforce b(s) to get close to binary values in {−1, 1},
where b(s) is the logit output of the hashing layer. The last two
terms in L H are essential for introducing the supervision into hashing code training. From these terms, the unbinarized hashing codesb(si ) and b(s j ) of two states si , s j will get similar only when they
have the same labels. By this way, the discriminative information
is effectively propagated for the learning hashing representations.

282

Main Track

AAMAS 2021, May 3-7, 2021, Online

binary code

expert state

Decoder

Encoderreconstructed
expert stateareconstructed
learner state

learner state

HashReward

Figure 1: Illustration of the HashReward model architecture which contains two modules: the autoencoder module and the discriminator
module. The red solid block represents the concatenation of the hashing code layer and the action signal a. The blue solid block represents
hidden dense layers of the discriminator module.

Table 1: The performance of each method on Atari after 10M timesteps. Boldface numbers indicate the best results. The state
space is 84 × 84 × 4.
BeamRider
Breakout
BoxingBattleZone
ChopperCommand
CrazyClimber
EnduroKangaroo
MsPacmanPong
Qbert
Seaquest
SpaceInvaders
UpNDown
ZaxxonExpert Reward

GAIL

VAIL

GIRIL

GAIL-AE

GAIL-AE-Up

GAIL-UHGAIL-UH-Up

HashReward-AE

HashReward

2139.20 ± 41.60
144.35 ± 29.27
95.70 ± 2.63
23000.00 ± 2549.51
3135.00 ± 145.86
95245.00 ± 2477.39
469.85 ± 18.21
4175.00 ± 94.21
3163.00 ± 160.88
21.00 ± 0.00
4750.00 ± 50.00
1835.00 ± 23.56
743.50 ± 26.03
34200.50 ± 2083.35
11190.00 ± 490.82

854.47±220.63
10.48±1.70
26.78±3.2411863.33±767.09
1469.33±135.22
35451.33±1002.73
26.53±26.35
1695.67±87.96
298.00±22.56-18.40±0.293634.42±388.09
761.13±38.89
341.67±23.38
21086.17±848.61
8130.33±626.00

615.63±258.67
24.32±2.60
2.47±1.55
7566.67±1503.43
1190.00±37.50
41170.67±5024.62
119.87±10.06
1482.00±427.65
1316.87±182.96-18.78±0.183260.62±209.12
826.93±40.99
346.70±27.42
31637.43±2301.67
8473.00±1507.70

2973.27±224.00
61.53±17.98-3.64±1.57
9070.00±2203.47
604.67±52.93
5020.00±599.03
0.00±0.00
32.00±2.83
121.23±75.75-20.14±0.07
1050.33±197.75
628.93±52.75
550.10±12.95
24399.07±9663.36
2644.33±63.73

638.04±95.95
17.32±6.83
26.05±19.83
11030.00±4734.64
1151.67±17.46
10321.00±1539.70
70.02±70.66
935.00±63.90
711.43±9.19-14.57±5.651126.25±416.21
1124.00±400.68
302.23±38.18
20252.53±10582.64
8577.67±852.58

2182.59±1424.69
28.70±0.22-5.47±23.43
9133.33±5255.45
1148.00±74.69
9590.00±5689.36
0.04±0.05
542.00±44.50
674.67±3.07-20.90±0.151303.62±33.38
1151.33±628.84
318.75±36.39
43972.33±1105.54
9062.00±1336.34

1412.91±230.91
1.04±0.52
0.59±1.17
4670.00±1432.29
1144.33±446.19
4049.00±907.23
0.00±0.00
80.00±94.36
655.00±41.36-17.89±0.79
360.33±111.00
689.93±5.04
309.83±74.85
4227.27±1264.39
1614.67±64.63

2089.55±1232.42
49.85±9.78-15.36±15.18
11400.00±3559.52
995.33±362.83
3159.33±2228.60
0.00±0.00
26.33±5.31
701.33±16.21-17.59±0.26
287.17±67.09
664.80±10.47
512.32±49.89
8715.40±4807.56
1546.67±28.77

596.00±8.1432.10±2.84-0.72±1.157043.33±1728.01
924.00±107.34
44766.67±21591.68
209.07±19.05
1352.00±249.99
726.93±42.11-17.97±0.94
4000.50±327.93
1339.80±188.92
496.33±67.44
64648.43±17684.03
9005.00±1404.70

1613.68±203.83
67.73±13.77
84.71±2.1315623.33±278.61
1522.67±78.36
63076.00±1841.86
219.88±72.06
1925.67±145.11
1463.07±52.68-5.25±4.794553.58±155.14
1400.73±67.41
546.48±42.09
36989.70±12739.30
10068.33±410.69

Table 2: The performance of each method on MuJoCo after 1M timesteps. The state space is 84 × 84 × 1.
Humanoid
HalfCheetah
HopperHumanoidStandup
Reacher

5.2

Expert Reward

GAIL

VAIL

GIRIL

GAIL-AE

GAIL-AE-Up

GAIL-UHGAIL-UH-Up

HashReward-AE

HashReward

1029.15 ± 53.95
1189.01 ± 140.28
2304.87 ± 287.45
110722.99 ± 6360.51-10.00 ± 2.83

360.13±9.53-1198.28±344.95
496.31±294.86
67946.69±3690.98-28.99±5.42392.69±9.87-138.48±36.00
2210.22±5.8877412.49±2948.07-74.98±13.82

207.27±20.07-280.27±159.74
975.13±17.02
81217.79±1013.14-13.53±1.31435.95±7.52-488.56±232.77
753.02±60.52
79006.35±2857.10-18.79±1.01422.90±28.37-706.58±339.33
1149.54±309.80
75210.63±2109.55-20.23±0.80

424.11±22.36-402.19±49.28
968.16±16.20
68248.25±3586.59-15.54±0.36

433.74±9.76-420.65±54.38
943.70±149.26
70330.63±2901.27-15.55±0.33

364.61±1.98-149.73±253.05
2116.94±35.24
82718.25±2706.36-100.10±25.82

370.92±3.65-78.92±168.05
2216.83±76.57
83423.17±1878.71-39.35±12.24

Results

Humanoid and Reacher. It also evinces that without supervision,
the change of feature could confuse the discriminator training.
HashReward-AE outperforms GAIL on most games, while remains
a gap compared with HashReward. This demonstrates that hashingis necessary for HashReward even with supervision. As expected,
HashReward outperform its contenders significantly and achieve
the best performance on most of environments (12/15 in Atari, 3/5
in MuJoCo), meanwhile gains expert-level reward on Boxing, Qbert,
SpaceInvaders, UpNDown, Zaxxon and Hopper.
It can be concluded that GAIL could not be improved with simple unsupervised DR; also the usage of supervision only from discriminator (regular term of loss in VAIL) to enhance DR part is
not enough; besides, another potential reason for not that satisfactory results of VAIL on Atari is that its discriminator does not

Experimental results in Atari are reported in Table 1 and Figure 2,
and that of the major methods in MuJoCo are shown in Table 1
and Figure 3. More results including the performances of othercontenders are reported in the supplementary material.
We can observe that VAIL outperforms GAIL on most environments, and achieves near-expert performance on UpNDown and
Hopper, but still fails compared with HashReward. GIRIL achieves
the best performance in BeamRider, SpaceInvaders and Reacher, but
its behavior is also unsatisfactory in some environments. For variants of GAIL, GAIL-AE outperform GAIL on several environments,
but fail on most compared with VAIL and HashReward. Meanwhile,
GAIL-UH only achieve satisfactory performance on BeamRider,283

Main Track

AAMAS 2021, May 3-7, 2021, Online

1e3

1e1

1.0

GAIL
GAIL-AE
GAIL-UH
VAIL
GIRIL
HR

8

3.0

Returns

2.5
2.0
1.5
1.0

6

4

2

0.5
0
0

1

2

Episodes

3

4

0.0

1e3

(a) BeamRider

1.0

1.5

2.0

Episodes

2.5

3.0

3.5

1.25

0.4

1.00

0.2

0.75

0.0

0.50

−0.2

0.25

−0.4

1e4

3.0

6

2.5

1

2

Episodes

3

4

0.5

0
0.2

0.4

0.6

0.8

Episodes

1.0

1.2

1.4

1.0

1.5

2.0

Episodes

0.2

0.4

0.6

0.8

Episodes

1.0

1.2

1.4

1e2

0
1

2

3

Episodes

4

5

6

1e3

−1.5

(k) Qbert

1.0

1.5

Episodes

−2.0

2.0

2.5

2

3

4

5

Episodes

6

7

0.0

1e3

0.2

0.4

0.6

Episodes

0.8

1.0

0.8

1.0

1e4

(j) Pong

1e4

1.0
0.8

3

0.6

2

0.4

1

0.2

0
0.00

1e3

1

4

1
0.5

1e3

1.2

2

0.0

7

−1.0

1e4

3

0.00
0

6

5

4

0.25

5

1e1

(i) MsPacman

5

0.50

4

−0.5

0

1e3

6

0.75

3

Episodes0.0(h) Kangaroo

7

2

0.5

0.0
0.0

1e3

1

(e) ChopperCommand

0.2

0.5

0

1e3

0.4

0.0

1.00

1

6

0.8

1.25

2

5

0.6

1e3

3

Episodes

4

1.2

0.5

1.50
4

3

1e3

(g) Enduro

1e3

2

1.0

(f) CrazyClimber

5

0.25
1

1.4

1.0

0.0

1e4

0.50

1.6

1.5

0.0
0.0

0.75

(d) BattleZone

1e3

1.5
1.0

1.00

(c) Boxing

2.0

1

1.25

0

1e3

4

2

1.50

0.00
0

2.0

3

1e3

1.75

1.50

0.6

1e2

7

5

Returns

1e4

1.75(b) Breakout

1e4

Returns

0.5

1e2

0.8

(l) Seaquest

0.25

0.50

0.75

1.00

1.25

Episodes

1.50

1.75

2.00

0.0
0

1e5

(m) SpaceInvaders

2

4

Episodes

6

8

0.0

1e3

0.2

(n) UpNDown

0.4

0.6

Episodes1e4(o) Zaxxon

Figure 2: Learning curves of on Atari after 10M timesteps, where shaded regions indicate the standard deviation. The state
space is 84 × 84 × 4.

1e2

1e3

4.5

1e3

9

0.0

4.0
3.5

−0.4

−1.5

2.5
2.0

−2.0

1.5

−2.5

1.0

−3.0
0.0

0.2

0.4

0.6

0.8

Episodes(a) Humanoid

1.0

1.2

1e4

1e2

8

1.5

−1.0

3.0

1e4

−0.2

2.0

−0.5

0.0

0.2

0.4

0.6

Episodes0.8(b) HalfCheetah

1.0

1e3

7

1.0

6

0.5

5

0.0

4
0.0

0.2

0.4

0.6

0.8

Episodes(c) Hopper

1.0

1e3

−0.6
−0.8
−1.0
−1.2
0.0

0.2

0.4

0.6

Episodes0.8(d) HumanoidStandup

1.0

1e3

0.0

0.2

0.4

0.6

0.8

Episodes

1.0

1.2

1.4

1e3

(e) Reacher

Figure 3: Learning curves of on MuJoCo after 1M timesteps, where shaded regions indicate the standard deviation. The state
space is 84 × 84 × 1.
utilize action signal, which is of a great importance signal for thesegames [26]; meanwhile, although GIRIL achieves decent performance on some environments, it does not solve DR problems directly, which may limit its performance; and HashReward provides
a powerful approach for tackling high-dimensional IL problems,
which can encourage the learner to generate expert comparable
policies in challenging IL tasks.

5.3

For contenders, they try to learn the ‘kill-shot’ but hit the ball by
the long side of the bar instead of the short side, except for GAILAE-Up, GAIL-UH, GAIL-UH-Up, GIRIL, and VAIL which miss theball. While the sequence of HashReward is the same as that ofdemonstrations, which shows that HashReward has successfully
learned the ‘kill-shot’.
Pseudo Reward Curves. In order to understand why HashReward outperforms its contenders and whether HashReward tackles
the discrimination-rewarding trade-off, first we analyze the true
reward and pseudo reward (generated by reward function) curvesof five basic methods for a single training process on Qbert, illustrated in Figure 5. For GAIL and GAIL-AE, the discriminator seems
to excessively focus on discriminating between the learner’s andexpert’s samples, such that the pseudo reward does not rise evenwhen the true reward has increased, which verifies our perspective

Is HashReward Meaningful?

Comparisons on Pong. To find out whether HashReward has
dug out the expert policy closely, we report the comparisons ofa sequence of expert demonstrations and the sequence generated
by each method during the same period of time on Pong, shownin Figure 4. We can observe that the expert hits the ball with theshort side of the bar, which is the ‘kill-shot’ and hard to imitate.

284

Main Track

AAMAS 2021, May 3-7, 2021, Online(a) Expert(b) GAIL(c) VAIL(d) GIRIL(e) AE(f) AE-Up(g) UH(h) UH-Up(i) HR-AE(j) HR

Figure 4: The sequence generated by each approach on Pong, with the comparison of expert demonstrations (the first column),
where ‘AE’, ‘AE-Up’, ‘UH’, ‘UH-Up’, ‘HR-AE’ and ‘HR’ indicate GAIL-AE, GAIL-AE-Up, GAIL-UH, GAIL-UH-Up, HashRewardAE and HashReward respectively. The timestamp for each row of images and seed for each environment are the same.

×10

GAIL-AE

3

True Reward

4.0

11

Pseudo Reward

×10

3.0
10

2.5

2.00
1.75

8.0

1.50

7.5

2.0

9

1.5

8

1.0
7

0

1

2

3

4

Episodes

5

6
×10

3

×10

2

×10
4.5

8.10

7.0

×10

1.00

3.5

0.95

8.00

1.00

7.95

4

3.0

0.25

0

1

2

3

4

Episodes

5

6
×10

1.5

4

6.5

2

0.85

3

6.0

2
5.5
1

1
5.0

0.70

7.80

3

5

3

0.75

7.85

3

0.90

0.80

7.90

×10

7.5

7.0

2.0
0.50

6.5

HashReward

3

8.05

1.25

0.75

VAIL

1

1.05

4.0

2.5

0.5
6

GAIL-UH

3

8.5

3.5

True Reward

GAIL
PR_agent
PR_expert

12

0

0

1000

2000

3000

4000

5000

Episodes

6000

0

1

2

3

4

Episodes

5

6
×10

3

0

0

1

2

3

4

Episodes

5

6
×10

3

Steps

Figure 5: Pseudo reward generated by discriminator and true reward curves of five basic approaches on Qbert. ‘PR_agent’ and‘PR_expert’ indicate pseudo reward for the agent and expert samples respectively. The blue curve denotes pseudo reward
provided by the discriminator, and the red one denotes the true reward.
100

100

100

100

80

80

80

80

60

60

60

60

40

40

40

40

20

20

20

20

0

Codes
(a) 0 Step

0

0

Codes
(b) 1e6 Steps

Codes(c) 1e7 Steps

0

Codes
(d) Expert

Figure 6: The 64-bit HashReward codes for samples from different policy steps as well as expert on Qbert game. The samples
are gathered from the same continual period within an episode with a fixed random seed.
on why GAIL (and with unsupervised DR) fails in such tasks. For
GAIL-UH, the discriminator fails to discriminate between learner’sand expert’s samples, as the pseudo reward curve for learner almost
overlaps with that for the expert in the whole training process. This
indicates that including supervised information in hashing code
training is indeed essential. For VAIL, the change of the pseudo

reward for agent and expert is similar to that of GAIL, which reveals
that the supervision of VAIL in DR is not enough. For HashReward,
we can observe that the change of HashReward ideally reflectsthat of the true reward. Furthermore, the over-discrimination of285

Main Track

AAMAS 2021, May 3-7, 2021, Online

Figure 7: t-SNE Visualizations of policy samples (blue points) and expert samples (orange points) on Qbert.
pseudo reward is successfully avoided by HashReward. More results from the rest environments and other methods can be foundin the supplementary material, which reflects similar phenomena.
64-bit Embeddings. Besides, to verify whether HashReward
generates meaningful codes, we train a 64-bit embedding versionof HashReward model on Qbert. Afterward we input samples generated from policies of different training stages (after 0 steps, 1e6
steps, and 1e7 steps) to the learned HashReward model to compare
their codes. To collect samples, we run each policy for an episodewith a fixed random seed and collect 100 samples within the sameinterval of time-steps. The codes are demonstrated in Figure 6. At
0 step and 1e6 step, the codes are clearly different from that of
the expert, showing that the policies are not sufficiently trained.
On the contrary, at 1e7 step, as the learner has dug out the latentexpert policy, their codes are close to each other. This shows that
HashReward learns a meaningful embedding space to reflect thequality of imitation.
t-SNE for Embeddings. Furthermore, Figure 7 illustrates tSNE [27] visualization of the embedding layer outputs from five of
the comparison methods under Qbert, and each figure contains 5000
policy and expert samples respectively. We show how embeddingsof expert demonstrations, as well as policy samples, evolve for thesame learning stages as above. We can observe that for GAIL and
GAIL-AE, the embeddings for individual samples remain isolated
throughout training, thus are easy to be fully discriminated. By
using information bottleneck loss, VAIL partially solves this problem by generating some local clustered structures, which howeverstill easy to be distinguished by the discriminator. For GAIL-UH,
even though the embeddings seem to be globally clustered, we canobserve that the policy and expert sample embeddings keep overlapping during the whole training process. This means that it fails
to keep high-dimensional discriminative information when performing DR. For HashReward, we can observe that both expert andpolicy embeddings are globally clustered. Meanwhile, the overlaps
between the two embedding sets increase along with the learning
process, revealing the improvement of policy learning. This verifies
HashReward could desirably dig out the hidden manifold structureof input space, which could lead to successful learning.
The above results reveal the close relationship between making
the discriminator provide ground-truth consistent reward signals
and properly dealing with the discrimination-rewarding trade-off.
Meanwhile, they also indicate that effective supervision is essential
for learning a proper discriminator, which is the key leading to thesuperior performance of HashReward in high-dim IL problems.

6

CONCLUSIONSIn this paper, we tackle the challenging problem of IL in highdimensional environments, under which even state-of-the-art IL algorithms fail. Based on theoretical and empirical studies, we identify
that such failure results from their improper treatment of tacklingthe discrimination-rewarding trade-off. Through this finding, we
propose a novel high-dimensional IL method named HashReward,
which utilizes supervised hashing to learn an effective discriminator, encouraging learners to dig out latent expert policies fromdemonstrations by providing efficient and stable reward signals.
Experiments under both Atari and MuJoCo environments verify theeffectiveness of HashReward, which outperforms state-of-the-art
contenders with significant gaps. We expect HashReward can also
provide inspirations in designing other hashing strategies containing effective semantic information, for solving other challenging IL
problems, e.g., exploration-demanding games like MontezumaRevenge. We will explore more on these possibilities.
Acknowledgment: This research was supported by the NSFC
(61673201, 61921006). The authors would like to thank Yang Yu,
Jieping Ye, and the anonymous reviewers for their insightful comments and suggestions.

286

Main Track

AAMAS 2021, May 3-7, 2021, Online

REFERENCES

8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18),
New Orleans, Louisiana, USA, February 2-7, 2018. 3342–3349.
[16] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom
Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2016. Continuous control
with deep reinforcement learning. In 4th International Conference on Learning
Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track
Proceedings, Yoshua Bengio and Yann LeCun (Eds.).
[17] Haomiao Liu, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2016. Deep Supervised Hashing for Fast Image Retrieval. In 2016 IEEE Conference on Computer
Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016.
2064–2072.
[18] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness,
Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg
Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen
King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015.
Human-level control through deep reinforcement learning. Nature 518, 7540
(2015), 529–533.
[19] Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine.
2019. Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow. In 7th International
Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May
6-9, 2019.
[20] John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp
Moritz. 2015. Trust Region Policy Optimization. In Proceedings of the 32nd
International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July
2015. 1889–1897.
[21] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
2017. Proximal Policy Optimization Algorithms. CoRR abs/1707.06347 (2017).
[22] Jing-Cheng Shi, Yang Yu, Qing Da, Shi-Yong Chen, and Anxiang Zeng. 2019.
Virtual-Taobao: Virtualizing Real-World Online Retail Environment for Reinforcement Learning. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence
Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in
Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1,
2019. 4902–4909.
[23] Alexander L. Strehl and Michael L. Littman. 2008. An analysis of model-basedInterval Estimation for Markov Decision Processes. J. Comput. Syst. Sci. 74, 8(2008), 1309–1331.
[24] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan,John Schulman, Filip De Turck, and Pieter Abbeel. 2017. #Exploration: A Study ofCount-Based Exploration for Deep Reinforcement Learning. In Advances in Neural
Information Processing Systems 30: Annual Conference on Neural Information
Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA. 2750–2759.
[25] Emanuel Todorov, Tom Erez, and Yuval Tassa. [n.d.]. MuJoCo: A physics engine
for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent
Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012.
5026–5033.
[26] Aaron Tucker, Adam Gleave, and Stuart Russell. 2018. Inverse reinforcement
learning for video games. CoRR abs/1810.10593 (2018).
[27] Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using
t-SNE. Journal of Machine Learning Research 9 (2008), 2579–2605.
[28] Haiyan Yin, Jianda Chen, and Sinno Jialin Pan. 2018. Hashing over Predicted
Future Frames for Informed Exploration of Deep Reinforcement Learning. In
Proceedings of the Twenty-Seventh International Joint Conference on Artificial
Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden. 3026–3032.
[29] Xingrui Yu, Yueming Lyu, and Ivor Tsang. 2020. Intrinsic Reward Driven Imitation
Learning via Generative Model. In International Conference on Machine Learning.
6672–6682.
[30] Pengchuan Zhang, Qiang Liu, Dengyong Zhou, Tao Xu, and Xiaodong He. 2018.
On the Discrimination-Generalization Tradeoff in GANs. In 6th International
Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April
30 - May 3, 2018, Conference Track Proceedings.

[1] Pieter Abbeel and Andrew Y. Ng. 2010. Inverse Reinforcement Learning. In
Encyclopedia of Machine Learning. 554–558.
[2] Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. [n.d.]. Generalization and Equilibrium in Generative Adversarial Nets (GANs). In Proceedings ofthe 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW,
Australia, 6-11 August 2017 (Proceedings of Machine Learning Research, Vol. 70),
Doina Precup and Yee Whye Teh (Eds.). 224–232.
[3] Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nandode Freitas. 2018. Playing hard exploration games by watching YouTube. In
Advances in Neural Information Processing Systems 31: Annual Conference on
Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018,
Montréal, Canada. 2935–2945.
[4] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. 2013. The
Arcade Learning Environment: An Evaluation Platform for General Agents. J.
Artif. Intell. Res. 47 (2013), 253–279.
[5] Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman,
Joel Z. Leibo, Jack W. Rae, Daan Wierstra, and Demis Hassabis. 2016. Model-Free
Episodic Control. CoRR abs/1606.04460 (2016).
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym. CoRR abs/1606.01540(2016).
[7] Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. 2019.
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement
Learning from Observations. In Proceedings of the 36th International Conference
on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA.
783–792.
[8] Daniel S. Brown, Wonjoon Goo, and Scott Niekum. 2019. Better-thanDemonstrator Imitation Learning via Automatically-Ranked Demonstrations. In
Proceedings of the 3rd Conference on Robot Learning.
[9] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, andDario Amodei. 2017. Deep Reinforcement Learning from Human Preferences.
In Advances in Neural Information Processing Systems 30: Annual Conference on
Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA,
USA. 4302–4310.
[10] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov.
2017. OpenAI Baselines.
[11] Aristides Gionis, Piotr Indyk, and Rajeev Motwani. 1999. Similarity Search inHigh Dimensions via Hashing. In VLDB’99, Proceedings of 25th International
Conference on Very Large Data Bases, September 7-10, 1999, Edinburgh, Scotland,
UK. 518–529.
[12] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot,
Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arnold,
John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. 2018. Deep Q-learning
From Demonstrations. In Proceedings of the Thirty-Second AAAI Conference on
Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial
Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in
Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018.
3223–3230.
[13] Jonathan Ho and Stefano Ermon. 2016. Generative Adversarial Imitation Learning.
In Advances in Neural Information Processing Systems 29: Annual Conference on
Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain.
4565–4573.
[14] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario
Amodei. 2018. Reward learning from human preferences and demonstrations in
Atari. In Advances in Neural Information Processing Systems 31: Annual Conference
on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018,
Montréal, Canada. 8022–8034.
[15] Qing-Yuan Jiang and Wu-Jun Li. 2018. Asymmetric Deep Supervised Hashing. In
Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the287