Imitation Learning with Human Eye Gaze via Multi-Objective Prediction

Ravi Kumar Thakur * 1 MD-Nazmus Samin Sunbeam * 1 Vinicius G. Goecks 2 Ellen Novoseller 2 Ritwik Bera 1
Vernon J. Lawhern 2 Gregory M. Gremillion 2 John Valasek 1 Nicholas R. Waytowich 2

Abstract
Approaches for teaching learning agents via human demonstrations have been widely studied andsuccessfully applied to multiple domains. However, the majority of imitation learning work utilizes only behavioral information from the demonstrator, i.e. which actions were taken, and ignores
other useful information. In particular, eye gazeinformation can give valuable insight towards
where the demonstrator is allocating visual attention, and holds the potential to improve agent
performance and generalization. In this work,
we propose Gaze Regularized Imitation Learning(GRIL), a novel context-aware, imitation learning
architecture that learns concurrently from both human demonstrations and eye gaze to solve tasks
where visual attention provides important context.
We apply GRIL to a visual navigation task, inwhich an unmanned quadrotor is trained to search
for and navigate to a target vehicle in a photorealistic simulated environment. We show that
GRIL outperforms several state-of-the-art gazebased imitation learning algorithms, simultaneously learns to predict human visual attention,and generalizes to scenarios not present in the
training data. Supplemental videos and code canbe found at https://sites.google.com/
view/gaze-regularized-il/.

1. Introduction
In the human-robot interaction field, imitation learning (IL),also called learning from demonstration, is widely used
to rapidly train artificial agents to mimic the demonstrator via supervised learning (Argall et al., 2009; Osa et al.,*Equal contribution 1 Department of Aerospace Engineering, Texas A&M University, College Station, Texas, USA
2
DEVCOM Army Research Laboratory, Aberdeen Proving
Ground, Maryland, USA. Correspondence to: Ravi Kumar Thakur<ravikt@tamu.edu>, MD-Nazmus Samin Sunbeam <mdsunbeam@tamu.edu>.
Interactive Learning with Implicit Human Feedback Workshop at
ICML 2023.

Figure 1. GRIL system diagram: The proposed multi-objective
learning architecture learns from human demonstrations consistingof actions and eye gaze data to train an imitation learning policy.

2018). IL using human-generated data has been widely
studied and successfully applied to multiple domains suchas self-driving cars (Codevilla et al., 2019), robot manipulation (Rahmatizadeh et al., 2018), and navigation (Silver
et al., 2010). Yet, while IL is a simple and straightforward
approach for teaching intelligent behavior, it suffers fromsample complexity issues when learning an end-to-end behavior policy directly from images, e.g. mapping images
from a robot’s camera to actions.
One avenue to improving IL sample efficiency and generalizability is to learn not only from the demonstrator’s actions,but also from additional signals such eye gaze. While the majority of existing work on IL ignores physiological data (Argall et al., 2009), eye gaze is a rich signal that has beenshown to strongly correlate with visual attention (Doshi &Trivedi, 2012), guide our actions, and filter parts of the environment perceived as relevant (Schütz et al., 2011). Thus,
eye gaze may provide important contextual information
about a person’s thought process and provide an indicationof visual attention that can be leveraged when training AI
agents. For example, when people provide demonstrations
for tasks such as flying a quadrotor via a joystick, eye gazeis often disregarded or ignored, even though it is necessary
to perform the task and can be easily collected via widely
available eye tracking hardware (Poole & Ball, 2006) atalmost no additional burden to the demonstrator. In ourwork, we leverage the demonstrator’s eye gaze, recorded

Gaze Regularized Imitation Learningwith minimal cost during the demonstrations, as a measureof attention in conjunction with human demonstrations toperform imitation learning with less demonstrator data.
Several previous works have utilized eye gaze to improve
imitation learning (Zhang et al., 2018; Saran et al., 2020;
Chen et al., 2019). However, these methods may still require significant human demonstrator time. For instance, thegaze-augmented Atari-HEAD dataset (Zhang et al., 2020)
collects an average of 5.85 hours of gameplay data per Atarigame. Furthermore, the applications considered in theseworks may be significantly simpler than the environment areal-world robot might encounter. For instance, the Atarisetup in Zhang et al. (2018; 2020); Saran et al. (2020); Thammineni et al. (2021) is synchronous and does not resemble
the physical world, the simulated driving setup in Chenet al. (2019) involves following a well-defined track, and thedrone task in Pfeiffer et al. (2022) involves following a reference trajectory (provided to the policy network) without
interacting with any objects in the environment.
We propose Gaze-Regularized Imitation Learning (GRIL), a
novel end-to-end algorithm for learning continuous control
from human demonstrations and eye gaze. GRIL jointly
learns to predict control commands and eye gaze, regularizing policy training via gaze prediction as shown in Figure 1.
This method has the potential to improve robust autonomy
and decrease the amount of demonstration data required tolearn reliable policies in visually complex environments.
We demonstrate GRIL in an autonomous quadrotor control
task in the AirSim (Shah et al., 2018b) simulator, training avisuomotor policy (i.e., combining perception and control)
to navigate a quadrotor to a target object. GRIL is trainedin supervised fashion via a dataset collected from humandemonstrators flying the quadrotor in the simulator, whichcontains the RGB images, human demonstrator’s eye gazecoordinates, and control commands at each timestep. The
proposed method jointly learns to predict gaze coordinatesand control commands. In the quadrotor navigation task,
we hypothesize that the gaze coordinates provide context
relevant to the target, identifying regions of interest in thescene and acting as an attention mechanism to guide thepolicy toward the target. Specifically, we contribute:
1. GRIL, an end-to-end, model-free algorithm with anovel multi-objective architecture for learning fromimages, demonstrator actions, and human eye gaze. To
our knowledge, GRIL is the first IL method to regularize policy learning via a multi-objective architecturethat jointly predicts gaze and policy actions.
2. A demonstration of our gaze-based approach using
an asynchronous, realistic quadrotor navigation taskwith high-dimensional state and action spaces and ahigh-fidelity, photorealistic simulator. To our knowledge, this is the most complex task considered in agaze-augmented imitation learning work, as it requires
continuous control in a photorealistic, outdoor environment without clear markers such as roads denoting
where the agent should travel.
3. A quantitative evaluation showing that GRIL is able
to significantly outperform three baseline methods: astandard behavior cloning model that does not use gaze
as well as two state-of-the-art gaze-augmented behavior cloning models: AGIL (Zhang et al., 2018) andCGL (Saran et al., 2020)
4. We show that after training with demonstrations inwhich the quadrotor navigates to a stationary target,
GRIL generalizes to a novel “seek and follow” task
in which the quadrotor must navigate to and follow amoving target.
5. We provide the first publicly-available gaze-augmented
demonstration dataset in a continuous control task.

2. Related work
Imitation Learning. Imitation learning (IL) is the problem of training a learning agent to act in an environment
given demonstrations. Behavioral cloning (Bratko et al.,
1995; Torabi et al., 2018) is a popular IL method that uses
supervised learning to predict demonstrator actions given
observations. Behavioral cloning is known to suffer fromcovariate shift, in which a model gives poor predictions instates not present in the training data (Ross et al., 2011).
In recent years, a number of other IL approaches havebeen proposed, for instance GAIL (Ho & Ermon, 2016),
IQ-learn (Garg et al., 2021), SQIL (Reddy et al., 2019),
ValueDICE (Kostrikov et al., 2019), Maximum Likelihood
IRL (Jain et al., 2019), EDM (Jarrett et al., 2020), and TREX (Brown et al., 2019). We note that advancements in IL
are orthogonal to this work, since our gaze regularizer canbe paired with any IL method.
Understanding Gaze Behavior. Recent works have proposed a number of methods for estimating human eye gazefrom images (Li et al., 2013; Xia et al., 2020; Wang & Sung,
2002; Cazzato et al., 2020). For instance, Xia et al. (2020)
proposed a periphery-fovea multi-resolution model to predict gaze and show that it improves prediction accuracy in asupervised learning task in a driving domain. While theseworks focus on modeling gaze, our work leverages gaze toimprove performance in IL tasks.
Saran et al. (2019) highlight the importance of understanding gaze behavior for use in IL. For several goal-oriented
robot manipulation tasks, the authors show that users primarily fixate on goal-related objects and that gaze can help
to resolve ambiguities in subtask classification. Meanwhile,
Guo et al. (2021) demonstrate that reinforcement learning
agents achieve better performance when they attend to simi-

Gaze Regularized Imitation Learninglar visual targets to humans, further indicating that humaneye gaze can be leveraged to guide policy learning.
Gaze-Augmented Imitation Learning. IL approaches
that augment demonstrations with the demonstrator’s eye
gaze have been shown to improve policy performance andgeneralization compared to IL methods without gaze (Zhang
et al., 2018; 2020; Saran et al., 2019; 2020; Thammineni
et al., 2021; Liu et al., 2021; Kim et al., 2020; 2021; 2022;
Pfeiffer et al., 2022; Chen et al., 2019) in domains such as
Atari, simulated driving, and robot manipulation.
These works propose a range of approaches for leveraging
gaze in IL. Several methods use predicted or actual gaze
to modulate either the control policy network or its input.
For instance, some approaches that crop an image observation around a user’s gaze (Kim et al., 2020; 2021; 2022)
have shown promise in robot manipulation tasks requiring
precision. Liu et al. (2021) similarly propose to directly
modulate input images based on gaze. Liu et al. (2021) and
Chen et al. (2019) consider using gaze to control dropout
rates in the policy network’s convolution layers. Evaluatedon a simulated driving task, Liu et al. (2021) show that boththe image modulation and dropout methods reduced generalization error compared to IL without gaze, with dropout
outperforming the image modulation method. Meanwhile,
Saran et al. (2019) also leverages gaze in robot manipulationand proposes an inverse reinforcement learning approachto learn rewards from demonstrations with gaze. However,
the method requires knowing the positions of all objects ofinterest to which a human might attend, and thus does notnecessarily generalize across domains.
A number of works (Zhang et al., 2018; Thammineni et al.,2021; Pfeiffer et al., 2022; Xia et al., 2020) estimate gazefrom image observations and then leverage the gaze predictions as a control policy input. Zhang et al. (2018) develop
Attention Guided Imitation Learning (AGIL), which trains agaze prediction network modeled as a human-like foveation
system and then uses the gaze predictions to train a policy.
Evaluated in Atari domains, AGIL outperforms a baseline
without gaze. However, to eliminate the effect of humanreaction time and fatigue, eye gaze was collected in a synchronous fashion in which the environment only advanced
to the next state once the human took an action, and gametime was limited to 15 minutes, followed by a 20-minute rest
period; this highlights the challenges of real-time humandata collection. Thammineni et al. (2021) build on AGIL
by introducing a gating model that selectively passes gazeinformation to the policy network when it predicts that gaze
will be useful. While this method is shown to outperform
AGIL, the contribution is orthogonal to ours, as such a gating module could be straightforwardly paired with GRIL.
Unlike these works, GRIL incorporates gaze as an auxiliary
loss, rather than as a policy input.

Saran et al. (2020) propose a coverage-based gaze loss
(CGL) for IL. This auxiliary loss penalizes the policy network for having low network activations in areas where thehuman gaze is focused. CGL outperforms AGIL (Zhang
et al., 2018) and dropout-based modulation (Chen et al.,2019) with the Atari-HEAD (Zhang et al., 2020) dataset.
While GRIL and CGL both regularize policy learning via
auxiliary losses, they are fundamentally different; whereas
CGL penalizes network activations, which requires setting
hyperparameters that transform the human gaze from coordinates to a heatmap, GRIL utilizes gaze directly by training
a two-headed network to jointly predict actions and gaze.
Moreover, GRIL’s approach to utilizing gaze is distinct from
CGL and AGIL in that both baselines predict and utilizea gaze heatmap, while GRIL does not utilize a heatmap.
Bypassing the need for a heatmap is desirable, since GRIL
does not need to specify a kernel hyperparameter and canuse an MSE loss to compare coordinates, which is simpler
than using a KL loss to compare distributions.
We believe that our work considers a more challenging
environment than these previous works on gaze-augmented
IL. Many works, including state-of-the-art approaches such
AGIL (Zhang et al., 2018) and CGL (Saran et al., 2020),
are exclusively tested in Atari domains, which can easily
be stopped and started to synchronously align with human
input, are temporally discrete, have discrete action spaces,and do not resemble the real world. In contrast, we consider
a quadrotor navigation application that requires continuous
control and complex scene understanding. Some works
consider car driving tasks (Chen et al., 2019; Xia et al.,2020), which also require continuous control; however, ourquadrotor control task is more complex than the driving
tasks in Chen et al. (2019); Xia et al. (2020), as their action
spaces have fewer degrees of freedom. While Pfeiffer et al.
(2022) also consider drone control, they train the drone tofollow reference trajectories such as a figure-8, which does
not involve interacting with other objects in the scene. Also,in addition to a gaze estimate, the policy network in Pfeiffer
et al. (2022) is given the reference trajectory and drone state(rotation and linear and angular velocity) as inputs. GRIL
does not utilize such information.
Lastly, while the Atari-HEAD dataset (Zhang et al., 2020)
provides a set of gaze-augmented demonstrations in Atari
domains, we are not aware of a prior publicly-available
dataset of gaze-augmented demonstrations in a continuous
control setting.

3. Gaze-Regularized Imitation LearningThis work considers a gaze-augmented visual imitation
learning setting, in which a learning agent aims to imitate ahuman demonstrator given the human’s visual observations,actions, and eye gaze.

Gaze Regularized Imitation Learning1
gaze
coords

1

64

2

25
6

1

12
8

1

fc5

fc6

51
2

fc4
1

fc3

deconv3

25

1
MobileNet

6

12

1

8

fc5

1

fc6

fc7

1

4

1

32

fc2

conv1

64

54

1

conv2

4

deconv2
deconv1

actions

51

2

fc4
1

fc3

Figure 2. GRIL learns to jointly predict gaze and control commands via a multi-headed convolutional neural network. Gaze prediction
provides context relevant to the visual scene and assists the model to predict well-performing control commands.

3.1. Problem Statement

3.2. Gaze-Regularized Architecture

We model the gaze-augmented visual imitation learning problem as an episodic partially-observable Markov
decision process without reward (POMDP\R), M =(S, O, A, P, Pe , µ, T ), where S is the underlying state
space, O is the observation space, A is the action space, P :S × A × S → [0, 1] yields the state transition probabilities,
Pe : S × O → [0, 1] gives the observation emission probabilities, µ : S → [0, 1] gives the initial state probabilities,
and T is the time horizon. A policy is a possibly-stochastic
mapping from observations to actions, π : O → A. The
learning agent interacts with the environment in rollout trajectories of the form τ = (o1 , a1 , . . . , oT , aT , oT +1 ).

Our proposed approach is a visuomotor policy based on amulti-headed convolutional neural network, as seen in Figure 2. The model is trained to take RGB images as input andto jointly estimate their corresponding control commands
and gaze coordinates. This joint estimation of gaze andcontrol regularizes the network during training to predictthe most suitable set of control commands. The gaze estimation also provides context to the model by highlighting
key properties of the image observations. The architecture
makes use of weight sharing to reduce the total number ofmodel parameters.

We assume that the observations o ∈ O are images in
RH×W ×C , with height H, width W , and number of channels C (e.g. C = 3 with RGB images). Furthermore, thelearning agent has access to a set of gaze-augmented demonstrations consisting of M observation-action-gaze triples:2
De = (oi , ai , gi )M
i=1 , where oi ∈ O, ai ∈ A, and gi ∈ R
are the demonstrator’s gaze coordinates. The demonstration data is assumed to be generated by an unknown humandemonstrator policy πh .
Learning Objective. The imitation learning objective is
to identify an optimal policy π ∗ with minimal discrepancy
from the demonstrator policy πh :π ∗ = argminπ

TX

Eo∼dπt [L(πh (o), π(o)))],(1)

We leverage a pre-trained MobileNet (Sandler et al., 2018)
to extract features from the images; while the MobileNet
weights are used to initialize our model, we further fine-tune
these weights using the human demonstrations. MobileNet
is trained on real-world images and is suitable for processing
outdoor images. We remove the model’s final layer and feed
its output features through another set of convolution layers.
The features extracted from the image are passed throughtwo different sets of dense layers, which respectively predict
control commands and gaze coordinates.
3.3. Training Criterion
We train the GRIL model via a linear combination of a gazeprediction loss, LGP , and a behavioral cloning loss, LBC :L(θ) = λ1 ∗ LGP (θ) + λ2 ∗ LBC (θ),

(2)

t=1

where dπt is the distribution over observations at timestept induced by following policy π, and L is a discrepancy
measure between two actions (e.g. MSE error).

where θ represents the set of trainable model parameters,and λ1 and λ2 are hyperparameters weighting the loss components. Each loss term is the mean-squared error between

Gaze Regularized Imitation Learning

the corresponding ground truth and predicted values:
M

LBC (θ) =1 X
2∥πaction (oi | θ) − ai ∥ ,M i=1

LGP (θ) =1 X
2∥πgaze (oi | θ) − gi ∥ ,
M i=1

(3)

M

(4)

where M is the number of training samples, πaction (o | θ)and πgaze (o | θ) respectively denote the outputs of the action
and gaze prediction heads given observation o and model
parameters θ, and recall that De = (oi , ai , gi )M
i=1 is thedemonstration dataset.

4. Experiments
4.1. Autonomous Drone Navigation
We evaluate the performance of GRIL in an autonomous
quadrotor navigation domain in which experiments wereconducted in simulated environments rendered by the Unreal Engine using Microsoft AirSim (Shah et al., 2018a), a
high-fidelity, photo-realistic drone simulator. We consider asearch and navigate task in which the quadrotor must seeka target vehicle that is initially out-of-view and navigate toward it in a cluttered forest simulation environment, seen in
Figure 3. The environment emulates sun glare and presents
trees and uneven rocky terrain as obstacles to navigationand visual identification of the target vehicle. We consider
two variants of the drone navigation task:
1. Stationary target: the target truck is in a location thatis fixed across episodes, and the target does not move.
2. Moving target: we use this task to evaluate how well
GRIL can generalize to a target tracking and following
task on which it was not previously trained. In thistask, the target vehicle moves along a fixed, predetermined path at a fixed speed. This task requires not only
seeking out the target vehicle, but also tracking andfollowing the vehicle.
4.2. Dataset Collection Procedure
While performing the task, the demonstrator was given
access to the quadrotor’s first-person view, with no additional information about its location or the target location.
Using an Xbox One joystick, the demonstrator controlled
the quadrotor throttle and yaw rate using the left joystickand controlled its forward and lateral velocity via the rightjoystick, as is standard for aerial vehicles. The eye gazedata collection was conducted using a screen-mounted eye
tracker, which was calibrated specifically for the user. Appendix A describes the eye tracking hardware setup, datacollection, and calibration procedure.
We define a set of 24 possible quadrotor starting locations,Figure 3. First-person view from the quadrotor illustrating the target vehicle (yellow truck), which the agent must find and navigate
toward, and the realistic cluttered forest simulation environment
rendered using Microsoft AirSim (Shah et al., 2018a).

which cover a pre-defined area of the map (see Figure 4),preventing invalid initial locations such as inside the groundor trees or on the target vehicle, while covering the desiredtask area. The human demonstrator collected 90 demonstration trajectories in which the drone navigated to the
fixed-location target. In each demonstration trajectory, thequadrotor’s starting location was sampled randomly fromthe set of 24 possible locations, and the quadrotor’s initialheading was also randomly sampled between 0 and 360
degrees. During the demonstrations, we recorded RGB
images (224x224x3) and eye gaze coordinates. We address observed imbalances in the dataset as described inAppendix B.
We evaluate each learned policy by rolling it out in the environment. To evaluate each policy, we use a set of 10 quadrotor starting points; these are distinct from the 24 starting
locations used during training and are depicted in Figure 4
in magenta. For each evaluation rollout starting position, thequadrotor’s initial heading is sampled randomly. For each
evaluated model, we perform 5 evaluation rollouts at eachof the 10 evaluation start points, for a total of 5 × 10 = 50
evaluation rollouts. Using the same set of evaluation starting
locations across all models helps to ensure evaluation consistency across approaches. Note that the randomly sampled
heading is held constant for the 5 rollouts in each location,
though it varies between locations.
4.3. Evaluation Metrics
We evaluate task performance in the evaluation rollouts via
the following metrics (Anderson et al., 2018):
1. Task completion rate (TCR): this is the percent of
episodes that are successful. In the stationary targettask, a rollout is successful if the quadrotor navigateswithin 5 meters of the target vehicle’s center of mass.

Gaze Regularized Imitation Learningfrom the authors’ implementation; we change the networkarchitecture to process our dataset and its associated outputs
and change the loss function from a classification loss to aregression loss.
The CGL baseline calculates an auxiliary loss that penalizes
the policy network’s final convolution layer for having low
activations where the human’s gaze is focused. Our auxiliary
loss predicts gaze rather than penalizing network weights.
As mentioned below and in the discussion, GRIL seems torequire significantly less tuning than CGL. As with AGIL,
we adapt the authors’ implementation to our task.
4.5. Results

Figure 4. Bird’s-eye view of the target vehicle location and drone
start locations. The magenta points are the drone start locationsin the demonstration dataset, the green points are the drone startlocations during evaluation, and the single yellow point is the fixedlocation of the target vehicle in the stationary target task.

In the moving target task, an episode is similarly successful if the quadrotor navigates within 5 meters of
the in-motion truck’s center of mass.
2. Collision rate (CR): the percent of episodes in which
the agent collides with the ground or any obstacles inthe environment such as rocks and trees. The current
episode is terminated when a collision occurs.
4.4. Comparison Methods
We compare GRIL with three imitation learning baseline methods: vanilla behavioral cloning (BC), AttentionGuided Imitation Learning (AGIL) (Zhang et al., 2018),
and behavior cloning with a context-aware gaze loss
(BC+CGL) (Saran et al., 2020). Appendix D specifies network architecture details for all baseline comparisons, while
Appendix C specifies hyperparameter values for all methods,
as well as hyperparameter ranges tested.
The BC baseline is trained to predict control commands
directly from input images without gaze, whereas GRIL
jointly predicts control commands and gaze coordinates.
Note that the BC and GRIL network architectures are analogous, except that GRIL has a gaze regularization head thatis not present in the BC model.
In AGIL, a gaze prediction network is trained to estimate agaze heatmap, which is then inputted to the policy network.
In contrast, GRIL leverages gaze prediction as a regularizer rather than a policy network input. Since AGIL wasdesigned for discrete control of a 2D Atari game, we developed a custom variant of AGIL for performing continuous
quadrotor control. Our implementation of AGIL is adapted

Algorithm Performance. Table 1 displays the performance of GRIL and all baseline comparisons. We see that
GRIL yields the strongest performance, achieving the highest task completion rates (TCR) and lowest collision rates(CR) compared to the baseline methods. Meanwhile, Figure 6 provides an example qualitative comparison between
the methods by depicting the evaluation rollout trajectories
corresponding to a particular starting location in the stationary task. At this evaluation location, GRIL and BC-CGL
successfully navigate to the target location, while BC mostly
crashes with the terrain early and AGIL drifts away.
Generalizing GRIL to Following a Moving Target. We
next evaluate how well GRIL generalizes to following amoving target when only trained with stationary target
demonstrations. In this moving target task, we evaluate
the same policies that were trained to search for and navigate toward a stationary target; the dataset in Section 4.2
does not include any demonstrations in which the target
moved. This task is more difficult than the stationary targettask, both because the policy must generalize to a task onwhich it was not trained and because the policy must trackand follow the moving target.
Table 1 shows the performance of GRIL and comparisonson the moving target task. During these evaluation rollouts,
GRIL, BC, and BC-CGL demonstrated a remarkable capability to generalize to the new task despite not having beenpreviously trained on it, often successfully searching for andfollowing the moving target. We see that GRIL outperforms
each of the baseline methods in the moving target task, indicating that GRIL is able to generalize to novel tasks morerobustly than previous state-of-the-art approaches. Qualitatively, we noticed that GRIL sometimes lost track of
the target if the vehicle left visual sight; however, we also
noticed that GRIL demonstrated recovery behaviors if thetarget re-appeared in sight. We provide videos of GRIL
and baseline methods performing the moving target task on
our supplemental website, https://sites.google.
com/view/gaze-regularized-il/.

Gaze Regularized Imitation Learning(a) “Motion leading” gaze pattern.

(b) “Target fixation” gaze pattern.

(c) “Saccade” gaze pattern.

(d) “Obstacle fixation” gaze pattern.

Figure 5. Visualization of a sequence of three frames illustrating characteristic gaze patterns observed in the human demonstrations,showed as magenta circles in the figure, which were also learned by the proposed model, showed as cyan circles (best viewed in color).
The larger, less transparent circle illustrates the current gaze observation and the smaller, more transparent circles represent gaze (and gazepredictions) from previous timesteps.

GRIL
TCR (%)

CR (%)BC-CGLTCR (%)

CR (%)BC
TCR (%)

CR (%)AGIL
TCR (%)

CR (%)

Stationary 80 (±9.9) 20 (±9.9) 64 (±14.8) 36 (±14.8) 40 (±13.7) 58 (±13.5) 10 (±10.0) 60 (±16.3)
Moving 40 (±12.3) 40 (±9.4) 36 (±14.8) 42 (±15.9) 30 (±12.7) 66 (±12.3) 14 (±10.3) 86 (±10.3)
Table 1. Performance of GRIL and baselines in the stationary and moving target tasks. Notably, the moving target task evaluates themodels’ ability to transfer to a new task, since the demonstration data only included a stationary target. For both tasks, we compare
the performance of GRIL, BC, AGIL (Zhang et al., 2018), and BC-CGL (Saran et al., 2020). We show the collision rate (CR) and taskcompletion rate (TCR); values are mean (± standard error) over 10 starting locations, with 5 rollouts per starting location each in thestatic and moving target tasks.

Figure 6. Evaluation rollout trajectories for GRIL and baselinecomparisons in the stationary task. For one of the evaluation
starting locations, we depict five evaluation rollouts corresponding
to each method. The task consists of navigating from the starting
location (purple dot) to the target vehicle (yellow dot).

Gaze prediction results. With respect to GRIL’s gaze prediction performance, we observe that several distinct gaze
patterns present in the human demonstrations were also
learned by GRIL’s gaze prediction head, as seen in Figure 5:a) a “motion leading” gaze pattern where gaze attends tothe sides of the images followed by a yaw motion in thesame direction, as illustrated in Figure 5(a). This pattern ismostly observed at the beginning of the episode when thetarget vehicle is not in the agent’s field-of-view; b) a “target
fixation” gaze pattern where gaze is fixated on the target during the final approach, as illustrated in Figure 5(b). In thispattern, gaze is fixed at the top of the target vehicle, independent of the agent’s current motion; c) a “saccade” gaze
pattern where gaze rapidly switches between fixation on
nearby obstacles (Salvucci & Goldberg, 2000), as illustrated
in Figure 5(c). This pattern is characteristic when there are
multiple obstacles between the current agent location andthe target vehicle; and d) an “obstacle fixation” gaze patternwhere the gaze attends to nearby obstacles when navigating
to the target, as illustrated in Figure 5(d). This pattern ismostly observed when the quadrotor is close to a potential
obstacle. This illustrates how GRIL’s gaze prediction headis able to capture and replicate similar visual attention cues
demonstrated by the user when performing the task.

5. Discussion
We propose the GRIL algorithm for leveraging human eye
gaze in a context-aware imitation learning framework. GRIL
leverages gaze via a novel, multi-objective optimization ap-

Gaze Regularized Imitation Learning

proach that jointly predicts control commands and gaze. The
gaze prediction loss helps to regularize the policy network,so that it focuses on relevant parts of the image observations.
This novel use of eye gaze makes the model sample efficient,practical, and generalizable. Our experiments demonstrate
GRIL’s performance in a continuous control setting featuring a quadrotor in an outdoor environment, in which GRIL
outperforms state-of-the-art baseline methods AGIL and
CGL. Though our experiments train the model to perform
search and navigation toward a single stationary target, weshow that GRIL can also perform the task with a moving
target, and furthermore, that GRIL generalizes to the newtask more robustly than baseline comparisons.
We speculate that, because GRIL is optimizing multiple
objectives with action and gaze prediction heads, it may
enable more robust and efficient learning. The benefit behind using a multi-objective optimization framework is twofold. Firstly, if a task is noisy or data is limited and highdimensional, it can be difficult for a model to differentiate
between relevant and irrelevant features. Multi-objective
learning helps the model to focus its attention on the featuresthat most matter, since optimizing multiple objectives simultaneously forces the model to learn relevant features thatare invariant across each objective (Caruana, 1993; AbuMostafa, 1990; Ruder, 2017). Finally, a multi-objective
learning framework, as proposed by this work, enables eye
gaze prediction loss to act as a regularizer by introducing
an inductive bias. As such, it reduces the risk of overfitting
as well as the Rademacher complexity of the model, i.e. its
ability to fit random noise (Baxter, 2000).
We hypothesize that GRIL may outperform CGL in part
because it incorporates gaze in a more direct manner requiring fewer hyperparameters. While GRIL utilizes thegaze coordinate directly, CGL requires a gaze heatmap, forwhich one must specify a resolution and degree of spread
about the gaze coordinates. CGL’s heatmap also results in amore complex loss function leveraging the KL divergence,
while GRIL relies on a simpler MSE loss. We also notethat CGL imposes no additional network parameters to fit,
whereas GRIL imposes an additional network head as it
simultaneously predicts both control commands and gazecoordinates.
In regards to why GRIL significantly outperforms AGIL, wehypothesize that AGIL requires significantly more demonstration training samples compared to GRIL due to its separate gaze prediction network. AGIL requires first traininga gaze prediction network to model human foveation bylearning a gaze prediction heatmap, and then uses the gazepredictions to train a separate policy model. In contrast,
GRIL combines both the gaze prediction network and policy network in a single model with branching network heads
and shared weights. This significantly reduces the numberof parameters to learn.
Future work. Future work will include evaluating GRIL in
additional environments, as well as to study the use of eye
gaze in physical robot experiments. We are also excited toleverage eye gaze regularization in other problem settings,
for instance multi-task learning and reinforcement learning.
Another interesting avenue for future work is the addition of
more human input modalities to the proposed approach, forinstance natural language, to condition the model to perform
multiple diverse tasks. The ability to use eye gaze data toleverage human visual attention opens the door to adapting
this research to learning unified policies that can generalize
across multiple human input types, task contexts, and task
objectives.

References
Abu-Mostafa, Y. S. Learning from hints in neural networks.
Journal of complexity, 6(2):192–198, 1990.
Anderson, P., Chang, A., Chaplot, D. S., Dosovitskiy, A.,
Gupta, S., Koltun, V., Kosecka, J., Malik, J., Mottaghi, R.,
Savva, M., et al. On evaluation of embodied navigation
agents. arXiv preprint arXiv:1807.06757, 2018.
Argall, B. D., Chernova, S., Veloso, M., and Browning, B.
A survey of robot learning from demonstration. Robotics
and autonomous systems, 57(5):469–483, 2009.
Baxter, J. A model of inductive bias learning. Journal of
artificial intelligence research, 12:149–198, 2000.
Bratko, I., Urbančič, T., and Sammut, C. Behaviouralcloning: phenomena, results and problems. IFAC Proceedings Volumes, 28(21):143–149, 1995.
Brown, D., Goo, W., Nagarajan, P., and Niekum, S. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International
conference on machine learning, pp. 783–792. PMLR,
2019.
Caruana, R. Multitask learning: A knowledge-based sourceof inductive bias. In Proceedings of the Tenth International Conference on International Conference on Machine Learning, ICML’93, pp. 41–48, San Francisco, CA,
USA, 1993. Morgan Kaufmann Publishers Inc. ISBN
1558603077. doi: 10.5555/3091529.3091535.
Cazzato, D., Leo, M., Distante, C., and Voos, H. When i look
into your eyes: A survey on computer vision contributions
for human gaze estimation and tracking. Sensors, 20(13):3739, 2020.
Chen, Y., Liu, C., Tai, L., Liu, M., and Shi, B. E. Gaze training by modulated dropout improves imitation learning. In

Gaze Regularized Imitation Learning2019 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS), pp. 7756–7761. IEEE, 2019.

Kingma, D. P. and Ba, J. Adam: A method for stochastic
optimization. arXiv preprint arXiv:1412.6980, 2014.

Codevilla, F., Santana, E., López, A. M., and Gaidon, A.
Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9329–9338,
2019.

Kostrikov, I., Nachum, O., and Tompson, J. Imitation learning via off-policy distribution matching. In International
Conference on Learning Representations, 2019.

Doshi, A. and Trivedi, M. M. Head and eye gaze dynamics
during visual attention shifts in complex environments.
Journal of vision, 12(2):9–9, 2012.
Garg, D., Chakraborty, S., Cundy, C., Song, J., and Ermon,
S. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:
4028–4039, 2021.
Guo, S. S., Zhang, R., Liu, B., Zhu, Y., Ballard, D., Hayhoe,M., and Stone, P. Machine versus human attention in
deep reinforcement learning tasks. Advances in Neural
Information Processing Systems, 34:25370–25385, 2021.

Li, Y., Fathi, A., and Rehg, J. M. Learning to predictgaze in egocentric video. In Proceedings of the IEEE
international conference on computer vision, pp. 3216–
3223, 2013.
Liu, C., Chen, Y., Liu, M., and Shi, B. E. Using eye gaze toenhance generalization of imitation networks to unseen
environments. IEEE Transactions on Neural Networks
and Learning Systems, 32(5):2066–2074, 2021. doi: 10.
1109/TNNLS.2020.2996386.
Osa, T., Pajarinen, J., Neumann, G., Bagnell, J. A.,
Abbeel, P., and Peters, J. An algorithmic perspectiveon imitation learning. Foundations and Trends® in
Robotics, 7(1-2):1–179, 2018. ISSN 1935-8253. doi:
10.1561/2300000053. URL http://dx.doi.org/
10.1561/2300000053.

Ho, J. and Ermon, S. Generative adversarial imitation
Pfeiffer, C., Wengeler, S., Loquercio, A., and Scaramuzza,learning. In Lee, D., Sugiyama, M., Luxburg, U.,
D. Visual attention prediction improves performance
Guyon, I., and Garnett, R. (eds.), Advances in Neuralof autonomous drone racing agents. Plos one, 17(3):Information Processing Systems, volume 29. Curran Ase0264471, 2022.
sociates, Inc., 2016. URL https://proceedings.
neurips.cc/paper/2016/file/
Poole, A. and Ball, L. J. Eye tracking in hci and usability
cc7e2b878868cbae992d1fb743995d8f-Paper.
research. In Encyclopedia of human computer interaction,
pdf.
pp. 211–219. IGI Global, 2006.
Jain, V., Doshi, P., and Banerjee, B. Model-free irl using
maximum likelihood estimation. In Proceedings of the
AAAI Conference on Artificial Intelligence, volume 33,
pp. 3951–3958, 2019.
Jarrett, D., Bica, I., and van der Schaar, M. Strictly batch
imitation learning by energy-based distribution matching.
Advances in Neural Information Processing Systems, 33:
7354–7365, 2020.
Kim, H., Ohmura, Y., and Kuniyoshi, Y. Using humangaze to improve robustness against irrelevant objects inrobot manipulation tasks. IEEE Robotics and Automation
Letters, 5(3):4415–4422, 2020.
Kim, H., Ohmura, Y., and Kuniyoshi, Y. Gaze-based dual
resolution deep imitation learning for high-precision dexterous robot manipulation. IEEE Robotics and Automation Letters, 6(2):1630–1637, 2021.
Kim, H., Ohmura, Y., and Kuniyoshi, Y. Memory-based
gaze prediction in deep imitation learning for robot manipulation. In 2014 IEEE international conference on
robotics and automation (ICRA), 2022.

Rahmatizadeh, R., Abolghasemi, P., Bölöni, L., and Levine,
S. Vision-based multi-task manipulation for inexpensive
robots using end-to-end learning from demonstration. In
2018 IEEE International Conference on Robotics andAutomation (ICRA), pp. 3758–3765. IEEE, 2018.
Reddy, S., Dragan, A. D., and Levine, S. Sqil: Imitation
learning via reinforcement learning with sparse rewards.
In International Conference on Learning Representations,2019.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online
learning. In Proceedings of the fourteenth international
conference on artificial intelligence and statistics, pp.
627–635. JMLR Workshop and Conference Proceedings,
2011.
Ruder, S. An overview of multi-task learning in deep neuralnetworks, 2017.
Salvucci, D. D. and Goldberg, J. H. Identifying fixations andsaccades in eye-tracking protocols. In Proceedings of the2000 symposium on Eye tracking research & applications,pp. 71–78, 2000.

Gaze Regularized Imitation Learning

Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., andChen, L.-C. Mobilenetv2: Inverted residuals and linear
bottlenecks. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), June
2018.

Zhang, R., Liu, Z., Zhang, L., Whritner, J. A., Muller, K. S.,
Hayhoe, M. M., and Ballard, D. H. Agil: Learning attention from human for visuomotor tasks. In Proceedings ofthe european conference on computer vision (eccv), pp.
663–679, 2018.

Saran, A., Short, E. S., Thomaz, A., and Niekum, S. Understanding Teacher Gaze Patterns for Robot Learning. In
Conference on Robot Learning (CoRL), 2019.

Zhang, R., Walshe, C., Liu, Z., Guan, L., Muller, K., Whritner, J., Zhang, L., Hayhoe, M., and Ballard, D. Atari-head:
Atari human eye-tracking and demonstration dataset. In
Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 6811–6820, 2020.

Saran, A., Zhang, R., Short, E. S., and Niekum, S. Efficiently guiding imitation learning algorithms with humangaze. In Proceedings of the AAAI Conference on Artificial
Intelligence, 2020.
Schütz, A. C., Braun, D. I., and Gegenfurtner, K. R. Eye
movements and perception: A selective review. Journalof Vision, 11(5):9–9, 09 2011. ISSN 1534-7362. doi: 10.
1167/11.5.9. URL https://doi.org/10.1167/
11.5.9.
Shah, S., Dey, D., Lovett, C., and Kapoor, A. AirSim: Highfidelity visual and physical simulation for autonomousvehicles. In Hutter, M. and Siegwart, R. (eds.), Field andService Robotics, pp. 621–635, Cham, 2018a. Springer
International Publishing. ISBN 978-3-319-67361-5. doi:
10.1007/978-3-319-67361-5 40.
Shah, S., Dey, D., Lovett, C., and Kapoor, A. Airsim: Highfidelity visual and physical simulation for autonomousvehicles. In Field and service robotics, pp. 621–635.
Springer, 2018b.
Silver, D., Bagnell, J. A., and Stentz, A. Learning fromdemonstration for autonomous navigation in complex unstructured terrain. The International Journal of Robotics
Research, 29(12):1565–1592, 2010.
Thammineni, C., Manjunatha, H., and Esfahani, E. T. Selective eye-gaze augmentation to enhance imitation learningin atari games. Neural Computing and Applications, pp.
1–10, 2021.
Torabi, F., Warnell, G., and Stone, P. Behavioral cloning
from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp.
4950–4957, 2018.
Wang, J.-G. and Sung, E. Study on eye gaze estimation.
IEEE Transactions on Systems, Man, and Cybernetics,
Part B (Cybernetics), 32(3):332–350, 2002.
Xia, Y., Kim, J., Canny, J., Zipser, K., Canas-Bajo, T., andWhitney, D. Periphery-fovea multi-resolution driving
model guided by human attention. In The IEEE WinterConference on Applications of Computer Vision, pp. 1767–
1775, 2020.

Gaze Regularized Imitation Learning

A. Details of the gaze data collection
A.1. Eye Tracker Calibration
Before an eye tracking recording is started, the user is taken through a calibration procedure. During this procedure, the eye
tracker measures characteristics of the user’s eyes and uses them together with an internal, anatomically-correct 3D modelof the human eye to calculate the requisite gaze data. This model includes information about shapes, light refraction, andreflection properties of the different parts of the eyes (e.g. cornea, placement of the fovea, etc.). During the calibration, theuser is asked to look at specific points on the screen. These specific dots are also known as calibration dots. During thisperiod several images of the human operator’s eyes are collected and analyzed. The resulting information is then taken into
account along with the eye model and the gaze coordinates (abcissa and ordinate in the screen frame) for each image.
A.2. Gaze Data Collection

Figure 7. Data collection setup used to record human gaze and joystick data illustrating the relative positioning between the user, task
display, joystick, and eye tracker. The user is given only the first-person quadrotor view while performing the visual navigation task.

The task was presented to the demonstrator on a 23.8 inches display with 1920x1080 pixel resolution, and eye gaze datacollection was conducted using a screen-mounted eye tracker positioned at a distance of 61cm from the demonstrator’s eye, as
seen in Figure 7. Before data collection, the height of the demonstrator’s chair was adjusted in order to position their head at
the optimal location for gaze tracking. The eye tracker sensor was calibrated according to an 8-point manufacturer-provided
software calibration procedure. The demonstrator avoided moving the chair and minimized torso movements during datacollection while moving the eye naturally. The demonstrator was also given time to acclimate to the task until they judged
for themselves that they were confident in performing it. The dataset was collected by a single human demonstrator.

B. Addressing Data Imbalance
Data imbalance. The size of the dataset is limited due to human participation. During exploratory data analysis on thecollected dataset, it was found that the yaw-axis actions were mostly centered around zero (see Figure 8), such that the agent
has many more examples of flying straight than of turning.
Data augmentation and undersampling. We performed data augmentations to address the dataset imbalance. First,
we created a horizontally-flipped version of each demonstration trajectory. The images were horizontally flipped withcorresponding changes in the horizontal gaze coordinate and control commands (i.e., roll and yaw are reversed, while pitch
and throttle are unchanged). Then we performed random undersampling of states with zero yaw, in which we rejected 90%of the zero yaw data samples. Figure 8 shows the distribution of control commands before and after the data augmentations
were applied.

Gaze Regularized Imitation Learning

Figure 8. The distribution of control commands reflects the forward motion bias of the dataset. Most of the states have yaw commands
centered around zero. This creates an imbalance in the dataset, which may result in a visuomotor policy where the vehicle learns to fly
only straight.

Weighted BC loss function. For GRIL and all baseline comparisons, we predict actions by weighting the mean squared
error BC loss LBC (θ) given by Equation (2). To help counter the forward motion bias in the dataset, the yaw control
command was weighted more highly than the other three control parameters.
M

Lweighted
(θ) =
BC

1 X
2
∥w ⊙ (πaction (oi | θ) − ai )∥ ,M i=1

which is analogous to the BC loss in (2) except that w is a vector in which the ith element weights the ith control commandcomponent, and ⊙ denotes an element-wise product. Recall that M is the number of experience tuples in the demonstration
dataset.
We manually tuned the weighting parameters w, and use a weight of 0.7 for the yaw command and weights of 0.1 for theother control components.

C. Hyperparameters
We keep the following hyperparameters constant across all methods. The loss function is optimized using the Adam (Kingma& Ba, 2014) optimizer with a learning rate of 0.0003 for 30 epochs and batch size of 32 data samples. For training themodel we have used an exponential decay function to schedule the decrease in learning rate. The decay occurs every 10,000
steps during model training.
For CGL, the loss function is a weighted sum of a BC term and the CGL term:
LCGL (θ) = λ1 ∗ LBC (θ) + λ2 ∗ LCGL (θ),
where λ1 and λ2 are hyperparameters that determine the relative weighting between the terms. We tune λ1 and λ2 manually,
and the optimized model uses λ1 = 0.9 and λ2 = 0.1. GRIL has analogous λ1 and λ2 hyperparameters that determine theweighting between the BC and gaze-regularization losses.
C.1. BC-CGL
We tuned the following two gaze hyperparameters for BC-CGL: the kernel size of the gaze heatmap and the loss weighting.
For the kernel size, the parameters considered were 3, 5, 10, and 15. For the loss weighting (action weight, gaze weight),
the weightings considered were (0.95, 0.05), (0.9, 0.1), (0.7, 0.3), and (0.5, 0.5). In the table, bolded values indicate thehyperparameter values found to be optimal.
Gaze Heatmap Kernel Size
3
5
10
15
Loss Weighting (action weight, gaze weight) (0.95, 0.05) (0.9, 0.1) (0.7, 0.3) (0.5, 0.5)

Gaze Regularized Imitation Learning

C.2. AGIL
The gaze hyperparameter for AGIL was the kernel size of the gaze heatmap. For the kernel size, the parameters consideredwere 3, 5, 10, and 15. In the table, the bolded value indicates the hyperparameter value found to be optimal.
Gaze Heatmap Kernel Size 3 5 10 15
C.3. GRIL
The gaze hyperparameter for GRIL was the loss weighting. For the loss weighting (action weight, gaze weight), theweightings considered were (0.95, 0.05), (0.9, 0.1), (0.7, 0.3), and (0.5, 0.5). In the table, the bolded entry indicates thehyperparameter combination found to be optimal.
Loss Weighting (action weight, gaze weight) (0.95, 0.05) (0.9, 0.1) (0.7, 0.3) (0.5, 0.5)

D. Baseline Model Architectures
We present in Figures 9, 10, 11, and 12 the network architectures used for each baseline method: BC, Attention-Guided
Imitation Learning (AGIL) (Zhang et al., 2018), and Behavior Cloning with a Context-aware Gaze Loss (BC+CGL) (Saranet al., 2020), respectively.

Figure 9. Standard behavior cloning model where only RGB images are used as input to the model.

Figure 10. AGIL policy model architecture where RGB images and gaze heatmaps are used as an input to the model. The heatmap at thebottom-left is generated as shown in Figure 11.

Gaze Regularized Imitation Learning

Figure 11. AGIL gaze prediction model architecture where RGB images are used as input and a predicted gaze heatmap is the output.
This model produces the heatmap seen in Figure 10.

Figure 12. BC+CGL where RGB images are being used as an input to the model.