Predicting Gaze in Egocentric Video by Learning
Task-dependent Attention Transition

arXiv:1803.09125v1 [cs.CV] 24 Mar 2018

Yifei Huang, Minjie Cai, Zhenqiang Li, Yoichi SatoThe University of Tokyo
Tokyo, Japan
Abstract
We present a new computational model for gaze prediction in egocentric videos by exploring patterns in temporal shift of gaze fixations (attention transition) that are dependent on egocentric manipulation tasks. Our assumption is that the high-level context of how a task is completed in a certain way has a strong influence on attention transition and should be modeled for gaze prediction
in natural dynamic scenes. Specifically, we propose a hybrid model based on deep neural networks which integrates
task-dependent attention transition with bottom-up saliency
prediction. In particular, the task-dependent attention transition is learned with a recurrent neural network to exploit
the temporal context of gaze fixations, e.g. looking at a
cup after moving gaze away from a grasped bottle. Experiments on public egocentric activity datasets show that ourmodel significantly outperforms state-of-the-art gaze prediction methods and is able to learn meaningful transitionof human attention.

1. Introduction
With the increasing popularity of wearable or action
cameras in recording our life experience, egocentric vision,
which aims at automatic analysis of videos captured froma first-person perspective, has become an emerging fieldin computer vision. In particular, as the camera wearer’s
point-of-gaze in egocentric video contains important information about interacted objects and the camera wearer’s intent, gaze prediction can be used to infer important regionsin images and videos to reduce the amount of computationneeded in learning and inference of various analysis tasks[7, 34].
This paper aims to develop a computational model for
predicting the camera wearer’s point-of-gaze from an egocentric video. Most previous methods have formulated gazeprediction as the problem of saliency detection, and computational models of visual saliency have been studied to thefind image regions that are likely to attract human attention. The saliency-based paradigm is reasonable because
it is known that highly salient regions are strongly correlated with actual gaze locations [22]. However, the saliency
model-based gaze prediction becomes much more difficultin natural dynamic scenes, e.g. cooking in a kitchen, where
high-level knowledge of the task has a strong influence onhuman attention.
In a natural dynamic scene, a person perceives the surrounding environment with a series of gaze fixations which
point to the objects/regions related to the person’s interactions with the environment. It has been observed that theattention transition is deeply related to the task carried out
by the person. Especially in object manipulation tasks, thehigh-level knowledge of an undergoing task determines a
stream of objects or places to be attended successively andthus influences the transition of human attention. For example, to pour water from a bottle to a cup, a person always first looks at the bottle before grasping it and thenchange the fixation onto the cup during the action of pouring. Therefore, we argue that it is necessary to explore thetask-dependent patterns in attention transition in order toachieve accurate gaze prediction.
In this paper, we propose a hybrid gaze prediction
model that combines bottom-up visual saliency with taskdependent attention transition learned from successively attended image regions in training data. The proposed model
is mainly composed of three modules. The first module
generates saliency maps directly from video frames. It
is based on a two-stream Convolutional Neural Network(CNN) which is similar to traditional bottom-up saliency
prediction models. The second module is based on a recurrent neural network and a fixation state predictor which
generates an attention map for each frame based on previously fixated regions and head motion. It is built based on
two assumptions. Firstly, a person’s gaze tends to be located on the same object during each fixation, and a large
gaze shift almost always occurs along with large head motion [18]. Secondly, patterns in the temporal shift between
regions of attention are dependent on the performed task

and can be learned from data. The last module is based ona fully convolutional network which fuses the saliency map
and the attention map from the first two modules and generates a final gaze map, from which the final prediction of 2D
gaze position is made.
Main contributions of this work are summarized as follows:• We propose a new hybrid model for gaze predictionthat leverages both bottom-up visual saliency and taskdependent attention transition.
• We propose a novel model for task-dependent attention transition that explores the patterns in the temporal shift of gaze fixations and can be used to predict theregion of attention based on previous fixations.
• The proposed approach achieves state-of-the-art gazeprediction performance on public egocentric activity
datasets.

2. Related Works
2.1. Visual Saliency Prediction.
Visual saliency is a way to measure image regions thatare likely to attract human attention and thus gaze fixation[1]. Traditional saliency models are based on the feature integration theory [32] telling that an image region with highsaliency contains distinct visual features such as color, intensity and contrast compared to other regions. After Itti etal.’s primary work [15] on a computational saliency model,
various bottom-up computational models of visual saliency
have been proposed such as a graph-based model [10] and aspectral clustering-based model [12]. Recent saliency models [20, 13, 21] leveraged a deep Convolutional Neural Network (CNN) to improve their performance. However, thesaliency models do not consider top-down image semantics or high-level context information such as certain objects closely related to a task that a viewer is doing and,as a result, these models often fail to model actual humanattention.
There is an increasing amount of work that have exploited top-down mechanisms or high-level context tomodel visual saliency. Early methods either tried to bias
bottom-up features according to high-level knowledge ofthe goal and objects [9, 33], or constructed a separate topdown stream for visual saliency and fused the two streams
afterwards [24, 2]. In [31], low-level features were combined with high-level scene context for saliency prediction.
More recently, high-level context has been considered indeep learning-based saliency models. In [27, 4], class labels were used to compute the partial derivatives of CNN
response with respect to input image regions to obtain aclass-specific saliency map. In [38], a salient object is detected by combining global context of the whole image andlocal context of each image superpixel. In [25], region-toword mapping in a neural saliency model was learned byusing image captions as high-level input.
However, none of the previous methods explored thepatterns in the transition of human attention inherent in acomplex task. In this work, we propose to learn the taskdependent attention transition on how gaze shifts between
different objects/regions to better model human attention innatural dynamic scenes.

2.2. Egocentric Gaze Prediction.
Egocentric vision is an emerging research domain incomputer vision which focuses on automatic analysis ofegocentric videos recorded with wearable cameras. Egocentric gaze is a key component in egocentric vision which
benefits various egocentric applications such as action
recognition [7] and video summarization [34]. Although
there is correlation between visually salient image regions
and gaze fixation locations [22], it has been found that traditional bottom-up models for visual saliency is insufficient
to model and predict human gaze in egocentric video [35].
Yamada et al. [36] presented a gaze prediction model by
exploring the correlation between gaze and head motion. In
their model, bottom-up saliency map is integrated with anattention map obtained based on camera rotation and translation to infer final egocentric gaze position. Li et al. [19]explored different egocentric cues like global camera motion, hand motion and hand positions to model egocentricgaze in hand manipulation activities. They built a graphicalmodel and further combined the dynamic behaviour of gazeas latent variables to improve the gaze prediction. However, their model is dependent on predefined egocentric cues
and may not generalize well to other activities where hands
are not always involved. Recently, Zhang et al. [37] proposed the gaze anticipation problem in egocentric videos.
In their work, a Generative Adversarial Network (GAN)
based model is proposed to generate future frames from acurrent video frame, and gaze positions are predicted on thegenerated future frames based on a 3D-CNN based saliency
prediction model.
In this paper, we propose a new hybrid model to predictgaze in egocentric videos, which combines bottom-up visual saliency with task-dependent attention transition. To
the best of our knowledge, this is the first work to explore
the patterns in attention transition for egocentric gaze prediction. By learning task-dependent attention transition, ourmodel can exploit the temporal context of gaze fixations
which greatly improves gaze prediction accuracy.

(
#"&'

$
#"&'

!"&'

Attention Transition Module

CONV

S-CNN

CONV

."

Fixation state predictor

CONV

#"%

Late Fusion

LSTM-based
weight predictor

CONV

Channel weight
extractor

)"+
ℒℒ-

#"$

/"&0,"

T-CNN
Feature Encoding Module

Saliency Prediction Module

#"%

)"*

#"$

)"
!"

Figure 1. The architecture of our proposed gaze prediction model. The red crosses in the figure indicate ground truth gaze positions.

3. Gaze Prediction ModelIn this section, we first give overview of the network architecture of the proposed gaze prediction model, and thenexplain the details of each component. The details of training the model are provided in the end.

3.1. Model Architecture
Given consecutive video frames as input, we aim to predict a gaze position in each frame. To leverage both bottomup visual saliency and task-dependent attention transition,
we propose a hybrid model that 1) predicts a saliency map
from each video frame, 2) predicts an attention map by exploiting temporal context of gaze fixations, and 3) fuses thesaliency map and the attention map to output a final gazemap.
The model architecture is shown in Figure 1. The feature encoding module is composed by a spatial Convolutional Neural Network (S-CNN) and a temporal Convolutional Neural Network (T-CNN), which extract latent representations from a single RGB image and stacked opticalflow images respectively. The saliency prediction module
generates a saliency map based on the extracted latent representation. The attention transition module generates anattention map based on previous gaze fixations and headmotion. The late fusion module combines the results ofsaliency prediction and attention transition to generate a final gaze map. The details of each module will be given in
the following part.

3.2. Feature Encoding
At time t, the current video frame It and stacked opticalflow Ot−τ,t are fed into S-CNN and T-CNN to extract latent
representations FtS = hS (It ) from the current RGB frame,
and FtT = hT (Ot−τ,t )from the stacked optical flow images

for later use. Here τ is fixed as 10 following [28].
The feature encoding network of S-CNN and T-CNN
follows the base architecture of the first five convolutional
blocks in Two Stream CNN [28], while omitting the final
max pooling layer. We choose to use the output feature mapof the last convolution layer from the 5-th convolutionalgroup, i.e., conv5 3. Further analysis of different choicesof deep feature maps from other layers is described in Section 4.4.

3.3. Saliency Prediction Module
Biologically, human tends to gaze at an image region
with high saliency, i.e., a region containing unique and distinctive visual features [30]. In the saliency prediction module of our gaze prediction model, we learn to generate avisual saliency map which reflects image regions that arelikely to attract human gaze. We fuse the latent representations FtS and FtT as an input to a saliency prediction decoder (denoted as S) to obtain the initial gaze prediction
map Gst (Eq. 1). We use the “3dconv + pooling” method of[8] to fuse the two input feature streams. Since our task isdifferent from [8], we modify the kernel sizes of the fusionpart, which can be seen in detail in Section 3.7. The decoder
outputs a visual saliency map with each pixel value within
the range of [0, 1]. Details of the architecture of the decoder
is described in Section 3.7. The equation for generating thevisual saliency map is:
Gst = S(FtS , FtT )(1)

However, a saliency map alone does not predict accurately where people actually look [35], especially in egocentric videos of natural dynamic scenes where the knowledge of a task has a strong influence on human gaze. To
achieve better gaze prediction, high-level knowledge about

!"#$

Mean
Pooling

Crop

ℒ345

LSTM-based weight predictor

Channel weight extractor%" = '"#$ %"#$ + 1 − '"#$ ,(%"#$ )

LSTM

%"

%"#$

0
/"#$

⋅
6"7

ℒ895
FC

FC

FC
reshape

1
/"#$

'"#$
/"0

Fixation state predictor

Figure 2. The architecture of the attention transition module.

a task, such as which object is to be looked at and manipulated next, has to be considered.

3.4. Attention Transition Module
During the procedure of performing a task, the taskknowledge strongly influences the temporal transition ofhuman gaze fixations on a series of objects. Therefore,
given previous gaze fixations, it is possible to anticipate theimage region where next attention occurs. However, directmodeling the object transition explicitly such as using object categories is problematic since a reliable and genericobject detector is needed. Motivated by the fact that different channels of a feature map in top convolutional layers correspond well to spatial responses of different highlevel semantics such as different object categories [5][39],we represent the region that is likely to attract human attention by weighting each channel of the feature map differently. We train a Long Short Term Memory (LSTM) model[11] to predict a vector of channel weights which is used
to predict the region of attention at next fixation. Figure 2
depicts the framework of the proposed attention transition
module. The module is composed of a channel weight extractor (C), a fixation state predictor (P), and a LSTM-based
weight predictor (L).
The channel weight extractor takes as input the latentS
representation Ft−1
and the predicted gaze point gt−1 fromS
the previous frame. Ft−1
is in fact a stack of feature maps
with spatial resolution 14 × 14 and 512 channels. Fromeach channel, we project the predicted gaze position gt−1
onto the 14×14 feature map, and crop a fixed size area withheight Hc and width Wc centered at the projected gaze position. We then average the value of the cropped feature
map at each channel, obtaining a 512-dimensional vector ofchannel weight wt−1 :S
wt−1 = C(Ft−1, gt−1 )(2)

where C(·) indicates the cropping and averaging operation,
wt−1 is used as feature representation of the region of attention around the gaze point at frame t − 1.
The fixation state predictor takes the latent representaTtion of Ft−1
as input and outputs a probabilistic score ofpTfixation state ft−1
= P (Ft−1) ∈ [0, 1]. Basically, the score
tells how likely fixation is occurring in the frame t − 1. The
fixation state predictor is composed by three fully connected
layers followed by a final softmax layer to output a probabilistic score for gaze fixation state.
We use a LSTM to learn the attention transition by learning the transition of channel weights. The LSTM is trained
based on a sequence of channel weight vectors extracted
from images at the boundaries of all gaze fixation periods
with ground-truth gaze points, i.e. we only extract one channel weight vector for each fixation to learn its transition between fixations. During testing, given a channel weight vector wt−1 , the trained LSTM outputs a channel weight vector
L(wt−1 ) that represents the region of attention at next gaze
fixation. We also consider the dynamic behavior of gazeand its influence on attention transition. Intuitively speaking, during a period of fixation, the region of attention tend
to remain unchanged, and the attended region changes only
when saccade happens. Therefore, we compute the regionof attention at current frame wt as a linear combination ofprevious region of attention wt−1 and the anticipated regionof attention at next fixation L(wt−1 ), weighted by the prep
dicted fixation probability ft−1:pp
· wt−1 + (1 − ft−1
) · L(wt−1 )wt = ft−1(3)

Finally, an attention map Gat is computed as the weighted
sum of the latent representation FtS at frame t by using theresulting channel weight vector wt :
Gat =

n
X
c=1

wt [c] · FtS [c](4)where [c] denotes the c-th dimension/channel of wt /FtS respectively.

3.5. Late Fusion
We build the late fusion module (LF) on top of thesaliency prediction module and the attention transition module, which takes Gst and Gat as input and outputs the predicted gaze map Gt .
Gt = LF (Gst , Gat )(5)

Finally, a predicted 2D gaze position gt is given as the spatial coordinate of maximum value of Gt .

3.6. Training
For training gaze prediction in saliency prediction module and late fusion module, the ground truth gaze map Ĝ
is given by convolving an isotropic Gaussian over the measured gaze position in the image. Previous work used either
Binary Cross-Entropy loss [17], or KL divergence loss [37]between the predicted gaze map and the ground truth gaze
map for training neural networks. However, these loss functions do not work well with noisy gaze measurement. A
measured gaze position is not static but continuously quivers in a small spatial range, even during fixation, and conventional loss functions are sensitive to small fluctuationsof gaze. This observation motivates us to propose a newloss function, where the loss of pixels within small distancefrom the measured gaze position is down-weighted. Moreconcretely, we modify the Binary Cross-Entropy loss function (Lbce ) across all the N pixels with the weighting term
1 + di as:
Lf (G, Ĝ) = −N
1 X(1 + di ) Ĝ[i] · log(G[i])+
N i=1

(6)

(1 − Ĝ[i]) · log(1 − G[i])where di is the euclidean distance between ground truth
gaze position and the pixel i, normalized by the image
width.
For training the fixation state predictor in the attention
transition module, we treat the fixation prediction of eachframe as a binary classification problem. Thus, we use theBinary Cross-Entropy loss function for training the fixation
state predictor. For training the LSTM-based weight predictor in the attention transition module, we use the meansquared error loss function across all the n channels:nLmse (wt , ŵt ) =1X(wt [i] − ŵt [i])2n i=1

where wt [i] denotes the i-th element of wt .

(7)

3.7. Implementation details
We describe the network structure and training details
in this section. Our implementation is based on the PyTorch [23] library. The feature encoding module follows
the base architecture of the first five convolutional blocks(conv1 ∼ conv5) of VGG16 [29] network. We remove
the last max-pooling layer in the 5-th convolutional block.
We initialize these convolutional layers using pre-trained
weights on ImageNet [6]. Following [28], since the input channels of T-CNN is changed to 20, we average theweights of the first convolution layer of T-CNN part. The
saliency prediction module is a set of 5 convolution layergroups following the inverse order of VGG16 while changing all max pooling layers into upsampling layers. Wechange the last layer to output 1 channel and add sigmoid
activation on top. Since the input of the saliency prediction
module contains latent representations from both S-CNN
and T-CNN, we use a 3d convolution layer (with a kernel
size of 1 × 3 × 3) and a 3d pooling layer (with a kernel sizeof 2 × 1 × 1) to fuse the inputs. Thus, the input and output sizes are all 224 × 224. The fixation state predictor isa set of fully connected (FC) layers, whose output sizes are
4096,1024,2 sequentially. The LSTM is a 3-layer LSTM
whose input and output sizes are both 512. The late fusion
module consists of 4 convolution layers followed by sigmoid activation. The first three layers have a kernel size of3 × 3, 1 zero padding, and output channels 32,32,8 respectively, and the last convolution layer has a kernel size of 1
with a single output channel. We empirically set both theheight Hc and width Wc for cropping the latent representations to be 3.
The whole model is trained using Adam optimizer [16]with its default settings. We fix the learning rate as 1e-7 andfirst train the saliency prediction module for 5 epochs forthe module to converge. We then fix the saliency prediction
module and train the LSTM-based weight predictor and the
fixation state predictor in the attention transition module.
Learning rates for other modules in our framework are all
fixed as 1e-4. After training the attention transition module,
we fix the saliency prediction and the attention transition
module to train the late fusion module in the end.

4. Experiments
We first evaluate our gaze prediction model on two public egocentric activity datasets (GTEA Gaze and GTEA
Gaze Plus). We compare the proposed model with otherstate-of-the-art methods and provide detailed analysis of ourmodel through ablation study and visualization of outputsof different modules. Furthermore, to examine our model’sability in learning attention transition, we visualize outputof the attention transition module on a newly collected test
set from GTEA Gaze Plus dataset (denoted as GTEA-sub).

Further results can be seen on our project page 1 .

4.1. Datasets
We introduce the datasets used for gaze prediction andattention transition.
GTEA Gaze contains 17 video sequences of kitchentasks performed by 14 subjects. Each video clip lasts forabout 4 minutes with the frame rate of 15 fps and an image resolution of 480 × 640. We use videos 1, 4, 6-22 as a
training set and the rest as a test set as in Yin et al. [19].
GTEA Gaze Plus contains 37 videos with the frame rateof 24 fps and an image resolution of 960 × 1280. In thisdataset each of the 5 subjects performs 7 meal preparation
activities in a more natural environment. Each video clip
is 10 to 15 minute long on average. Similarly to [19], gazeprediction accuracy is evaluated with 5-fold cross validation
across all 5 subjects.
GTEA-sub contains 227 video frames selected from thesampled frames of GTEA Gaze Plus dataset. Each selected
frame is not only under a gaze fixation, but also contains
the object (or region) that is to be attended at the next fixation. We manually draw bounding boxes on those regions
by inspecting future frames. The dataset is used to examine whether or not our model trained on GTEA Gaze Plus
(excluding GTEA-sub) has successfully learned the taskdependent attention transition.

4.2. Evaluation Metrics
We use two standard evaluation metrics for gaze prediction in egocentric videos: Area Under the Curve (AUC) [3]and Average Angular Error (AAE) [26]. AUC is the area
under a curve of true positive rate versus false positive rate
for different thresholds on the predicted gaze map. It is
a commonly used evaluation metric in saliency prediction.
AAE is the average angular distance between the predicted
and the ground truth gaze positions.

4.3. Results on Gaze Prediction
4.3.1

Baselines.

We use the following baselines for gaze prediction:• Saliency prediction algorithms: We compare ourmethod with several representative saliency prediction
methods. More specifically, we used Itti’s model [14],Graph Based Visual Saliency (GBVS [10]), and a deepneural network based saliency model as the current
state of the art (SALICON [13]).
• Center bias: Since egocentric gaze data is observed tohave a strong center bias, we use the image center as
the predicted gaze position as in [19].
1 https://cai-mj.github.io/project/egocentric_gaze_prediction

Metrics
Itti et al. [14]GBVS [10]SALICON [13]Center bias
Yin ta al. [19]DFG [37]
Our full model

GTEA Gaze Plus

GTEA Gaze

AAE (deg) AUC AAE (deg) AUC
19.9
14.7
15.6
8.6
7.9
6.6
4.0

0.753
0.803
0.818
0.819
0.867
0.952
0.957

18.4
15.3
16.5
10.2
8.4
10.5
7.6

0.747
0.769
0.761
0.789
0.878
0.883
0.898

Table 1. Performance comparison of different methods for gazeprediction on two public datasets. Higher AUC (or lower AAE)
means higher performance.

• Gaze prediction algorithms: We also compare ourmethod with two state-of-the-art gaze prediction methods: the egocentric cue-based method (Yin et al. [19]),
and the GAN-based method (DFG [37]). Note that although the goal of [37] is gaze anticipation in future
frames, it also reported gaze prediction in the current
frame.
4.3.2

Performance Comparison.

The quantitative results of different methods on two datasetsare given in Table 4.3.2. Our method significantly outperforms all baselines on both datasets, particularly on the
AAE score. Although there is only a small improvement on
the AUC score, it can be seen that previous method of DFG[37] has already achieved quite high score and the spaceof improvement is limited. Besides, we have observed fromexperiments that high AUC score does not necessarily meanhigh performance of gaze prediction. The overall performance on GTEA Gaze is lower than that on GTEA Gaze
Plus. The reason might be that the number of training samples in GTEA Gaze is smaller and over 25% of ground truth
gaze measurements are missing. It is also interesting to seethat the center bias outperforms all saliency-based methods
and works only slightly worse than Yin et al. [19] on GTEA
Gaze Plus, which demonstrates the strong spatial bias ofgaze in egocentric videos.
4.3.3

Ablation Study.

To study the effect of each module of our model, and the effectiveness of our modified binary cross entropy loss (Equation 6), we conduct an ablation study and test each component on both GTEA Gaze Plus and GTEA Gaze datasets.
Our baselines include: 1) single-stream saliency prediction
with binary cross entropy loss (S-CNN bce and T-CNNbce), 2) single-stream saliency prediction with our modified
bce loss (S-CNN and T-CNN), 3) two-stream saliency pre-

SP

AT

Our full model

Ground truth

International Conference on Computer V

1a

1b

Frame 7113

…

…

…

…

…

…

…

…

2a

2b

3a

Frame 4116

3b

Frame 4122

…

…

…

…

4a

Frame 9651

4b

Frame 9659

Figure 3. Visualization of predicted gaze maps from our model. Each group contains two images from two consecutive fixations, where ahappens before b. We show the output heatmap from the saliency prediction module (SP) and the attention transition module (AT) as wellas our full model. The ground truth gaze map (the rightmost column) is obtained by convolving an isotropic Gaussian on the measured
gaze point.

GTEA Gaze plus

Metrics 













9.90
10.08
9.72
9.6
9.53
9.43
9.51
7.58

0.854
0.854
0.857
0.859
0.860
0.861
0.857
0.898

Table 2. Results of ablation study.

diction with bce loss (SP bce), 4) two-stream input saliency
prediction with our modified bce loss (SP), 5) the attention
transition module (AT), and our full model.
Table 4.3.3 shows the results of the ablation study. The
comparison of the same framework with different loss functions shows that our modified bce loss function is more suitable for the training of gaze prediction in egocentric video.
The SP module performs better than either of the singlestream saliency prediction (S-CNN and T-CNN), indicatingthat both spatial and temporal information are needed for
accurate gaze prediction. It is important to see that the AT
module performs competitively or better than the SP module. This validates our claim that learning task-dependent
attention transition is important in egocentric gaze prediction. More importantly, our full model outperforms all separate components by a large margin, which confirms that thebottom-up visual saliency and high-level task-dependent attention are complementary cues to each other and should beconsidered together in modeling human attention.
Visualization.

Figure 3 shows qualitative results of our model. Group (1a,1b) shows a typical gaze shift: the camera wearer shifts his
attention to the pan after turning on the oven. SP fails tofind the correct gaze position in (1b) only from visual features of the current frame. Since AT exploits the high-leveltemporal context of gaze fixations, it successfully predicts
the region to be on the pan. Group (2a, 2b) demonstrates a“put” action: the camera wearer first looks at the target location, then puts the can to that location. It is interesting that
AT has learned the camera wearer’s intention, and predicts
the region at the target location rather than the more salient
hand region in (2a). In group (3a, 3b), the camera wearer
searches for a spatula after looking at the pan. Again, AT
has learned this context which leads to more accurate gazeprediction than SP. Finally, group (4a, 4b) shows that SP
and AT are complementary to each other. While AT performs better in (4a), and SP performs better in (4b), the full

0.893
0.906
0.905
0.906
0.918
0.928
0.940
0.957



5.61
6.15
5.57
6.07
5.63
5.52
5.02
4.05

 

 

  

AAE (deg) AUC AAE (deg) AUC

S-CNN (bce)
T-CNN (bce)
S-CNN
T-CNN
SP (bce)SP
AT
Our full model

4.3.4



GTEA Gaze




 






 





























Figure 4. AUC and AAE scores of cross task validation. Five different experiment settings (explained in the text below) are compared to study the differences of attention transition in different
tasks.

model combines the merits of both AT and SP to make better prediction. Overall, these results demonstrate that the attention transition plays an important role in improving gazeprediction accuracy.
4.3.5

Cross Task Validation.

To examine how the task-dependent attention transition
learned in our model can generalize to different tasks undersame (kitchen) scene, we perform a cross validation acrossthe 7 different meal preparation tasks on GTEA Gaze Plus
dataset. We consider the following experiment settings:• SP: The saliency prediction module is treated as a
generic component and trained on a separate subset ofthe dataset. We also use it as a baseline for studying
the performance variation of different settings.
• AT d: The attention transition module is trained andvalidated under different tasks. Average performanceof 7-fold cross validation is reported.
• AT s: The attention transition module is trained andvalidated on two splits of the same task. Average performance of 7 tasks is reported.
• SP+AT d: The late fusion on top of SP and AT d.
• SP+AT s: The late fusion on top of SP and AT s.
Quantitative results of different settings are shown in
Figure 4. Both AUC and AAE scores show the same performance trend with different settings. AT d works worse
than SP, while AT s outperforms SP. This is probably due
to the differences of gaze behavior contained in different
tasks. However, SP+AT d with the late fusion module canstill improve the performance compared with SP and AT s,even with the context learned from different tasks.

4.4. Examination of the attention transition module
We further demonstrate that our attention transition module is able to learn meaningful transition between adjacent gaze fixations. This ability has important applications
in computer-aided AR system, such as implying a person

an office or in a manufacturing factory, and to study thegeneralizability of our model in different task domains.

References

Figure 5. Qualitative results of attention transition. We visualize
the predicted heatmap on the current frame, together with the current gaze position (red cross) and ground truth bounding box ofthe object/region of the next fixation (yellow box).

where to look next in performing a complex task. We conduct a new experiment on the GTEA-sub dataset (as introduced in Section 4.1) to test the attention transition moduleof our model. Since here we focus on the module’s abilityof attention transition, we omit the fixation state predictor
in the module and assume the output of the fixation statepredictor as ft = 0 in the test frame. The module takes wt
calculated from the region of current fixation as input andoutputs an attention map on the same frame which represents the predicted region of the next fixation. We extracta 2D position from the maximum value of the predicted
heatmap and calculate its rate of falling within the annotated bounding box as the transition accuracy.
We conduct experiments based on different latent representations extracted from the convolutional layer: conv5 1,
conv5 2, and conv5 3 of S-CNN. The accuracy based on
the above three convolutional layers are 71.7%, 83.0%, and86.8% respectively, while the accuracy based on random
position is 10.7%. We also tried using random channel
weight as the output of channel weight predictor to computeattention map based on the latent representation of conv5 3,
and the accuracy is 9.4%. This verifies that our model canlearn meaningful attention transition of the performed task.
Figure 5 shows some qualitative results of the attention transition module learned based on layer conv5 3. It can be seenthat the attention transition module can successfully predict
the image region of next fixation.

5. Conclusion and Future Work
This paper presents a hybrid model for gaze predictionin egocentric videos. Task-dependent attention transition islearned to predict human attention from previous fixations
by exploiting the temporal context of gaze fixations. The
task-dependent attention transition is further integrated witha CNN-based saliency model to leverage the cues from bothbottom-up visual saliency and high-level attention transition. The proposed model achieves state-of-the-art performance in two public egocentric datasets.
As for our future work, we plan to explore the taskdependent gaze behavior in a broader scale, i.e. tasks in[1] A. Borji and L. Itti. State-of-the-art in visual attention
modeling. IEEE transactions on pattern analysis andmachine intelligence, 35(1):185–207, 2013.
[2] A. Borji, D. N. Sihite, and L. Itti. Probabilistic learning of task-specific visual attention. In CVPR, 2012.
[3] A. Borji, H. R. Tavakoli, D. N. Sihite, and L. Itti. Analysis of scores, datasets, and models in visual saliency
prediction. In ICCV, 2013.
[4] C. Cao, X. Liu, Y. Yang, Y. Yu, J. Wang, Z. Wang,
Y. Huang, L. Wang, C. Huang, W. Xu, et al. Look andthink twice: Capturing top-down visual attention withfeedback convolutional neural networks. In ICCV,2015.
[5] L. Chen, H. Zhang, J. Xiao, L. Nie, J. Shao, W. Liu,and T.-S. Chua. Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning. 2017.
[6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and
L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[7] A. Fathi, Y. Li, and J. M. Rehg. Learning to recognize
daily actions using gaze. In ECCV, 2012.
[8] C. Feichtenhofer, A. Pinz, and A. Zisserman. Convolutional two-stream network fusion for video action
recognition. In CVPR, 2016.
[9] S. Frintrop, G. Backer, and E. Rome. Goal-directed
search with a top-down modulated computational attention system. In Joint Pattern Recognition Symposium, pages 117–124. Springer, 2005.
[10] J. Harel, C. Koch, and P. Perona. Graph-based visualsaliency. In NIPS, 2007.
[11] S. Hochreiter and J. Schmidhuber. Long short-term
memory. Neural computation, 9(8):1735–1780, 1997.
[12] X. Hou, J. Harel, and C. Koch. Image signature: Highlighting sparse salient regions. IEEE transactions on
pattern analysis and machine intelligence, 34(1):194–201, 2012.
[13] X. Huang, C. Shen, X. Boix, and Q. Zhao. Salicon:
Reducing the semantic gap in saliency prediction by
adapting deep neural networks. In ICCV, 2015.
[14] L. Itti and C. Koch. A saliency-based search mechanism for overt and covert shifts of visual attention.
Vision research, 40(10-12):1489–1506, 2000.
[15] L. Itti, C. Koch, and E. Niebur. A model of saliencybased visual attention for rapid scene analysis. IEEE

Transactions on pattern analysis and machine intelligence, 20(11):1254–1259, 1998.
[16] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,
2014.
[17] J. Kuen, Z. Wang, and G. Wang. Recurrent attentional
networks for saliency detection. In CVPR, 2016.
[18] M. F. Land. The coordination of rotations of the eyes,
head and trunk in saccadic turns produced in naturalsituations. Experimental brain research, 159(2):151–
160, 2004.
[19] Y. Li, A. Fathi, and J. M. Rehg. Learning to predictgaze in egocentric video. In ICCV, 2013.
[20] Y. Lin, S. Kong, D. Wang, and Y. Zhuang. Saliency
detection within a deep convolutional architecture. In
AAAI Workshops, 2014.
[21] J. Pan, E. Sayrol, X. Giro-i Nieto, K. McGuinness,and N. E. O’Connor. Shallow and deep convolutional
networks for saliency prediction. In CVPR, 2016.
[22] D. Parkhurst, K. Law, and E. Niebur. Modeling therole of salience in the allocation of overt visual attention. Vision research, 42(1):107–123, 2002.
[23] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang,
Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, andA. Lerer. Automatic differentiation in pytorch. 2017.
[24] R. J. Peters and L. Itti. Beyond bottom-up: Incorporating task-dependent influences into a computationalmodel of spatial attention. In CVPR, 2007.
[25] V. Ramanishka, A. Das, J. Zhang, and K. Saenko. Topdown visual saliency guided by captions. In CVPR,
2017.
[26] N. Riche, M. Duvinage, M. Mancas, B. Gosselin, and
T. Dutoit. Saliency and human fixations: state-of-theart and study of comparison metrics. In ICCV, 2013.
[27] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint
arXiv:1312.6034, 2013.
[28] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In
NIPS, 2014.
[29] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition.
arXiv preprint arXiv:1409.1556, 2014.
[30] Y. Sugano, Y. Matsushita, and Y. Sato. Appearancebased gaze estimation using visual saliency. IEEE
transactions on pattern analysis and machine intelligence, 35(2):329–341, 2013.

[31] A. Torralba, A. Oliva, M. S. Castelhano, and J. M.
Henderson. Contextual guidance of eye movements
and attention in real-world scenes: the role of globalfeatures in object search. Psychological review,
113(4):766, 2006.
[32] A. M. Treisman and G. Gelade. A feature-integration
theory of attention. Cognitive psychology, 12(1):97–
136, 1980.
[33] D. Walther and C. Koch. Modeling attention to salient
proto-objects. Neural networks, 19(9):1395–1407,
2006.
[34] J. Xu, L. Mukherjee, Y. Li, J. Warner, J. M. Rehg,
and V. Singh. Gaze-enabled egocentric video summarization via constrained submodular maximization. In
CVPR, 2015.
[35] K. Yamada, Y. Sugano, T. Okabe, Y. Sato, A. Sugimoto, and K. Hiraki. Can saliency map models predict
human egocentric visual attention? In ACCV, 2010.
[36] K. Yamada, Y. Sugano, T. Okabe, Y. Sato, A. Sugimoto, and K. Hiraki. Attention prediction in egocentric video using motion and visual saliency. In
Pacific-Rim Symposium on Image and Video Technology, pages 277–288. Springer, 2011.
[37] M. Zhang, K. Teck Ma, J. Hwee Lim, Q. Zhao, and
J. Feng. Deep future gaze: Gaze anticipation on egocentric videos using adversarial networks. In CVPR,
2017.
[38] R. Zhao, W. Ouyang, H. Li, and X. Wang. Saliency
detection by multi-context deep learning. In CVPR,
2015.
[39] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, andA. Torralba. Learning deep features for discriminative
localization. In CVPR, 2016.