Heatmap overlay using neutral body model
for visualizing the measured gaze distributionsof observers
Michiko Inoue, Fuyuko Iwasaki, Masashi Nishiyama, and Yoshio Iwai
Graduate School of Engineering, Tottori University, Japanmi.inoue@tottori-u.ac.jp

Abstract. We propose a method for visualizing where an observer’s
gaze focuses on a subject in a still image using a neutral human bodymodel. Generally, two-dimensional (2D) heatmaps are superimposed on
still images to visualize an observer’s gaze distribution, which indicateswhere an observer looks when observing a subject. To investigate gaze
distributions, eye-tracking researchers need a method to directly compare
the 2D heatmaps because body pose and shape differ among subjects.
Thus, a comparison of the gaze distributions using the 2D heatmaps istime-consuming if there is no acceptable method to handle the body poseand shape variations. Instead, our visualization method superimposes athree-dimensional (3D) heatmap representing the gaze distribution on
the surface of a neutral human body, which has a fixed pose and shapefor all subjects to visualize the locations at which an observer’s gazefocuses. Experimental results show that our visualization method allows
eye-tracking researchers to compare gaze distributions more directly thanthe conventional visualization method using 2D heatmaps on still images.
Keywords: Gaze distribution · 3D heatmap · neutral body model ·visualization · vertex attention probability.

1

Introduction

There are many situations in which people gather together, such as parties andconferences. Eye-tracking researchers in the computer vision and cognitive science fields investigate the gaze distributions of people in these situations, because
these gaze distributions indicate the behavior of the observers when they lookat the bodies of other people. For example, consider a situation in which someobservers judge the aesthetics of subjects attending a ceremonial party. In thissituation, the eye-tracking researchers measure the gaze distributions of the observers during this judgement. The researchers then compare the measured gaze
distributions of the subjects to determine whether the gaze behaviors of the observers differ. To perform this comparison, it is necessary to visualize the gaze
distribution so that the researchers can directly compare the locations on thesubjects where the observers focus their attention.

2

M. Inoue et al.

(a)

(b)

(c)

Fig. 1. (a) Measuring and visualizing where an observer’s gaze focuses when they lookat the subjects in the still images. (b) Superimposing the measured gaze distributions
onto the images using 2D heatmaps. (c) Superimposing the distributions on a neutral
human body model using 3D heatmaps.

We here consider how to compare whether there are differences in the measured gaze distributions of observers observing an attractive female and malesubject. The body poses and shapes of the female and male subject are very
different, as shown in the examples in Fig. 1(a). In these examples, the woman
has both hands down, and the man has one hand up. Moreover, the woman isslender, and the man is muscular. As these examples illustrate, subjects have various body poses and various body shapes in such situations. Hence, eye-tracking
researchers need a visualization method that can directly compare the gaze distributions among subjects, even when they have different body poses and body
shapes.
Two-dimensional (2D) heatmaps are often superimposed on still images tovisualize where an observer’s gaze focuses on a subject. Figure 1(b) shows examples of this visualization. In fact, eye-tracking researchers have commonly used
2D heatmaps to represent gaze distributions [8, 6, 4]. However, body poses andshapes differ among subjects when this method is used. To compare where thegazes of observers focus in the examples in Fig. 1(b), eye-tracking researchers
must pay attention to the different poses of the hands and arms and the different
body shapes of the man and woman. Therefore, a visualization that superimposes
2D heatmaps on still images is time-consuming for eye-tracking researchers toanalyze them. Other visualization methods [9, 10] have been proposed in which
3D heatmaps representing the gaze distributions are superimposed on the surface of artificial objects by aligning the shapes of known objects. It is easier for
eye-tracking researchers to directly compare gaze focus on artificial objects withrigid body shapes in various poses. However, applying these methods [9, 10] toHeatmap overlay using neutral body model for visualizing gaze distributions

3

human subjects is restricting because humans are non-rigid and have various
body shapes in different poses.
Here, we propose a visualization method that superimposes 3D heatmaps on
the surface of a neutral human body to directly compare the locations of observergazes when the observers look at subjects in still images. We use a neutral humanbody model with a body pose and shape that are normalized among subjects.
Our visualization method allows eye-tracking researchers to directly compare
the gaze distributions on the common 3D surface of the neutral human bodymodel, even if the body poses and shapes of the subjects in still images differ.
Figure 1(c) shows examples of 3D heatmaps representing gaze distributions bysuperimposing them on the 3D surface of the neutral human body model. In ourvisualization, it is not necessary to consider the differences in the body poses andshapes of the subjects. In addition, the observer gaze distributions can be directlycompared by simply looking at the spatially aligned 3D heatmaps superimposedon the surface of the neutral human body model.

2

Our 3D heatmap-based visualization method

2.1

Overview

In our method, the 3D heatmaps represent which parts of the subject’s bodythe gazes of observers focus. The 3D heatmaps are superimposed on the surface
of a neutral human body model. Figure 2 shows an overview of our method.
Our method generates images visualizing the measured gaze distributions of theobserver using steps S1 to S3 and the neutral human body model connecting
these steps.
S1. Computation of the pixel attention probability
We calculate the probability that the gazes of observers focus on the 2D
positions of the pixels in region of the subject in the still image. We call thisthe pixel attention probability.
S2. Computation of the vertex attention probability
Using the relationship between the 2D positions of the pixels of the subject
region and the 3D positions of the surface of the neutral human model, we
calculate the probability that the gazes focus on the vertices of the bodymodel. We call this the vertex attention probability.
S3. 3D heatmap overlay
We superimpose the 3D heatmap representing the vertex attention probabilities onto the neutral human body model so that eye-tracking researchers
can directly compare the gaze distributions among subjects. To do this, we
generate an image visualizing each gaze distribution using the 3D heatmap.
Neutral human body model
We use a neutral human body model with a pre-determined constant bodypose and shape parameters to fix the pose and shape of the subjects. The
model consists of a set of 3D vertices and the adjacencies between them.

4

M. Inoue et al.
S1. Pixel attention probability

S2. Vertex attention probability

𝑝 𝒙𝑗 𝑜, 𝑡,

𝑝 𝒗𝑙 𝑜, 𝑡,

Represents how much the gazes focus
on each pixel 𝒙𝑗

Represents how much the gazes focus
on each vertex 𝒗𝑙 of
the neutral human body model 𝒱

Time
𝑡
Gaze

Marginalized by𝑜 ∈ 𝒪 and 𝑡 ∈ 𝒯
Subject region
in the image

Set of observers

Set of times

𝒪

Observer𝑜

Neutral human body model

𝒯 = 1, ⋯ , 𝑇

𝑝 𝒗𝑙
S3. 3D heatmap overlay

𝒗𝑗 = 𝑚 𝒙𝑗 ;2D position of
pixel 𝒙𝑗 in

Visualizes 𝑝 𝒗𝑙
on the surfaceof the neutral human body model
3D position 𝒗𝑗
on the surfaceof the model

Vertex 𝒗𝑙 ∈ 𝒱 of the model

Visualization image
using the 3D heatmap

Fig. 2. Overview of our method. When observers look at a certain subject in a still
image, we measure where their gazes focus on the subject, and visualize the gaze
distribution using the 3D heatmap representing the vertex attention probability.

Our method transforms the 2D positions of the pixels in the subject regionof the still image to the 3D positions on the surface of the neutral humanbody model.
Here, we discuss the issue of the conventional visualization method using the
2D heatmap representing the pixel attention probability. When body poses andshapes differ among subjects, the alignment of bodies in the subject regions
causes gaps among the still images. Therefore, it is difficult to directly compare
the pixel attention probability between subjects. Our method reduces the gap
between body poses and shapes using neutral human body model with a normalized body pose in a normalized shape to represent the gaze distributions
using the 3D heatmaps. Our visualization method enables the vertex attention
probability among subjects with various body poses and shapes to be directlycompared. In the following, we describe the pixel attention probability in Section 2.2, the neutral human body model in Section 2.3, the vertex attention

Heatmap overlay using neutral body model for visualizing gaze distributions

5

probability in Section 2.4, and the 3D heatmap for generating the visualization
image in Section 2.5.
2.2

Pixel attention probability

Pixel attention probability indicates how much the measured gazes of the observers focus on each pixel in the subject region of the still image. Suppose thatthe subject region I is the set of pixels xj , and the gaze is measured from observer o at pixel xt in the subject region I at time t. We denote the probability
that the gaze is measured at a pixel xj ∈ I byp(xj |o, t, I) ∼ N (xj |xt , Σp ),(1)

where N (xj |xt , Σp ) is a bivariate normal distribution
with mean xt and covari
ance matrix Σp . We assume Σp = diag σp2 , σp2 . Here, p(xj |o, t, I) is assumed
to follow a normal distribution. The observer is not only looking at the pixel
xt where the gaze is measured and but also looking at the surrounding pixels.
Hence, to approximate this, we use the normal distribution. Pixcel attention
probability p(xj |o, t, I) satisfies the following equation:Xp(xj |o, t, I) = 1.
(2)
xj ∈I

Note that eye-tracking researchers must handle pixel attention probability carefully because it is difficult to directly compare pixel attention probabilities among
subjects because body poses and shapes differ among subject regions in still images, as described in Section 1.
2.3

Neutral human body model

Several human body models [5, 2, 7] have been proposed to represent the humanbody with various poses and shapes. In these human body models, pose andshape are represented by continuous parameters in function space. We set up
the neutral human body model using constant values for the parameters of bodypose and shape for all subjects. This model allows us to transform different bodyposes and shapes into a common body representation for all subjects.
The neutral human body model V consists of meshes based on a set of vertices vl and the adjacencies between vertices, as described in [5, 2, 7]. To indicate
how much the gazes focus on a vertex vl , we could simply map pixel attention
probability of each pixel xj in the subject region of a still image to a vertex vl of
the neutral human body model using nearest neighbors. However, the vl vertices
are discrete, and this simple mapping will cause aliasing. For this reason, we use
the 3D position vj that exists continuously on the surface of the neutral humanbody model.
The method for mapping the 2D position of pixel xj ∈ I in the subject
region of a still image to a 3D position vj on the surface of a neutral humanbody model is as follows. Specifically, we transform it using function m() as
vj = m(xj ; I).

(3)

6

M. Inoue et al.

The function m() first estimates the body pose and shape parameters for eachsubject in the still image using an existing method, e.g. [3, 11, 1, 12]. Next, the estimated pose and shape parameters are converted to constant values for the neutral human body model. In this transformation, the 2D position of the pixel xj
can be automatically mapped to the 3D position vj .
2.4

Vertex attention probability

We define the vertex attention probability, which represents how much an observer o focuses his/her gaze on a vertex vl of the neutral human body model V
when looking at the subject region I at time t asX
p(vl |o, t, I) =p(vl |xj , I)p(xj |o, t, I),
(4)
xj ∈I

where p(vl |xj , I) is the probability that the gaze focuses on the 3D vertex vl of
the neutral human body model given the 2D pixel xj ∈ I. We transform the 2D
position of the pixel xj on the still image to the 3D position vj on the neutral
human body model surface using Eq.(3) asX

p(vl |o, t, I) =p(vl |vj , I)p(xj |o, t, I).

(5)

xj ∈I

Obtaining this probability using geodetic distances is computationally expensive. Therefore, we make the following assumption to reduce computation time.
First, we suppose that the pixel at which the gaze focuses is transformed to 3D
position vj on the neutral human body model. A smaller Euclidean distancefrom vj to vertex vl on the neutral human body model means that the probability that the gaze is measured at that vertex will be higher. Based on thisassumption, we express p(vl |vj , I) ≃ N (vl |vj , Σv ) using a normal distribution
and calculate the vertex attention probability p(vl |o, t, I) asp(vl |o, t, I) ≃X

N (vl |vj , Σv )p(xj |o, t, I),

(6)

xj ∈I

where N (vl |vj , Σv ) is a trivariate normal distribution with mean vj and covari
ance matrix Σv . In our method, the covariance matrix is Σv = diag σv2 , σv2 , σv2 .
Note that p(vl |o, t, I) satisfies the following equation:Xp(vl |o, t, I) = 1.

(7)

vl ∈V

Here, we further enhance the comparison of gaze distributions in eye-tracking
research. It is time-consuming to individually check the locations of gaze focus
for each observer o and at each time t. We hence marginalize the probabilities
using the set of observers O and set of measurement times T . Given subject

Heatmap overlay using neutral body model for visualizing gaze distributions

7

region I in the still image, we calculate the marginal probability that the gazes
are measured at the vertex vl of the neutral human body model as follows:Xp(vl |I) =p(vl |o, t, I)p(o)p(t).
(8)
o∈O,t∈T

Let O be the number of elements in set O and T be the number of elements in
set T . Approximating p(o) using the uniform distribution 1/O and p(t) using
the uniform distribution 1/T , Eq.(8) is converted top(vl |I) =

1
OT

X

p(vl |o, t, I).

(9)o∈O,t∈T

Note that p(vl |I) satisfies the following equation:Xp(vl |I) = 1.

(10)

vl ∈V

We call p(vl |I) in Eq.(9) the vertex attention probability marginalized by theset of observers O and set of measurement times T .
2.5

3D heatmap for generating the visualization image

To visualize the gaze distributions on a neutral body model for direct comparison
among subjects, we represent the vertex attention probability p(vl |I) using a
3D heatmap and overlay it onto the surface of the model. First, we consider asimple, vertex-only heatmap visualization. Figure 3(a) shows examples of thevisualization images using the vertex-only heatmap. In the figure, vertices witha higher probability of concentrated gazes are redder in hue, and those with alower probability of focused gazes are closer to blue. In these visualization images,
the front and back vertices of the neutral human body model are both visible,
making it difficult to compare the gaze distributions among subjects. Thus, we
continuously interpolate the colors representing the high and low probabilities
of focused gazes at 3D positions vj using a mesh on the surface of the neutral
human body model. Figure 3(b) shows examples of the visualization images using
the 3D mesh heatmap. This heatmap representation prevents the front and backof the neutral human body model from being visible simultaneously, making iteasier to directly to compare the gaze distributions among subjects.

3

Experiments

3.1

Visualization images generation conditions

We evaluated the effectiveness of our method using visualization images generated by the following methods.
– Conventional method (M2d ): We overlaid the 2D heatmap representing the
pixel attention probability described in Section 2.2 onto the still image.

8

M. Inoue et al.

(a)Vertex attention probability: Low

(b)
High

Fig. 3. Examples of visualization images using 3D heatmaps to represent vertex attention probabilities. (a) Using vertices only. (b) Using a 3D mesh.

– Our method (M3d ): We overlaid the 3D heatmap representing the vertex
attention probability described in Section 2.4 onto the surface of the neutral
human body model.
Note that the conventional method M2d is equivalent to a visualization of themeasured gaze distributions of observers, such as the methods used in existing
analytical studies [8, 6, 4].
For our method M3d , we used SMPL [5] to implement the neutral humanbody model described in Section 2.3. In the neutral human model used in thisexperiment, we set the pose and shape parameters of SMPL to their default
values1 . These default values specify that the body pose is one with arms outstretched and legs slightly open, and the body shape is average. The numberof SMPL vertices used was 6890. Adjacency was represented by the meshes oftriangles connecting the three vertices. We used DensePose [3] to estimate thebody pose and shape parameters of SMPL for each subject in the still image forfunction m() of Eq.(3), as described in Section 2.3.
1https://smpl.is.tue.mpg.de

Heatmap overlay using neutral body model for visualizing gaze distributions

3.2

9

Gaze measurement

To obtain measured gaze distributions, we asked several observers the following
question Q so that they would evaluate the aesthetics of a subject in still images.
Q: Do you think that the subject’s hands are beautiful?
We instructed them to answer “yes” or “no.” The number of observers was 24(12 men and 12 women), and the mean age of the observers was 22.4 ± 1.0 years
old.
We first explained question Q to the observers and showed them a randomly
selected still image on a display for 7 s. Figure 4(a) shows the still image given
to the observers as a stimulus. Subject 1 is a slender woman with her right handdown and left leg slightly shifted forward. Subject 2 is a man with his righthand raised and his legs shoulder-width apart. Subject 3 is a petite woman withher right hand on her hip and her heels together. Figure 4(b) shows the subject
regions in the still images.
We measured each observer’s gaze using an eye tracker device while displaying the still image. This gaze measurement was repeated until all still images
were viewed. The observers were seated 65 cm from the display. We allowed theobservers to adjust the chair height so that the eye height was between 110 and
120 cm. We used a 24-inch display with a resolution of 1920 × 1080 pixels anda Gazepoint GP3 HD eye tracker with a 60 Hz sampling rate. The spatial resolution error of the eye tracker device is approximately one degree, as describedin the specifications. We displayed the still image at a random position on thedisplay to avoid center bias.
3.3

Visualization results

Figure 4(c) shows the visualization images generated using the conventional
method M2d , which superimposes the 2D heatmaps representing the pixel attention probability on the still images. Because of the variation in the bodyalignment among subjects, we must consider the differences in the body posesand shapes when comparing the gaze distributions. With these differences inmind, we can infer the following when comparing the results of Fig. 4(c). For
subject 1, the observers’ gazes focused most often on the right hand, followedby the head. For subject 2, the observers’ gazes focused most often on the righthand, followed by the head and left hand. For subject 3, the observers’ gazes
focused most often on the right and left hands, followed by the head. Fromthese results, we conclude that when question Q is given to observers looking at
subjects 1 through 3, the gazes mostly focus on the hands and then on the head.
Figure 4(d) shows the visualization images generated using our method M3d .
Our visualization results show that the body poses and shapes of the neutral
human body model are the same for all subjects. These 3D heatmaps makethe body alignment equal for all subjects so that when analyzing differences in
the gaze distributions, eye-tracking researchers only need to compare the same
positions on the surface of the model. The gaze distributions for subjects 1

10

M. Inoue et al.

Subject 1 Subject 2 Subject 3(a) Still images

Subject 1 Subject 2 Subject 3(b) Subject regions

Subject 3
Subject 2
Subject 1(c) Visualization images using 𝑀2𝑑 for each subject.

Subject 1
Subject 2
Subject 3(d) Visualization images using 𝑀3𝑑 for each subject.
Attention probability:

Low

High

Fig. 4. Visualization image results showing the attention probabilities of the measured
gaze distributions of observers looking at the still images.

through 3 reveal that the gazes most often focus on the hand, followed by thehead. We can reach this conclusion more directly using our method M3d thanwhen using the conventional method M2d .
3.4

Subjective assessment of the visualization images

Conditions of subjective assessment
We conducted a subjective assessment to determine whether the visualization images generated by the conventional method M2d or those generated
by our method M3d enable the gaze distributions for various subjects to be
directly compared. Sixteen eye-tracking researchers participated in the subjective assessment (13 men and three women). These researchers are graduate
students studying human and computer vision, including gaze measurement

Heatmap overlay using neutral body model for visualizing gaze distributions

Eye-tracking
researcher

11

Display
65 inch(b) Conventional method 𝑀2𝑑90 cm
150 cm(a) Arrangement(c) Our method 𝑀3𝑑Fig. 5. Setup of the subjective assessment.

and analysis. Their average age was 23.4 ± 1.5 years old. Figure 5(a) shows
the setup of the subjective assessment. The researchers stood upright at a
distance of 150 cm horizontally from the display. The size of the display(LG, OLED65E9PJA) was 65 inches, and the height from the floor to thedisplay was 90 cm. The researchers compared the results obtained using theconventional method M2d (Figure 5(b)) with the results obtained using our
method M3d (Figure 5(c)). Visualization images of each method were shownon the display for 60 s each. We randomized the order of displaying thevisualization images generated by M2d and M3d . We asked the researchers
to choose the visualization images they felt would better directly facilitate
the comparison of gaze distributions among the subjects. The researchers
replied with one of the following answers: M2d , M3d , or neutral.
Result of the subjective assessment
Figure 6 shows the result of the subjective assessment. Five eye-tracking
researchers chose the conventional method, two chose neutral, and nine chose
our method. Our method M3d obtained the highest results. Some researchers
chose the conventional method M2d because they were familiar with the 2D
heatmaps on still images, which made it easy to identify the body parts
focused on by gaze for each subject. In contrast, some researchers chose ourmethod M3d because the body parts such as the hands, torso, and legs arecompletely aligned for all subjects, so it is easy to compare the differencesin the gaze distributions without having to pay attention to the body posesand shapes. From these results, we confirmed that our method M3d enables
eye-tracking researchers to more directly compare differences in the gaze
distributions, even when the body poses and shapes are different, than theconventional method M2d .

12

M. Inoue et al.

Conventional method 𝑀2𝑑NeutralOur method 𝑀3𝑑
0

4

8

12

16

Number of eye-tracking researchers

Fig. 6. Result of the subjective assessment comparing visualization images generatedusing the proposed and conventional methods.

4

Conclusions

We proposed a method of superimposing 3D heatmaps on the surface of a neutral
human body model to visualize where the gazes of observers focus when theylook at subjects in still images. For the visualization images obtained using our
method, we confirmed in a subjective assessment that eye-tracking researchers
could directly compare the gaze distributions, even when the body poses andshapes differed among subjects.
In future work, we will expand the evaluation to consider the case in which
various shape characteristics, such as the weight, of the subjects in still images
change. We also plan to consider the error estimation due to the influence ofgender and clothing according to the neutral human body model. Furthermore,
we intend to develop a method for calculating the vertex attention probability
when there are multiple subjects in a single still image.

Acknowledgment
This work was partially supported by JSPS KAKENHI Grant No. JP23K11145.
We would like to thank Mr. Ken Kinoshita for his cooperation in our evaluationand data collection.

References
1. Choutas, V., Müller, L., Huang, C.P., Tang, S., Tzionas, D., Black, M.J.: Accurate
3d body shape regression using metric and semantic attributes. In: Proceedingsof the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp.
2718–2728 (2022)
2. Georgios, P., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A.A.A., D.Tzionas,
Black, M.J.: Expressive body capture: 3D hands, face, and body from a single
image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition. pp. 10975–10985 (2019)
3. Güler, R., Neverova, N., Kokkinos, I.: Densepose: Dense human pose estimation in
the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition. pp. 7297–7306 (2018)

Heatmap overlay using neutral body model for visualizing gaze distributions

13

4. Irvine, K.R., McCarty, K., Pollet, T.V., Cornelissen, K.K., Tovée, M.J., Cornelissen, P.L.: The visual cues that drive the self-assessment of body size: Dissociation
between fixation patterns and the key areas of the body for accurate judgement.
Body Image 29, 31–46 (2019)
5. Matthew, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.: SMPL: A
skinned multi-person linear model. ACM Transactions on Graphics 34(6), 1–16
(2015)
6. Nummenmaa, L., Jari, H., Santtila, P., Hyönä, J.: Gender and visibility of sexual
cues influence eye movements while viewing faces and bodies. Archives of Sexual
Behavior 41(6), 1439–1451 (2012)
7. Osman, A.A.A., Bolkart, T., Tzionas, D., Black, M.J.: SUPR: A sparse unified partbased human representation. In: Proceedings of European Conference on Computer
Vision. pp. 1–18 (2022)
8. Piers, C., Peter, H., Kiviniemi, V., Hannah, G., Martin, T.: Patterns of eye movements when male and female observers judge female attractiveness, body fat andwaist-to-hip ratio. Evolution and Human Behavior 30(6), 417–428 (2009)
9. Takahashi, R., Suzuki, H., Chew, J., Ohtake, Y., Nagai, Y., Ohtomi, K.: A system
for three-dimensional gaze fixation analysis using eye tracking glasses. Journal ofComputational Design and Engineering 5(4), 449–457 (2017)
10. Wang, X., Lindlbauer, D., Lessig, C., Maertens, M., Alexa, M.: Measuring thevisual salience of 3d printed objects. IEEE Computer Graphics and Applications
36(4), 46–55 (2016)
11. Xiu, Y., Yang, J., Tzionas, D., Black, M.J.: Icon: Implicit clothed humans obtained
from normals. In: Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition. pp. 13296–13306 (2022)
12. Zhang, H., Tian, Y., Zhou, X., Ouyang, W., Liu, Y., Wang, L., Sun, Z.: Pymaf: 3d
human pose and shape regression with pyramidal mesh alignment feedback loop.
In: Proceedings of the IEEE/CVF International Conference on Computer Vision.
pp. 11446–11456 (2021)