Journal of Intelligent & Robotic Systems
https://doi.org/10.1007/s10846-019-01052-8

First-person Video Analysis for Evaluating Skill Levelin the Humanitude Tender-Care TechniqueAtsushi Nakazawa1· Yu Mitsuzumi2 · Yuki Watanabe1 · Ryo Kurazume3 · Sakiko Yoshikawa4 · Miwako Honda5

Received: 30 August 2018 / Accepted: 12 June 2019
© The Author(s) 2019

Abstract
In this paper, we describe a wearable first-person video (FPV) analysis system for evaluating the skill levels of caregivers.
This is a part of our project that aims to quantize and analyze the tender-care technique known as Humanitude by using
wearable sensing and AI technology devices. Using our system, caregivers can evaluate and elevate their care levels bythemselves. From the FPVs of care sessions taken by wearable cameras worn by caregivers, we obtained the 3D facial
distance, pose and eye-contact states between caregivers and receivers by using facial landmark detection and deep neuralnetwork (DNN)-based eye contact detection. We applied statistical analysis to these features and developed algorithms
that provide scores for tender-care skill. In experiments, we first evaluated the performance of our DNN-based eye contact
detection by using eye contact datasets prepared from YouTube videos and FPVs that assume conversational scenes. We
then performed skill evaluations by using Humanitude training scenes involving three novice caregivers, two Humanitude
experts and seven middle-level students. The results showed that our eye contact detection outperformed existing methods
and that our skill evaluations can estimate the care skill levels.
Keywords Dementia · Care · Deep neural network (DNN) · Skill evaluation · Wearable system · Computer vision · First
person video

 Atsushi Nakazawa

1 Introduction

nakazawa.atsushi@i.kyoto-u.ac.jp
Yu Mitsuzumi
yu.mitsuzumi.ae@hco.ntt.co.jp
Ryo Kurazume
kurazume@ait.kyushu-u.ac.jp
Sakiko Yoshikawa
yoshikawa.sakiko.4n@kyoto-u.ac.jp
Miwako Honda
honda-1@umin.ac.jp
1

Graduate School of Informatics, Kyoto University,
Kyoto, Japan

2

NTT Communication Science Laboratories, Nippon Telegraphand Telephone Corporation (NTT), Kanagawa, Japan

3

Faculty of Information Science and Electrical Engineering,
Kyushu University, Fukuoka, Japan

4

Kokoro Research Center, Kyoto University, Kyoto, Japan

5

Geriatric Research Division, National Hospital Organization
Tokyo Medical Center, Tokyo, JapanAs the elderly population increases, the number of people
suffering from dementia continues to grow. As a result, thecare that needs to be administered to them is becoming increasingly important in social terms [8, 25, 42]. The populationof people in Japan afflicted with dementia is expected to
exceed seven million by 2025. A more serious problem isthe shortage of caregivers. The number of caregivers thatwill be necessary in 2025 is estimated to be 2.53 million,
but the actual number is estimated to be 2.15 million [29].
Dementia occurs when the brain is damaged by maladies
such as Alzheimer’s disease and Lewy body dementia
and produces a set of symptoms that include memory
loss and difficulties with thinking, problem-solving, andverbal communication. Dementia can be accompanied bypsychosis, agitation and aggression; thus, caring for people
with dementia is quite difficult [7].
Two approaches can be cited as ways to alleviate thedifficulties this poses for caregivers. The first is providing
patients with customized treatment, which can slow the
progression of dementia and prevent side effects such as

J Intell Robot Syst

infections. The second is reducing the burden on caregivers
to prevent them from ”burning out.” Dementia can cause
symptoms similar to those of mental illnesses, known
as behavioral and psychological symptoms of dementia(BPSD), so caregivers’ working conditions can be harsh. As
a result, the number of caregiving staff members that leaveand become burned out is increasing.
Humanitude tender-care style: Due to this social
background situation, the caregiving style Humanitude has
been spotlighted by care professionals and family caregivers
since it can reduce the occurrence of BPSD events andcaregivers’ burden [20]. Humanitude was developed by Y.
Geneste and R. Marescotti 35 years ago [18] and has been
introduced in more than 600 hospitals and nursing homes in
Europe. Humanitude primarily uses a combination of fourcommunication skills: gaze, verbal communication, touch,and helping care receivers to stand up. Several studies havereported that the cost-efficiency of introducing Humanitude
is around 20 times that of care without it because of a
40% decrease in the use of psychotropic drugs and in thenumber of care staff members who leave [22]. In recent
years, Humanitude has become popular in Japan; in thepast three years, more than 2,600 people took Humanitude
training over the course of more than 30 training sessions.
Computational tender-care science project: Since webelieve improved care techniques using robotics and computer vision technologies are valuable to both caregivers
and care receivers, we started a comprehensive project thataims to (a) quantize and visualize the Humanitude skills,(b) reveal the brain mechanism behind Humanitude-based
communications and (c) develop a system that will help
people to learn Humanitude skills (Fig. 1). This paper
shows one of the topics in (a) skill quantization, but we
briefly introduce all topics below.
For (a), we developed a system that automatically finds the
skill elements by using wearable sensors that capture learner’s and care receivers’ behaviors. The system uses data
mining and recognition algorithms developed to enhancecomputer vision and machine learning. We then obtained theessence of Humanitude skills through multi-modal analysis.
For (b), we tried to reveal why using Humanitude facilitates communication with people with dementia (PwD)
and why it reduces BPSD through cognitive neuroscientificapproaches. Namely, we conducted functional neuroimaging studies to find the differences between younger people
and elderly people, and between healthy people and people
suffering from Alzheimer’s disease while giving emotional
stimuli such as facial pictures that show eye contact ordynamic facial expressions.
For (c), on top of the (a) and (b) findings, we developed atender-care education platform that presents the caregivers’
skill level to learners by using our care-skill evaluation

systems. With these systems, learners will be able toevaluate their current tender-care skill levels easily and at
a low cost by themselves. It follows that this platform will
be suitable for non-professional caregivers such as family
caregivers, as well as for professional caregivers who wantto periodically refresh their caregiving skills.
Wearable sensing technique for care behavior analysis:
Since Humanitude consists of communication behaviors
from close distances such as gaze and touch, we use
wearable sensing devices to extract events in which such
behaviors occur. We observed care techniques that do and do
not use Humanitude and found behavioral differences andtheir outcomes through statistical computational analysisbased approaches, as illustrated in Fig. 2.
This paper describes the first step of this project– a system for extracting face-to-face communication
behavioral skills by using a head-mounted camera worn by
a caregiver. From camera images obtained using mutual
facial distances, we obtained poses and eye contact states byusing facial parts tracking and deep neural network (DNN)based eye contact detection algorithms. We compared thesebehavioral elements from those we ascertained amongcare novices, middle-level Humanitude-care learners andHumanitude-care experts. Although many attempts havebeen made to obtain the aforementioned information byusing third-person view videos, the videos were analyzed byvideo annotators. This required considerable cost and time
and there was the risk of obtaining biased results caused byannotator subjectivity [21]. To address these problems, thecontributions and limitations of this paper are as follows:
1. It describes our development of a prototype system thatuses wearable cameras and image analysis for care skill
evaluation.
2. It describes our development of DNN-based eye
contact detection algorithms that outperform existing
approaches by using eye contact datasets based on
YouTube videos and first-person video (FPVs).
3. It describes how we obtained an FPV dataset while
conducting Humanitude training sessions for novices,
middle-level caregivers and expert caregivers and foundthe differences among them regarding face-to-face distance and pose (angle) as well as eye contact frequency.
4. It describes how we performed unsupervised principal
component analysis (PCA) for the features obtained
from FPVs and found significant correlation between
caregiver levels and PCA scores.
5. The present research represents a preliminary analysis
that uses a relatively small number of datasets. It will bethus be followed by our analysis of extensive studies using
a much larger number of samples. It will be also important
to use chronological behavioral data for the same individuals to observe how the skills were acquired or forgetten.

J Intell Robot Syst
Fig. 1 Project overview. The
Humanitude tender-care
technique was analyzed fromboth computational andcognitive neuroscientificapproaches(a) Computaonal approach
Behavior analysis throughobserva(b) Neuroscien
Tender-care (Humanitude)- Obtain rule-based knowledge
representaons.

- Obtain and analyze brain/facialmuscle acvies- Develop cognive models fortender-care communicaons

Knowing how and why
Humanitude works

Computaonal behavioral
sensing- Quanze gaze behaviors (eyetracking rst-person vision)- Sense touching and talking skills.

(c) Skill transfer

Analyzing physical behavior
with Humanitude

Devepment of Tender-care
educaon plaorm- Introduce developed

Systemevaluaon

method/devices/informaoninto the2 Related WorkIn this section, we show related studies regarding caregiver’s
burden and effects of intervention, care skill evaluation,
first-person video-based skill evaluations and eye contact
detection from video images.
Caregiver’s burden and eﬀects of interventions: The burdenof caregiver for dementia patients have been reported in a
lot of literature [2, 10, 12, 15, 16, 23, 33, 41]. According
to the recent meta-review paper[2], the larger caregivers
burden is related to 1) female sex, 2) low education, 3)
cohabitation with care recipient, 4) caregiving time andeffort, 5) financial stress, 6) lack of choice and inability tocontinue regular employment. As the result, caregiver tends
to having larger risk in mortality, weight loss, poor self-careand sleep deprivation.
Effects of interventions for reducing caregiver’s burden have been reported as well [1, 13, 15, 19, 31, 32].
Interventions are categorized into several types: psychoeducational intervention, psychosocial intervention, cognitive
behavior therapy, respite, caregiver support groups, anticholinergic and antipsychotic drugs, and skill training. As

Usual care

Humanitude care
(Tender-care)

Behavioral features
Gaze: target object,distance, pose
Verbal: context, tone,
Featurepitch, speed, intensity
extracon Touch: locaon, hand used,
power, stroke
Skill elements [Ishikawa16]

approach

Hospitals
Nursing homes
Private homes

the results, practical interventions to reduce caregiver’s burden are 1) encouraging caregivers to function as a memberof the care team, 2) encouraging caregivers to improveself-care and maintain their health, 3) providing education and information, 4) coordinating for assistance withcare, 5) encouraging caregiver to access respite care and 6)
using the supports of technology [2]. Specifically, there areseveral reports that skills training such as coping skill training (CST) may reduce the caregiver strain, depression andfatigue in caregivers of the patient with cancers [9, 31].
Care skill quantization: There are several approaches thatuse care skill quantization. In computer science, Ishikawa
et al. developed a method of care skill evaluation basedon the knowledge of care experts [21]. They categorizedcare skills into three layers: intramodality, intermodality andmultimodal-interaction. Intramodality consists of behavior
primitives such as gaze, speech, touch, nodding andknocking on a door. Intermodality shows the relationships
among intramodalities, such as comprehensiveness ofcare, waiting for elderly people’s actions and consistency.
Multimodal-interaction consists of actions that develop
a relationship between actors, such as eye contact andStascal analysis
ﬁnd stascal diﬀerences in behaviors

Feature vectors- Automated feature detecon- Finding new skill elements
Ambient sensing technology

Time-series analysis
ﬁnd temporal relaons o ehaviors

Mul-modal analysis
ﬁnd relaons of diﬀeren ypes of signals(e.g., gaze and verbal, gaze and touch)

Skill level

cameras, sensors embedded in environments

Wearable sensing technology
wearable cameras and sensors

Fig. 2 Computational skill extraction obtained using behavioral observation and wearable sensors. We quantized care skill elements with andwithout Humanitude and found the differences between them

J Intell Robot Systdistance

Vectorquanzaon

Mutual facial distanceFPV of a caregiver

Care receiver

Unsupervised
analysis (PCA)

Vectorquanzaon

Feature vector

Care giver

Care skill
score

Mutual facial pose

Facial parts

OpenFace

Eye images

Eye contact
detectionusing DNN

Fig. 3 Overview of the care skill estimation method using first-person video

verbal/nonverbal dialogue. They also developed a web
interface that shows care learners care skills in visual form
to confirm the effectiveness of the system.
First-person video analysis for skill evaluation: There havebeen a number of studies on action recognition andprediction using FPVs [17, 28, 37, 39, 40]. However,few studies have been conducted for skill evaluation. In
recent years, Bertasius et al. showed a method to assess abasketball player’s performance from FPVs. They designed
and used temporal CNN and long short-term memory(LSTM) architecture to evaluate whether a particular playin basketball was good or not from a player’s FPV [5].
In the medical field, Hei et al. proposed a method forevaluating skill in robotic surgical operations from video
images. Their method tracks the keypoints of surgical robot
instruments by using cloud sourcing or hourglass networks
and evaluates the skill by support vector machine analysis[26].
Video-based eye contact detection: Detecting and making
eye contact are important for understanding social communication and designing communication robots. Therefore,
several studies in this area have been conducted. Smith

Fig. 4 Facial pose parameters obtained from OpenFace

et al. [38] proposed an algorithm to detect gaze-locking(looking at a camera) faces using eye appearances and PCA
plus multiple discriminate analysis. Ye et al. developed apioneering algorithm that detects mutual eye gaze using
wearable glasses [43, 45]. In recent years, deep-learningbased approaches are being implemented for eye-contact
detection. Mitsuzumi et al. developed the DNN-based eye
contact deteciton algorithm (DeepEC)[30] that uses only
cropped eye regions for eye-contact detection and performed better than existing methods. Eunji et al. develop the
DNN-based PiCNN detector that accepts the facial regionand output both facial postures and eye contact states [11].
Zhang et al. presented an eye-contact detection algorithm
based on their deep neural network (DNN) based gaze estimations [48]. In robotics, Petric et al. developed an eye
contact detection algorithm that uses facial images takenwith a camera embedded in a robot’s eyes [34] to develop
robot-assisted ASD-diagnosis systems. These eye contact
detection algorithms depend on facial landmark detection
libraries or gaze estimation algorithms with which it isassumed that subject faces are not occluded.
Image-based gaze estimation algorithms have also been
recently studied, although they differ in scope from ourdetection algorithms. The current trend in this area is
deep learning-based approaches, namely, learning and predicting gaze directions according to datasets that describe
the relation between facial images, facial landmarks, andgaze points. For example, Lu et al. developed a headpose-free gaze estimation method by synthesizing eye
images from small samples. However, their method requires
personal-dependent eye image samples taken under experimental setups [27]. Zhang et al. proposed a DNN algorithm
that inputs eye images and 3D head poses obtained fromfacial landmark points [46]. They also developed a DNNbased algorithm using full facial images without occlusions [47]. Krafka et al. developed a DNN-based eye gaze

J Intell Robot Syst

Target Frame

Target Frame

N-1 Previous Frames●t = T-N+1

t=T
Facial parts detection

IR

Facial parts detection

Eye images + GCN

Eye images + GCN
●

●

● ●

●

●

Left eye(60x36)

Conv(3x3, 20)

Conv(3x3, 20)

Conv(3x3, 20)

Conv(3x3, 20)

Maxpool 1/2

Maxpool 1/2

Conv(3x3, 50)

Conv(3x3, 50)

Conv(3x3, 50)

Conv(3x3, 50)

Maxpool 1/2

Maxpool 1/2

FC (512)FC (512)fR(IR)

fL(IL)FC (1024)

DEEPECDEEPECfR(IR)

fL(IL)

LSTM (R-EYE)

512(515)512(515)512(515)512(515)512(515)512(515)

LSTM (L-EYE)

Skip
Connections

DEEPECt=T

t = T-N+2

Facial Pose esmaon
Facial parts detection(in TempEC-HP)

IL

Right eye
(60x36)●

Facial pose (TempEC-HP)

Eye images + GCN●

full connected (2048 or 2060)
FC (1024)

full connected (2048 or 2060)

output(a) Singleframe Eye Contact
detecon (DEEPEC)

output(b) Proposed algorithm (TempEC, TempEC-HP)

Fig. 5 Eye contact detection algorithms. a The single-frame eye contact detection algorithm (DeepEC) first detects the eye regions of thetarget image frame and obtains a pair of right and left eye images.
The eye contact state is obtained by only the CNN that inputs the eye
images. b Proposed temporal eye contact detection algorithms that use
multiple (i.e. N) image frames. First, they detect facial landmarks with
OpenFace face detection, with which they then obtain eye regions ineach of the N frames. The resulting N pairs of eye images are inputtedto CNNs that have a structure similar to that of DeepEC. These CNNs
are followed by an LSTM network, which learns the temporal state of
the eyes. Finally, the target eye contact state is obtained by the following fully connected networks, which use not only the LSTM’s outputs
but also the CNN’s outputs of the target frame (t = T ) with skip
connections

estimation algorithm that inputs full facial images as well as
eye images [24].
In contrast, our detection algorithms only output binary(eye-contacted/averted) information. However, they do not
require personal-dependent samplings and are robust to
facial occlusions, which frequently occur in FPVs in
caregiving and communication scenarios. This was achieved
by designing a CNN that only uses images taken around theeyes.

3 Proposed Method
The flow of our skill evaluation is illustrated in Fig. 3.
From a first-person camera worn by a caregiver we obtained
mutual facial distances, mutual facial poses and eye contact
states. Then, we estimated tender-skill scores through anunsupervised analysis. In the following subsections, we firstdescribe the first-person camera hardware and then ouralgorithms we used for analyzing FPVs.

J Intell Robot Syst
Table 1 First person eye-contact dataset (Youtube dataset)
Name

Duration
(sec)

Fps

Image size
(H,V)

Avec
Azis
Derek
Elle
Emma
WataruJames
Kendall
Liza
Neil
SelenaMai
Taylor

350
479
490
391
555
55
380
447
447
687
460
337
581

29.97
23.98
23.98
29.97
23.98
29.97
23.98
23.98
23.98
23.98
23.98
29.12
23.98

1280×720
1280×720
1280×720
1280×720
1280×720
540×360
1280×720
1280×720
1280×720
1280×720
1280×720
1280×720
1280×720

3.1 Hardware

3.2 Face Detection and 3D Pose Estimation
We then obtained facial positions, poses and eye locations
from the input FPVs. We used the OpenFace library [4] andobtained 3D facial positions, poses and 68 facial landmark
points. We computed the cameras’ focal lengths from thecamera projection matrices and used them to estimate 3D
facial positions and rotations.

3.3 Histograms of Facial Distances and Poses
To quantize the face-to-face communication behaviors
between caregivers and care receivers, we encoded themutual facial distances and poses obtained from OpenFace
as illustrated in Fig. 4. Namely, we computed the
1 , . . . , f 11 ], f1
9histograms fdist = [fdist
rx = [frx , . . . , frx ],
dist
1
9
1
9
fry = [fry , . . . , fry ] and frz = [frz , . . . , frz ] that represent
the mutual facial distances and poses from all frames in acare session. The bins were set to every 0.1 [m] from 0.0 to
1.0 [m] for the distance feature and 20 [deg] from -90 to +90

avertedeye contacted

We used two types of head-mounted first-person camera
systems. One was a Pivothead Kudu camera [35], which isequipped with a front-view camera in the middle of a pairof glasses. The camera takes full HD (1920 × 1080 pixels)

videos at 30 fps. The other was a Pupil Labs camera system[36], whose frontal camera also takes full HD (1920 × 1080
pixels) videos at 30 fps. The cameras’ projection matrices
were obtained by using the MATLAB camera calibrator.

Aziz

Derec

Elle

Emma

WataruJames

Kendall

Liza

Neil

SelenaMai

TaylorImaizumi*

Kitazumi*

Ogawa*

avertedeye contacted

avertedeye contacted

Avec

Fig. 6 Eye-contact dataset using publicly available videos from YouTube or conversational scnario (names with *)J Intell Robot Syst
Table 2 First person eye-contact dataset (Conversation dataset)
Name

Duration
(sec)

Fps

Image size
(H,V)

Imaizumi
Kitazumi
Ogawa

220
711
1865

29.97
29.97
29.97

1920×1080
1920×1080
1920×1080[deg] for angular features. The distances larger than 1.0 [m]
1 indicateswere voted to the last bin. Thus, for example, fdist
the number of frames where the mutual facial distanceswere from 0.0 to 0.1 [m] and fr4x indicates the number offrames where the mutual facial rotation (rx ) was from -30
to -10 [deg].

3.4 Visualization
After obtaining histograms, we normalized the histogramand applied principal component analysis (PCA). While
many data analysis or machine learning techniques havebeen proposed, we used PCA for its simplicity andreliability in exploratory data analysis (EDA). Since wetried to find tender-care technique skills in a bottom-up(data-driven) manner, this nature of PCA fitted our task
better than other more complicated methods such as nonlinear or supervised-learning based approaches.
We here denote fs as a D × 1 column vector that
represents a normalized histogram of either fdist , frx , fry or
frz of a subject s ∈ {1, . . . , M}. Histograms of all subjects
can be decomposed by using D × 1 column eigenvectors{e1 , . . . , eM } and eigenvalues {λ1,1 , . . . , λD,M }, where D isthe dimension of the histogram:⎤⎡
λ1,1 · · · λ1,M
 
⎢
 1
⎥
f · · · fM = e1 · · · eD ⎣ ... . . . ... ⎦ .
λD,1 · · · λD,M

(a)

(b)

We plot the eigenvalues of all subjects to visualize the
distribution of their behaviors, as well as to analyze theelements of eigenvectors to find the relation between skill
levels and behavioral features.

3.5 Eye Contact Features
Another feature is counting eye contact bids, which wasintroduced by Ye et al. [44]. They assume ”eye contactbids”, i.e., situations when a subject wearing an FPV camera
is gazed at by other subjects. Since the definition of eye
contact is making mutual eye gaze—two people look at
each other at the same time – eye contact bids are not
the same as actual eye contact. If we want to accurately
detect eye contact, two people must wear FPV cameras
or a caregiver must use an eye gaze tracker (EGT) device
that detects observers’ gaze information. However, from thepractical point of view it is difficult to use two FPV cameras
or an EGT device since 1) it is difficult for subjects withdementia to wear such devices and 2) even for caregivers
it is difficult to use eye trackers in actual care scenes
due to their noticeable appearance, calibration requirements
and headmount drift. Therefore, rather than accurately
detecting eye contact, we tried to measure and use eye
contact bids for evaluating care skills. We used facial
poses and eye images for detecting eye contact bids using
DNNs.
Figure 5 illustrates the existing eye contact detection
algorithm (Fig. 5a) and proposed TempEC and TempECHP (Fig. 5b). The TempEC uses only eye images whereas
the TempEC-HP uses both eye images and 3D facial poses.
These algorithms consist of the following components.
Eye region detection: From the landmark points detectedby OpenFace, we obtained the right and left eye regionsin the target frame, from which we obtained each eye
image used as input for the CNN, after gray-scaling and(c)Fig. 7 Experimental scenes and example FPV frames of care learning scenes. a Data capturing, b first-person camera and recorder (Pupil Labs +smartphone) and (c) example frames (OpenFace annotations overlaid)

Avec
Aziz
Derek
Elle
Emma
WataruJames
Kendall
Liza
Neil
SelenaMai
TaylorImaizumi*
Kitazumi*
Ogawa*
Total

Name

8,940
6,711
4,550
6,458
6,302
1,306
3,270
4,331
7,440
8,904
6,052
2,565
9,444
2,343
20,36429,220128,200

Detected

Frames

10,498
11,508
11,765
11,73913,325
1,675
9,125
10,735
10,73916,487
11,043
9,840
13,945
6,594
21,33637,917208,271

Face

Total

Table 3 The results of Experiment 1

4,671
5,058
3,710
4,005
3,878
1,049
2,110
3,094
6,097
5,677
3,634
1,330
5,489
1,664
18,56028,032
98,058

Contacted

Eye0.88780.9386
0.88650.897
0.81960.87850.89570.87720.93130.84040.83220.73490.80850.565
0.90240.92420.8512

Precision

DeepEC [30]0.82620.86950.87580.73710.82780.91620.423
0.84230.85620.88940.71790.77620.66740.88730.65930.72070.7808

Recall
0.85590.90270.88110.80920.82370.897
0.57460.85940.89220.86420.77090.755
0.73120.69040.762
0.80980.8319F1
0.78810.88380.84950.85220.752
0.836
0.83060.81020.88380.42230.77670.698
0.74730.51650.79930.86220.7693

Precision
0.85840.90770.92920.77730.8821
0.96150.48030.894
0.91560.9031
0.78870.82310.73080.95430.75610.78010.8339RecallDeepEC+CRF [30]0.82170.89560.88760.813
0.81190.89440.60860.85
0.89440.57550.78260.75540.73890.67020.77710.81910.7876F1
0.87640.935
0.92440.84790.83340.84920.88390.8836
0.95390.84960.80180.70220.88890.61880.94730.90180.8561

Precision
0.888
0.89440.89240.90790.87010.97620.91020.917
0.894
0.97290.75170.94590.80280.91480.36120.771
0.8544

Recall

TempEC(Proposed)

0.88220.91420.90810.87690.85130.90830.89680.9
0.923
0.90710.776
0.806
0.84370.73820.523
0.83130.8706F1

0.88330.94710.87610.85560.84410.83070.91750.86540.93590.79520.7736
0.63150.8141
0.63330.90770.87140.8364

Precision

0.9241
0.9636
0.94780.94860.72730.99640.81970.91390.9144
0.97530.92820.988
0.95710.93120.88180.979
0.9248

Recall

TempEC-HP(Proposed)

0.90320.95530.9106
0.89970.78140.906
0.86580.889
0.925
0.87610.84390.77050.87980.75390.89450.92210.8781F1J Intell Robot Syst

J Intell Robot Syst
Table 4 t-test results (p-value) for Experiment 1
Methods

Precision

RecallF1 -score

DeepEC - DeepEC+CRF
DeepEC - TempEC
DeepEC - TempEC-HP
DeepEC+CRF - TempEC
DeepEC+CRF - TempEC-HP
TempEC - TempEC-HP

0.001∗∗
0.296
0.085
0.001∗∗
0.006∗∗
0.013∗

0.000∗∗
0.035∗
0.000∗∗
0.308
0.004∗∗
0.039∗

0.176
0.085
0.001∗∗
0.049∗
0.001∗∗
0.118

algorithm, each LSTM accepts 10 vectors corresponding toa series of eye images and outputs one 512-dimensionalfeature vector. In the TempEC-HP algorithm, a series of
3D vectors that represent 3D head positions are additionallyinput to LSTM.
However, we found that a naı̈ve LSTM could not perform
satisfactorily. To solve this problem, we prepared fully
connected layers that had 2048 (512×4) units at the last
frame, which accepted the outputs of the left and right
DeepEC’s and the LSTM’s cell state vectors. Because the
DeepEC results for the current frames are directly used foreye contact detection, and since temporal inference is also
merged to the fully connected layers, we were ultimately
able to obtain better results than the naı̈ve implementationsof DeepEC and LSTM.

The numbers with ∗ and ∗∗ indicates respectively ≤ 0.05 and ≤ 0.01

normalizing with global contrast normalization (GCN).
Using the landmarks as a basis, we obtained the coordinates
of four corner points that determine the eye region. At thistime, we applied a 10% margin to the height and width of
the region so that facial landmark detection errors can be
accepted.

4 Dataset
We prepared two dataset for learning and evaluating thealgorithms. The first one was an eye contact video dataset,
which we prepared by using publicly available videos
from YouTube and our original FPV videos that assume
conversational scenarios. The second one was obtained incare learning scenes. We recorded the FPV videos equippedto the caregiver during Humanitude care teaching classes.

4.1 First-person Eye Contact Video Dataset
The first-person eye contact video dataset was used forevaluating the eye contact detection performance. The
ground-truth eye contact states (1 or 0) were provided forevery video frame in the dataset. We asked three people toannotate the eye-contact states and set the eye contact status
as 1 if more than two annotators thought the eye contact was
engaged at the frame.

1.0

1.0

0.8

0.8

0.8

0.6

0.4

0.2

TempEC (AUC=0.85)
TempEC-HP (AUC=0.87)
DeepEC (AUC=0.82)
0.2

0.4

0.6

0.8

1.0

True Positive Rate

1.0

True Positive Rate

True Positive Rate

Deep temporal eye contact detection: Given the images ofboth eye regions and the 3D facial pose, we implemented
our two deep temporal eye contact detection algorithms, as
shown in Fig. 5b. The algorithms use ten continuous video
frames – the target frame and nine preceding frames – tomake predictions.
As shown in Fig. 5b, each of these eye image pairs ItR , ItL
were input to the CNN. This CNN had the same structure as
DeepEC with the exception of the last two fully connected
layers; namely, it had two streams and six layers— two
convolution layers followed by four max pooling layers. The
CNN outputs a pair of 512-dimensional feature vectors foreach eye image fR (ItR ) and fL (ItL ).
These feature vectors were input to two separate LSTM
networks for the left and right eye images. In the TempEC

0.6

0.4

0.2

TempEC (AUC=0.88)
TempEC-HP (AUC=0.88)
DeepEC (AUC=0.84)
0.2

0.4

0.6

0.8

0.6

0.4

0.2

TempEC (AUC=0.77)
TempEC-HP (AUC=0.82)
DeepEC (AUC=0.76)
0.2

0.4

0.6

0.8

False Positive Rate

False Positive Rate

False Positive Rate(a) Result of all dataset(b) Result of Youtube dataset(c) Result of conversation dataset

Fig. 8 ROC curves of (a) The test results with all dataset, b The test results with videos from Youtube, c The test results obtained during
conversation

J Intell Robot Syst
Table 5 The result of Experiment 2
Name

Expert A
Expert B
Middle A
Middle B
Middle C
Middle D
Middle E
Middle F
Middle G
Novice A
Novice B
Novice C

Total
framesFacedetected

Eye contacted
framesAv. mutual facial
distance [mm]Av. mutual facial pose [deg]
rx
ry

rz

3442
4852
3214
2659
2024
2775
4506
3062
2485
6287
6168
4710

2148 (62.4%)2070 (42.7%)
1857 (57.8%)1248 (46.9%)
1171 (57.9%)1108 (39.9%)2547 (56.5%)2176 (71.1%)
683 (27.5%)
1648 (26.2%)
2675 (43.4%)
1002 (21.3%)

1667 (48.4%)
1321 (27.2%)
1087 (33.8%)1011 (38%)
914 (45.2%)
817 (29.4%)
1989 (44.1%)
2114 (69%)
856 (34.4%)553 (8.8%)810 (13.1%)
699 (14.8%)

314.3
435.1
406.8
404.4
470.6
489.3
617.8
232.5
686.5
485.0
422.2
587.2

6.69
−9.31
−7.01
2.80
−7.80
−0.26
−5.30
7.91
−7.50
−12.98−10.08−4.92−23.10−19.98−30.61−37.09−33.80−26.71−31.63−22.04−29.99−39.37−40.19−25.00a) First-person eye contact video Youtube (Youtube
dataset)We used 13 videos in which a person talked into a camera from Youtube. We took a consensus of the annotations
and made ground-truth data. A list of the videos and theirproperties is shown in Table 1 and example frames areshown in Fig. 6.
b) First-person eye contact video dataset during conversation (Conversation dataset)
We additionally prepared first-person-view videos while
two individuals were conversing. The scenarios were taken
in a lab environment in which two participants weretalking, where one of them wore a Pivothead Kudu firstperson camera. A list of the videos and their properties isshown in Table 2 and Figure 6. We took three video clips
from six participants and two test-video clips from twoparticipants.

4.2 FPVs of Care-learning Scenes
To verify the applicability of the proposed algorithms, weprepared first-person videos of a) two Humanitude careexperts (instructors), b) seven middle-level Humanitude
caregivers and c) three novice Humanitude caregivers asshown in Table 5 and Fig. 7. In all videos, caregivers
were equipped with the Pupil Labs first-person camera andperformed the same task:
Step 1
Step 2
Step 3

Approach the simulated patient while making eye
contact,
Perform the care, andLeave the care receiver.

7.37
−0.75
3.57
0.46
−2.30
−4.07
0.91
4.50
−9.80
−7.37
−2.08
6.03

5 Experiments
In the first of two experiments we performed, we
evaluated the performance of eye contact (bids) algorithms,
comparing the two proposed approaches and an existing
approach (DeepEC [30]). The second experiment was
performed for an actual Humanitude care training scene.
In it, we obtained data from a novice caregiver and a
Humanitude care expert and compared the results throughthe use of an unsupervised learning algorithm.

5.1 Experiment 1: Evaluation of eye contact
detection performance
As mentioned, we first conducted an experiment to compare
the performances of the proposed algorithms and an existingalgorithm by using the datasets. One video was chosen for
testing and the others were used for learning. We iterated
this step for 16 videos and obtained the average performancefor them.
The learning of the networks with DeepEC, TempEC and
TempEC-HP was conducted as follows. We first computed
the bounding rectangles of eyes using the facial landmark
points obtained by OpenFace. The obtained eye images
were then rescaled such that the image was (60 × 36)
pixels. We used static CNN hyper-parameters for all of theexperiments. Specifically, the drop-out rate was 0.5 andLeaky ReLU activation function’s α was set to 0.01. We
used a Nadam optimizer [14] with the learning rate set to0.001, the decay as 0.004 per epoch and β1 and β2 as 0.9
and 0.999, respectively. Learning was performed on a GPUbased workstation (Intel Core i7-7800X CPU 3.50GHz,
128GB RAM, NVIDIA GeForce1080Ti-11G). On average,

rotation (rz)rotation (ry)rotation (rx)distance

J Intell Robot Syst

Fig. 9 (Left) Histograms of mutual facial distances and poses (rx , ry
and rz ). (Right) PCA score results where the x and y axes show the firstand second components. There are clear thresholds between novices

and others at about x = 0.16 for the distance PCA score, and between
experts and others at about x = −0.18 in rz PCA score

it took 230 sec, 1900 sec and 1920 sec for learning 1 epochof DeepEC, TempEC and TempEC+HP, respectively. The
learning of CRF of DeepEC-CRF took about 57 sec. We
took leave-one-out cross-validation strategy for splitting thelearning data and test data.

The results are given in Table 3 and t-test results between
four algorithms are given in Table 4. The results showthat TempEC+HP algorithms generally outperform otherapproaches. Namely, the TempEC algorithm thoroughly
outperforming DeepEC in precision = 0.8561, recall =J Intell Robot Syst

Frm 1

Frm 2

Frm 3

Frm 4

Frm 5

Frm 6

Frm 7

Frm 8

1
1
1
1

1
1
1
1

1
1
1
1

0
1
0
0

0
1
0
0

0
1
0
0

0
1
0
0

0
1
0
0

Frm 1

Frm 2

Frm 3

Frm 4

Frm 5

Frm 6

Frm 7

Frm 8

1
1
1
1

1
1
1
1

0
1
0
0

0
1
0
0

0
1
0
0

0
1
0
0

0
1
0
0

0
1
0
0

Frm 1

Frm 2

Frm 3

Frm 4

Frm 5

Frm 6

Frm 7

Frm 8

0
0
0
0

0
0
0
0

0
0
0
0

0
0
0
0

1
0
1
1

1
0
1
1

1
0
1
1

1
0
1
1

Avec

Ground Truth
DeepEC+CRF
TempEC
TempECHP

Emma

Ground Truth
DeepEC+CRF
TempEC
TempECHP

Kendall

Ground Truth
DeepEC+CRF
TempEC
TempECHP

Fig. 10 Example differences among DeepEC+CRF, TempEC and TempEC-HP. Adding CRF to DeepEC tends to ‘smoothen’ the temporal
inference while the other two algorithms correctly estimate the state change

0.8544 and F1 score = 0.8706. TempEC-HP outperformed
TempEC, DeepEC and DeepEC+CRF in recall and F1 score, with a recall of 0.9248 and F1 score of 0.8781 on
average. Figure 8 shows the area under the curve (AUC)of our algorithm is larger than that of DeepEC. TempECHP had the best AUC (0.870) followed by TempEC (AUC

= 0.85). With respect to accuracy, TempEC-HP achieved
a 25% improvement in miss detection rate, with 0.1751
in comparison to DeepEC’s 0.2330. Overall, the recall
performances increased by introducing temporal inference(CRF or LSTM). In addition, the proposed methods using
LSTM outperforms the method using CRF.

Related tonovices

Related toexperts(a) 1st PCA component (distance)
Fig. 11 1st components of PCA of the distance and rz histograms.
Negative elements are related to the experts’ behaviors while positiveelements are related to the novices’ behaviors. In the distance histogram, bins 2 and 3 (0.2 - 0.4 [m]) are related to experts while bins(b) 1st PCA component (rz)
4 and 5 (0.4 - 0.6 [cm]) are related to novices. Regarding the rz histogram, bins 4 and 5 (-30 - +10 [deg]) are related to experts while bins
2 and 3 (-70 - -30 [deg]) are related to novices

J Intell Robot Syst

5.2 Experiment 2: Observation of Face-to-Face
Communication Behavior During Care Learning
ScenesIn the second experiment we compared the occurrence of
Humanitude care skill between novice and expert caregivers
using the FPVs of care learning scenes. We obtained thenumber of eye contact frames, mutual facial distances andposes from a care scene dataset and compared the results.
Analysis and results: The occurrences of eye contact
frames, average mutual facial distance and poses (angles)
are shown in Table 5 and normalized histograms of eachfeature are shown in Fig. 9 (left). We applied PCA to
the histograms. The resulting PCA scores are shown in
Fig. 9 (right), where the x-axis shows the scores of thefirst component and the y-axis indicates the scores of thesecond PCA component. From the eye contact rates and
PCA analysis results, we were able to clearly distinguish thescores of novices and experts for eye contact rate, mutual
facial distance and rz PCA scores. There were significant
differences in eye contact rate between the expert & middlelevel and novice groups (p = 0.0452), and clear thresholds
at about x = 0.16 for mutual facial distance and at about
x = −0.18 for the rz PCA scores. In the mutual facial
distance category, the histograms showed that the expert
caregivers and most of the middle-level ones approached
the care receiver such that the distance was less than 30[cm]. In the mutual facial pose category, there were clear
dissimilarities in the z-rotation, which is the rotation of
the care receiver’s face in the FPV image plane (the plane
perpendicular to the facial frontal direction) as shown in
Fig. 4. Namely, the average and peak z-rotation values of theexperts and the middle-level caregivers were located around
0 [deg] while those of novices were much larger.

6 Discussion and Conclusion
In this section, we will discuss the main points we havemade for the proposed image-based eye contact detection
algorithms and wearable care-skill evaluation system anddraw conclusions from them.
Image-based eye contact detection: We developed eyecontact-detection algorithms that use temporal features aswell as static image features. Our algorithms show better
performance for various types of datasets. They combine
CNNs and LSTM and successfully learned both static features and temporal dependence. In experiments, the proposed TempEC and TempEC-HP algorithms outperformed
the DeepEC algorithm. In particular, TempEC-HP achieved
a 25% improvement in the miss-detection rate over existingalgorithms.

In a preliminary experiment, we found that a simple
concatenation of CNNs and LSTM was not effective.
We concluded that such a primitive combination was notsuitable for learning both static and temporal features at thesame time. Thus, in the final estimation step we introduced
a skip connection that jumps over the LSTM networks
and directly links the CNN outputs to the final fully
connected layers. This structure improved our algorithms’
performance, as the experiment 1 results showed. They
showed that the skip connection enables the algorithms to
successfully learn both static and temporal features at thesame time.
Surprisingly, some tests showed that TempEC performed
better than TempEC-HP. This was contrary to what we had
expected because we believed the facial pose information in
TempEC-HP would help in detecting eye contact for various
face directions. However, these results do not indicatethat facial pose information is useless. In our TempECHP algorithm, 3D facial pose estimation is based on facial
landmarks, the detection of which is mostly accurate but
has a certain degree of error. This error is not significant,which is why it is not a problem when used to obtaineye regions. However, in facial pose estimation, such asmall error sometimes causes a large incorrect gap between
two contiguous frames. The facial pose of two adjacentframes should be close because a human’s face cannot movevery much in a short time (namely, 0.03 sec because thevideos were recorded at 30 fps). Due to this problem, facialpose estimation is occasionally not sufficiently reliable,
which causes TempEC-HP to perform poorly. Hence, theperformance of TempEC-HP can be improved by using
a more accurate facial detection or facial pose estimation
algorithm.
Another notable finding was that introducing temporal
inference increased the algorithms’ recall performance,
which means the temporal information contributed to‘overlooked’ effects of eye contact. Our experiments
showed that adding a conditional random field (CRF) to the
DeepEC algorithm could not improve its results. Several
examples (Fig. 10) showed that CRF tends to ”smoothen”
temporal inference in DeepEC, which may help to avoid
the ‘jittering’ effects of single frame estimations but does
not solve the temporal inference problem fundamentally.
Thus, we believe our current algorithms, which combine theinternal states of single frame recognition and LSTM, are abetter solution.
Our results show that our algorithms enable excellent eye
contact detection performance to be achieved. They also
show the potential of temporal learning of eye behavior,
with which we can evaluate the care skills of caregivers.
Evaluation of wearable care-skill estimation system: Unsupervised analysis results of mutual facial distances andJ Intell Robot Syst

facial poses enabled us to find significant differences
between novices, middle-level and expert Humanitude caregivers. Specially, we found a clear threshold in eye contact
frequency and PCA scores of facial distance and rz -rotation
histograms, which indicate that the important skills in
Humanitude tender-care are related to a) frequent eye contact, b) a nearest mutual facial distance of less than 30 [cm]and c) mutual facial poses being in the same direction. Thiscan also be seen from the 1st PCA components of the distance and rz histograms (Fig. 11). In the distance histogram,bins 2 and 3 (0.2 - 0.4 [m]) are related to the experts whilebins 4 and 5 (0.4 - 0.6 [m]) are related to the novices. For
the rz histogram, bins 4 and 5 (-30 - +10 [deg]) are related
to the experts while bins 2 and 3 (-70 - -30 [deg]) are related
to the novices. These skills are a part of Humanitude gazeskill: caregivers should communicate to the care receivers
while keeping eye contact from a close distance and possessing the same facial angles of the care receiver’s face.
This is based on the idea of Humanitude care methodologythat all behaviors are considered to imply non-verbal messages. To have the eye contact straight in front of the carereceiver expresses the fairness, and the distance between
caregivers and care receivers reflects their friendliness. The
study results show that the experts expressed fairness andfriendliness much more than the novices. This skill is a coreskill with which to establish a good relationship that leads
to high-quality care.
Open issues and future work: Though our analyses canquantize and visualize the Humanitude care communication
skills, several open issues remain to increase the analysis
quality, as we ascertained from the responses of Humanitude
experts. The first point is the face detection stabilityof OpenFace. Specifically, OpenFace cannot detect carereceivers’ faces when x or y rotations are quite large(e.g., looks at the size of the faces). The second pointis the temporal analysis. The estimation of facial posesand distances is currently performed frame-by-frame and
TempEC considers only 1/3 seconds as the temporal
duration. However, it has been reported that the duration ofeye contact is about three seconds in typical communication
scenes [6] and that a longer duration of mutual gaze isoften effective in communicating with a dementia patient[3]. Thus, temporal inference using a longer duration can be
expected to be effective in care-skill evaluation as well.
The tender-care concept involves multi-modal skills
including gazing, speaking and touching. As the initial
step in computational care communication analysis, we
treated face-to-face communication skills. We are currently
developing a method for detecting and analyzing voice
signals and sensing touch behaviors through the use ofwearable contact sensors or vision analysis. We believe that
our findings for tender-care skills and systems for obtaining
care skills will prove to be important and usable, not only
for increasing the skills of caregivers but also for designingand evaluating care robots’ behaviors.
Acknowledgements All the experiments were conducted in compliance with the protocol which was reviewed and approved by the
ethical committee of Unit for Advanced Studies of the Human Mind,
Kyoto University (Permit Number: 30-P-4). This work was supported
by JST CREST Grant Number JPMJCR17A5 and JSPS KAKENHI
17H01779, Japan.
Open Access This article is distributed under the terms of theCreative Commons Attribution 4.0 International License (http://
creativecommons.org/licenses/by/4.0/), which permits unrestricted
use, distribution, and reproduction in any medium, provided you give
appropriate credit to the original author(s) and the source, provide alink to the Creative Commons license, and indicate if changes were
made.

References
1. Acton, G.J., Kang, J.: Interventions to reduce the burden of
caregiving for an adult with dementia: a meta-analysis. Res. Nurs.
Health 24(5), 349–360 (2001)
2. Adelman, R.D., Tmanova, L.L., Delgado, D., Dion, S., Lachs,
M.S.: Caregiver burden: a clinical review. Jama 311(10), 1052–
1060 (2014)
3. Alzheimer’s Society: Factsheet: Communicating. https://www.alzheimers.org.uk/site/scripts/documents info.php?
documentID=130, [Online; accessed 18-Nov-2016] (2016)
4. Baltrusaitis, T., Robinson, P., Morency, L.P.: (2016) OpenFace:
An Open source facial behavior analysis toolkit. 2016 IEEE
Winter Conference on Applications of Computer Vision WACV.
https://doi.org/10.1109/WACV.2016.7477553 (2016)
5. Bertasius, G., Park, H.S., Stella, X.Y., Shi, J.: Am I a Baller?
Basketball Performance Assessment from First-Person Videos. In:2017 IEEE International Conference on Computer Vision (ICCV),
pp. 2196–2204. IEEE (2017)
6. Binetti, N., Harrison, C., Coutrot, A., Johnston, A., Mareschal, I.:
Pupil dilation as an index of preferred mutual gaze duration. Royal
Society Open Science 3(7). https://doi.org/10.1098/rsos.160086,
http://rsos.royalsocietypublishing.org/content/3/7/160086.full.pdf(2016)
7. Biquand, S., Zittel, B.: Care giving and nursing, work conditions
and humanitude®. Work 41(Supplement 1), 1828–1831 (2012)
8. Boseley, S.: Dementia research funding to more than double to$66m by 2015. The Guardian (2012)
9. Campbell, L.C., Keefe, F.J., Scipio, C., McKee, D.C., Edwards,
C.L., Herman, S.H., Johnson, L.E., Colvin, O.M., McBride, C.M.,
Donatucci, C.: Facilitating research participation and improving
quality of life for african american prostate cancer survivors andtheir intimate partners: a pilot study of telephone-based coping
skills training. Cancer: Interdiscip. Int. J. Amer. Cancer Soc.
109(S2), 414–424 (2007)
10. Casado, B., Sacco, P.: Correlates of caregiver burden amongfamily caregivers of older korean americans. J. Gerontol. Ser. B:
Psychol. Sci. Soc. Sci. 67(3), 331–336 (2011)

J Intell Robot Syst
11. Chong, E., Chanda, K., Ye, Z., Southerland, A., Ruiz, N., Jones,
R.M., Rozga, A., Rehg, J.M.: Detecting gaze towards eyes innatural social interactions and its use in child assessment. Proc.
ACM Interact. Mob. Wearab. Ubiquit. Technol. 1(3), 43 (2017)
12. Clyburn, L.D., Stones, M.J., Hadjistavropoulos, T., Tuokko, H.,
et al.: Predicting caregiver burden and depression in alzheimer’s
disease. J. Gerontol. Ser. B 55(1), S2–S13 (2000)
13. Coon, D.W., Thompson, L., Steffen, A., Sorocco, K., GallagherThompson, D.: Anger and depression management: psychoeducational skill training interventions for women caregivers of a
relative with dementia. Gerontologist 43(5), 678–689 (2003)
14. Dozat, T.: Incorporating nesterov momentum into adam. http://
cs229.stanford.edu/proj2015/054 report.pdf, [Online; accessed
25-Aug-2018] (2016)15. Dunkin, J.J., Anderson-Hanley, C.: Dementia caregiver burden:
a review of the literature and guidelines for assessment andintervention. Neurology 51(1 Suppl 1), S53–S60 (1998)
16. Etters, L., Goodall, D., Harrison, B.E.: Caregiver burden among
dementia patient caregivers: a review of the literature. J. Am.
Acad. Nurse Pract. 20(8), 423–428 (2008)
17. Fathi, A., Hodgins, J.K., Rehg, J.M.: Social Interactions: a FirstPerson Perspective. In: 2012 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 1226–1233. IEEE
(2012)
18. Gineste, Y., Pellissier, J.: Humanitude: comprendre la vieillesse,
prendre soin des hommes vieux. A. Colin (2007)
19. Given, B., Sherwood, P.R., Given, C.W.: What knowledge andskills do caregivers need? J. Soc. Work. Educ. 44(sup3), 115–123(2008)
20. Honda, M., Ito, M., Ishikawa, S., Takebayashi, Y., Tierney, L.:
Reduction of behavioral psychological symptoms of dementia bymultimodal comprehensive care for vulnerable geriatric patientsin an acute care hospital: a case series. Case reports in medicine
2016 (2016)
21. Ishikawa, S., Ito, M., Honda, M., Takebayashi, Y.: The skill
representation of a multimodal communication care method forpeople with dementia. JJAP Conf. Proc. 011616, 4 (2016)
22. Ito, M., Honda, M.: An examination of the influence ofhumanitude caregiving on the behavior of older adults withdementia in japan. In: Proceedings of the 8th International
Association of Gerontology and Geriatrics European Region
Congress (2015)
23. Kim, H., Chang, M., Rose, K., Kim, S.: Predictors of caregiverburden in caregivers of individuals with dementia. J. Adv. Nurs.
68(4), 846–855 (2012)
24. Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar,
S., Matusik, W., Torralba, A.: Eye Tracking for Everyone. In: The
IEEE Conference on Computer Vision and Pattern Recognition(CVPR) (2016)
25. Larson, E.B., Yaffe, K., Langa, K.M.: New insights into thedementia epidemic. Engl. J. Med. 369(24), 2275–2277 (2013)
26. Law, H., Ghani, K., Deng, J.: Surgeon technical skill assessment
using computer vision based analysis. In: Machine Learning for
Healthcare Conference, pp. 88–99 (2017)
27. Lu, F., Sugano, Y., Okabe, T., Sato, Y.: Gaze estimation from eye
appearance: a head pose-free method via eye image synthesis.
IEEE Trans. Image Process. 24(11), 3680–3693 (2015)28. Ma, M., Fan, H., Kitani, K.M.: Going deeper into first-person
activity recognition. In: Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, pp. 1894–1903 (2016)29. Ministry of Health, Labour and Welfare, Japan: Supply anddemand estimation for nursing care personnel for 2025. https://www.mhlw.go.jp/stf/houdou/0000088998.html, [Online; accessed
11-Aug-2018] (2015)30. Mitsuzumi, Y., Nakazawa, A., Nishida, T.: Deep Eye Contact
Detector: Robust Eye Contact Bid Detection Using Convolutional
Neural Network. In: Proceedings of the British Machine Vision
Conference (BMVC) (2017)31. Northouse, L.L., Katapodi, M.C., Song, L., Zhang, L., Mood,
D.W.: Interventions with family caregivers of cancer patients:
meta-analysis of randomized trials. CA: Cancer J. Clin. 60(5),317–339 (2010)
32. Ostwald, S.K., Hepburn, K.W., Caron, W., Burns, T., Mantell,R.: Reducing caregiver burden: a randomized psychoeducational
intervention for caregivers of persons with dementia. Gerontologist 39(3), 299–309 (1999)
33. Papastavrou, E., Kalokerinou, A., Papacostas, S.S., Tsangari, H.,
Sourtzi, P.: Caring for a relative with dementia: family caregiverburden. J. Adv. Nurs. 58(5), 446–457 (2007)
34. Petric, F., Miklić, D., kovačić, Z.: Probabilistic eye contact
detection for the robot-assisted asd diagnostic protocol. In:
Lončarić S., Cupec, R. (eds.) Proceedings of the Croatian Compter
Vision Workshop, Year 4, Center of Excellence for Computer
Vision, pp. 3–8. University of Zagreb, Osijek (2016)35. Povithead: Pivothead KUDU. http://www.pivothead.com/,[Online; accessed 29-Aug-2016] (2016)36. Pupil Labs: Pupil labs camera system. https://pupil-labs.com/
pupil/, [Online; accessed 25-Aug-2018] (2018)37. Singh, S., Arora, C., Jawahar, C.: First person action recognition
using deep learned descriptors. In: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, pp
2620–2628 (2016)38. Smith BA, Yin Q, Feiner SK, Nayar SK: Gaze locking: passive eye
contact detection for human-object interaction. In: Proceedings ofthe 26th annual ACM symposium on User interface software andtechnology, pp. 271–280. ACM (2013)
39. Soo Park, H., Shi, J., et al.: Force from motion: decoding physicalsensation in a first person video. In: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, pp.
3834–3842 (2016)40. Su, S., Hong, J.P., Shi, J., Park, H.S.: Predicting Behaviors ofBasketball Players from First Person Videos. In: CVPR, vol. 2, pp.
3 (2017)
41. Win, K.K., Chong, M.S., Ali, N., Chan, M., Lim, W.S.: Burden
among family caregivers of dementia in the oldest-old: an
exploratory study. Front. Med. 4, 205 (2017)
42. World Health Organization, et al.: Dementia: Fact sheet N 362(2012)
43. Ye, Z., Li, Y., Fathi, A., Han, Y., Rozga, A., Abowd, G.D., Rehg,
J.M.: Detecting eye contact using wearable eye-tracking glasses.
In: Proceedings of the 2012 ACM Conference on Ubiquitous
Computing, pp. 699–704. ACM (2012a)
44. Ye, Z., Li, Y., Fathi, A., Han, Y., Rozga, A., Abowd, G.D.,
Rehg, J.M.: Detecting eye contact using wearable eye-tracking
glasses. In: Proceedings of the 2012 ACM Conference on
Ubiquitous Computing, UbiComp’12, pp 699–704. ACM, New
York, https://doi.org/10.1145/2370216.2370368 (2012b)
45. Ye, Z., Li, Y., Liu, Y., Bridges, C., Rozga, A., Rehg, J.M.:
Detecting Bids for Eye Contact Using a Wearable Camera. In:2015 11Th IEEE International Conference and Workshops On
Automatic Face and Gesture Recognition (FG), vol. 1, pp. 1–8.
IEEE (2015)
46. Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: Appearancebased gaze estimation in the wild. In: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, pp.
4511–4520 (2015)
47. Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: It’s written all over your face: Full-face appearance-based gaze

J Intell Robot Systestimation. arXiv:1611.08860, https://perceptual.mpi-inf.mpg.de/
wp-content/blogs.dir/12/files/2016/11/zhang16 arxiv.pdf (2016)
48. Zhang, X., Sugano, Y., Bulling, A.: Everyday eye contact detection using unsupervised gaze target discovery. In: Proceedings ofthe 30th Annual ACM Symposium on User Interface Software andTechnology, pp. 193–203. ACM (2017)
Publisher’s Note Springer Nature remains neutral with regard tojurisdictional claims in published maps and institutional affiliations.
Atsushi Nakazawa is an associate professor in the Department ofInformatics at the Kyoto University. He received his doctorate fromthe Osaka University in 2001 in Systems Engineering. Afterward,
he worked in Institute of Industrial Science, University of Tokyoand then in Cybermedia Center, Osaka University. From 2013, hejoined the Kyoto University. From Oct. 2017, he becomes a program
investigator (PI) of the JST CREST project “Computational andcognitive neuroscientific approaches for understanding the tendercare”. His research interests are in human behavior/mental analysis
using computer vision, eye tracking, eye imaging and motion capturesystems. Dr. Nakazawa got the best paper award in International Conf.
on Virtual Systems & Multimedia (VSMM2004) and Japan Robotics
Society (RSJ).
Yu Mitsuzumi is an employee at NTT Communication Laboratories.
He received Master of Infromatics from Kyoto University in 2019. His
research interests include computer vision, machine learning and deepneural nets.
Yuki Watanabe is a master course student in the Department ofInformatics at the Kyoto University. He received B.E from Faculty ofEngineering, Kyoto University in 2018, then he joined the Department
of Informatics, Kyoto University. His research interests include
computer vision, deep neural nets and computer animation.

Ryo Kurazume is a Professor at the Graduate School of Information
Science and Electrical Engineering, Kyushu University. He received
his M.Eng. and PhD in Mechanical Engineering from Tokyo Instituteof Technology in 1989 and 1998. He was a director of the Robotics
Society of Japan (RSJ) from 2009 to 2011 and 2014 to 2015,
and a director of the Society of Instrument and Control Engineers(SICE) from 2013 to 2015. Currently he is a chairman of the Japan
Society of Mechanical Engineers (JSME) Robotics and Mechatronics
Division. He received JSME Robotics and Mechatronics Academic
Achievement Award in 2012, RSJ Fellow in 2016, SICE System
Integration Division Academic Achievement Award in 2017, andJSME Fellow in 2018. His current research interests include legged
robot control, computer vision, multiple mobile robots, service robots,
medical imaging, and biometrics.

Sakiko Yoshikawa is a professor of Psychology, Kokoro ResearchCenter, Kyoto University. She received her Ph.D. from Kyoto
University in Education. After working at the Psychology Department
of Otemon-Gakuin University, she joined the Kyoto University in
1997. From 2007 to 2018 she worked as a first director of Kokoro
Research Center that promotes interdisciplinary research on the humanmind. Her recent research interests include basic mechanisms offace-to-face communication and recognition of social signals fromfaces. She received a Distinguished Paper Award from the Japan
Psychological Association and a Best Paper Award of JPA annual
convention in 2017.

Miwako Honda is director of Geriatric Research Center, National
Hospital Organization Tokyo Medical Center. After graduating the
Department of Medicine, Tsukuba University at 1993, she became
the clinical fellow of general medicine, Kameda General Hospital,
then worked as the Clinical fellow of infectious disease, International
Medical Center of Japan during 1997-1998, followed by the Resident
of internal medicine, Thomas Jefferson University, Philadelphia, andclinical fellow of geriatrics, at Cornel University. From 2002 to 2011,
she joined International Medical Center of Japan. From 2011, she
worked in the current position. She wrote more than five books
regarding geriatric care and Humanitude.