Development of dementia care training system based on augmented
reality and whole body wearable tactile sensor
Tomoki Hiramatsu1 , Masaya Kamei1 , Daiji Inoue1 , Akihiro Kawamura2 , Qi An2 , and Ryo Kurazume2
Abstract— This study develops a training system for a multimodal comprehensive care methodology for dementia patients
called Humanitude. Humanitude has attracted much attention
as a gentle and effective care technique. It consists of fourmain techniques, namely, eye contact, verbal communication,
touch, and standing up, and more than 150 care elements.
Learning Humanitude thus requires much time. To provide aneffective training system for Humanitude, we develop a training
system that realizes sensing and interaction simultaneously by
combining a real entity and augmented reality technology. To
imitate the interaction between a patient and a caregiver, we
superimpose a three-dimensional CG model of a patient’s face
onto the head of a soft doll using augmented reality technology.
Touch information such as position and force is sensed using
the whole body wearable tactile sensor developed to quantify
touch skills. This training system enables the evaluation of eye
contact and touch skills simultaneously. We build a prototypeof the proposed training system and evaluate the usefulness ofthe system in public lectures.

I. INTRODUCTION
The burden of caring for elderly people whose cognitive
function has declined due to dementia has rapidly increasedin hospitals and nursing facilities. Care for elderly dementia
patients is difficult and thus there is a shortage of caregivers
and nurses. Such care also leads to exhaustion and burnout among staff and greatly decreases the physical functionof the patients themselves. To tackle these issues, a multimodal comprehensive care methodology called Humanitude[1] has attracted much attention for the care of elderly
dementia patients. Humanitude mainly consists of four caretechniques: eye contact, verbal communication, touch, andstanding up. Although some reports have shown that Humanitude is quite effective in suppressing the behavioral andpsychological symptoms of dementia (BPSD) in patients andmakes it possible to provide nursing care safely, calmly, andeffectively, the training of the various techniques used in
Humanitude simultaneously is very difficult. To overcome
this problem and facilitate the use of Humanitude not only
in hospitals and care facilities but also homes, we havebeen developing a system for the quantification of careskills and an effective training system for Humanitude using
information technology [2][3].
*This work was supported by JST CREST Grant Number JPMJCR17A5,
Japan
1 Tomoki Hiramatsu, Masaya Kamei, and Daiji Inoue are with Graduate
School of Information Science and Electrical Engineering, Kyushu University, 744 Motooka, Nishi-ku, Fukuoka 819-0395, Japan {hiramatsu,

inoue}@irvs.ait.kyushu-u.ac.jp

1 Akihiro Kawamura and Ryo Kurazume are with Faculty of Information Science and Electrical Engineering, Kyushu University, 744
Motooka, Nishi-ku, Fukuoka 819-0395, Japan {kawamura, anqi,

kurazume}@ait.kyushu-u.ac.jp

In this paper, we discuss measurements of the eye contact
and touch skills used in Humanitude and propose a training
system for Humanitude based on augmented reality (AR)technology. We built a whole body wearable tactile sensor
to measure touch skill during a care action. To quantify touchskill, we need to measure the position and force of a touch
applied to a patient’s body by a caregiver. The sensor consistsof conductive fiber sheets and a capacitance measurement
circuit. The proposed training system for Humanitude uses
HoloLens (Microsoft) and a soft doll equipped with thetactile sensor. To imitate the interaction between a patientand a caregiver, we superimpose a three-dimensional CG
model of a patient’s face onto the head of the soft dollusing AR technology. The gaze direction, blinking, and facial
expressions of the patient are dynamically controlled. The
eye contact between the patient and the caregiver is detected
and evaluated on-line. The position and force of a touch
applied by the caregiver are measured by the whole body
wearable tactile sensor.
II. H UMANITUDE
Humanitude is a multimodal comprehensive care methodology for dementia patients developed by Gineste andMarescotti [1]. This methodology consists of four main techniques, namely, eye contact, verbal communication, touch,and standing up, and more than 150 care elements. Several
studies have reported that the cost efficiency of introducing
Humanitude is around 20 times higher than that of carewithout it because of a 40% decrease in the use of psychotropic drugs and the number of care staff who leave[4]. One study [5] reported a case series for three dementia
patients and showed that the duration of the behavioral
and psychological symptoms of dementia (BPSD) decreased
from 25.0%, 25.4%, and 66.3% for conventional care to 0%,0%, and 0.3% for Humanitude, respectively. Humanitude
has been introduced in more than 600 hospitals and nursing
homes in Europe and has recently become popular in Japan.
In the past three years, more than 2,600 people in Japan haveundergone Humanitude training, which includes more than
30 training sessions.
III. D EVELOPMENT OF WHOLE BODY WEARABLE
TACTILE SENSOR

In this section, we describe the developed whole body
wearable tactile sensor for measuring the position and force
of a touch applied to a patient’s body by a caregiver. Two
methods are considered to be used to obtain the touch
information. In the first method, a sensor glove that can

A. Tactile sensor based on conductive fiber sheets
We built a whole body wearable tactile sensor that consistsof conductive fiber sheets and capacitance measurement
circuits. Figure 1 shows a conductive fiber sheet with conductive fibers (Tsuchiya Inc.). By stacking two conductive
fiber sheets, the force applied to the sheets can be measured
by reading the difference in capacitance between the two
sheets. Since this sensor is made from fiber sheets, it can beeasily cut and folded to fit the shape of a body. The tactile
sensor consists of square sensor cells. The minimum size ofa sensor cell (maximum resolution) is 10 × 10 mm2 . By
connecting several sensor cells, the resolution ranges from10 × 10 mm2 to 30 × 30 mm2 .
10mm

Circuitis 20 × 20 mm2 and the sampling rate is 3.5Hz for wholesensor cells.

Fig. 2.

whole body wearable tactile sensor

Figure 3 shows sensor values for various forces measured
by a force gauge (Fig. 1). It is clear that the sensor cell has
non-linearity and the measurement range is about 0.1 to 500
N or more.
800
600

Sensor value

Sensor value

measure finger tactile pressure [6] and a motion capturesystem [7] are used. The timing and pressure are measured
by the sensor glove and the touch position is estimatedfrom the patient’s posture and the hand position of the
caregiver measured by the motion capture system. The two
sensor systems (tactile sensor and motion capture) must be
synchronized, making this method complex.
In the second method, a distributed tactile sensor wornby the patient is used. Since the position and force can bemeasured by a single sensor, the system is simple and easy
to use. However, conventional force distribution sensors suchas BPMS (Nitta) [8] or pliance (novel GmbH) [9] are quite
expensive and difficult to cut or fold, making them difficult
to use in a training system at care facilities or homes. A
piezoelectric fabric sensor [10], pressure sheets using carbon
microcoils [11] and conductive ink [12], and carbon nanotube
elastomers [13] have been proposed. However, these sensors
are small and thus cannot cover the whole area of a patient’s
body.
In this study, we developed a wearable tactile sensor thatcovers the whole body (arms, legs, and torso). The touchskill of a caregiver is quantified through the visualization of
the force applied to the patient’s body.

400
200
0

0

100

200 300
Force [N]Fig. 3.

400

500

140
120
100
80
60
40
20
0

0

10

20
30
Force [N]40

50

Sensor values for various forces

Figures 4 and 5 show the force distribution measured by
the whole body wearable tactile sensor when a Humanitude
instructor and a trainee give a posture change motion for apatient. In these figures, yellow and blue areas indicate highand low forces, respectively. In Fig. 4, it can be seen that the
Humanitude instructor uses not only the hands but also theupper arm to lift the patient’s body. The instructor did not
grasp the patient’s leg but pulled behind the thigh with theright palm. On the other hand, the trainee grasped a backof the knee and applied strong force. Figure 6 shows theforce measured by the cell that indicates the maximum force
during the contact. The left figure shows the force appliedby the left hand of the Humanitude instructor and the rightfigure shows the force by the left hand of the trainee. As
shown in this figure, the Humanitude instructor lifted thethigh more gently compared with the trainee.
IV. T RAINING SYSTEM FOR H UMANITUDE BASED ON AR
TECHNOLOGY

Fig. 1. Conductive fiber sheets (Tsuchiya Inc.) and sensor calibration by
a force gauge

Next, we built a whole body wearable tactile sensor byforming the conductive fiber sheets into a shirt and pants.
Figure 2 shows the components of the whole body wearable
tactile sensor. This sensor consists of six tactile sensor sheets
and capacitance measurement circuits. The sensor data aretransmitted to a PC via Bluetooth and a wire. The numbersof sensor cells are 5,754 in total, which are 1,282 for body,
1,320 for two arms, and 3,152 for two legs. The resolution

The methodology of Humanitude consists of four maintechniques and more than 150 care elements. Therefore,
learning Humanitude requires much time. To train care techniques effectively, the quantification of care skills performedby trainees is crucial. However, it is almost impossible todirectly attach many sensors to a dementia patient. Therefore,a training doll is sometimes utilized in the role of a patient,but the interaction between a patient and a caregiver, which
is important for Humanitude, cannot be simulated using thissystem. Instead of a doll, a person can imitate a dementia
patient during training. However, in Humanitude, the face of

200

150

150

100

100

Force [N]Force [N]

200

50
0

-100

-100
0
Left arm

0
-50

-50

Right arm

50

5Time [s]100(a) Humanitude instructor

5Time [s]10(b) Trainee

Fig. 6. Force applied by the left hand of the Humanitude instructor andthe left hand of the trainee to the patient’s thigh and the back of the knee

Note that the training system based on AR technology was
originally developed by Exawizards Inc.
Fig. 4. Sensor measurement of posture change by a Humanitude instructor

Virtual face

HoloLens

Real body

Soft doll
Wearableortactile sensor

Right arm

Left arm

AR marker
Fig. 7. Training system for Humanitude using a virtual face and real body

A. Evaluation of touch skill using whole body wearable
tactile sensor
Fig. 5.

Sensor measurement of posture change by a trainee

the caregiver must be placed quite close to the patient’s face(< 200 mm) and the caregiver has to keep eye contact withthe patient for as long as possible, which is quite unnaturalin daily life. Therefore, the person playing the role of thepatient needs special skills and thus has to be trained.
To solve these problems, we constructed a training systemthat realizes sensing and interaction simultaneously by combining a real entity and AR technology (Fig. 7). This system
consists of a soft doll equipped with a whole body wearable
tactile sensor (see Section 3) and a head-mounted AR display(HoloLens, Microsoft). The virtual face of a patient, witheyes that gaze and blink, is generated and superimposedon the head of the soft doll using AR technology. The
caregiver can see the facial expression of the patient throughthe AR display. Eye contact can be detected if we assume
that the facial direction of the caregiver coincides with thedirection of the head-mounted AR display. In addition, thecaregiver can touch the surface of the doll’s body and gettactile feedback. The position and force of the touch aremeasured by the whole body wearable tactile sensor on thedoll. Therefore, we can quantify the eye contact and touchskills of a caregiver using the proposed training system.

In order to quantify the touch skill of Humanitude, thewhole body wearable tactile sensor was attached to the bodyof the soft doll (Fig. 7). In Humanitude, a caregiver should
touch a patient’s body as softly as possible and never hit or
grasp the body. In addition, the caregiver should start thetouch at the patient’s arms or back and then move the touch
position slowly from the back to the front of the patient’s
body. With the whole body wearable tactile sensor, we canmeasure the position and force distribution of the touch
applied by a trainee and evaluate the quality of the touchskill.
To evaluate touch skill, we developed the decision tree
from observations of the care actions of expert caregivers.
For example, the instructor does not touch the front of thepatient’s body, but initially touches the back and the shoulder
gently. Thus, we divided the touch sensor on upper body
into 10 parts, and determined the positive and negative parts
to be touched. The proper force applied to the patient isalso measured through the demonstration by the instructor. In
most cases, the instructor touched the patient’s arm with lessthan 8 N and body with less than 2 N, and we determined thethreshold to be 8 N and 2 N, respectively. Also, the instructor
spreads his/her fingers and touches the body over a large area,
so contact areas are also included to determine touch skills(> 5000 mm2 ). According to the decision tree, if the contact

position, the force, and the contact area are appropriate, themessage “Excellent!” is displayed in the AR display in realtime, as shown in Fig. 8. However, if not, the warnings suchas “Please touch the patient’s back”, “Please touch a widerarea”, or “Too strong!” are displayed.

9XIRULD
9+ROR/HQV
0LFURVRIW$5 PDUNHU,QLWLDOUHJLVWUDWLRQXVLQJ$5PDUNHU

3LQFKPRWLRQ

3UHFLVHUHJLVWUDWLRQXVLQJSLQFKPRWLRQ
Fig. 9.
Fig. 8.

Registration of facial image on a doll head [16][15]Touch skill instructions

B. Evaluation of eye contact skill using head-mounted AR
display
As mentioned, the face of the patient is generated andsuperimposed on the head of the doll using a head-mounted
AR display. This allows virtual interaction between the avatarof the patient and the caregiver. The eye direction, blinking,and facial expression can be controlled using the training
system. The facial model in this system was developed using
the open-source software MakeHuman[14]. The blinking
and changes in eye direction of the avatar occur randomly.
However, once eye contact with the caregiver is established,
the avatar keeps eye contact as long as the caregiver keeps
looking at the patient’s eyes.
The relative position of the faces of the patient and the
caregiver is measured as shown in Fig. 9. The initial relative
position is determined by detecting the AR markers on
the body of the doll using the cameras on HoloLens. Thisinitial calibration process was developed using the AR toolkit
Vuforia[15]. After the initial calibration, precise calibrationis conducted manually by applying a pinch motion to the
AR model to precisely place the facial region on the headof the doll. Figure 10 shows the superimposed face and theactual body of the doll after calibration. Trainees can see
the superimposed facial images on the head of the doll with
HoloLens.
Figure 11 shows six facial expressions implemented inthis system: happiness, anger, surprise, fear, and sadness.
Each facial expression corresponds to the touch and theeye contact scores. When the eye contact is established, thehappiness facial expression appears. On the other hand, if the
trainee touches inappropriate position of the body such as
the chest, the surprise facial expression appears. The angler
facial expression is displayed when the trainee grasps or
touches body strongly.

Fig. 10.

Superimposed facial image

We evaluated eye contact skill in terms of gaze, distance,
and inclination, as shown in Table I. The scores are shownin the AR display as “Poor”, “Good”, or “Excellent”.
TABLE IE VALUATION INDICES
Index
Gaze
Distance
Inclination

Description
Difference between eye directions of avatar and trainee
Distance between faces of avatar and trainee
Relative angle between faces of avatar and trainee

1) Gaze score: HoloLens does not have a built-in eye
tracking function. We thus assumed that the direction of
HoloLens coincides with the eye direction of a trainee.
The gaze score is assigned as “Excellent” if the directionof HoloLens remains toward the avatar’s face and the eye
direction of the avatar remains toward HoloLens. “Good”
is assigned if the direction of HoloLens remains toward
the avatar’s face but the eye direction of the avatar does
not remain toward HoloLens. “Poor” is assigned if neither
direction is toward the target. The threshold for determining
eye contact is 15 degrees and 25 degrees in the first andsecond experiments in Section V-B, respectively.

C. Two training modes

Happiness

Anger

Sadness

Fear
Fig. 11.

SurpriseWe develop two kinds of modes for training Humanitude:
free training mode and scenario-based training mode.
1) Free training mode: In this mode, a trainee performs
care action freely and the evaluation score during the careaction is displayed in AR display on-line. The obtained
scores are displayed as a number of asterisks, as shown in
Fig. 13, where one, two, and three asterisks correspond to“Excellent”, “Good”, and “Poor”, respectively. In this mode,the care action of the trainee in daily work is evaluated fromthe viewpoint of Humanitude.

Facial expressions

2) Distance score: The distance between the avatar’s face
and the trainee’s face can be measured using the positioning
tracking function of HoloLens. In Humanitude, it is recommended that the distance between faces be around 200 mm.
“Excellent”, “Good”, and “Poor” are assigned if the distanceis less than 300 mm, between 300 mm and 600 mm, andmore than 600 mm, respectively3) Inclination score: The inclination score is adopted
from the concept of mutual facial position [2]. This scorereflects the relative inclination angle between the faces ofthe patient and the caregiver. Observations of care actions
by experts in Humanitude indicate that they keep the verticaldirection of their faces parallel to the vertical direction of
the patient’s face, as shown in Fig. 12(a). In the developed
system, in addition to the alignment of the vertical directionsof faces, we introduce a score of the relative position (height)of faces, as shown in Fig. 12(b). The caregiver should lookat the patient’s face from the same height, and thus thisscore decreases if the trainee looks from above or below thepatient’s face. The sum of these two scores is the inclination
score (“Excellent”, “Good”, or “Poor”). “Excellent”, “Good”,
and “Poor” are assigned if the relative angle is less than 10
degrees, between 10 degrees and 20 degrees, and more than
20 degrees, respectively.

Fig. 13.

Free training mode. Asterisks indicate each score.

2) Scenario-based training mode: In this mode, a trainee
performs care action according to the instruction by thesystem. The scenario implemented currently is as follows:
1) The trainee approaches the patient (doll) from the left
side along the arrow displayed on the HoloLens andestablishes eye contact.
2) The trainee touches gently the patient’s shoulder.
3) The patient changes his gaze to the right.
4) The trainee changes the standing position and catches
the gaze again while touching the patient’s shoulders.
5) The trainee rifts the patient’s left arm to encourage
standing up.
Figure 14 shows the images from the trainee’s viewpoint.
The scores of eye contact and gentle touch are displayed on
the left corner of the display.
V. T RAINING EXPERIMENT
Finally, we conducted training experiments of care skillsof Humanitude. Firstly, Humanitude instructors used thedeveloped care training system and we received valuablecomments to improve the system such as the natural responseof a dementia patient. After modifying the system according
to the comments, 12 people tested the system. Six of them
have dementia patients in their families.
A. Evaluation by instructors

Fig. 12.

Relative facial positions

Two Humanitude certified instructors, a medical doctor,
and three trainees tried the developed system and evaluatedthe validity for the training of care skills. Figure 15 showsthe experiment by the certified instructor. After the trials, weinterviewed them and received various comments. Most ofcomments are positive, such as this system is very suitablefor beginners since he/she has never experienced of keeping
eye contact with the patients from such as a close distance.

1. Approach while
catching gaze

2. Touch the shoulder
gently3. The patient changes
eye direction

4. Change position
while touching

can catch the gaze more easily. In addition, we revised thescenario so that the instruction can be understood moreintuitively and the direction of approach is indicated by
the red arrow. The evaluation score and comments after thesecond public lecture was greatly improved and the positive
comments were increased.
TABLE III MPRESSION OF THE TRAINING SYSTEM OBTAINED BY THE INTERVIEWAFTER THE TRAINING

Naturalness (1:Low - 5:High)
Usefulness (1:Low - 5:High)

5. Rift the arm tostand up

1st
1.8
3.3

2nd
3
4.3

TABLE IIIQ UESTIONNAIRE RESULTS
Fig. 14.

Scenario-based training mode

They also said this system is useful for training hand-eye
coordination, which is critical for Humanitude but hard to be
trained by conventional voice instruction. On the other hand,
instructors pointed out some weaknesses to be improved.
For example, the facial expressions are very important toknow the patient’s emotion. The patients tend to maintaineye contact once the eye contact has been established.

Fig. 15.

Question
Positive
I could feel the eye contact
I could learn the face distanceI could learn how to catch gaze
I could learn how to touch gently
Negative
I could not learn the eye contact
I could not feel the facial expressions
I could not learn the proper force
The field of view of HoloLens was narrow

Votes (≤ 6)
1st
2nd
1
3
1
5
0
2
2
2
1st
2nd
5
2
6
2
2
2
4
4

Trials by instructors

B. Training experiment by Humanitude trainees
According to the comments mentioned above, we improved the system so that the patient changes his facial
expressions and keeps eye contact once the eye contact has
been established as mentioned in Section 3.
After modifying the system, we tested the system in two
public lectures for dementia care. 6 people participated thefirst public lecture. They have dementia patients in theirfamilies and care for them every day at home. In the second
public lecture, 6 people who are interested in dementia careparticipated and tested.
Tables II and III show the results of interview after the
training. The evaluation score and comments after the firstpublic lecture was very harsh. This is because they could
hardly catch the gaze of the patients due to the inappropriate
threshold value for judging the eye contact. In the second
public lecture, the threshold was adjusted so that the trainee

Fig. 16.

Training in public lecture

VI. C ONCLUSIONSIn this paper, we proposed a whole body wearable tactile
sensor for quantifying the touch skill used in Humanitude.
In addition, we proposed a training system that combines anactual entity and AR technology. This system consists of asoft doll equipped with a whole body wearable tactile sensor
and a head-mounted AR display. It realizes sensing and theinteraction used in Humanitude simultaneously. We think the
proposed system is suitable for beginners since the distanceand the eye contact used in Humanitude are very special andhave never been experienced for most of trainees. In future
work, we will introduce the proposed training system to public lectures or courses of Humanitude at nursing schools. We
will also perform comparative experiments before and aftertraining to more carefully investigate whether the proposedsystem is sufficiently realistic and useful for actual training.
ACKNOWLEDGMENT
This work was supported by JST CREST Grant Number
JPMJCR17A5, Japan.
R EFERENCES[1] Y. Gineste and J. Pellissier, Humanitude : Comprendre la vieillesse,
prendre soin des Hommes vieux. Armand Colin, 2007.
[2] A. Nakazawa, R. Kurazume, M. Honda, S. Wataru, S. Ishikawa,
S. Yoshikawa, and M. Ito, “Computational tender-care science: Computational and cognitive neuroscientific approaches for understanding
the tender care,” IUI Workshop on Symbiotic Interaction and Harmonius Collaboration for Wisdom Computing, vol. 1, pp. 1–9, 2018.
[3] A. Nakazawa, R. Kurazume, M. Honda, S. Wataru, S. Ishikawa,
S. Yoshikawa, and M. Ito, “Computational tender-care science: Computational and cognitive neuroscientific approaches for understanding
the tender care,” in Workshop on Elderly Care Robotics - Technology
and Ethics (WELCARO) in ICRA2018, pp. –, 2018.
[4] M. Ito and M. Honda, “An examination of the influence of humanitude
caregiving on the behavior of older adults with dementia in japan,” inProceedings of the 8th International Association of Gerontology andGeriatrics European Region Congresss, 2015.
[5] M. Honda, M. Ito, S. Ishikawa, Y. Takebayashi, and L. T. Jr.,
“Reduction of behavioral psychological symptoms of dementia bymultimodal comprehensive care for vulnerable geriatric patients in anacute care hospital: A case series,” Case reports in medicine, vol. 2016,
p. 4813196, 2016.
[6] PPS, “Fingertps.””https://pressureprofile.com/
fingertps”.
[7] Vicon, “Motion capture system.” ”https://www.vicon.com/”.
[8] Nitta, “Bpms system.””https://www.nitta.co.jp/
product/sensor/bpms/”.
[9] novel gmbh, “pliance sensors.””http://novel.de/novelcontent/sensors”.
[10] Y. Tajitsu, “Smart piezoelectric fabric and its application to control ofhumanoid robot,” Ferroelectrics, vol. 499, no. 1, pp. 36–46, 2016.
[11] H. S. Han, J. Park, T. D. Nguyen, U. Kim, C. T. Nguyen, H. Phung,and H. R. Choi, “A highly sensitive dual mode tactile and proximity
sensor using carbon microcoils for robotic applications,” in 2016 IEEE
International Conference on Robotics and Automation (ICRA), pp. 97–
102, IEEE, 2016.
[12] G. Ponraj and H. Ren, “Estimation of object orientation using conductive ink and fabric based multilayered tactile sensor,” in 2018 IEEE
International Conference on Robotics and Automation (ICRA), pp. 1–
7, IEEE, 2018.
[13] J. O’Neill, J. Lu, R. Dockter, and T. Kowalewski, “Practical, stretchable smart skin sensors for contact-aware robots in safe and collaborative interactions,” in 2015 IEEE International Conference on Robotics
and Automation (ICRA), pp. 624–629, IEEE, 2015.
[14] The MakeHuman team., “MakeHuman community.” http://www.
makehumancommunity.org/.
[15] PTC Inc., “Vuforia Developer Portal.” https://developer.
vuforia.com/.
[16] Microsoft Corporation, “Microsoft HoloLens.” https://www.
microsoft.com/en-us/hololens.