sensors
ArticleToward Capturing Scientific Evidence in Elderly Care: Efficient
Extraction of Changing Facial Feature Points †Kosuke Hirayama 1 , Sinan Chen 1, * , Sachio Saiki 2 and Masahide Nakamura 1,3
1

2

3

*
†



Citation: Hirayama, K.; Chen, S.;
Saiki, S.; Nakamura, M. Toward
Capturing Scientific Evidence inElderly Care: Efficient Extraction of
Changing Facial Feature Points.
Sensors 2021, 21, 6726. https://doi.org/10.3390/s21206726

Graduate School of System Informatics, Kobe University, 1-1 Rokkodai-cho, Nada, Kobe 657-8501, Japan;
hirayama@ws.cs.kobe-u.ac.jp (K.H.); masa-n@cs.kobe-u.ac.jp (M.N.)
Department of Data & Innovation, Kochi University of Technology, 185 Miyanigutu, Tosayamada-cho,Kami-shi 782-8502, Japan; saiki.sachio@kochi-tech.ac.jp
RIKEN Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan
Correspondence: chensinan@ws.cs.kobe-u.ac.jp; Tel.: +81-78-803-6295
This paper is an extended version of the conference paper: Hirayama, K.; Saiki, S.; Nakamura, M. Capturing
User-Defined Facial Features for Scientific Evidence of Elderly Care. In Proceedings of the PerFlow 2020,
Online, 23 March 2020.

Abstract: To capture scientific evidence in elderly care, a user-defined facial expression sensing
service was proposed in our previous study. Since the time-series data of feature values have been
growing at a high rate as the measurement time increases, it may be difficult to find points of interest,
especially for detecting changes from the elderly facial expression, such as many elderly people canonly be shown in a micro facial expression due to facial wrinkles and aging. The purpose of thispaper is to implement a method to efficiently find points of interest (PoI) from the facial feature
time-series data of the elderly. In the proposed method, the concept of changing point detection
into the analysis of feature values is incorporated by us, to automatically detect big fluctuations orchanges in the trend in feature values and detect the moment when the subject’s facial expression
changed significantly. Our key idea is to introduce the novel concept of composite feature value toachieve higher accuracy and apply change-point detection to it as well as to single feature values.
Furthermore, the PoI finding results from the facial feature time-series data of young volunteers andthe elderly are analyzed and evaluated. By the experiments, it is found that the proposed method isable to capture the moment of large facial movements even for people with micro facial expressions
and obtain information that can be used as a clue to investigate their response to care.

Academic Editor: Alicja
Wieczorkowska

Keywords: scientific long-term care; facial expression; points of interest; changing point detection

Received: 30 August 2021
Accepted: 5 October 2021
Published: 10 October 2021

Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims inpublished maps and institutional affiliations.

Copyright: © 2020 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms andconditions of the Creative Commons
Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

1. Introduction
Japan is facing a super-aged society [1]. The number of elderly people who need careis increasing, which leads to a chronic lack of care resources. Under this circumstance, theJapanese Government has declared the practice of scientific long-term care as a national
policy [2,3]. It aims at optimal use of care resources by corroborating the effect of care with
scientific evidence. To achieve scientific long-term care, it is essential to evaluate the effectof care objectively and quantitatively. As an objective metric to quantify the care effect, we
focus on facial expressions [4–6]. The facial expressions give useful hints to interpret theemotions of a person under care. As a practical example, there is a systematic methodology
called facial expression analysis [7]. To analyze one’s face and emotions automatically,
emerging image processing technologies can be used. Especially, in recent years, owing
to powerful machine learning technologies, various companies have provided cognitive
application programing interfaces (API) for face [8,9], which automatically recognize andanalyze faces within a given picture. These APIs produce values of typical emotions suchas happiness and sadness, which would be helpful for the evaluation of the care effect. In
actual medical and care scenes, subtle changes of a face that may not be covered by the

Sensors 2021, 21, 6726. https://doi.org/10.3390/s21206726

https://www.mdpi.com/journal/sensors

Sensors 2021, 21, 6726

2 of 20

existing APIs have to be detected. This is because the facial expressions of elderly people
under care tend to be weakened due to functional and/or cognitive impairment. Moreover,the emotion analysis conducted by the existing APIs is a black box [10]. Therefore, it isdifficult for caregivers to understand why and how the value of the emotion is measured.
In addition, analysis of subtle facial changes and explainable emotion analysis become
possible, and the user’s expertise in emotion analysis also can be improved.
The main contribution of this paper is to implement a method to efficiently findpoints of interest from the facial feature time-series data of the elderly. In the existing
approach with the facial expression sensing service, since the time-series data of feature
values will greatly increase as the measurement time increases, a method to efficiently findpoints of interest in changes of the facial expressions is desperately needed, especially forevaluating elderly people who can only be shown in a micro facial expression due to facial
wrinkles and aging. Analyzing the data obtained, it is possible to analyze them from two
perspectives: the macro perspective of how the subject’s emotions changed throughout
the care and the micro perspective of when the subject’s emotions changed during care.
From the macro perspective, how the care satisfied the subject and how the care suitable
for the subject can be considered, and from the micro perspective, it is possible to infer
to what kind of conversations or contents the subject responded positively. This kind ofanalysis is useful for evaluating the effectiveness of care and for providing optimal careaccording to the preferences of each subject. However, since the facial expression sensing
service measures feature values on a second-by-second basis, the time-series data obtained
can be enormous, and it may be difficult to find a specific Point of Interest (PoI) [11–13],
especially when analyzing them from a micro perspective.
The previous version of this paper was published as a conference paper [14]. The
most significant change made to this version is the addition of a method for extracting thepoints of interest from the facial feature time-series data more efficiently. In the proposed
method, the moment is regarded as PoIs by us when the subject’s expression changed
significantly. To find them, the concept of Changing Point Detection is adopted [15,16] andchanging points automatically detected that indicate the moments at which the feature
values fluctuated significantly or the tread of the values changed and find the moments
at which the facial expressions of the subject have changed significantly. This makes it
possible to easily find PoIs, from the time-series data and provide data that contribute to
the analysis from the micro perspective described earlier. Based on the proposed method,
test data for evaluation created from the subject’s face and the data measured from thevideo of subjects receiving care are analyzed using Change Finder (i.e., a changing point
detection method [17]). Moreover, the effect of the changing point detection method is
discussed by comparing it with the actual behavior. In addition, by creating a composite
feature value [18], which is a combination of more than two feature values instead of single
feature values, and applying a changing point detection to it as well, we attempted toobtain information that is closer to fluctuations of actual facial expressions.
The remainder of this paper is organized as follows. Section 2 describes the preliminary
and previous study of developing “Face Emotion Tracker (FET)” with a cognitive API.
Section 3 introduces the related work of Changing Point Detection and Change Finderfrom recent years. Section 4 produces a complete description of the proposed method. The
preliminary experiment and actual experiment are respectively presented in Section 5 andSection 6, followed by conclusions in Section 7.
2. Preliminary and Previous Study
As described in Section 1, to evaluate the effect of care quantitatively and objectively
is important for scientific long-term care. However, the evaluation methods of care tend
to depend on a subjective scale, such as observations and questionnaires. Therefore, it
is difficult to use the obtained data as scientific evidence. Moreover, these assessment
methods place a heavy burden on care evaluators, care recipients, or both. In addition,
since scientific care is premised on using large-scale data, efficient data collection is required.

Sensors 2021, 21, 6726

3 of 20In the past, to measure care effects objectively, an experiment that measured the changes in
the facial expressions of people undergoing care was conducted [19]. In this experiment,
facial features such as the height of eyebrows and the opening of eyes were measured fromimages based on the method called “facial expression analysis” of P. Ekman et al. [7] andso on. However, they measured the features from the recorded video manually. Moreover,in the report, they concluded that “. . . Currently, we have not generalized this study
successfully because of the limited number of cases. . . . To establish an objective evaluation
method, we have to collect and analyze both the data of the objective people and the
caregiver more”. From the above, to practice scientific long-term care, it is important
to develop a computer-assisted service that enables us to automatically collect data that
contribute to the quantitative assessment of care.
With the rapid technological growth of cloud computing, cloud services to process
various tasks which were previously executed on edge sides can be used [20]. Under such akind of background, various services utilizing artificial intelligence, which has become very
popular in recent years, are currently available as a cloud service. One of the actual casesof cloud service using artificial intelligence is cognitive services. Cognitive services extract
useful information from visual [21], sound [22], linguistic [23], and knowledge data [24]by analyzing them and enable computers to recognize these types of information. Using
cognitive services, functions about cognition can be performed which were previously
difficult for computers. Many public cloud services provide these kinds of services as“cognitive API” [25]. For example, cognitive APIs for vision can extract various informationfrom an input image [26–28]. Cognitive APIs can automatically quantify emotions, whichscientific long-term care might utilize.
In the past, our research group developed a real-time emotion analysis service “Face
Emotion Tracker (FET)” using a cognitive API and attempted a quantitative evaluation ofcare [29]. In this previous study, care using a virtual agent [30] was conducted for 5 elderly
people living in an elderly facility. During care, FET captures pictures of a subject andsends them to a cognitive API (Microsoft Face API [31]) to analyze the emotion. A part of
the result of the FET experiment is shown in Figure 1 as time-series graphs (some outliers
has been eliminated). FET outputs 8 kinds of emotions (“anger”, “contempt”, “disgust”,“fear”, “happiness”, “neutral”, “sadness”, “surprise”) from 0 to 1 as probabilities, thistime, paying attention to the values of “neutral” and “happiness”. Subject A was relatively
expressive, and the values of “neutral” and “happiness” appear alternately during care.
Subject B was 99 years old, and the change in facial expression was not so apparent becauseof the decline of facial muscles. As a result, in contrast to subject A, only neutral wasobserved at almost all timings. However, the accompanying person who observed thesubjects evaluated that subject B felt better and better as the care progressed. Therefore, it
can be said that there was a gap between the obtained results and the actual ones. Thus, thereliability of the result of emotion analysis with cognitive API may be degraded dependingon the characteristics of the subject.
When observing the facial expressions of care targets, many cases of people with poor
facial expressions for their actual emotions will be encountered. The factors involved are
aging, dementia [32], the sequela of stroke [33], Amyotrophic Lateral Sclerosis (ALS) [34],Parkinson’s disease [35], and so on. Hence, taking some measures will be needed. In
addition, the cognitive API which FET used is built with machine learning technology.
Therefore, the process in which a result is outputted from an input cannot be followed. As aresult, we may face difficulty when using the obtained data as evidence for the effect of care.
To capture scientific evidence in elderly care, facial expressions are regarded as an objective
indicator of the effects of care, which led to proposing a user-defined facial expression
sensing service in the previous study [14]. The facial expression sensing service extracts
the coordinates of characteristic parts of the face (i.e., feature points) such as eyebrows,
eyes, and mouth from the face image. Then, the service measures the length connecting
two feature points as a feature value and records it as time-series data. By tracking thechange of these values, the degree of facial expression change from facial movement can beSensors 2021, 21, 6726

4 of 20

measured. As a result, even when the person shows only a subtle facial expression change,
the service will be able to measure and record it as opposed to overlooking it. The aim of
this paper is to contribute to explicable emotion analysis and quantitative evaluation ofcare by examining the degree of facial expression change as clear numerical values andinferring the emotional changes associated with the movement of facial expression.

Subject A - FET Result

1
0.8
0.6
0.4
0.2
0
00:00

02:00

04:00

06:00

08:00

10:00

12:00

14:00

16:00
neutral

18:00
happiness

Subject B - FET Result

1
0.8
0.6
0.4
0.2
0
00:00

02:00

04:00

06:00

08:00

10:00

12:00

14:00

16:00

18:00
20:00
neutral

22:00
happiness

Figure 1. A part of the result of the FET experiment.

3. Related Work
Since the Facial Expression Sensing Service measures feature values on a second-bysecond basis, the time-series data obtained can be enormous, and it may be difficult to find
PoI, especially when analyzing from a micro perspective. Changing point detection is thetechnique for detecting changing points in time-series data. This falls into two categories,
offline detection and online detection. Offline detection uses a batch process for data that havebeen accumulated and find changing points. On the other hand, online detection judges if
the data are changing point each time new data are presented. An an example of an offline
detection method is a method based on statistical tests by Guralnik and Srivastava [36]. In
this method, the point of change in the time-series data is analyzed assuming that all thetime-series data to be analyzed are available. In this process, a time-series model has to be
applied, such as an autoregressive model or a polynomial regression model, to the previous
and next data for all data points, which causes a problem that requires a large amount ofcomputation. In this study, Change Finder [17,37,38] (i.e., one of the online change-point
detection methods) is adopted. Since the amount of calculation of Change Finder is linear
to the number of data to be used, Change Finder can detect changing points quickly [39].
Change Finder is a changing point detection method characterized by an auto regression model (AR model) and two phases of learning using smoothing. Change Finder
has a mechanism for detecting changes of the model that the time-series data follow andcalculates the degree of change with a changing score. The changing score will be highwhen the degree of change is high. Moreover, the changing score will be low when the
degree of change is low. Change Finder has two AR models. The first model learns originaltime-series data. The second model learns the degree of changes, which is calculated based

Sensors 2021, 21, 6726

5 of 20on the first model as time-series data. This leads to removing changes raised by small
noises. Moreover, using a discounting learning algorithm called Sequentially Discounting AR model learning (SDAR) algorithm to train AR models, Change Finder achieves
processing speed as fast as Change Finder can process online. Moreover, whereas AR
models require stationarity of data, Change Finder can manipulate non-stationary data byadopting discounting learning. Change Finder processes the one-dimensional time-series
data obtained by Facial Expression Sensing Service as follows:
Step 1: Change Finder trains the first AR model on the input data. In this step, using
SDAR algorithm, Change Finder updates the mean, covariance, variance, andautoregression coefficients, which the AR model has. The equations of update are
given as follows:µt = (1 − r )µt−1 + rxt
Ct,i = (1 − r )Ct−1,i + r ( xt − µt )( xt−i − µt )(1)(i = 0, . . ., k)(2)k∑ at,j Ct,i− jCt,i =(i = 1, . . ., k)

(3)

j =1

Here, the order of the AR model is k, input data for times t(= 1, 2, · · · ) are xt ; the average
to be calculated this time is µt ; the auto regression coefficient is at,i ; the covariance is Ct,i ;the variance is σt , and the discounting rate is r (0 < r < 1).
Equation (3) is called the Yule–Walker equation. In this regard, let Ct,−i = Ct,i . By
solving this equation, at,j is obtained, and then, the following is calculated:
k

x̂t =

∑ at,i (xt−i − µt ) + µt

(4)

i =1

σt = (1 − r )σt−1 + r ( xt − x̂t )( xt − x̂t )(5)

Step 2: In this step, Change Finder calculates the score using normal probability density
distribution based on average and variance that results from Step 1. Let theprobability density be pt ( xt ), then pt ( xt ) and the score yt are calculated as follows.
pt ( xt ) = √



1
1
exp −
( xt − x̂t )2
2σt
2πσt

yt = − log pt−1 ( xt )

(6)
(7)

Step 3: In this step, Change Finder smoothes scores that result from Step 2 to remove
noises.
∑iw=−01 yt−iwHere, the width of smoothing is w, and the result of smoothing is Scoret .
Scoret =(8)

Step 4: Using Scoret that results from Step 3 as new time-series data, Change Finder
performs the second round of learning, calculating the score and smoothing just as
in Step 1–Step 3. Then, scores that result from the second round of Step 3 as thechanging score are obtained.
4. Methodology
4.1. Purpose and Approach
In this study, the analysis focuses on the micro-perspective described in Section 1 andaims to efficiently investigate when the subject reacted during care. For this purpose, we
consider extracting PoI, as the moments when large fluctuations are recognized in thetime-series data of feature values obtained by Facial Expression Sensing Service.

Sensors 2021, 21, 6726

6 of 20

A simple way of extracting PoIs is to look for points whose feature values exceed
a certain threshold. However, this method considers the absolute values of the feature
values, even if a subject’s expression has changed significantly, for example, when a neutral
face becomes a smiley face, those facial changes may be discarded if the change in feature
values is small. Moreover, since feature values are based on facial movements, the values
are always fluctuating little by little. Therefore, depending on the threshold setting, a large
number of points that exceed the threshold may be extracted, and thus the efficiency ofanalysis will not be improved. Considering these factors, it cannot be said that this method
can capture facial expression changes based on reality, and extracted points will not be sovaluable as PoIs. Therefore, in this study, the change detection algorithm called Change
Finder described in Section 3 and attempts to extract PoIs based on accurately capturing
the change in the feature values are adopted.
The key idea of this study is to introduce the concept of composite feature value andapply change-point detection to it as well as to single feature values. Composite feature
value is a scale created from multiple feature values according to a certain purpose of
the analysis. In this way, analysis can be performed more efficiently than considering thechanges in each feature value. Furthermore, since multiple facial parts move when facial
expression changes, the composite feature value that combines multiple feature values will
enable the extraction of PoIs with high validity that reflect the actual facial movements.
The specific method is as follows.
Step 1: Create a composite feature value X (t) = f ( x1 (t), x2 (t), · · · ) by combining multiple features x1 (t), x2 (t), · · ·Step 2: Input X (t) into Change Finder and obtain the change score S(t).
Step 3: Set a threshold γ, and regard a point t0 with γ < S(t0 ) as a candidate of PoI.
4.2. Composite Feature ValueIn this study, from the perspective of measuring the effectiveness of care, we consider
defining “Smile scale” as a composite feature that indicates how much the subject enjoys
the care. In creating Smile scale, facial movements suggesting feelings of enjoyment andhappiness are considered. Based on Ekman et al.’s book on facial expression analysis [7]and the attempt to observe the features of faces showing various emotions taken withthe cooperation of volunteers [40], the following features of facial expressions showing
happiness are summarized. From this description, this study focuses on 4 of the 6 feature
values used in the experiments of preliminary study (see Figure 2) as the smile scale:“Mouth width” h9 , “Eyebrow height” a1 , “Eyes opening” a3 , and “Mouth corner lowering”
a4 . Out of the four features that we focus on, when compared with the features of facial
expressions described earlier, the three features except for the “Mouth width” h9 show
more happiness when the value is smaller. Moreover, the absolute amount of variationdiffers for each feature value. For example, the movement around the mouth should belarger than the movement of eyes opening and closing. Therefore, it is necessary to assign
certain weights to each feature.
••••

Eyebrows will be lowered in whole;Eyes narrowed;
Mouth corner will be pulled back and raised;
Cheeks will be raised.

To determine the weights, we first measured the feature values of neutral and smilingfaces, using the same face images of this thesis author as in the preliminary experiment.
Next, we randomly selected 10 data in each state and calculated the average of the feature
values and differences between the two states. The samples of the face images used in
the calculations are shown in Figure 3 for the neutral state and Figure 4 for the smilingstate. The results are shown in Table 1. The “Ratio of difference” shows the ratio of the“Difference” of each feature value compared to the “mouth width” h9 . The results showthat the absolute amount of change in “Eyebrow height” a1 and “Eyes opening” a3 is small,Sensors 2021, 21, 6726

7 of 20

about half of that of “Mouth width” h9 and “Mouth corner lowering”. a4 . Therefore, Smile
scale X is determined as follows.
X = 2h9 − a1 − a3 − 2a4(9)Considering that the degree of facial movement varies among individuals, using Ratioof change in Table 1 directly as a coefficient was avoided. Note that the standardized
single feature values to calculate X was used. Specifically, for the original feature value
x (t), which has the mean µ x and standard deviation σx , z(t) is calculated by the following
formula and is used as the feature value:z(t) =

3
1

x (t) − µ x
σx

4
10

2

5

(1)
6

7

8

(2)
(3)
(4)
(5)
(6)

1+ 2
= ─ : Eyebrows height2
3+ 4
2 = ─ : Eyebrows lifting
2
5+ 6
3 = ─ : Eyes opening
2
7+ 8
4 = ─ : Mouth corner lowering2
9 : Mouth width
10 : Eyebrows distance1

9

Figure 2. Feature values for the experiment.

Figure 3. Samples of neutral face images to consider smile scale.

Figure 4. Samples of smiling face images to consider smile scale.

(10)Sensors 2021, 21, 6726

8 of 20

Table 1. Average of 10 feature values for the neutral and the smiling faces of the author.

Mouth
Width h9

Eyebrows
Height a1

Eyes
Opening a3

Mouth Corner
Lowering a4

Average of neutral
Average of smiling

0.81990.89140.42700.39810.09010.0503

1.0431
0.9817

Difference
Ratio of difference

0.07151.0000−0.0289
0.4035−0.0398
0.5561−0.06140.85784.3. Experiment Detail
We attempted to extract PoIs for single feature values and composite features value
Smile scale, using Change Finder on the test data created from the face of this thesis’author and data obtained from facial images of elderly people receiving care. In thisexperiment, we use the Change Finder library for Python [41]. The library takes thefollowing parameters: order of the AR model order, discounting rate r, and width ofsmoothing size w. These parameters are used directly for the first AR model training. In
the second AR model training, the width of the smoothing size will be round(w/2.0). In
each experiment, the parameters are set to order = 1, r = 0.025, and w = 5. Among thechange scores S(t) calculated by the Change Finder, as shown in Figure 5, we define thepoint t with the highest score as the time between the score exceeding a certain thresholdand it falling below the threshold as the PoI candidate. By examining the appearance of theface around PoI candidates, we can confirm whether an obtained PoI candidate is a valid
as PoI. In this series of experiments, we determine the threshold γ by γ = µS + σS using
the mean µS and standard deviation σS of the change score S.
Change score

PoI candidate

Thresholdγ

Time

Figure 5. Definition of PoI candidate.

5. Preliminary Experiment5.1. Outline
As a preliminary experiment, we conducted a PoI extraction experiment using thefeature value data obtained from the face image of the author of this paper was conducted.
The test data based on a video in which the author repeated a neutral and a smile expression
several times was created. The video was taken by using the built-in camera of a laptop
computer while the author was looking at the screen. We measured the feature values
at intervals of 1 s and removed obvious outliers. Missing data were filled with previous
data. Then Change Finder was applied to the single features and Smile scale of this data tocalculate change score and extract PoI candidates.
In this experiment, the ratios n p /n and no /n were first calculated to estimate howmuch more efficient the facial expression analysis becomes. Here, n is the number of whole
data; n p is the number of PoIs, and no is the number of data exceeding the threshold γ. The
smaller these values are, the fewer the number of data to be checked, and the more efficient
the analysis will be. Next, to confirm the validity of the obtained PoI candidates, the timingof facial expression changes recorded by eye observation was compared with them. First,
we looked for the timing, as shown in Figure 6, when the face (a) becomes happy looking

Sensors 2021, 21, 6726

9 of 20and (b) changes to not happy looking by visual inspection, and recorded them as “event
point”. In this experiment, event points were recorded only when obvious facial movement
was seen. For example, gradually returning from a smiling face to a neutral face over along period, about 30 s to 1 min was not recorded because it was difficult to determine
where to place an event point. Next, PoI candidates were picked whose timing matched an
event point, and we defined it as a correct PoI and defined its number as a number of correct.
In this experiment, if a PoI candidate was within 10 s before or after an event point, it wasconsidered that they are matched. Note that a number of correct will be counted only once
for each event point. Then, for the number of PoIs n p , the number of correct nc , and thenumber of event points ne , the following ratios were checked.
••

Recall: Ratio of event points that could be captured nc /ne ;Precision: Ratio of valid PoI candidates nc /n p .

The higher the Recall, the less the change in facial expression is missed. Moreover,the higher the Precision, the more accurate the extraction of PoI is with less noise. The
data at the beginning of the change score are not useful because the training of the SDAR
algorithm has just started. Therefore, the data in the first 30 s from the calculation of thethreshold γ and each ratio were excluded.

Figure 6. Example of event points (a) the face becomes happy looking, (b) the faces changes to not happy looking. (“Elapsedtime” corresponds to change score graphs in Figure 7).

5.2. Results
Table 2 shows the calculation results of each ratio. Figure 7 shows the change score
for Smile scale. The red line is the change score, the light blue line is the raw value ofeach feature value, and the black dashed line is the threshold γ. Red dots indicate PoI
candidates. Among them, dots with double circles indicate correct PoIs. The coloredvertical lines indicate the event points, where yellow lines indicate when the subject’s facebecame happy looking and light blue lines dashed line indicates when the subject’s facebecame not happy looking. The left axis indicates the value of the change score, the rightaxis indicates the feature value, and the horizontal axis shows the elapsed time. n p /n
was less than 0.02 in all cases, and no /n was about 0.1 at the highest. In other words, ourmethod was able to narrow down the data to be searched to less than one-tenth. When
Change Finder was applied to the smile scale, the number of correct was the highest and
Recall was also the highest. Looking at the graphs, PoI candidates near the event points ataround 2 min, 3 min, and 15 min and 30 s were obtained only from the smile scale, andSensors 2021, 21, 6726

10 of 20these were not obtained for any single feature value. In addition, for the case of Smile scale,the number of PoI candidates not around event points was the lowest, so the Precision
was the highest. From these results, it can be said that PoI candidates of composite feature
value have a higher probability to catch important moments than PoI candidates of single
feature values.
Table 2. Various parameters and ratios measured (test data: number of whole data n = 1114, number of event points ne = 12).

Feature Type

SingleComposite

Feature Name

Mouth Width h9

Eyebrows Height a1

Eyes Opening a3

Mouth Corner Lowering a4

Smile Scale

np
no

16
55

12
80

16
75

13
113

18
78

n p /nno /n0.01440.04940.01080.07180.01440.06730.01170.10140.01620.0700

nc

4

3

4

6

11

RecallPrecision

0.33330.25000.25000.25000.33330.25000.50000.46150.91670.6111Figure 7. The graph showing the change score for Smile scale (test data).

In the graphs of each feature value, there are PoI candidates in the 6 to 9 min where
there are no event points. Although at these points the author does not show smiles, subtle
facial movements were observed such as widening eyes and holding the mouth. Change
Finder seemed to react to these facial movements. As long as the data are based on detailed
facial movement tracking, this kind of noise will occur more or less. However, when using
Smile scale, the subtle variation in each feature value is suppressed a little. Therefore, it
is considered that Change Finder extracts few unimportant points compared to the caseusing single feature values, and as a result, the Precision is increased.
As a comparison with the method of this experiment, the PoI candidates were extracted
and each scale calculated using the raw values of the Smile scale. As in the case in which
the change score was used, the timing with the highest value exceeding a certain threshold
was considered as a PoI candidate. We set the threshold value to µ + nσ using the mean µand standard deviation σ of all Smile scale’s raw values. The results are shown in Table 3.
When we set the low threshold (n = 0, n = 0.5), the values of n p /n and no /n were muchhigher than those using the change score. Moreover, many PoI candidates were detected,
and although Recall was high, Precision was low. When we set the high threshold (n = 2),
the number of correct decreased and the precision was low because small changes wereignored. In the case of n = 1, although Recall was relatively high, many PoI candidates
were detected in the areas where Smile scale value wiggled, and the precision was lower
than that using the change score. From this, it can be said that using the change scoreis more efficient and accurate than analyzing the raw feature data while changing thethreshold value.

Sensors 2021, 21, 6726

11 of 20

Table 3. Various parameters and ratios of the experiment measured using the raw Smile scale value(test data: number of whole data n = 1144, number of event points ne = 12).

Thresholdµµ + 0.5σµ+σµ + 2σ

np
no

69
355

24
227

22
173

11
77

n p /nno /n0.06030.31030.02100.19840.01920.15120.0096
0.0673

nc

8

8

9

5

RecallPrecision

0.66670.11590.66670.33330.75000.40910.41670.4545

From this result, it was confirmed that applying Change Finder to feature value data
and using composite features are effective for the test data. In the next section, we willconduct the same analysis using actual images of elderly people during care and verify
whether the analysis method of this preliminary experiment can be applied to general data.
6. Actual Experiment6.1. Overview
In this experiment, we measure a feature value using images of five subjects during
care obtained in the experiment of FET [29] and attempt to extract PoIs on them using
Change Finder as in the preliminary experiment. Then, we examine whether the analysis
can be more efficient and whether the obtained PoI candidates are valid. We divided
expressive subjects (subjects A, C, and D) and subjects with poor facial expression (subjects
B and E) into subject group 1 and subject group 2, respectively, and discussed each group.
Since the face images of each subject were taken at intervals of about 3 s, the obtained feature
value had similar intervals. To adjust the experimental conditions with the preliminary
experiment, in advance, we filled the feature value data with their previous data, to place
the data at intervals of 1 s. Moreover, we removed obvious outliers. As in the preliminary
experiment, we excluded the data in the first 30 s from the calculation of the threshold γ
and from each ratio.
6.2. Results
Here, we explain the results of the analysis for subject group 1. In addition to calculating the change score of Change Finder, for Subject Group 1, since we could examine facial
expression changes visually, as in the preliminary experiment, we recorded event points
and calculated Recall and Precision. Tables 4–6 show the calculation results of each ratio
for each subject. Figures 8–10 are the graphs showing the change scores of the Smile scale.
The legends of each graph are the same as that of the preliminary experiment. For eachsubject, no /n was about 0.08 at the highest for any of the feature values. Considering thatthe data to be investigated can be reduced to less than 8% of the total data, it can be said
that the analysis was efficient. In this experiment, the original data were heavily padded,
so n p /n is not very helpful, but the values were all less than 0.02.
For subjects C and D, the highest values of Recall and Precision were obtained whenusing the Smile scale. Moreover, for both subjects, there were event points that could be
detected only when using the Smile scale. From these results, it can be said that using thecomposite feature value was useful for subjects C and D, as the preliminary experiment. In
both subjects, Recall was 0.5, which means that half of the event points were detected as
PoI candidates. The graph of the Smile scale shows that although the value rises and falls
around the event points, not all of them were extracted as PoI candidates. In other words,
it can be considered that Change Finder does not simply react to all local changes of feature
value. However, considering that the data to be investigated was greatly narrowed down
and that the extracted PoI candidates corresponded to the event points to some extent,

Sensors 2021, 21, 6726

12 of 20it can be said that although there were some missing event points, Change Finder wasable to efficiently scoop up the appropriate timing as PoI from the constantly changing
feature data. As for the subject A, compared to Smile scale, “Mouth width” has a higher
value in both Recall and Precision, and “Mouth corner lowering” has a higher value in
Recall. Looking at the graph of Smile scale, compared to the large changes in feature values
around 4 min, 10 min 30 s, and 14 min, the change score did not react to the small valuechanges between 6 min 30 s and 7 min, and corresponding event points were overlooked.
On the other hand, they were detected when using “mouth width”. Although the resultsof single feature values are more likely to catch unimportant points as PoI candidates thancomposite feature values, they can be considered to be effective in complementing theresults of composite feature value depending on the characteristics of the input data.
Table 4. Various parameters and ratios measured (Subject A: number of whole data n = 1129, number of event points
ne = 11).

Feature Type

SingleComposite

Feature Name

Mouth Width h9

Eyebrows Height a1

Eyes Opening a3

Mouth Corner Lowering a4

Smile Scale

np
no

13
61

17
76

19
86

20
93

12
62

n p /nno /n0.01150.05400.0151
0.06730.01680.07620.01770.08240.0106
0.0549

nc

6

3

3

7

5

RecallPrecision

0.54550.46150.27270.17650.27270.15790.63640.35000.45450.4167

Table 5. Various parameters and ratios measured (Subject C: number of whole data n = 1075, number of event points
ne = 16).

Feature Type

SingleComposite

Feature Name

Mouth Width h9

Eyebrows Height a1

Eyes Opening a3

Mouth Corner Lowering a4

Smile Scale

np
no

12
67

12
45

15
77

10
61

15
76

n p /nno /n0.01120.06230.01120.04190.01400.07160.00930.05670.01400.0707

nc

6

4

6

4

8

RecallPrecision

0.37500.50000.25000.33330.37500.40000.25000.40000.50000.5333

Table 6. Various parameters and ratios measured (Subject D: number of whole data n = 769, number of event points ne = 14).

Feature Type

SingleComposite

Feature Name

Mouth Width h9

Eyebrows Height a1

Eyes Opening a3

Mouth Corner Lowering a4

Smile Scale

np
no

10
42

8
38

7
36

6
61

12
54

n p /nno /n0.01300.05460.01040.04940.0091
0.04680.00780.07930.01560.0702

nc

5

2

2

3

7

RecallPrecision

0.35710.50000.14290.25000.14290.28570.21430.50000.50000.5833

Sensors 2021, 21, 6726

13 of 20

Figure 8. Graph of the change score for Smile scale (Subject A).

Figure 9. Graph of the change score for Smile scale (Subject C).

Figure 10. Graph of the change score for Smile scale (Subject D).

We describe the results of the analysis for subject group 2. For subject group 2, it wasdifficult to find event points visually, so we only calculated the change score of Change
Finder and discussed it, confirming the facial expression around detected PoI candidates
The calculation results of n p /n and no /n are shown in Tables 7 and 8 for each subject.
Figures 11 and 12 is the graphs showing the change scores of Smile scale. The legendsof the graphs are the same as those of the preliminary experiment and the experiment ofsubject group 1. For each subject, no /n was always less than 0.1. As in the case of subject
group 1, it can be said that this method was able to narrow down the data significantly.
Subject B had hardened facial muscles and no obvious change in facial expression comparedto the other subjects. Although each feature value continued to fluctuate, it was not possible
to identify the timing when the values rose or fell clearly or when the trend of the timeseries changed. Around the PoI candidate using single feature values, it was possible toconfirm the facial movements corresponding to each feature value, but there were no clear
signs that emotional changes can be inferred. Examining the faces near the PoI candidates
using the smile scale, we mainly observed mouth movements such as holding the mouthand opening the mouth. While there was no particular tendency of variation in every single
feature, the Smile scale might be strongly reflected by “Mouth width” and “Mouth corner
lowering”, which have a high coefficient in the Smile scale, and this made Change Finder
sensitive to these fluctuations. Subject E showed symptoms of euphoria and generally
appeared to be happy from the beginning to the end of the care. When we examined theSensors 2021, 21, 6726

14 of 20

faces near the PoI candidate extracted from the smile scale, we were able to capture themoments when the expressions moved significantly. In the first half to the middle part ofthe care (before 14 min), large smile scenes could be seen. In the latter part of the care, there
was a candidate point of interest at 18 min and 30 s, and from this time, facial movements
including smiling were not seen very much until about 22 min and 30 s. According to thecare record, the subject was listening to her favorite music for about 4 min from 18 min
and 20 s, and it can be assumed that her facial expression temporarily faded as a result oflistening to the music intently.
Table 7. Various parameters and ratios measured (Subject B: number of whole data n = 1303).

Feature Type

SingleComposite

Feature Name

Mouth Width h9

Eyebrows Height a1

Eyes Opening a3

Mouth Corner Lowering a4

Smile Scale

np
no

23
95

19
86

19
88

21
129

26
115

n p /nno /n0.01770.07290.0146
0.06600.0146
0.06750.0161
0.09900.02000.0883

Table 8. Various parameters and ratios measured (Subject E: number of whole data n = 1351).

Feature Type

SingleComposite

Feature Name

Mouth Width h9

Eyebrows Height a1

Eyes Opening a3

Mouth Corner Lowering a4

Smile Scale

np
no

14
72

16
70

20
100

14
71

17
82

n p /nno /n0.01040.05330.01180.05180.01480.07400.01040.05260.01260.0607

Smile scale
10

20

5
15

0

10

5
10

5

15
0
00:00 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00

Figure 11. Graph of the change score for the Smile scale (Subject B).

Smile score
10

25

5

20

0

15

5

10

10

5

15

0

00:00 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00

Figure 12. Graph of the change score for the Smile scale (Subject E).

6.3. Discussion
For subject group 1, we confirmed that the extracted PoI candidates matched thevisually confirmed event points, (i.e., the change in timing of facial expression, to someSensors 2021, 21, 6726

15 of 20extent). Although there were gaps in accuracy depending on the data used and someoverlooked event points, this method seems to be useful in that it can narrow down a hugeamount of time-series data and provide clues to efficiently find PoIs. In common withthe three subjects of subject group 1, closely examining the graph of the change score, we
found some small increase in the score around the event point, although it does not exceed
the threshold γ. If the threshold is lowered, such points will be detected, but the numberof PoI candidates will also increase, and the cost to examine appropriate PoI will increase.
Therefore, it will be necessary for users to determine the threshold value by considering
their own needs, the time and cost they can take for analysis, and so on. Regarding thecomposite feature value, looking at the results for the Smile scale of subject group 1, thefailure to detect event points when the face changes to not happy looking is more noticeable
than when the face becomes happy looking. For example, in the case of Subject D’s Smile
scale, the former was detected in 5 out of 7 event points (There is one event point of happy
looking on the right edge of the graph), while the latter was detected only in 2 out of 6 event
points. The Smile scale was created to estimate how happy subjects feel. However, since
the facial movements corresponding to emotions such as happiness, disgust, and surpriseare unique, it would be necessary to create a composite feature value for each emotion,
especially if we want to capture the “unpleasant look”. For subject group 2, we could not
successfully extract PoI candidates that showed a large change in facial expression fromthose whose facial movements were small. However, we could see some facial movements
like mouth movements around the PoI candidates, so even though it is difficult to estimate
emotions, it is possible to estimate whether the interaction was successful without losing
the subject’s interest by investigating the “magnitude of response to external stimulus”. On
the other hand, for the subject who seemed to be euphoric, the moments when she showeda big smile or became serious were successfully captured as PoI candidates.
We have conducted experiments based on the images of the subject during careobtained in the FET experiment. In this section, we attempt to extract the PoI candidates
using the original measurement results of the FET experiment and compare the resultswith those obtained by using the feature values obtained by the Facial Expression Sensing
Service. FET can quantify various kinds of emotions (see Section 2). In this experiment, we
focus on “happiness” from the viewpoint of estimating the effect of care. For the time-series
data of the “happiness” value, as in the previous experiments, the time-series data at 1-s
intervals and the extracted PoI candidate were interpolated using Change Finder with thesettings described in Section 6.1. We calculated n p /n and no /n for each subject. The results
are shown in Table 9. For subject group 1, we calculated Recall and Precision. The results
are shown in Table 10. Moreover, the graphs of results of subjects in each subject group arealso shown in Figures 13 and 14. The right vertical axis of each graph shows the value of“happiness”. Other than that, legends are the same as in the previous experiments.
Table 9. Values of n p /n and no /n from change score using FET “happiness” data.

Subject A
Subject B
Subject C
Subject D
Subject E

n

np

no

n p /n

no/n

1132
1303
1075
772
1351

9
18
15
10
20

54
84
70
53
90

0.00800.01380.01400.01300.01480.04770.06450.0651
0.06870.0725

Table 10. Recall and precision from change score using FET “happiness” data of Subject Group 1.

Subject A
Subject C
Subject D

ne

nc

np

RecallPrecision

11
16
14

6
4
5

9
15
10

0.54550.25000.35710.6667
0.26670.5000Sensors 2021, 21, 6726

16 of 20In this experiment, no /n was lower than 0.1 for all subjects, which means that wewere able to focus on the important data, significantly, the FET data as well as the datafrom the Facial Expression Sensing Service. In subject group 1, for subjects A and D, Recall
and Precision were higher than those with feature values. For subject C, the result was not
good. If the data tend to keep high or low values, as in the case of subject C, the subtle
changes will be lost and PoIs will not be detected well. For subject group 2, the change
score run along with the change in the value of “happiness”. The value of “happiness” isalmost always around 0 or 1, and it tends to change sharply at some moment. Therefore,such instantaneous value is mainly detected as a PoI candidate, and the detailed changes
in facial expressions at other moments are lost. Moreover, we compared the change scores
and PoI candidates for each subject in subject group 2 with those using the Smile scale. The
result are shown in Figure 15. From these graphs, it can be seen that there are some gaps in
the timing of the PoI candidates between the two cases. The gaps are especially seen in
subject B. This suggests that by using the data from the Facial Expression Sensing Service,
it is possible to perform different analyses of the case using FET data. From these results,
we conclude that the analysis based on the method of this experiment can be used for FET
data for expressive people. For the people with poor facial expression, we will be able toconduct care analysis based on subtle facial changes that cannot be captured by FET byusing the data of the Facial Expression Sensing Service.
happiness (Subject A)

happiness (Subject C)

happiness (Subject D)

Figure 13. Graphs of the change score for FET’s “happiness” value (Subject group 1).

Sensors 2021, 21, 6726

17 of 20

happiness (Subject B)

happiness (Subject E)

Figure 14. Graphs of the change score for FET’s “happiness” value (Subject group 2).

Smile scale

Smile scale

Figure 15. Comparison of change score between Smile scale and FET “happiness” value (Subject group 2).

7. Conclusions
In order to capture scientific evidence in elderly care, this paper specifically focusedon the changes in the facial expressions of the elderly. The main outcomes include thefollowing: (1) Proposing to apply the Changing point detection method using Change
Finder with the feature value data obtained from Facial Expression Sensing Service toSensors 2021, 21, 6726

18 of 20

efficiently find PoIs. (2) Conducting a complete experiment to analyze and evaluate theperformance of the proposed method whether on the facial expressions of young men or
elderly people. Through the experiments, it was found that the proposed method cannot
only efficiently find appropriate PoIs from a huge amount of time series data but also
extract the moments when the face moved remarkably. Meanwhile, for processing thespecial case with incomplete raw data to the maximum, improvement and verificationof the proposed method are required. As future work, more facial data from elderly
people will be analyzed and evaluated integrating the proposed method and care nursing
knowledge, in order to realize explainable scientific care for elderly people.
Author Contributions: Writing—original draft preparation, K.H.; writing—review and editing, S.C.
and S.S.; supervision, M.N.; validation, S.C. All authors have read and agreed to the publishedversion of the manuscript.
Funding: This research received no external funding.
Institutional Review Board Statement: This study was approved by the Research Ethics Committeeof Kobe University Graduate School of System Informatics (No. R01-02).
Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.
Acknowledgments: This research was partially supported by JSPS KAKENHI Grant Numbers
JP19H01138, JP18H03242, JP18H03342, JP19H04154, JP19K02973, JP20K11059, JP20H04014, JP20H05706
and Tateishi Science and Technology Foundation (C) (No.2207004).
Conflicts of Interest: The authors declare no conflict of interest.

References
1.
2.
3.
4.
5.
6.
7.
8.
9.

10.
11.

12.

13.
14.

Okamura, H. Mainstreaming Gender and Aging in the SDGs; At a Side Event to the High Level Political Forum; Ambassador and
Deputy Representative of Japan to the United Nations: New York, NY, USA, 2016; Volume 13.
For Japan’s Economic Revitalization (No.26), H. Future Investment Strategy (2017). Available online: http://www.kantei.go.jp/
jp/singi/keizaisaisei/dai26/siryou.pdf (accessed on 15 September 2021).
Tsutsui, T.; Muramatsu, N. Care-needs certification in the long-term care insurance system of Japan. J. Am. Geriatr. Soc. 2005,
53, 522–527. [CrossRef] [PubMed]
Ekman, P. Facial expressions. Handb. Cogn. Emot. 1999, 16, e320.
Ekman, P.; Keltner, D. Universal facial expressions of emotion. In Nonverbal Communication: Where Nature Meets Culture;
Segerstrale, U.P., Molnar, P., Eds.; Psychology Press: Hove, UK, 1997; Volume 27, p. 46.
Dimberg, U.; Thunberg, M.; Elmehed, K. Unconscious facial reactions to emotional facial expressions. Psychol. Sci. 2000, 11, 86–89.
[CrossRef] [PubMed]
Ekman, P.; Friesen, W.V. Unmasking the Face; Spectrum-Prentice Hall: Englewood Cliffs, NJ, USA, 1975.
Khan, S.; Akram, A.; Usman, N. Real Time Automatic Attendance System for Face Recognition Using Face API and OpenCV.
Wirel. Pers. Commun. 2020, 113, 469–480. [CrossRef]
Kim, H.; Kim, T.; Kim, P. Interest Recommendation System Based on Dwell Time Calculation Utilizing Azure Face API. In
Proceedings of the 2018 International Conference on Computer, Information and Telecommunication Systems (CITS), Colmar,
France, 11–13 July 2018; pp. 1–5.
Duclos, J.; Dorard, G.; Berthoz, S.; Curt, F.; Faucher, S.; Falissard, B.; Godart, N. Expressed emotion in anorexia nervosa: What is
inside the “black box”? Compr. Psychiatry 2014, 55, 71–79. [CrossRef] [PubMed]
Liu, B.; Fu, Y.; Yao, Z.; Xiong, H. Learning geographical preferences for point-of-interest recommendation. In Proceedings of the
19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, IL, USA, 11–14 August 2013;
pp. 1043–1051.
Yuan, Q.; Cong, G.; Ma, Z.; Sun, A.; Thalmann, N.M. Time-aware point-of-interest recommendation. In Proceedings of the 36th
International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland, 28 July–1 August
2013; pp. 363–372.
Gao, H.; Tang, J.; Hu, X.; Liu, H. Content-aware point of interest recommendation on location-based social networks. In
Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, USA, 25–30 January 2015.
Hirayama, K.; Saiki, S.; Nakamura, M.; Yasuda, K. Capturing User-defined Facial Features for Scientific Evidence of Elderly Care.
In Proceedings of the 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom
Workshops), Austin, TX, USA, 23–27 March 2020; pp. 1–6.

Sensors 2021, 21, 6726

15.
16.
17.
18.
19.
20.
21.

22.
23.
24.
25.

26.
27.
28.
29.

30.
31.
32.

33.

34.

35.

36.
37.

38.

39.

19 of 20

Lee, B.; Erdenee, E.; Jin, S.; Nam, M.Y.; Jung, Y.G.; Rhee, P.K. Multi-class multi-object tracking using changing point detection. In
Proceedings of the European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 2016; pp. 68–83.
Miranda, V.; Zhao, L. Topological Data Analysis for Time Series Changing Point Detection. In Proceedings of the International Conference
on Natural Computation, Fuzzy Systems and Knowledge Discovery; Springer: Berlin/Heidelberg, Germany, 2019; pp. 194–203.
Saranya, R.; Rajeshkumar, S. A Survey on Anomaly Detection for Discovering Emerging Topics. Int. J. Comput. Sci. Mob. Comput.
(IJCSMC) 2014, 3, 895–902.
Li, R.K.; Tu, Y.M.; Yang, T.H. Composite feature and variational design concepts in a feature-based design system. Int. J. Prod.
Res. 1993, 31, 1521–1540. [CrossRef]
Sato, K.; Sugino, K.; Hayashi, T. Development of an objective evaluation method for the nursing-care education based on facial
expression analysis. Med. Health Sci. Res. Bull. Tsukuba Int. Univ. 2010, 1, 163–170.
Moniz, A.; Gordon, M.; Bergum, I.; Chang, M.; Grant, G. Introducing Cognitive Services. In Beginning Azure Cognitive Services;
Springer: Berlin/Heidelberg, Germany, 2021; pp. 1–17.
Kasaei, S.; Sock, J.; Lopes, L.S.; Tomé, A.M.; Kim, T.K. Perceiving, learning, and recognizing 3d objects: An approach to cognitive
service robots. In Proceedings of the AAAI Conference on Artificial Intelligence, Hilton New Orleans Riverside, New Orleans,
LA, USA, 2–7 February 2018; Volume 32.
Mertes, C.; Yang, J.; Hammerschmidt, J.; Hermann, T. Challenges for Smart Environments in Bathroom Contexts-Video and Sound
Example; Bielefeld University: Bielefeld, Germany, 2016.
Ogiela, U.; Ogiela, L. Linguistic techniques for cryptographic data sharing algorithms. Concurr. Comput. Pract. Exp. 2018,30, e4275. [CrossRef]
Chen, M.; Herrera, F.; Hwang, K. Cognitive computing: Architecture, technologies and intelligent applications. IEEE Access 2018,6, 19774–19783. [CrossRef]
Chen, S.; Saiki, S.; Nakamura, M. Proposal of home context recognition method using feature values of cognitive API. In
Proceedings of the 2019 20th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking
and Parallel/Distributed Computing (SNPD), Toyama, Japan, 8–11 July 2019; pp. 533–538.
Chen, S.; Saiki, S.; Nakamura, M. Toward Affordable and Practical Home Context Recognition: Framework and Implementation
with Image-based Cognitive API. Int. J. Netw. Distrib. Comput. 2019, 8, 16–24. [CrossRef]
Chen, S.; Saiki, S.; Nakamura, M. Integrating Multiple Models Using Image-as-Documents Approach for Recognizing FineGrained Home Contexts. Sensors 2020, 20, 666. [CrossRef] [PubMed]
Chen, S.; Saiki, S.; Nakamura, M. Toward flexible and efficient home context sensing: Capability evaluation and verification ofimage-based cognitive APIs. Sensors 2020, 20, 1442. [CrossRef] [PubMed]
Sako, A.; Saiki, S.; Nakamura, M.; Yasuda, K. Developing Face Emotion Tracker for Quantitative Evaluation of Care Effects. In
Proceedings of the Digital Human Modeling 2018 (DHM 2018), Held as Part of HCI International 2018, Las Vegas, NV, USA,
15–20 July 2018; Volume LNCS 10917, pp. 513–526.
Tokunaga, S.; Tamamizu, K.; Saiki, S.; Nakamura, M.; Yasuda, K. VirtualCareGiver: Personalized Smart Elderly Care. Int. J. Softw.
Innov. (IJSI) 2016, 5, 30–43.
Azure, M. Face API-Facial Recognition Software | Microsoft Azure. Available online: https://azure.microsoft.com/en-us/
services/cognitive-services/face/ (accessed on 30 August 2021).
Asgarian, A.; Zhao, S.; Ashraf, A.B.; Browne, M.E.; Prkachin, K.M.; Mihailidis, A.; Hadjistavropoulos, T.; Taati, B. Limitations
and Biases in Facial Landmark Detection D an Empirical Study on Older Adults with Dementia. In Proceedings of the CVPR
Workshops, Long Beach, CA, USA, 15 June 2019; pp. 28–36.
Batista, A.X.; Bazán, P.R.; Conforto, A.B.; Martin, M.d.G.M.; Simon, S.; Hampstead, B.; Figueiredo, E.G.; Miotto, E.C. Effects ofmnemonic strategy training on brain activity and cognitive functioning of left-hemisphere ischemic stroke patients. Neural Plast.
2019, 2019, 1–16. [CrossRef] [PubMed]
Bandini, A.; Green, J.R.; Taati, B.; Orlandi, S.; Zinman, L.; Yunusova, Y. Automatic detection of amyotrophic lateral sclerosis (ALS)from video-based analysis of facial movements: Speech and non-speech tasks. In Proceedings of the 2018 13th IEEE International
Conference on Automatic Face & Gesture Recognition (FG 2018), Xi’an, China, 15–19 May 2018; pp. 150–157.
Bandini, A.; Orlandi, S.; Escalante, H.J.; Giovannelli, F.; Cincotta, M.; Reyes-Garcia, C.A.; Vanni, P.; Zaccara, G.; Manfredi, C.
Analysis of facial expressions in parkinson’s disease through video-based automatic methods. J. Neurosci. Methods 2017, 281, 7–20.
[CrossRef] [PubMed]
Guralnik, V.; Srivastava, J. Event detection from time series data. In Proceedings of the Fifth ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining, San Diego, CA, USA, 15–18 August 1999; pp. 33–42.
Yamanishi, K.; Takeuchi, J. A Unifying Framework for Detecting Outliers and Change Points from Non-Stationary Time Series
Data. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,
Edmonton, AB, Canada, 23–26 July 2002; pp. 676–681.
Yamanishi, K.; Takeuchi, J. Discovering Outlier Filtering Rules from Unlabeled Data: Combining a Supervised Learner with an
Unsupervised Learner. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery andData Mining, San Francisco, CA, USA, 26–29 August 2001; pp. 389–394.
Yamanishi, K. Anomaly Detection with Data Mining; Kyoritsu Shuppan Co., Ltd.: Tokyo, Japan, 2009.

Sensors 2021, 21, 6726

40.
41.

20 of 20

Ohta, T.; Tamura, M.; Arita, M.; Kiso, N.; Saeki, Y. Facial expression analysis: Comparison with results of Paul Ekman. J. Nurs.
Shiga Univ. Med Sci. 2005, 3, 20–24.
GitHub-Shunsukeaihara/Changefinder. Available online: https://github.com/shunsukeaihara/changefinder (accessed on 15
September 2021).