A concept for emotion recognition systems for children with profoundintellectual and multiple disabilities based on artificial intelligence
using physiological and motion signals

Hiroki Tanabea, Toshihiko Shiraishia*, Haruhiko Satob, Misato Niheic,
Takenobu Inoued, Chika KuwabaraeaGraduate School of Environment and Information Sciences, Yokohama National

University, Yokohama, Japan; bFaculty of Rehabilitation, Kansai Medical University,
Hirakata, Japan; cGraduate School of Frontier Sciences, The University of Tokyo,
Kashiwa, Japan; dDepartment of Assistive Technology, Research Institute of National
Rehabilitation Center for Persons with Disabilities, Tokorozawa, Japan; eCenter for
Developmental Disabilities of Yokosuka, Yokosuka, Japan

Abstract
Purpose: This study proposes a concept for emotion recognition systems forchildren with profound intellectual and multiple disabilities (PIMD) based on
artificial intelligence (AI) using physiological and motion signals.
Methods: First, the heartbeat interval (R–R interval, RRI) of a child with PIMD
was measured, and the correlation between the RRI and emotion was briefly
tested in a preliminary experiment. Then, a concept based on AI for emotion
recognition systems for children with PIMD was created using physiological andmotion signals, and an emotion recognition system based on the proposed
concept was developed using a random forest classifier taking as inputs the RRI,eye gaze, and other data acquired using low physical burden sensors.
Subsequently, the developed emotion recognition system was evaluated,
validating the proposed concept. Finally, we proposed a validated concept foremotion recognition systems.
Results: A correlation was found between the RRI and emotion. The emotion
recognition system was created based on the proposed concept and tested.
According to the results, the recognition rate of “negative” and “not negative” of
70.4% ± 6.1% (Mean ± S.D.) of the developed emotion recognition system washigher than 48.5% ± 5.0% of an unfamiliar person used as a control.
Conclusion: The results indicate that the proposed concept for the emotion
recognition systems is useful for communicating with children with PIMD.
Keywords: children with profound intellectual and multiple disabilities; emotion
recognition; heartbeat interval; artificial intelligence; physiological signal;
motion signal; communication assist system

Implications for rehabilitation•

A new concept based on artificial intelligence for emotion recognition systems
for children with profound intellectual and multiple disabilities (PIMD) using
physiological and motion signals is proposed.

•

An emotion recognition system based on the proposed concept developed usinga random forest classifier taking as inputs the heartbeat interval, eye gaze, andother data acquired using low physical burden sensors was tested in terms of theemotion recognition rate.

•

The recognition rate of “negative” and “not negative” of the developed system(i.e., 70.4% ± 6.1%) is higher than that of an unfamiliar person (i.e., 48.5% ±5.0%).

•The proposed concept for emotion recognition systems may be useful forcommunicating with children with PIMD.

Introduction
Profound intellectual and multiple disabilities (PIMD) is characterized by an individual
who is diagnosed with a low intelligence quotient and more than one disability [1]. InJapan, children aged 15 years old or below who have 25 points or higher on the “scoreof children with PIMD” and maintain the score for 6 months or longer are diagnosed
with PIMD [2]. Children with PIMD exhibit poor emotion expression, which is theprocess of sending out information about one’s emotional state, including emotion
signals in facial expressions, voice tone, and speech rate [3–6]. This condition makes
emotion recognition and nonverbal emotional communication difficult. Emotion
recognition is defined as the accurate recognition, decoding, or perception of received
nonverbal emotional information [4,7]. Emotion expression and recognition are apathway of nonverbal emotional communication [7].
A questionnaire study revealed that it is difficult for teachers to find movements
as the emotion expression of all the movements of children with PIMD and recognize
their emotional information because their face, eye, head, hand, arm, and leg
movements and those of their other body parts can be extremely weak or at times absent[8]. This difficulty brings embarrassment to teachers in terms of the way they teachchildren with PIMD [9,10]. However, these movements can be clues for recognizing
their emotional information, communicating with them, and teaching them. For example,
the reciprocation and the stopping time of the eye movements of a child with PIMD
differ according to the situation (e.g., medical treatment and daily life support) [11].
Accordingly, conventional studies on assistive devices used in children with
PIMD, which utilized motion signals, were conducted. One study developed a systemthat detects motion every second in a tablet placed in front of the face based on thechanges in the pixel values around the mouth and the eyes and alerts the user with abuzzer [12]. For other assistive devices, another study developed a system forrecognizing a user’s intention through eye gaze [13], brain signals in the motor cortex[14], voice signals [15], and tongue motion [16]. However, recognizing the user’semotion only by capturing the body motion is impossible, and it is only a subjective
judgment of the emotion by others.
Both motion and physiological signals are studied to understand the emotions ofchildren with PIMD. A heartbeat has the advantages of easy measurement compared toother physiological signals and low physical burden for children with PIMD [17]. It
may be helpful for emotion recognition because some emotions of able-bodied
individuals are related to the heart rate variability under certain conditions [18–20]. In a
study focusing on the response of two children with PIMD to a verbal stimulation, one
child exhibited a heartbeat response without a body movement response, while the other
did not show both responses; instead, the child exhibited a mouth movement response toanother smell stimulation [21]. Considering that the body movements of children with
PIMD are extremely weak, and their motion and physiological signals depend on
individuals, models using the data acquired from sensors for emotion recognition havebeen proposed only for able-bodied people [22,23]. The characteristics of thephysiological signals of children with cerebral palsy, many of whom are judged ashaving PIMD, are different from those of able-bodied people [24]. An emotion
recognition system should be tailored to each individual because the condition of eachchild with PIMD is unique, as is the part of the body that can be moved.
Artificial intelligence (AI) is defined as the science and engineering of making
intelligent machines, especially intelligent computer programs [25]. In recent years, AI
has been rapidly developed [26], exhibiting the potential to recognize the emotions ofchildren with PIMD from motion and physiological signals (e.g., heartbeat) through its
high classification performance. AI can also solve the dependency of the relationship of
motion and physiological signals to their emotions on each individual by its learning
ability.

In this study, we create a concept based on AI for emotion recognition systems
for children with PIMD using physiological and motion signals, develop and test an
emotion recognition system based on the proposed concept, and propose a validated
concept. The proposal of this concept is novel and significant as the first step of the
future works on the emotion recognition for and the nonverbal emotional
communication with children with PIMD because, to the best of our knowledge, noprevious study has described the feasible emotion recognition systems for them.
Methods
As part of an early-stage proposal of a concept for emotion recognition using
physiological and motion signals, we developed herein an emotion recognition systemprototype based on the proposed concept and tested this system to validate the concept
using one participant. The procedure for the proposed concept comprised three parts:
1) Preliminary experiment to explore the relationship between heartbeat and emotion;2) Creating a concept and developing a prototype system for emotion recognition;3) Testing the developed system and proposing a validated concept for emotion
recognition.
Preliminary experiment to explore the relationship between heartbeat andemotionThe physiological and motion signals of one participant were measured during
outpatient physical therapy, which the participant had been regularly receiving from oneof this paper’s authors, CK, once or twice a month for eight years and eleven months
when invited to participate in the study (Figure 1). Two of this paper’s authors, CK andHS, distributed flyers and introduced the study to the parents of children with PIMD.
The children included in this study met the following criteria: (1) Communication

Function Classification System (CFCS) levels IV–V [27]; (2) with ages above 5 years
and 0 months; (3) Gross Motor Function Classification System (GMFCS) levels IV–V
[28,29]. The participant is a 10 year-old boy who has cerebral palsy with profoundintellectual disability and epilepsy. He is bedridden and classified as a level five on the
GMFCS. He also wears a tracheal cannula to aspirate the sputum frequently. No
voluntary movements were observed, except for occasional tongue movements, which
also appeared to be involuntary. He seemed unable to communicate his emotion
verbally and through facial expressions, but his parent sometimes identified his
emotions (e.g., “He likes it.” or “He was stressed.”).
The participant was observed as changing emotion during the therapy. This
therapy primarily consisted of his practice of the motion range of the joints in the supine
position and occasionally in the seated position through the help of the physicaltherapist and was under the observation of his parent. His electrocardiogram (ECG) andacceleration in the chest and electromyogram (EMG) and acceleration on the right
digitorum and emoris were measured using paste-type wearable sensors (BioStamp,
MC10) with 1000 Hz ECG and EMG sampling frequencies and 31.25 Hz acceleration.
His therapy scene, eye gaze, and facial expression were recorded using three video
cameras with 30 or 60 fps. A session consisted of a set of therapy for 40 min and three
sessions were conducted on separate days in one year in this study. Each signal wasmeasured and tested in one session of them to determine whether or not it would
correlate with the final emotion label expressing the estimated emotion, which will be
described below.
The study protocol was approved by the Research Ethics Committee of Kitasato
University School of Allied Health Sciences (2020-012, 19 June 2020). Before testing,the child’s parent was informed about the procedure and the purpose of the study andsigned an informed consent form.
Creating a concept and developing a prototype system for emotion recognition
A concept for emotion recognition
We created a concept for emotion recognition, in which AI recognizes the emotions ofchildren with PIMD using physiological and motion signals. We hypothesized thatphysiological signals, especially a heartbeat, would show a correlation with the emotionof children with PIMD and aimed to validate it using the abovementioned procedure.
For the emotion recognition, we focused on the participant’s motion signals (e.g., eyeand tongue motions) and the EMG in his muscles that are weakly movable because
motion is important in emotion recognition [12]. Emotions could be recognized using
physiological and motion signals, but the correlation between emotion andphysiological and motion signals would be extremely complicated. AI recently showedsignificant advances in categorizing a set of data into classes and may be able to solve
the abovementioned complexity.
Physiological and motion signals for emotion recognition systems
Based on the proposed concept for emotion recognition systems based on AI using thephysiological and motion signals, we developed an emotion recognition systemprototype that can be used in real time with a personal computer, a tablet, or asmartphone to realize smooth communication with both familiar and unfamiliar people.
Eighteen types of signals, including eye gaze, tongue motion, EMG, and heartbeat
interval, R–R interval (RRI), which is the time elapsed between two successive R–
waves on ECG, were used as the physiological and motion signals featuring emotion(Table 1). These signals were measured using low physical burden sensors, such aswearable sensors and video cameras, during the participant’s outpatient physical therapy,
as described above.
As a physiological signal, the RRI was obtained from the ECG signals [30], andgeneral features (e.g., mean and median) were used every 10 s. The slope a, intercept b,
and coefficient of determination R2 in the linear approximation of the RRI time series
for 10 s was used to consider the temporal change in the RRI.
Certain frequency bands of the heart rate variability tend to correlate with acertain nervous system activity. Note that the correlation for children with cerebral
palsy is not the same as that for able-bodied children [24]. We used low-frequency
power (LF) and high-frequency power (HF), which are frequency activities standardized
in 0.04–0.15 and 0.15–0.40 Hz frequencies, respectively, and the LF/HF ratio [30,31].
Although the HF range for pediatric participants who breathe faster than adults should
be appropriately adjusted [24], the peak of the participant’s spectrum was confirmed toexist in the standard HF range of 0.15–0.40 Hz, and no adjustment was needed.
The eye gaze, tongue motion, breathing, and EMG were expressed in numericalform and used as the motion signals because of their importance. The eye gaze wasexpressed as 1 and 0 in the upward and downward directions, respectively. The tongue
motion was expressed as 1 and 0 in motion and no motion, respectively. Breathing was
expressed as 1 when taking a deep breath and 0, otherwise. The integral of the EMG(iEMG) is an indicator of the muscle activity intensity; hence, the iEMG on the right
digitorum and the right femoris of the participant for 10 s was used [32].
Random forest classifier based on AI for the emotion recognition system
A random forest (RF), which is an ensemble learning method for classification using
many decision trees, was employed to classify the acquired physiological and motion
signal data into a certain emotion through AI [33]. The RF algorithm would be suitable

for the emotion recognition systems for children with PIMD because it requires lessdata for learning and is more explainable than other algorithms (e.g., artificial neuralnetworks (deep learning)). In this work, the RF classifier was implemented in theemotion recognition system using Scikit-learn, which is a Python machine learning
library [34].
To learn the system and tailor it to the participant, the RF requires a dataset ofpaired inputs and labeled outputs that correspond to the physiological and motion
signals and emotion labels for each emotion, respectively. However, recognizing theemotions of children with PIMD on site and in real time is extremely difficult; hence,emotion labels cannot be obtained in general ways. Emotion labeling in the learning
process that does not require a real time process differing from the test process of thesystem is, therefore, important. Furthermore, a reliable emotion labeling procedure isrequired.
The desires to act during the outpatient physical therapy were labeled as “again”
for the participant wanting to do it again, “not again” for the participant not wanting to
do it again, “neutral” for neither of the two, and “unknown” for being unable to judge it[35]. Judging the negative emotion of the participant is relatively easy and reliable, andthe RRI is related to a negative emotion rather than positive and neutral emotions forable-bodied people [36]. Thus, “not again” was converted into the emotion label of“negative” expressed by −1; “again” and “neutral” were converted into “not negative”
expressed by 1; and “unknown” was expressed by 0. This simplified the emotion labels
to a one-dimensional value of −1, 0, or 1.
An estimator, who was likely to provide reliable emotion labels, created theemotion labels. The estimator may have perceived the created emotion as confident in
one scene, but not in another. Therefore, the confident indices included “not confident,”

“little confident,” “pretty confident,” and “confident” expressed numerically as 1, 2, 3,
and 4, respectively. Multiplying an emotion label value by a confident index yielded anemotion label value with confidence.
For more reliable labels, emotion labels with confidence were created by threeestimators, namely a physical therapist who regularly treats the participant and two coauthors of this study, including a physical therapist, who repeatedly watched videos andwere blinded to the results from the other data (e.g., RRI, ECG, and EMG). The sum of
the emotion labels estimated by the three estimators every 10 s was calculated in therange from −12 to 12 to increase the reliability like ensemble prediction [37]. By trial
and error, the final emotion label was obtained by setting the threshold value to 3 (25%of the maximum value of 12 in three persons) and estimating the summed value of 3 ormore as “not negative,” that of −3 or less as “negative,” and the other value as“unknown.” The usage of the values by the three estimators has the following
advantage; the summed value is high if one of the estimators has confidence in theemotion labels, resulting in reliable final estimation labels.
Figure 2 presents an overview of the developed emotion recognition system.
Testing the developed system and validating the concept for the emotion
recognition systems
We tested the developed emotion recognition system by comparing its recognition ratewith that of an unfamiliar person in the same experiment as in the preliminary
experiment. The unfamiliar person was selected under the condition that he did not havemuch knowledge of the participant and had no particular expertise in emotion
recognition considering the need of assistive devices in emotion recognition in actual
cases. As well as estimators, he created the final emotion labels using the thresholdvalue of 1 (25% of the maximum value of 4 in one person), which was obtained as thebest effort by trial and error, every 10 s by watching the video in the timeline.
The concept for the emotion recognition systems was validated by using the Fmeasure, a measure frequently used for imbalanced data [31], as an evaluation metric tomeasure the system recognition rate considering the imbalance in the final obtained data
(i.e., “unknown” did not indicate any emotion in the system, making it not muchmeaningful, and the average of the F-measure values of “negative” and “not negative”
was used as the system recognition rate).
Analyses
The recognition rates of the emotion recognition system and the unfamiliar person wereobtained through a four-fold cross-validation and statistically analyzed. In the four-foldcross-validation, the data were randomly divided into four folds of approximately equal
size [38]. The first fold was treated as a validation set. The learning for the RF classifier
was fit on the three remaining folds. This procedure was repeated for four times, andeach time, a different fold of the data was treated as a validation set. The results wereexpressed as mean ± S.D. t-Test was performed to determine the statistical significancesof the differences. p-Values less than 0.05 were considered significant.
Results
Preliminary experiment to explore the relationship between heartbeat andemotionThe RRI was obtained from the ECG. The median of the RRI was selected as arepresentative value of the state every 10 s (Figure 3). A significantly decrease in the
median of the RRI synchronized with the estimators’ judgment from the video that theparticipant may have a “negative” emotion.

Testing the developed emotion recognition system
Figure 4 depicts the time series of the emotion label value with confidence estimated byeach estimator, summed emotion label value of the three estimators, and final emotion
label value. Figure 5 compares the time series of the emotion label value withconfidence estimated by the unfamiliar person and the final emotion label value.
The system recognition rate was obtained as the F-measure average of “negative”
and “not negative” through the four-fold cross-validation. The system recognition ratesof “negative” and “not negative” (i.e., 70.4% ± 6.1% (mean ± S.D.)) were significantly
higher than those of the unfamiliar person (i.e., 48.5% ± 5.0% (p < 0.05)) (Table 2).
Discussion
Potential of emotion recognition using physiological and motion signals
In the preliminary experiment, the participant seemed “negative” from the video whenthe median of the RRI decreased. In some of the “negative” scenes, muscle stretching
was performed, and the participant appeared to be in pain. The time series of the medianof the RRI shown in Figure 3 partially agreed with those of the summed emotion label
value of the three estimators in Figure 4(d), especially in the low RRI median range. A
conventional study found that pain changes the heart rate [39]. Hence, the RRI may berelated to the emotions of children with PIMD and beneficial for emotion recognition.
By contrast, when the median of the RRI decreased in another scene, the participant’s
posture was changed from a supine position to a sitting position. The autonomic
nervous activity changed with the changing posture [40]. The result in this study wasconsistent with it, indicating that the RRI may change because of both emotion andposture changes, and no simple one-to-one correspondence exists between the medianof the RRI and emotion. Meanwhile, the parent believed that it was negative when theparticipant’s eyes raised, and that it was a response from the participant when the
tongue moved. Recognizing emotions by AI requires the use of not only the RRI, butalso of other physiological and motion signals, such as eye gaze and tongue motion.
Validity of the proposed concept and usefulness of the developed system
The emotion recognition system based on our proposed concept, which uses the
RF classifier as AI taking physiological and motion signals as the inputs, recognized the“negative” and “not negative” emotions of children with PIMD by 70.4% significantly
higher recognition rate compared to the unfamiliar person, who had 48.5%. The finding
suggests that the proposed concept may be valid. This also implies that the emotion
recognition system based on the proposed concept can be useful as a real time assistivesystem that will enable children with PIMD to communicate with others, improve their
quality of life, and allow their caregivers to more easily care for them. Although nopractical communication assistive system has yet been made available for emotion
recognition for children with PIMD, many gaze-based assistive systems have been used
as conventional ones for nonverbal children with severe physical impairments. One
study on children with severe physical impairments indicated that gaze-based assistivesystems provide an opportunity for people involved with these children. These systems
allow them to better understand the children’s needs, including the “not again.” This is arelief because parents need not worry much about others not being able to understandtheir children’s needs [41]. Although children with PIMD cannot use gaze-based
assistive systems due to their disabilities and their need of emotion recognition systems,
the performance of our emotion recognition system in terms of judging “negative” or“not negative” expression can provide the same benefit.
When the emotion recognition system is used as a communication tool between
children with PIMD and others, the emotions must be recognized using the dataacquired from sensors on site and in real time. Low physical burden sensors are used in
the system and can be integrated into medical devices that children with PIMD use daily.
We confirmed that the RF classifier for the emotion recognition has a short computation
time that is sufficient for communicating with children in real time using a standard
laptop computer. Therefore, the emotion recognition system is feasible as acommunication tool for hardware and software.
Limitation and future works
Obtaining the true emotions of children with PIMD is extremely difficult. Therefore, theemotions are estimated by the estimators, who are likely to provide reliable emotions,and the system for outputting the estimated emotions from the inputs of physiologicaland motion signals is obtained through the AI learning process. In the future work, it
will be possible to make the estimated emotion closer to the true emotion by increasing
the number of estimators to integrate the emotion label, which will improve the system
recognition rate. A correlation hint between the physiological and motion signals andthe emotion of children with PIMD may be found if the number of participants isincreased, and the system is modified.
Four major issues must be addressed in the future. The first one is the
improvement of the system recognition rate by using other classifiers, such as artificial
neural networks, considering real time processing and the required amount of data. The
second issue is the analysis of data collected on different days and the identification ofdifferences in the physiological characteristics based on that day. The third one is thedetermination of the reason for the system’s ability to recognize emotion and thevalidation of this reason. The fourth and last issue is the usage of automatic image
processing to obtain the eye gaze and tongue motion instead of manual image
processing.

Conclusion
In this study, we created a concept based on AI for the emotion recognition systems forchildren with PIMD by using physiological and motion signals, developed and tested anemotion recognition system based on the proposed concept using the RF classifier thattook as inputs the RRI, eye gaze, and other data acquired using low physical burden
sensors, and proposed the validated concept. The “negative” and “not negative”
recognition rate of the developed system (i.e., 70.4% ± 6.1%) is higher than that of theunfamiliar person (i.e., 48.5% ± 5.0%). The result suggests that the proposed concept
for the emotion recognition systems may be useful and valid for communicating withchildren with PIMD.
Disclosure Statement
No potential conflict of interest was reported by the author(s).
Funding
This work was supported by JSPS KAKENHI Grant Number 19K11330.
References[1]

Chou YC, Chiao C, Fu LY. Health status, social support, and quality of life
among family carers of adults with profound intellectual and multiple
disabilities (PIMD) in Taiwan. J Intellect Dev Disabil. 2011;36(1):73–79.

[2]

Suzuki Y, Tatsuno M, Yamada M. [Seriously Handicapped Children with
Intensive Medical Cares. Definition and Problems]. J Child Health.
1995;54(3):406–410. Japanese.

[3]

Mesquita B, Frijda NH. Cultural variations in emotions: a review. Psychol Bull.
1992 Sep;112(2):179-204.

[4]

Hutchison A, Gerstein L, Emotion recognition, emotion expression, and cultural
display rules: implications for counseling. J Asia Pac Couns. 2017;7(1):19–35.

[5]Mencap. Communicating with people with profound and multiple learning
disabilities (PMLD) [Internet]; [cited 2022 feb]. Available from:https://www.jpaget.nhs.uk/media/104336/Communicating_with_people_with_P
MLD__a_guide__1_.pdf.

[6]Adams D, Oliver C, The expression and assessment of emotions and internal
states in individuals with severe or profound intellectual disabilities. Clin
Psychol Rev. 2011;31(3):293–306.

[7]

Ekman P, Freisen WV, Ancoli S. Facial signs of emotional experience. J Pers
Soc Psychol. 1980;39(6):1125–1134.

[8]

Nozaki Y, Kawasumi R. [Challenges for teachers instructing children withsevere motor and intellectual disabilities and medical care dependent groups].
Annu Bull Grad Sch Educ Tohoku Univ. 2012;60(2):225–241. Japanese.

[9]

Matsuda T. [Support of school children]. Jpn J Dev Disabil. 2003;25:150–158.
Japanese.

[10]

Okazawa S, Kawasumi R. [Educational approaches to a boy with profound andmultiple disabilities]. Jpn Assoc Spec Educ. 2005;43:203–214. Japanese.

[11]

Abe S, Tsujino M, Nakata S, et al. [Nonverbal communication of children withprofound intellectual and multiple disabilities having visual and auditory
disorders]. J Jpn Soc Nurs Sci Natl Health Serv. 2007;3(1):38–42. Japanese.

[12]

Araki H, Iwamoto Y, Arima M. [Development of communication support
system for children (persons) with severe motor and intellectual disabilities]. Jpn
J Ergon. 2016;52(Suppl):290–291. Japanese.

[13]

Sesin A, Adjouadi M, Cabrerizo M, et al. Adaptive eye-gaze tracking usingneural-network-based user, profiles to assist people with motor disability, J
Rehabil Res Dev. 2008;45(6):801–817.

[14]

Falk TH, Guirgis M, Power S, et al. Taking NIRS-BCIs outside the lab: towards
achieving robustness against environment noise. IEEE Trans Neural Syst
Rehabil Eng. 2011;19(2):136–146.

[15]

Harada A, Landay JA, Malkin J, et al. The vocal joystick evaluation of voicebased cursor control techniques for assistive technology. Disabil Rehabil Assist
Technol. 2008;3(1):22–34.

[16]

Struijk LNSA, Lontis ER, Gaihede M, et al. Development and functional
demonstration of a wireless intraoral inductive tongue computer interface forseverely disabled persons, Disabil Rehabil Assist Technol. 2017;12(6):631–640.

[17]Kitajima Y. [Studies on children with severe motor and intellectual disabilities
using physiological psychology indices]. Jpn Assoc Spec Educ. 2005;43:225–
231. Japanese.

[18]

Sakamoto K, Sakashita S, Yamashita K, et al. [Effect of TV viewers’ emotional
states on physiological and psychological measurements]. Panasonic Tech J.
2013;59(1): 29–34. Japanese.

[19]

Murase C, Kawamoto R, Sugimoto S, [Changing of emotions by the stimulationof visual and auditory senses –an analysis of heart rate variability (HRV)–]. J
UOEH. 2004;26(4):461–471. Japanese.

[20]

Valderas MT, Bolea J, Laguna P, et al. Human emotion recognition using heart
rate variability analysis with spectral bands based on respiration. 2015 37th
Annual International Conference of the IEEE Engineering in Medicine andBiology Society (EMBC). 2015;6134–6137.

[21]

Ooe H, Yamada T, Kobayashi I. [Behavior in
intellectual

person withprofoundand multiple disabilities]. Proc Jpn Assoc Spec Educ 43rd Conf.

2005:222. Japanese.
[22]

Hayano J, Tanabiki T, Iwata S, et al. Estimation of emotions by wearable
biometric sensors under daily activities. 2018 IEEE 7th Global Conference on
Consumer Electronics (GCCE), 2018;268–269.

[23]

Garg A, Chaturvedi V, Kaur AB, et al. Machine learning model for mapping ofmusic mood and human emotion based on physiological signals. Multimed
Tools Appl. 2022;81:5137–5177.

[24]

Gasior JS, Zamunér AR, Silva LEV, et al. Heart rate variability in children andadolescents with cerebral palsy–a systematic literature review, J Clin Med.
2020;9(4): 1141-1–25.

[25]

McCarthy J. WHAT IS ARTIFICIAL INTELLIGENCE? [Internet]; [cited 2022
sept]. Available from: http://wwwformal.stanford.edu/jmc/whatisai/whatisai.html.

[26]

Arrieta AB, Diaz-Rodriguez N, Ser JD, et al. Explainable artificial intelligence(XAI): concepts, taxonomies, opportunities and challenges toward responsible
AI. Inf Fusion. 2019;58:82–115.

[27]

Hidecker MJ, Paneth N, Rosenbaum PL, et al . Developing and validating thecommunication function classification system for individuals with cerebral palsy.
Dev Med Child Neurol. 2011;53:704–710.

[28]Palisano R, Rosenbaum P, Walter S, Russell D, Wood E, Galuppi B (1997)
Development and reliability of a system to classify gross motor function inchildren with cerebral palsy. Dev Med Child Neurol 39:214–223.

[29]

Palisano RJ, Rosenbaum P, Bartlett D, et al . Content validity of the expanded
and revised gross motor function classification system. Dev Med Child Neurol.
2008;50:744–750.

[30]

Malik M, Bigger JT, Camm J, et al. Heart rate variability: standards ofmeasurement, physiological interpretation and clinical use. Eur Heart J.
1996;17(3):354–381.

[31]

Li G, Yu Y, Visual saliency based on multiscale deep features, 2015 IEEE
Conference on Computer Vision and Pattern Recognition (CVPR). 2015;5455–
5463.

[32]

Nakamura H. [Back to Basics Series: Biomedical Measurement 1(Electromyography)]. Trans Jpn Soc Med Biol Eng. 2019;57(4,5):149–157.
Japanese.

[33]

Habe H, [Random forests: fundamentals and recent trends, J ITE.
2016;70(9):788–791. Japanese.

[34]

Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: machine learning in
Python. J Mach Learn Res. 2011;12:2825–2830.

[35]

Atkin K, Lorch MP. An ecological method for the sampling of nonverbal
signalling behaviours of young children with profound and multiple learning
disabilities (PMLD). Dev Neurorehabil. 2016;19(4):211–25.

[36]

Choi KH, Kim J, Kwon OS, et al. Is heart rate variability (HRV) an adequate
tool for evaluating human emotions? – A focus on the use of the international
affective picture system (IAPS). Psychiatry Res. 2017 May;251:192–196.

[37]

Japan Meteorological Agency. About TCC (Ensemble Prediction System)[Internet]; [cited 2022 sept] Available from:https://ds.data.jma.go.jp/tcc/tcc/about/products/aboutModel.html.

[38]

James G, Witten D, Hastie T, et al. An introduction to statistical learning. New
York (NY): Springer, 2013.

[39]

Riquelme I, Rosário RS, Vehmaskoski K, et al. Autonomous nervous systemregulation of pain in children with cerebral palsy, Brain Inj. 2021;35(3):356–362.

[40]Pagani M, Lombarsi F, Guzzetti S, et al. Power spectral analysis of heart rate
and arterial pressure variabilities as a marker of sympatho-vagal interaction in
man and conscious dog. Circ Res. 1986;59(2):178–193.

[41]

Borgestig M, Rytterström P, Hemmingsson H. Gaze-based assistive technology
used in daily life by children with severe physical impairments – parents’experiences. Dev Neurorehabilit. 2017;20(5):301–308.

Table 1. List of the acquired physiological and motion signals.
Mean of RRI

S.D. of RRI

Maximum of RRI

Minimum of RRI

Skew of RRI

Kurtosis of RRI

Median of RRI

LF

HF

LF/HF

a of RRI = at + bb of RRI = at + b

R2 of RRI = at + b

Eye gaze

Tongue motion

Deep breathiEMG (right leg)

iEMG (right arm)

Table 2. Emotion recognition rate of “negative” and “not negative” [%].
1

2

3

4

Mean ± S.D.

61.6

68.1

75.7

76.4

*70.4 ± 6.1

Unfamiliar person 56.8

45.9

43.5

47.6

48.5 ± 5.0

Proposed system*p < 0.05

Figure 1. Overview of the experiment.

Final emotion labels by threeestimators as “negative,”“not negative,” and “unknown”
18 types ofphysiological andmotion signals

Learning
Random forestclassifier

Figure 2. Overview of the developed emotion recognition system.

MusclestretchingMusclestretching

Figure 3. Time series of the median of RRI.

(a)

(b)

(c)

(d)

(e)

Figure 4. Time series of emotion label values by estimators. (a–c) The emotion label
value with confidence estimated by each estimator. (d) The summed emotion label
value of three estimators. (e) The final emotion label value.

Figure 5. Time series of emotion label values by an unfamiliar person. (a) The emotion
label value with confidence is estimated by an unfamiliar person. (b) The final
emotion label value.