Background

January 6, 2024
2024 IEEE International Conference on Consumer Electronics

2● Individual’s good work-life balance are being recommended [1]● Many changes in work style can be burdensome for workers[2]● There is still room to consider ways of working

Indoor Environment Control Methodfor Improving Well-Being
using Human Thermal Stress Estimatedby Yuragi Learning● Increasing productivity while taking into account personal well-being [3]● Biometric information can estimate stress state[6]● Using skin potential activity, skin temperature, and pulse wave

Keita Hara1, Risa Yoshida1, Daichi Kominami1,
Yuichi Ohsita1, Michiko Kaihotsu2 and Masayuki Murata1
Osaka University1
Daikin industry2

Remote workingChildcare

Nursing care[1] “Abenomics: For future growth, for future generations, and for a future japan (Online available).
[2] D. E. Guest, “Human resource management and employee well-being: Towards a new analytic framework,” Human resource management journal, 2017.
[3] A. Sakuraya, et al., “Job crafting, work engagement, and psychological distress among japanese employees: A cross-sectional study,” BioPsychoSocial medicine, 2017.
[6] P. Schmidt, et al., “Introducing wesad, a multimodal dataset for wearable stress and affect detection,” in Proc. ACM ICMI, 2018.

Purpose and Problems

3● Purpose

Proposal

4● Realizing well-being space by 3 steps● Estimating the stress state of each individualfrom human biological information● Controlling the indoor environment
based on human stress‣ Individual can stay comfortably

1. Sense various biometric information using biometric sensors
2. Estimate a person’s psychological state from the biometric information
3. Intervene indoor space by controlling actuators‣ To relieve the stress state● We implement this system and conduct experiments
Image of comfortable room

● Problems②● The biological information from wearable sensors contains noise● Stress affects biometric information differently for each individual‣ Using Yuragi learning to solve these problems
Even in the same indoor environment●●

Changed modality
Electrodermal activity: HIGHSkin temperature: LOW

Changed modality●●
Subject A

Subject B①③

Electrodermal activity: LOW
Skin temperature: HIGH

Overview of system

Image of Stress affects biometric information differently person to person

Wearable Stress and Affect Detection (WESAD)[6] 5● Biometric information can estimate human emotional state● Various biometric information from 15 subjects‣ Blood volume pulse (BVP), Electrocardiogram (ECG),
Electrodermal activity (EDA), Electromyogram (EMG),
Respiration (RESP), Skin temperature (TMP),
and 3-axis accelerometer (ACC)● Their self-reported is used as emotional-state label‣ Neutral, Stress, Amusement● They achieved high accuracy in estimating psychological states● Using machine learning methods on these data
● Decision trees, Random forests, AdaBoost,Linear discriminant analysis, K-nearest neighbor methods● Our research use as reference this study● Our research uses various biometric information too● Our research uses Yuragi learning[7] for estimating stress state[7] M. Murata and K. Leibnitz, Fluctuation-Induced Network Control and Learning: Applying the Yuragi Principle of Brain and Biological Systems. Springer, 2021.

Bayesian attractor model (BAM)● BAM models the decision process that human brain makes● BAM decides which of choices the observed information corresponds to‣ This decision-making can be made even from noisy information
Observation
Observation xt is input to BAM

Generative model of zt- Generative model of zt has K attractors- Each of K choices are pre-stored on each attractor

State estimation- Estimate P(zt | xt)- Using unscented Kalman filter

DecisionP(zt = Ჶn| xt) > λ ?

Overview of BAM● Decisions are made according to confidence level● Confidence level means that how zt is close to one of the attractors Ჶn● Adopt Attractor whose confidence level exceeds the threshold λ‣ This allows decisions to be made with fewer errors for noisy
observations

6

Multimodal recognition based on Yuragi learning

7● Using BAM for decision making machine learning methods

Experimental setup● Our research group has succeeded in various applications [7]● Can be run on relatively small systems● Indoor environment is set based on PMV[14]‣ Comfortable room was set as PMV = 0(20℃, 40% humidity)‣ Uncomfortable room was set as PMV = +2
(30℃, 70% humidity)● Composed of frameworks that use BAM● Input observed information (preprocessing)● Set attractor (training)● Make decision● Experimental steps

Experiment Room

● Acquisition of tain data (10min * 2)
● Actuator-control experiment (10min)‣ Estimated subject’s state is checked every 10 seconds‣ Actuator changes a room environment according to this state● In this study, integrating multiple biometric outputs

data acquisition

10 minRoom1(PMV = 0)

Subject
Overview of Estimating

Devices8● Two rooms were set up as experimental environments[14] “ISO 7730:2005 (Online available).

9● Empatica E4 wristband [9]Control experiment

10 min10 minRoom2(PMV = +2)

Room2(PMV = +2)

Experiment flow

Result10● Stress state are estimated and control execution is confirmed● It used for acquiring ACC, BVP, EDA, IBI, TMP● Green line represents the estimated state at each time step
● Improved accuracy by combining multiple biometric data
● In this experiment, the control always runs

● Laptop computer● Installs a streaming server which receives biometric information‣ Connected to the E4 wristband via Bluetooth● Conducts Yuragi learning● Sends control commands to an actuator by the estimated result

Stress state is estimated andAir is blowing● Daikin assisted circulator [10]● The circulator is connected to a Raspberry Pi 3 model B (PI3)‣ its operation can be changed by serial commands sent from the PI3‣ Blowing wind when stress state is estimatedIntegrate

Integrated result

E4 wristband and assisted circulator

Result of Estimation[9] “E4 wristband,” https://www.empatica.com/en-int/research/e4/, (Accessed on Nov. 1, 2023).
[10] “Daikin assisted circulator,” https://launch.ac.daikin.co.jp/onlineshop/ detail03, (Accessed on Nov. 1, 2023).

Conclusion● We proposed a Yuragi Learning based stress estimation method● Integrating multiple types of biometric information● Experiments have shown that the system can operate corectry● Future works● How to control the actuators to completely remove stress‣ Pursuit of adequacy of actuating methods● How to estimate and control them for different people at the same time
● Another experiment senario‣ Control PMV by changing exercise‣ In this research we control PMV by temperature and humidity

11

References (1/3)

14● [1] “Abenomics: For future growth, for future generations, and for a futurejapan,”https://www.japan.go.jp/abenomics/_userdata/abenomics/pdf/1901_abenomics.pdf (Accessed on Nov. 1, 2023).
● [2] D. E. Guest, “Human resource management and employee well-being:
Towards a new analytic framework,” Human resource management journal,
vol. 27, no. 1, pp. 22–38, 2017.
● [3] A. Sakuraya, A. Shimazu, H. Eguchi, K. Kamiyama, Y. Hara, K. Namba, and N.
Kawakami, “Job crafting, work engagement, and psychological distress among
japanese employees: A cross-sectional study,” BioPsychoSocial medicine, vol.
11, no. 1, pp. 1–7, 2017.
● [4] T. Kogo, M. Tsujikawa, Y. Kiuchi, A. Nishino, and S. Hashimoto, “Model
predictive control of shallow drowsiness: Improving productivity of officeworkers,” in Proceedings of Annual International Conference of the IEEE
Engineering in Medicine and Biology Society, Jul. 2019, pp. 2459–2465.

References (2/3)

15● [5] S. Colenberg, T. Jylha, and M. Arkesteijn, “The relationship between ¨interior office space and employee health and well-being–a literature review,”
Building Research & Information, vol. 49, no. 3, pp. 352–366, 2021.
● [6] P. Schmidt, A. Reiss, R. Duerichen, C. Marberger, and K. Van Laerhoven,“Introducing wesad, a multimodal dataset for wearable stress and affectdetection,” in Proceedings of the 20th ACM international conference on
multimodal interaction, 2018, pp. 400–408.
● [7] M. Murata and K. Leibnitz, Fluctuation-Induced Network Control andLearning: Applying the Yuragi Principle of Brain and Biological Systems.
Springer, 2021.
● [8] S. Bitzer, J. Bruineberg, and S. J. Kiebel, “A Bayesian attractor model forperceptual decision making,” PLoS Computational Biology, vol. 11, no. 8, p.
e1004442, 2015.
● [9] “E4 wristband,” https://www.empatica.com/en-int/research/e4/, (Accessedon Nov. 1, 2023).

References (3/3)

16● [10] “Daikin assisted circulator,” https://launch.ac.daikin.co.jp/onlineshop/
detail03, (Accessed on Nov. 1, 2023).
● [11] A. Greco, G. Valenza, A. Lanata, E. P. Scilingo, and L. Citi, “cvxEDA: A
convex optimization approach to electrodermal activity processing,” IEEE
transactions on biomedical engineering, vol. 63, no. 4, pp. 797– 804, 2015.
● [12] “YLPF: Yuragi Learning Platform Core Module,” https://github.com/
nbic-ist-osaka-u-ac-jp/YLPF-Core (Accessed on Nov. 1, 2023).
● [13] “bayesian-attractor-model-python-cpp-,” https://github.com/
d-kominami/bayesian-attractor-model-python-cpp- (Accessed on Nov. 1,2023).
● [14] “ISO 7730:2005,” https://www.iso.org/standard/39155.html (Accessed onNov. 1, 2023).