A Framework for Daily Mental Attitude Monitoringof Workers∗

Kai Suwabe† , Takashi Maegawa† , Teruhiro Mizumoto‡ , Hirohiko Suwa† , and Keiichi Yasumoto††Nara Institute of Science and Technology, Nara, Japan,‡
Osaka University, Osaka, Japan,∗
kai.suwabe@ubi-lab.com

Abstract—Monitoring mental health is important for people
at work, as well as in other aspects of our lives. This is because
mental states affect workers’ health and labor productivity. Many
governments in developed countries are working to support
workers’ mental health, but previous methods are based onquestionnaires and interviews, which are not suitable for dailyuse. Therefore, we propose ”Work Attitude PLR (Personal LifeRecord) collection platform” that can continuously estimate
and record ”Work Attitude” (a composite measure of work
engagement and recovery experience, etc.) using multimodal
information. In this paper, we describe the framework of ourproposal and show the performance of the estimation model that
positively suggest the feasibility of our method.
Index Terms—Work style reform, Stress, Work engagement,Recovery, Multi-modal information, Emotion recognition

TABLE IW ORK ATTITUDE SCALES
Measurement category
RecoveryExperience [5]Recovery State [6]Work Engagement [7]Scale
RE1RE2RE3RE4RS1RS2RS3WE1
WE2
WE3

Summary
Psychological detachment
Relaxation
Mastery experiences
Control during leisure time
Feeling physically refreshedFeeling mentally refreshed
Sleep efficiency
Vigor
Dedication
Absorption

II. P ROPOSAL M ETHOD
I. I NTRODUCTION
It is important for workers to maintain a healthy mental state
to ensure labor productivity. For this reason, many developed
countries, including Japan, where many workers are engaged
in long hours of work and suffer from poor health and highlevels of stress, are monitoring workers’ mental health andtrying to create healthier work environments. However, themonitoring methods currently used in practice are interviews
and questionnaires [1], and are not intended to be used on adaily basis.
With the spread of smartphones, there has been growinginterest in using mobile devices to monitor users’ mental
states and promote self-care [2], and methods for estimating
psychological states using smart devices have been proposed[3] [4]. However, many previous works use information fromcontact devices such as smartwatches, which places a highburden on the user. In addition, most studies have focused on
negative indicators such as stress and depression, and not on
positive indicators of worker well-being. Even when stress ispresent, it is considered not to be a problem when workers
are highly motivated to work and recover from stress, so both
positive and negative indicators need to be included in orderto maintain workers’ mental health.
Therefore, we developed the Work Attitude scale, which
can evaluate both positive and negative mental attitudes ofworkers toward their work. We also propose a ”Work Attitude
PLR (Personal Life Record) collection platform” that cancontinuously estimate Work Attitude based on multimodal
information obtained from only smartphones.

A. Work Attitude
We developed the Work Attitude scale with the cooperationof clinical psychologists to assess the mental state of workers
from both positive and negative perspectives. The Work Attitude scale is a composite of three existing scales: Recovery
Experience [5], Recovery State [6], and Work Engagement [7].
These scales are summarized in Table I.
B. Work Attitude PLR collection platform
Fig. 1 shows an overview of Work Attitude PLR collection
platform. The system consists of a smartphone and a server.
Smartphones collect activity data and selfie video data(about 10 seconds). The activity data are the distance traveled
and the number of steps taken. The selfie video data is theworker’s brief message about how he or she is feeling at thatmoment. The smartphone sends these data to the server.
Data Collection

Work Attitude Estimation ServerWork Attitude

DataFeedback to Workers

WorkerDataFeature
Extraction

Features

Work Attitude
EstimationWork Attitude

Processing in Server

Fig. 1. Overview of Work Attitude PLR collection platform.

The server extracts features from the received data andinputs them to the Work Attitude estimation model. In feature
extraction, emotion values are calculated from each of the face
images, voice, and text (the speech content) channel of theselfie video data using the Face API (Microsoft), Empath API(Empath), and Cotoha API (NTT Communications), respectively. The features of the face image channel are the mean,
median, variance, and maximum of the emotion values foreach of the 5 images extracted from the video. As a result,
78 types of data are used as features. The estimation model isconstructed using Random Forest. The output Work Attitude
is recorded in a database as a PLR and fed back to the worker.
III. E XPERIMENTS & R ESULTS

IV. C ONCLUSION
For daily mental attitude monitoring of workers, we proposed Work Attitude, which can represent both positive andnegative perspectives, and Work Attitude PLR collection platform, which estimates, stores, and feeds back Work Attitude.
Experimental results showed significantly higher F1 scoresfor several measures, and feature selection further enhanced
this performance. The frequency distributions of the selectedfeatures, which differ from scale to scale, indicate the effectiveness of using multimodal information in our method.
In the future, we will collect large-scale data to ensure a sufficient number of samples, and then conduct a demonstration
experiment using data that has been processed by re-sampling.
ACKNOWLEDGMENT
This work was supported by MEXT ”Innovation Platform
for Society 5.0” Program Grant Number JPMXP0518071489.

Fig. 2. Estimation performance. (Permutation test, uncorrected, *: P<0.01;+: P<0.05)
RE2

RE44080

RE33550255040

Times50

Times60

Times30402040

153020102010105000

FeatureRS1

RS1

smile_manger_m
contempt_m
disgust_m
fear_mhappiness_m
neutral_msadness_m
surprise_m
smile_v
anger_v
contempt_v
disgust_v
fear_vhappiness_v
neutral_v
sadness_v
surprise_v
smile_max
anger_max
contempt_max
disgust_max
fear_maxhappiness_max
neutral_max
sadness_max
surprise_max
smile_med
anger_med
contempt_med
disgust_med
fear_medhappiness_med
neutral_med
sadness_med
surprise_med
calm
anger
joy
sorrowenergyPos/Neu/NegPN
PN
kanashiifuankonomashii
koufunyorokobu
annshiniya
StepFeature

RS3

70600Feature

RS280

70

10

smile_manger_m
contempt_m
disgust_m
fear_mhappiness_m
neutral_msadness_m
surprise_m
smile_v
anger_v
contempt_v
disgust_v
fear_vhappiness_v
neutral_v
sadness_v
surprise_v
smile_max
anger_max
contempt_max
disgust_max
fear_maxhappiness_max
neutral_max
sadness_max
surprise_max
smile_med
anger_med
contempt_med
disgust_med
fear_medhappiness_med
neutral_med
sadness_med
surprise_med
calm
anger
joy
sorrowenergyPos/Neu/Neg
PN
PN
kanashiifuankonomashii
koufunyorokobu
annshiniya
Step
Step_log
Distance
Distance_log
Sleep_efficiency

Distance_log
Sleep_efficiency

Step_log
Distance

30

20

smile_manger_m
contempt_m
disgust_m
fear_mhappiness_m
neutral_msadness_m
surprise_m
smile_v
anger_v
contempt_v
disgust_v
fear_vhappiness_v
neutral_v
sadness_v
surprise_v
smile_max
anger_max
contempt_max
disgust_max
fear_maxhappiness_max
neutral_max
sadness_max
surprise_max
smile_med
anger_med
contempt_med
disgust_med
fear_medhappiness_med
neutral_med
sadness_med
surprise_med
calm
anger
joy
sorrowenergyPos/Neu/Neg
PN
PN
kanashiifuankonomashii
koufunyorokobu
annshiniya
Step30Feature

RE4

70

60

Step_log
Distance

RE27060

smile_manger_m
contempt_m
disgust_m
fear_mhappiness_m
neutral_msadness_m
surprise_m
smile_v
anger_v
contempt_v
disgust_v
fear_vhappiness_v
neutral_v
sadness_v
surprise_v
smile_max
anger_max
contempt_max
disgust_max
fear_maxhappiness_max
neutral_max
sadness_max
surprise_max
smile_med
anger_med
contempt_med
disgust_med
fear_medhappiness_med
neutral_med
sadness_med
surprise_med
calm
anger
joy
sorrowenergyPos/Neu/NegPN
PN
kanashiifuankonomashii
koufunyorokobu
annshiniya
Step
Step_log
Distance
Distance_log
Sleep_efficiency

Times

RE380

RE170

Distance_log
Sleep_efficiency

RE180

35

RS260

RS330

50

25

40

2040

Times

Times

Times

5030

1520

1020

10

510

Featuresmile_manger_m
contempt_m
disgust_m
fear_mhappiness_m
neutral_msadness_m
surprise_m
smile_v
anger_v
contempt_v
disgust_v
fear_vhappiness_v
neutral_v
sadness_v
surprise_v
smile_max
anger_max
contempt_max
disgust_max
fear_maxhappiness_max
neutral_max
sadness_max
surprise_max
smile_med
anger_med
contempt_med
disgust_med
fear_medhappiness_med
neutral_med
sadness_med
surprise_med
calm
anger
joy
sorrowenergyPos/Neu/NegPN
PN
kanashiifuankonomashii
koufunyorokobu
annshiniya
Step
Step_log
Distance
Distance_log
Sleep_efficiency

Distance_log
Sleep_efficiency

Step_log
Distance0
smile_manger_m
contempt_m
disgust_m
fear_mhappiness_mneutral_msadness_m
surprise_m
smile_v
anger_v
contempt_v
disgust_v
fear_vhappiness_v
neutral_v
sadness_v
surprise_v
smile_max
anger_max
contempt_max
disgust_max
fear_maxhappiness_max
neutral_max
sadness_max
surprise_max
smile_med
anger_med
contempt_med
disgust_med
fear_medhappiness_med
neutral_med
sadness_med
surprise_med
calm
anger
joy
sorrowenergyPos/Neu/Neg
PN
PN
kanashiifuankonomashii
koufunyorokobu
annshiniya
Step0
smile_manger_m
contempt_m
disgust_m
fear_mhappiness_m
neutral_msadness_m
surprise_m
smile_v
anger_v
contempt_v
disgust_v
fear_vhappiness_v
neutral_v
sadness_v
surprise_v
smile_max
anger_max
contempt_max
disgust_max
fear_maxhappiness_max
neutral_max
sadness_max
surprise_max
smile_med
anger_med
contempt_med
disgust_med
fear_medhappiness_med
neutral_med
sadness_med
surprise_med
calm
anger
joy
sorrowenergyPos/Neu/NegPN
PN
kanashiifuankonomashii
koufunyorokobu
annshiniya
Step
Step_log
Distance
Distance_log
Sleep_efficiency

0Feature

WE1Feature

WE240

WE340

WE1353030

WE23530WE3252020

Times25

Times

Times25

2015151010550015

Times selected30

Features10Feature

FeatureStep_log
Distance

Distance_log

smile_manger_m
contempt_m
disgust_m
fear_mhappiness_m
neutral_msadness_m
surprise_m
smile_v
anger_v
contempt_v
disgust_v
fear_vhappiness_v
neutral_v
sadness_v
surprise_v
smile_max
anger_max
contempt_max
disgust_max
fear_maxhappiness_max
neutral_max
sadness_max
surprise_max
smile_med
anger_med
contempt_med
disgust_med
fear_medhappiness_med
neutral_med
sadness_med
surprise_med
calm
anger
joysorrowenergyPos/Neu/Neg
PN
PN
kanashiifuankonomashii
koufunyorokobu
annshiniya
Step0
smile_manger_m
contempt_m
disgust_m
fear_mhappiness_m
neutral_msadness_m
surprise_m
smile_v
anger_v
contempt_v
disgust_v
fear_vhappiness_v
neutral_v
sadness_v
surprise_v
smile_max
anger_max
contempt_max
disgust_max
fear_maxhappiness_max
neutral_max
sadness_max
surprise_max
smile_med
anger_med
contempt_med
disgust_med
fear_medhappiness_med
neutral_med
sadness_med
surprise_med
calm
angerjoysorrowenergyPos/Neu/NegPN
PN
kanashiifuankonomashii
koufunyorokobu
annshiniya
Step
Step_log
Distance
Distance_log

Step_log
Distance

Distance_log

5

smile_manger_m
contempt_m
disgust_m
fear_mhappiness_m
neutral_msadness_m
surprise_m
smile_v
anger_v
contempt_v
disgust_v
fear_vhappiness_v
neutral_v
sadness_v
surprise_v
smile_max
anger_max
contempt_max
disgust_max
fear_maxhappiness_max
neutral_max
sadness_max
surprise_max
smile_med
anger_med
contempt_med
disgust_med
fear_medhappiness_med
neutral_med
sadness_med
surprise_med
calm
angerjoysorrowenergy
Pos/Neu/Neg
PNPN
kanashiifuankonomashii
koufunyorokobu
annshiniyaStep

We evaluated the performance of our estimation model.
9 male subjects participated in the 2-week experiment.
Subjects were asked to take video only and to act as usual
otherwise. As a result, a total of 155 samples were obtained.
In addition, subjects answered the Work Attitude value usedas the ground-truth in questionnaires after the video recording.
The results were merged into 2 classes for RS3 and into 3classes for the other scales to ensure a sufficient sample size.
The model was evaluated by leave-1 sample-out crossvalidation in two ways: with all features and with feature
selection. Feature selection was based on Gini importance.
The estimation was performed sequentially by adding features
one by one, starting with the most important features, and thesubset of features with the best performance was adopted.
The F1 scores for each scale of the Work Attitude estimation
model are shown in Fig. 2. The dashed bars indicate the F1score calculated by interpolating the corresponding precision
as 0 because there were no cases classified into a particularclass. The F1 score of RE1 was significantly higher than thechance level. For RE2, RS1, and RS2, significance was shown
only when features were selected.
In addition, Fig. 3 shows the frequency distribution of eachfeature selected. Here, different trends were observed for eachscale. This suggests that our proposed method using multimodal information is suitable for Work Attitude estimation.

FeatureFig. 3. Distribution of the number of times each feature was selected.

R EFERENCES[1] P. D. Harvey, B. R. Greenberg, and M. R. Serper, “The affective
lability scales: development, reliability, and validity,” Journal of clinicalpsychology, vol. 45, No. 5, pp. 786–793, 1989.
[2] V. P. Cornet, and R. J. Holden, “Systematic review of smartphonebased passive sensing for health and wellbeing,” Journal of Biomedical
Informatics, vol. 77, pp. 120–132, 2018.
[3] C. Amenomori, T. Mizumoto, H. Suwa, Y. Arakawa, and K. Yasumoto,“A Method for Simplified HRQOL Measurement by Smart Devices,”
Wireless Mobile Communication and Healthcare, vol. 247, pp. 91–98,
2018.
[4] N. Jaques, S. Taylor, A. Azaria, A. Ghandeharioun, A. Sano and etal., “Predicting students’ happiness from physiology, phone, mobility,and behavioral data,” Int Conf Affect Comput Intell Interact Workshops,
2015.
[5] S. Sonnentag, and C. Fritz, “The Recovery Experience Questionnaire:
Development and Validation of a Measure for Assessing Recuperationand Unwinding From Work,” Journal of Occupational Health Psychology, vol. 12, No. 3, pp. 204–221, 2007.
[6] C. Binnewies, S. Sonnentag, and E. J. Mojza, “Daily performance atwork: feeling recovered in the morning as a predictor of day-level job
performance,” Journal of Organizational Behavior,J. Organiz. Behav, vol.
30, pp. 67–93, 2009.
[7] W. B. Schaufeli, A. Shimazu, J. Hakanen, M. Salanova, and H. De Witte,“An ultra-short measure for work engagement: The UWES-3 validation
across five countries,” European Journal of Psychological Assessment,
vol. 35, pp. 577–591, 2019.