37th International Symposium on Automation and Robotics in Construction (ISARC 2020)

Incident Detection at Construction Sites
via Heart-Rate and EMG Signal of Facial Muscle
Mizuki Sugimotoa, Shunsuke Hamasakia, Ryosuke Yajimaa, Hiroshi Yamakawaa
Kaoru Takakusakib, Keiji Nagatania, Atsushi Yamashitaa, and Hajime Asamaaa The University of Tokyo, Japanb Asahikawa Medical University, Japan

E-mail:m-sugimoto@robot.t.u-tokyo.ac.jp
yajima@robot.t.u-tokyo.ac.jp
kusaki@asahikawa-med.ac.jp
yamashita@robot.t.u-tokyo.ac.jp

hamasaki@robot.t.u-tokyo.ac.jp
yamakawa@robot.t.u-tokyo.ac.jp
keiji@i-con.u-tokyo.ac.jp
asama@robot.t.u-tokyo.ac.jp

Abstract Fatal accidents occur at construction sites. Incidents involving dangerous situations but not reaching the categoryof fatal accidents also take place. When an incident occurs,
workers typically generate an on-site physiological reaction.
In this paper, an automatic detection system is proposed toautomatically identify incidents by measuring biological signals related to emotions of on-site workers, such as heart rate
and the masseter muscle. In the first stage of this study, some
virtual reality (VR) video-based experiments were conductedwith some wearable sensors to confirm whether the aforementioned biological signals are suitable to detect incidents.
While watching the VR videos, biological signals of the subjects were measured using diﬀerent wearable sensors. The
experimental results corroborated that the proposed sensing
method is suitable to detect construction-site incidents.

Keywords Construction site; Safety; Biological signals; Incident detection

1

Introduction

According to a survey by the Ministry of Health, Labourand Welfare, there has been a slight increase in the numberof occupational accidents in Japan. In particular, more fatal workplace accidents occur in the construction industry
than in other industries [1]. The reasons for this are theshortage of workers caused by the decline in the population due to the low birth rate, and the aging of the on-site
workers. Therefore, the need to establish a safer working
environment is becoming urgent. To this end, we need todetect an “incident”, i.e., a situation in which there is a riskof an accident or other danger, and conduct an analysis of
the cause of the problem.
There are two main methods for detecting incidents, as
shown in Figure 1: detection from the external environment through devices such as a camera, and detection of
hazards perceived by field workers. For the latter method,
biological signals related to physiological responses are886

Figure 1. Two main methods for detecting incidents.
often used to detect and register dangerous situations perceived by the workers at the site. In this study, this method
was used because we finally would visualize the hazards
perceived by the field workers not only for incident detection but also for removing the mental burden of theworkers in the future.
Previous studies related to ours reported a constructionsite incident detection method based on the heart rate [2].
In this study, the subjects’ heart rate was measured using a chest-belt smartwatch, and incidents were detectedusing the measured heart-rate variability. There are alsoprevious studies for traﬃc incidents, where the heart rateof drivers is used to detect incidents[3][4]. The heart
rate varies according to the activity of sympathetic andparasympathetic nerves [5]. This is related to a person’smental state. Remarkably, the sympathetic nervous system is dominant during tension, resulting in a higher heartrate. In other words, the heart rate is higher when theuser feels in danger. However, it is generally known thatthe heart rate increases not only during mental tension
but also during exercise[6]. At construction sites, workers
often carry heavy materials and perform other tasks that
require strenuous exercises. Therefore, the problem of uncertainty about whether an elevated heart rate is causedby exercise or worker s unsafety feelings is expected tooccur.

37th International Symposium on Automation and Robotics in Construction (ISARC 2020)

In this study, to solve the aforementioned problems, weaimed to detect incidents at construction sites by using various biological signals related to physiological responses
in addition to heart rate. To achieve this, we created virtual
reality (VR) videos that simulated incidents that can occur
at construction sites. We also conducted experiments using the VR videos. Subjects were asked to watch the VR
videos while their physiological responses were measured
in terms of heart rate and electromyography (EMG) of themasseter muscle. The detectability of construction-site
incidents using these biological signals was verified.

Figure 2. Outline of the proposed method.

2 Proposed Method
2.1 Biological Signals Used
The final goal of this study was to detect situations where
workers feel in danger in the field through a wearable
sensor. The sensor measures their biological responses
and correlates them with the work content. To achievethis, it is necessary to select the biological signals for
detecting the scene where a worker feels in danger. Typicalbiological signals related to human emotions include theheart rate and the masseter muscle.
The heart rate fluctuates according to sympathetic andparasympathetic, i.e., autonomous, nervous activity [5].
This activity is related to the human psyche and is associated with a higher heart rate owing to the dominance of
sympathetic nerves during tension. In other words, it isbelieved that the heart rate increases when the worker is ina tense state due to potential danger to himself.
Facial muscles, such as the masseter muscle, are known
to be related to human emotions [7]. Therefore, it ispossible to estimate workers’ emotions toward danger bymeasuring the activity of the masseter muscle.
Overall, heart rate and EMG of the masseter muscle can constitute physiological indicators to detect situations where a worker feels danger. In this study, weused these indicators as candidates for the biological response. We conducted indoor VR experiments to test
whether these candidate biological signals can be used todetect construction-site incidents.

x = (h, f )T .

(1)

Using this x as inputs, a Gaussian naive Bayes classifier, which is often employed to classify time series datasuch as those generated in this study, was used to determine whether an incident occurs at a construction site ornot. A Gaussian naive Bayes classifier assumes the independence of each feature and simultaneously estimates theprobability from each feature. The label with the highest probability constitutes the output. The classifier isexpressed according to the following formula:ŷ= argmax[p(y|x)],D∏= argmax[p(y)p(xd |y)],

(2)
(3)

d=1

where ŷ is the output label, p(y|x) is the posterior probability of correct label y given the input vector x, p(y)is the prior probability of correct label y, p(xd |y) is thelikelihood, xd is the feature in the d-th dimension, and D
is the dimension of x.
To use the classifier, we need a unique parameter foreach probability distribution. The optimal value of theeigen-parameters is found by maximum likelihood estimation, with the feature matrix of the training data as X
and the corresponding correct label vector as y. The maximum likelihood function L(X, y) is expressed as follows:

2.2 Detection Method
In this study, heart rate and EMG of the masseter muscle
were used as features, and a Gaussian naive Bayes classifier
was used to detect construction-site incidents, as shown in
Figure 2.
As in previous studies, we used the average heart rate h
at a given time t as a feature of heart rate [2]. Given that the
maximum amplitude is generally used for the evaluationof EMG, the masseter EMG feature in our study was the
maximum amplitude f at a given time t. The feature vector
x used in this study is expressed as follows:

L(X, y) =
=

N
∏
n=1
N
∏
n=1

p(yn )p(xn |yn ),[p(yn )

D∏p(xnd |yn )],

(4)

(5)

d=1

where N is the number of training data, yn is the correctlabel in the n-th training data, xn is the feature vector in
the n-th training data, and xnd is the feature in the d-th
dimension of the n-th data.

887

37th International Symposium on Automation and Robotics in Construction (ISARC 2020)

A Gaussian naive Bayes classifier assumes that the likelihood p(x|y) follows a Gaussian distribution and finds
the optimal value of the intrinsic parameters such that thelikelihood function L(X, y) is maximized for each event.
The Gaussian distribution is represented by the following
equation:p(x|y) = √

1
2πσ 2

exp(−

x−µ
),
2σ 2

(6)

where σ 2 is the distribution of the feature x when the correct label is y and µ is the mean of the feature x when
the correct label is y. A Gaussian naive Bayes classifier estimates the correct label from the input features bycomputing the above equation.
In this study, the aforementioned methods were used toclassify whether incidents occur at a construction site.

3

VR Experience

3.1

Outline of Experience• One type of non-accident scene where a hydraulicexcavator waits at a long enough distance from aworker (standby)

In this study, we conducted experiments in which subjects watched a simulated environment of a construction
site through VR videos and simultaneously obtained biological signals from a wearable sensor. Subjects sat in
a designated position in the laboratory, wore a wearable
sensor for biometric measurements, and watched a VR
video. To understand which scenes subjects felt as dangerous, we gave them a controller and instructed them topress the button on the controller when they felt a situation
as dangerous; we recorded the time when they pressed the
button. This experiment was conducted with the approvalof the Ethics Committee of the University of Tokyo.
3.2

Figure 3. Example of VR image.

VR videosIn the construction industry, personal injuries involving contact between workers and construction equipment
or machines such as automobiles are very common [8].
Therefore, in this study, we prepared VR videos of incident scenes where a subject and an excavator are likely tocome into contact. Specifically, we prepared the following
eight videos.

• One type of non-accident scene where a hydraulicexcavator turns in the opposite direction with respect
to a worker (non-accidental turning)• One type of non-accident scene where a hydraulicexcavator moves forward and away from a worker(forward)• One type of non-accident scene where a hydraulicexcavator crosses at a long enough distance from aworker (crossing)In addition, we prepared videos so that the subject could
see papers near the subject’s hand, simulating that the
subject is working at a site while looking at papers such as
instructions. The subjects were instructed in advance tolook at the papers before them and to pay attention to theconstruction machines in front of them, as if they were in aconstruction site. An example of such prepared VR images
is shown in Figure 3. These VR images were filmed at
the Public Works Research Institute using Ricoh Theta V,
operated by Hitachi Construction Machinery ZAXIS120.

• Two types of incident scenes in which a hydraulic ex- 3.3 Devices
cavator moves backwards unaware of the presence
The wearable sensors used in this experiment are shownof a worker and almost comes into contact within Figure 4. The S&ME DL-310 was used to measure thethe worker, changing the stopping position (longheart rate, as shown in Figure 4a, and the S&M DL-140
distance backward and short-distance backward)
was used to obtain the EMG, as shown in Figure 4b. The• Two types of incident scenes where a hydraulic ex- S&ME’s DL-310 amplifies the R wave of the cardiac signal
cavator turns unaware of the presence of a worker detected by the sensor with a filter amplifier, and outputs
and the tip of the bucket almost comes into contact this pulse. EMG of the masseter muscle was measured
with the worker at diﬀerent distances (long-distance by the S&M’s DL-140 with electrodes attached to thetemporal area of the face. The biological signals acquired
turning and short-distance turning)888

37th International Symposium on Automation and Robotics in Construction (ISARC 2020)(a) S&ME DL-310(b) S&ME DL-140(a) Heart-rate variability in subject A.

(c) S&ME DL-3100

Figure 4. Sensors employed in the experiments.
from these sensors were recorded in an S&ME data logger(DL-3100), as shown in Figure 4c. This S&ME DL-3100
has a sampling frequency of 1000 Hz and 16-bit A/Dconversion resolution and can store measurement data in
its on-board memory.
(b) EMG of the masseter muscle in subject A.
Vive Cosmos was used to present the VR videos. This
Figure 5. Biological signals of subject A.
head-mounted display is also equipped with headphones;
thus, subjects could hear the sound recorded at the time offilming and watch the video.
following equation:vut

3.4 Biological Signals Processing
After applying a bandpass filter in the frequency range
from 0.16 to 500 Hz to the cardiac telegraphic signal in
the amplifier section, the biometric information was transmitted to the receiver and recorded in the device. Simultaneously, a pulse waveform synchronized with the R wave
was generated in the amplifier section and was recorded in
the biometric device concurrently with the cardiac signal.
Given that the synchronized pulse rises at the time positionof the R wave, the R-R interval was calculated from thetime position of the rise of the pulse, and the heart rate wascalculated from the inverse of the pulse. The time at the
midpoint between consecutive R waves was used to record
the heart rate.
In this study, the root mean square (RMS) method wasused, which is commonly used for analyzing an EMG ofthe masseter muscle. Considering the frequency range of
EMGs in previous studies, a 5-Hz high-pass filter was used
to process the EMGs [9]. Given that humans usually chew
in a range from 0.1 seconds to several seconds when theychew, the frame length in this study was set to 100 ms andthe interval between frames was set to 1 ms. Denoting
the value of RMS at sample point n as S(n), we obtain theS(n) =1 ∑ 2
f (n + i),
N i=0
N

(7)

where N is the frame length and f(n+i) is the signal of themasseter muscle at sample point n+i.
3.5 Result
Three male students in their twenties were the subjectsof this study.
As an example of experimental results, the variationsof heart rate and EMG of the masseter muscle for subject
A while he was watching the short-distance turning VR
video are shown in Figure5. Figure5a shows the heart rateof subject A. The horizontal axis represents the time of the
VR video and the vertical axis represents the heart rate.
Figure5b shows the variation of the EMG of the massetermuscle for subject A. The horizontal axis shows the timeof the VR video and the vertical axis shows the amplitude
variation of the EMG of the masseter muscle. The area
in red in both Figure5a and Figure5b is the interval of theincident. Figure5 shows that the heart rate ranged from 73
to 86 bpm during the incident, and there was only a slight889

37th International Symposium on Automation and Robotics in Construction (ISARC 2020)

diﬀerence from the time before the incident occurred. In
contrast, the amplitude of the EMG of the masseter muscle
reached up to 0.38 mV, which is approximately twice aslarge as before the incident occurred. This suggests thepossibility of detecting incidents from the EMG of themasseter muscle.
In this study, among the experimental data of 3 subjects
watching 8 VR videos, 3 subjects watching 4 VR videos(short-distance backward, short-distance turning, standby,and crossing) were used as training data, and 3 subjects
watching the remaining 4 videos (long-distance backward,
long-distance turning, forward, and no-accident turning)were used as evaluation data. Correct labels were assigned for the experimental data as incidents at the time ofthe simulated incident in the VR video and no incidents
otherwise.
The detection of construction-site incidents by the proposed method was evaluated using accuracy and recall.
Accuracy was calculated by comparing the estimation results of the proposed method with the correct labels of thetest data and dividing the number of correct answers by
the total number of test data. Thus, it provides the percentage of correct estimation results. Recall is calculatedby dividing the number of test data correctly detected asincidents by the proposed method by the number of test
data actually labeled as incidents. Thus, it indicates thecompleteness of the incident detection. The equations for
accuracy A and recall R are as follows:

method. Therefore, the recall of the proposed method washigher. The purpose of this study was to detect incidents,
and it is important to detect incidents comprehensively forachieving a safe construction-site environment. Therefore,
it is desirable to have a method with high recall and accuracy. Given that the recall of the heart-rate-only method
is 0, it is assumed, according to such method, that no incidents were detected at all. The proposed method is more
eﬀective than the heart-rate-only method because the recall is higher and more comprehensive, even though theaccuracy of the proposed method is lower than that of theheart-rate-only method. The reason why we could not detect an incident using the heart-rate-only method in theseenvironments may be because heart rate responds after ashort period of time after an incident occurs. Generally,
the heart rate is known to respond about 10 to 20 seconds
after an incident occurs, but in the VR video used in theseexperiments, the video ended within 10 seconds after theincident occurred. It is thought that we could not detect
incidents using the heart-rate-only method. In the future,
we will consider creating a VR movie with more than 20
seconds left after an incident and using it for experiments.

4 Conclusion

In this study, a method for detecting construction-site
incidents through a Gaussian naive Bayesian classifier using heart rate and masseter EMG as features was developed. Indoor experiments in which subjects experienced
the simulated environment of a construction site through
C
D
,
(8)
A =
VR videos were conducted to obtain biological signals.
DT
The performance of the proposed method was evaluated
DC
I
R =
,
(9) using the data obtained from the experiments. A comparDTI
ison between the proposed method and a heart-rate-only
method confirmed that the proposed method was more
where DC is the total number of correct data, DT is the eﬀective in detecting incidents comprehensively. In the
total number of evaluated data, DC
I is the number of correct future, we will aim to establish a method for detecting
data detected as incidents, and DTI is the number of data incidents in which workers and construction machines arelabeled as incidents.
close to come into contact with each other through verifiThe accuracy and recall of the proposed method are cation of experiments using VR videos.
shown in Table 1. In addition, for comparison, the accuracy and recall obtained by the method of estimating References
only the heart rate as a feature are also shown in Table 1.
[1] Ministry of Health, Labour and Welfare: “Fiscal Year 2017
Accuracy was 0.48 for the proposed method and 0.87 forWorkers’ Accident Situation”, https://www.mhlw.go.jp/stf/houdou/
0000209118.html (Access: 7th July, 2019).
the heart-rate-only method. Therefore, the accuracy of theheart-rate-only method was higher. By contrast, recall was [2] Ryuichi Imai, Daisuke Kamiya, Haruka Inoue, Shigenori Tanaka,
0.57 for the proposed method and 0 for the heart-rate-only
Jun Sakurai, Takuya Fujii Nobuya Honda and Makoto Ito: “ReTable 1. Accuracy and recall of the proposed method andheart-rate-only method
MethodProposed Method
Heart-Rate-Only

Accuracy
0.48
0.87

search for Experimenting Various Possibilities Grapsing Near Miss
Incidents and Physical Fatigue Using Smart Watch in Construction
Fields”, Journal of Japan Society of Civil Engineers F3, Vol. 74,
No. 2, pp. 167-177, 2018.
[3] Masayasu Tanaka, Fumiaki Obayashi, Toshiya Arakawa, Shinji
Kondo and Kazuhiro Kozuka: “Detection of Driver’s SurprisedState Based on Blood Pressure and Consideration about Sensitivityof Surprised State”, Proceeding of the 2nd International Conferenceon Intelligent Systems and Image Processing 2014, pp. 56-60, 2014.

Recall0.57
0890

37th International Symposium on Automation and Robotics in Construction (ISARC 2020)[4] Toshiya Arakawa, Masayasu Tanaka, Fumiaki Obayashi, Shinji
Kondo and Kazuhiro Kozuka: “Probability of Driver’s State Detection Based on Systolic Blood Pressure”, Proceeding of 2015
IEEE International Conference on Mechatronics and Automation(ICMA), pp. 2106-2111, 2015.
[5] Julian F. Thayera, Fredrik Åhs, Mats Fredrikson, John J. Sollers III
and Tor D. Wager: “A Meta-Analysis of Heart Rate Variability and
Neuroimaging Studies: Implications for Heart Rate Variability asa Marker of Stress and Health”, Nueroscience and Biobehavioral
Reviews, Vol. 36, No. 2, pp. 747-756, 2012.
[6] Tasuku Sato, Toshihiro Ishiko, Junichiro Aoki, Tatsuo Shimisu andTakashi Maejima: “Exercise Change of Heart Rate, Blood Pressure
and Respiatory Rate in Relation to Sex and Age”, Japanese Journalof Physical Fitness and Sports Medicine, Vol. 26 No. 4 pp. 165-176
1977.
[7] John T. Cacioppo, Richard E. Petty, Mary E. Losch and Hai Sook
Kim: “Electromyographic Activity Over Facial Muscle Regions
CanDiﬀerentiate the Valence and Intensity of Aﬀective Reactions”,
Journal of Psrsonality and Social Psychology, Vol. 50, No. 2,
pp. 260-268, 1986.
[8] Japan Construction Occupational Safety and Health Association:“Fatal Construction Accidents by Type of Construction Work andType of Accident, 2018”, https://www.kensaibou.or.jp/safe_tech/statistics/construction/h30.html (Access: 19th Februally 2020).
[9] Roberto Merletti and Politecnico di Torino: “Standards for Reporting EMG Data”, J Electromyogr Kinesiol, Vol. 9, No. 1, pp. 3-4,
1999.

891