37𝑡 ℎ International Symposium on Automation and Robotics in Construction (ISARC 2020)

Investigation of Changes in Eye-Blink Rate by VR Experimentfor Incident Detection at Construction Sites
Shunsuke Hamasaki𝑎 , Mizuki Sugimoto𝑎 , Ryosuke Yajima𝑎 ,Atsushi Yamashita𝑎 , Keĳi Nagatani𝑎 and Hajime Asama𝑎𝑎 School of Engenerring, The University of Tokyo, Japanhamasaki@robot.t.u-tokyo.ac.jp, m-sugimoto@robot.t.u-tokyo.ac.jp, yajima@robot.t.u-tokyo.ac.jp,
yamashita@robot.t.u-tokyo.ac.jp, keĳi@i-con.u-tokyo.ac.jp, asama@robot.t.u-tokyo.ac.jp
Abstract Productivity and safety are in a trade-oﬀ relationship, andthe improvement of the safety management system of construction sites is a pressing issue. Therefore, it is important
to know and analyze information about incidents at real construction sites. However, it is diﬃcult to gather information
about these incidents from workers’ self-reports. Therefore,
in this research, we take an approach to view the workers as
the sensors distributed in construction site and detect theseincidents with the reaction of the workers. Biological signals
such as heart rate, sweating, and muscle activity are the signals generally used to detect an emotional reaction; however,requiring workers to attach electrodes to their body during
work is not suitable. Thus, we focused on blinks since they
can be detected without electrodes attached to the skin. This
study aims to investigate changes that occur in human blinks
during an incident at a construction site. For safety purposes, this study used VR technology to simulate an incident
at a construction site. During the simulation, an image ofthe subject’s eyes was taken by the camera installed in thehead-mounted display. The results of this study suggest thathumans who face an incident have lower blink rates because
they gaze at the cause of the incident.

Figure 1. Concept of the ﬁnal goal

movable ranges too narrowly for safety reasons would adversely aﬀect work eﬃciency. This means developmentof technologies for improving safety without lowering efﬁciency is necessary.
To improve safety without lowering eﬃciency, identifying when danger occurs at the construction site is ofutmost importance. It is also generally said that there aremany “incidents” behind the serious accidents that cause
fatalities. Thus, the motivation for our study is to collect information on these incidents. At many construction
sites, these incidents are currently collected through reports from workers. However, these are uncertain methods
because they are subjective on the worker’s memory.
Keywords There are two possible methods for detecting incidents
Blink; Construction sites; Head-mounted display; VRfrom construction sites. One is by observing the entire
construction site using cameras or LiDAR, and the otheris through the measurement of workers. In this study, we
1 Introduction
focused on the latter method.
A previous study detected incidents through the meaFocusing on current industrial accidents in Japan, thenumber of fatalities in the construction industry accounts surement of workers via heart rate variability. This method
for more than 30% of fatal accidents in all industries. The is also actively conducted, especially for detecting stressnumber of fatalities due to accidents in construction indus- [1], sleepiness and fatigue[2], and the feeling of “traﬃctries has been on a downward trend; however, the decline near-miss” (surprising)[3, 4] among car driver subjects
has stagnated over the last decade. This shows that en- using heart rate and other indicators. Heart rate is related
suring the safety of workers at construction sites is an to a person’s mental state. During tension, the sympathetic
important and urgent theme. On the other hand, improve- nervous system is more dominant resulting to higher heart
ment in labor productivity is also an urgent challenge. In rates. However, it is generally known that the heart rate inmany situations, productivity and safety are in a trade-oﬀ creases not only during mental tension but also during exrelationship. For instance, the areas, where workers and ercise and work. Furthermore, attaching many electrodes
construction machines are active, are often separated for to ﬁeld workers is not considered as a suitable method.
safety reasons in construction sites. However, restricting Therefore, we focused on the element of eye blinks.

1191

37𝑡 ℎ International Symposium on Automation and Robotics in Construction (ISARC 2020)

Blinking is associated with various factors such as prevention of corneal dryness, eye puriﬁcation, eyelid muscle
refreshment, and mental conditions such as anxiety andtension/attention[5]. A study relates blinking to stress indriving and using a personal computer[6, 7]. Blinking
is not only measured by electrodes attached to the skin,
but a method has also been developed for measuring itfrom images, which is suitable for this study since it is a
non-contact procedure.
The aim of this study is to detect incidents that occur
at construction sites. We aim to detect them by measuring blinks of workers when faced with dangerous incidents. Figure 1 shows our concept. As the ﬁrst step, it
is necessary to investigate what kind of changes humanblink shows when human experiences a dangerous incident. This study uses VR technology as a safety precaution
during the simulation of such incidents.

For comparison, we calculated the blink rate, which
represent the number of blinks per minute. We compared
the blink rate both when the incident is occurring and notoccurring.

2 Method

The videos shown to the subject in the VR experimentmust reproduce the incident. We focused on incidents
related to collision of construction machines and humans,
which is a very common accident at construction sites. To
prepare videos that are impressive to the subjects, we usedreverse playback. We carefully brought the construction
machine and the 360-degree camera close to each otherand shot a video of the construction machine moving awayfrom the high-speed 360-degree camera.
We prepared the following four types of incident VR
videos:VR technology was used to simulate the incident.
Speciﬁcally, the subject wore a head-mounted display
while viewing the VR image. A small camera was also
installed in the head-mounted display to observe the stateof blinking during the experiment.
Simple image processing was performed to detect thestate of blinking. Since the camera was ﬁxed to the headmounted display, the position of the eyes can be easily
speciﬁed manually. First, the brightness and contrast wereadjusted for the obtained image, including the eye region.
Next, the edges of the upper and lower eyelids were clearlyvisualized by bilateral smoothing and adaptive binarization. After image processing, the number of blinks wascounted visually. Similarly, the high-speed opening andclosing of the eyelids that occurred during 100-200 ms wasconsidered to be a “blink”. Figure 2 shows an example ofthis simple image processing.

3 VR Experiment
3.1 Outline of Experiment
In this experiment, participants watched the simulation
environment of the construction site through a VR video.
During the simulation, an image of the subject’s eyes was
taken by the camera installed in the head-mounted display.
In order to know the scene where the subject perceived
danger, a gamepad was used and the subject was instructedto press a button during a dangerous scene.
3.2 VR video• Two types of incident videos in which the hydraulicexcavator goes backwards without noticing the presence of the worker and almost comes into contact withthe worker. There are two types of stop positions: (i)long-distance backward and (ii) short-distance backward.
• Two types of incident videos in which the hydraulicexcavator turns unaware of the presence of the worker,
and the tip of the bucket almost comes into contact
with the worker. There are two types of distances
to the excavator: (i) Long-distance turning and (ii)
Short-distance turning.
For that incident VR video, we compared the blink rate
before and after the incident occurs. However, discrepancies may arise when subjects expect that an incident willoccur.
Therefore, we prepared four types of VR videos whereincidents do not occur.

Figure 2. Simple image processing to blink detection• One type of non-incident scene where the hydraulicexcavator is stationary at a suﬃcient distance fromthe worker (stand by).

1192

37𝑡 ℎ International Symposium on Automation and Robotics in Construction (ISARC 2020)

Figure 4. Two cameras installed inside the headmounted display
Figure 3. Sample Scene of a VR Video• One type of non-incident scene where the hydraulic 200 ms. To synchronize the VR video presentation andexcavator turns in the opposite direction from the camera control, an integrated experimental environment
was constructed using Unity.
worker (non-accidental turning).
The inside of the head-mounted display is shown in• One type of non-incident scene where the hydraulic Fig. 4 where the two red circles are the cameras. Since theexcavator moves forward and away from the worker inside of the head-mounted display is suﬃciently dark, the(forward).
camera does not interfere with the subject’s VR watching.
• One type of non-incident scene where the hydraulicexcavator crosses at a distance far enough from the 3.4 Participants
worker (crossing).
Three healthy male volunteers took part in the experiment and supplied informed consent. This study was apIn addition, we prepared the videos so that the subject
proved by the research ethics committee of the University
could see the paper near the subject’s hand. This replicatesof Tokyo.
a scene where the subject is working at the site whilelooking at the paper, such as progress schedule charts. Thesubjects were instructed in advance to look at the paper 4 Results and Discussion
in front of them while paying attention to the construction
The results of the VR experiment are shown in Fig. 5.
machines as if they were in a construction site. TheseTheblue bar stands for the average of subjects’ blink rateeight types of images were about one-minute long eachbeforethe incident occurs, while the red bar stands forand presented to the subjects in random order. An example
theaverageof subjects’ blink rate during the incident. The
scene of the prepared VR videos is shown in Fig. 3.
timingoftheincident was determined from the video once
All these VR videos were shot at the Public Works
theheavyequipment
started to move.
Research Institute. We used Theta V from Richo andThree
results
excludingshort-distance backward reZAXIS120 from Hitachi Construction Machinery as thevealed
that
theblinkrate
duringthe incident tended to360-degree camera and hydraulic excavator, respectively.
be lower than that in the safe state. First, we must consider
the reason why the blink rate did not decrease in Short3.3 Experimental Equipment
distance backward. Interestingly, the blinks that seemed
The head-mounted display used in the experiment was to be voluntary were concentrated in the results of one
Vive COSMOS from HTC. This included headphones for subject which shows the higher blink rate. This is considered to be the reason for the higher average of blink rate
the audio output of the virtual environment.
The two small cameras installed inside the head- in Short-distance backward. Voluntary blinks caused bymounted display were Raspberry Pi Camera Module V2. some reasons unrelated to VR incidents such as eye itching
To capture a wider area inside the head-mounted display, become noise in this experiment. Hence, increasing thethe lens of the cameras was replaced with a 195 属 wide- number of subjects and performing statistical processing
angle lens. Two cameras were controlled by the Raspberry are recommended to potentially eliminate noise unrelated
Pi 3B with a frame rate of 30 fps. This frame rate was se- to VR incidents.
lected since blinks are generally captured between 150 andNext, we discuss the decreasing tendency observed in1193

37𝑡 ℎ International Symposium on Automation and Robotics in Construction (ISARC 2020)(a) Long-distance turning(b) Short-distance turning(c) Long-distance backward(d) Short-distance backward

Figure 5. Blink rate during incident and safe time
the three graphs. The ﬁrst possibility is that the blink rate
gradually decreases while watching a VR video for one
minute. In other words, the blink rate is high until subjects
adjust to the brightness of the VR image, and it decreases as
they adjusted to it. Therefore, we calculated the blink ratein the “stand-by” condition where subject is farthest fromthe incident by dividing one minute into 10 seconds each
(0-10 s, 10-20 s, 20-30 s, 30-40 s, 40-50 s, and 50-60 s).
According to the results of the calculation, the blink rate
under the stand-by condition remained almost constant.
Therefore, this consideration was excluded. From this, it
was considered that the decrease in the blink rate shownin the three graphs is related to the incident.
Another possible cause for the decreasing trend is thatthe subjects gazed at the cause of the incident (excavator).
It is well known that the blink rate of a person who isgazing at something decreases, and this can explain thedecreasing trend properly. The purpose of this study was to
examine changes that occur in human blinks when humans
experience an incident at a construction site. The resultsof this research suggest that humans during an incident
have lower blink rates because they gaze at the cause ofthe incident.
During application of this approach for incident detection at construction sites, it is necessary to make a distinction between blink rate reduction due to incidents fromthose due to gaze irrelevant to incidents. In this case, acombination of saccades and facial expressions is considered eﬀective.

References[1] J. Healey and R. W. Picard: “SmartCar: detecting driver stress”,
Proceedings 15th International Conference on Pattern Recognition.
ICPR-2000, Vol. 4, pp. 218-221, 2000.
[2] M. Patel, S. K. L. Lal, D. Kacanagh and P. Rossiter: “Applying
neural network analysis on heart rate variability data to assess
driver fatigue” Expert Systems with Applications, Vol. 38, Issue 6,
pp. 7235-7242, 2011.
[3] Masayasu Tanaka, Fumiaki Obayashi, Toshiya Arakawa, Shinji
Kondo and Kazuhiro Kozuka: “Detection of Driver’s SurprisedState Based on Blood Pressure and Consideration about Sensitivityof Surprised State”, Proceeding of the 2nd International Conferenceon Intelligent Systems and Image Processing 2014, pp. 56-60, 2014.
[4] Toshiya Arakawa, Masayasu Tanaka, Fumiaki Obayashi, Shinji
Kondo and Kazuhiro Kozuka: “Probability of Driver’s State Detection Based on Systolic Blood Pressure”, Proceeding of 2015
IEEE International Conference on Mechatronics and Automation(ICMA), pp. 2106-2111, 2015.
[5] J. A. Stern, L. C. Walrath and R. Goldstein: “The Endogenous
Eyeblink”, Psychophysiol, Vol. 21, No. 1, pp. 22-33, 1984.
[6] R. Mork, H. K. Falkenberg, K. I. Fostervold and H. M. S. Throud:“Visual and psychological stress during computer work in healthy,
young females-physiological responses” International Archives ofOccupational and Environmental Health, Vol. 91, pp. 811-830,
2018.
[7] C. S. Hsieh and C. C. Tai: “An Improved and Portable Eye-Blink
Duration Detection System to Warn of Driver Fatigue” Instrumentation Science and Technology, Vol. 41, Issue 5, pp. 429-444, 2013.

5 Conclusion
In this study, we investigated the changes that occur inhuman blinks when humans experience an incident at aconstruction site. The VR experiment was performed using a VR video simulating an incident scene. Our results
suggest that humans during an incident have lower blink
rates because they gaze at the cause of the incident. However, increasing the number of subjects is recommended
to further verify the suggestions obtained in this study.

1194