UBICOMP/ISWC ’16 ADJUNCT, SEPTEMBER 12-16, 2016, HEIDELBERG, GERMANY

Eye Blink as an Input Modality for a
Responsive Adaptable Video System
Abstract

Benjamin Tag

Naohisa Ohta

Keio University

Keio University

Yokohama, Kanagawa, Japan

Yokohama, Kanagawa, Japantagbenja@kmd.keio.ac.jp

naohisa@kmd.keio.ac.jp

Junichi Shimizu

Kai Kunze

Keio University

Keio University

Yokohama, Kanagawa, Japan

Yokohama, Kanagawa, Japan

jun.shimi@kmd.keio.ac.jp

kai.kunze@gmail.com

Chi Zhang

Kazunori Sugiura

Keio University

Keio University

Yokohama, Kanagawa, Japan

Yokohama, Kanagawa, Japannora.zhang@kmd.keio.ac.jp

uhyo@kmd.keio.ac.jp

We propose a unique system that allows real-time
adaption of video settings to a viewer’s physical state.
A custom made program toggles between videos
according to the average eye blink frequency of eachviewer. The physical data is harnessed with J!NS MEME
smart glasses that utilize electrooculography (EOG). To
the best of our belief, this is the first adaptable
multimedia system that responds in real time tophysical data and alters technical settings of videocontents.

Author Keywords
Eyewear; Smart Glasses; Activity Recognition; Eye
Blink; Adaptable Video; Psychophysics

ACM Classification Keywords
Human-centered computing~Ubiquitous and mobilecomputing.
Permission to make digital or hard copies of part or all of this workfor personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial
advantage and that copies bear this notice and the full citation on thefirst page. Copyrights for third-party components of this work must
be honored. For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
Ubicomp/ISWC'16 Adjunct , September 12-16, 2016, Heidelberg,
Germany
ACM 978-1-4503-4462-3/16/09.
http://dx.doi.org/10.1145/2968219.2971449

205

Introduction and Motivation
Content producers, technology companies, anddistributers have constantly been looking for new ways
to improve the quality of displayed contents since thebirth of the first moving images. The digitalization ofcontent production and the development ofinterconnected networks have created new distribution
models for multimedia contents. In combination withthe continuous evolution of production equipment andUBICOMP/ISWC ’16 ADJUNCT, SEPTEMBER 12-16, 2016, HEIDELBERG, GERMANY

the simplification of production workflows the demandfor highly personalized, new contents has grown
relentlessly.

Figure 1: J!NS MEME showing
electrodes around the nose.

The system with the highest degree of individualizationis, without any doubt, the human body. Within thissystem, our eyes take the role of gate keeper
responsible for receiving and rejecting visual
information. The endlessly available stream ofinformation is naturally interrupted by our eye blink.
Eye blinking is a necessary process for lubricating andcleaning the eye balls, but also to cut out visual stimuli,e.g. during times of rest and sleep. Blinking usually
happens unintentionally, but can be controlled insituations where a delay might be crucial such as inmoments of stress, fear, or danger [3].
Researchers at the University of Nottingham have beenutilizing blinking for a film experience with their project#Scanners. The sequence of scenes of their movie isinfluenced by the viewer’s eye blink and EEG [5]. Even
though #Scanners allows every viewer to watch theirown individual version of a movie, the production effort
necessary for creating such a complex content
structure is comparably high. Moreover, the constantscene cutting with every single eye blink tends to cause
flustered experiences, especially for users with a rather
high natural blink frequency (BF). This in return has an
impact on the EEG and thus the actual experience.
A similar approach is taken by #AlmostForgot, asmartphone application that utilizes the user’s
heartbeat for adjusting the tempo of a piece of music.
While watching an animated video, the user’s heartbeat
is measured with the phone’s backside camera [1].

206

Our approach tackles the issue of customized content
from a new direction. We are addressing the form
before the substance, i.e. the technical characteristics
before the actual contents (e.g. story) of a video.

Our ApproachTag et al. investigated the impact of different frame
rates (FR) on viewers’ BF. Videos with 120fps, 60fps,
and 30fps were shown to participants while their eye
blink is logged with J!NS MEME glasses (Figure 1). The
obtained results revealed a lower BF for higher FR video[6]. A different study by Kuroki has shown that our EEG
patterns are closest to those of our natural EEG when
we are watching video presented in 240fps. Therefore,
it can be concluded that high frame rate video, whichdisplays motion more similar to reality, causes lessstrain on the visual system of the viewer [4].
Build upon these findings, we produced a video thatwas rendered in 60fps (V1), 30fps (V2), and 15fps (V3)respectively with appropriate retiming applied in order
to avoid temporal distortions. We are currently using
lower frame rates in order to keep the computational
workload small, compared to a 30fps, 60fps, and120fps system. As Figure 2 depicts, all three videos are
simultaneously played, but only one is displayed onscreen for the viewer. A Quartz Composer program
toggles between these three videos. The eye blink data
is input through a custom patch written for thisprogram. Before showing the videos to the viewer, welet participants listen to music for some time while
wearing the J!NS MEME smart glasses. During this time
we log the natural blink, so that we can define every
individual’s BF baseline. The standard human BF isgiven with an average of 17 blinks/minute [2].

SESSION: POSTERS

Figure 3: EOG Original Data, 6
blinks, 2 gazes up.

Figure 2: Schematic system algorithm. Variations in blink ratefrequency over/under certain thresholds trigger video shift.

Figure 4: SMA Adjustment.

Nevertheless, natural characteristics, sleepiness, stress,
monitor work, etc. cause every person to present witha different baseline frequency. After obtaining thebaseline frequency (fb) we input a lower (A) and anupper (B) BF threshold for every user. When BFs aregoing either over or under the respective thresholds theprogram switches to a different video. (A) and (B) arecalculated with the following equations:(1) A = fb - (fb/2)

Figure 5: Comparison of sampledhorizontal and vertical EOG values.

(2) B = fb + (fb/2)

The program constantly processes the input BF andtoggles between V1, V2, and V3 accordingly. In our setup
every presentation starts with the standard frame rate
video V2. If the BF goes below A the program will switch
to video V1. The lower frame rate of V1 will then triggera higher BF. Since all three videos are synchronously
played with the same speed and length, there are noleaps or glitches in the story.

207

To detect the eye blink we use off-the-shelf electrooculography (EOG) smart glasses called J!NS MEME.
Three embedded electrodes can detect the vertical andhorizontal eye movement of the wearer. We implementa robust blink detection algorithm using these twovalues. The vertical and horizontal EOG data of aparticipant blinking six times and looking up two times
can be seen in Figure 3. Due to the eye blinking
relatively affecting the vertical EOG data, it is difficult
to distinguish between an actual blink and a glance up.
Moreover, the original EOG data is rather noisy. For
extracting eye blinking data correctly, we adjust theSMA (Simple Moving Average) to 10 samples for eachdata set (Figure 4). We then sample 0.1 sec. andcalculate the difference. After that, we sum up thesevalues and compare the vertical and horizontal values(Figure 5). Finally, the eye blink can be clearly detectedby setting a threshold for the vertical and horizontal
EOG data (Figure 6).

Implementation and Future Works
The proposed mechanism proposes a new, unique video
system that reacts to the physical state and perceptionof its users. It responds in real-time by adjusting
settings for an individually tailored experience.
Responsive VR Graphics
Especially in virtual reality environments that require
head mounted displays (HMD), it is of crucial
importance to avoid overexertion of the visualapparatus. Blink rate frequency can be used to adjust
the frame rate of the screens in the HMD to guarantee
longer and less debilitating usage. Computational
power can be used more efficiently and with the current
generation of HMDs a wider range and finer tuning offrame rates is also possible.

UBICOMP/ISWC ’16 ADJUNCT, SEPTEMBER 12-16, 2016, HEIDELBERG, GERMANY

VDU-work
The proposed system is a simple but effective addition
to today’s predominantly existing screen-dependentwork-, entertainment- and study environments. Lessstrain and exertion for the eye would mean more
effective and healthier hours of work and pleasure.

Figure 6: Detected eye blinks after
setting threshold.

Medical Applications
Medical Monitors can adapt to the individual
characteristics of each physician and therefore makecamera supported surgeries and diagnoses safer bytaking physical stress off of the doctor.
Cinema/Video Therapy
Physical data can be used to understand the viewer’s
emotional state, and video settings can be adjusted
accordingly. Thereby, certain emotions can beaddressed more effectively, and the occurrence ofoverly stressful situations can be avoided. We
furthermore believe that an adjustable multimedia
therapy is able to help with the treatment of stress andburnout related conditions by fostering meditation in
more relaxing environments.
Prospectively, the physical data used can be extended
by data obtained from facial skin temperature(cognitive load), galvanic skin response (emotionalresponse), and electrical brain activity (e.g. attention).
This will allow researchers to draw more complex
pictures of the users’ physical and emotional state.

Acknowledgements
We would like to thank all our supporters and advisorsthat helped with the development of this system.

208

References1.

https://almostforgot.me/m/.

2.

Anna Rita Bentivoglio, Susan B. Bressman,Emanuele Cassetta, Donatella Carretta, PietroTonali, and Alberto Albanese. 1997. Analysis ofBlink Rate Patterns in Normal Subjects. MovementDisorders, Vol. 12, No. 6: 1028-1034

3.

Gabriel S. Dichter, Andrew J. Tomarken, and Brian
R. Baucom. 2002. Startle modulation before,
during and after exposure to emotional stimuli.
International Journal of Psychophysiology, 43:
191-196.

4.

Yoshihiko Kuroki, Haruo Takahashi, Masahiro
Kusakabe, and Ken-ichi Yamakoshi. 2014. Effects ofMotion Image Stimuli with Normal and High FrameRates on Electroencephalographic Spectra. Journalof the Society for Information Display 22, 4: 177–227.

5.

Matthew Pike, Richard Ramchurn, Steve Benford,and Max L. Wilson. 2016. #Scanners: Exploring theControl of Adaptive Films using Brain-Computer
Interaction. In Proceedings of the 2016 CHI
Conference on Human Factors in Computing
Systems (CHI '16). ACM, New York, NY, USA,5385-5396. DOI=http://dx.doi.org/10.1145/2858036.2858276

6.

Benjamin Tag, Junichi Shimizu, Chi Zhang, Kai
Kunze, Naohisa Ohta, and Kazunori Sugiura. 2016.
In the Eye of the Beholder: The Impact of Frame
Rate on Human Eye Blink. In Proceedings of the2016 CHI Conference Extended Abstracts on
Human Factors in Computing Systems (CHI EA'16). ACM, New York, NY, USA, 2321-2327.
DOI=http://dx.doi.org/10.1145/2851581.2892449