JINSense: Repurposing Electrooculography Sensors on Smart
Glass for Midair Gesture and Context Sensing
Hui-Shyong Yeo

Juyoung Lee

Woontack Woo

hsy@st-andrews.ac.uk
University of St Andrews
St Andrews, UK

ejuyoung@kaist.ac.kr
KAIST
Daejeon, Republic of Korea

wwoo@kaist.ac.kr
KAIST
Daejeon, Republic of Korea

Hideki Koike

Aaron Quigley

Kai Kunzekoike@c.titech.ac.jp
Tokyo Institute of Technology
Tokyo, Japana.quigley@unsw.edu.au
University of New South Wales
Sydney, Australia

kai@kmd.keio.ac.jp
Keio University
Tokyo, Japan

ABSTRACTIn this work, we explore a new sensing technique for smart eyewear
equipped with Electrooculography (EOG) sensors. We repurpose
the EOG sensors embedded in a JINS MEME smart eyewear, originally designed to detect eye movement, to detect midair hand
gestures. We also explore the potential of sensing human proximity,
rubbing action and to diferentiate materials and objects using thissensor. This new found sensing capabilities enable a various typesof novel input and interaction scenarios for such wearable eyewear
device, whether it is worn on body or resting on a desk.

Figure 1: JINS MEME EOG electrodes placement on the nose
pad and nose bridge. This placement makes the it non obvious and the device looks like a normal pair of glass.

CCS CONCEPTS• Human-centered computing → Gestural input.

KEYWORDS
Smart eyewear, Electrooculography, Midair gesture, Context sensing
ACM Reference Format:
Hui-Shyong Yeo, Juyoung Lee, Woontack Woo, Hideki Koike, Aaron Quigley,
and Kai Kunze. 2021. JINSense: Repurposing Electrooculography Sensors on
Smart Glass for Midair Gesture and Context Sensing. In CHI Conference on
Human Factors in Computing Systems Extended Abstracts (CHI ’21 Extended
Abstracts), May 8–13, 2021, Yokohama, Japan. ACM, New York, NY, USA,
6 pages. https://doi.org/10.1145/3411763.3451741

1

INTRODUCTION

Wearable computing is becoming pervasive in our life with theproliferation of wearable devices such as smartwatches, ftness
trackers, and smart eyewear. In particular, smart eyewear such as
the Google Glass [5], JINS MEME [8–10], North Focals [13], Snap’sSpectacles are becoming popular. It is also rumored that both Apple and Facebook are working on Augmented Reality (AR) glasses.
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor proft or commercial advantage and that copies bear this notice and the full citationon the frst page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan© 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-8095-9/21/05. . . $15.00
https://doi.org/10.1145/3411763.3451741

These eyewear devices normally include various sensors, including
inertial measurement unit (IMU), microphone and optionally camera(s) for sensing and tracking the context, environment and hand,under various real-world conditions. Some of these eyewears arealso equipped with output capabilities such as display or speakerfor immersive interaction in AR and VR environment.
However, due to privacy concern [4], people are wary of eyewear
with obvious form factor, especially those equipped with camera(s),such as the Google Glass, Snap’s Spectacles or Microsoft HoloLens.
Therefore, there is another trend of smart eyewear with subtle
design, where it looks like a pair or normal glass, yet it can still
provide basic but useful functionality, such as the North Focals(minimal display for notifcation) or JINS MEME (no display).
For example, JINS MEME is a smart eyewear that looks just likea basic pair of glass [9, 10] (Figure 1). Yet, it has Electrooculography(EOG) electrodes that are built into the nose pad and nose bridge,
which is not obvious at all. The EOG sensors on JINS MEME are
used for sensing signal related to human eye, such as eye motion,eye movement [3, 15] and eye blink [16]. On a higher level, it can bealso used for tracking alertness [17], cognitive and social interaction
assessments [2], activity recognition [7, 19], fatigue [17], readingwords [6], among the others.
Recent work shown that it is possible to repurpose the EOG
sensors on the eyewear for facial action detection [14] and nose
touching interaction [11]. These research inspire our work here.
However, previous work are contact-based sensing where the electrodes touch the skin. Here, in JINSense, we try to achieve noncontact based sensing. In particular we are interested in (i) midair
hand gesture sensing (ii) human proximity sensing and (iii) materialdiferentiation.

CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan

Yeo et al.

Figure 2: The raw EOG sensor signals show unique pattern when diferent hand gestures are performed. This fgure shows thegesture left, right, up and down.

Figure 3: This fgure shows the EOG signal for gesture hover and pull, push + pull quickly, rotate clockwise and counterclockwise.

Figure 4: This fgure shows the EOG signal for gesture palm wiggling, fnger alternating up down and palm alternating up
down.

2

RELATED WORKIn related work, midair gesture are typically detected using camera, microphone, capacitive sensor, infrared sensor or even radio
frequency (e.g., radar sensor). To the best of our knowledge, EOG
sensor has not been explored for midair gesture detection.

3

SENSOR BACKGROUND

Our eyes behave as dipoles with a constant electrical potential withthe cornea acting as positive and the retina acting as the negativepole. The magnitude of this corneoretinal potential (CRP) is in therange of 0.4mV to 1.0mV [1]. If we move our eyes also the dipole
and the electrical potential moves. This movement can be captured
using electrodes taped around the eye (usually measuring the potential change left/right for horizontal and up/down for verticalmovement). This procedure is called Electrooculogram.
In the JINS MEME hardware, there are three EOG electrodes,
one placed on the nose bridge and one each placed on the left
and right nose pad. We can measure potential �� as the potential

between electrodes left and bridge (Figure 1), and �� as the potential
between electrodes right and bridge [18]. Then we can calculate��������� = −(�� + �� )/2 as is the up-down eye movement and�ℎ��������� = �� − �� as the side-to-side eye movement, assumingthat eye balls move in the same direction.
Nonetheless, the equations above only apply to eye related signal
when the device is worn as usual, where the EOG electrodes directly
touch the user skin, and therefore the extracted raw data consists ofmainly eye motion signal. However, when the user is not wearing
the device, such as when being left on a desk or hang in the front
pocket, the electrodes work in a way that is similar to electric feld
sensing. It senses the disturbance in electric potential diferencein the surrounding air. These disturbances can be caused by hand
gesturing nearby the sensor, or human body moving nearby thesensor. We can exploit this phenomenon and repurpose the sensor
for sensing new modalities such as hand gesture detection, humanproximity sensing and material diferentiation. Example signals forthese gestures are shown in Figure 2, 3 and 4. From these fgures,
we can observe the diference between signals of diferent gestures.

JINSense: Repurposing Electrooculography Sensors on Smart Glass for Midair Gesture and Context Sensing

4

IMPLEMENTATION

To explore whether we can recognize midair gestures using thisbuilt-in EOG sensor on the JINS MEME device, we developed a prototype system that can collect, train, and classify diferent gesturesfrom the signals. In this system, we stream raw data from the JINSMEME device over Bluetooth to a remote laptop for real-time signal
processing and machine learning classifcation. We implemented
the software in Python with libraries. In particular, we used Pygame
for the user interface, SciPy for signal processing, and Scikit-learn
for recognizing midair gestures with machine learning classifers.
Initially, we took a simple approach using heuristic rules as a
frst attempt at achieving real-time gesture recognition without
relying on machine learning. This approach calculates the peakand valley of the signal with predefned thresholds and classifes
the gestures. However, the accuracy was far from usable, hence weproceed to employ machine learning approach.
Next, for the machine learning approach, our signal processing
pipeline segments the data into one-second windows and we use 4
signals from the JINS MEME: EOG left (�� ), right (�� ), horizontal(�ℎ��������� ) and vertical (��������� ). Note that the horizontal andvertical of the EOG signals are derived from the combination ofeach side and bridge electrode, but horizontal and vertical signals
are calculated by the left and right signals. We collected fve trials
per gesture for training and for the online test, we ran the classifcation every 30ms. We checked the maximum absolute valuesof �� and �� with threshold to distinguish gesture trials and used
Random Decision Forest to classify gestures, as it was known tobe fast and has low memory footprint [12]. We perform two typesof classifcation on the data; one for classifying whether input has
occurred and the other one for classifying the type of input (e.g.,
gesture left, right or horizontal).
We extracted 50 features in total from the four EOG signals, including 9 individual features from each signal and 14 combination
features. These are common statistical features used when classifying time series data. The features are (i) deviation of FFT, (ii) root
mean square, (iii) percentage of positive and (iv) negative values,(v) number of positive and (vi) negative peaks of original signal
and (vii) root mean square, (viii) percentage of positive and (ix)negative values of the frst diference between subsequent valuesof original signal. As the signal strength generated from gestures
are not so signifcant, we also added combination features which
are derived from two diferent sensor signals. For the combination
features, we picked aggregation of root mean square values fromtwo combinations left/right and horizontal/vertical electrodes, andwe chose Pearson correlation coefcient value and the maximum
cross-correlation value from all possible combinations of signals.

4.1

CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan

Figure 5: A demonstration of real-time midair gesture recognition using machine learning classifer. The JINS MEME
smartglass is resting on a desk. The user performed a swipe
gesture and is correctly detected as a vertical swipe.

Figure 6: When the smart glass in placed in front pocket or
collar, it becomes a gesture sensor to detect hand gestureson the go. In this case, the user swiped down using his hand,
and the gesture is detected as vertical.

Gesture Recognition

Although we have not evaluated the system accuracy, our initialtest shows promising result. It could recognize left/right gesturesrobustly, but not for up/down gestures. Therefore, we group bothup/down gestures into a single vertical gesture (ignore direction). In
total, we can robustly detect left, right and horizontal gestures (Figure 5), which is enough for basic interaction. For example, Google
Soli radar in the Pixel 4 smartphone could only detect left, rightand push gesture, yet it is enough for basic interaction.

Based on our observation, other hand motions such as push, pull,rotate, wiggle also generate unique signal, as shown in Figure 3 and4. We believe these gestures can be robustly detected with bettermachine learning technique that can extract better features, suchas deep neural network, which we left for future work.

CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan

Yeo et al.

Figure 7: This fgure shows the EOG signal (left) when an empty plastic bottle is moving on top of the sensor and (right) whena plastic bottle flled with water is moving on top of the sensor. The EOG signal clearly shows diference between the two
conditions and the latter has a strong signal.

Figure 8: This fgure shows the EOG signal (left) when a plastic bottle is moving on top of the sensor and (right) when theplastic bottle is rubbed with cloth from a long distance.

4.2

Proximity Sensing And Material
Diferentiation

In addition, our preliminary test also shows that it can potentially
support human proximity sensing and material diferentiation [20],as shown in Figure 7, 8 and video fgure, although we have notthoroughly tested it. Our visualization clearly shows that the signal
is unique and changing according to human proximity and diferent
material moving around the sensor. Please refer to the video fgure
for demonstration. We are still in the process to collect more data
and to train a robust recognizer to validate these claims.
In one example (Figure 7), an empty plastic bottle will generatesignal with diferent amplitude when moved around the sensor,
when compared to the same plastic bottle flled with water. Interestingly, the empty bottle causes the sensor to pickup a stronger signal
with higher amplitude. With this, it might be possible to detect thewater level of a container, or dryness of an object.
In another example (Figure 8), rubbing an empty plastic bottle
from a long distance can still be detected by the EOG sensor (upto 1 meter in our test). The EOG signal shows unique pattern thatcan be leveraged for human action or activity recognition. We
hypothesize this may be related to triboelectric efect or static
electricity generated when rubbing cloth on plastic bottle.
In the video fgure, we also show how diferent object and material when moved above the EOG sensor will cause the sensor
to pickup diferent signal, which can be potentially used to differentiate object and material. In addition, human body movingaround the eyewear also caused the sensor to pick up some signal,
albeit rather weak. Again, more research and experiment have to
be conducted to validate these claims.

5

APPLICATIONS AND USE CASES

While our approach may be limited to non-wearing situation only at
the moment, we suggest a few compelling use cases and scenarios:
Interaction with IoT devices – Before going to bed, a person
who wear glass usually removes the glass and put it on a bed-side
desk. When she wakes up, she can perform diferent midair hand
gestures to interact with devices, such as swipe to dismiss alarm,
push to read out emails, wiggle to read out current time. In thisJINSense: Repurposing Electrooculography Sensors on Smart Glass for Midair Gesture and Context Sensing

scenario, users are not required pick up the phone and perform
precise touch input, especially when the user is still sleepy.
Interaction while user is on-the-move -– When a user fold
her sunglasses and put in the front pocket or collar (Figure 6), theeyewear turns into a front-facing gesture sensor for on-the-moveinteraction using simple hand motion. Users can perform a swipe
gesture to dismiss notifcation, or to skip to next song.
Interaction with eyewear when it is docked — The eyewear
can be left on an ofce desk or docked in car when not needed.
Then it becomes a powerful gesture sensor for desktop environmentor car infotainment input controller. Because the eyewear also
supports proximity sensing, it can intelligently turn of a phone ormonitor display to save battery when no human presence detected,
similar to Google Pixel 4 presence sensing using radar.
Multi-purposes smart sensing -– Since the signals show difference when diferent materials are moved around the sensor, thesystem can be potentially used for material recognition, water level
detection and dryness detection.

6

LIMITATIONS AND FUTURE WORK

Currently, we have not evaluated the system. In future work, we
aim to evaluate the system by technical evaluation and user study.
While the user is wearing the device, because the electrodes aretouching the skin directly, the EOG signal from eye motion and skinmotion will be the primary source of signal and it is overshadowing
the midair signal, making it difcult to extract useful features causedby midair motion. Thus our current sensing technique would notwork while user is wearing the device. Potential solution could be
multiplexing the sensor. We are also exploring ways to insulate theelectrodes or to extend the area of electrodes to the outer frame of
the glass using conductive material.
For future versions of the glasses it is possible to only focus on
the electrodes part without the glass frame. The sensing apparatus
can then be very small in size and therefore it can be easily attached
on any device such as smartwatch, smartphone, smart speaker orhome assistant (e.g., Google Home) to support midair gestures. It
can be also attached on diferent body parts or apparels (e.g., button,belt), further enabling compelling use cases. The API makes it easy
to prototype interaction and this may be a useful toolkit for the
HCI and UbiComp research community as a whole.
Finally, we are exploring more compelling use cases that are
uniquely enable by our sensing technique. This includes touchlessinteraction, discreet interaction and context-aware interaction.

7

CONCLUSION

In this work, we have explored new ways to exploit and repurpose
existing EOG sensors on smart eyewear to achieve new sensing
capabilities, such as midair gesture detection, human proximity
sensing and material diferentiation. Our preliminary explorations
show that the technique is indeed working for gesture detection,and the captured EOG signals look promising for human proximity
sensing and material diferentiation. We believe this sensing technique will be useful because there is no need to add new sensors
to a device such as smart eyewear, but instead it is possible reuse
existing sensors for new capabilities.

CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, JapanACKNOWLEDGMENTS
This work was partly supported by JSPS Kakenhi Kiban B Grant No.
18H03278 and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea
government (MSIT) (No.2019-0-01270, WISE AR UI/UX Platform
Development for Smartglasses).

REFERENCES[1] Andreas Bulling, Daniel Roggen, and Gerhard Tröster. 2009. Wearable EOG
Goggles: Eye-Based Interaction in Everyday Environments. In CHI ’09 Extended
Abstracts on Human Factors in Computing Systems (Boston, MA, USA) (CHI EA’09). Association for Computing Machinery, New York, NY, USA, 3259–3264.
https://doi.org/10.1145/1520340.1520468[2] George Chernyshov, Kirill Ragozin, Benjamin Tag, and Kai Kunze. 2019. EOG
Glasses: An Eyewear Platform for Cognitive and Social Interaction Assessments inthe Wild. In Proceedings of the 21st International Conference on Human-Computer
Interaction with Mobile Devices and Services (Taipei, Taiwan) (MobileHCI ’19).
Association for Computing Machinery, New York, NY, USA, Article 67, 5 pages.
https://doi.org/10.1145/3338286.3344418[3] Murtaza Dhuliawala, Juyoung Lee, Junichi Shimizu, Andreas Bulling, Kai Kunze,
Thad Starner, and Woontack Woo. 2016. Smooth Eye Movement Interaction
Using EOG Glasses. In Proceedings of the 18th ACM International Conference on
Multimodal Interaction (Tokyo, Japan) (ICMI ’16). Association for Computing Machinery, New York, NY, USA, 307–311. https://doi.org/10.1145/2993148.2993181[4] Google. 2019. Explorers. Retrieved December 26, 2019 from https://sites.google.
com/site/glasscomms/glass-explorers.
[5] Google. 2019. Glass. Retrieved December 26, 2019 from https://www.google.
com/glass/start/.
[6] Shoya Ishimaru, Kai Kunze, Koichi Kise, and Andreas Dengel. 2016. The Wordometer 2.0: Estimating the Number of Words You Read in Real Life Using
Commercial EOG Glasses. In Proceedings of the 2016 ACM International Joint
Conference on Pervasive and Ubiquitous Computing: Adjunct (Heidelberg, Germany) (UbiComp ’16). Association for Computing Machinery, New York, NY,
USA, 293–296. https://doi.org/10.1145/2968219.2971398[7] Shoya Ishimaru, Kai Kunze, Yuji Uema, Koichi Kise, Masahiko Inami, and Katsuma
Tanaka. 2014. Smarter Eyewear: Using Commercial EOG Glasses for Activity
Recognition. In Proceedings of the 2014 ACM International Joint Conference on
Pervasive and Ubiquitous Computing: Adjunct Publication (Seattle, Washington)(UbiComp ’14 Adjunct). Association for Computing Machinery, New York, NY,
USA, 239–242. https://doi.org/10.1145/2638728.2638795
Retrieved December 26, 2019 from https://jins[8] JINS. 2019. JINS MEME.
meme.com/en/.
[9] Kai Kunze, Kazutaka Inoue, Katsutoshi Masai, Yuji Uema, Sean Shao-An Tsai,
Shoya Ishimaru, Katsuma Tanaka, Koichi Kise, and Masahiko Inami. 2015. MEME:
Smart Glasses to Promote Healthy Habits for Knowledge Workers. In ACM
SIGGRAPH 2015 Emerging Technologies (Los Angeles, California) (SIGGRAPH ’15).
Association for Computing Machinery, New York, NY, USA, Article 17, 1 pages.
https://doi.org/10.1145/2782782.2792491[10] Kai Kunze, Katsuma Tanaka, Shoya Ishimaru, Yuji Uema, Koichi Kise, andMasahiko Inami. 2015. MEME: Eye Wear Computing to Explore Human Behavior. In Adjunct Proceedings of the 2015 ACM International Joint Conference
on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers (Osaka, Japan) (UbiComp/ISWC’15
Adjunct). Association for Computing Machinery, New York, NY, USA, 361–363.
https://doi.org/10.1145/2800835.2800900[11] Juyoung Lee, Hui-Shyong Yeo, Murtaza Dhuliawala, Jedidiah Akano, Junichi
Shimizu, Thad Starner, Aaron Quigley, Woontack Woo, and Kai Kunze. 2017.
Itchy Nose: Discreet Gesture Interaction Using EOG Sensors in Smart Eyewear.
In Proceedings of the 2017 ACM International Symposium on Wearable Computers(Maui, Hawaii) (ISWC ’17). Association for Computing Machinery, New York, NY,
USA, 94–97. https://doi.org/10.1145/3123021.3123060[12] Jaime Lien, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten
Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous
Gesture Sensing with Millimeter Wave Radar. ACM Trans. Graph. 35, 4, Article
142 (July 2016), 19 pages. https://doi.org/10.1145/2897824.2925953[13] North. 2019. Focals. Retrieved December 26, 2019 from https://www.bynorth.
com/focals.
[14] Soha Rostaminia, Alexander Lamson, Subhransu Maji, Tauhidur Rahman, andDeepak Ganesan. 2019. W!NCE: Unobtrusive Sensing of Upper Facial Action
Units with EOG-Based Eyewear. Proc. ACM Interact. Mob. Wearable UbiquitousTechnol. 3, 1, Article 23 (March 2019), 26 pages. https://doi.org/10.1145/3314410[15] Junichi Shimizu, Juyoung Lee, Murtaza Dhuliawala, Andreas Bulling, Thad
Starner, Woontack Woo, and Kai Kunze. 2016. Solar System: Smooth Pursuit
Interactions Using EOG Glasses. In Proceedings of the 2016 ACM International

CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan

Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (Heidelberg,
Germany) (UbiComp ’16). Association for Computing Machinery, New York, NY,
USA, 369–372. https://doi.org/10.1145/2968219.2971376[16] Benjamin Tag, Junichi Shimizu, Chi Zhang, Naohisa Ohta, Kai Kunze, andKazunori Sugiura. 2016. Eye Blink as an Input Modality for a Responsive Adaptable Video System. In Proceedings of the 2016 ACM International Joint Conference
on Pervasive and Ubiquitous Computing: Adjunct (Heidelberg, Germany) (UbiComp ’16). Association for Computing Machinery, New York, NY, USA, 205–208.
https://doi.org/10.1145/2968219.2971449[17] Benjamin Tag, Andrew W. Vargo, Aman Gupta, George Chernyshov, Kai Kunze,
and Tilman Dingler. 2019. Continuous Alertness Assessments: Using EOG Glasses
to Unobtrusively Monitor Fatigue Levels In-The-Wild. In Proceedings of the 2019
CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk)(CHI ’19). Association for Computing Machinery, New York, NY, USA, Article
464, 12 pages. https://doi.org/10.1145/3290605.3300694[18] Yuji Uema and Kazutaka Inoue. 2017. JINS MEME Algorithm for Estimation andTracking of Concentration of Users. In Proceedings of the 2017 ACM International

Yeo et al.

Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the2017 ACM International Symposium on Wearable Computers (Maui, Hawaii) (UbiComp ’17). Association for Computing Machinery, New York, NY, USA, 297–300.
https://doi.org/10.1145/3123024.3123189[19] Yuji Uema, Takashi Ohnishi, and Kazutaka Inoue. 2018. Measuring Study Activity
across High School Students Using Commercial EOG Glasses. In Proceedings ofthe 2018 ACM International Joint Conference and 2018 International Symposium
on Pervasive and Ubiquitous Computing and Wearable Computers (Singapore,
Singapore) (UbiComp ’18). Association for Computing Machinery, New York, NY,
USA, 275–278. https://doi.org/10.1145/3267305.3267670[20] Hui-Shyong Yeo, Gergely Flamich, Patrick Schrempf, David Harris-Birtill, andAaron Quigley. 2016. RadarCat: Radar Categorization for Input & Interaction.
In Proceedings of the 29th Annual Symposium on User Interface Software andTechnology (Tokyo, Japan) (UIST ’16). Association for Computing Machinery,
New York, NY, USA, 833–841. https://doi.org/10.1145/2984511.2984515