Eyewear 2021 The Forth Workshop on Eyewear Computing –Augmenting Social Situations and Democratizing Tools
Kirill Ragozin
Kai Kunzekirill,kai@kmd.keio.ac.jp
Keio University
Yokohama, Japan

Yuji Uema

J!NS Inc.
Yokohama, Japanyuji-uema@jin-co.com

Teresa Hirzle

Ulm University
Ulm, Germany
teresa.hirzle@uni-ulm.de

Enrico Ruzio

Ulm University
Ulm, Germany
enrico.rukzio@uni-ulm.deBenjamin Tag

University of Melbourne
Melbourne, Australia
benjamin.tag@unimelb.edu.au

Jamie A Wardjamie@jamieward.net
Goldsmiths University of London
London, United Kingdom

Figure 1: Eyewear Prototypes and Use case examples. From facial gesture recognition, over somatosensory feedback in VR tolarge scale social sensing approaches, like the data set recordings during UbiComp 2019 and Theater performance recordings [8,17, 18]

ABSTRACT
Head-worn sensing, especially embedded in augmented and virtual reality (AR/VR) head-mounted displays and smart glasses iscurrently increasingly moving away from niche applications andsmall-scale research prototypes to large-scale consumer adoption(e.g. Oculus Quest 2, Hololens 2, J!NS MEME, Bose Frames). Significant progress in sensing technologies and modalities have lead to a
constant increase of commercially available products and unobtrusive, affordable research prototypes. These recent advances allow
us to extend the Eyewear Community to enable large scale in-situ
studies, as one of the favored research methodologies in Ubiquitous
Computing. One manifestation of this can already be observed inlarge scale dataset recording, eyewear student competitions andprogramming seminars.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
UbiComp-ISWC ’21 Adjunct, September 21–26, 2021, Virtual, USA© 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-8461-2/21/09. . . $15.00
https://doi.org/10.1145/3460418.3479267

In this workshop we focus on supporting these large-scale usesof eyewear computing, discussing lessons learned from early deployment and how to empower the community with better hardware/software prototyping tools as well as the establishment ofopen data sets.
In addition, we will discuss long-term psychological and physical impacts and risks of the technology that become increasingly
important with a wider distribution of devices to consumers. The
proposed workshop will bring together researchers from a wide
range of disciplines, such as mobile and ubiquitous computing, activity recognition eye tracking, optics, human vision and perception,
usability, as well as systems engineering research. This year it will
also bring in researchers from neuroscience, psychology and otherfields that might want to apply or use the research systems. The
workshop is a continuation from 2016/2018/2019 and will focus on
discussing on how to democratizing tools for researchers who want
to apply eyewear computing (sensing/interaction) in their fields,
yet are no wearable computing experts or computer scientists.

CCS CONCEPTS• Human-centered computing → Interaction design; Interaction design process and methods.

KEYWORDS
datasets, eyewear computing, interactions, activity recognition

458

UbiComp-ISWC ’21 Adjunct, September 21–26, 2021, Virtual, USA

Ragozin et al.

ACM Reference Format:
Kirill Ragozin, Kai Kunze, Teresa Hirzle, Benjamin Tag, Yuji Uema, Enrico
Ruzio, and Jamie A Ward. 2021. Eyewear 2021 The Forth Workshop on
Eyewear Computing – Augmenting Social Situations and Democratizing
Tools. In Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021
ACM International Symposium on Wearable Computers (UbiComp-ISWC ’21
Adjunct), September 21–26, 2021, Virtual, USA. ACM, New York, NY, USA,
4 pages. https://doi.org/10.1145/3460418.3479267

1

INTRODUCTION

Our head holds most of our senses - sight, hearing, taste, smell - andthrough nonverbal gestures, like facial affect and eye gaze, it allows
us to communicate with others. Additionally, The face is the main
way in which we express personality and character, and is crucial
to our ability to interact socially. Consequently, head and face arean obvious choice to place sensors and actuators to track activities,
enable new user interactions, and study social interactions [2, 15].
For a long time, head- and face-based wearables have been restricted to special purpose, niche applications. While hearing aids
and mobile headsets are widely accepted as head-worn devices,
users in public spaces often consider novel head-attached sensors
and devices as uncomfortable, irritating, or stigmatizing. Yet, given
a number of recent technological developments, we believe eyewear
computing will soon transition into mainstream technology andbecome a prominent research field. In essence, because the face is
so important to humans, under every day circumstances people
are highly sensitive to ergonomic and social acceptance aspects ofanything placed there. It is telling that in early wearable computing
ergonomic studies the head was not even included [4]. Early egocentric vision devices were rather bulky, expensive, and their battery
life severely limited their usage [7]. Building on existing work inwearable computing, recent commercial egocentric vision devices
and mobile eye trackers, such as Google Glass, SMI ETG, and J!NS
meme, pave the way for a new generation of "Smart Eyewear" thatis light-weight, low-power, convenient to use, and increasingly
looks like ordinary glasses adding to its unobtrusiveness [6, 14].
We motivate the continuation of this workshop series throughsignificant progress in sensing technologies and modalities leading
to a constant increase of commercially available products and unobtrusive, affordable research prototypes. These recent advances
allow to extend the Eyewear Community to enable large scalein-situ studies, as one of the favored research methodologies in
Ubiquitous Computing. One manifestation of this can already beobserved in large scale dataset recording, eyewear student competitions and programming seminars(for example the MobileHCI 2019
competiton 1 ).
Advances in real-life tracking of physical and cognitive activities and states (e.g. reading, walking, detection of fatigue, cognitive load) enable applications on areas like quantified self for themind [16, 19]. And with increasing ubiquity of the technology, newopportunities arise for applications that track social behaviours andinteractions between groups of people in real-world settings [3, 23].
The workshop is a continuation of the first three EyeWear workshops at UbiComp (2016 in Heidelberg, 2018 in Singapore, and 2019
in London). These three workshops focused mostly on enabling
1 https://mobilehci.acm.org/2019/call-for-student-competition/

459

Figure 2: Demonstrators of eyewear interactions using different software prototyping tools. On the left an example formicro-gestures, on the right smooth pursuit systems [20, 22]technology on activity recognition and interaction paradigms. We
will still discuss enabling sensors and actuators, yet in contrast to
2016/2018/2019, we will focus more on open eyewear platform tools(hardware and software know-how) as well as the development andsharing of public datasets.

2

TOPICS

Eyewear Sensing and Actuation Technologies: Driven by use
cases, which sensing modalities are the most interesting to be integrated in Smart Eyewear? What are the important activities andmeasures to focus on (e.g. fatigue detection, concentration tracking)?
Eyewear Interactions: Best practices and explorations for interaction mechanisms and technologies related to smart eyewear
control/input (see also Figure 2) For example eye-gaze based interactions, head movements, and facial micro-gestures.
Open Eyewear Tools and Dataset Best Practices: Introducing to several hardware /software tools used by the community
and the co-organizers[2, 8]. Where does it make sense to integratethem? What are advantages, disadvantages of the tools and where
are they still lacking. In addition we will discuss and share datasets(
Fig ??). A concrete result of the workshop will be a list of softwaretools and open datasets related to eyewear computing.
Novel Use Cases: Which activities are the most useful for application cases and how can we support them from the community.
Potential directions include exploring social interactions in performances (interpersonal synchrony, entrainment) as well as medical
applications (e.g. disease: Alzheimer, Parkinson monitoring) [1, 13].
Impact and Risks of Long-Term Sensing: How can we use
these real life recordings of physical, physiological and cognitive
signals on the head? What are potential negative effects of long term
usage, ranging from perceptual issues regarding the use of wearable
displays (binocular rivalry, vertical and horizontal gaze comfort,
instrument myopia, eye strain, accommodation-convergence mismatch) over potential skin irritations during the deployment ofelectrodes or other sensors necessary to touch the skin to privacy
issues and negative social impacts[11, 12].

Eyewear 2021 The Forth Workshop on Eyewear Computing – Augmenting Social Situations and DemocratizingUbiComp-ISWC
Tools’21 Adjunct, September 21–26, 2021, Virtual, USA

Figure 3: Examples for publicly available data-sets and experimental setup instances utilizing smart eyewear [5, 8–10, 21]3

ACTIVITIESWe propose a one-day workshop on Saturday the 25th September with presentation sessions in the morning, development of
scenarios in the early afternoon, and group discussions on fundamental challenges in the late afternoon. In the following we describe
pre-workshop preparations and the post-workshop follow up.

3.1

Pre-Workshop activities

Prior to the workshop all selected attendees will be asked to fill out
the online form regarding their timezone preferences, platform restrictions and accessibility requirements, professional background,and interest in topics connected to smart eyewear. The call andother information (social network links, discord info) will be posted
on http://eyewear.pro/eywear2021/. We will setup a discord server
for all participants and will start exchanging software tools to workwith during the workshop. This will start about 1-2 months before
the workshop. We are also planning to stream some sensor data to
the participants and let them interact with the eyewear data in realtime.

to sort out the challenges and opportunities for an Open Eyewear
Platform, we will create an affinity diagram analysis of the gatheredideas. Group analysis will start at 15:00 and will end at 15:30.
3.2.5 Group Discussion (15:30-17:30). After the group analysis, wewill have a longer coffee break (15:30-16:00) followed by a discussion round on identified challenges and opportunities (16:00-17:00).
Organizers will actively engage with the audience to stimulate
discussions. We will summarize the key experiences from the workshop and will plan follow up activities (17:00-17:30).

3.3

4
3.2

Workshop activities

3.2.1 Welcome(10:00-11:30). The workshop will start with an introduction given by the workshop organizers in which they willsummarize the workshop’s motivation, goals, and outline (10:0010:15). This will be followed by short introductory presentationsof the workshop participants (in Pecha Kucha style) to get familiar
with each other and to get to know the topics they are working on.
The presentation session will be broken into two parts with a short
break in between. This will allow enough time to discuss different
ideas coming out from the presentations.
3.2.2 Keynote (11:30-12:30). The keynote will be given by Prof.
Thad Starner (Georgia Tech, Google Glass Development Lead).
3.2.3 Demonstration Session. We encourage participants to demonstrate research prototypes related to their submissions. Dependingon the number of demonstrations we will hold a separate session
or do small show-and-tells in small breaks.
3.2.4 Scenario Development (13:00-15:30). After the presentations,grouped participants will develop new scenarios. All participants
will write notes, which will posted on a virtual whiteboard. In order460

Post-Workshop Follow Up

Organizers will document the outcome of the above analysis, andmake this information available to the all participants througha shared Google Drive folder. Participants will be invited to an
existing Slack channel where they can share papers relevant to
the workshop themes. We will start a Data-set directory on http://eyewear.pro/ based on the participants submissions during theworkshop to make it easier to access useful datasets and tools.

ORGANIZERS

Kirill Ragozin is a postdoctoral researcher at Keio University
Graduate School of Media Design. His major research contributions
are in mixed reality thermal interactions. His research interests
include eye tracking and interaction design with smart glasses.
Kai Kunze is a Professor at Keio University Graduate School ofMedia Design. His major research contributions are in pervasive
computing with a focus on augmenting human abilities.
Teresa Hirzle is a fourth-year Ph.D. student at Ulm University.
Her research interests lie in analysing the impact of head-worn technology (in particular VR) on user comfort and developing suitablemeasurement tools thereof.
Benjamin Tag is a postdoctoral researcher and associate lecturer at the School of Computing and Information Systems at the
University of Melbourne. His research focuses on the conceptualisation of digital emotion regulation, and investigation of humancognition using biometric sensors and psychological test methods.
Yuji Uema is a researcher at JINS Inc., where he develops smart
eyewear and conducts feasibility studies with special focus on HCI,
education and medical application. His Ph.D. research at The University of Tokyo includes the analysis and estimation of cognitive
load based on eye blinks and eye movement.

UbiComp-ISWC ’21 Adjunct, September 21–26, 2021, Virtual, USA

Ragozin et al.

Enrico Rukzio is a Professor of Media Informatics at Ulm University. His research focuses on mobile and wearable interaction,
computerized eyewear and automotive user interfaces
Jamie A Ward is a lecturer at Goldsmiths, University of London.
He works on wearable computing, with contributions to topics likesocial neuroscience, activity recognition, performance evaluation,
and applications to real-world problems in health, industry, andthe arts.

REFERENCES[1] Oliver Amft, Jesus Favela, Stephen Intille, Micro Musolesi, and Vassilis Kostakos.
2020. Personalized Pervasive Health. IEEE Annals of the History of Computing 19,
03 (2020), 11–13.
[2] Oliver Amft, Florian Wahl, Shoya Ishimaru, and Kai Kunze. 2015. Making regular
eyeglasses smart. IEEE Pervasive Computing 14, 3 (2015), 32–43.
[3] Mihai Bâce, Sander Staal, and Andreas Bulling. 2020. How far are we fromquantifying visual attention in mobile HCI? IEEE Pervasive Computing 19, 2(2020), 46–55.
[4] Kerry Bodine and Francine Gemperle. 2003. Effects of functionality on perceived
comfort of wearables. In ISWC 2003, Vol. 1530. 17–00.
[5] Andreas Bulling, Ozan Cakmakci, Kai Kunze, and James M Rehg. 2016. Eyewear
computing–augmenting the human with head-mounted wearable assistants(Dagstuhl Seminar 16042). In Dagstuhl Reports, Vol. 6. Schloss Dagstuhl-LeibnizZentrum fuer Informatik.
[6] Andreas Bulling and Thorsten O Zander. 2014. Cognition-aware computing.
Pervasive Computing, IEEE 13, 3 (2014), 80–83.
[7] Ozan Cakmakci and Jannick Rolland. 2006. Head-worn displays: a review. Display
Technology, Journal of 2, 3 (2006), 199–216.
[8] George Chernyshov, Kai Kunze, Benjamin Tag, Jamie A Ward, and Yuji Uema.
2019. http://eyewear. pro: an open platform to record and analyze large scale data
sets from smart eyewear. In Adjunct Proceedings of the 2019 ACM International
Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the2019 ACM International Symposium on Wearable Computers. 264–265.
[9] Aman Gupta, Finn L Strivens, Benjamin Tag, Kai Kunze, and Jamie A Ward. 2019.
Blink as you sync: Uncovering eye and nod synchrony in conversation using
wearable sensing. In Proceedings of the 23rd International Symposium on Wearable
Computers. 66–71.
[10] Antonia Hamilton, Paola Pinti, Davide Paoletti, and Jamie A Ward. 2018. Seeing
into the brain of an actor with mocap and fNIRS. In Proceedings of the 2018 ACMInternational Symposium on Wearable Computers. 216–217.
[11] Teresa Hirzle, Maurice Cordts, Enrico Rukzio, and Andreas Bulling. 2020. A Survey of Digital Eye Strainin Gaze-Based Interactive Systems. In ACM Symposium
on Eye Tracking Research and Applications. 1–12.
[12] Teresa Hirzle, Maurice Cordts, Enrico Rukzio, Jan Gugenheimer, and Andreas
Bulling. 2021. A Critical Assessment of the Use of SSQ as a Measure of General
Discomfort in VR Head-Mounted Displays. In Proceedings of the 2021 ACM CHI
Virtual Conference on Human Factors in Computing Systems-CHI, Yokohama, Japan.
8–13.
[13] Teresa Hirzle, Jan Gugenheimer, Florian Geiselhart, Andreas Bulling, and Enrico
Rukzio. 2019. A design space for gaze interaction on head-mounted displays. In
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems.
1–12.
[14] Reinhold Kliegl and et al. 2006. Tracking the mind during reading. Journal ofexperimental psychology 135, 1 (2006), 12.
[15] Jochen Kuhn, Paul Lukowicz, and al et. 2016. gPhysics - Using Smart Glasses for
Head-Centered, Context-Aware Learning in Physics Experiments. IEEE Transactions on Learning Technologies 9, 4 (2016), 304–317.
[16] Kai Kunze, Katsutoshi Masai, Masahiko Inami, Ömer Sacakli, Marcus Liwicki,
Andreas Dengel, Shoya Ishimaru, and Koichi Kise. 2015. Quantifying reading
habits: counting how many words you read. In UbiComp 2015. ACM, 87–96.
[17] K. Masai, K. Kunze, D. Sakamoto, Y. Sugiura, and M. Sugimoto. 2020. Face
Commands - User-Defined Facial Gestures for Smart Glasses. In 2020 IEEE
International Symposium on Mixed and Augmented Reality (ISMAR). 374–386.
https://doi.org/10.1109/ISMAR50242.2020.00064[18] Kirill Ragozin, Dingding Zheng, George Chernyshov, and Danny Hynds. 2020.
Sophroneo: Fear not. A VR Horror Game with Thermal Feedback and Physiological Signal Loop. In Extended Abstracts of the 2020 CHI Conference on Human
Factors in Computing Systems. 1–6.
[19] Soha Rostaminia, Alexander Lamson, Subhransu Maji, Tauhidur Rahman, andDeepak Ganesan. 2019. W! nce: Unobtrusive sensing of upper facial action units
with eog-based eyewear. Proceedings of the ACM on Interactive, Mobile, Wearable
and Ubiquitous Technologies 3, 1 (2019), 1–26.
[20] Chisa Saito, Katsutoshi Masai, and Maki Sugimoto. 2020. Classification of spontaneous and posed smiles by photo-reflective sensors embedded with smart

461

eyewear. In Proceedings of the Fourteenth International Conference on Tangible,
Embedded, and Embodied Interaction. 45–52.
[21] Moe Sugawa, Taichi Furukawa, George Chernyshov, Danny Hynds, Jiawen Han,
Marcelo Padovani, Dingding Zheng, Karola Marky, Kai Kunze, and Kouta Minamizawa. 2021. Boiling Mind: Amplifying the Audience-Performer Connection
through Sonification and Visualization of Heart and Electrodermal Activities. In
Proceedings of the Fifteenth International Conference on Tangible, Embedded, andEmbodied Interaction. 1–10.
[22] Mélodie Vidal, Andreas Bulling, and Hans Gellersen. 2013. Pursuits: spontaneousinteraction with displays based on smooth pursuit eye movement and moving
targets. In Proceedings of the 2013 ACM international joint conference on Pervasive
and ubiquitous computing. 439–448.
[23] Jamie A Ward, Daniel Richardson, Guido Orgs, Kelly Hunter, and Antonia Hamilton. 2018. Sensing interpersonal synchrony between actors and autistic children
in theatre using wrist-worn accelerometers. In ACM ISWC, Proceedings of. ACM,
148–155.

A

CALL FOR PARTICIPATION

The human face, holding the majority of human senses, provides
versatile information about a person’s cognitive and affective states.
Using head-worn technology, user states, such as reading, walking,detection of fatigue or cognitive load, can be recognized and enable
new application scenarios, such as quantified self for the mind.
Besides, significant progress in sensing technologies and modalities have led to a constant increase in unobtrusive and affordablehead-worn sensing devices, such as smart glasses like Google Glassor J!NS meme. With the resulting increasing ubiquity of the technology, new opportunities arise for applications that track social
behaviours and interactions between groups pf people in real-world
settings.
This workshop aims to identify key factors in large-scale usesof eyewear computing. More precisely, we are going to summarize
lessons learned from early deployment, focus on ways to empower
the community with high-quality hardware and software prototyping tools, and will specifically discuss the establishment of open
source datasets. With the wider distribution of head-worn sensing
technology to the public, long-term impacts of the technology become increasingly important. Therefore, we also welcome topics
that are concerned with physical or psychological aspects of headworn sensing devices. We invite submissions of position papers(2-4 pages in the ACM sigconf format, excluding references) thatcover topics such as, but are not limited to:• Open Eyewear Tools and Datasets• Eyewear sensing and actuation technologies• Smart Eyewear interactions• Application scenarios of head-worn sensing/interaction devices• Impact and Risks of long-term sensing• Smart Eyewear User Experience Designs
Submissions will go through a single-phase review process withat least 2 reviewers. They will be assessed based on their relevance,
originality, and their potential of initiating a fruitful discussion at
the workshop. Note, that position papers are not expected to present
finished research projects. We rather ask for thought-provoking
ideas or initial explorations of a topic. Position papers will be reviewed by two of the workshop organizers. At the workshop, accepted submissions will be presented in a 5-min prerecorded video,
following the Pecha Kucha style. At least one author of an acceptedsubmission must attend the workshop.