ISSN 2186-7437

NII Shonan Meeting Report

No. 174

Augmented Perception (ver 1.0)
Yuta Itoh, The University of Tokyo, Japan
Kai Kunze, Keio University, Japan
Jason Orlosky, Augusta University, USA & Osaka
University, JapanMay 29–June 1, 2023

National Institute of Informatics
2-1-2 Hitotsubashi, Chiyoda-Ku, Tokyo, Japan

Augmented Perception (ver 1.0)
Organizers:
Yuta Itoh (The University of Tokyo, Japan)
Kai Kunze (Keio University, Japan)
Jason Orlosky (Augusta University, USA & Osaka University, Japan)
May 29–June 1, 2023

1

Background and introduction
In the past years, there have been two successful Shonan Meetings over Augmented Reality (AR) and Human-Computer Interaction (HCI): the first Shonan
Meeting on Perceptual Issues in Augmented Reality (AR) organized by Christian Sandor, Dieter Schmalstieg, and Edward J. Swan II in 2016; and the first
Shonan Meeting on AR in HCI organized by Yuta Itoh, Kai Kunze, Alexander
Plopski, and Christian Sandor in 2018. Building on the success of the previous
meetings held every two years, we propose to organize a Shonan Meeting on
Augmented Perception in 2020.
AR technology extends our perceived real world by providing users with additional information and/or interaction schemes. A typical example of such technology is optical see-through near-eye displays (OST-NED) [Sutherland 1968]
that can overlay a virtual image into the user’s view. As another example, we
can use interactive techniques in combination with computer vision algorithms
to improve human vision, allowing individuals to see in different spectrums or
highlight objects of interest in the world that would otherwise be difficult to
notice or invisible. Moreover, vision is not the only sensory functions that AR
can enhance. Past AR research works explore AR displays for various sensory
modal [Kim 2018] including hearing, taste, smell, and touch. In other words,
AR technology is by nature a technology that changes or extends our perception. With the rapid development of AR technology, we believe in realizing afuture where advanced AR devices virtually enhance our ability in various tasks
by assisting our perception. While AR technology has steadily developed to be
a commodity tool in our lives [Gartners 2019], not many people have realized its
potential in extending human perception and how it could change our future life
[Rekimoto 1999, Papagiannis 2017]. As a natural consequence, we believe that
there is an emerging need to explore augmented perception research in more
depth.
The concept of augmenting perception [Schmidt 2017] is a subtopic of awider concept of extending human intellect proposed in the 1960s [Engelbart
1962]. Assisted by augmented perception devices, the users may be able toprocess more complex data and/or sense subtle or imperceptible informationthat our raw sensory system can handle. One example of augmented perception
technology is the ChromaGlasses [Langlotz 2018] system, which assists color
blindness by enhancing colors perceived by the user. The system analyzes thefirst-person view user and partially modulates the color of the person’s view via
see-through image overlay.
The major challenge in augmented perception is that the system of humanperception and AR displays can be represented as a loop using various sensory models. For example, to provide optimized visual stimuli, the developmentof the ChromaGlasses described above required an understanding of the color
perception of a color-blind individual, augmented reality technology, and cognition. Thus, augmented perception technologies must tackle this loop of humanperception and the real-world environment for given tasks. To investigate thepotential of augmented perception we have included people from various disciplines and backgrounds, ranging from electronics to neuroscience.
This seminar thus aims to rethink how information technology including
AR and HCI could modify and extend our perception. The seminar provides aunique opportunity among excellent researchers and students in AR, HCI, Hu2man Augmentation, and cognitive science from all over the world. Not only will
participants come from diverse cultural backgrounds, but many participants also
have a different and unique research field that will complement the discussions
during the meeting.
With this diverse group of participants, we explore how perception augmentation would affect future society. More specifically, our goal is to establish
future directions of research and narrow down topics for projects that we cancarry out within the next several years. Much like the previous meetings, we expect that participants will build lasting relationships, start new collaborations,and be able to change their way of thinking after listening to the others.

3Overview of the meeting
Overall, the meeting provided a platform for collaboration, knowledge sharing,and exploration of emerging themes and challenges in the fields of perceptual
augmentation, accessibility in XR, and display technologies. Participants engaged in fruitful discussions, shared their expertise, and laid the groundworkfor future research and collaborative efforts in these areas.
The meeting included a ”Super Collabo Session,” which aimed to foster collaboration among participants. The session provided an opportunity for attendees to find potential collaborators, initiate joint research papers, and explore
common themes. Two specific topics were discussed during this session. The
first topic involved augmenting perception to improve abilities and skills. Participants explored the possibilities of using technology to enhance human perception and its potential impact on various fields. The second topic, presented
as a dream scenario, centered around inhibiting muscle activity using HumanComputer Interaction (HCI) technology. The participants considered methods
such as sonification or Electrical Muscle Stimulation (EMS) amplification oferrors to make individuals aware of their muscle activity.
Another significant topic discussed during the meeting was the proposed
workshop at ISMAR (International Symposium on Mixed and Augmented Reality) with a focus on Accessibility for XR (Extended Reality). The workshop
aimed to address the challenges and advancements in making XR technologies
more accessible. Attendees had the opportunity to ask questions and refine thetopics for further discussions throughout the day. The VERA project, emphasizing the experimental environment, was included, and participants were encouraged to share their project ideas for immediate collaboration. In addition,
sessions were organized to explore considerations for starting research related
to accessibility in XR and to discuss XR assistive technologies for individuals
with low vision or blindness, such as an AR app for the visually impaired.
In another workshop session, we delved into the exploration and applications of Generative AI tools and their potential to significantly enhance various
activities, including research, design, storytelling, and support for individuals
with disabilities. Key discussion points were the potential of Generative AI
tools to produce a nuanced perception of the world and their application inimage descriptions. The integration of AI tools, including image recognition,
image captioning, and GPT-based chat systems, can provide real-time support.
These can be particularly useful in aiding people who have trouble expressing
themselves verbally or need additional explanation support. We examined thetransformative impact of AI on design education, suggesting the potential for
design engineers to shift towards roles of quality assurance. The Generative AI
tool DALL-E was mentioned in this context, hinting at the evolving dynamics
of the design industry. The core aspect of communicating with Generative AI
was explored, highlighting that it reacts to a series of words rather than individual concepts. An example was drawn from the current use of Google Glass indetecting symptoms of unconscious persons for paramedics, where written descriptions, understood by GPT, are highly beneficial. The LERF concept was
introduced as a tool to generate richer or automated instructions, especially in
critical scenarios such as emergency situations. This application positions AI asa social equalizer, although it is important to remember that despite its sophistication, AI like GPT is fundamentally performing lexical analysis without actual
4comprehension. Lastly, we acknowledged the current limitations of GenerativeAI, particularly in the realm of physics and spatial reasoning. As this technology evolves, these are areas where future developments and improvements may
emerge.
We also had a focused ”Display Session.” This session delved into more detailed discussions on topics covered earlier in the day. The participants exploredvarious aspects, including identifying killer applications, ideal display characteristics (e.g., form factor), and different use cases. Considerations were made forfactors such as field of view, immersion, hands-free form factor, smartphone versus laptop usage, interaction methods, sensor integration, screen types, and thetrade-offs between slim and bulky devices. The discussions also encompassed
applications related to visual assistance, exploring how XR technologies can beutilized in this context.
Finally, we had a “bowl of questions” session that spanned multiple days.
Each participant submitted a question and led the discussion regarding solutions
to that question. The list of participants and their fields of expertise are listed
below.

5

List of Participants• Gerd Bruder, University of Central Florida
Areas: perception, cognition, 3DUI• Ozan Cakmakci, Google
Areas: Optics, waveguides• Jingyuan Cheng, University of Science and Technology of China
Areas: Smart textile• Henry Duh, Hong Kong Polytechnic University
Research topics: simulator sickness, UX/UI• Andre Hinkenjann, Bonn-Rhein-Sieg University of Applied Sciences
Research areas: (Neural) Graphics, interaction, field visualization (e.g.,magnetic)• Takefumi Hiraki, Cluster Metaverse Lab
Research areas: Projection-based AR, Haptics• Yuichi Hiroi, The University of Tokyo
Research areas: See-through displays, vision augmentation• Yuta Itoh, The University of Tokyo
Research areas: See-through displays, vision augmentation• Daisuke Iwai, Osaka University
Research area: Projection mapping• Jeeeun Kim, Texas &M University
Research areas: fabrications, programed design• Kiyoshi Kiyokawa, Nara Inst. of Sci. & Tech. (NAIST)
Research areas: displays, perception correction• Hideki Koike, Professor of Tokyo Institute of Technology
Research areas: vision-based HCI, AR sports, ski training• Ernst Kruijff, Bonn-Rhein-Sieg University of Applied Sciences
Research areas: human-factors driven design, multisensory 3DUI
• Kai Kunze, Keio University
Research areas: HCI, VR, vision augmentation• Tobias Langlotz, Unversity of Otago
Research areas: HCI, visual augmentation, vision augmentation, guidance• Pedro Lopes, University of Chicago
Research areas: mobile, wearable integrated sensors, physical skill transfer, muscle stimulation• Paul Lukowicz, DFKI and University of Kaiserslautern
Research areas: sensing, human activity recognition, shared perception

6• Päivi Majaranta, Tampere University
Research areas: gaze-based HCI, multimodal HCI• Katsutoshi Masai, Kyushu University
Research areas: Affective computing and interaction, augmented perception, face perception
Additional context: context of facial expression sensed is also important.
• Jason Orlosky, Augusta University and Osaka University
Research areas: XR and learning, XR support to reduce cognitive disparity• Alexander Plopski, TU Graz
Research areas: visual perception in AR displays• Enrico Rukzio, Ulm University
Research areas: mobile, VR interaction, visual alterations• Maki Sugimoto, Keio University
Research areas: human augmentation in VR, sensing technology• Jonathan Sutton, University of Otago
Research areas: see-through displays, visual assistance• Greg Welch, University of Central Florida
Research areas: XR, motion sensing/tracking, physical-virtual• Guanghan Zhao, Tokyo Institute of Technology
Research areas: CV, VR/AR, HCI, Visually Induced Motion Sickness.
• Zhang Qing, The University of Tokyo
Research areas: visual information processing, visual perception programming

Summary of Researcher Topics and Discussions
The following section of slides includes topic introductions from each of theparticipants. These topics were also discussed in the various sessions throughout
the seminar and were used to spur various questions in the “Bowl of Questions”
session.

7

Introduction slides
Shonan Meeting 174

Who are you?

Gerd Bruder
What part of your work is most relevant to the Seminar?

My name is Gerd (/ˈɡɛrd/), my favorite color isyellow, and I live in Florida!
Research Associate Professor at the
University of Central Florida
Focus on perception and cognition in AR/VR;
3D user interfaces
PhD 2011 (WWU) on perceptual illusions andredirection, and Habilitation 2017 (UHH) on
perceptually-inspired user interfaces

What is the most important topic/grand challenge we should tackle?
● AR for augmenting human perception○

Exceed sensory limitations; accessibility re. low vision, low hearing, etc.

● AR for augmenting human cognition○

AI-supported cognition, personal AI, shared cognition, accessibility, …

Ozan Cakmakci
Who are you?

What part of your work is most relevant to the Seminar?

Ozan Cakmakci, Google

Interested in optical design of eyewear displays.
Was part of the Google Glass Optics team.
Currently running an optical design groupfocusing on AR optics.
Google Glass

Spent a summer at Canon R & D, with aninventor of freeform prism optics, in Tokyo,
Japan.

Curved lightguides (SPIE 2019)

Enjoy playing bass guitar in my free time andspend time with my daughter.
6 months left to finish.
To be published by SPIE press.

Fast electromagnetic propagation methods

Holographic see-through pancake optics (Optics Express, 2021)

What is the most important topic/grand challenge we should tackle?
Barriers to consumer adoption?

Jingyuan ChengWho are you?

What part of your work is most relevant to the Seminar?
Whole-body pressure sensing

Jingyuan CHENG (⛬ᩗཎ) jingyuan@ustc.edu.cn
pose

University of Science and Technology of China(https://en.wikipedia.org/wiki/University_of_Science_and_Technology_of_China)
gy
gy_o
y_o
y
y_
_o
_
of_C
ff_Ch
_Chin
_C
iina
na
a))

self
interaction

2002 Bachelor, Applied (nuclear) Physics, China
2007 Ph.D., Physical Electronics, China
2007-2016, Ubiquitous Computing, Germany (9 years in Prof.
Lukowicz’s group )objectinteraction

Since 2016, matrix-wise smart textile and health related
applications,
MultidisciplinaryChina
in the same lab: Textile, Electronics and Algorithms

humaninteraction

What is the most important topic/grand challenge we should tackle?

A small office (1 sensor/cm2) = 105 channels

Yet-to-ready textile industry:

➔
➔
➔
➔

New material for fibres and threads
New textile machines
New textile-electronics interface
Vs: large factories, low profit, very fine work division(the worst enemy for small-scale experiments)

Few examples from existing wearables:

➔
➔
➔
➔

Henry DuhMillions of channels per person
Multi-layers (modalities)
Multi-view from multiple devices
Vs: high wearability, low power&cost, missing algorithms

André
Hinkenjann
Who are you?

What part of your work is most relevant to the Seminar?

* Professor for computer graphics and interactive
environments @ Bonn-Rhein-Sieg University of Applied
Sciences, Adjunct @ UNB

Neural scene representations
ntations(“KiloSDF” for SDFs/geometry)eometry)* Founding Director of the Institute of Visual Computing @BRS-U* Doing graphics (ray tracing) since end of 80s* Surfing VR waves since 1993 (Fraunhofer IAO, GMD /FhG IMK, BRS-U), HMDs, power walls, CAVEs, work
benches, tiled walls,..
* Research interest: efficient rendering, neuralrepresentations, interaction, systems

Work on augmenting magnetic fields
for pupils

What is the most important topic/grand challenge we should tackle?
Capturing light is easy, as light can be projected on sensors.
● How do we efficiently capture other perceivable stimuli that cannot be
projected? (Or can they?)

Takefumi Hiraki● How do we store them?

● How to we reproduce them?

Who are you?

What part of your work is most relevant to the Seminar?

●

Reality-augmentation media research for environment and body augmentation●●

Senior Research Scientist @ Cluster Metaverse Lab○ Cluster is the Japanese biggest VR SNS platform company
Assistant Professor @ University of Tsukuba (cross-appointed)○ PI @ Phygital Media Lab
Research Interest○ Projection mapping○ Visuo-Haptic augmented reality○ High-speed projection○ Soft robotics

What is the most important topic/grand challenge we should tackle?
Considering devices/technologies that extend human perception in our daily life,
especially for the metaverse platforms

Yuichi Hiroi
Filling the gap of VR/AR/MR research situation between academia and industry

CV:

Yuichi Hiroi

What part of your work is most relevant to the Seminar?

● Postdoc researcher @Rekimoto Lab, U-Tokyo

Enhancing individual vision by indistinguishable VR/AR○

Will join in Metaverse Lab, Cluster Inc. from July 2023

● 2022 Ph.D.Eng. from Tokyo Institute of Technology (Supervisor: Yuta Itoh)● Topic: AR Display, Vision Augmentation
Occlusion-capable OST-HMDs

Appearance Reproduction in AR

Vision Augmentation

What is the most important topic/grand challenge we should tackle?
Realization of vision augmentation according to individual sensitivity, beyond
ophthalmological constraints“What we see” is different individually;
how can VisAug confront the problem?
Visual Snow w/ Autism Spectrum Disorder (ASD)

Yuta ItohWho are you?
●●●

What part of your work is most relevant to the Seminar?

https://augvislab.github.io/people/yuta-itoh2013~2016: TU Munich, (Ph.D.)
2016~2017: Keio University,
2017~2021: Tokyo Tech,
2021~ Project Assistant Professor at Rekimoto Lab, U-Tokyo
See-through Displays, Human/Vision augmentation● D
Di
Display
is
isp
sp
plla
ay hardware
harrd
ha
d
dw
wa
w
are
re
Opticaldevices
● Op
O
pti
tic
ca
al d
de
evice
es
Vision
● V
Vi
isi
s io
on
n augmentation
aug
ugmen
ntta
n
attio
on

What is the most important topic/grand challenge we should tackle?
● How to make AR displays smaller and more usable for the daily life

Daisuke Iwai
Who are you?
●
●
●
●
●
●
●
●

What part of your work is most relevant to the Seminar?
SIGGRAPH ASIA 13

2007: PhD (Osaka University)
2008: PostDoc@Bauhaus University Weimar
2011: Visitor@ETH/Disney
2011~: Associate Professor@Osaka University—
Projection mapping
Computational displays
Computational fabrication

ISMAR 21 (Best paper)Eurographics 18

ISMAR 15

UIST 22IEEE VR 23

Augmented Human 17

Scientific Reports 14

IEEE TVCG 18

IEEE Access 18

CHI 20What is the most important topic/grand challenge we should tackle?
● Text-based real world manipulation

Jeeeun Kim“Golden, Baroque style”
Text-to-projection

Text-to-fabrication

Who are you?

●●●

Texas A&M University, direct HCIed Lab
Spend time to fight with fabrication machines
Use materials and machine parameters to fabricate different properties that humans canperceive (see, touch, and feel)

What part of your work is most relevant to the Seminar?

What is the most important topic/grand challenge we should tackle?

KiyoshiKiyokawa●●●●●

What are the properties of interest to be perceived differently so need customized?
What are types of information to be embedded, for what purpose?
How can that information be designed, embedded, and measured?
How people’s different perceptions could be presented in the tool for different contexts?
How fabricated physical properties and embedded information be reprogrammed real time?

Who are you?

What part of your work is most relevant to the Seminar?

● Professor at the CARE Laboratory, Nara Inst. of Sci. & Tech. (NAIST)(please google care lab japan)● VR / AR / Multimodal interaction, Human augmentation, …

● Display hardware (Hop)● Perception correction (Step)● Perception augmentation (Jump)

What is the most important topic/grand challenge we should tackle?

● How can we make smart glasses smart?
● How can we continuously diagnose vision in a daily life?

Hideki Koike
Initial results: Estimation ofvisual acuity from EOG (96%of accuracy) @ EMBC 2023

Who are you?

What part of your work is most relevant to the Seminar?

Name: Hideki Koike
Affiliation: Professor of Tokyo Institute of Technology
Research Topics: Vision-based HCI, Interactive Surfaces,
Human Augmentation

Motion capturing

Visualization for
Piano training

Haptic glove

Hobby: Alpine Ski, Playing Electric Bass

VR/AR Ski Training

AR Golf Training

Medical Training in VR

What is the most important topic/grand challenge we should tackle?
-

How VR/AR provides efficient training environment for experts (not novices)?
How each channel (e.g. visual, audio, haptics) or their combination could be
used?

Ernst Kruijff

Who are you?

What part of your work is most relevant to the Seminar?

● Research Professor for Human-Computer Interaction and co-director Institute ofVisual Computing, Bonn-Rhein-Sieg University of Applied Sciences● Adjunct Professor SIAT / Simon Fraser University● Basic research in perceptual and cognitive issues in AR● Multi-camera navigation techniques for AR● View management in narrow and wide FOV displays● Main topic: human-factors driven design, implementation and evaluation ofmultisensory 3D user interfaces● Research sind mid-nineties on 3D user interfaces (Utrecht University, BauhausUni Weimar, Fraunhofer IMK, TU Graz, Cure) - work on Augmented and Virtual
ys
yst
y
st
stems
ms
m
s
Reality ssystems
G
ra
az
z ((S
(Sc
S
Sc
chm
hm
hma
ma
alst
lsst
stieg
ie
ie
eg
g) o
nu
un
nco
n
cco
onve
nv
n
ve
ven
nti
ttio
on
ona
na
n
al 3D
3D useruse
err interfaces
e
int
in
nter
ter
te
erfac
ffa
ac
aces
es
● PhD TU Gr
Graz(Schmalstieg)
onunconventional○○

Effect of FOV on search tasks
Multisensory techniques for (search task) guidance (audio-tactile), dynamic conditions

What is the most important topic/grand challenge we should tackle?
●●●●

How to create wearable wider FOV and multifocal AR displays?
What are the longitudinal effects of AR?
How to adapt AR view management for dynamic (outdoor) conditions?
How to go beyond visual-only AR - non-visual augmentations, integration ofvarious modalities, cross-modal effects ?

Kai Kunze
Who are you?

Tobias Langlotz
What part of your work is most relevant to the Seminar?

What is the most important topic/grand challenge we should tackle?

[LSEVI]SY#EWWMWXERXTVSJIWWSVSJGSQTWGMEXXLIYRMZIVWMX]SJGLMGEKS
VIWIEVGLMRXIKVEXIHIZMGIW[MXLFSH]

Pedro Lopes

TIVGITXMSR[SVOTL]WMGEPWOMPPWWQIPPXEWXIWIRWISJEKIRG]',- 9-78 9-78 ',- ',-  ',- 
4WMRZMXEXMSRWYFQMXXS%'179- 7TEXMEP9WIV-RXIVJEGIW[LMGLXEOIWTPEGIMR7MHRI]3GXGSPSGEXIH[MXL-71%6

QSWXMQTSVXERXXSTMGKVERHGLEPPIRKI[IWLSYPHXEGOPI#
T
T K
K[LEXTEVXSJ]SYV[SVOMWQSWXVIPIZERXXSXLIWIQMREV#[LEXIPWIMWEXSHHW[MXLQEMRKSEPWSJ%6#
HIZMGIWXLEXEVIIKPMKLX[IMKLXRSXIRGYQFIVMRKWGEPI[IPPƏWSQIMHIEWXSQSZIXLMWJSV[EVH
7PMT 7XMGO;-4Paul Lukowicz

Augmented Perception

Shared Perception

Päivi Majaranta
Who are you?

What part of your work is most relevant to the Seminar?

Päivi Majaranta, Senior Research Fellow at● Gaze interaction for compensating (or augmenting) impaired abilities,
including work done in www.cogain.org● Multimodal HCI, e.g., improving gaze interaction with haptic feedback

Tampere University, Finland● M.Sc. in Computer Science (1998)● Ph.D. in Interactive Technology (2009)● Teaching courses on human-technology interaction● Research interests include human-technology interaction,gaze-based HCI, multimodal interfaces, humanaugmentation, animal-computer interaction
I like walking in forests, dogs, reading, playing computer gamesWhat is the most important topic/grand challenge we should tackle?

Katsutoshi
Masai

Utilize augmented perception to● Improve communication / interaction between people● Enhance empathy between people – or any living beings

Not forgetting basic research, ethical issues, societal research, etc. reminded in our IJHCS 2019 paper by Raisamo,
Rakkolainen, Majaranta et al. on Human augmentation: Past, Present and Future, including call for research

Katsutoshi Masai(Assistant Professor @ Kyushu University)

What part of your work is most relevant to the Seminar?
Augmenting communication using perception of self-faceEconomics -> Media Design -> Computer Science
Affective computing and interaction

What is the most important topic/grand challenge we should tackle?

We are all different. Emotion/Perception is something
personal.

And sometimes difficult to explain in words: and perhaps
we are even not aware of them. How can we handle
such sensitive and personal things ?

Jason Orlosky

So, personalization/ unconscious-self is my interested
topic

Who am I?

What part of your work is most relevant to the Seminar?

2006 BS Computer Engineering @ Georgia Tech2009 Healthcare IT for 3 Years2011 Japanese Language @ University of Georgia
2016 PhD in computer science @ Osaka University
2021 Associate professor @ Augusta University

Changing the way we learn via XR
Confused Learner

Perhaps the first person to play tennis the AR Rift

Automated Support

Self-regulation &Team-based study

ARVR Lab @ Augusta University

What is the most important topic/grand challenge we should tackle?
1) Change people via XR.
2) Use scalable XR support to reduce cognitive disparity.
We don’t need “నmoreᚍ,” we need “‫ه‬betterҗ.” I.e. improve how people treat eachother. If we just produce more stuff, the world gets bigger but doesn’t change.

-

Positivity
Mutual understanding
Compassion

Can do this via-

Intelligent tutors
Moral guides (agents)
Interfaces that help people understand each other

Shout out to the “AR forGood” Workshop folks~

Alexander
Plopski

Improved learning
outcomes

Who are you?

What part of your work is most relevant to the Seminar?

2016 - PhD @ Osaka University
2016-2019 - Assistant Professor 㸾 NAIST
2019-2022 - PostDoc @ University of Otago
March 2023 - Assistant Professor @ TU Graz-

Focus on eye gaze and AR with Optical See-Through Headsets-

Human Computer Interaction
Perception of AR Content

Impact on perception

Impact on learning

What is the most important topic/grand challenge we should tackle?
-

What use will AR have in everyday life?
--

Can it replace the smartphone?
How to make it cheap enough?
Can we avoid another “glasshole”?

Personalization of the experience
How much is too much?

Enrico Rukzio

Sight

Hyperreality

Enrico Rukzio: Who are you?

What part of your work is most relevant to the Seminar?

Professor in HCI at Ulm University (Germany)
Research on mobile & wearable interaction+ augmented and virtual reality

Enjoy mountain hiking, singing, sailingand watching football (FC Heidenheim)
my free time.
Spend time with my children.

Social Implications of Fragmented Augmented Reality: Imbalanced & Tampered
AMP-D (CHI 2014)

VairR (CHI 2017)

Facetouch (CHI 2017)

Movelet (ISWC 2018)

InScent (ISWC 2017)

Two days ago with
Rhianna Rukzio at
teamLab Planets TOKYO

Visual Alterations in Interpersonal Communication(CHI 2021)

Displaying Personal Information in
Augmented Reality (CHI 2022)

What is the most important topic/grand challenge we should tackle?

1. Relevance of AI chatbots
on Augmented Perception

Maki Sugimoto

ChatGPT

2. Potential of Auditory Augmented Reality
Pervasiveness of wearable auditory devices vs. wearable visual devices
Apple AirPods

1. Potential of revisiting Handheld Augmented Reality
for Augmented Perception

Maki Sugimoto: Who are you?

What part of your work is most relevant to the Seminar?

WƌŽĨĞƐƐŽƌĂƚĞƉƚ͘/ŶĨŽƌŵĂƚŝŽŶĂŶĚŽŵƉƵƚĞƌ^ĐŝĞŶĐĞ͕<ĞŝŽhŶŝǀ͘͘ZĞƐĞĂƌĐŚdŽƉŝĐƐ
^ĞŶƐŝŶŐdĞĐŚŶŽůŽŐŝĞƐĨŽƌZͬsZ,ƵŵĂŶƵŐŵĞŶƚĂƚŝŽŶŝŶsZ^ƉĂĐĞŽŶƚĂĐƚ͗ŵĂŝůƚŽ͗ƐƵŐŝŵŽƚŽΛŝĐƐ͘ŬĞŝŽ͘ĂĐ͘ũƉ,ŽďďŝĞƐ
ƌŝǀŝŶŐ
WŚŽƚŽŐƌĂƉŚ
'ͬtĞďĞƐŝŐŶͬWƌŽŐƌĂŵŵŝŶŐ
D͘&ƵŬƵŽŬĂ͕&͘EĂŬĂŵƵƌĂ͕͘sĞƌŚƵůƐƚ͕D͘/ŶĂŵŝ͕D͘<ŝƚĂǌĂŬŝĂŶĚD͘^ƵŐŝŵŽƚŽ͕Η^ĞŶƐŽƌǇƚƚĞŶƵĂƚŝŽŶǁŝƚŚĂsŝƌƚƵĂůZŽďŽƚŝĐƌŵŽŶƚƌŽůůĞĚhƐŝŶŐ&ĂĐŝĂůDŽǀĞŵĞŶƚƐ͕ΗŝŶ/dƌĂŶƐĂĐƚŝŽŶƐŽŶsŝƐƵĂůŝǌĂƚŝŽŶĂŶĚŽŵƉƵƚĞƌ'ƌĂƉŚŝĐƐ͕ϮϬϮϯ;ĂƌůǇĐĐĞƐƐͿ͕ĚŽŝ͗Ϭ͘ϭϭϬϵͬds'͘ϮϬϮϯ͘ϯϮϰϲϬϵϮ͘What is the most important topic/grand challenge we should tackle?

JonathanSuttonMiura, R., Kasahara,S., Kitazaki, M., Verhulst, A., Inami, M., and Sugimoto, M. (2021). MultiSoma:
Distributed Embodiment with Synchronized Behavior and Perception, Augmented Humans 2021

Who are you?

What part of your work is most relevant to the Seminar?

Postdoc from the University of Otago, HCI Lab
2022 Ph.D. from University of Otago
Research focus on OSTHMDs and using them to assist
vision.

Providing assistance to our visual perception using ‘Computational Glasses’

What is the most important topic/grand challenge we should tackle?
How to determine the augmentationand degree of augmentation that is
appropriate in a given situation.

Greg Welch
Professor and AdventHealth Endowed Chair in Healthcare Simulation
University of Central Florida

Who am I?

Virtual Experience Research Accelerator (VERA)

Academic Background•••Purdue University (B.S. Electrical Technology)
UNC-Chapel Hill (M.S. Ph.D. Computer Science)

Professional experience

•
•
•
•
•

NASA’s Jet Propulsion Laboratory (Voyager)
Northrop-Grumman (airborne ECM)
The University of North Carolina at Chapel Hill
The University of Central Florida

Smart Grid

Research Interests (examples)

•
•
•
•

Stochastic “stuff” (e.g., Kalman filter)
Physical-Virtual “stuff” (humans, AR+IoT, etc.)
Nascent: “Human Filter,” stimuli tradeoffs, …

Mysteries, music, travel, running, biking,home automation, …

Physical-Virtual Stuff

Virtual Experience Research Accelerator (VERA)

Tactile Telepresence for Isolated
Patients

Physical-VirtualPh
i l Vil Patient
P i
Simulator
Si lVisual → Aural
XR + AI…

Physical-Virtual Wound Simulator

Blowing in theWindWho are you?

Guanghan
Zhao
What part of your work is most relevant to the Seminar?

● Post doctor at Tokyo Institute of Technology● Received PhD in information technology from Osaka University● Research Interests:
Computer Vision, Virtual Reality, Augmented Reality, Human-Computer
Interaction and Visually Induced Motion Sickness.

What is the most important topic/grand challenge we should tackle?

Visual modification/indication in XR.

Remote meeting & collaboration
Provide a good-enough experience for daily usages:● Immersive● Interactive● Low-costPotential directions:● Real-time 360 video streaming from portable cameras to MR HMDs.
● Remote interactions via robot arms.

Qing Zhang(ᙇΎ)
Email: qzkiyoshi@gmail.com
Personal homepage:https://qzkiyoshi.github.io/○

What part of your work is most relevant to Augmented Perception?

○

What is the most important topic/grand challenge we should tackle during this seminar?

The time we without the “Human Augmentation” Devices

https://www.linkedin.com/pulse/mobile-phone-newexternal-organ-anmol-vishwakarama

Meeting Schedule

Figure 1: Meeting Schedule

19

Summary of Discussions and New findings
The main causes of misperception in AR/VR were discussed, including factors
such as distances, sizes, speed, and time. While it is challenging to solve theseissues completely, several approaches were suggested. One approach involved
a hyper-realistic rendition to force a perception shift. Another idea was tocreate a small set of standardized calibrations, although it was acknowledgedthat perception is often person-dependent and may require customized models.
Considerations for additional references and psychological perspectives were alsomentioned. It was noted that misperception could have serious consequencesin certain fields, such as space exploration or military training. The use of eye
tracking and models based on eye parameters were proposed for calibration,and the Varjo HMD system was highlighted as a potential solution for depth
perception in microscopic images.
The question of over-relying on visuals in AR was raised, and the importance of incorporating associated sounds was discussed. It was suggested thatsound could enhance the overall experience and provide additional cues for perception. Advances in AI-generated sound and spatial audio were mentioned.
While visuals remain crucial, the combination of visual and sound elements wasemphasized as essential for achieving a fully immersive and accurate experience.
Special AR ”for good” applications were explored, including areas such as
phobia therapy, blind assistance, education, and visualization of invisible phenomena. The potential of AR to improve various aspects of life, from healthcare
to science education, was highlighted. It was acknowledged that these applications require careful design and consideration of user needs, but they holdsignificant potential for positive impact.
The discussion revolved around triggering the next AR hype and the factorsthat would contribute to widespread adoption. The need for better AR devices,killer apps, and a focus on user experience and interaction were mentioned. Examples of potential killer apps, such as face recognition or viewing convenience,
were discussed. The challenge of finding the right balance between functionality
and form factor was emphasized, along with the need for seamless integration
with existing technologies.
The question of using the right measures to test AR environments undercontrolled and uncontrolled conditions was raised. The limitations of controlled
studies in capturing real-world experiences were acknowledged, and the importance of conducting studies in more practical and dynamic environments was
highlighted. Suggestions were made for in-the-wild experiments and the need for
longer-term studies to assess the effects of HMD use. The lack of well-acceptedusability studies and guidelines was also noted, calling for further research inthis area.
To make people want to use AR glasses on a daily basis, several features
were considered crucial. The workshop discussion focused on this question, andthe detailed summary was not provided.
The question of making real AR systems common was raised. The need foradvancements in technology, form factor, visualization, and interaction design
was discussed. The importance of addressing user needs, improving user interfaces, and creating practical and useful applications that integrate seamlessly
into daily life was emphasized.
Ensuring equality in AR adoption, particularly in terms of financial acces20

sibility, was discussed. Suggestions included cross-platform accessibility, availability of affordable technologies, and finding funding for socio-economicallydisadvantaged individuals. The importance of policy discussions, guidelines,
and considering the needs of individuals with disabilities was highlighted.
The question of XR UI/interaction with GenAI embedded and presenting
rich information in a limited display size was raised. Design patterns, interactive
content generation, and automated level/difficulty generation based on user
performance were discussed. The potential for AI to enhance gaming experiences
and customize content based on user preferences was highlighted.
The impact of AI, specifically Chat GPT, on VR programming jobs was examined. It was recognized that AI can change programming methodologies andtasks but also noted that human involvement and expertise remain necessary.

21

Identified issues and future directions
In the realm of Augmented Perception and Extended Reality (AR/VR/XR),
several important future directions emerge from the findings. One crucial aspect
is addressing misperception in AR/VR experiences. To achieve this, personalized calibration methods and eye-based models could be explored, potentiallyleveraging eye tracking for closed-loop calibration. By customizing the experience based on individual perception, the accuracy of AR/VR rendering can beenhanced.
Another direction involves integrating sound and visuals to create more immersive experiences. Advances in AI-generated sound and spatial audio technologies can play a vital role in enhancing the overall perception and immersion
in AR/VR environments. By incorporating sound cues and leveraging the powerof AI, developers can create more realistic and engaging experiences.
Special AR applications with a focus on healthcare, education, phobia therapy, and accessibility offer great potential for positive impact. Collaboratingwith domain experts and designing user-centered experiences can result in meaningful applications that address specific needs. By leveraging AR to augment
healthcare practices, facilitate learning, or assist individuals with disabilities,
the field can make significant strides towards improving lives.
Driving widespread adoption of AR requires advancements in AR devices,the development of killer apps, and a focus on optimizing user experiences.
Researchers and developers need to continue pushing the boundaries of technology, form factors, and interaction design to create AR solutions that seamlessly
integrate into users’ daily lives.
Evaluating AR environments also presents a challenge, as controlled studies may not fully capture real-world experiences. To overcome this limitation,
researchers should explore in-the-wild experiments and establish usability guidelines. Balancing controlled studies with practical, dynamic environments will
provide a more comprehensive understanding of AR usability and effectiveness.
Improving AR glasses to provide practical value and enhance user experienceis another crucial direction. This involves refining form factors, visualization
techniques, and interaction design to create AR glasses that users will want toincorporate into their daily routines.
The integration of AI, such as Chat GPT, holds potential for enhancing
XR user interfaces and interactions. AI-driven design patterns, personalized
content generation, and automated level/difficulty adjustments based on user
performance could significantly enhance XR experiences and customization. It
is important to recognize that despite the advancements in AI and automation,
human expertise remains vital in VR programming. While AI can assist in
various aspects, human involvement and expertise are still necessary to createcompelling and meaningful VR experiences.
By focusing on these future directions, i.e., addressing misperception, enhancing multi-sensory experiences, developing impactful applications, driving
adoption, improving evaluation methodologies, optimizing user experience, advancing AR technologies, leveraging AI in XR, and acknowledging the role ofhuman expertise, the participants gained ideas on how to further improve thefields of Augmented Perception and Extended Reality.

22Bibliography
1. Ivan E. Sutherland. A Head-Mounted Three Dimensional Display. In
Proceedings of the Fall Joint Computer Conference, Part I, pages 757–
764, New York, USA, December 1968.
2. Kim, K., Billinghurst, M., Bruder, G., Duh, H. B. L., & Welch, G. F.
(2018). Revisiting trends in augmented reality research: A review of the
2nd decade of ISMAR (2008–2017). IEEE Transactions on Visualization
and Computer Graphics, 24(11), 2947-2962.
3. Schmidt, Albrecht. “Augmenting human intellect and amplifying perception and cognition.” IEEE Pervasive Computing, 16(1), 6-10, 2017.
4. Engelbart, Douglas C. “Augmenting human intellect: A conceptual framework.” Menlo Park, CA (1962).
5. Rekimoto, Jun, & Nagao, Katashi. “The world through the computer:
Computer augmented interaction with real world environments.” In Designing Communication and Collaboration Support Systems, 1999.
6. Papagiannis, Helen. Augmented human: How technology is shaping the
new reality. O’Reilly Media, Inc., 2017.
7. Langlotz, T., Sutton, J., Zollmann, S., Itoh, Y., & Regenbrecht, H. (2018,
April). “ChromaGlasses: Computational glasses for compensating colour
blindness.” In Proceedings of the 2018 CHI Conference on Human Factors
in Computing Systems (p. 390). ACM.

23