Immersive Inclusivity at CHI: Design and Creation of Inclusive
User Interactions Through Immersive Media
Bektur Ryskeldiev

Yoichi Ochiai

University of Tsukuba, Mercari R4D
Japan
bektour@digitalnature.slis.tsukuba.ac.jp

University of TsukubaJapanwizard@slis.tsukuba.ac.jp

Jie LiYamen Saraiji

CWI
Netherlands
jie.li@cwi.nl

Avatarin IncJapanyamen@avatarin.com

Mark Billinghurst

Suranga Nanayakkara

University of Auckland
New Zealandmark.billinghurst@auckland.ac.nz

University of Auckland
New Zealands.nanayakkara@auckland.ac.nz

Koki Kusano

Merpay, Mercari R4D
Japankoki.kusano@mercari.com

Kai Kunze

Keio UniversityJapan
kai@kmd.keio.ac.jp

Yusuke Sugano

University of TokyoJapansugano@iis.u-tokyo.ac.jp

Tatsuya Honda

Fujitsu LimitedJapanhonda-tatsuya@ed.tmu.ac.jp

Figure 1: (a) Proposed immersive inclusive design placement in relation the felds of Human-Computer Interaction, immersive
media, and accessibility. (b, c) Examples of immersive inclusive design applications: (b) using a wheelchair remotely controlled
through Virtual Reality (VR) telepresence interface for assistance [3], (c) applying a headworn hair clip for translation ofauditory signals into haptic vibration [4]ABSTRACT
Immersive media is becoming increasingly common in day-to-day
scenarios: from extended reality systems to multimodal interfaces.
Such ubiquity opens an opportunity for building more inclusive
environments for users with disabilities (permanent, temporary,Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor proft or commercial advantage and that copies bear this notice and the full citationon the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan© 2021 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-8095-9/21/05.
https://doi.org/10.1145/3411763.3441322

or situational) by either introducing immersive and multimodal
elements into existing applications, or designing and creating immersive applications with inclusivity in mind. Thus the aim of
this workshop is to create a discussion platform on intersections
between the felds of immersive media, accessibility, and humancomputer interaction, outline the key current and future problemsof immersive inclusive design, and defne a set of methodologies
for design and evaluation of immersive systems from inclusivity
perspective.

CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, JapanCCS CONCEPTS• Human-centered computing → Human computer interaction (HCI); Accessibility technologies; Collaborative and socialcomputing.

KEYWORDS
assistive technology, collaborative technology, accessibility, artifcial intelligence, augmented reality, virtual reality, mixed reality,
extended reality, human-computer interaction, inclusive design,internet of things, augmented human

Ryskeldiev and Ochiai, et al.

2 WORKSHOP PLAN
2.1 Workshop Goals
The aim of this workshop is to address these questions by creating adiscussion platform among researchers working in the felds of immersive and interactive media, accessibility, and human-computerinteraction. The main goals of this discussion are:• Identify the current and future key problems in inclusive
design from the perspective of immersive media; defne when
making a system more immersive leads to a more inclusive
user experience, and how immersive systems can be designed
and built with inclusivity in mind,• Outline the immersive inclusive design taxonomy, in particular how immersive inclusive design principles combine withthe concepts of assistive augmentation and accessibility,• Collect and describe a set of use-cases on immersive inclusive
design where this approach was or could have been appliedon existing problems and applications,• Propose and outline potential evaluation methodologies for
immersive inclusive design research, create an understanding on how existing evaluation methodologies for accessibility and immersive systems could be applied in immersive
inclusivity context• Outline and expand community of immersive inclusive design researchers and practitioners, encourage a dialogue between academic researchers and industry practitioners, leading to continuous bilateral value and experience exchange.

ACM Reference Format:
Bektur Ryskeldiev, Yoichi Ochiai, Koki Kusano, Jie Li, Yamen Saraiji, Kai
Kunze, Mark Billinghurst, Suranga Nanayakkara, Yusuke Sugano, and Tatsuya Honda. 2021. Immersive Inclusivity at CHI: Design and Creation ofInclusive User Interactions Through Immersive Media. In CHI Conference on
Human Factors in Computing Systems Extended Abstracts (CHI ’21 ExtendedAbstracts), May 8–13, 2021, Yokohama, Japan. ACM, New York, NY, USA,
4 pages. https://doi.org/10.1145/3411763.3441322

1

BACKGROUND

Immersive media (e.g., extended reality (XR) and multimodal) interfaces are becoming increasingly common in day-to-day scenarios:
from facial tracking and location-based entertainment to voiceand gesture-controlled smart assistants and home systems. As suchinteractions are becoming more seamless and natural, recent studies are suggesting that in the upcoming years we will see evencloser integration between human and technology [8], and artifcial
intelligence (AI) [9].
Thus, by applying immersive and multimodal forms of interactions to existing interfaces, we also open new opportunities fordesigning and establishing inclusive environments and applications
for users with disabilities (either permanent, temporary, or situational). This approach has been particularly explored in works by“Augmented Human” and “Assistive Augmentation” communities.
Where, for instance, in [5] Huber et al. discuss that by combining
immersive, wearable, and multimodal interfaces with user’s mind,
body, and behavior, it is possible to enhance user’s perception as
well as have an efect on their abilities or disabilities.
Recent examples of such enhancements in academic research include using mobile Augmented Reality (AR) interface for assistancein understanding dynamic menus in vending machines and kiosks[6], combining spatial audio and AR for real time recognition ofsounds and speech [1], and making VR experiences accessible forusers with vision disabilities through haptic and auditory feedback[10]. Similarly, due to increasing availability of personal mobile andinternet of things (IoT) devices, recent commercial applications had
also featured immersive inclusive design solutions on navigation
through spatialized auditory feedback [7], personal wearables forsound recognition with haptic feedback [4], and using multimodal
handheld devices to make live musical performances accessible tousers with hearing disabilities [2].
As intersections between immersive media and inclusive design
are becoming more commonplace, we outline two main questions:“How can immersive media make existing environments, systems, andinterfaces more inclusive?” and “How can immersive systems be builtwith inclusivity in mind?”

2.2

Pre-Workshop Plans

2.2.1 Website and Social Media. Workshop overview, call for papers, deadlines, submission form, organizer, and contact information will be displayed on the website https://iiworkshop2021.
github.io. Given the current pandemic situation and online-onlyformat of the conference, we are aiming for fully remote participation, with video calls for talks and workshop activities, and aseparate message group. Further details on remote participation will
be posted on the website upon the workshop proposal acceptanceand discussion with CHI organization committee.
Prior to the workshop all selected attendees will be requested
to fll out the online form regarding their timezone preferences,
platform restrictions and accessibility requirements, professional
background, and interest in topics connected to immersive inclusive
design. This information will be used for scheduling the workshop,
arranging the online communication format, and creation of discussion groups on several chosen topics (e.g., immersive inclusive
design taxonomy, applications, evaluation methods, and ethicalissues) on which the attendees should create a short presentation
to be shown during the workshop. All attendees will also be added
to a private Slack group for discussion, note sharing, and further
planning of the workshop.
2.2.2 Recruitment. We plan to recruit a diverse set of scholars andpractitioners primarily from the following communities: HCI (ACMSIGCHI), Computer Graphics (ACM SIGGRAPH), Accessibility (ASSETS, IPSJ SIG AAC), and Augmented Human. Furthermore, we
hope that this workshop would also attract researchers, developers,

Immersive Inclusivity at CHI

and UI/UX designers who have experience or are currently working
in the industry.
The call for papers (CFP) will be mainly disseminated throughsaid communities’ mailing lists and social media, and posted on
http://www.wikicfp.com/cfp/. We intend to select up to 30 participants who will be asked to submit a 2-4 page position paper on
one or several topics listed in the CFP. The selection and reviews
will be carried out by organizers and invited reviewers.

2.3

Workshop StructureThis workshop is based on a one-day four-hour program. Tentative
schedule as follows:• 9:00-9:15 Welcome and introduction: Workshop opening, introduction of the topics, program, organizers and participants• 9:15-09:45 Opening keynote talk by one of the workshop
organizers on inclusive design and immersive media• 09:45-10:00 Short break• 10:00-11:00 Lightning talks: Series of short position papertalks, 2-3 minutes each
• 11:00-11:15 Short break• 11:15-11:45 Group ideation: Participants work in (preformed) groups on topics introduced prior to the workshop.
Each group prepares a short (2-3 slides) presentation on thetopic. The groups will be done via Zoom breakout rooms
and Discord channels.
• 11:45-12:00 Short break• 12:00-12:45 Group presentations: Each group presents ashort 2-3 minute talk on their chosen topic with about 3-5
minutes on discussion• 12:45-13:00 Workshop discussion and closing: Organizers summarize the conducted workshop and present potential plans for future collaborations.

3POST-WORKSHOP PLANS

Outputs such as position papers and discussion contents (presentations and notes) will be published on the workshop website. The
workshop process, discussions, and outcomes will be also collected,summarized and published as a blog post on the workshop’s website,
as well as submitted for publication to ACM Interactions magazine.
As we intend to support the created immersive inclusive design community, we plan to also create a mailing list to coordinate furtherworkshops or SIG meetings at ACM CHI or SIGGRAPH sponsoredconferences.
Finally, we expect this workshop to outline and broaden theimmersive inclusive design research community, as well as createa bilateral experience exchange between academic research andindustry which will lead to potential future collaborations.

4ORGANIZERS• Bektur Ryskeldiev is a Postdoctoral Researcher at Researchand Development Center for Digital Nature, University ofTsukuba, and a Research Fellow at HCI group, Mercari R4D.
His work is primarily focused on the problems of telepresence, remote collaboration, XR interactions, and inclusive
design. He is also a member of ACM SIGGRAPH Immersive

CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japanand Interactive Media Committee, and ACM SIGGRAPH
International Resources Committee (IRC).
• Yoichi Ochiai is an Associate Professor at the University ofTsukuba and CEO at Pixie Dust Technologies Inc. He is oneof the principle investigators at JST CREST xDiversity grant
which is focused on applications of computer graphics, AI,and multimodal interactions for inclusive design.
• Koki Kusano is a UX Researcher in Merpay, Inc. He studies
tools and methods for supporting Human-Centered Designand applies them in practice as a UX researcher. He hasvarious work experiences related to research and development of design methods, UX and qualitative UX research, aswell as lecturing on Human-Centered Design for the masterprogram at Keio University.
• Jie Li is a postdoctoral researcher at Distributed Interactive
Systems group of The Dutch National Research Institute forMathematics and Computer Science (CWI). She holds a PhD
degree in Human Information communication Design from
Delft University of Technology, and is specialised in UX andHCI research. She is currently working on develop subjective
metrics for assessing experience in social VR, building andevaluating new social VR experiences in diverse domains(e.g., medical consultation, co-design, museum).
• Yamen Saraiji is an Avatar Core Director at Avatarin Inc.
He has previously taught as a Project Senior Assistant Professor at Keio University Graduate School of Media Design. His
research experience includes working in the felds of telepresence, robotics, artifcial intelligence, augmented human,and human-computer interaction.
• Kai Kunze works as a Professor at the Graduate School ofMedia Design, Keio University, Yokohama, Japan. He is a pioneer researcher in the emerging Eyewear Computing feld.
His current research includes augmenting humans, quantifying cognitive states and amplifying human senses. Heis a founding member of the Superhuman Sports AcademySociety, Japan.
• Mark Billinghurst is Professor at the University of South
Australia in Adelaide, Australia, and also at the University of
Auckland in New Zealand, directing the Empathic Computing Laboratory. He earned a PhD in 2002 from the Universityof Washington and researches how virtual and real worlds
can be merged, publishing over 550 papers on AugmentedReality, remote collaboration, Empathic Computing, and related topics. In 2013 he was elected as a Fellow of the Royal
Society of New Zealand, and in 2019 was given the ISMAR
Career Impact Award in recognition for lifetime contribution
to AR research and commercialization• Suranga Nanayakkara is an Associate Professor at theAuckland Bioengineering Institute, the University of Auckland (UoA). Suranga founded and directs the “Augmented
Human Lab” to explore ways of creating ‘enabling’ humancomputer interfaces as natural extensions of our body, mindand behaviour• Yusuke Sugano is an associate professor at Institute ofIndustrial Science, The University of Tokyo. His researchinterests focus on computer vision and human-computerinteraction. He received his Ph.D. in information science

CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japanand technology from the University of Tokyo in 2010. Hewas previously an associate professor at Graduate Schoolof Information Science and Technology, Osaka University, a
postdoctoral researcher at Max Planck Institute for Informatics, and a project research associate at Institute of IndustrialScience, the University of Tokyo.
• Tatsuya Honda is a user interface designer at Fujitsu Limited. Honda began working with deaf people in college asa sign language interpreter and volunteer. Looking to help
the deaf, he started work in college on what would becomea product called Ontenna (a play on the Japanese words
for sound and antenna), a device that translates sound, likemusic, into vibrations and light.

5CALL FOR PARTICIPATION

Immersive media is becoming increasingly common in day-to-day
scenarios: from extended reality systems to multimodal interfaces.
Such ubiquity opens an opportunity for building more inclusive environments for users with disabilities (either permanent, temporary,or situational) by introducing immersive and multimodal elements
into existing applications, or designing and creating immersive
applications with inclusivity in mind.
This ACM CHI 2021 Online Workshop on Design and Creation of Inclusive User Interactions Through Immersive Media invites its participants to open a discussion on intersections
between the felds of immersive media, accessibility, and humancomputer interaction, outline the key current and future problemsof immersive inclusive design, defne a set of methodologies fordesign and evaluation of immersive systems from inclusivity, andcreate a bilateral value exchange dialogue between researchers andpractitioners in academia and industry.
We welcome all interested applicants, including students andprofessionals with experiences in such felds as Human-Computer
Interaction (HCI), social computing, UI/UX design and research,accessibility, immersive media, Augmented, Virtual, and Extended
Reality (AR / VR / XR), Artifcial Intelligence, and Augmented Human. We intend this workshop to be engaging and informative toall participants regardless of level of familiarity with the workshop
topic.
Please note that due to ACM CHI 2021 virtual format, this workshop will be also conducted online. To participate in the workshop,
we ask applicants to submit 2-4 page position papers in ACM Master Article (single column) format, on the topics that include, but
are not limited to:

Ryskeldiev and Ochiai, et al.

• Immersive media and inclusive design• Accessibility• Assistive augmentation• Augmented, Virtual, Mixed, and Extended Reality (AR / VR /MR / XR)• Human-Computer Interaction• Human-Computer Integration (HInt)• Augmented Human• HCI and Artifcial Intelligence
Submissions deadline is February 21st 2021 Anywhere On Earth(AoE), acceptance notifcations will be sent out on March 5th 2021.
Workshop will happen on May 8th 2021. At least one of the accepted
authors is expected to attend both workshop and the conference.

ACKNOWLEDGMENTS
This work was supported by JST CREST Grant Number JPMJCR1781,
Japan.

REFERENCES[1] Ru Guo, Yiru Yang, Johnson Kuang, Xue Bin, Dhruv Jain, Steven Goodman, LeahFindlater, and Jon Froehlich. 2020. HoloSound: Combining Speech and SoundIdentifcation for Deaf or Hard of Hearing Users on a Head-Mounted Display.
ACM, New York, USA. https://doi.org/10.1145/3373625.3418031[2] Hakuhodo. 2018. Making music accessible with a “sound-free concert”. Article. Retrieved October 14, 2020 from https://www.hakuhodo-global.com/news/makingmusic-accessible-with-a-sound-free-concert.html.
[3] Satoshi Hashizume, Ippei Suzuki, Kazuki Takazawa, Ryuichiro Sasaki, and Yoichi
Ochiai. 2018. Telewheelchair: The remote controllable electric wheelchair system
combined human and machine intelligence. In Proc. of the 9th Augmented HumanInt. Conf. 1–9.
[4] Tatsuya Honda and Makoto Okamoto. 2014. User interface design of soundtactile. In Int. Conf. on Computers for Handicapped Persons. Springer, 382–385.
[5] Jochen Huber, Jun Rekimoto, Masahiko Inami, Roy Shilkrot, Pattie Maes, Wong
Meng Ee, Graham Pullin, and Suranga Chandima Nanayakkara. 2014. Workshop
on assistive augmentation. In CHI’14 Extended Abstracts on Human Factors in
Computing Systems. 103–106.
[6] Junhan Kong, Anhong Guo, and Jefrey P. Bigham. 2019. Supporting OlderAdults in Using Complex User Interfaces with Augmented Reality. In The 21stInt. ACM SIGACCESS Conf. on Computers and Accessibility (Pittsburgh, PA, USA)(ASSETS ’19). Association for Computing Machinery, New York, NY, USA, 661–663.
https://doi.org/10.1145/3308561.3354593[7] Microsoft. 2018. Soundscape. Article. Retrieved October 14, 2020 from https://www.microsoft.com/en-us/research/product/soundscape/.
[8] Florian Floyd Mueller, Pedro Lopes, Paul Strohmeier, Wendy Ju, Caitlyn Seim,
Martin Weigel, Suranga Nanayakkara, Marianna Obrist, Zhuying Li, Joseph Delfa,
et al. 2020. Next Steps for Human-Computer Integration. In Proc. of the 2020 CHIConf. on Human Factors in Computing Systems. 1–15.
[9] Jun Rekimoto. 2019. Homo Cyberneticus: The Era of Human-AI Integration.
(2019). arXiv:1911.02637[10] Yuhang Zhao, Cynthia L Bennett, Hrvoje Benko, Edward Cutrell, Christian Holz,
Meredith Ringel Morris, and Mike Sinclair. 2018. Enabling people with visual
impairments to navigate virtual reality with a haptic and auditory cane simulation.
In Proc. of the 2018 CHI Conf. on Human Factors in Computing Systems. 1–14.