# Multisensory Augmented Reality

Kasun Karunanayaka, Anton Nijholt, Thilina Halloluwa, Nimesha Ranasinghe, Manjusri Wickramasinghe, Dhaval Vyas

► To cite this version:

Kasun Karunanayaka, Anton Nijholt, Thilina Halloluwa, Nimesha Ranasinghe, Manjusri Wickramasinghe, et al.. Multisensory Augmented Reality. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.558-563, 10.1007/978-3-030-85607-6_77. hal-04291242

HAL Id: hal-04291242

https://inria.hal.science/hal-04291242v1

Submitted on 17 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Multisensory Augmented Reality

Kasun Karunanayaka¹[0000-0002-3933-9832], Anton Nijholt²[0000-0002-5669-9290],

Thilina Halloluwa¹[0000-0003-0132-6355], Nimesha Ranasinghe⁴[0000-0003-3962-8084],

Manjusri Wickramasinghe¹[0000-0002-0725-5124], and Dhaval Vyas³ [0000-0002-9356-0976]

¹ School of Computing, University of Colombo, Colombo, Sri Lanka

² University of Twente, Enschede, Netherlands

³ University of Queensland, St Lucia, Australia

⁴ School of Computing and Information Science, University of Maine, USA
ktk@ucsc.cmb.ac.lk, a.nijholt@utwente.nl, tch@ucsc.cmb.ac.lk
r.ranasinghe@maine.edu, mie@ucsc.cmb.ac.lk, d.vyas@uq.edu.au

**Abstract.** Multisensory augmented reality enables content developers to generate more realistic, sensory rich user experiences. Applications and research related to multisensory augmented reality expands through several areas such as education, medicine, human-machine interactions, human-food interactions, marketing, and neuroscience. Aim of this workshop is to gather researchers and industry representatives who are involved in multisensory augmented reality research to discuss the current state-of-the-art in the field, define future research directions, form new collaborations, and come up with future publication plans. We believe that this workshop would enhance the participants’ experience of the Interact 2021 and encourage more participants to attend the main conference.

**Keywords:** Multisensory Augmented Reality, Multisensory Internet, Multimodal Interfaces, HCI, Multisensory User Experiences.

## 1 Introduction

Augmented Reality (AR) research is usually based on a definition provided by Ronald Azuma [2]: AR (1) combines real and virtual objects in a real environment, (2) registers (aligns) real and virtual objects, (3) and runs interactively, in three dimensions, and in realtime. This definition does not limit AR to specific technologies and neither does it limit AR to the sight sense. The definition applies to all senses. That is, AR systems can provide artificially generated visual, auditory, tactile, olfactory, and gustatory experiences or combinations of these experiences superimposed on reality (Multisensory AR). Although we often speak of virtual ‘imagery’ that is overlaid on a user’s ‘view’ of the physical environment, the other senses and their cross-modal properties can be addressed as well to design useful, convincing, and attractive AR environments. When, according to the definition, we want to register or align real and virtual objects we should take into account that the virtual objects have other than visual properties or don’t have visual properties at all. We can have virtual touch, sound, smell, and taste
objects that are superimposed on reality and define an augmented reality world that is experienced by a user's senses.

During the past decades, Multisensory Augmented Reality (MAR) have been used to deliver immersive user experiences in different areas. Traditional Internet communication was previously dominated by text, images, audio and video. However, digitization of other sensory modalities such as touch [24], taste [4, 17], and smell [4, 16] transforming the current Internet communication into a multisensory Internet communication. In relation to teaching and training, MAR can be used to provide enhanced interactions using virtual objects, user guidance, and force feedback [9, 10, 19]. Some previous research related to human-food Interaction suggested that multisensory augmented reality can be used to generate or modify our perceptions by altering color, textures, shapes, taste, and smell of food and beverages that we eat and drink [4, 17, 21, 22]. On the other hand, digital sensory marketing can be considered as another important area where MAR can be used to create and enhance customer experiences [15]. MAR has also been used to develop new interactions and communication methods related to smart cities and smart communities [1, 12, 13, 14, 17]. MAR can also enhance our gaming experiences [23] as well as it can be used to improve industry and automation activities [8, 11].

At present, the field of MAR is primarily focused on technology developments, such as developing new ways to simulate multiple senses and create new interactions by combining AR and human senses. However, little is known about the impact of the addition of each stimulus, let alone multiple stimuli in MAR experiences. Future explorations of MAR, apart from expanding into different application areas, will help study technology-mediated perception, cognition, and human behaviors. These studies will assist in uncovering secrets in the human brain, behavior, neuroscience, and physiology.

# 2 Workshop Objectives, Target Audience, and Topics

The first workshop on Multisensory Augmented Reality (http://usehci.org/mar2021/) for Interact 2021 was proposed to gather researchers from diverse disciplines such as human-computer interaction, augmented reality, virtual reality, interaction design, gaming, psychology, and neuroscience. This workshop will be held on 30th August 2021 as a one-day workshop in conjunction with the Interact 2021 conference in Bari, Italy.

The structure of this workshop is focused on creating research partnerships and identifying collaborative projects. During the workshop, recent advances in Multi-sensory Augmented Reality will be discussed, which will be the key trends, challenges, and opportunities in conducting research in these intersections. Our overall goal is to have a focused workshop that will initiate projects which will extend beyond the workshop itself. We believe that the intimate nature of workshops at INTERACT is an ideal venue for this type of workshop.

Submissions were sought on the topics including MAR interfaces, applications of MAR, new sensing and actuation Technologies, multimodal or crossmodal perception
in AR, smart wearable technologies, MAR based digital and sensory marketing, MAR in human food interaction, and communication protocols for MAR. The workshop's Call for Participation were distributed via several mailing lists related to HCI, UX, and related areas interested in multisensory research. Additionally, the call for participation was distributed via various social media channels and media arts groups. The workshop website (http://usehci.org/mar2021/) was used to share more information and update interested participants.

# 3 Summary of Position Papers

We have received eight submissions for the workshop. Submissions were reviewed by the workshop organizers and after careful consideration, five position papers were accepted. Either full paper versions or abstracts of those position papers are published on the MAR2021 website (https://usehci.org/mar2021/proceedings). Key contributions of those position papers are described below.

Thomas and Holmquist have proposed a ‘Generalised Architecture for Multisensory Augmented Reality’ [19]. This approach would entail three components: data to be displayed, preferred modality in which it is to be displayed, and a transformation function between first and second. As a result of the literature review and outcomes of their focus group findings, authors have suggested that audio, visual, haptics, and temperature modalities are the most desirable modalities for future devices, although they have suggested further investigation on smell and taste as well.

‘The Value of Sound within a Multisensory Approach to AR in the Arts’ by Bilbow, Kiefer, and Chevalier [3] discussed the potential of using sound in creating coherent, immersive, and embodied multisensory AR experiences. First, this paper explains a theoretical framework which investigates sensory modalities afforded by AR. Second, it describes two artworks which investigates embodied experiences in AR artworks. Authors suggested that DIY approaches to augmented reality are essential for creative work, and they further speculate on how art can contribute to future theory, technologies and practice in the field.

Guedes’s (2021) paper is about ‘Designing Multisensory Experiences in Museums for People with Disabilities’ [6]. This position paper questioned about how accessible and feasible current museums are for people with disabilities. Then the paper discussed how technology can support different forms of interacting, understanding, and sense-making of artworks and multimedia content before, during, and after a museum visit. One example application proposed by the author is called AIMuseum, which provides an accessible and inclusive solution that seeks to enhance museums and exhibitions with Augmented Reality and Screen Readers. In addition to that it discusses several available opportunities for designing, implementing, and evaluating tools for and with people with disabilities to support their experience in museums while catering to different abilities.

‘Beyond visual guidance in semi-manual industrial assembly of wooden house sections’ by Tobisková, Malmsköld, and Pederson [20] proposed a multimodal AR based guidance system for semi-manual industrial assembly of wooden house sections. This
proposed system incorporate vision, hearing, touch, smell senses together with guid-
ance through subtle cues for supporting the truss assembly operations. Authors have
reported the findings of their brainstorming sessions related to the process of assem-
bling wood trusses and mapping AR guidance modalities which is useful.

The paper titled ‘The use of Augmented Reality to deliver enhanced user experiences in fashion industry’ by Jayamini et. al. [7] reviewed how the fashion design industry has acquired novel technologies such as AR, VR, Mixed Reality, and Machine learning to address the limitations of traditional fashion experience and user experience. Literature review section of this paper provides a detailed overview of the recent developments related to fashion design, apparel self-customization, and enhancing customer shopping experiences. They argue that MAR has a huge potential in transforming the fashion industry towards digitalization by introducing the applications such as virtual fashion shows, virtual showrooms, virtual fitting rooms, virtual fit-or-sizing tools, digital jewelry, virtual stylists, and magazine catalogs.

4 Expected Outcomes and Future Plan

One of the main expected outcomes of this workshop is to gather researchers who are working on similar research topics, present each other’s research and research interest, discuss improvements to current research, form new research ideas, and start new research collaborations. In this way, it is possible to build multidisciplinary research teams and plan future research work and publications. We expect that this workshop would serve the purpose of knowledge gathering and dissemination related to MAR and enhance Interact 2021 participation and experience. To stimulate additional discussions, all accepted submissions to the workshop, presentations, and inspiring discussions will be made public on the website (http://usehci.org/mar2021/).

5 Workshop Organizers

**Kasun Karunanayaka** is a Senior Lecturer at the University of Colombo School of Computing, Sri Lanka. Kasun obtained his PhD in Electrical and Computer Engineering from the National University of Singapore (NUS) in 2014. He conducts research related to Mixed Reality, Multisensory Communication, and Physical Computing.

**Anton Nijholt** received his PhD in computer science from the Vrije Universiteit in Amsterdam. He held positions at various universities, inside and outside the Netherlands. In 1989 he was appointed full professor at the University of Twente in the Netherlands. His main research interests are multimodal interaction with a focus on entertainment computing, affect, humour and brain-computer interfacing.

**Thilina Halloluwa** is a Senior Lecturer at the University of Colombo School of Computing. He has a PhD from the Science and Engineering Faculty at the Queensland University of Technology, Brisbane. Within the fields of Human-Computer Interaction (HCI) and Computer-Supported Cooperative Work (CSCW), his research focuses on Designing for Underserved Communities, Smart Agriculture, Gamification, and Technology Enhanced Education.
**Nimesha Ranasinghe** is an Assistant Professor at the School of Computing and Information Science and directs the Multisensory Interactive Media lab (MIM lab - www mimlab info/) at the University of Maine. He completed his Ph.D. at the Department of Electrical and Computer Engineering, National University of Singapore (NUS), in 2013. His research interests include Multisensory Interactive Media, Human-Computer Interaction, Augmented and Virtual Reality.

**Manjusri Wickramasinghe** is a Senior Lecturer at the University of Colombo School of Computing, Sri Lanka. He obtained his PhD in Information Technology from Monash University in 2016. His research interests are Machine Learning, Computer Vision and Graphics, and Serious Games.

**Dhaval Vyas** is an ARC DECRA Fellow (2018-2021) and a Senior Lecturer in the School of Information Technology and Electrical Engineering, at the University of Queensland, Australia. He received a PhD in Human-Computer Interaction from the University of Twente, the Netherlands. His research spans across the areas of Human-Computer Interaction (HCI) and Computer-Supported Cooperative Work (CSCW).

## References

1. Arsovski, S., Osipyan, H., Cheok, A.D. and Muniru, I.O., 2018, November. Internet of speech: a conceptual model. In *3rd International Conference on Creative Media, Design and Technology (REKA 2018)* (pp. 359-363). Atlantis Press.

2. Azuma, R.T.: A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments 6, 4, 355-385 (1997).

3. Bilbow, S., Kiefer, C., and Chevalier, C. 2021, The Value of Sound within a Multisensory Approach to AR in the Arts, *In Proceedings of the Multisensory Augmented Reality Workshop, Interact 2021.*, https://usehci.org/mar2021/proceedings/

4. Cheok AD, Karunanayaka K. Virtual taste and smell technologies for multisensory internet and virtual reality. Cham: Springer; 2018 Jun 29.

5. Edirisinghe, C., Podari, N. and Cheok, A.D., 2018. A multi-sensory interactive reading experience for visually impaired children; a user evaluation. *Personal and Ubiquitous Computing*, pp.1-13.

6. Guedes, L. S., 2021, Designing Multisensory Experiences in Museums for People with Disabilities, *In Proceedings of the Multisensory Augmented Reality Workshop, Interact 2021*, https://usehci.org/mar2021/proceedings/

7. Jayamini, C., Sandamini, A., Pannala, T., Kumarasinghe, P., Perera, D., Karunanayaka, K., 2021, The use of Augmented Reality to deliver enhanced user experiences in fashion industry, *In Proceedings of the Multisensory Augmented Reality Workshop, Interact 2021*. https://usehci.org/mar2021/proceedings/

8. Karunanayaka, K., Saadiah, H., Shahroom, H. and Cheok, A.D., 2017. Methods to develop a low-cost laboratory olfactometer for multisensory, psychology, and neuroscience experiments. In *IECON 2017-43rd Annual Conference of the IEEE Industrial Electronics Society* (pp. 2882-2887). IEEE.

9. Liyanage, S.U., Jayaratne, L., Wickramasinghe, M. and Munasinghe, A., 2019, March. Towards an Affordable Virtual Reality Solution for Cardiopulmonary Resuscitation Training. In *2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)* (pp. 1054-1055). IEEE.
10. Lu, J., Li, L. and Sun, G.P., 2010, August. A multimodal virtual anatomy e-learning tool for medical education. *In International Conference on Technologies for E-Learning and Digital Entertainment* (pp. 278-287). Springer, Berlin, Heidelberg.

11. Malý, I., Sedláček, D. and Leitao, P., 2016, July. Augmented reality experiments with industrial robot in industry 4.0 environment. In *2016 IEEE 14th International Conference on Industrial Informatics (INDIN)* (pp. 176-181). IEEE.

12. Nijholt, A., 2020. *Making Smart Cities More Playable*. Springer Singapore.

13. Nijholt, A., 2020. Virtual and Augmented Reality Animals in Smart and Playful Cities (Invited Paper), 2020 IEEE Joint 9th International Conference on Informatics, Electronics & Vision (ICIEV) and 2020 4th International Conference on Imaging, Vision & Pattern Recognition (ICIVPR), Kitakyushu, Japan, 2020, 1-7.

14. Nijholt, A., 2021. Humorous and Playful Social Interactions in Augmented Reality. Contributions a la Lingüística y a la Comunicación Social. Tributo a Vitelio Ruiz Hernández. *Sixteenth International Symposium on Social Communication*, Centro de Lingüística Aplicada, Santiago de Cuba, Cuba, 167-173.

15. Petit O, Cheok AD, Spence C, Velasco C, Karunanayaka KT. Sensory marketing in light of new technologies. *In Proceedings of the 12th International Conference on Advances in Computer Entertainment Technology* 2015 Nov 16 (pp. 1-4).

16. Ranasinghe, N., Nguyen, T.N.T., Liangkun, Y., Lin, L.Y., Tolley, D. and Do, E.Y.L., 2017, October. Vocktail: A virtual cocktail for pairing digital taste, smell, and color sensations. *In Proceedings of the 25th ACM international conference on Multimedia* (pp. 1139-1147).

17. Ranasinghe, N., Jain, P., Thi Ngoc Tram, N., Koh, K.C.R., Tolley, D., Karwita, S., Lien-Ya, L., Liangkun, Y., Shamaiah, K., Eason Wai Tung, C. and Yen, C.C., 2018, April. Season traveller: Multisensory narration for enhancing the virtual reality experience. *In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems* (pp. 1-13).

18. Tache, R., Abeykoon, H.A., Karunanayaka, K.T., Kumarasinghe, J.P., Roth, G., Fernando, O.N.N. and Cheok, A.D., 2012, March. Command Center: Authoring tool to supervise augmented reality session. *In 2012 IEEE Virtual Reality Workshops (VRW)* (pp. 99-100). IEEE.

19. Thomas, T., and Holmquist, L. E., 2021, The Value of Sound within a Multisensory Approach to AR in the Arts, *In Proceedings of the Multisensory Augmented Reality Workshop, Interact 2021*. https://usehci.org/mar2021/proceedings/

20. Tobisková, N, Malmsköld, L., Pederson, T., 2021, Beyond visual guidance in semi-manual industrial assembly of wooden house sections, *In Proceedings of the Multisensory Augmented Reality Workshop, Interact 2021*. https://usehci.org/mar2021/proceedings/

21. Velasco, C., Karunanayaka, K. and Nijholt, A., 2018. Multisensory human-food interaction. *Frontiers in psychology, 9*, p.796.

22. Velasco, C., Nijholt, N., Spence, C., Narumi, T., Motoki, K., Huisman, G., and Obrist, M., Multisensory Approaches to Human-Food Interaction. *In Proceedings of the 2020 International Conference on Multimodal Interaction (ICMI '20)*. ACM, New York, NY, USA, 2020, 878-880.

23. Yamabe, T. and Nakajima, T., 2013. Playful training with augmented reality games: case studies towards reality-oriented system design. *Multimedia Tools and Applications, 62*(1), pp.259-286.

24. Zhang, E.Y., Cheok, A.D., Arsovski, S. and Muniru, I.O., 2017. Exploring the role of robotic kissing "kissenger" in digital communication through alan turing's imitation game. In *IECON 2017-43rd Annual Conference of the IEEE Industrial Electronics Society* (pp. 2870-2875). IEEE.