# Multisensory Experiences: Where the Senses Meet Technology

Marianna Obrist

<figure><img src="image_1.png" /></figure>

## To cite this version:

Marianna Obrist. Multisensory Experiences: Where the Senses Meet Technology. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.9-13, 10.1007/978-3-030-85623-6_2. hal-04329567

HAL Id: hal-04329567

https://inria.hal.science/hal-04329567v1

Submitted on 8 Dec 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Multisensory Experiences: Where the Senses meet
## Technology

Marianna Obrist¹[0000-0002-4009-1627]

¹ Department of Computer Science, University College London, Gower Street, London, WC1E
6BT, United Kingdom
m.obrist@ucl.ac.uk

**Abstract.** Multisensory experiences, that is, experiences that involve more than one of our senses, are part of our everyday life. We often tend to take them for granted, at least when our different senses function normally (normal sight functioning) or are corrected-to-normal (using glasses). However, closer inspection to any, even the most mundane experiences, reveals the remarkable sensory world in which we live in. While we have built tools, experiences and computing systems that have played to the human advantages of hearing and sight (e.g., signage, modes of communication, visual and musical arts, theatre, cinema and media), we have long neglected the opportunities around touch, taste, or smell as interface/interaction modalities. Within this keynote I will share my vision for the future of computing/HCI and what role touch, taste, and smell can play in it.

**Keywords:** Multisensory experiences, human-computer interaction, novel interaction modalities, multisensory technology, senses, touch, taste, smell.

## 1 Introduction

"The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it" (Weiser, 1995). Extending this vision of Mark Weiser, I believe that the most profound future digital technologies are those that unite all the main senses (sight, hearing, touch, taste, smell) into compelling experiences. Here below I outline some relevant developments we can observe today that make this vision for multisensory technologies more feasible than ever, but also leave us still with many challenges and responsibilities to consider.

First, it is key to remind ourselves that our life experiences are multisensory in nature. Think about this moment. You may be reading this article while immersed in a sound atmosphere. There may be a smell in the environment, even if you are not aware of it, and you may be drinking a cup of coffee or eating something, while touching the means through which you read the article. All these different sensory inputs, but perhaps more, influence the experience that you have about reading the article. But what if we could design such multisensory arrangement to create a given, intended, experience? (Velasco & Obrist, 2021).
## 2 Defining Multisensory Experiences

Multisensory experiences are increasingly changed and enabled through technology (e.g. Lin et al. 2020), and particularly through advances in the field of HCI (e.g. Cornelio et al. 2021, Maggioni et al. 2019, 2020, Vi et al. 2020). Beyond HCI, there are many other disciplines that are contributing to the continuous growth of our understanding of the senses, multisensory experiences, and the relationship between the senses and technology (see Velasco & Obrist 2020).

To account for this growing interdisciplinary interest in multisensory research and practice and enable a dialogue across stakeholders we defined multisensory experiences as “...impressions formed by specific events, whose sensory elements have been carefully crafted by someone. For example, to create the impression of a sunflower, colours, shapes, textures, and smells are considered. The senses are placed at the centre of the formation of the impression of the sunflower, even in the absence of a real flower.” (Velasco & Obrist, 2020, p. 15).

Imagine the experience of a sunflower. Technologies such as virtual and augmented reality (VR, AR) enable the creation of an impression of a sunflower, even in the absence of a real flower. This is enabled through advances in sensory devices and interface that allow us to carefully select and combine sensory stimuli in a virtual or mixed reality environment. The sunflower example is only one of many experiences we can design for and to illustrate a multisensory experience. You can think of any other experiences you have in your everyday life such as eating an apple, watching a movie, or reading this article.

It is worth noting that whilst most of people's experiences are multisensory per se, multisensory experiences are different in that there is intentionality in them. In other words, while, say, walking in a forest or jungle (event) involves a number of sensory elements, a multisensory experience, as we put forward, is carefully designed by someone. Take, as an example, a walk in the jungle that has been designed by a landscape architect in order to evoke specific impressions in a given receiver. Hence, when one refers to multisensory experiences, the “design” part is implied. For this same reason, multisensory experiences and multisensory experience design are often used interchangeably (see further details in Velasco & Obrist, 2020, and a primer in Velasco & Obrist, 2021).

## 3 Designing Multisensory Experiences

Technology has revolutionised how we communicate, work, relax, and share experiences. In effect, technology has not only changed the way in which we experience the world around us but has become an experience in itself. Do you remember the one time you forgot your smartphone at home, the Internet connection did not work, or your computer just did not turn on and you stared at a black screen? Experiencing those moments of being “online” and “offline” can be frustrating. It can disconnect us from others and at the same time remind us about what it means to be human, without extensions or augmented capabilities, in an increasingly digital world. Even events that were only partially influenced by technology until recently such as eating, have become
increasingly a playground for introducing technology to provide us with completely
new dining experiences.

How would it feel to eat your dinner wearing special glasses that make your vegetables look more appealing? How would it feel like to eat in zero gravity without the use of any plateware? Food just floats in front of you and the only thing you need is your mouth to catch the food. It may sound like science fiction, but advances in levitation technology make it possible. You could, for example, levitate a piece of pear to impress your dinner date, as Anakin Skywalker did to impress Princess Amidala in Star Wars: Attack of the Clones. Unexpected and novel experiences can become a reality through technological advances, and thus not only change the experience, but become the experience itself (see TastyFloats by Vi et al. 2017, LeviSense by Vi et al. 2020).

Now, consider how to design a multisensory experience that has no real-world
sensory representation. For example, dark matter is often discussed in the general media
but difficult to grasp for non-experts. How can we use sensory elements to design an
experience that makes the invisible visible, feelable, and much more? Together with
astrophysicists from Imperial College, we carefully crafted a multisensory experience
of Dark Matter. This experience is facilitated through the integration of multiple
technologies, including an ultrasonic mid-air haptic device to create tactile sensations
on people's hands (Obrist et al. 2013, Carter et al. 2013), a scent-delivery device to
release the smell at specific moments (Maggioni et al. 2019), a projector to create the
visual effect of the universe inside the dome, and noise-cancelling headphones to follow
the audio narrative (see details in Trotta et al. 2020). Within the field of science
communication, there are increased efforts to make those scientific concepts more
accessible through the use of technology, sensory experiences, and new ways of
storytelling (Hajas et al. 2020).

We only start to understand the design space for multisensory experiences and enabling technologies. With the continuous design and development of multisensory technologies, questions arise such as to what extent are those technologies going to change humans and our everyday lives and become an extension and augmentation of our human capabilities (physical, perceptual, and cognitive) and consequently of ourselves. The growing degree of integration between humans and technology, starting from today's mixed reality spaces, makes us wonder how technology will keep changing us humans (Mueller et al. 2020).

4 Responsible Innovation around the Senses & Technology

The excitement about multisensory experiences in academia and industry, opens a plethora of opportunities but also needs to be met with responsibility. However, there is, to date, little discussion on the implications of multisensory experiences (e.g. on our body see Brianza et al. 2019, creating illusions Pittera et al. 2019).

Considering the above introduced definition of multisensory experiences, we postulated the three laws of multisensory experiences. These laws focus on acknowledging and debating publicly different questions that are at the heart of the definition of multisensory experiences, namely, the why (the rationale/reason), what
(the impression), when (the event), how (the sensory elements), who (the someone), and whom (the receiver), associated with a given multisensory experience.

The three laws indicate (Velasco and Obrist, 2020, p. 79):

I. Multisensory experiences should be used for good and must not harm others.

II. Receivers of a multisensory experience must be treated fairly.

III. The someone and the sensory elements must be known.

The first law aims to guide the thinking process related to the question: Why and what impressions and events we want to design for? The answer to this question, should always be: Reasons, events, and impressions must not cause any harm to the receiver, nor anyone else. Multisensory experiences should be used for good. The second law aims to make people reflect about the questions: Who are we designing for? Should we design differently for different receivers? The first question helps to identify the receiver and its characteristics. The final law seeks to address two questions. First, who is crafting the multisensory experience? Second, what sensory elements we select and why? With this law we call for transparency in terms of who designs, what knowledge guides the design, and what sensory elements are chosen to craft an impression. Although it is possible that not all information may be provided upfront to the receiver, they must have easy access to such information if they want.

In summary, today is one of the best moments to design multisensory experiences in that both science and technology are evolving faster than ever and providing us with a deeper understanding of our senses as well as multisensory technologies to stimulate and extend them. Nevertheless, there are still many questions and unknown answers when looking from the present into the future. For example, the growing degree of integration between humans and technology, makes us wonder how technology will keep changing us humans and consequently also inform the design of humanoid robots and future artificial intelligent (artificial intelligence, AI) systems. While humanoid robots still have a long way to go before becoming part of everyday lives, they are surely increasingly becoming ready for it, especially with AI making progress. With it, humanoid robots will be equipped with all the main human senses: sight, hearing, touch, smell, and even taste when it will, if at all, become necessary for them to fully function. Advances in sensor technology facilitate those development.

Only time will tell how the future relationship between humans and technology will look like, although we can be assured that it will be both promising and challenging in the context of multisensory experiences.

## Acknowledgements

This work was mainly funded by the European Research Council (ERC) under the European Unions Horizon 2020 Research and Innovation Program under Grant No: 638605. I would like to thank all members of the SCHI Lab for their fantastic contributions to multisensory HCI. Special thanks go to Dr Carlos Velasco, co-author of our book on Multisensory Experiences, that provides the foundation for this keynote.

## References

1. Brianza G., Tajadura-Jiménez A., Maggioni E., Pittera D., Bianchi-Berthouze N., Obrist M. (2019). As Light as Your Scent: Effects of Smell and Sound on Body Image Perception. In:
Lamas D., Loizides F., Nacke L., Petrie H., Winckler M., Zaphiris P. (eds) Human-Computer Interaction – INTERACT 2019. INTERACT 2019. Lecture Notes in Computer Science, vol 11749. Springer, Cham.

2. Carter, T., Seah, S. A., Long, B., Drinkwater, B., and Subramanian, S. UltraHaptics: multi-point mid-air haptic feedback for touch surfaces. In Pro. *UIST'13*, ACM, (2013) 505--514.

3. Cornelio, P., Velasco, C. and Obrist, M. (2021). Multisensory Integration as per Technological Advances: A review. *Front. Neurosci.* | doi: 10.3389/fnins.2021.652611.

4. Hajas, D., Ablart, D., Schneider, O., and Obrist, M. (2020). I can feel it moving: Science Communicators Talking about the Potential of Mid-Air Haptics. In *Frontiers in Computer Science*. 20 Nov 2020, https://doi.org/10.3389/fcomp.2020.534974.

5. Lin, Y-J, Punpongsanon, P., Wen, X., Iwai, D., Sato, K., Obrist, M., Mueller, S. (2020). FoodFab: Creating Food Perception Tricks using Food 3D Printing. In *Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20)*. ACM, New York, NY, USA. 1-13. DOI: https://doi.org/10.1145/3313831.3376421

6. Maggioni, E., Cobden, R., and Obrist, M. (2019). OWidgets: A Toolkit to Enable Smell-based Experience Design. In *International Journal of Human-Computer Studies*, Volume 130, 248-260.

7. Maggioni, E., Cobden, R., Dmitrenko, D., Hornbæk, K., and Obrist, M. (2020). SMELL SPACE: Mapping out the Olfactory Design Space for Novel Interactions. ACM Trans. Comput.-Hum. Interact. 27, 5, Article 36 (October 2020), 26 pages. DOI: https://doi.org/10.1145/3402449

8. Mueller, F., Lopes, P., Strohmeier, P., Ju, W., Seim, C., Weigel, M., Nanayakkara, S., Obrist, M., Li, Z., Delfa, J., Nishida, J., Gerber, E.M., Svanaes, D., Grudin, J., Greuter, S., Kunze, K., Erickson, T., Greenspan, S., Inami, M., Marshall, J., Reiterer, H., Wolf, K., Meyer, J., Schiphorst, T., Wang, D., and Maes, P. (2020). Next Steps in Human-Computer Integration. In *Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20)*. ACM, New York, NY, USA. 1-15. DOI: https://doi.org/10.1145/3313831.3376242

9. Pittera, D., Gatti, E., and Obrist, M. (2019). I'm Sensing in the Rain: Illusion of Raindrops Through Multiple Incongruent Mid-Air Tactile Stimulation. In *Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19)*. ACM, New York, NY, USA, Paper 132, 15 pages.

10. Trotta, R., Hajas, D., Camargo-Molina, J. E., Cobden, R., Maggioni, E. and Obrist, M. (2020). 'Communicating cosmology with multisensory metaphorical experiences'. *JCOM'19* (02), N01. https://doi.org/10.22323/2.19020801.

11. Velasco, C. and Obrist, M. (2020). Multisensory Experiences: where the senses meet technology. *Oxford University Press* (30 Sept. 2020).

12. Velasco, C. and Obrist, M. (2021). Multisensory Experiences: A Primer. *Front. Comput. Sci.*, 25 March 2021 | https://doi.org/10.3389/fcomp.2021.614524.

13. Vi, T.C., Ablart, D., Gatti, E., Velasco, C., and Obrist, M. (2017). Not just seeing, but also feeling Art: Mid-air Haptic Experiences integrated in a Multisensory Art Exhibition. In *International Journal of Human-Computer Studies*, Volume 108, 1-14. https://doi.org/10.1016/j.ijhcs.2017.06.004

14. Vi, C.T., Marzo, A., Memoli, G., Maggioni, E., Ablart, D., Yeomans, M., and Obrist, M. (2020). LeviSense: A Platform for the Multisensory Integration in Levitating Food and Insights into its Effect on Flavour Perception. In *International Journal of Human-Computer Studies*. DOI: https://doi.org/10.1016/j.ijhcs.2020.102428

15. Weiser, M. (1991). The Computer for the 21st Century. *Scientific American*.